...

Using current command sets on current nmis devices

General Node management from the Command Line

The tool opnode_admin.pl lets you list, export, create, modify or delete nodes from the command line. Editing operations involve an export to a JSON file, which you then modify using an editor or other tool of choice, followed by an update of the node using that JSON file. The opnode_admin.pl tool is pretty self-explanatory and documented on this page.

Dealing with slow nodes: using Device Presets

opConfig cannot let one misbehaving device block progress for all others; therefore all interactions with remote devices are subject to a timeout (configuration item opconfig_command_timeout, default: 20 seconds). This timeout is shared by all nodes.

From version 3.0.7 onwards, opConfig  offers Device Presets: a more fine-grained mechanism for adjusting the timing for each node.
Device presets are configured in conf/opCommon.nmis, under the key opconfig_device_presets. Here is the default setting as shipped with opConfig 3.0.7:

'opconfig_device_presets' => {
  'normal' => { wake_up => 0, settling_time => 0, timeout_factor => 1 },
  'slow' => { wake_up => 2, settling_time => 5, timeout_factor => 1.25 },
},

A device preset can provide adjustments using the following three properties:

• wake_up controls whether a slow login process (that has timed out) should be 'revived' by sending an 'enter' keystroke to wake the device.
  The default is 0, or no extra 'wake up' keystrokes and if the login times out,  the connection is terminated.
  With a positive value N for wake_up, opConfig  will send a wake up keystroke, then wait for a prompt up to the configured timeout, and retry this sequence up to N times.
  If no prompt is detected during that sequence, the connection is terminated.
  Please note that the default for versions before 3.0.7 for wake_up was 1, ie. one login retry. Under certain circumstances and with devices responding excessively slowly this could lead to misinterpreted command outputs and data corruption.
• settling_time controls whether opConfig should wait an extra N seconds for stray output after the login is done but before issuing the first command.
  This setting is meant for compensating for overly slow and/or complex logins with extra banners and the like, which can cause opConfig to mis-trigger on a premature, undesirable prompt if the device in question responds overly slowly. Any output that arrives between the login and the end of the settling period is consumed and discarded. The default values is 0, or no extra compensation delay.
• timeout_factor allows the setup of a longer or shorter interaction timeout for particular devices.
  This option accepts positive values, and the configured opconfig_command_timeout is scaled by that factor for any device that uses this device preset.
  The default value is 1, i.e. no scaling. In the example above and with the default opconfig_command_timeout of 20 seconds, devices marked with the "slow" preset would get a timeout of 25 seconds.

In the GUI, assigning a device preset to a device is done on the Edit Node page, on the Connection tab.
On the command line  you can use opnode_admin.pl act=set node=<somenode> entry.connection_info.device_preset=<presetname> to select a preset for a node.

A node without preset  is subject to the default values.

Managing Command Sets

As mentioned above opConfig lets you organize whatever commands you'd like it to run into an arbitrary number of groups which we call command sets.

Command sets are stored in the single file conf/command_sets.nmis in opConfig before 2.2.4, and since 2.2.4 opConfig supports this and individual command set files in conf/command_sets.d/.

It is recommended that you use individual files (as that makes things easier to maintain and more robust, e.g. should one file have syntactical problems). For backwards compatibility the old single file is consulted first, then any individual ones. opConfig loads a command set when it's seen for the first time, and any subsequent occurrences of a clashing config set (i.e. same name) will be ignored but logged.

A command set definition consists of these types of information:

• meta-information like the command set's name (which can be used in CRON to determine when / how often this command set is run),
• optional control information like scheduling constraints and an expiration policy for the commands in that set,
• filters for selecting which devices/nodes the command set should apply to. These are discussed in more detail below.
• and finally the list of the actual commands to be run (which may also have certain modifiers and meta-data attached to them).

Here is an excerpt from the default command set that  ships with opConfig:

Managing Command Sets

As mentioned above opConfig lets you organize whatever commands you'd like it to run into an arbitrary number of groups which we call command sets.

Command sets are stored in the single file conf/command_sets.nmis in opConfig before 2.2.4, and since 2.2.4 opConfig supports this and individual command set files in conf/command_sets.d/.

It is recommended that you use individual files (as that makes things easier to maintain and more robust, e.g. should one file have syntactical problems). For backwards compatibility the old single file is consulted first, then any individual ones. opConfig loads a command set when it's seen for the first time, and any subsequent occurrences of a clashing config set (i.e. same name) will be ignored but logged.

A command set definition consists of these types of information:

• meta-information like the command set's name (which can be used in CRON to determine when / how often this command set is run),
• optional control information like scheduling constraints and an expiration policy for the commands in that set,
• filters for selecting which devices/nodes the command set should apply to. These are discussed in more detail below.
• and finally the list of the actual commands to be run (which may also have certain modifiers and meta-data attached to them).

Here is an excerpt from the default command set that  ships with opConfig:

  'IOS_HOURLY' => {
    'os_info' => {
      'version' => '/12.2|12.4|15.\d+/',
      'os' => 'IOS'
    },
        'purging_policy' => {
            'keep_last' => 1000,
            'purge_older_than' => 2592000, # 30 days
            'autoprotect_first_revision' => 'true', 
        },    
        
        'scheduling_info' => {
      'run_commands_on_separate_connection' => 'false',
    },    
    'commands' => [
      {
        'multipage' => 'true',
        'privileged' => 'false',
        'command' => 'show interfaces',
        'tags' => [ 'HOURLY', 'performance', 'interfaces' ],
      },
      {

  'IOS_HOURLY' => {
    'os_info' => {
      'version' => '/12.2|12.4|15.\d+/',
      'os' => 'IOS'
    },
        'purging_policy' => {
            'keep_lastmultipage' => 1000'true',
            'purge_older_than'privileged' => 2592000, # 30 days
    'false',
        'autoprotect_first_revisioncommand' => 'trueshow ip interface',
  
      'tags' => [ }'HOURLY', 'detect-change', 'configuration'  ],
      },
... and  
        'schedulinglots more

Defining which nodes a command set applies to

This is normally done by filtering by OS info, but not necssarily limited to OS info only:

#filter using both regular expression and string equality checks:
'os_info' => {
      'run_commands_on_separate_connection'version' => 'false/12.2|12.4|15.0/',
    },    
    'commands'os' => [
      {
        'multipage'IOS'
},
# or just the specific os family/type and that's all:
'os_info' => { 
 'os' => 'true',
        'privileged' => 'false',
        'command' => 'show interfaces',
        'tags' => [ 'HOURLY', 'performance', 'interfaces' ],
      },
      {
        'multipage' => 'true',
        'privileged' => 'false',
        'command' => 'show ip interface',
        'tags' => [ 'HOURLY', 'detect-change', 'configuration' ],
      },
... and lots more

Defining which nodes a command set applies to

This is normally done by filtering by OS info, but not necssarily limited to OS info only:

#filter using both regular expression and string equality checks:
'os_info' => {
 'version' => '/12.2|12.4|15.0/',
 'os' => 'IOS'
},
# or just the specific os family/type and that's all:
'os_info' => { 
 'os' => '/(Linux|CentOS|Ubuntu)/'
},

The examples above specify that the node in question must have an os_info property set, with sub-properties  os and version in the first example and just os in the second.

Plain string values indicate string comparison, but regular expressions can be given too. A node is only considered for this particular command set if all filtering expressions match. You can filter on any node property, not just properties from os_info (but the default command sets only use os_info).

Prior to version 3.1.1, opConfig considered command sets without filter blocks as disabled; for these versions you may want a 'wildcard' filter matching anything whatsoever,  which can be achieved by adding an os_info filter block with a 'match-everything' regular expression for os, e.g 'os' => '/.*/'.

From opConfig 3.1.1 onwards a command set without filter is interpreted as to apply to all nodes without any restriction.

...

By default opConfig stores all command outputs in its MongoDB database, and only there.

In recent versions two new options let you fine-tune the behaviour and make use of on-disk storage where desired:

• Shadowing outputs on disk is controlled by the option shadow_file,
• and the choice between database storage and disk storage is made via the option store_internal.

Disk or Database?

If you set  the store_internal option to "true" or omit  it altogether, then all command outputs are stored in the database.
With this option set to false, opConfig creates a separate file on disk for each revision of the command.

The storage location for such files is configured with the config option opconfig_external_store (default /usr/local/omk/var/opconfig/external).
The actual  files are then stored in the directory <opconfig_external_store>/<node name>/<command name>/<revision>.
In version 3.1.1 and newer, an additional symbolic link latest that points to the most recent revision is maintained in that directory.

For externally stored data the visible 'command output' in the GUI is made up from the size and the SHA256 checksum of the contents, and change detection (and the GUI) uses this data instead of the (possibly binary or huge) file contents. This produces much more coarse change detection, but works with binary files. In the GUI you'll see the made up 'command output', and a button to download the actua contents.

All other opConfig capabilities work normally for externally stored data; e.g. scheduling, tags, purging of old revisions, revision creation itself and so on.

Please note that in versions before 3.1.1 store_internal is only supported for Tracking Files.

Commands or Files that cannot fit into the Database?

MongoDB doesn't support documents greater than 16 megabytes. In opConfig 3.1.1 and newer, any command output that is larger than this limit is automatically reconfigured for storage on disk, i.e. store_internal is overridden and set to "false" for just this one revision.

Both Disk and Database?

In opConfig 3.1.1 an option for shadowing command output on disk was added: if you set the property shadow_file to true (in the command's block, or in the command set's scheduling_policy section), then opConfig will store the data both in the database and also on disk (in the same place as externally stored command outputs).

Please note that shadow_file has no effect for commands that have been configured for disk storage.

Tracking Files

opConfig version 3.0.3 introduces a new capability: Arbitrary files can now be downloaded from a node (with SCP), stored and tracked by opConfig.
Here is an snippet from the example command set named file_store.nmis that ships with opConfig:

# ...other command set structure
scheduling_info => 
{
  # indicates work to be performed by and on the opConfig host
  run_local => 'true',        
},
commands => [
  {
    command => '_download_file_ scp:///var/log/secure', 
    store_internal => 'false',
    tags => [ 'detect-change', 'other', 'custom', 'tags' ], 
  },
  {
    command => '_download_file_ scp://file_in_user_homedir',
    store_internal => 'true', # is the default,
    tags => [ 'detect-change' ],
  },
],...

To instruct opConfig to track one or more files, you have to

1. set up a command set with the scheduling property run_local set to true,
2. and add a separate special _download_file_ command for every file you want  to track.
3. If the file data is binary or larger than 16 megabytes, set store_internal to false.

The run_local option indicates that all commands in this command set are to be run on the opConfig server, instead of on the node in question.

The special/pseudo-command _download_file_ requires an scp URI to be given. Note that the first two "/" characters in scp:///some/abs/path.txt belong to the URI, and the path in question is /some/abs/path.txt. In the second example above, the path is a plain relative file_in_user_homedir which means scp will look for this file in the user's home directory.

The store_internal option works as described in the previous section.

Grouping commands into sessions

By default all commands for a node will be run sequentially, one after another within the same session/connection to that device.  This can save time as connecting to the device (and authenticating) often takes longer than running all the commands one by one. 

But this is not always the case, some devices have weird shell command processors which don't take well to this kind of scripted remote control; and then there are certain commands that do run for a longer period of time (for example system statistics gatherers like vmstat or top).  These commands would cause long delays in a sequential run, and opConfig lets you adjust this behaviour to your liking.

Separate sessions for some commands

You can specify which commands should have their own separate connections. Separate connections can be opened in parallel, if opconfig-cli.pl is run with mthread=true.

If you would like to change the default behavior and have opConfig run all commands in all command set in separate session/connection, simply modify opCommon.nmis like this:

# to run each command in it's own session, change this to true
 'opconfig_run_commands_on_separate_connection' => 'false',

It makes a lot more sense to tell opConfig to run a set of commands on their own or, in most cases, just a single command on its own.  Here is how to make a command or command set have their own sessions/connections:

# for all commands in a set, define this in the command set - note the S in run_commandS_...!
'scheduling_info' => {
 'run_commands_on_separate_connection' => 'true'
},

# for just a specific command, set this for the command in question - no S in run_command_...!
commands => [
 {
 'command' => 'show version',
 'run_command_on_separate_connection' => 'true',
 }]

Dealing with particular slow commands: Recovering from timeouts

opConfig doesn't block indefinitely if a command on a device doesn't return a response; instead the session times out after a configurable period (see config  item opconfig_command_timeout, default 20 seconds) and the command is marked as failed.

From version 3.0.7 onwards, opConfig handles timeouts in long-lived sessions robustly; earlier versions could, under certain circumstances, get confused into associating the wrong outputs with commands if a timeout had occurred earlier during that session.

Version 3.0.7 lets you decide how to react to a command timing out:

• If your command set activates the  attempt_timeout_recovery option, then opConfig  will attempt to re-synchronise the session state for a little while longer.
  It does so by looking for your device's prompt for a little while. If none is forthcoming (ie. the problematic command is still blocking progress), then opConfig will send a newline to "wake" the device. This is repeated a few times, unless a prompt is found.
• If that option is off or the resynchronisation is unsuccessful, then opConfig will close the session (and open a new one, if further commands are to be run for this node).

The value of the attempt_timeout_recovery option must be a nonnegative integer; it defines how many wait-and-wake-up cycles opConfig should perform. Each such cycle takes up to opconfig_command_timeout seconds.

You may set the attempt_timeout_recovery for all commands belonging to a set or for individual commands. Individual commands' setting override the ones for the set. The default if neither are given is zero, i.e. no extra recovery grace period.

Here is an example config set snippet, whose commands all get one extra timeout cycle for recovery, except  some_brittle_command gets up to five extra timeout periods.

...

/(Linux|CentOS|Ubuntu)/'
},

The examples above specify that the node in question must have an os_info property set, with sub-properties  os and version in the first example and just os in the second.

Plain string values indicate string comparison, but regular expressions can be given too. A node is only considered for this particular command set if all filtering expressions match. You can filter on any node property, not just properties from os_info (but the default command sets only use os_info).

Prior to version 3.1.1, opConfig considered command sets without filter blocks as disabled; for these versions you may want a 'wildcard' filter matching anything whatsoever,  which can be achieved by adding an os_info filter block with a 'match-everything' regular expression for os, e.g 'os' => '/.*/'.

From opConfig 3.1.1 onwards a command set without filter is interpreted as to apply to all nodes without any restriction.

Controlling how the command output is stored

Anchor
Shadow_File Shadow_File

By default opConfig stores all command outputs in its MongoDB database, and only there.

In recent versions two new options let you fine-tune the behaviour and make use of on-disk storage where desired:

• Shadowing outputs on disk is controlled by the option shadow_file,
• and the choice between database storage and disk storage is made via the option store_internal.

Disk or Database?

If you set  the store_internal option to "true" or omit  it altogether, then all command outputs are stored in the database.
With this option set to false, opConfig creates a separate file on disk for each revision of the command.

The storage location for such files is configured with the config option opconfig_external_store (default /usr/local/omk/var/opconfig/external).
The actual  files are then stored in the directory <opconfig_external_store>/<node name>/<command name>/<revision>.
In version 3.1.1 and newer, an additional symbolic link latest that points to the most recent revision is maintained in that directory.

For externally stored data the visible 'command output' in the GUI is made up from the size and the SHA256 checksum of the contents, and change detection (and the GUI) uses this data instead of the (possibly binary or huge) file contents. This produces much more coarse change detection, but works with binary files. In the GUI you'll see the made up 'command output', and a button to download the actua contents.

All other opConfig capabilities work normally for externally stored data; e.g. scheduling, tags, purging of old revisions, revision creation itself and so on.

Please note that in versions before 3.1.1 store_internal is only supported for Tracking Files.

Commands or Files that cannot fit into the Database?

MongoDB doesn't support documents greater than 16 megabytes. In opConfig 3.1.1 and newer, any command output that is larger than this limit is automatically reconfigured for storage on disk, i.e. store_internal is overridden and set to "false" for just this one revision.

Both Disk and Database?

In opConfig 3.1.1 an option for shadowing command output on disk was added: if you set the property shadow_file to true (in the command's block, or in the command set's scheduling_policy section), then opConfig will store the data both in the database and also on disk (in the same place as externally stored command outputs).

Please note that shadow_file has no effect for commands that have been configured for disk storage.

Tracking Files

opConfig version 3.0.3 introduces a new capability: Arbitrary files can now be downloaded from a node (with SCP), stored and tracked by opConfig.
Here is an snippet from the example command set named file_store.nmis that ships with opConfig:

# ...other command set structure
scheduling_info => 
{
  # indicates work to be performed by and on the opConfig host
  run_local => 'true',        
},
commands => [
  {
    command => '_download_file_ scp:///var/log/secure', 
    store_internal => 'false',
    tags => [ 'detect-change', 'other', 'custom', 'tags' ], 
  },
  {
    command => '_download_file_ scp://file_in_user_homedir',
    store_internal => 'true', # is the default,
    tags => [ 'detect-change' ],
  },
],...

To instruct opConfig to track one or more files, you have to

1. set up a command set with the scheduling property run_local set to true,
2. and add a separate special _download_file_ command for every file you want  to track.
3. If the file data is binary or larger than 16 megabytes, set store_internal to false.

The run_local option indicates that all commands in this command set are to be run on the opConfig server, instead of on the node in question.

The special/pseudo-command _download_file_ requires an scp URI to be given. Note that the first two "/" characters in scp:///some/abs/path.txt belong to the URI, and the path in question is /some/abs/path.txt. In the second example above, the path is a plain relative file_in_user_homedir which means scp will look for this file in the user's home directory.

The store_internal option works as described in the previous section.

Grouping commands into sessions

By default all commands for a node will be run sequentially, one after another within the same session/connection to that device.  This can save time as connecting to the device (and authenticating) often takes longer than running all the commands one by one. 

But this is not always the case, some devices have weird shell command processors which don't take well to this kind of scripted remote control; and then there are certain commands that do run for a longer period of time (for example system statistics gatherers like vmstat or top).  These commands would cause long delays in a sequential run, and opConfig lets you adjust this behaviour to your liking.

Separate sessions for some commands

You can specify which commands should have their own separate connections. Separate connections can be opened in parallel, if opconfig-cli.pl is run with mthread=true.

If you would like to change the default behavior and have opConfig run all commands in all command set in separate session/connection, simply modify opCommon.nmis like this:

# to run each command in it's own session, change this to true
 'opconfig_run_commands_on_separate_connection' => 'false',

It makes a lot more sense to tell opConfig to run a set of commands on their own or, in most cases, just a single command on its own.  Here is how to make a command or command set have their own sessions/connections:

# for all commands in a set, define this in the command set - note the S in run_commandS_...!
'scheduling_info' => {
 'run_commands_on_separate_connection' => 'true'
},

# for just a specific command, set this for the command in question - no S in run_command_...!
commands => [
 {
 'command' => 'show version',
 'run_command_on_separate_connection' => 'true',
 }]

Dealing with particular slow commands: Recovering from timeouts

opConfig doesn't block indefinitely if a command on a device doesn't return a response; instead the session times out after a configurable period (see config  item opconfig_command_timeout, default 20 seconds) and the command is marked as failed.

From version 3.0.7 onwards, opConfig handles timeouts in long-lived sessions robustly; earlier versions could, under certain circumstances, get confused into associating the wrong outputs with commands if a timeout had occurred earlier during that session.

Version 3.0.7 lets you decide how to react to a command timing out:

• If your command set activates the  attempt_timeout_recovery option, then opConfig  will attempt to re-synchronise the session state for a little while longer.
  It does so by looking for your device's prompt for a little while. If none is forthcoming (ie. the problematic command is still blocking progress), then opConfig will send a newline to "wake" the device. This is repeated a few times, unless a prompt is found.
• If that option is off or the resynchronisation is unsuccessful, then opConfig will close the session (and open a new one, if further commands are to be run for this node).

The value of the attempt_timeout_recovery option must be a nonnegative integer; it defines how many wait-and-wake-up cycles opConfig should perform. Each such cycle takes up to opconfig_command_timeout seconds.

You may set the attempt_timeout_recovery for all commands belonging to a set or for individual commands. Individual commands' setting override the ones for the set. The default if neither are given is zero, i.e. no extra recovery grace period.

Here is an example config set snippet, whose commands all get one extra timeout cycle for recovery, except  some_brittle_command gets up to five extra timeout periods.

%hash = (  
  'my_first_config_set' =>  {
    'os_info' =>  {   
	# ...node conditions
    },
    # add one extra timeout period for all commands
    'scheduling_info' => {
       'attempt_timeout_recovery' => 1,
    },
    'commands' => [
       { 'command' => "uptime" },
	   # any number of commands, treated equal wrt. timeouts
       # but the following command is more prone to timing out, so we give it up to 5 x timeout
       { 'command' => 'some_brittle_command',
          'attempt_timeout_recovery' => 5,
       },

Dealing with slow nodes: using Device Presets

opConfig cannot let one misbehaving device block progress for all others; therefore all interactions with remote devices are subject to a timeout (configuration item opconfig_command_timeout, default: 20 seconds). This timeout is shared by all nodes.

From version 3.0.7 onwards, opConfig  offers Device Presets: a more fine-grained mechanism for adjusting the timing for each node.
Device presets are configured in conf/opCommon.nmis, under the key opconfig_device_presets. Here is the default setting as shipped with opConfig 3.0.7:

'opconfig_device_presets' => {
  'normal' => { wake_up => 0, settling_time => 0, timeout_factor => 1 },
  'slow' => { wake_up => 2, settling_time => 5, timeout_factor => 1.25 },
},

A device preset can provide adjustments using the following three properties:

• wake_up controls whether a slow login process (that has timed out) should be 'revived' by sending an 'enter' keystroke to wake the device.
  The default is 0, or no extra 'wake up' keystrokes and if the login times out,  the connection is terminated.
  With a positive value N for wake_up, opConfig  will send a wake up keystroke, then wait for a prompt up to the configured timeout, and retry this sequence up to N times.
  If no prompt is detected during that sequence, the connection is terminated.
  Please note that the default for versions before 3.0.7 for wake_up was 1, ie. one login retry. Under certain circumstances and with devices responding excessively slowly this could lead to misinterpreted command outputs and data corruption.
• settling_time controls whether opConfig should wait an extra N seconds for stray output after the login is done but before issuing the first command.
  This setting is meant for compensating for overly slow and/or complex logins with extra banners and the like, which can cause opConfig to mis-trigger on a premature, undesirable prompt if the device in question responds overly slowly. Any output that arrives between the login and the end of the settling period is consumed and discarded. The default values is 0, or no extra compensation delay.
• timeout_factor allows the setup of a longer or shorter interaction timeout for particular devices.
  This option accepts positive values, and the configured opconfig_command_timeout is scaled by that factor for any device that uses this device preset.
  The default value is 1, i.e. no scaling. In the example above and with the default opconfig_command_timeout of 20 seconds, devices marked with the "slow" preset would get a timeout of 25 seconds.

In the GUI, assigning a device preset to a device is done on the Edit Node page, on the Connection tab.
On the command line  you can use opnode_admin.pl act=set node=<somenode> entry.connection_info.device_preset=<presetname> to select a preset for a node.

A node without preset  is subject to the default values.

...

Privileged Mode

Many types of devices distinguish between a normal and a privileged/superuser/elevated mode, and allow certain commands only in privileged mode. opConfig needs to know whether that applies to your device and which commands are affected.

...