Date: Fri, 29 Mar 2024 13:56:44 +0000 (UTC) Message-ID: <786794274.4151.1711720604073@skald.opmantek.com> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_4150_1522362410.1711720604073" ------=_Part_4150_1522362410.1711720604073 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html
The main concepts to bear in mind are nodes, credential sets, comman= ds, changes and revisions:
Nodes are devices/computers that opConfig knows about, and which it = is configured to run commands for.
As opConfig needs to connect= to the node in question and execute commands on that node, the node needs = to be configured with access credentials. In opConfig these are stored inde= pendent from the nodes in what opConfig calls credential sets.
= li>Commands are normally comman= d line constructs which will be executed on the node in question.
(Some are "passive commands"= like "audit-import" which are not actually run on the node but the result = is associated with node.
Commands can be grouped and collected into what opConfig calls a "co=
mmand set". Command sets are configured to apply only to particular OS and =
maybe versions or platforms.
The command output is captur= ed and stored by opConfig.
Command outputs are compared against the previous revision, an= d if different it's saved as a new revision in opConfig. It could als= o be a one-shot command which is not analyzed in great detail (e.g. a proce= ss listing or some other diagnostic command)
A command can be marked for change detection in which case more deta= iled analysis occurs for changes.
The tool opnode_admin.pl
lets you list, export, create, mod=
ify or delete nodes from the command line. Editing operations involve an ex=
port to a JSON file, which you then modify using an editor or other tool of=
choice, followed by an update of the node using that JSON file. The <=
code>opnode_admin.pl tool is pretty self-explanatory and documented on this=
page.
As mentioned above opConfig lets you organize whatever commands you'd li= ke it to run into an arbitrary number of groups which we call command sets.=
Command sets are stored in the single file conf/command_sets.nmis<=
/code> in opConfig before 2.2.4.
Since 2.2.4 opConfig supports this and individual command set f=
iles in conf/command_sets.d/
and the files are now .json =
rather than .nmis.
It is recommended that you use individual files (as that makes things ea= sier to maintain and more robust, e.g. should one file have syntactical pro= blems). For backwards compatibility the old single file is consulted first,= then any individual ones. opConfig loads a command set when it's seen= for the first time, and any subsequent occurrences of a c= lashing config set (i.e. same name) will be ignored but logged.
A command set definition consists of these types of information:
Here is an excerpt from the default command set that ships with opConfig= (in .nmis, pre 2.2.4 format).
'IOS_HOURLY' = =3D> { 'os_info' =3D> { 'version' =3D> '/12.2|12.4|15.\d+/', 'os' =3D> 'IOS' }, 'purging_policy' =3D> { 'keep_last' =3D> 1000, 'purge_older_than' =3D> 2592000, # 30 days 'autoprotect_first_revision' =3D> 'true',=20 }, =20 =20 'scheduling_info' =3D> { 'run_commands_on_separate_connection' =3D> 'false', }, =20 'commands' =3D> [ { 'multipage' =3D> 'true', 'privileged' =3D> 'false', 'command' =3D> 'show interfaces', 'tags' =3D> [ 'HOURLY', 'performance', 'interfaces' ], }, { 'multipage' =3D> 'true', 'privileged' =3D> 'false', 'command' =3D> 'show ip interface', 'tags' =3D> [ 'HOURLY', 'detect-change', 'configuration' ], }, ... and lots more
Below is much the same, but in the post 2.2.4, json format.
"IOS_HOURLY" : { "scheduling_info" : { "run_commands_on_separate_connection" : "false", "attempt_timeout_recovery" : 1 }, "purging_policy" : { "purge_older_than" : 2592000, "keep_last" : 1000, "autoprotect_first_revision" : "true" }, "commands" : [ { "command" : "show interfaces", "privileged" : "false", "tags" : [ "HOURLY", "performance", "interfaces" ], "multipage" : "true" },
This is normally done by filtering by OS info, but not necessarily limit= ed to OS info only:
# Filter using bo= th regular expression and string equality checks: 'os_info' =3D> { 'version' =3D> '/12.2|12.4|15.0/', 'os' =3D> 'IOS' }, # Or just the specific os family/type and that's all: 'os_info' =3D> {=20 'os' =3D> '/(Linux|CentOS|Ubuntu)/' },
The examples above specify that the node in question must have an os= _info property set, with sub-properties os and version in the first example and just os in the second.
Plain string values indicate string comparison, but regular expressions = can be given too. A node is only considered for this particular command set= if all filtering expressions match. You can filter on any node property, not just properties= from os_info (but the default command sets only use os_info).
Prior to version 3.1.1, opConfig considered command sets without filter =
blocks as disabled; for these versions you may want a 'wildcard' filter mat=
ching anything whatsoever, which can be achieved by adding an
e.g os
,'os' =3D> '/.*/'=
.
From opConfig 3.1.1 onwards a command set without filter is interpreted = as to apply to all nodes without any restriction.
By default opConfig stores all command outputs in its MongoDB database, = and only there.
In recent versions two new options let you fine-tune the behaviour and m= ake use of on-disk storage where desired:
shadow_file=
,store_internal
.If you set the store_internal
option to "true" o=
r omit it altogether, then all command outputs are stored in the data=
base.
With this option set to false, opConfig creates a separate file on disk for=
each revision of the command.
The storage location for such files is configured with the config option=
opconfig_external_store
(default /usr/local/omk/var/opc=
onfig/external
).
The actual files are then stored in the directory <opconfig_=
external_store>/<node name>/<command name>/<revision><=
/code>.
In version 3.1.1 and newer, an additional symbolic link latest=
code> that points to the most recent revision is maintained in that directo=
ry.
For externally stored data the visible 'command output' in the GUI is ma= de up from the size and the SHA256 checksum of the contents, and change det= ection (and the GUI) uses this data instead of the (possibly binar= y or huge) file contents. This produces much more coarse change detection, = but works with binary files. In the GUI you'll see the made up 'command out= put', and a button to download the actual contents.
All other opConfig capabilities work normally for externally stored data= ; e.g. scheduling, tags, purging of old revisions, revision creation itself= and so on.
Please note that in versions before 3.1.1 store_internal is only supported for Tracking Files.
MongoDB doesn't support documents greater than 16 megabytes. In opConfig=
3.1.1 and newer, any command output that is larger than this limit is auto=
matically reconfigured for storage on disk, i.e. store_internal<=
/code> is overridden and set to "false" for just this one revision.
In opConfig 3.1.1 an option for shadowing command output on disk was add=
ed: if you set the property shadow_file
to true (in the c=
ommand's block, or in the command set's scheduling_policy
=
section), then opConfig will store the data both in the database a=
nd also on disk (in the same place as externally stored comm=
and outputs).
Please note that shadow_file
has no effect for commands tha=
t have been configured for disk storage.
opConfig can handle Large Command Outputs, but there are some things to = take into account:
For example,= a file up to 100 MB can be obtained with a cat command, b= ut there are more efficient ways to track the file content. Using a command= output will be done with ssh, but opConfig can use scp for that purpose (S= ee below section - Tracking Files). Choosing the right pro= tocol can make a difference.
opConfig version 3.0.3 introduces a new capability: Arbitrary files can =
now be downloaded from a node (with SCP), stored and tracked by opConfig.
Here is an snippet from the example command set named file_store.nmis=
that ships with opConfig:
# ...other comman= d set structure scheduling_info =3D>=20 { # indicates work to be performed by and on the opConfig host run_local =3D> 'true', =20 }, commands =3D> [ { command =3D> '_download_file_ scp:///var/log/secure',=20 store_internal =3D> 'false', tags =3D> [ 'detect-change', 'other', 'custom', 'tags' ],=20 }, { command =3D> '_download_file_ scp://file_in_user_homedir', store_internal =3D> 'true', # is the default, tags =3D> [ 'detect-change' ], }, ],...
To instruct opConfig to track one or more files, you have to
run_local<=
/code> set to true,
_download_file_
command fo=
r every file you want to track.=
store_internal
to false.The run_local
option indicates that all commands in th=
is command set are to be run on the opConfig server, instead =
of on the node in question.
The special/pseudo-command _download_file_
requires an scp =
URI to be given. Note that the first two "/" characters in scp:=
///some/abs/path.txt
belong to the URI, and the path in question=
is /some/abs/path.txt
. In the second example above, the =
path is a plain relative file_in_user_homedir
which means scp =
will look for this file in the user's home directory.
The store_internal
option works as described in the pr=
evious section.
By default all commands for a node will be run sequentially, one after a= nother within the same session/connection to that device. This can sa= ve time as connecting to the device (and authenticating) often takes longer= than running all the commands one by one.
But this is not always the case, some devices have weird shell command p=
rocessors which don't take well to this kind of scripted remote control; an=
d then there are certain commands that do run for a longer period of time (=
for example system statistics gatherers like vmstat
or t=
op
). These commands would cause long delays in a sequential ru=
n, and opConfig lets you adjust this behaviour to your liking.
You can specify which commands should have their own separate connection=
s. Separate connections can be opened in parallel, if opconfig-cli.pl=
is run with mthread=3Dtrue
.
If you would like to change the default behavior and have opConfig run&n= bsp;all commands in all command set in separate sess= ion/connection, simply modify opCommon.nmis like this:
# to run each com= mand in it's own session, change this to true 'opconfig_run_commands_on_separate_connection' =3D> 'false',
It makes a lot more sense to tell opConfig to run a set of commands on t= heir own or, in most cases, just a single command on its own. Here is= how to make a command or command set have their own sessions/connections:<= /p>
# for all command= s in a set, define this in the command set - note the S in run_commandS_...= ! 'scheduling_info' =3D> { 'run_commands_on_separate_connection' =3D> 'true' }, # for just a specific command, set this for the command in question - no S = in run_command_...! commands =3D> [ { 'command' =3D> 'show version', 'run_command_on_separate_connection' =3D> 'true', }]
opConfig doesn't block indefinitely if a command on a device doesn't ret=
urn a response; instead the session times out after a configurable period (=
see config item opconfig_command_timeout
, default 2=
0 seconds) and the command is marked as failed.
From version 3.0.7 onwards, opConfig handles timeouts in long-lived sess= ions robustly; earlier versions could, under certain circumstances, get con= fused into associating the wrong outputs with commands if a timeout had occ= urred earlier during that session.
Version 3.0.7 lets you decide how to react to a command timing out:
attempt_timeout_recovery<=
/code> option, then opConfig will attempt to re-synchronise the sessi=
on state for a little while longer.
It does so by looking for your device's prompt for a little while. If none =
is forthcoming (ie. the problematic command is still blocking progress), th=
en opConfig will send a newline to "wake" the device. This is repeated a fe=
w times, unless a prompt is found.
The value of the attempt_timeout_recovery
option must be a =
nonnegative integer; it defines how many wait-and-wake-up cycles opConfig s=
hould perform. Each such cycle takes up to opconfig_command_time=
out
seconds.
You may set the attempt_timeout_recovery
for all comma=
nds belonging to a set or for individual commands. Individual commands' set=
ting override the ones for the set. The default if neither are given is zer=
o, i.e. no extra recovery grace period.
Here is an example config set snippet, whose commands all get one extra =
timeout cycle for recovery, except some_brittle_command gets up to five extra timeout periods.
%hash =3D ( =20 'my_first_config_set' =3D> { 'os_info' =3D> { =20 =09# ...node conditions }, # add one extra timeout period for all commands 'scheduling_info' =3D> { 'attempt_timeout_recovery' =3D> 1, }, 'commands' =3D> [ { 'command' =3D> "uptime" }, =09 # any number of commands, treated equal wrt. timeouts # but the following command is more prone to timing out, so we give = it up to 5 x timeout { 'command' =3D> 'some_brittle_command', 'attempt_timeout_recovery' =3D> 5, },
opConfig cannot let one misbehaving device block progress for all =
others; therefore all interactions with remote devices are subject to a tim=
eout (configuration item opconfig_command_timeout
, defaul=
t: 20 seconds). This timeout is shared by all nodes.
From version 3.0.7 onwards, opConfig offers Device Presets: =
a more fine-grained mechanism for adjusting the timing for each node.
Device presets are configured in conf/opCommon.nmis
, under the=
key opconfig_device_presets
. Here is the default setting as s=
hipped with opConfig 3.0.7:
'opconfig_device_= presets' =3D> { 'normal' =3D> { wake_up =3D> 0, settling_time =3D> 0, timeout_fa= ctor =3D> 1 }, 'slow' =3D> { wake_up =3D> 2, settling_time =3D> 5, timeout_fact= or =3D> 1.25 }, },
A device preset can provide adjustments using the following three = properties:
wake_up
controls whether a slow login process (that =
has timed out) should be 'revived' by sending an 'enter' keystroke to wake =
the device.wake_up
, opConfig will send =
a wake up keystroke, then wait for a prompt up to the configured timeout, a=
nd retry this sequence up to N times.wake_up was 1, ie. one login retry. Under certain circumstances and with devic=
es responding excessively slowly this could lead to misinterpreted command =
outputs and data corruption.
settling_time
controls whether opConfig should wait =
an extra N seconds for stray output after the login is done but before issu=
ing the first command.timeout_factor
allows the setup of a longer or short=
er interaction timeout for particular devices.opconfig_comm=
and_timeout
is scaled by that factor for any device that uses this d=
evice preset.opconfig_command_timeout
of 20 seconds, devices m=
arked with the "slow" preset would get a timeout of 25 seconds.In the GUI, assigning a device preset to a device is done on the E=
dit Node page, on the Connection tab.
On the command line you can use opnode_admin.pl act=3Dset =
node=3D<somenode> entry.connection_info.device_preset=3D<presetnam=
e>
to select a preset for a node.
A node without preset is subject to the default values.
Many types of devices distinguish between a normal and a privileged/supe= ruser/elevated mode, and allow certain commands only in privileged mode. op= Config needs to know whether that applies to your device and which commands= are affected.
In opConfig 3.0.2 and newer, every credential set indicates whether it g=
rants direct and permanent access to privileged mode (always_privileg=
ed
true), or whether opConfig needs to switch between normal and pri=
vileged mode (possibly repeatedly). Older versions only support the dual, m=
ode-switching setup.
Commands affected by this distinction need to be marked with the p=
rivileged
property, in which case opConfig will attempt to gain elev=
ated/superuser privileges before attempting to run the command.
When connected to a device in always privileged mode, opConfig ignores the =
privileged
property.
Before version 3.0.3 opConfig would always open an interactive session t= o the device, then issue commands one by one. Amongst other things this req= uires a working phrasebook for the type of device, and imposes certa= in constraints - but when it works it's very efficient.
Recently we've run into a few devices (types and versions), where remote= controlling the shell processor is unreliable - which can cause opCo= nfig to fail to properly communicate with the device, e.g. if the prompt ca= nnot be determined reliably.
Version 3.0.3 introduces an alternative, but for SSH only, which bypasse= s the shell/cli command processor on the device as much as possible.
You can adjust this behavior for a whole command set (or an individual c=
ommand) with the command set property run_commands_noninteractiv=
ely
. Default is false; if set to true, opConfig will issue every sin=
gle command 'blindly' and independently in a new SSH connection just for th=
is command. For transport type Telnet this property is ignored.
This option is similar to what run_commands_on_separate_connection=
does, except interactive prompts and the like are not relevant here=
: opConfig starts a new ssh client for each command, and each ssh client te=
rminates after returning the output of the command execution on the node.=
p>
opConfig version 3.1.1 has introduced a new plugin mechanism for collect= ing configuration data, which can help with devices whose command-line inte= rface is found lacking for any reason; Please consult this separate documen= t for information about how Plugins in opConfig work.
opConfig 2.2. (and newer) have a flexible purging subsystem which is described in det=
ail on a separate page here. The example above shows roughly how it's c=
ontrolled: a command or command set can have a section called p=
urging_policy
which controls whether and when a revision should be r=
emoved from the database.
Not all output is worthy of change detection; configuration information = generally is while performance information generally is not. The configurat= ion for a router, for example, is - but the usage counters on an interface = likely are not. As mentioned above, opConfig can deal with both of these ty= pes of commands, the "one-shot" ones as well as the change-indicating ones.=
In a command set you tell opConfig which commands belong to what class b=
y setting (or omitting) the tag detect-change
.
If the tag is present, then change detection is run on this command's ou= tput: If and only if the output is different from the previously stor= ed revision, a new revision is stored along with detailed information of wh= at the changes were. If the tag is not defined, then a new revision is crea= ted if the output is different - but no detailed changes are tracked, and t= he opConfig GUI will only report the command's revision in the list of "com= mands" but not in the list of "configuration changes".
'tags' =3D> ['= detect-change'],
Related to this is the question of what changes that were found should b= e considered important or relevant. opConfig can "ignore" unimportant= differences in a command's output if you provide it with a set of command = filters for a command:
'privileged' =3D&= gt; 'false', 'command' =3D> 'ifconfig -a', 'tags' =3D> [ 'DAILY', 'configuration', 'detect-change' ], 'command_filters' =3D> [ '/RX packets/', '/TX packets/', '/RX bytes/', '/TX bytes/' ]
In the example above, the output of the "ifconfig -a"
comma=
nd would be checked and any changed lines that match TX/RX packets or TX/RX=
bytes (i.e. the interface counters) are ignored. Any other change=
s that remain after applying the filters are used to figure out whether to =
create a new revision or not.
Please note that command filters are possible for both one-shot commands=
and commands with change-detect
enabled, and behave the=
same for both.
In opConfig 3.1.1, support for post-processing plugins was added which p= rovides facilities for more flexible filtering (e.g. whole blocks of data m= ay be excluded programatically). The Plugins in opConfig page describes how to make use of this = new functionality.
opConfig 2.2 and newer can raise an event with NMIS if a change was dete=
cted for a node and a particular command. The event name is "Node Configura=
tion Change Detected" and includes details of the change which has been det=
ected. To activate this feature you have to give the command in=
question both the tags detect-change
and repo=
rt-change
. You may also give a fixed event severity (values as defin=
ed in NMIS), like in this example:
'privileged' =3D&= gt; 'false', 'command' =3D> 'chkconfig', 'tags' =3D> [ 'DAILY', 'configuration', 'detect-change', 'report-change'= ], 'report_level' =3D> 'Minor',
To enable or disable this feature in general edit /usr/local/nmis8/conf/= Config.nmis.
'log_node_con= figuration_events' =3D> 'true',
If set to true the feature is enabled; if set to false the feature is di= sabled.
In this case, the Redhat/Centos command chkconfig
(=3D list=
of system services to automatically start on boot) will be checked for cha=
nges, and if any are found then a "Node Configuration Change" event with th=
e context node in question, the element "chkconfig" and the serverity "Mino=
r" will be raised in the local NMIS.
If you want a more dynamic event severity, then you can use r=
eport_level_min_changes
which selects a severity based on the number=
of changes that were found:
{ =20 'privileged' =3D> 'true', 'command' =3D> 'vgdisplay', 'tags' =3D> [ 'DAILY', 'configuration', 'detect-change', 'report-change= ' ], 'report_level_min_changes' =3D> { 1 =3D> "Normal", 3 =3D> "Minor", 10 =3D> "Major" }, }
In this example, changes in the vgdisplay
command outp=
ut would result in an event of severity Normal if there are 1 or 2 changes,=
Minor for 3 to 9 changes, and Major for 10 or more.
As seen in the examples above, commands can have a number of pre-defined=
tags like detect-change
or report-change; but you are encouraged to set your own custom tags to characterize the =
commands according to your preferences.
The default command set shipped with opConfig uses two primary tags for =
categorization, DAILY
and HOURLY
.These =
are used in the example cron invocations of "opconfig-cli.pl act=3Dru=
n_command_sets
" so that all commands with tag DAILY
are=
run only once a day, whereas the HOURLY
ones run once an hour=
. Commands with neither tag will not run periodically, for example there ar=
e numerous commands tagged with troubleshooting
which will onl=
y be run in reaction to an event detected in opEvents.
Using tags in this fashion for categorization is much more convenient th=
an having to tell opconfig-cli.pl
which command sets to r=
un by name (which is also possible).
Information on what commands are supported is printed when no options ar= e specified. All options support debug=3Dtrue for debug output and de= bug=3D9 for extremely verbose output.
Listed below are the possible options for the act=3DX
=
argument:
Info
This option is only available from versions < 4.0.0. From opConfig 4.= 0.0, the data is shared between applications.
Grabs one or more nodes from NMIS. Identical in function to "Refresh" or= "Import Nodes from NMIS" in the GUI.
Automatically Setting a Nodes opConfig= attributes
When a device is imported we automatically set a number of the devices o= pConfig attributes such as "connection_info.personality" "os_info.os" etc.<= /p>
The matching of NMIS device attributes and the subsequent setting of opC= onfig attributes is controlled via omk/conf/OS_Rules.nmis
You can extend this file to automatically set all sorts of opConfig devi= ce attributes when an import is done, e.g you might set all cisco devices t= o use ssh by setting 'connection.transport' =3D> 'SSH',
For more information see Create an OS Rule= and OS Rules Help Text
Tests a single given node (argument node=3DnodeX) and attempts to determ= ine its Transport and Credential Set configuration. The node must already h= ave a Personality set.
opconfig-cli can be used to test connections to help debug situations th= at don't make any sense.
An example of how it can be used:
bin/opconfig-cli.= pl act=3Dtest_connect host=3D192.168.88.254 transport=3DTelnet personality= =3Dios username=3Dtestuser password=3Dtestpass
The options for transport and personality are given above.
It is also possible to test an existing connection from the connections.= nmis file by specifying node=3Dnode_name , if any options are specified on = the command line along with the node they will override the settings loaded= from the connections.nmis file.
command=3D"some command" can also be specified to test the output of a s= pecific command.
This command will run all applicable command sets for all nodes (by defa= ult).
Options:
Requries node=3Dnode_name command=3D"command name" and returns output co= llected from the last run of this command
Shows the diff from the output of 2 revisions of stored output (does not= run them, only queries). The command line would look similar to get_= command_output with the edition of revision_1=3D and revision_2=3D
eg. act=3Ddiff_command_outputs node=3Dnode1 command=3D"show run" revisio= n_1=3D10 revision_2=3D13
This action imports audit information from Open-AudIT Enterprise, for=
all devices known to Open-AudIT.
This requires a licensed Version 1.4 of Open-AudIT Enterprise or newe=
r. The audit information will show up in opConfig as command "audit", and t=
he command output will be the full audit data set in JSON format. Nod=
es will be consolidated as much as possible with your currently existing no=
de definitions.
To enable this functionality, set the necessary options in <=
code>conf/opCommon.nmis and run opconfig-cli.pl act=3Dimport_a=
udit
periodically or as a one-off. The required config settings are =
as follows:
'opconfig_audit_i= mport' =3D> 1, # to enable this altogether 'opconfig_audit_import_url_base' =3D> "http://my.oae.system.com/omk/oae/= ", 'opconfig_audit_import_user' =3D> "nmis", # needs ro-access 'opconfig_audit_import_password' =3D> "somepassword",
These operations are documented on the separate page about Compliance Management= a>.