Page History

...

Controlling how the command output is stored

Anchor
Shadow_File Shadow_File

In opConfig 3.1.1 an option for shadowing command output on disk was added: if you set the property shadow_file to 1 or true (in the command's block, or in the command set's scheduling_policy section), then opConfig will store the data both in the database and also on disk, in the same location and fashion as documented in the Tracking Files section below.

Tracking Files

opConfig version 3.0.3 introduces a new capability: Arbitrary files can now be downloaded from a node (with SCP), stored and tracked by opConfig.
Here is an snippet from the example command set named file_store.nmis that ships with opConfig:

# ...other command set structure
scheduling_info => 
{
  # indicates work to be performed by and on the opConfig host
  run_local => 'true',        
},
commands => [
  {
    command => '_download_file_ scp:///var/log/secure', 
    store_internal => 'false',
    tags => [ 'detect-change', 'other', 'custom', 'tags' ], 
  },
  {
    command => '_download_file_ scp://file_in_user_homedir',
    store_internal => 'true', # is the default,
    tags => [ 'detect-change' ],
  },
],...

To instruct opConfig to track one or more files, you have to

1. set up a command set with the scheduling property run_local set to true,
2. and add a separate special _download_file_ command for every file you want  to track.
3. If you want the file data to be treated as binary, set store_internal to false.

The run_local option indicates that all commands in this command set are to be run on the opConfig server, instead of on the node in question.

The special/pseudo-command _download_file_ requires an scp URI to be given. Note that the first two "/" characters in scp:///some/abs/path.txt belong to the URI, and the path in question is /some/abs/path.txt. In the second example above, the path is a plain relative file_in_user_homedir which means scp will look for this file in the user's home directory.

If you leave the store_internal option set to true (or omit  it altogether), then the normal storage  behaviour is selected: opConfig assumes your file contents are text, treats them as the 'command output' for this special command, and stores the output in the database. Hence, you'll see the whole file contents in the GUI, and change detection will be performed line-by-line. This cannot work for binary files, and cannot work for large files (above 16 megabytes) either (16 megabytes is MongoDB's size limit for individual documents).

On the other hand with this option set to false, opConfig stores a separate copy of the the file for each revision (under the directory configured with config option opconfig_external_store, usually /usr/local/omk/var/opconfig/external). The 'command output' is made up from the size and the SHA256 checksum of the file contents, and change detection (and the GUI) uses this data instead of the (binary or huge) file contents. This produces much more coarse change detection, but works with binary files. In the GUI you'll see the made up 'command output', and a button to download the actual file data.

All other opConfig capabilities work normally for file tracking commands; e.g. scheduling, tags, purging of old revisions, revision creation itself and so on.

The resulting  files are stored in the directory /usr/local/omk/var/opconfig/external/<node name>/<command name>/<revision>. opConfig 3.1.1 and newer also maintain a symbolic link latest that points to the most recent revision.

By default opConfig stores all command outputs in its MongoDB database, and only there.

In recent versions two new options let you fine-tune the behaviour and make use of on-disk storage where desired:

• Shadowing outputs on disk is controlled by the option shadow_file,
• and the choice between database storage and disk storage is made via the option store_internal.

Disk or Database?

If you set  the store_internal option to "true" or omit  it altogether, then all command outputs are stored in the database.
With this option set to false, opConfig creates a separate file on disk for each revision of the command.

The storage location for such files is configured with the config option opconfig_external_store (default /usr/local/omk/var/opconfig/external).
The actual  files are then stored in the directory <opconfig_external_store>/<node name>/<command name>/<revision>.
In version 3.1.1 and newer, an additional symbolic link latest that points to the most recent revision is maintained in that directory.

For externally stored data the visible 'command output' in the GUI is made up from the size and the SHA256 checksum of the contents, and change detection (and the GUI) uses this data instead of the (possibly binary or huge) file contents. This produces much more coarse change detection, but works with binary files. In the GUI you'll see the made up 'command output', and a button to download the actua contents.

All other opConfig capabilities work normally for externally stored data; e.g. scheduling, tags, purging of old revisions, revision creation itself and so on.

Please note that in versions before 3.1.1 store_internal is only supported for Tracking Files.

Commands or Files that cannot fit into the Database?

MongoDB doesn't support documents greater than 16 megabytes. In opConfig 3.1.1 and newer, any command output that is larger than this limit is automatically reconfigured for storage on disk, i.e. store_internal is overridden and set to "false" for just this one revision.

Both Disk and Database?

In opConfig 3.1.1 an option for shadowing command output on disk was added: if you set the property shadow_file to true (in the command's block, or in the command set's scheduling_policy section), then opConfig will store the data both in the database and also on disk (in the same place as externally stored command outputs).

Please note that shadow_file has no effect for commands that have been configured for disk storage.

Tracking Files

opConfig version 3.0.3 introduces a new capability: Arbitrary files can now be downloaded from a node (with SCP), stored and tracked by opConfig.
Here is an snippet from the example command set named file_store.nmis that ships with opConfig:

# ...other command set structure
scheduling_info => 
{
  # indicates work to be performed by and on the opConfig host
  run_local => 'true',        
},
commands => [
  {
    command => '_download_file_ scp:///var/log/secure', 
    store_internal => 'false',
    tags => [ 'detect-change', 'other', 'custom', 'tags' ], 
  },
  {
    command => '_download_file_ scp://file_in_user_homedir',
    store_internal => 'true', # is the default,
    tags => [ 'detect-change' ],
  },
],...

To instruct opConfig to track one or more files, you have to

1. set up a command set with the scheduling property run_local set to true,
2. and add a separate special _download_file_ command for every file you want  to track.
3. If the file data is binary or larger than 16 megabytes, set store_internal to false.

The run_local option indicates that all commands in this command set are to be run on the opConfig server, instead of on the node in question.

The special/pseudo-command _download_file_ requires an scp URI to be given. Note that the first two "/" characters in scp:///some/abs/path.txt belong to the URI, and the path in question is /some/abs/path.txt. In the second example above, the path is a plain relative file_in_user_homedir which means scp will look for this file in the user's home directory.

The store_internal option works as described in the previous sectionFurthermore, in opConfig 3.1.1 and newer, any command output that is larger than 16 megabytes is automatically reconfigured for storage on disk. In these versions the configuration option store_internal is also supported for any command whatsoever, not just _download_file_<xyz>, but using the shadow_file facility is more likely to be useful for  'normal' commands.

Grouping commands into sessions

...