...

• 5 minutes to read.
• 15-30 minutes to put into production.

Methodology to create an Automation

What do we want to automate, how do we detect the condition we want to detect.  A simple analogy would be that if the doctors suspects you have a broken bone, they send you to get an x-ray, which confirms the injury or shows that the bone is not broken.  This could be referred to as a diagnostic or test.

Detection

In this case I wanted to confirm if the log4j Java library was installed on the server, unfortunately, the software does not use a Linux package manager, so we can not use RPM and APT commands.  There is a simple way to verify if the software was installed, look in / (root directory and all child directories) to see if there were any files containing the name log4j.

...

Now I want to run this command quickly and easily on 50 Linux servers.  A new command set was needed which I called "Linux_Log4j", I created a new command set file for this and similar things called "Linux_Software_Installed.nmis".

Linux_Software_Installed Command Set

Command sets in opConfig are stored in /usr/local/omk/conf/command_sets.d by default.  I copied an existing one and edited it to make it reflect what I needed, importantly this needed to have os_info matching Linux only and I needed to change the two commands, in the most recent version of opConfig for NMIS9 these files are JSON.

...

Change detection with change reporting is incredibly powerful, automated change detection to ensure compliance.

Linux_Software_Installed.nmis

The final command set looks like this:

{
   "Linux_Log4j" : {
      "commands" : [
         {
            "privileged" : "true",
            "command" : "Log4jSearch",
            "exec" : "sudo find / -name \"*log4j*\"",
            "tags" : [
               "HOURLY",
               "Linux",
               "operations",
               "detect-change",
               "report-change"
            ]
         }
      ],
      "scheduling_info" : {
         "run_commands_on_separate_connection" : "false"
      },
      "os_info" : {
         "os" : "/(Linux|CentOS|Ubuntu)/"
      }
   }
}

Running the command set

Because it is tagged with "HOURLY" the command set will run automatically every hour.  If you want to run it manually for testing, you run the following command:

Check for any errors, if all good, run manually for all nodes or wait an hour or so.

...

The /usr/local/omk/conf/opCommon.json file can be edited and the value for opconfig_command_timeout increased to a suitable number of seconds.

Running as non-privileged

You may not have (or want to use) the privileged user (using sudo). In this case, a more suitable exec string is below (and remember to set "privileged": "false").

"exec" : "find / -name \"*log4j*\" 2>/dev/null",

Diagnose

Now I can go to the opConfig GUI and find the matching nodes.

Access the Commands Overview

From the opConfig menu, select "Views → Views → Recent Commands Overview" and you should be seeing a screen which looks like the one below, first we can see how many instances of "perl versionLog4jSearch" we have collected.

In the box enter "perl versionLog4jSearch" change the select to "Command" and click "Go", you will have a list of nodes and the command name, all of these are samples we can not check for.  Step 2 is to click to click on the "Advanced" button on the right.

Image Added

Click on the Node Name to see the command output.

Image Added

And here we can see this node has some possible files of concern.

Remediation

In this case remediation requires one of the operations team to install updated versions of Log4j or the packages from vendors using it. The Opmantek development team use Vagrant to automate this kind of activity and this issue will be resolved quickly.

Conclusion

Using Operational Process Automation methodology of detect, diagnose and action, Opmantek was able to identify the servers requiring the change quickly (about 15 minutes) and then complete the remediation.