This guide will go over a few settings that will help to make your Linux environment with NMIS9 more secure and stable.
Set the OMK NMIS9 VM to run services & scripts as user with omkadmin being the owner:group of /usr/local/omk
These steps will allow omkadmin as a user and as a group able to execute OMK scripts. To do this, we will follow the steps below:
To begin, we will find files and directories in /data/omk with setUID bit set
sudo find /data/omk -perm /+4000
Set Cronjobs to run as omkadmin
Be sure to set all OMK cronjobs to run as the user omkadmin and not as the root user.
General settings to ensure omkdadmin is the user running commands:
# stop services while we make this change:
sudo /usr/local/omk/bin/checkomkdaemons.sh stop;
# Add approved users, such as nmis, to omkadmin group as needed:
sudo usermod -a -G omkadmin nmis;
# this directory exists on the OMK NMIS9 VM:
OMK_DIR=/data/omk
# Set OMK directory structure writable by group:
sudo chown -R omkadmin:omkadmin "${OMK_DIR:-FAIL_HERE}";
sudo find "${OMK_DIR:-FAIL_HERE}" -type d -exec chmod 0770 {} \;;
# Set user and group able to write files:
sudo find "${OMK_DIR:-FAIL_HERE}" -type f -exec chmod 0660 {} \;;
# Set scripts executable by user and group:
# This command is purely precautionary: this directory is not likely to exist
sudo find "${OMK_DIR:-FAIL_HERE}/script" -type f -exec chmod 0770 {} \;;
# This command is purely precautionary: this directory is not likely to exist
sudo find "${OMK_DIR:-FAIL_HERE}/bin" -type f -exec chmod 0770 {} \;;
OMK_DIR=/usr/local/omk
# These command are exactly as for /data/omk:
# Set OMK directory structure writable by group:
sudo chown -R omkadmin:omkadmin "${OMK_DIR:-FAIL_HERE}";
sudo find "${OMK_DIR:-FAIL_HERE}" -type d -exec chmod 0770 {} \;;
# Set user and group able to write files:
sudo find "${OMK_DIR:-FAIL_HERE}" -type f -exec chmod 0660 {} \;;
# Set scripts executable by user and group:
# This command should succeed: this directory is likely to exist
sudo find "${OMK_DIR:-FAIL_HERE}/script" -type f -exec chmod 0770 {} \;;
# This command should succeed: this directory is likely to exist
sudo find "${OMK_DIR:-FAIL_HERE}/bin" -type f -exec chmod 0770 {} \;;
# The following commands should be executed after any of the the above commands
# to ensure PAR directory structure is re-created with PAR's own permissions set:
# Set sticky bit on $PAR_GLOBAL_TMPDIR directory and only executable by root.
# This is a more secure implementation of the linux /tmp/ directory implementation which also uses
sticky bit, but with chmod 1777:
sudo chmod 1700 "${PAR_GLOBAL_TMPDIR:-FAIL_HERE}";
# Delete existing PAR subdirectories as we may have set incorrect permissions on this directory
structure when executing the previous commands.
# The PAR subdirectories are re-created automatically by PAR upon being deleted (at execution of any PAR
script exe by that user):
sudo rm -rf "${PAR_GLOBAL_TMPDIR:-FAIL_HERE}"/*;
A good check that PAR is working as envisaged is when a normal user can execute OMK PAR exe scripts (other than those scripts that explicitly require execution by sudo with elevated permissions) and when there aren't any directories found of pattern /tmp/par-*/. (Assuming that environment variable "$PAR_GLOBAL_TMPDIR" points to an existing and user accessible directory).
After the above changes, below is the PAR directory structure after executing ./omk/bin/patch_config.exe command as nmis, root then omkadmin. Each user gets their own accessible PAR directory structure:
ls -lAth /usr/local/omk/var/lib/common/ drwx--S---. 3 omkadmin omkadmin 4.0K Jan 22 08:54 par-6f6d6b61646d696e drwx--S---. 9 root omkadmin 4.0K Jan 22 08:52 par-726f6f74 drwx--S---. 3 nmis omkadmin 4.0K Jan 22 08:50 par-6e6d6973 # and /tmp/ is okay too: sudo ls -lAth /tmp/par-* ls: cannot access /tmp/par-*: No such file or directory
Services
At this stage we have services running as root, relying on root privileges to access directory structures with user and group being omkadmin. This works, but isn't ideal, so I attempted to get the systemd services running as user and group omkadmin and with minor changes got the services started with omkadmin as user and group:
ps aux|grep omkadmin|grep -v grep|grep -v ps; root 8542 0.0 0.0 156852 5668 ? Ss 17:10 0:00 sshd: omkadmin [priv] omkadmin 8545 0.0 0.0 156852 2596 ? S 17:10 0:00 sshd: omkadmin@pts/0 omkadmin 8546 0.0 0.0 116308 2976 pts/0 Ss 17:10 0:00 -bash omkadmin 16596 0.5 3.5 744052 282988 ? Ss 18:16 0:02 opmantek.pl-webserver -f -p /var/run /omkd/opmantek.exe.pid -r omkadmin 16674 0.0 3.6 750236 291808 ? S 18:17 0:00 opmantek.pl-webserver -f -p /var/run /omkd/opmantek.exe.pid -r omkadmin 16675 0.0 3.6 749948 291368 ? S 18:17 0:00 opmantek.pl-webserver -f -p /var/run /omkd/opmantek.exe.pid -r omkadmin 16676 0.0 3.6 749100 290476 ? S 18:17 0:00 opmantek.pl-webserver -f -p /var/run /omkd/opmantek.exe.pid -r omkadmin 16677 0.0 3.6 749104 290464 ? S 18:17 0:00 opmantek.pl-webserver -f -p /var/run /omkd/opmantek.exe.pid -r omkadmin 16678 0.0 3.6 749048 290520 ? S 18:17 0:00 opmantek.pl-webserver -f -p /var/run /omkd/opmantek.exe.pid -r omkadmin 16734 0.0 2.0 516592 162804 ? Ss 18:17 0:00 opconfigd omkadmin 16735 2.0 2.2 533916 178636 ? S 18:17 0:07 opconfigd worker omkadmin 16768 0.0 1.7 397504 136556 ? Ss 18:18 0:00 opeventsd omkadmin 16769 7.8 2.2 529444 178300 ? S 18:18 0:24 opeventsd. worker omkadmin 16790 0.0 2.1 528736 174528 ? S 18:18 0:00 opeventsd.tail.tivoli. log omkadmin 16791 1.8 2.1 528868 175092 ? S 18:18 0:05 opeventsd.tail.event. log omkadmin 16792 0.0 2.1 528736 174528 ? S 18:18 0:00 opeventsd.tail.trap. log omkadmin 16794 0.0 2.1 528736 174524 ? S 18:18 0:00 opeventsd.tail.cisco. log omkadmin 16796 0.0 2.1 528736 174528 ? S 18:18 0:00 opeventsd.tail.winlogd. log omkadmin 16841 0.0 1.4 339892 112900 ? Ss 18:18 0:00 opchartsd omkadmin 16842 0.1 1.9 505236 157140 ? S 18:18 0:00 opchartsd worker
The changes needed to get this working are needed to overcome the fact that normal users, such as omkadmin, don't have write access to the /var/run/ directory for their pids:
In /usr/local/omk/conf/opCommon.json set each *_pid entry as follows (note that each service has its own pid directory under /var/run):
- "opchartsd_pid" : "/var/run/opchartsd/opchartsd.pid"
- "opconfigd_pid" : "/var/run/opconfigd/opconfigd.pid"
- "opeventsd_pid" : "/var/run/opeventsd/opeventsd.pid"
Note: that omkd service sets its pid directory as a parameter in the command that starts the service
Each service file needs slight adjustments mostly to do with each service having its own pid directory, for example for opchartsd.service under [service]:
User=omkadmin Group=omkadmin PIDFile=/var/run/opchartsd/opchartsd.pid PermissionsStartOnly=true ExecStartPre=/bin/sh -c "mkdir -p /var/run/opchartsd/" ExecStartPre=/bin/sh -c "chown omkadmin:omkadmin /var/run/opchartsd/" ExecStartPre=/bin/sh -c "chmod 0700 /var/run/opchartsd/"
'User=' line is a new entry
'Group=' line is a new entry
'PIDFile=' is edited to include the pid directory: /var/run/opchartsd/
'PermissionsStartOnly=' instructs that the service should only start with omkadmin as user and group this is necessary so the 'ExecStartPre=' lines are executed as root
'ExecStartPre=' lines are new entries
- opconfigd and opeventsd services have the same pattern of change, the difference being each services pid directory name
- omkd service has the same pattern of change, the difference again being for the pid directory name but has one extra line to edit for the path/to/pid parameter passed on the commandline to start omkd service:
- ExecStart=/usr/local/omk/script/opmantek.pl -f -p /var/run/omkd/opmantek.exe.pid -r
- There may be deeper issues where OMK services do need root privileges, for example to execute a command that will only run as root, but on the surface this has been working on ESXi instance pollux on vm named omkadmin@omk-vm9-centos7-runs-as-omkadmin-20220118 at IP 192.168.88.119 since 18 January 2022 ...
Logrotate
/etc/logrotate.d/omk-rotate.conf needs to be edited and each section of the logrotate script needs to include:
create 0660 omkadmin omkadmin su omkadmin omkadmin
Check for issues using a cronjob
We allow group nmis as user omkadmin is a member of group nmis. If we exclude group nmis and only check for NOT group omkadmin, the following are returned soon after:
sudo find /data/omk/ ! -group omkadmin ! -regex '/data/omk/var/lib/common/par-.+' -exec ls -lAh '{}' \;;
-rwxrwx---. 1 nmis nmis 21K Feb 5 23:15 /data/omk/var/opcore/registry/opConfig.json
-rwxrwx---. 1 nmis nmis 367 Feb 5 23:15 /data/omk/var/opcore/registry/open-audit.json
-rwxrwx---. 1 nmis nmis 105K Feb 5 23:15 /data/omk/var/opcore/registry/opEvents.json
-rwxrwx---. 1 nmis nmis 1.4K Feb 5 23:15 /data/omk/var/opcore/registry/opHA.json
-rwxrwx---. 1 nmis nmis 36K Feb 5 23:15 /data/omk/var/opcore/registry/opCharts.json
-rwxrwx---. 1 nmis nmis 112 Feb 5 23:14 /data/omk/var/opevents/file_state/_usr_local_nmis9_logs_event.log.json
-rwxrwx---. 1 nmis nmis 104 Feb 5 12:02 /data/omk/var/opevents/file_state/_usr_local_nmis9_logs_cisco.log.json
Create cronjob /etc/cron.d/omk_check_omkadmin_user_group:
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# m h dom mon dow user command
30 * * * * root find /data/omk ! -group omkadmin ! -group nmis ! -regex '/data/omk/var/lib/common/par-.+' -exec ls -lAh '{}' \;;
32 * * * * omkadmin find /data/omk ! -writable -exec ls -lAh '{}' \;;
34 * * * * root find /data/omk -perm /+2000 -exec ls -lAh '{}' \;;
36 * * * * root find /data/omk -perm /+4000 -exec ls -lAh '{}' \;;
40 * * * * root find /usr/local/omk ! -group omkadmin ! -group nmis ! -regex '/usr/local/omk/var/lib/common/par-.+' -exec ls -lAh '{}' \;;
42 * * * * omkadmin find /usr/local/omk ! -writable-exec ls -lAh '{}' \;;
44 * * * * root find /usr/local/omk -perm /+2000 -exec ls -lAh '{}' \;;
46 * * * * root find /usr/local/omk -perm /+4000 -exec ls -lAh '{}' \;;
Reboot the VM to start clean - starts the OMK services that were stopped earlier
sudo reboot;
Weaning this VM off of SetGID (if it was ever set - originally set in first of these tests)
SetGID may be seen as a security risk so we don't SetGID bit anymore as part of the solution. Undo SetGID on /data/omk/ and /usr/local/omk directories if it was ever set:
# stop services while we make this change:
sudo /usr/local/omk/bin/checkomkdaemons.sh stop;
# Unset SetGID on directories:
# /data/omk directory exists on the OMK NMIS9 VM:
sudo sh -c 'for F in $(find /data/omk -perm /+2000);do echo "${F}";chmod g-s "${F}";done;'
# Run this command a second time to confirm all have been fixed:
sudo sh -c 'for F in $(find /data/omk -perm /+2000);do echo "${F}";chmod g-s "${F}";done;'
# These command are exactly as for /data/omk:
sudo sh -c 'for F in $(find /usr/local/omk -perm /+2000);do echo "${F}";chmod g-s "${F}";done;'
# Run this command a second time to confirm all have been fixed:
sudo sh -c 'for F in $(find /usr/local/omk -perm /+2000);do echo "${F}";chmod g-s "${F}";done;'
# The following commands should be executed after any of the the above commands
# to ensure PAR directory structure is re-created with PAR's own permissions set:
# Set sticky bit on $PAR_GLOBAL_TMPDIR directory and only executable by root.
# This is a more secure implementation of the linux /tmp/ directory implementation which also uses sticky bit, but with chmod 1777:sudo chmod 1700 "${PAR_GLOBAL_TMPDIR:-FAIL_HERE}"
# Delete existing PAR subdirectories as we may have set incorrect permissions on this directory structure when executing the previous commands.
# The PAR subdirectories are re-created automatically by PAR upon being deleted (at execution of any PAR script exe by that user):
sudo rm -rf "${PAR_GLOBAL_TMPDIR:-FAIL_HERE}"/*
# reboot the VM to start clean - starts the OMK services that were stopped earlier:
sudo reboot;