Table of contents
The NOC team of a service provider needed to get very specific alerts sent to their instance of NetCool from NMIS.
This solution leverages the nocSyslog.nmis from the NMIS Events sent to IBM Tivoli Netcool using syslog solution.
- every X period of time, 5-15 minutes they want to collect the average interface utilisation for every interface being collected
- the average input and output interface utilisation will be calculated for the last X period of time 5 mins to X hours
- the highest utilisation of input or output will be selected, this is the interface utilisation
- the interface utilisation will be compared to a threshold level (should this be static or multi-level, e.g. just use the existing NMIS threshold values or other values)
- if the interface utilisation exceeds the threshold a special NOC event will be created.
- this can be the same as the existing NMIS events, but it needs to include the group name for the node
- all the regular features of NMIS interface alerts are excepted, which would include adding the interface description, bandwidth of the interface and the values of the threshold.
1388903720,NODENAME,Proactive Interface Utilisation,Critical,INTERFACENAME,INTERFACE DESCRIPTION Bandwidth=7168000: Value=93.17 Threshold=93,GROUPNAME
This event will either be sent as a syslog or logged to a file, either of which will be processed by NetCool.
An NMIS utility script which finds all the interfaces on all the nodes, performs the calculation on the interfaces and sends events to the configured syslog server.
For testing a local syslog server can be used and the facility could be local3 (configurable).
In production the syslog should be sent using TCP to ensure it arrives. The NetCool team would then process the received syslog event into NetCool.
The code for this solution is included in the NMIS9 contrib folder which is available in the installation or from NMIS9@GitHub, it will be in the folder nmis9/contrib/interface_util_alerts
A good option to install is to create a util folder e.g. /usr/local/nmis9/util and then create a symbolic link so the file will run with the correct paths, and copy the nocSyslog.nmis to the NMIS9 conf folder
Some handy commands (sudo might be required):
Update nocSyslog.nmis with your needed config, the options should be self explanatory except for extra_logging, which if enabled will give you some logging to nmis.log with when events are sent over syslog to NetCool.
To run it on a single node.
To run on all nodes, just run the script.
To put into production, set a cron.d file, e.g. /etc/cron.d/interface_util_alerts
Check the nmis.log for debug and info messages, check the configured syslog target to see the events.
Local Syslog Testing
To test locally, add the following to /etc/rsyslog.conf for testing and restart syslogd.