Provide a SNMP trap handling solution that can scale to 300 traps per second.
This solution leverages snmptrapd to initially pull the trap off the wire, apply access control, translate, then forward it to rsyslog. rsyslog then puts the translated trap in a log file to be processed by opEvents. opEvents then applies filtering, parsing and actions as appropriate.
Step #1 - Configure snmptrapd to forward traps to rsyslog
Below is an example of configuring snmptrapd to send traps to rsyslog. The '-Ls' flag tells snmptrapd to send logging output to syslog. Using '-Ls2' specifies that snmptrapd will send it with the local2 facility value. The facility value is what rsyslog keys on for routing decisions. Please review the snmptrapd and snmpcmd man pages.
Verify there is not reference to a traphandle in /etc/snmp/snmptrapd.conf
Step #2 - Configure rsyslog to route traps into a specified log file
We need the traps to be placed into a specified log file that opEvents will process. The following example states that all messages with a facility of local2 will be placed in the /usr/local/nmis8/logs/snmptrap.log file. Please review the rsyslog.conf man page.
Most likely we will not want these messages to also go to /var/log/messages. We can edit /etc/rsyslog.conf to prevent this from happening. An example for facility local2 follows; notice the 'local2.none' statement.
Step #3 - Configure opEvents to process SNMP trap log file
We need to tell opEvents to process the newly created snmptrap.log file. This is done in /usr/local/omk/conf/opCommon.nmis. Be careful with this file; in reality it is a perl hash, any syntax error will render the OMK server dead. After modifying this file check it for syntax errors with the following command 'perl -c /usr/local/omk/conf/opCommon.nmis'. If you are not scared you should be
Something like the following example needs to be added to the opevents section of opCommon.nmis.
EventParserRules.nmis is where parsing generally occurs. In this case we are anticipating some complex maneuvers; so we are going to tell EventParserRules to send this to an opEvents plugin where complexity is better dealt with. Remember all that big bad syntax talk? Same applies here.
Something like the following example needs to be added to EventParserRules.nmis
Install SNMP trap parser plugin
Install an opEvents parser plugin such as: snmpTrap.pm . This perl module will be placed in /usr/local/omd/conf/parser_plugins.
Step #4 - Restart associated daemons
Restart the following daemons:
Step #5 - Verify
- Use tcpdump to observe snmptraps being recieved by the server
- Use the ps command to ensure snmptrapd, rsyslog, omkd, and opeventsd are running with the proper options
- Tail /usr/local/nmis/logs/snmptraps.log file
- Tail /usr/local/omk/log/opEvents.log
- Via the GUI; check opEvents views-> raw logs
- Via the GUI; check opEvents views -> events