Related Topics
Table of Contents
| Table of Contents |
|---|
Purpose
Provide a SNMP trap handling solution that can scale to 300 traps per second.
...
| Code Block |
|---|
SNMP Trap Processing - Line Diagram snmptrapd--> rsyslog--> /var/log/nmis/snmptrap.log --> opEvents --> Blacklist --> EventParserRules --> SnmpTrapParserPlugin.pm |
Deployment Steps
Step #1 - Configure snmptrapd to forward traps to
...
syslog
RHEL/Centos - Edit /etc/sysconfig/snmptrapd
Below is an example of configuring snmptrapd to send traps to rsyslog. The '-Ls' flag tells snmptrapd to send logging output to syslog. Using '-Ls2' specifies that snmptrapd will send it with the local2 facility value. The facility value is what rsyslog keys on for routing decisions. Please review the snmptrapd and snmpcmd man pages.
| Code Block | ||
|---|---|---|
| ||
OPTIONS="-n --OQ Ls2 -p /var/run/snmptrapd.pid -m ALL -M /usr/local/nmis8/mibs/traps" |
Debian - Edit /etc/default/snmptrapd
Below is an example of configuring snmptrapd to send traps to rsyslog. The '-Ls' flag tells snmptrapd to send logging output to syslog. Using '-Ls2' specifies that snmptrapd will send it with the local2 facility value. The facility value is what rsyslog keys on for routing decisions. Please review the snmptrapd and snmpcmd man pages.
| Code Block | ||
|---|---|---|
| ||
TRAPDOPTS="-n --OQ Ls2 -p /var/run/snmptrapd.pid -m ALL -M /usr/local/nmis8/mibs/traps"
/etc/systemd/system/snmptrapd.service.d/override.conf
-n -OQ -Ls2 -p /var/run/snmptrapd.pid -m ALL -M /usr/local/nmis9/mibs/traps
[Service]
ExecStart=
ExecStart=/usr/sbin/snmptrapd -n -OQ -Ls2 -p /var/run/snmptrapd.pid -m ALL -M /usr/local/nmis9/mibs/traps |
Verify /etc/snmp/snmptrapd.conf
...
Most likely we will not want these messages to also go to /var/log/messages. We can edit /etc/rsyslog.conf to prevent this from happening. An example for facility local2 follows; notice the 'local2.nonepwd
' statement.
| Code Block | ||
|---|---|---|
| ||
*.info;mail.none;authpriv.none;cron.none;local2.none /var/log/messages |
Step #3 - Configure opEvents to process SNMP trap log file using a plugin
Modify opCommon.nmis
We need to tell opEvents to process the newly created snmptrap.log file. This is done in /usr/local/omk/conf/opCommon.nmis. Be careful with this file; in reality it is a perl hash, any syntax error will render the OMK server dead. After modifying this file check it for syntax errors with the following command 'perl -c /usr/local/omk/conf/opCommon.nmis'. If you are not scared you should be 
...
| Code Block | ||
|---|---|---|
| ||
'opevents_logs' => {
'snmptraps' => [
'<nmis_logs>/snmptrap.log'
], |
In the case of opEvents 3, it needs to be adapted to json format.
| Code Block |
|---|
"opevents_logs" : {
"snmptraps" : [
"<nmis9_logs>/snmptrap.log"
], |
Modify EventParserRules.nmis
...
Install SNMP trap parser plugin
Install an opEvents parser plugin such as: snmpTrap.pm . This perl module will be placed in /usr/local/omdomk/conf/parser_plugins.
The plugin is not always needed. Traps can be processed using the event handler nmis traplog, but the plugin can parse more complex snmp traps.
Step #4 - Restart associated daemons
...
- Use tcpdump to observe snmptraps being recieved by the server
- Use the ps command to ensure snmptrapd, rsyslog, omkd, and opeventsd are running with the proper options
- Tail /usr/local/nmis/logs/snmptraps.log file
- Tail /usr/local/omk/log/opEvents.log
- Via the GUI; check opEvents views-> raw logs
- Via the GUI; check opEvents views -> events
Related Topics
Alternative Solution
A plugin is not always needed for snmp trap processing. The plugin should be necessary just when we need to process really complex traps.
Using the built in traplog parser, we would modify the Step 3 for the following:
Step #3 - Configure opEvents to process SNMP trap log file using a built in parser
Modify opCommon.nmis (abi3)/opCommon.json (abi4)
We need to tell opEvents to process the newly created snmptrap.log file. This is done in /usr/local/omk/conf/opCommon.nmis. Be careful with this file; in reality it is a perl hash, any syntax error will render the OMK server dead. After modifying this file check it for syntax errors (Just for the .nmis file) with the following command 'perl -c /usr/local/omk/conf/opCommon.nmis'. If you are not scared you should be
| Code Block |
|---|
"opevents_logs" : {
"traplog" : [
"<nmis9_logs>/snmptrap.log"
], |
Modify EventParserRules.nmis
EventParserRules.nmis is where parsing generally occurs. In this case we are anticipating some complex maneuvers; so we are going to tell EventParserRules to send this to an opEvents plugin where complexity is better dealt with. Remember all that big bad syntax talk? Same applies here.
We would need to review the trap format. Usually they look like the following:
| Code Block |
|---|
May 14 16:59:21 localhost snmptrapd[17772]: 2021-05-14 17:04:21 UDP: [127.0.0.1]:38166->[127.0.0.1]:162 [UDP: [127.0.0.1]:38166->[127.0.0.1]:162]:#012RFC1213-MIB::sysUpTime.0 = 0:0:00:00.00#011SNMPv2-MIB::snmpTrapOID.0 = BGP4-MIB::bgpBackwardTransition#011OPMANTEK-MIB::omkNotifications = "Events" |
Based on this, we will need to add the following rules to EventParserRules.nmis/EventParserRules.json, in order to be processed:
| Code Block |
|---|
"traplog" : {
"1" : {
"IF" : "SNMPv2-MIB::snmpTrapOID",
"THEN" : {
"6" : {
"THEN" : [
"capture(date)"
],
"DESCRIPTION" : "first match date/time",
"IF" : "(\\d{4}-\\d\\d-\\d\\d \\d\\d:\\d\\d:\\d\\d)"
},
"12" : {
"THEN" : [
"capture(host)"
],
"DESCRIPTION" : "host captured",
"IF" : "(\\d+\\.\\d+\\.\\d+\\.\\d+)"
},
"68" : {
"THEN" : [
"set.event(OMK Notifications)",
"set.stateful(OMK Notifications)",
"set.state(up)",
"set.priority(2)"
],
"IF" : "OPMANTEK-MIB::omkNotifications"
}
... |
We can add as many rules and captures as we need. Here you can find further information.
...
opEvents - Centralized Logging Solution
...
opEvents - Syslog Handling - Adding a New Vendor
...