Purpose

This article will provide an example of opConfig collecting specific command output based on event characteristics.  The command output becomes embedded into the event allowing network operators to more quickly identify root cause and affect resolution.

Use Case

For this example we'll take actions if an event's stateful element is a BGP Peer.  opEvents will fire scripts that result in useful information for engineers to follow up on.

Related Pages

Configuration

opEvents

In this case the necessary configuration for opEvents is contained in /usr/local/omk/conf/EventActions.nmis.  There are two sections we will be concerned with; script and policy.

Let's have a look at the script section first.  For this example we want to ping the node that reported the problem, the BGP peer it's having an issue with, and leverage opConfig to gather related command output.  Notice the arguments passed to opconfig-cli.pl, there is a command set name; IOS_TS_BGP.  In the opConfig step we'll define what these specific router commands are.

 	'script' => {
               'ping_node' => {
                        arguments => '-c 5 node.host',
                        exec => '/bin/ping',                  
                        output => 'save'
                },
                'ping_neighbor' => {
                        arguments => '-c 5 event.element',
                        exec => '/bin/ping',
                        output => 'save'
                },
                'troubleshoot_bgp' => {
                        arguments => 'act=run_command_sets names=IOS_TS_BGP nodes=node.name print_command_output=true mthread=false debug=0 quiet=1',                    
                        exec => '/usr/local/omk/bin/opconfig-cli.pl',
                        output => 'save'
                },
	}

In the policy section we define a rule that matches stateful properties that contain 'BGP Peer'.  If there's a match it will fire the scripts defined above.

         'policy' => {
                '10' => {
                        IF => 'event.any',
                        THEN => {
                                '10' => {
                                        IF => 'event.stateful =~ "BGP Peer"',
                                        THEN => 'priority(8) AND script.ping_node() AND script.ping_neighbor() AND script.troubleshoot_bgp()',
                                        BREAK => 'false'
                                },
                        },
          }

opConfig

The /usr/local/omk/conf/command_sets.d directory is where we have files detailing exec commands to be executed on nodes.  For this example we are concerned with ios.nmis.  Our IOS_TS_BGP command set will be running the following commands.

If you have a large number of BGP routes we do not recommended 'show ip bgp'.

   'IOS_TS_BGP' => {
  
      'os_info' => {
      'version' => '/12.2|12.4|15.\d+/',
      'os' => 'IOS',
    },
      'scheduling_info' => {
      'run_commands_on_separate_connection' => 'false',
      'attempt_timeout_recovery' => 1,
    },
                
    'purging_policy' => {
      'keep_last' => 1000,
      'purge_older_than' => 2592000, # 30 days
      'autoprotect_first_revision' => 'true', 
    },    
    'commands' => [
                        {
        'multipage' => 'true',
        'privileged' => 'true',
        'command' => 'show ip bgp',
        'tags' => [ 'troubleshooting', 'routing', 'detect-change' ],
      },
                        
      {
        'multipage' => 'true',
        'privileged' => 'true',
        'command' => 'show ip bgp summary',
        'tags' => [ 'troubleshooting', 'routing', 'detect-change' ],
      },
                        
      {
        'multipage' => 'true',
        'privileged' => 'true',
        'command' => 'show ip route summary',
        'tags' => [ 'troubleshooting', 'routing', 'detect-change' ],
      },
    ],
  },

Testing and Verification

The easiest way to test this configuration is to administratively shutdown a BGP peer.  After the next NMIS collect cycle a BGP Peer Down alert will be processed by opEvents.  Here's an example from our lab.

Notice the Actions section is notifying us that our scripts fired.  Scrolling down we can view the script output; it's now embedded into the event.