Child pages
  • opEvents input sources

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

For the natively-supported log formats (except nmis_json_dir) only the actual parsing is hard-coded; the act of subsequent further extraction and collection of relevant details is configurable - but of course opEvents ships with a substantial set of default normalisation rules. Event normalisation consists of associating a log entry with a node, extracting details, determining whether the event is stateful or stateless, followed by optional additional enrichment from external sources.

Normalisation is for snmptraps and logs are controlled by the configuration files file  conf/EventSyslogRulesEventParserRules.nmis, EventNmisRules.nmisEventTivoliRules.nmis and EventTrapRules.nmis, all of which have a similar format. Here is an example config fragment from the syslog rules:

Code Block
'rules' => {
 '1' => {
	 event => 'Interface Down',
	 regex => qr/LINEPROTO-5-UPDOWN:.+down/,
	 stateful => 'Interface',
	 priority => 1,
 },
 ...
 '10' => {
 	 regex => qr/Interface (\w+[\d\/\.]+)/,
 	 name => 'element',
 },

The key component is the rules section, which controls what details are extracted from a log entry and how they are saved a sevent properties. There are a few ways of augmenting the event with information:

...

    1. only to the event property named by the directive variable if that directive is present (e.g. 'variable' => 'node'),
    2. or to the whole, unsplit input log entry.
  • Parser types except nmis_eventlog and nmis_slavelog: if a rule block contains a regex directive which matches, then any key-value entries for event, prioritystate or stateful in that rule block will be copied to the event as static properties.

(You might also encounter the deprecated legacy format of using directives name and value to set just one property to a fixed value.)

In the example above, rule 1 will be active if a "line protocol down" log entry is detected, and in that case it'll add properties "priority", "event", and "stateful", all with static values. Rule 10 will be active if the log entry contains "Interface <something>", and it'll copy over the matched <something> as the value of the property named "event".

All normalisation rules are checked in sequence of their numeric key, and all the ones whose regex directive matches will contribute to the new event's properties. Normalisation and enrichment then continues using information from NMIS; events are associated with the relevant nodes, stateful deduplication is performed etc.

. The next section Generic Extensible Parser documents how this works.

There are also some built-in parsers for NMIS logs  EventNmisRules.nmis, and even tivoli via EventTivoliRules.nmis which have a similar format.  Discussed in the last section.

Further enrichment can be performed using policy actions (using the tag.tagname() action), enrichment statements in correlation rules or from external databases.Please note that the log  

There is also the option to directly create Events using the Rest API which uses a similar json formats to the file format nmis_json_dir which is not subject to normalisation; instead the contents of these are expected to be normalised already.

Command-line Event Creation

To provide a simple interface for external programs, opEvents also can create an event "on the fly" with event details from command-line arguments or a JSON file.

To create an event on the fly, you have to call opeventsd.pl with the argument act=create-event, which causes it to use all further key=value pairs in the arguments to construct an event, like this example:

Code Block
opeventsd.pl act=create-event event=testevent node="somenode" details="this is just a test event" action_required=1 action_checked=0 priority=4

Your event is expected to contain all required event properties and no further normalisation is performed.  The option action_required should be set to 1 so that opEvents will process the event with Action Policies, or 0 to have opEvents not process with action policies.

Alternatively you can save your desired event's properties in a file in JSON format, and use act=create-json to instruct opeventsd to create an event from it:

Code Block
opeventsd.pl act=create-json path=./myevent_in_format.json

Generic Extensible Parser

...

If a plugin named 'PluginName' is available, its parse_enrich function is executed and if successful, the modifications made by the function replace the event properties. Parsing then continues normally.

Command-line Event Creation

To provide a simple interface for external programs, opEvents also can create an event "on the fly" with event details from command-line arguments or a JSON file.

To create an event on the fly, you have to call opeventsd.pl with the argument act=create-event, which causes it to use all further key=value pairs in the arguments to construct an event, like this example:

Code Block
opeventsd.pl act=create-event event=testevent node="somenode" details="this is just a test event" action_required=1 action_checked=0 priority=4

Your event is expected to contain all required event properties and no further normalisation is performed.  The option action_required should be set to 1 so that opEvents will process the event with Action Policies, or 0 to have opEvents not process with action policies.

Alternatively you can save your desired event's properties in a file in JSON format, and use act=create-json to instruct opeventsd to create an event from it:

Code Block
opeventsd.pl act=create-json path=./myevent_in_format.json

Built-in Parsers


Code Block
'rules' => {
 '1' => {
	 event => 'Interface Down',
	 regex => qr/LINEPROTO-5-UPDOWN:.+down/,
	 stateful => 'Interface',
	 priority => 1,
 },
 ...
 '10' => {
 	 regex => qr/Interface (\w+[\d\/\.]+)/,
 	 name => 'element',
 },

The key component is the rules section, which controls what details are extracted from a log entry and how they are saved a sevent properties. There are a few ways of augmenting the event with information:

  • if both regex and name directives are present and if the regex matches and captures something from the log entry, then a named property (with name from the name directive) will be created, with the value being the captured content.
  • The regex matching is performed on most of log input, but different across the various parsers:
    For parser typesother than nmis_eventlog and nmis_slavelog the regex is applied to the event details property, which at this point holds most of the input log entry (usually everything except node and timestamp)
    For the event log parsers, the match is applied to:
    1. only to the event property named by the directive variable if that directive is present (e.g. 'variable' => 'node'),
    2. or to the whole, unsplit input log entry.
  • Parser types except nmis_eventlog and nmis_slavelog: if a rule block contains a regex directive which matches, then any key-value entries for event, prioritystate or stateful in that rule block will be copied to the event as static properties.

(You might also encounter the deprecated legacy format of using directives name and value to set just one property to a fixed value.)

In the example above, rule 1 will be active if a "line protocol down" log entry is detected, and in that case it'll add properties "priority", "event", and "stateful", all with static values. Rule 10 will be active if the log entry contains "Interface <something>", and it'll copy over the matched <something> as the value of the property named "event".

All normalisation rules are checked in sequence of their numeric key, and all the ones whose regex directive matches will contribute to the new event's properties. Normalisation and enrichment then continues using information from NMIS; events are associated with the relevant nodes, stateful deduplication is performed etc.