| Table of Contents |
|---|
opEvents provides the Event Action Policy as a flexible mechanism for reacting to events. This document briefly describes how to configure the service, the policy language and the currently supported actions.
Action Policy Language
The action policy is configured in conf/EventActions.nmis, primarily in the section named policy. The policy consists of any number of nested if-this-then-that clauses, which specify the conditions an event must conform to and what actions to take in case of a match. Further configuration sections specific to particular actions can be present in the same file.
...
| Code Block |
|---|
'policy' => {
'1' => {
IF => 'node.configuration.customer eq "important"',
THEN => { # a sub-policy
'10' => {
IF => 'node.roleType eq "core" and event.event =~ qr{Down}',
THEN => ['log.disaster()', AND 'escalate.twentyfourseven()'], # the recommended format for specifying actions is a list
BREAK => 'true'
},
'20' => {
IF => 'node.roleType eq "distribution" and event.event =~ qr{Down}',
THEN => 'priority(+2) AND email(admin)', # do avoid this legacy single-string format!
BREAK => 'false'
},
BREAK => 'false'
},
'2' => ...
}, |
...
The IF expression is basically any arbitrary Perl expression, but tokens of the form event.name or node.name <name>, node.<name> or macro.<name> are substituted with the respective event or /node/macro property value. The special wildcards Macro properties are defined in the configuration file opCommon.nmis in the section macro.
The special wildcards event.any and node.any are replaced by a logical true value. Furthermore, tokens that match extdb.queryname.column match enrich.<extdb>.<queryname>.<column> will be substituted with the result of an external enrichment query. In opEvents 3 all Perl operators, parentheses and so on can be used in IF expressions.
If your IF expression does require text that could be misinterpreted as a substitution token (e.g. the "Nr.1" in IF => 'event.details eq "NTP Server Nr.1"'), then you should escape the dotted expression with a backslash (e.g. "NTP Server Nr\.1").
Please note that in versions before 2.2.2, any misidentified unparseable tokens were flagged as errors and were not included in the final expression to be tested.
In version 2.2 and newer, tokens of the form macro.macroname are also substituted by the value of the named macro (which can be defined in the configuration file opCommon.nmis in the section macro).
Please note that for maximum robustness you should express any regular expression in IFs as /regexp contents/ or qr{for maximum robustness you should express any regular expression in IFs as /regexp contents/ or qr{regexp contents}, NOT as "regexp contents": the doublequoted variant only works for very simple patterns.
...
The THEN clause is executed if and only if the IF expression evaluates as true (ie. non-zero, non-blank, defined). The THEN clause contains either
- a nested sub-policy,
- or a list of one or more action invocations,
- or a single string that specifies any number of action invocations separated by the token "
AND" (space, AND, space).
This legacy format should be avoided. Providing a list of actions is both more robust and faster.
The order of action invocations is relevant, but the token " AND " is just a separator: all : All given actions in a THEN clause will be executed in the given order and regardless of success or failure of prior ones.
All action invocations follow the same patterns: actionname(argument), actionname.subtype() or actionname.subtype(argument). The Please note that the empty set of parentheses must not be omitted.
Policy evaluation starts at the outermost policy level, and proceeds in order of the numeric rule identifiers. All rules on the same nesting level are evaluated one after the other, unless a successful rule has its BREAK option set to true: in this case the rules after the successful one are skipped. No BREAK option present is interpreted as BREAK is false.
...
- No policy actions are performed for events with the property
action_checkedset to 1 or for events that are (already) acknowledged.
The former can be controlled by custom parser rules, the latter is mostly affected by the configuration optionsopevents_auto_acknowledgeandopevents_auto_acknowledge_up:
With auto-acknowledge enabled, a stateful down event is automatically acknowledged when the corresponding up event arrives. In that case, the up event itself is also automatically acknowledged if and only ifopevents_auto_acknowledge_upis set. - If the configuration option
opevents_no_action_on_flapis set to true inconf/opCommon.nmis, then no actions are performed on the down event that is involved in a flap event, and the down event is acknowledged. This is the default behaviour. - Policy action handling is delayed by
state_flap_windowseconds for all stateful events, so that state flaps can be detected before any actions are performed. - Policy action handling is delayed for synthetic events, if the event creation rule sets the property
delayedaction.
Supported Policy Actions
| Action Name | Description |
|---|---|
| log.logtype() | Log the event to a file, as plain text or in JSON format |
| script.scriptname() | Execute a user-defined script, possibly capturing the output |
| escalate.policyname() | Mark this event for escalation using a particular escalation policy |
| email(contactname) | Email the event details to a particular contact |
| syslog.targetserver(prio) | Send the event as Syslog message to a Syslog server, optionally overriding the event priority |
| nmissyslog.targetserver(prio) | Send the event as Syslog message to an NMIS Syslog server, in the format expected by NMIS |
| priority(adjustment) | Change the priority of the event Adjustment can be a number between 0 and 10 for fixed assignment, or +number or -number for relative adjustment. |
| tag.tagname(value) | Set a custom event property's value for static enrichment. Tagname is the name of the property to modify and must be a single string without spaces. Values are not restricted. (In the database the custom tag will be stored as "tag_tagname", hence you cannot overwrite opEvents-internal properties with this action. As a consequence, if your policy has IFs that need a tag's value, then these need to reference the tag with the 'long form' "tag_tagname".) In opEvents 2.0.2 and newer the tagname " kb_topic" is special and controls linking to external data sources. |
| acknowledge() | Acknowledges the event in question (which stops all escalation activity for the event). Supported in opEvents 2.0.3 and newer. |
| watchdog.set(waittime) watchdog.disable() | Creates or updates a watchdog timer for the node associated with the current event. The timer is set to expire in waittime seconds from now. If the timer is not disabled or updated before the expiration time, then a synthetic event named "Watchdog Timer expired" is generated. Note that all four watchdog actions are disabled if the current event itself is a watchdog expiration event. |
| element_watchdog.set(waittime) element_watchdog.disable() | Similar to the previous, but for watchdog timers that are specific to both the node and the element (e.g. an interface) of the current event. Element watchdog timers are independent of node watchdogs and of each other: Updating or disabling an element watchdog for say, eth1 doesn't affect a timer for lo0 for the same node. |
Notes for watchdog and element_watchdog
...
The script action lets you execute a program of your choice, and optionally captures and saves that program's output with the event. As usual, the section script of conf/EventActions.nmis contains the required configuration directives:
...
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
...
|
The path to the program file must be given in the the exec option. Arguments can be passed to the program; simply add them to the arguments option. Any tokens of the form event.name or node.name will be replaced by the named event or node property, respectively. If the option output is set to save, then the output of the program execution is captured and saved with the event in question; otherwise the output is discarded.
Please Note:
- opEvents versions up to 2.0.3 do not support long-running programs in script actions, and opeventsd blocks until the action program terminates.
- From version 2.0.4 onwards,
scriptaction handling is asynchronous and parallel, and the event status gets updated whenever processing of a script action completes.
Because of the asynchronous processing your action policy does not have access to anyscript.<scriptname>event properties. - Up to version 2.0.6, script actions are excuted using the system shell.
- As a consequence you have to ensure your script
argumentsare shell-safe, ie. that spaces are escaped or suitably quoted, that quotes line up and that the arguments do not contain unescaped shell metacharacters (",',`,!, &...). - The exit code of the external program is not captured, only its output on STDOUT (and STDERR, unless the
execargument disposes of STDERR explicitely with a "2>..." construct). Argument substitution for
event.name andnode.name may need to be disabled (if your arguments need to contain a verbatim "event.sometext" string.
This can be done by escaping the "." with an (escaped) backslash. For exampleCode Block arguments => 'node.host event\\.event ...and other stuff to feed the program'
will cause the argument to contain the unsubstituted text 'event.event'. Node the use of single quotes.
- As a consequence you have to ensure your script
- Since the refresh of opEvents 2.0.6 on 2016-11-01, script actions are no longer executed using a shell, but directly by
opeventsdinstead.
This is much safer from a security perspective, and also generally faster.- It is recommended (but not required) that you change your script configuration to use the new list format for
arguments(andexec), as shown in the example above (see "future_proof").
If you use the list format, then each token is analysed for potential property substitution and then passed on to your program, separate from all other tokens.
Spaces, backticks or other shell metacharacters are thus no longer problematic in an event or node property. - You can continue using the single-string
argumentsorexec, but then opEvents will perform the necessary word-splitting and minimal amendments for backwards-compatibility only:
If yourargumentsstring contains quoted tokens like"--some_program_arg=event.event", the surrounding double (or single) quotes are stripped.
Please note that this is not performed for quotes anywhere else in your arguments string.
I.e. with an arguments string like--weird_argument=don't, the single quote will be passed through to your program as-is. - If you need to disable substitution (to pass in strings like "event.sometext" verbiatim), escape the "." with a backslash.
As a much better alternative you can also put verbatim arguments in theexeclist, because only theargumentslist is subject to substitution. - It is now possible to select whether the script exitcode should be captured and saved with the event.
This is enabled by default, unless you addexitcode=> 'false' to your script configuration. - It is now also possible to select which combination of STDOUT and STDERR output of a script should be captured and saved.
The config propertyoutputcovers STDOUT, the propertystderrSTDERR.stderrdefaults to the value ofoutput, if not given explicitely.
Adding"2>&1" to your script arguments is no longer supported. - Should you absolutely require shell features in your script action, simply use
/bin/shas theexecand set theargumentsto your liking, but
please note that this is substantially less secure than direct execution ifevent.Xornode.Ysubstitutions are involved.
- It is recommended (but not required) that you change your script configuration to use the new list format for
- opEvents version 2.2.2 and newer also supports the
max_triesparameter which determines how often a failed script action may be retried; ifmax_triesis not set, then the default value 3 is used, i.e. up to three attempts to perform the action. Please note that action failure in this context means a script exceeding the maximum configured runtime or opEvents encountering a problem with starting the script, but not a script returning a nonzero exit code.
...