Generally speaking, the overall goal is to provide centralised logging services for the purposes of operations, compliance and audit. All systems should send log messages to the central server, where the logs will be kept in original form for the required period of time and the logs will be availble available for event, incident and problem management purposes.

...

There are several considerations for creating a centralised logging solution.

• What devices , and operating systems will be sending logs.
• What applications will be sending logs.
• What protocol will be used to send log messages.
• What timezone are each of the devices sending logs in.
• What criticality of logs is required.

...

You should complete a table as below to catalogue all the devices which will be sending logs.

Applications Sending Logs

The following applications logs need to be send centrally.

Logging Protocol

syslog has proven to be a very robust protocol for large scale log management. 

...

The requirement is to send level 6 and above.

Full details for syslog severity levels https://en.wikipedia.org/wiki/Syslog#Severity_level

...

There are several good choices for Windows, but NXLOG has proven to meet all the requirements, almost all other systems include embedded syslog systems

syslog Facility

The best reference is: https://en.wikipedia.org/wiki/Syslog#Facility

We are primarily concerned with the facilities localX facilities.  Logs will also grow at different rates and having them in separate files will allow for more granular control.  The following table summarises which log files will end up in which files.

Alternate file naming can be supported if required, e.g. cisco.log instead of local7.log.

...

local0.*        /usr/local/nmis8/logs/local0.log
local1.*        /usr/local/nmis8/logs/local1.log
local2.*        /usr/local/nmis8/logs/local2.log
local3.*        /usr/local/nmis8/logs/local3.log
local4.*        /usr/local/nmis8/logs/local4.log
local5.*        /usr/local/nmis8/logs/local5.log
local6.*        /usr/local/nmis8/logs/local6.log
local7.*        /usr/local/nmis8/logs/local7.log

Configure syslog reception for remote Linux hosts

Based on the table above the following would be the configuration

$template LinuxLogs,"%timegenerated%.%timegenerated:::date-subseconds% %HOSTNAME% %syslogtag%%msg%\n"
if      $fromhost-ip != '127.0.0.1' \
        and $syslogseverity <= '6' \
        and $syslogfacility <= '15' \
then    /usr/local/nmis8/logs/linux.log;LinuxLogs

Optionally handling things with the WORD LINUX in the tag

if      $fromhost-ip != '127.0.0.1' \
        and $syslogtag contains 'LINUX' \
        and $syslogseverity <= '6' \
        and $syslogfacility <= '15' \
then    /usr/local/nmis8/logs/linux.log;LinuxLogs

Handling Different Times and Time Zones

...

# enable the imfile module for file monitoring
$ModLoad imfile
$WorkDirectory /var/spool/rsyslog

# Monitor the file
$InputFileName /var/log/messages
$InputFileTag :
$InputFileStateFile messages_log
$InputFileSeverity error
$InputFileFacility local6
$InputFilePollInterval 1
$InputFilePersistStateInterval 1
$InputRunFileMonitor
 
# forward these logs to another server
local6.*                                @192.168.1.7:514

Remote Linux Server

The following rsyslog config will send all syslogs which are sourced localy with a severity 0-6 to the remote server.

$template LinuxMnemonic,"%timereported% %HOSTNAME% %syslogfacility-text%-%syslogseverity%-%syslogtag%%msg%\n"
if $fromhost-ip == '127.0.0.1' and $syslogseverity <= '6' then @192.168.1.7;LinuxMnemonic

Here we are using a Linux Mnemonic like the Cisco Syslog so that we do not lose the original facility and severity when the message is fowarded.

Optionally send everything with the WORD LINUX in the tag

$template LinuxMnemonic,"%timereported% %HOSTNAME% LINUX-%syslogfacility-text%-%syslogseverity%-%syslogtag%%msg%\n"

Example Topology

Image Modified 

In the example above all syslog messages received with a facility of local7 will be forwarded to the master Primary server at 10.215.1.5.  When this message is forwarded from the poller to the masterPrimary, the poller will insert its own timestamp into the message. 

### syslog message from the node to the poller server

02:23:37.250516 IP 10.10.1.1.58415 > 10.215.1.7.syslog: SYSLOG local7.notice, length: 100
E....Y.....+
..
..../...l./<189>90: *Feb  1 11:23:35.623: %SYS-5-CONFIG_I: Configured from console by hero on vty0 (10.215.1.5)
^C

### syslog message from the poller server to the masterprimary server

11:23:37.273514 IP 10.215.1.7.35902 > 10.215.1.5.syslog: SYSLOG local7.notice, length: 126
E.....@.?.#.
...
....>.....j<189>Feb  1 02:23:37 10.10.1.1 90: *Feb  1 11:23:35.623: %SYS-5-CONFIG_I: Configured from console by hero on vty0 (10.215.1.5)

...

### /usr/local/omk/config/opCommon.nmis

    'opevents_max_action_queue_age' => 3600,
    'opevents_max_event_age' => 7200,
    'opevents_max_time_delta' => 1800,
    'opevents_monthly_report_recipients' => [],
    'opevents_monthly_report_title' => 'Monthly Summary Report',

...

Setting the 'opevents_max_time_delta' to a value of 1800 seconds will instruct opEvents to do the following:

• If the syslog message has a timestamp that is more than 1800 seconds off from the current server time:
  • Accept the syslog message
  • Remove and replace the timestamp with its own time stamp.

References

https://en.wikipedia.org/wiki/Syslog#Facility

http://wiki.rsyslog.com/index.php/Sysklogd_drop-in_with_remote_logs_separated_by_dynamic_directory

http://man7.org/linux/man-pages/man5/rsyslog.conf.5.html

http://www.rsyslog.com/doc/v7-stable/configuration/property_replacer.html

http://www.rsyslog.com/doc/v7-stable/concepts/multi_ruleset.html

http://fibrevillage.com/sysadmin/221-rsyslog-rules-examples-on-linux

http://superuser.com/questions/532587/configure-rsyslog-server-to-log-incomming-messages-with-time-of-the-rsyslog-serv

http://people.redhat.com/pvrabec/rpms/rsyslog/rsyslog-example.conf

Appendix A: Upgrading rsyslog of RHEL and CentOS

...

yum install rsyslog

Related Topics

• opEvents - Syslog Handling - Adding a New Vendor

• opEvents - Syslog Handling - Adding a New Format

• SNMP Traps with Cisco and Other devices

• High Volume SNMP Trap Processing