Versions Compared

Old Version 9

changes.mady.by.user Christopher Gatlin

Saved on

compared with

New Version Current

changes.mady.by.user Robin Twiggs

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Purpose

Purging may not have been set up at installation time.  If the OMK database has grown large it's a good idea to purge it in 90 day increments until the desired retention period is realized.

Determine the Purge Older Than Value

In order to purge from the database in 90 day increments the date of the oldest record is required.

Code Block
[root@opmantek ~]# mongo -u opUserRW -p op42flow42 nmis --eval 'db.rawlogs.findOne()'
MongoDB shell version v3.4.9
connecting to: mongodb://127.0.0.1:27017/nmis
MongoDB server version: 3.4.9
{
        "_id" : ObjectId("5543df5db5d690352f1eb2cb"),
        "entry" : "2015-05-01T16:17:19\t10.73.225.166\tUDP: [10.73.225.166]:14445->[10.72.0.31]\tSNMPv2-MIB::sysUpTime.0=23:5:15:52.26\tSNMPv2-MIB::snmpTrapOID.0=SNMPv2-SMI::enterprises.8072.4.0.3\tSNMPv2-MIB::snmpTrapEnterprise.0=SNMPv2-SMI::enterprises.8072.4",
        "time" : NumberLong(1430511439),
        "eventid" : ObjectId("5543df5db5d690352f1eb2ca"),
        "type" : "nmis_traplog"
}

Notice the 'time' property above; 1430511439.  The date command may be used to reveal when this was.

Code Block
[root@opmantek ~]# date --date @1430511439
Fri May  1 20:17:19 UTC 2015

select an event retention period and manually purge the OMK database.

Related Documents

Purging of old data in opEvents

Determine Retention Interval

Select a retention period that provides infrastructure engineers with necessary event information and also complies with your organizations security and compliance policies. 

Run the opEvents Purge Command

The purge action is run with the opevents-cli.exe command line utility.  When using this utility the retention period is relative to the current time and is referred to as 'purge_older_than'.  The value passed with the command may be expressed in minutes, hours, or days (combinations are also possible).  Using days as an example, if the retention period were 1 year the command line option for the retention period would like like this:The purge command can be passed a value in minutes with the option 'events_purge_older_than'. 

Code Block
languagetext
### Current unix epoch time
[root@opmantek ~]# date +%s
1513316674

### Let X be The difference of current time and oldest record time converted to minutes
>>> (1513316674-1430511439)/60
1380087.25

### There are 129600 minutes in 90 days.  Subtract 129600 minutes from X.
>>> 1380087 - 129600
1250487

### Based on this the Retention period
events_purge_older_than_=365d

### Entire command
/usr/local/omk/bin/opevents-cli.exe act=purge events_purge_older_than should be 1250487

Run the opEvents Purge Command

=365d


We recommend running the purge First run the command in simulate mode first to ensure the math is correctdetermine if the desired outcome will be achieved. 

Info

For large databases this operation may take some time.


Code Block
[root@opmantekroot@poller-office ~]# /usr/local/omk/bin/opevents-cli.exe act=purge events_purge_older_than=52560365d debug=1 simulate=true
[Fri Dec 15 0522:5416:3128 2017] [debug] cli[3137711499] Purging unwanted database indices
[Fri Dec 15 0522:5416:3128 2017] [debug] cli[3137711499] DbPurging purgedisabled offor rawlogs: 0 entries (of 7161464) in rawLogs are older than 1y270d50m
Db purge of rawlogs: 0 entries (of 7161464) in rawLogs are older than 1y270d50m
Purging disabled for rawlogs
[Fri Dec 15 0522:5416:3128 2017] [debug] cli[31377] Simulated purging rawlogs: would remove 0 (of 7161464) entries from rawLogs
Simulated purging rawlogs: would remove 0 (of 7161464) entries from rawLogs
[Fri Dec 15 05:54:31 2017] [debug] cli[31377] 11499] Purging disabled for archivelogs
Purging disabled for archivelogs
[Fri Dec 15 0522:5416:3328 2017] [debug] cli[3137711499] Db purge of events: 46920440 entries (of 506091317947) in events are older than 36d12h1y
Db purge of events: 46920440 entries (of 506091317947) in events are older than 36d12h1y
[Fri Dec 15 0522:5416:3328 2017] [debug] cli[3137711499] Simulated purging events: would remove 46920440 (of 506091317947) entries from events
Simulated purging events: would remove 46920440 (of 506091317947) entries from events
[Fri Dec 15 0522:5416:3728 2017] [debug] cli[3137711499] Db purge of events: 128313340 entries (of 1380315613105) in actionlog are older than 36d12h1y
Db purge of events: 128313340 entries (of 1380315613105) in actionlog are older than 36d12h1y
[Fri Dec 15 0522:5416:3728 2017] [debug] cli[3137711499] Simulated purging events: would remove 128313340 (of 1380315613105) entries from actionlog
Simulated purging events: would remove 128313340 (of 1380315613105) entries from actionlog
[Fri Dec 15 0522:5416:3728 2017] [debug] cli[3137711499] Purging disabled for reports
Purging disabled for reports

Once satisfied that the desired action will occur; remove the debug simulate option and run the command again.

Continue this 90 day cycle until the desired retention period is attained.

Recover Disk Space - Compact MongoDB

Recover Disk Space - Compact MongoDB

The old records have been removed, but the database is still taking the same amount of storage.  In order to recover the disk space we will need to issue the MongoDB 'compact' command.  This will rewrite and defragment the database. The safe approach is to shut down all daemons that are writing the the database prior to issuing the 'compact' command.

Shutdown OMK Services

Code Block
service opeventsd stop
service opflow stop
service opconfigd stop
service nmisd stop
service omkd stop

Enter the mongo Shell and Compact opEvents Collections

Code Block
[root@opmantek ~]# mongo -u opUserRW -p op42flow42 --authenticationDatabase=admin nmisng
MongoDB shell version v3.4.9
connecting to: mongodb://127.0.0.1:27017/nmis
MongoDB server version: 3.4.9

> db.runCommand({compact: 'events'})
{ "ok" : 1 }
> db.runCommand({compact: 'rawlogs'})
{ "ok" : 1 
> db.runCommand({compact: 'actionlog'})
{ "ok" : 1 }
>

Start OMK Services

Code Block
service omkd start
service nmisd start
service opeventsd start
service opconfigd start
service opflowd start

Set Purging Policy

Now that the database has been cleaned up, set the purging policy in opCommon.  That procedure is found here: Purging of old data in opEvents


Related Topics

...