Skip to end of metadata
Go to start of metadata

NMIS8 is a Network Management System which performs performance management.  NMIS8 collects SNMP data from routers, switches, firewalls, servers, and many more types of IT devices.  It stores the collected data in a performance database, which is an RRD file using RRDTool.  

This article will describe how much data NMIS8 stores and how to modify NMIS8 to store more data.

How much data does NMIS8 Store?

RRDTool is a round robin database, which stores data in a binary file which is effectively circular, you define how many elements of data you want to keep and keep feeding data and it will keep adding elements up to that many of entries, when it is full, the oldest entries are overwritten.  To keep summarised data you define a summarisation scheme, so the data is rolled up into the next level.  In NMIS8 we define how much data RRDTool keeps by using the NMIS8 modelling system.  The files for the NMIS8 models are stored in /path/to/nmis8/models and specifically the default database definitions are kept in /path/to/nmis8/models/Common-database.nmis

By default NMIS8 polls for performance data every 5 minutes and data in the following scheme:

Data SummaryDays Kept For
raw data (5 minute polls)8 days
30 minute averaged data32 days
120 minute averaged data189 days
daily averaged data1890 days (5 years)

This summarisation scheme was chosen to get a good balance of data stored and disk space used.  The file sizes created using this scheme are as follows:

RRD TypeFile Size
Interface (ifInOctets and ifOutOcts)581112 bytes
Packet data1933120 bytes
HC Packet data (new)2319408 bytes
MIB-2 IP data1739976 bytes
NMIS Health data3091984 bytes

What about statistical summarisation?

NMIS8 does not just keep the AVERAGES, it also keeps the MAXIMUM and MINIMUM for each data point, so in effect you have the range of data seen for a given summarisation, this is very important statistically, and many systems do not maintain this information.  When doing analysis of performance data, it is important to use the averages (MEAN) as well as the MAXIMUM and MINIMUM (RANGE).

Can NMIS8 Store More Performance Data?

Yes, NMIS8 can be configured to store as much performance data as you have disk for.  This is very easy to do when you first install an NMIS system, but can be done later as well. (If you are using the NMIS8 Virtual Machine please check out our instructions on Resizing NMIS VMs.) To store more data, first you need to determine how much more data you would like to store, then plug those numbers into the RRD Calculator Spreadsheet rrd_calc2.xls, attached for your convenience, this will tell you what you need to change in the NMIS8 model file  /path/to/nmis8/models/Common-database.nmis

The default Common-database.nmis file contains this:

This will need to be changed to suit your new scheme, you might like to only keep more interface data, or to change all of them to keep more data.

How would I configure NMIS8 to keep 32 days of raw data?

To configure NMIS8 to keep 32 days of raw data (5 minute) and 96 days of 30 minutes of interface data you would make the following change to the interface section.

Keep (days)Summarise (minutes)Model EntryDefault ValueNew Value
32 days5 minutesrows_day23049216
96 days30 minutesrows_week15364608

The entry for the Common-database.nmis model would look like this after the change:

This change could be made to all the entries in the Common-database.nmis file if required.

NMIS8 will now create new RRD files with the new scheme defined in the model file.  The resulting file sizes from the change above are:

RRD TypeOld File SizeNew File Size% Increase
Interface (ifInOctets and ifOutOcts)581112 bytes1391364 bytes239%
Packet data1933120 bytes4264508 bytes220%

How would I configure NMIS8 to poll every 1 minute and keep 8 days of raw data?

To configure NMIS8 to keep 8 days of raw data (with 1 minute polls) and 32 days of 30 minutes of interface data you would make the following change to all the sections.  

Keep (days)Summarise (minutes)Model EntryDefault ValueNew Value
8 days1 minutesrows_day230411520
32 days30 minutesrows_week15361536 (no change)

I can get these results using the RRD Calculator (which has been updated after ~12 years to include the NMIS8 steps/rows) this is the screen shot, I have changed the default of 5 minutes to 1 minute.

You will need to change the step and polling interface in the top of the file the defaults are hbeat (heartbeat) 900 and poll 300:

The new values from the spreadsheet are 180 and 60, so the change is:

The interface entry for the Common-database.nmis model would look like this after the change, the terms steps and rows are from RRDTool, using the year, month, etc is really a nominal name, these are the types and levels of summarisation, typically we summarise by day, week, month, year, but you can do others if needed:

This change should be made to the following sections:

  • reachability
  • interface
  • default
  • metrics

After the change to 1 min data for 8 days, the RRD files are larger, an important consideration for overall disk usage.

RRD TypeOld File SizeNew File Size% Increase
Interface (ifInOctets and ifOutOcts)581112 bytes

1244664 bytes

214%
Packet data1933120 bytes

4973616 bytes

257%

 

Can I migrate existing NMIS8 RRD files to the new scheme

Yes and no, increasing the amount of data you store is not problem, changing the polling interval to 1 minute creates a logical problem.  To change to 1 minute polling might be able to save the existing data but it will be 5 minute polling data, so the graphs would be wrong, you can delete all the existing files and start anew, and NMIS will just create them with the new values defined in the Common-database.nmis file.

To increase the data you can take existing NMIS8 RRD files and resize them using RRDTool Resize, the documentation for RRDTool resize is available on the RRDTool website http://oss.oetiker.ch/rrdtool/doc/rrdresize.en.html, in summary it would work like this.

If I wanted to change the amount of data being kept in an existing file, I would first determine the RRA Number for the entry to update, so I would run RRDTool Info on the file the output looks like this:

From all of this data I am interested in the following six lines:

These entries define the storage of the day and week summaries for RRA entries AVERAGE, MIN and MAX.

The RRDTool resize command can only resize one RRA at a time, and works on adding rows, so we need to add 6912 rows to the daily value (9216 - 2304) and add 3072 rows to the weekly value (4608 - 1536).  The following are the commands required to resize this RRD.

This takes a second or two to complete.  This process can be scripted easily enough, and has been, this is included in the NMIS8 distribution in /path/to/nmis8/admin/rrd_resize.pl, the script will determine what RRA's need to be updated and then resize them accordingly.

Where are NMIS RRD Files Stored?

NMIS stores the RRD files in the folder /path/to/nmis8/database, then in various sub-directories depending on what is required.  The script rrd_file_update.pl has the code to traverse a directory structure.  NMIS can easily be configured to store RRD files in another location, this is controlled by the NMIS config item database_root which by default is set to '<nmis_data>/database'.

Conclusion

NMIS8 is a powerful network management system which can keep as much data as you have disk space to store, the defaults make sense for most organisations and provide a good balance between statistical granularity and disk space.

 

  • No labels