NMIS8 Collection Time

 
1
0
-1

I have a single host with 8000+ interfaces and do require data from all of these interfaces every 5min.
NMIS is unable to complete the collection within 5min, it currently takes around 8-10min.

There is only this one host in NMIS, it appears collection is a single threaded process as only see one CPU core being utilised?

Is there any optimisations I can make to decrease collection time?

    CommentAdd your comment...

    3 answers

    1.  
      3
      2
      1

      Hello Ricky,

      NMIS will happily scale to handle the largest of networks.

      There are lots of things you can do to poll more devices / interfaces and make the most of the server you have.  Please take a look through the following wiki entries which discuss scale, scaling, performance and handling large numbers of interfaces.  If you are only using one CPU you likely want to increase the number of polling threads you are running which is set in your nmis cron entry.

      Scaling NMIS Polling

      Particularly have a look at varying the number of threads you use in the article above:

       /usr/local/nmis8/bin/nmis.pl type=collect mthread=true maxthreads=<MAX THREADS BASED ON YOUR BASELINE>

      Scaling NMIS polling - how NMIS handles long running processes

       

      Sometimes it is the devices themselves which are too slow to respond If this the case have a look here:

      Managing Nodes with High Interface Counts

      Let us know how you go with these and we can provide some more tips to scale things further.

      1. Nick Day

        The commentary here is now out of date as NMIS now has a more sophisticated scheduling system.

        The wiki articles linked should be up to date with new methods for configuring number of threads aetc.

      CommentAdd your comment...
    2.  
      2
      1
      0

      Ricky,

      There are some options for optimising the interface collection, are you wanting to collect interface utlisation and packet stats?  NOT collecting packet stats would increase collection time a fair bit, that is a simple modelling change.

      There are some other optimisations which would be a little more involved but could be done.  Please reach out if you would like some more details.

      Regards

      Keith

      1. Ricky Owens

        Thanks Keith, Yes we do require utilisation & packet stats. Certainly interested to hear about any other optimisations that could be done. Regards Ricky

      2. Keith Sinclair

        Ricky, The first thing you can do is to modify the model and condense the collection of SNMP data for interfaces, this will halve the total number of SNMP queries for interfaces (not really a problem for smaller interface counts). Basically you tell NMIS that when it collects the in/out octets in that same poll also collect the packet data. You should also consider precisely which SNMP variables are supported by the device, e.g. do you want the multicast/broadcast/unicast breakdown or not? Less data collected means faster. This can all be optimised with device modelling, no coding required. The next round of optimisation would require coding, the simplest option is to "pack the PDU's", so instead of collecting one interface per poll, collect multiple, depending on your end device you might find that collecting 3-5 interfaces at a time, still fits into a single IP packet. This coding isn't hard, just needs some thought on the best way to get those indexes lined up and then to pack the PDU. Another coding option which would work if you are collecting ALL interfaces on a single device, is to use SNMPBULKGET, many devices support this well, and the queries are much faster, this could be tested with NET-SNMP walks to see the difference and monitor the load on the device. This would depend on ALL interfaces being collected, as that is how the SNMP BULK feature works. What is REALLY important is to rationalise which data you want to collect, please keep in mind that MANY interfaces don't actually support ifInDiscards, ifOutDiscards, ifInErrors and ifOutErrors. Generally speaking only physical interfaces would be able to see the packet errors, so things like sub-interfaces would not have them, BUT might have 0 in the MIB. You need to look at what data you need to solve your business problem and collect that. As I said, most of that can be done with NMIS modelling. If you need some help Opmantek offers services for doing this kind of thing. Regards Keith

      CommentAdd your comment...
    3.  
      1
      0
      -1

      SAME issue ,

       

      i have a server with small baseline and aroud 200 devices .WHat is the max thread i can set for it .

      1. Nick Day

        You can have as many threads as suits you. See my other reply at the bottom.

      CommentAdd your comment...