Page History

...

cp install/01opmantek.conf /etc/apache2/conf.d/
service apache2 restart

...

Data

...

Storage Size 

Anchor
Considerations_for_NetFlow_Data_Storage Considerations_for_NetFlow_Data_Storage

NetFlow data can get away on you, there could be several Gigabytes of NetFlow data each hour, day or month, this is all very dependant on where you are generating netflow from, the number of active users and the types of applications they are using.  A longer discussion on this will be written up, but in summary, if you are expecting large amounts of NetFlow data, then you will want to consider storing the MongoDB database and the NetFlow data into a filesystem with 50 gigabytes or more.  During development we found 20 gigabytes was enough for our purposes, but caused problems with the flow files mainly. If you are using the Opmantek Opmantek NMIS8 Virtual Machine please  please check out our instructions on on Resizing NMIS VMs.

Size

opFlow is highly configurable and customisable, so it is easy to just have the flowd data be a seperate filesystem and update the configuration accordingly. 

'<opflow_dir>' => '/data/opflow',

mongodbpath=/data/mongodb

logfile "/data/opflow/flowd" 

Alternate Installation Directory

opFlow can be installed into another directory if required, e.g. /opt/opmantek, the same process applies, but a few files will need to be changed.

Edit opFlow.nmis and opCommon.nmis and change the <omk_base> to be the new, e.g.

uses fixed sizes for raw flows and conversations, when the setup tool (opflow_setup.pl) is run the database files will immediately be pre-allocated to the configured size and will never grow after this.  It is important that the sizes you choose fit on the partition you choose, the setup tool will warn you if they will not fit.  The default settings are: (found in /usr/local/opmantek/conf/opFlow.nmis)

'opflow_db_conversation_collection_size' => 16106127360, #15G
'opflow_db_flow_collection_size' => 5368709120 # 5G

Please adjust them appropriately before running the setup tool. (which is done later in the instructions)

The output from the setup tool may tell you to run it again after adjusting your config with force=1 (opflow_setup.pl setup=db_config force=1) # again, only if required

Location

opFlow is highly configurable and customisable, so it is easy to just have the flowd data be a seperate filesystem and update the configuration accordingly. 

'<opflow_dir>'<omk_base>' => '/optdata/opmantekopflow',

...

...

...

mongodbpath=/data/mongodb

...

logfileopmantek/htdocs/"ScriptAlias /cgi-omk/ "/usrdata/local/opmantek/cgi-bin/"

<Directory "/usr/local/opmantek/cgi-bin"> 

Install flowd NetFlow Daemon

You will need to compile the NetFlow Daemon flowd, the source code is included with opFlow.

Install Required Packages

yum install byacc   

apt-get install byacc

Compiling Flowd

cd /usr/local/opmantek/source
tar xvf flowd-0.9.1.tar.gz
cd flowd-0.9.1
./configure
make
make install
cd ../../

Install MongoDB

Please follow the instructions on the MongoDB installation page to install the latest supported version of MongoDB.

Opmantek Setup for Flowd

Now that you have the binaries for flowd we have a bunch of Opmantek goodness to make it work easily.  The following commands get this running

opflow/flowd" 

Alternate Installation Directory

opFlow can be installed into another directory if required, e.g. /opt/opmantek, the same process applies, but a few files will need to be changed.

Edit opFlow.nmis and opCommon.nmis and change the <omk_base> to be the new, e.g.

'<omk_base>' => '/opt/opmantek',

Edit the Apache include file, which if already copied to /etc/httpd/conf.d will be /etc/httpd/conf.d/01opmantek.conf and change the following lines to the new installation location

Alias /opmantek/ "/usr/local/opmantek/htdocs/"

ScriptAlias /cgi-omk/ "/usr/local/opmantek/cgi-bin/"

<Directory "/usr/local/opmantek/cgi-bin"> 

Install flowd NetFlow Daemon

You will need to compile the NetFlow Daemon flowd, the source code is included with opFlow.

Install Required Packages

yum install byacc   

apt-get install byacc

Compiling Flowd

cd

adduser _flowd
\cp /usr/local/opmantek/install/flowd.conf /usr/local/etc/flowd.conf
cp /usr/local/opmantek/install/flowd.init.d /etc/init.d/flowd
mkdir /usr/local/var
mkdir /usr/local/var/run
mkdir /var/opflow/
chkconfig flowd on
service flowd start 
 

source
tar xvf flowd-0.9.1.tar.gz
cd flowd-0.9.1
./configure
make
make install
cd ../../

Install MongoDB

Please follow the instructions on the MongoDB installation page to install the latest supported version of MongoDB.

Opmantek Setup for Flowd

Now that you have the binaries for flowd we have a bunch of Opmantek goodness to make it work easily.  The following commands get this running

adduser _flowd
\cp /usr/local/opmantek/install/flowd.conf /usr/local/etc/flowd.conf
cp /usr/local/opmantek/install/flowd.init.d /etc/init.d/flowd
mkdir /usr/local/var
mkdir /usr/local/var/run
mkdir /var/opflow/
chkconfig flowd on
service flowd start 
 

adduser: Please enter a username matching the regular expression configured
via the NAME_REGEX[_SYSTEM] configuration variable. Use the `--force-badname'
option to relax this check or reconfigure NAME_REGEX.

adduser --force 

adduser: Please enter a username matching the regular expression configured
via the NAME_REGEX[_SYSTEM] configuration variable. Use the `--force-badname'
option to relax this check or reconfigure NAME_REGEX.

adduser --force _flowd

#change this line: 
. /etc/init.d/functions
#to this: 
. /lib/lsb/init-functions

sysv-rc-conf flowd on
service flowd start

...

1. Ensure mongod is not running with the --auth switch, if you are using the mongod.init.d script included in opFlow run  run:

   Code Block
   /etc/init.d/mongod stop; /etc/init.d/mongod start_no_auth;

2. Create the user, currently the opFlow user requires access to both the admin database and it's own. Start up the mongo shell and type these commands:

   Code Block
   use admin; db.addUser('opUserRW', 'op42flow42'); // these are the defaults, change them as well as your opFlow.nmis file use nmis; // again this is a default, it doesn't need to be changed db.addUser('opUserRW', 'op42flow42'); // these are the defaults, change them as well as your opFlow.nmis file, it should match the above user command

3. Restart the mongo server with authentication, --auth auth (again, only if you are using the mongod.init.d script included in opFlow)

   Code Block
   /etc/init.d/mongod stop; /etc/init.d/mongod start;

...

cd ~
# get release and untar it
wget opFlow-release.tgz
tar -zxvf opFlow-release.tgz
# make a backup of existing installation
cp -r /usr/local/opmantek /usr/local/opmantek-DAY-MON-YEAR
# stop the existing daemon so update can happen
service opflowd stop
# update the files
cp -r ./opmantek/*  -r ./opmantek/* /usr/local/opmantek/*

opFlow database now uses capped sizes, please adjust the following config settings to make sure they fit on your partition and are the size you want: (the defaults are listed)

'opflow_db_conversation_collection_size' => 16106127360, #15G
'opflow_db_flow_collection_size' => 5368709120 # 5G

Now run the setup tool:

/usr/local/opmantek/bin/opflow_setup.pl setup=db_config

Inspect the output and be sure you have configured the sizes the way you want. If you are unsure of the current size running opflow_setup.pl setup=db_config will inform you.

When you are happy with your configuration, re-run the setup tool with force=1: (this may take some time to complete if your existing database is large)

/usr/local/opmantek/*

...

/bin/opflow_setup.pl setup=db_config force=1

Now update the configuration files run the following commands to do that.

...