Netflow plugin

[DreamHunter]

Hello everyone. I have developed new plugin for cacti with name "NetFlow". I was inspired by other development - "FlowViewer". TBH: I even tried to contact with that developer, but he not answered on my requests... Last update of FlowViewer says about end of life...

So I want to present you: NetFlow plugin. You can get it on the sourceforge site. It's still beta because of i had no testers for my system. So i would be very glad your tips about it.
I've started develop it in may 2013. It's includes collector's script and web interface. Web interface can be used as standalone, but also i've implemented feature of using as cacti plugin.

Current version is 0.0.7

Link to my project: https://sourceforge.net/projects/netflow/

FreeBSD port: http://master-dl.sourceforge.net/projec ... ows.tar.gz
FreeBSD installation instructions: http://master-dl.sourceforge.net/projec ... README.txt

Centos RPM: http://master-dl.sourceforge.net/projec ... x86_64.rpm
CentOS installation instructions: http://master-dl.sourceforge.net/projec ... README.txt

Examples for Cisco Flexible Netflow configuration: http://forums.cacti.net/viewtopic.php?p=264037#p264037

Windows installation How-To: http://forums.cacti.net/viewtopic.php?p=264518#p264518

CHANGELOG:

0.0.7
27.07.2016 - Added database optimization. Less detalisation, but more performance.
27.07.2016 - Added access lists to avoid transfering from invalid sensors. (look /nflows/collector/threaded.pl -> @allowed variable that 0.0.0.0 by default)
27.07.2016 - Added scales for graphs.
27.07.2016 - Changed database structure.

0.0.6
05.05.2015 - Separated tables for each device should increase perfomance
05.05.2015 - Collector cleans old data every hour. No need to use cron - crutches
05.05.2015 - No need to create any table in database. Collector creates all required tables

0.0.5
14.10.2014 - Collector works as daemon now (freebsd)
14.10.2014 - Project can be installed as a port in freebsd (see the freebsd filefolder)
14.10.2014 - Fixed some bugs

0.0.4
07.10.2013 - Integration as plugin Cacti! (http://www.cacti.net/)
07.10.2013 - Finished developing time intervals on the web-interface
07.10.2013 - Fixed some bugs

0.0.3
09.09.2013 - Added interfaces discovering via SNMP
09.09.2013 - Improved Netflow-monitor interface
09.09.2013 - Changed database structure
09.09.2013 - Changed README file

0.0.2
20.08.2013 - Fixed chart generator (/nflows/collector/php)
20.08.2013 - Changed/Corrected README.TXT
20.08.2013 - Deprecated "mysql_" methods changed to "mysqli_"
20.08.2013 - Project moved in to "Netflow" project

0.0.1
19.08.2013 - Fixed a lot of bugs

My plans:

1) Implement Ipv6 protocol
2) Improve performance
3) Improve Access-lists
4) Improve UI.

[mcutting]

This plugin looks like it has some real potential, although I can't see any real integration with Cacti as such. By this, I mean no hooks to existing devices etc - unless I am missing something ?

Would be happy to be a tester for this.

[DreamHunter]

This plugin looks like it has some real potential, although I can't see any real integration with Cacti as such. By this, I mean no hooks to existing devices etc - unless I am missing something ?

Would be happy to be a tester for this.

Basically yes, you right. This system uses separated database and different structure of data.
But there are reasons:
1) NetFlow data structure totally different compare to SNMP data. RRDTools are not applicable here. Also there are different principles of transfering/gathering the data. Impossible to implement Cacti's pooler.
2) My project started as standalone project. I just took a first step towards the development of Cacti plugin. So I want to check the demand for it.

[DreamHunter]

added detailed guide how to install the tool. https://sourceforge.net/p/netflow/wiki/How-to-install/

[JorisFRST]

Hi,

I'm also interested.
Can this be run on a seperate box with the web end as a cacti plugin ?
Just concious of the disk/cpu impact of netflows.

[DreamHunter]

Wanna inform cacti community that my program got a user friendly ports (FreeBSD) and rpms (Centos). Also there are performance improvements.
Working on Ubuntu release.

[weday0]

Can this be installed on Windows? I have a customer with a Cacti instance on Windows so curious if this can be added to that setup as well.

Thanks,

[DreamHunter]

Can this be installed on Windows? I have a customer with a Cacti instance on Windows so curious if this can be added to that setup as well.

Thanks,

Yes you can install it in Windows. But you have to do it manually. Later i'll create some kind of "how-to".

in few words: you need install perl and additional modules to be able use the system. Of course Apache, PHP and mysql must be installed as well.

P.S. There are my e-mail address on sourceforge project page. Mail me.

[DreamHunter]

Flexible netflow config for my program:

Step1. Create a template.

Cisco ASR:

Code: Select all

!
flow record ipv4flow
 match ipv4 source address
 match ipv4 destination address
 match ipv4 protocol
 collect timestamp sys-uptime first
 collect timestamp sys-uptime last
 collect routing next-hop address ipv4
 collect interface input snmp
 collect interface output snmp
 collect counter bytes
 collect counter packets
 collect transport source-port
 collect transport destination-port
 collect transport tcp flags
 collect ipv4 tos
 collect routing source as
 collect routing destination as
 collect ipv4 source mask
 collect ipv4 destination mask
!

Cisco 65xx:

Code: Select all

!
flow record ipv4flow
 match ipv4 tos
 match ipv4 protocol
 match ipv4 source address
 match ipv4 destination address
 match transport source-port
 match transport destination-port
 collect routing source as
 collect routing destination as
 collect routing next-hop address ipv4
 collect ipv4 source mask
 collect ipv4 destination mask
 collect transport tcp flags
 collect interface input
 collect interface output
 collect counter bytes
 collect counter packets
 collect timestamp sys-uptime first
 collect timestamp sys-uptime last
!

Step 2. Create exporters:

Code: Select all

!
flow exporter ipv4exp1
 destination <COLLECTOR_IP_ADDR>
 source Loopback0
 transport udp 9999
!

Step 3. Create monitor with using exporters and template:

Code: Select all

!
flow monitor ipv4mon
 exporter ipv4exp1
 cache timeout active 60
 record ipv4flow
!

Step 4. Apply settings to interface:

Code: Select all

!
interface XXXXXXXXEthernetx/x/x
 ip flow monitor ipv4mon input
 ip flow monitor ipv4mon output
!

[DreamHunter]

Ok there... Now it's time to explain how to launch my program in Windows systems.

1) First we have to download netflow program:
Link to project page: https://sourceforge.net/projects/netflo ... =directory

1.png (100.38 KiB) Viewed 28082 times

Link to downloads page: https://sourceforge.net/projects/netflo ... rce=navbar

2.png (36.8 KiB) Viewed 28082 times

2) Unpack nflows.0.0.7.tar.gz and move unpacked files somewhere. For example into c:\

3) Second we need - perl for windows:
Link to download page: http://strawberryperl.com/

3.png (333.38 KiB) Viewed 28082 times

4) Cause of this is windows OS, we have to restart our computer.

5) If we will try to launch our program, we will get the following message:

4.png (10.97 KiB) Viewed 28077 times

6) Cause of this is a windows OS, we need to replace some lines in collector script.

Open C:\nflows.0.0.7\collector\threaded.pl in wordpad (not in notepad!!!) and replace the following lines:

Code: Select all

use Proc::Daemon;
use Proc::PID::File;

# Daemonize
if ($ARGV[0] ne "nodaemon") {
    Proc::Daemon::Init();
}

# Exit if daemon already running
if (Proc::PID::File->running()) {
print "The program is already running\n";
exit 0;
}

by this code:

Code: Select all

use Win32::Daemon;

# Tell the OS to start processing the service...
    Win32::Daemon::StartService();

# Wait until the service manager is ready for us to continue...
    while( SERVICE_START_PENDING != Win32::Daemon::State() )
    {
        sleep( 1 );
    }

    # Now let the service manager know that we are running...
    Win32::Daemon::State( SERVICE_RUNNING );

If you willtry install required modules instead, you will get the following error:

[DreamHunter]

7) Third part is installing MySQL server:

Link for download page: https://dev.mysql.com/downloads/windows/installer/

6.png (57.89 KiB) Viewed 28074 times

I advice to choose installation type - Only server. And don't forget about root password!
tip: after install I've got a bug: MySQL server tries to create pid file in the programdata directory:

Code: Select all

2017-01-15T10:50:51.723567Z 0 [ERROR] mysqld: Can't create/write to file 'C:\ProgramData\MySQL\MySQL Server 5.7\Data\noname-??.pid' (Errcode: 2 - No such file or directory)
2017-01-15T10:50:51.723567Z 0 [ERROR] Can't start server: can't create PID file: No such file or directory

So it seems you will need to add some settings to file
C:\ProgramData\MySQL\MySQL Server 5.7\my.cnf

Code: Select all

[mysqld]
tmpdir=c:/temp
pid_file=c:/temp/mysql.pid

this is only because of stupid windows. don't blame MySQL and yourself. (AND DO NOT USE NOTEPAD!!!)

8 Now we need create database for our system:
C:\>cd Program Files\MySQL\MySQL Server 5.7\bin
C:\Program Files\MySQL\MySQL Server 5.7\bin>mysql.exe -u root -p
Enter password:

Code: Select all

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 11
Server version: 5.7.17-log MySQL Community Server (GPL)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CREATE DATABASE IF NOT EXISTS `flow` CHARACTER SET utf16;

Code: Select all

Query OK, 1 row affected (0.00 sec)

mysql> USE `flow`;

Code: Select all

Database changed

mysql> CREATE USER 'netflow'@'localhost' IDENTIFIED BY PASSWORD '*993AA45E0B64915AFBD1A5BE5713FD509A8E6C2C';

Code: Select all

Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON `flow` . * TO 'netflow'@'localhost' WITH GRANT OPTION;

Code: Select all

Query OK, 0 rows affected (0.00 sec)

mysql> exit

Code: Select all

Bye

9) Now we can check our collector:

Code: Select all

C:\Program Files\MySQL\MySQL Server 5.7\bin>cd \nflows.0.0.7\collector
C:\nflows.0.0.7\collector>perl threaded.pl

Expected output:

Code: Select all

C:\nflows.0.0.7\collector>perl threaded.pl
Smartmatch is experimental at threaded.pl line 64.
Smartmatch is experimental at threaded.pl line 64.
given is experimental at threaded.pl line 66.
when is experimental at threaded.pl line 67.
when is experimental at threaded.pl line 68.
Connect to DB via socket...
Check database structure:
Table "devices"
1. Field list
   device_id,device_header,device_description,device_data,device_snmpstr - OK
2. Structure check:
   device_id, int(10) unsigned, NO, PRI, , auto_increment - OK
   device_header, varchar(100), YES, , ,  - OK
   device_description, varchar(100), YES, , ,  - OK
   device_data, varchar(100), YES, , ,  - OK
   device_snmpstr, varchar(100), YES, , ,  - OK
Table "devices" - DONE
Table "interfaces"
1. Field list
   id,device_id,interface_id,interface_name,interface_description,interface_moni
toring - OK
2. Structure check:
   id, bigint(20) unsigned, NO, PRI, , auto_increment - OK
   device_id, int(10) unsigned, NO, , ,  - OK
   interface_id, int(10) unsigned, NO, , ,  - OK
   interface_name, varchar(256), YES, , ,  - OK
   interface_description, varchar(256), YES, , ,  - OK
   interface_monitoring, tinyint(1), NO, , 0,  - OK
Table "interfaces" - DONE
Table "ip4temp"
1. Field list
   id,device_id,dtime,srcaddr,dstaddr,nexthop,input,output,dpkts,doctets,srcport
,dstport,tcp_flags,prot,tos,src_as,dst_as,src_mask,dst_mask - OK
2. Structure check:
   id, bigint(20) unsigned, NO, PRI, , auto_increment - OK
   device_id, int(10) unsigned, NO, , ,  - OK
   dtime, int(10) unsigned, YES, , ,  - OK
   srcaddr, int(10) unsigned, YES, , ,  - OK
   dstaddr, int(10) unsigned, YES, , ,  - OK
   nexthop, int(10) unsigned, YES, , ,  - OK
   input, smallint(5) unsigned, YES, , ,  - OK
   output, smallint(5) unsigned, YES, , ,  - OK
   dpkts, int(10) unsigned, YES, , ,  - OK
   doctets, int(10) unsigned, YES, , ,  - OK
   srcport, smallint(5) unsigned, YES, , ,  - OK
   dstport, smallint(5) unsigned, YES, , ,  - OK
   tcp_flags, tinyint(3) unsigned, YES, , ,  - OK
   prot, tinyint(3) unsigned, YES, , ,  - OK
   tos, tinyint(3) unsigned, YES, , ,  - OK
   src_as, smallint(5) unsigned, YES, , ,  - OK
   dst_as, smallint(5) unsigned, YES, , ,  - OK
   src_mask, smallint(5) unsigned, YES, , ,  - OK
   dst_mask, smallint(5) unsigned, YES, , ,  - OK
Table "ip4temp" - DONE
Table "ip4temp1"
1. Field list
   id,device_id,dtime,srcaddr,dstaddr,nexthop,input,output,dpkts,doctets,srcport
,dstport,tcp_flags,prot,tos,src_as,dst_as,src_mask,dst_mask - OK
2. Structure check:
   id, bigint(20) unsigned, NO, PRI, , auto_increment - OK
   device_id, int(10) unsigned, NO, , ,  - OK
   dtime, int(10) unsigned, YES, , ,  - OK
   srcaddr, int(10) unsigned, YES, , ,  - OK
   dstaddr, int(10) unsigned, YES, , ,  - OK
   nexthop, int(10) unsigned, YES, , ,  - OK
   input, smallint(5) unsigned, YES, , ,  - OK
   output, smallint(5) unsigned, YES, , ,  - OK
   dpkts, int(10) unsigned, YES, , ,  - OK
   doctets, int(10) unsigned, YES, , ,  - OK
   srcport, smallint(5) unsigned, YES, , ,  - OK
   dstport, smallint(5) unsigned, YES, , ,  - OK
   tcp_flags, tinyint(3) unsigned, YES, , ,  - OK
   prot, tinyint(3) unsigned, YES, , ,  - OK
   tos, tinyint(3) unsigned, YES, , ,  - OK
   src_as, smallint(5) unsigned, YES, , ,  - OK
   dst_as, smallint(5) unsigned, YES, , ,  - OK
   src_mask, smallint(5) unsigned, YES, , ,  - OK
   dst_mask, smallint(5) unsigned, YES, , ,  - OK
Table "ip4temp1" - DONE

10) Start perl service as daemon:

create service by using sc command:

Code: Select all

sc create netflow binPath= "C:\Strawberry\perl\bin\perl.exe c:\nflows.0.0.7\collector\threaded.pl"

now you have netflow service.

That's it folks!

P.S.: I hate windows. Really... This system are not about server software. This is a system for user applications like a Internet browser, PC games and multimedia players...

[hillda01]

Hi,

I'm quite a newbie to Cacti so please excuse me...

Is there an "idiots guide" to setting this netflow type information up within the Cacti server.

I know you can get netflow data in systems like Solarwinds but that is HUGELY expensive!

Regards,

Dave.