Help with the flowview plugin

[mikygee]

Hello,

I don't have a standard configuration so I would like a little help with this plugin.

I use flowd as a collector but there are others nfsen.
My flowd collector runs independantly from cacti and creates one file, /netflow/flows where everything is stored.

I would like cacti to read this file and display the stored informations.

1) In Settings > Misc I have

Code: Select all

Flow Tools Binary Path = /bin
Flow Tools Work Directory = /tmp
Flows Directory = /netflow/flows

With

Code: Select all

# ls /var/www/bin/ | grep flow
flow-cat
flow-filter
flow-nfilter
flow-print
flow-stat

2) My file is growing slowly

Code: Select all

# du -k /netflow/flows
66      /netflow/flows
# du -k /netflow/flows
86      /netflow/flows

3) I think I don't need to run any listener, do I ? I already have my file /netflow/flows

Does anyone have an advice ?

Also I have noticed in the plugin directory .sql files, what are they ?

Thank you

[TheWitness]

You have to create the listeners to the correct sub-directories of the netflow directory structure. However, you don't need to migrate the init.d script as you have full control of them. The tricky part is selecting the correct directory mapping algorithm to use. This is where people make the most mistakes. Presently we don't 'detect' the configuration for you.

TheWitness

[mikygee]

Hello,

Thank you for your answer. Could you help me to configure the listener ?

I have

Code: Select all

Device Name = ANYNAME
Directory = /netflow/flows
Allowed Host = 0
Port = 12345
Nesting = /
Netflow Version = v5

My tcpdump shows that I receive my netflow packets

Code: Select all

# tcpdump -i lo0 port 12345
tcpdump: listening on lo0, link-type LOOP
15:30:04.703378 127.0.0.1.53364 > 127.0.0.1.12345: udp 1464 (DF) [tos 0x10]
15:30:34.705466 127.0.0.1.53364 > 127.0.0.1.12345: udp 504 (DF) [tos 0x10]

[TheWitness]

What's the structure of the directory for this listener?

[mikygee]

I don't understand your question.

Actually I have watched your video, and another one also but everyone assumes that we use linux. I use OpenBSD and Cacti is in a chrooted Apache.

I also don't understand what is the role of this listener thumbnail, I understand Filters, I understand Schedules but not that one.

I have my pf that sends netflow data on 127.0.0.1:12345, it can also be an external firewall such as asa, netscreen etc.
I have my collector that is OpenBSD, on it I run flowd which collect the netflow data and populate one file (/var/www)/netflow/flows (real path into brackets). So I suppose this is flowd the listener. I just start it like this

Code: Select all

# flowd

When you ask me "What's the structure of the directory for this listener" I don't understand because I think everything is stored in the file named below.

[TheWitness]

Listeners define to Cacti where your data is located. Your chroot environment will require access to those locations. In the end, the UI uses the permissions it has to run the flowtools. The listener tells the flowtools command what the location of the flow captures is.

[mikygee]

Hello,

I understood that flowtools has to be installed in the chroot. That's what I listed in 1) of my first message.

My data is in the chroot (/var/www)/netflow/flows

When you write "listener" I understand it refers to a daemon. And for me this daemon is flowd

Code: Select all

# netstat -f inet -a
Proto   Recv-Q Send-Q  Local Address          Foreign Address        (state)
udp          0      0  localhost.12345        *.*

But when I see the Gui, it seems to refer to something else

In your video, you wrote

Code: Select all

local local /YYYY-MM-DD 0 2055 ....

What is YYYY-MM-DD, it's a directory or a file ?
Could you do a ls -l in you netflow directory ?
I don't understand what is this Nesting (Directory Structure that will be used for the flows for this device.) My netflow data file is just called flows

I had set my flow directory like this (has seen inside the chroot)

Code: Select all

Flows Directory = /netflow/flows

There are a few pieces from this puzzle I'm missing

[TheWitness]

Ok, so we are closer then, but you did not describe the directory structures completely that you are using. So, we know that it is in '/var/www/netflow/flows'. So, we then know that the base directory under Console->Settings should be '/var/www/netflow'. Your base directory under the listener should be called 'flows', and then it's simply the 'nesting' that is needed to be determined.

Again, since you are managing the startup of the listeners yourself, you only need these entries so that the flowtools can find the directory, format and version of the flow information.

Now, I did make the assumption that 'flows' is the specific flow directory for a single flow. If you start to send additional flows to the Cacti server, I'm also assuming the second and subsequent would be something like '/var/www/netflow/flows1..2..n'

TheWitness

[mikygee]

Hello,

Thank you again for your help.

I have understood the misunderstanding between us.
I was using a daemon called flowd
http://www.mindrot.org/projects/flowd/
but you expected me to use flow-capture
I think they both offer the same functionnality

Now that I started flow-capture

Code: Select all

flow-capture -S 1 -w /var/www/netflow/ 0/0/12345

I can see the directory structure you were refering to

Code: Select all

# ls /var/www/netflow/2011/2011-12/2011-12-09/
ft-v05.2011-12-09.060204+0100        ft-v05.2011-12-09.080001+0100        ft-v05.2011-12-09.100001+0100        ft-v05.2011-12-09.120001+0100

so now I have configured the listener like this

Code: Select all

local 	/netflow 	/YYYY/YYYY-MM/YYYY-MM-DD 	0 	12345 	NetFlow version 5 	0 	1 Minute 	2 Days

I have 2 new problems:
1) When I do a filter using the listener local, no lines/data are displayed
2) When selecting flow bar I have the flash error Open Flash Chart / JSON Parse Error [Syntax Error] / Error at character 0, line 1:

I suspect that 1) might be a path problem, because of chroot but I don't know how to debug it.
However when I'm chrooting my user I have the result for this command

Code: Select all

$ /bin/flow-cat /htdocs/metrologie/cacti/netflow/2011/2011-12/2011-12-09/ft-v05.2011-12-09.060204+0100 | flow-stat -f8
#  --- ---- ---- Report Information --- --- ---
# Fields:    Total
# Symbols:   Disabled
# Sorting:   None
# Name:      Destination IP
#
# Args:      flow-stat -f8
# IPaddr         flows                 octets                packets
#
192.168.1.246    27                    18684                 92
192.168.1.253   335                   505340                2089

[TheWitness]

You can use Firefox and Firebug to view the Ajax responses and find the error with the XML format that Open Flash Charts is receiving from the server. Something still seems wrong with your config.

[mikygee]

Hello,

I went to

I unselected all four Table / Bytes Bar / Packets Bar / Flows Bar started firebug and then selected Flows Bar

The system tries to load two pages

Code: Select all

http://127.0.0.1/cacti/plugins/flowview/open-flash-chart.swf
http://127.0.0.1/cacti/plugins/flowview/flowview.php?session=YTo0OntzOjc6ImNvbW1hbmQiO3M6MzQ3OiIvYmluL2Zsb3ctY2F0IC10ICIxMi8wOC8yMDExIDIyOjMwOjQ0IiAtVCAiMTIvMDkvMjAxMSAwNjozMDo0NCIgL2h0ZG9jcy9tZXRyb2xvZ2llL2NhY3RpL25ldGZsb3cvL2h0ZG9jcy9tZXRyb2xvZ2llL2NhY3RpL25ldGZsb3cvMjAxMS8yMDExLTEyLzIwMTEtMTItMDggL2h0ZG9jcy9tZXRyb2xvZ2llL2NhY3RpL25ldGZsb3cvL2h0ZG9jcy9tZXRyb2xvZ2llL2NhY3RpL25ldGZsb3cvMjAxMS8yMDExLTEyLzIwMTEtMTItMDkgIHwgL2Jpbi9mbG93LW5maWx0ZXIgLWYgL3RtcC9GbG93Vmlld2VyX2ZpbHRlcl8xMzIzNDA4NjQ0IC1GRmxvd1ZpZXdlcl9maWx0ZXIgfCAvYmluL2Zsb3ctc3RhdCAtZjEwIC1TMyI7czo0OiJwb3N0IjthOjI2OntzOjU6InF1ZXJ5IjtzOjE6IjAiO3M6MTE6ImRldmljZV9uYW1lIjtzOjMyOiIvaHRkb2NzL21ldHJvbG9naWUvY2FjdGkvbmV0ZmxvdyI7czoxMDoic3RhcnRfZGF0ZSI7czowOiIiO3M6MTA6InN0YXJ0X3RpbWUiO3M6ODoiLTggSE9VUlMiO3M6MTA6InRvc19maWVsZHMiO3M6MDoiIjtzOjg6ImVuZF9kYXRlIjtzOjA6IiI7czo4OiJlbmRfdGltZSI7czozOiJOT1ciO3M6OToidGNwX2ZsYWdzIjtzOjA6IiI7czo5OiJwcm90b2NvbHMiO3M6MDoiIjtzOjE0OiJzb3VyY2VfYWRkcmVzcyI7czowOiIiO3M6MTE6InNvdXJjZV9wb3J0IjtzOjA6IiI7czo5OiJzb3VyY2VfaWYiO3M6MDoiIjtzOjk6InNvdXJjZV9hcyI7czowOiIiO3M6MTI6ImRlc3RfYWRkcmVzcyI7czowOiIiO3M6OToiZGVzdF9wb3J0IjtzOjA6IiI7czo3OiJkZXN0X2lmIjtzOjA6IiI7czo3OiJkZXN0X2FzIjtzOjA6IiI7czoxMToic3RhdF9yZXBvcnQiO3M6MjoiMTAiO3M6MTE6ImZsb3dfc2VsZWN0IjtzOjE6IjEiO3M6MTc6InJlc29sdmVfYWRkcmVzc2VzIjtzOjE6IlkiO3M6MTA6InNvcnRfZmllbGQiO3M6MToiNCI7czoxMjoiY3V0b2ZmX2xpbmVzIjtzOjI6IjIwIjtzOjEzOiJjdXRvZmZfb2N0ZXRzIjtzOjE6IjAiO3M6NjoiYWN0aW9uIjtzOjQ6InZpZXciO3M6OToibmV3X3F1ZXJ5IjtzOjA6IiI7czo3OiJjaGFuZ2VkIjtzOjE6IjEiO31zOjc6ImV4cGlyZXMiO2k6MTMyMzQwODk0NDtzOjU6InRpdGxlIjtzOjg6Ik5ldyBGbG93Ijt9&action=chartdata&exclude=0&type=flows&title=Source/Destination%20IP

I don't see any error in firebug, however in the Apache logs I see

Code: Select all

[Fri Dec  9 22:48:14 2011] [error] ALERT-SIMULATION - configured GET variable value length limit exceeded - dropped variable 'session' (attacker '127.0.0.1', file '/htdocs/cacti/plugins/flowview/flowview.php')

And in my php.ini I have

Code: Select all

suhosin.simulation = On

So this error should be harmless

I'm not sure I understand everything you said in your last message. What additional steps should I do to troubleshoot further ?

[TheWitness]

Hosing is the operative word. Grew for memory limit in the flowview pluming make it something less.

[mikygee]

Sorry I didn't get what you meant: hoing...pluming...

There was this first problem

1) When I do a filter using the listener local, no lines/data are displayed

Which internal commands do the php script to retrieve the data from the flow files ?
I would like to check manually, what the script does in the backend.

[TheWitness]

Here is the pluming:

Code: Select all

$path = db_fetch_cell("SELECT value FROM `settings` WHERE name = 'path_flows_dir'");
$from = $device['allowfrom'];
$folder = $device['folder'];
shell_exec($tools_path . "/flow-capture -w $path/$folder 0/$from/$port -S5 -V$v -z $comp -n $rotate -e $expire -N $nest");

That is also where the flow-tools look for data.

TheWitness

[mikygee]

Thank god it works. As you said in your second message I did put the correct Directory or Nesting. I don't have the time today to troubleshoot what's wrong exactly. As for the flash problem, it didn't work because it didn't have any data to work with.

There is another minor problem but for now I have a quick question. When I display the flows with UDP/TCP port, it display the correct ports (162 for example) but writes unknown (supposed to be snmp I think). How does it try to resolve a port name ?
I have copied /etc/services in the /etc of the chroot but it didn't help