Traffic monitor stops working after a reboot

[DutchSamurai]

Hi,

I noticed that if I reboot the pc that is running cacti (windows host, pfsense and cacti in VM) cacti becomes unable to monitor the SNMP traffic. I need to delete my whole entry and add all data sources and graphs again to make it work.

Why is this happening and is there any way to fix this? I noticed a ton of things in cacti are bugged, some pretty fundamental like this.

[phalek]

Change the logging level to "debug" and check the cacti logfile after a reboot.

What errors are being displayed ?
Is the poller actually running at all ?
Where do you store your database, in memory or on the disk ?

[DutchSamurai]

edit: added a screenshot showing how the cpu is still updated.

This is what I get running snmpwalk after the reboot.

snmpwalk -v 2c -c test 192.168.2.2 1.3.6.1.2.1.2.2.1.2.1
IF-MIB::ifDescr.1 = STRING: em0

Cacti working

Graph

If I look inside that folder there are lots of rrd files so I assume Cacti is saving logs.

Code: Select all

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=1453443305 \
--end=1453446905 \
--title='Test - Traffic - em0' \
--rigid \
--base=1000 \
--height=400 \
--width=1000 \
--alt-autoscale-max \
--lower-limit='0' \
COMMENT:"From 2016/01/22 15\:15\:05 To 2016/01/22 16\:15\:05\c" \
COMMENT:"  \n" \
--vertical-label='bytes per second' \
--slope-mode \
--font TITLE:10: \
--font AXIS:8: \
--font LEGEND:8: \
--font UNIT:8: \
DEF:a="/var/www/html/rra/test_traffic_in_79.rrd":'traffic_in':AVERAGE \
DEF:b="/var/www/html/rra/test_traffic_in_79.rrd":'traffic_out':AVERAGE \
AREA:a#00CF00FF:"Inbound"  \
GPRINT:a:LAST:" Current\:%8.2lf %s"  \
GPRINT:a:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:a:MAX:"Maximum\:%8.2lf %s"  \
COMMENT:"Total In\:  153.18 MB\n"  \
LINE1:b#002A97FF:"Outbound"  \
GPRINT:b:LAST:"Current\:%8.2lf %s"  \
GPRINT:b:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:b:MAX:"Maximum\:%8.2lf %s"  \
COMMENT:"Total Out\: 3.02 MB" 

Log: before reboot.txt


After pfsense reboot, Cacti isn't processing interface traffic anymore. It is logging cpu utilization.

Graph

Code: Select all

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=1453443885 \
--end=1453447485 \
--title='Test - Traffic - em0' \
--rigid \
--base=1000 \
--height=400 \
--width=1000 \
--alt-autoscale-max \
--lower-limit='0' \
COMMENT:"From 2016/01/22 15\:24\:45 To 2016/01/22 16\:24\:45\c" \
COMMENT:"  \n" \
--vertical-label='bytes per second' \
--slope-mode \
--font TITLE:10: \
--font AXIS:8: \
--font LEGEND:8: \
--font UNIT:8: \
DEF:a="/var/www/html/rra/test_traffic_in_79.rrd":'traffic_in':AVERAGE \
DEF:b="/var/www/html/rra/test_traffic_in_79.rrd":'traffic_out':AVERAGE \
AREA:a#00CF00FF:"Inbound"  \
GPRINT:a:LAST:" Current\:%8.2lf %s"  \
GPRINT:a:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:a:MAX:"Maximum\:%8.2lf %s"  \
COMMENT:"Total In\:  154.35 MB\n"  \
LINE1:b#002A97FF:"Outbound"  \
GPRINT:b:LAST:"Current\:%8.2lf %s"  \
GPRINT:b:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:b:MAX:"Maximum\:%8.2lf %s"  \
COMMENT:"Total Out\: 3.31 MB" 

Log: after reboot.txt

[phalek]

The polling issue is caused due to the re-caching of the device, so actually rebooting the 192.168.2.2 device only will result in the same effect.

One solution is to actually have the SNMP query Re-Index method set to "none" and do a manual re-indexing via cron. ( Makes the polling times faster as well ).

You probably did set it to "uptime goes backwards", at least an "index count changed" would be the better solution if you don't want to do a manual re-indexing.

Unfortunately, this cannot be changed on the interface itself, so you either have the remove these graphs/snmp queries and re-add them, or update the database using the following SQL statements:


Update the reindex method for a host for all data queries to “none” ( 0 = none, 3 = “Verify All Fields” )

Code: Select all

UPDATE host_snmp_query SET reindex_method=0 WHERE host_id=6;

Delete the entries from the poller_reindex table:

Code: Select all

DELETE FROM poller_reindex WHERE host_id=6;

Then it should monitor/poll the device after a reboot.

[DutchSamurai]

Thank you.

From where am I supposed to run those commands?

Code: Select all

[root@localhost ~]# DELETE FROM poller_reindex WHERE host_id=10;
-bash: DELETE: command not found

Where can I add the cron job? cron.d/0hourly has 1 in it but I'm clueless as to what that is as changing the poller and cron interval in the settings doesn't change the cron value in the file.

[phalek]

From the mySQL command line ( these are SQL statements ).

The CLI command for the re-indexing is in the cli directory. THe file is poller_reindex_hosts.php

[DutchSamurai]

By the way my current install just stopped working. I've tried deleting and reading my device but no interface traffic is getting logged. It's just a thin blue line at the bottom of the graph same as on the screenshot.

I did try the verify all fields query on my previous tests but not the index count changed one.

[DutchSamurai]

Ahh how we love mysql and their -u -p commands that don't work.

I had enough of this for the week. Will continue with the neverending list of awkwardness after the weekend.

Sorry for sounding so grumpy and ungrateful, I'm happy you are helping me out. Some of this shit just doesn't work like it should.

[DutchSamurai]

Tried all your suggestions but none of them work unfortunately. On top of that Cacti decided to just die and start reporting inaccurate results on the traffic monitor. Fresh install didn't help. As I noticed before SNMP and cacti seems to be a small disaster. If you're lucky it'll work, but don't expect it to.

[DutchSamurai]

Okay after some more testing I discovered rebooting the pfsense box will break cacti interface monitoring. The device comes back up in cacti but the interface monitor will just show a flat blue line and the device uptime is incorrect, showing 400+ days online. Rebuilding the php poller or mysql database doesn't do anything. Well, apart from breaking the ability to monitor the cpu...

Now, restart the pfsense snmp service and interface traffic is showing up again. Looks like Cacti isn't properly picking up the interface OIDs after a reboot.

[phalek]

Can you post a screenshot of the lower part of your pfsense device page ? It should show the "Associate data queries" and the other things you've added for monitoring.