My graphs stopped

[jcoulam]

Last Friday my graphs just stopped collecting data. Many people have access to this box but everyone says they have not changed anything. So I need to debug this and fiure out what the deal is.
I checked all the permissions because when I look at the files in the rra/ directory they all had the friday date and user is "cactiuser" and group was "cacti"
After messing with all the settings clearing all the cache, no change so I upgraded from 0.8.6c to 0.8.6d still no change. In the log file I see:

05/11/2005 10:46:36 AM - CMDPHP: Poller[0] Host[13] SNMP: v1: 172.22.0.101, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.235, output: 1696498852
05/11/2005 10:46:36 AM - CMDPHP: Poller[0] Host[13] SNMP: v1: 172.22.0.101, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.234, output: U
05/11/2005 10:46:36 AM - CMDPHP: Poller[0] Host[13] WARNING: Result from SNMP not valid. Partial Result:
05/11/2005 10:46:36 AM - CMDPHP: Poller[0] Host[13] SNMP: v1: 172.22.0.101, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.234, output: U
05/11/2005 10:46:36 AM - CMDPHP: Poller[0] Host[13] WARNING: Result from SNMP not valid. Partial Result:
05/11/2005 10:46:36 AM - CMDPHP: Poller[0] Host[13] SNMP: v1: 172.22.0.101, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.233, output: U
05/11/2005 10:46:36 AM - CMDPHP: Poller[0] Host[13] WARNING: Result from SNMP not valid. Partial Result:
05/11/2005 10:46:36 AM - CMDPHP: Poller[0] Host[5] SNMP: v1: 192.168.133.251, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.22, output: 0
05/11/2005 10:46:36 AM - CMDPHP: Poller[0] Host[13] SNMP: v1: 172.22.0.101, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.233, output: U
05/11/2005 10:46:36 AM - CMDPHP: Poller[0] Host[13] WARNING: Result from SNMP not valid. Partial Result:
05/11/2005 10:46:36 AM - CMDPHP: Poller[0] Host[13] SNMP: v1: 172.22.0.101, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.232, output: U
05/11/2005 10:46:36 AM - CMDPHP: Poller[0] Host[13] WARNING: Result from SNMP not valid. Partial Result:
05/11/2005 10:46:36 AM - CMDPHP: Poller[0] Host[13] SNMP: v1: 172.22.0.101, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.232, output: U
05/11/2005 10:46:36 AM - CMDPHP: Poller[0] Host[13] WARNING: Result from SNMP not valid. Partial Result:
05/11/2005 10:46:36 AM - CMDPHP: Poller[0] Host[13] SNMP: v1: 172.22.0.101, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.231, output: U
05/11/2005 10:46:36 AM - CMDPHP: Poller[0] Host[13] WARNING: Result from SNMP not valid. Partial Result:

What is the "Partial Result" error if I do an snmpwalk it seems to be collecting everything just fine. Can anyone give anymore debugging tips?

[rony]

How big is your log file? Make sure it's not to large.

Did someone upgrade PHP or NET-SNMP or perhaps RRDtool?

[jcoulam]

One of the first things I did was delete the log file because it was pretty big.

All the vesions of PHP etc are the same.

What are some reasons why the rrd files are not being created?

Jeff

[rony]

Post one round of polling from the log file.

So, clear the log, wait for it to poll and post that. I'm looking for CACTI2RRD lines in the log.

[jcoulam]

I have attached my log file.
I created a new rra/ directory
I created a new MySQL database
Still does not write any .rrd files. If I run the poller.php, that will write .rrd files so I deleted my old crontab and re-did that still nothing.

[rony]

What version of rrdtool are you running?

Have you check the permissions on the rra and log directory to make sure that the user running the poller can read and write to them?

Run the following as the poller user and post the output:

Code: Select all

/usr/bin/rrdtool create /var/www/html/cacti/rra/rack_133_traffic_in_75.rrd   --step 300   DS:traffic_in:COUNTER:600:0:100000000  DS:traffic_out:COUNTER:600:0:100000000  RRA:AVERAGE:0.5:1:600  RRA:AVERAGE:0.5:6:700  RRA:AVERAGE:0.5:24:775  RRA:AVERAGE:0.5:288:797  RRA:MAX:0.5:1:600  RRA:MAX:0.5:6:700  RRA:MAX:0.5:24:775  RRA:MAX:0.5:288:797  

[jcoulam]

I am running rrdtool-1.0.49-3

[rony]

No, not the rrd, I want the result of the command..

But the fact that you have the RRD is interesting, so it created it?

[jcoulam]

lol
I cut and pasted the command that you sent and there was no output, it just created the file.

[chicago]

I had the same problem...

Turns out PHP was running out of memory!...

had to increase the max memory for PHP

[rony]

Um, what is your memory setting in PHP?

Should be 32 or 64 for cacti.

[chicago]

think i went 64 , have a lot of switchs and routers

[jcoulam]

I changed the memory limit to 64 still no graphs

[TheWitness]

I noticed from your log file that you are continually attempting to create RRD files. This is either a permissions problem or a duplicate DS problem in you UI. Please edit your Interfaces Data Template and look for duplicate traffic_in and traffic_out entries. If you find them, delete all but ID's 54 and 55. You can see the id's if you hover over the RED "X" and look at your status bar.

TheWitness

[jcoulam]

Ok, I have down to the cron job. If I "su cactiuser" then "php poller.php" the rrd files are created and the graphs have data.

My cron looks like this:
*/5 * * * * cactiuser /usr/bin/php /var/www/html/cacti/poller.php > /dev/null 2>&1

I took off the > /dev/null 2>&1 so I could get the cron emails and the emails show "/bin/sh: cactiuser: command not found"

My /etc/crontab
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin

I checked my /var/log/messages and it shows the the cron runs every 5 minutes and I get an email every 5 minutes.

Any suggestions?