Inexplicably, our Cacti graphs stopped updating the other day. Watching the Cacti logs on debug mode, I'm not seeing rrdtool information anywhere. The last messages in the log after a poller run:
06/30/2006 10:33:25 AM - PHPSVR: Poller[0] DEBUG: PHP Script Server Shutdown request received, exiting
06/30/2006 10:33:25 AM - CMDPHP: Poller[0] Time: 203.3228 s, Theads: N/A, Hosts: 79
Where else can I look for help goading Cacti to start updating my graphs again?
rrdtool not running after poller
Moderators: Developers, Moderators
rrdtool not running after poller
Did you ever solve this problem?
I think I may have a similar problem.
Cacti has been graphing away happily for 9 months or so, doing a great job. I went on vacation for a week and upon returning found that the .rrd files were no longer updating as of Thursday June 29, at approx 11PM. ALL of them.
Something changed, obviously, but I can't figure out what.
I'm running
Cacti 0.8.6g
PHP 5.1.1
RRDtool 1.0.50
Linux version 2.4.21-20.EL (bhcompile@tweety.build.redhat.com) (gcc version 3.2.3 20030502 (Red Hat Linux 3.2.3-42)) #1 Wed Aug 18 20:58:25 EDT 2004
My permissions are correct, I think.
[root@yyy rra]# ls -latr /var/www/html/cacti/rra
-rw-r--r-- 1 cactiuser cactiuser 94660 Jun 29 23:06 xyz_hdd_total_827.rrd
-rw-r--r-- 1 cactiuser cactiuser 94660 Jun 29 23:06 xyz_hdd_total_826.rrd
-rw-r--r-- 1 cactiuser cactiuser 94660 Jun 29 23:06 xyz_hdd_total_825.rrd
<snip>
I am able to do rrdtool updates to the .rrd files manually as the cactiuser, and the values that I enter appear on the graphs. But other than that, my graphs are faithfully graphing NaN for everything because the .rrd files are not being updated by Cacti.
[root@yyy rra]# crontab -l -u root | head -1
*/5 * * * * cactiuser php /var/www/html/cacti/poller.php > /dev/null 2>&1
I have on DEBUG level logging, but do not see anything that looks like an error occurring. What I do see is that there is periodic CACTI2RRD: /usr/bin/rrdtool update messages all the way up to 6/29/2006 11:06:19 PM and then they stop.
Here is the last CACTI2RRD entry and a few lines after that:
06/29/2006 11:06:19 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/www/html/cacti/rra/xxx_hdd_total_790.rrd --template hdd_used:hdd_total 1151636702:0:0
06/29/2006 11:06:19 PM - SYSTEM STATS: Time:77.3641 Method:cmd.php Processes:1 Threads:N/A Hosts:35 HostsPerProcess:35 DataSources:972 RRDsProcessed:4411
06/29/2006 11:06:19 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q /var/www/html/cacti/poller_commands.php]
06/29/2006 11:06:19 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q /var/www/html/cacti/poller_export.php]
06/29/2006 11:07:06 PM - CMDPHP: Poller[0] Host[2] PING: UDP ping timed out
06/29/2006 11:07:06 PM - CMDPHP: Poller[0] Host[2] SNMP: SNMP not performed due to setting or ping result.
06/29/2006 11:07:06 PM - CMDPHP: Poller[0] Host[7] PING: Host is Alive
06/29/2006 11:07:06 PM - CMDPHP: Poller[0] Host[7] SNMP: Host responded to SNMP
06/29/2006 11:07:06 PM - CMDPHP: Poller[0] Host[7] RECACHE: Processing 3 items in the auto reindex cache for 'xxx'.
06/29/2006 11:07:06 PM - CMDPHP: Poller[0] Host[7] DS[152] CMD: sh /var/www/html/cacti/scripts/remote_linux_load.sh xxx,
output: 1min:4.14 5min:4.36 15min:4.52
06/29/2006 11:07:06 PM - CMDPHP: Poller[0] Host[7] DS[153] CMD: sh /var/www/html/cacti/scripts/remote_linux_uptime.sh xxx, output: 44
[cactiuser@xyz log]$ php /var/www/html/cacti/poller.php
07/07/2006 05:11:36 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q /var/www/html/cacti/cmd.php 0 47]
Waiting on 1/1 pollers.
And it hangs there waiting. I am not sure what I am supposed to see here, but from some of the other posts here, I think more.
[cactiuser@yyy log]$ php /var/www/html/cacti/cmd.php
07/07/2006 05:01:24 PM - CMDPHP: Poller[0] PHP Script Server Started Properly
07/07/2006 05:01:24 PM - CMDPHP: Poller[0] Host[1] PING: Host is Alive
07/07/2006 05:01:24 PM - CMDPHP: Poller[0] Host[1] SNMP: Host responded to SNMP
07/07/2006 05:01:24 PM - CMDPHP: Poller[0] Host[1] RECACHE: Processing 3 items in the auto reindex cache for 'xyz'.
07/07/2006 05:01:25 PM - CMDPHP: Poller[0] Host[1] DS[3] CMD: perl /var/www/html/cacti/scripts/linux_memory.pl MemFree:, output: 70360
07/07/2006 05:01:25 PM - CMDPHP: Poller[0] Host[1] DS[4] CMD: perl /var/www/html/cacti/scripts/linux_memory.pl SwapFree:, output: 1416048
rrdtool is where cacti thinks it should be.
Also the post-broken (after being broken) logs show cmd.php running:
[cactiuser@xyz log]$ grep "CMD: /usr/bin/php, ARGS: -q" cacti.log | cut -d"-" -f2- | sort | uniq
POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q /var/www/html/cacti/cmd.php 0 47]
But the pre-broken logs show poller_commands.php and poller_export.php running as well as cmd.php.
[cactiuser@xyz log]$ grep "CMD: /usr/bin/php, ARGS: -q" cacti.log.8 | cut -d"-" -f2- | sort | uniq
POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q /var/www/html/cacti/cmd.php 0 46]
POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q /var/www/html/cacti/poller_commands.php]
POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q /var/www/html/cacti/poller_export.php]
Post-broken logs show no rrdtool activity and pre-broken logs show many rrdtool updates, as already mentioned.
I have recently cleared Poller cache, but no help. My View Poller Cache has 995 items in it.. Are those supposed to still show up after running the Clear Poller Cache action?
Any ideas would be appreciated.
Thanks,
Jason
I think I may have a similar problem.
Cacti has been graphing away happily for 9 months or so, doing a great job. I went on vacation for a week and upon returning found that the .rrd files were no longer updating as of Thursday June 29, at approx 11PM. ALL of them.
Something changed, obviously, but I can't figure out what.
I'm running
Cacti 0.8.6g
PHP 5.1.1
RRDtool 1.0.50
Linux version 2.4.21-20.EL (bhcompile@tweety.build.redhat.com) (gcc version 3.2.3 20030502 (Red Hat Linux 3.2.3-42)) #1 Wed Aug 18 20:58:25 EDT 2004
My permissions are correct, I think.
[root@yyy rra]# ls -latr /var/www/html/cacti/rra
-rw-r--r-- 1 cactiuser cactiuser 94660 Jun 29 23:06 xyz_hdd_total_827.rrd
-rw-r--r-- 1 cactiuser cactiuser 94660 Jun 29 23:06 xyz_hdd_total_826.rrd
-rw-r--r-- 1 cactiuser cactiuser 94660 Jun 29 23:06 xyz_hdd_total_825.rrd
<snip>
I am able to do rrdtool updates to the .rrd files manually as the cactiuser, and the values that I enter appear on the graphs. But other than that, my graphs are faithfully graphing NaN for everything because the .rrd files are not being updated by Cacti.
[root@yyy rra]# crontab -l -u root | head -1
*/5 * * * * cactiuser php /var/www/html/cacti/poller.php > /dev/null 2>&1
I have on DEBUG level logging, but do not see anything that looks like an error occurring. What I do see is that there is periodic CACTI2RRD: /usr/bin/rrdtool update messages all the way up to 6/29/2006 11:06:19 PM and then they stop.
Here is the last CACTI2RRD entry and a few lines after that:
06/29/2006 11:06:19 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/www/html/cacti/rra/xxx_hdd_total_790.rrd --template hdd_used:hdd_total 1151636702:0:0
06/29/2006 11:06:19 PM - SYSTEM STATS: Time:77.3641 Method:cmd.php Processes:1 Threads:N/A Hosts:35 HostsPerProcess:35 DataSources:972 RRDsProcessed:4411
06/29/2006 11:06:19 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q /var/www/html/cacti/poller_commands.php]
06/29/2006 11:06:19 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q /var/www/html/cacti/poller_export.php]
06/29/2006 11:07:06 PM - CMDPHP: Poller[0] Host[2] PING: UDP ping timed out
06/29/2006 11:07:06 PM - CMDPHP: Poller[0] Host[2] SNMP: SNMP not performed due to setting or ping result.
06/29/2006 11:07:06 PM - CMDPHP: Poller[0] Host[7] PING: Host is Alive
06/29/2006 11:07:06 PM - CMDPHP: Poller[0] Host[7] SNMP: Host responded to SNMP
06/29/2006 11:07:06 PM - CMDPHP: Poller[0] Host[7] RECACHE: Processing 3 items in the auto reindex cache for 'xxx'.
06/29/2006 11:07:06 PM - CMDPHP: Poller[0] Host[7] DS[152] CMD: sh /var/www/html/cacti/scripts/remote_linux_load.sh xxx,
output: 1min:4.14 5min:4.36 15min:4.52
06/29/2006 11:07:06 PM - CMDPHP: Poller[0] Host[7] DS[153] CMD: sh /var/www/html/cacti/scripts/remote_linux_uptime.sh xxx, output: 44
[cactiuser@xyz log]$ php /var/www/html/cacti/poller.php
07/07/2006 05:11:36 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q /var/www/html/cacti/cmd.php 0 47]
Waiting on 1/1 pollers.
And it hangs there waiting. I am not sure what I am supposed to see here, but from some of the other posts here, I think more.
[cactiuser@yyy log]$ php /var/www/html/cacti/cmd.php
07/07/2006 05:01:24 PM - CMDPHP: Poller[0] PHP Script Server Started Properly
07/07/2006 05:01:24 PM - CMDPHP: Poller[0] Host[1] PING: Host is Alive
07/07/2006 05:01:24 PM - CMDPHP: Poller[0] Host[1] SNMP: Host responded to SNMP
07/07/2006 05:01:24 PM - CMDPHP: Poller[0] Host[1] RECACHE: Processing 3 items in the auto reindex cache for 'xyz'.
07/07/2006 05:01:25 PM - CMDPHP: Poller[0] Host[1] DS[3] CMD: perl /var/www/html/cacti/scripts/linux_memory.pl MemFree:, output: 70360
07/07/2006 05:01:25 PM - CMDPHP: Poller[0] Host[1] DS[4] CMD: perl /var/www/html/cacti/scripts/linux_memory.pl SwapFree:, output: 1416048
rrdtool is where cacti thinks it should be.
Also the post-broken (after being broken) logs show cmd.php running:
[cactiuser@xyz log]$ grep "CMD: /usr/bin/php, ARGS: -q" cacti.log | cut -d"-" -f2- | sort | uniq
POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q /var/www/html/cacti/cmd.php 0 47]
But the pre-broken logs show poller_commands.php and poller_export.php running as well as cmd.php.
[cactiuser@xyz log]$ grep "CMD: /usr/bin/php, ARGS: -q" cacti.log.8 | cut -d"-" -f2- | sort | uniq
POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q /var/www/html/cacti/cmd.php 0 46]
POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q /var/www/html/cacti/poller_commands.php]
POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q /var/www/html/cacti/poller_export.php]
Post-broken logs show no rrdtool activity and pre-broken logs show many rrdtool updates, as already mentioned.
I have recently cleared Poller cache, but no help. My View Poller Cache has 995 items in it.. Are those supposed to still show up after running the Clear Poller Cache action?
Any ideas would be appreciated.
Thanks,
Jason
rrdtool not running after poller
Tried changing
[root@scenmon1 php]# grep -i memory_limit /etc/php.ini
#memory_limit = 8M ; Maximum amount of memory a script may consume (8MB)
#memory_limit = 16M ; Maximum amount of memory a script may consume (8MB)
memory_limit = 32M ; Maximum amount of memory a script may consume (8MB)
And running by hand. The process finished but no updated to .rrds were made.
[root@scenmon1 php]# grep -i memory_limit /etc/php.ini
#memory_limit = 8M ; Maximum amount of memory a script may consume (8MB)
#memory_limit = 16M ; Maximum amount of memory a script may consume (8MB)
memory_limit = 32M ; Maximum amount of memory a script may consume (8MB)
And running by hand. The process finished but no updated to .rrds were made.
I've just run into this, and I think I've found the problem, or at least an ugly way to resolve the problem.
It appears that in lib/poller.php, the database is queried to see what data needs to be added to the rrds. If you have a bunch of data in the poller_output table, the output from the select statement never returns. I didn't did log enough to determine if it was a timeout, or what. I do know that if you delete all rows from poller_output that span the last rrd update time through the current time, everything starts to work again.
This seems to come up often enough that DEBUG output in lib/poller.php.process_poller_output would be of benefit.
- Sam
It appears that in lib/poller.php, the database is queried to see what data needs to be added to the rrds. If you have a bunch of data in the poller_output table, the output from the select statement never returns. I didn't did log enough to determine if it was a timeout, or what. I do know that if you delete all rows from poller_output that span the last rrd update time through the current time, everything starts to work again.
This seems to come up often enough that DEBUG output in lib/poller.php.process_poller_output would be of benefit.
- Sam
Who is online
Users browsing this forum: No registered users and 2 guests