Graphing dies after reboot

[itallushrt]

Hi and sorry for what is probably a redundant topic, but I've tried to search for a similar question / solution and was unable to find anything relevant.

To make a long story short I've inherited a box running Cacti 0.8.6h, rrdtool 1.2.12 (built from ports), and mysql 4.0.26 (built from ports).

This system was running fine until a recent reboot of the server. Since then cacti is not graphing any new data.

cron is running, it's calling the proper script :

*/5 * * * * cacti /usr/local/bin/php /usr/local/cacti/poller.php >/dev/null 2>/dev/null

I see it running in the process list. I see activity in the log file ...

A snippet from the last time it ran :

04/28/2006 10:06:30 AM - CMDPHP: Poller[0] Host[54] DS[2605] SNMP: v2: sdf-cab100-asw1.mgmt.XXXXX.com, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.112, output: 1520088804425
04/28/2006 10:06:30 AM - CMDPHP: Poller[0] Time: 87.7725 s, Theads: N/A, Hosts: 16


Permissions on the rra directory are fine, and I even chmod 777 to make sure.

Mysql is running and there is a db which user permissions, etc. Remember this was a previously working system.

I'm not sure where to look next.

Thanks in advance.

[fmangeant]

Hi

can you check the server time ? Because RRDtool won't let you update files "in the past".

[itallushrt]

date / time on the server looks good as this server polls an ntp server for its time.

su-2.05b# date
Fri Apr 28 10:50:19 EDT 2006

[itallushrt]

I've changed logging to debug yet I'm not seeing any errors in the log. What should I be looking for?

Any clues?

[gandalf]

I've changed logging to debug yet I'm not seeing any errors in the log. What should I be looking for?

Any clues?

Please check in sequence
- required hosts are checked to be up
- SNMP requests are made and return "valid" data
- rrdtool update are found in the logs
Then, please check "last used" date of the rrd files. To check their contens, please peform

Code: Select all

rrdtool fetch <rrd file> AVERAGE

on them, looking at the last few lines
Reinhard

[itallushrt]

1. hosts are certainly up and I can snmpwalk the devices from the cacti server.

2. snmp request seem valid

3. I see NO rrdtool updates in the logs. (I am running debug level logging)

4. The last updated time of all the log files in the rra dir are time stamped with the time just before the reboot of the server.

Code: Select all

su-2.05b# rrdtool fetch xxxxxxxxxxxxxx_traffic_in_4279.rrd AVERAGE
                    traffic_out          traffic_in

1146405600: NaN NaN
1146405900: NaN NaN
1146406200: NaN NaN

[itallushrt]

I failed to mention that the graphs display all current values as NaN

[gandalf]

2. snmp request seem valid

This answer is derived from reading the cacti.log?

3. I see NO rrdtool updates in the logs. (I am running debug level logging)

Please post your cacti.log from ONE pollers run in DEBUG mode

4. The last updated time of all the log files in the rra dir are time stamped with the time just before the reboot of the server.

You surely verified, that the poller is running every 5 min from crontab?
Reinhard

[itallushrt]

I've attached the output of the latest poll cycle with debug logging level.

Also, yes, as previously stated this was a working system and a recent reboot rendered it dead in the water. So yes, the poller is being called by cron every five minutes.

One new thing I've noticed is that there seems to be many instances of the poller running at once .. I'm not sure if this is typical or not.

Code: Select all

su-2.05b# more /etc/crontab 
# /etc/crontab - root's crontab for FreeBSD
#
# $FreeBSD: src/etc/crontab,v 1.21.2.3 2000/12/08 10:56:07 obrien Exp $
#
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin
HOME=/var/log


**** truncated ****



# Polling scripts
*/5 * * * * cacti /usr/local/bin/php /usr/local/cacti/poller.php >/dev/null 2>/dev/null



#########################################

and

#########################################




su-2.05b# ps -aux

**** truncated ****

cacti      1600  2.3  0.5 20792 18484  ??  S    11:15AM   0:03.18 /usr/local/bin/php -q /usr/local/cacti-0.8.6h/cmd.php 26 54
cacti      1595  0.0  0.5 19980 17552  ??  S    11:15AM   0:00.35 /usr/local/bin/php /usr/local/cacti/poller.php
cacti      1593  0.0  0.0   628  428  ??  Is   11:15AM   0:00.00 /bin/sh -c /usr/local/bin/php /usr/local/cacti/poller.php >/dev/nu
root       1591  0.0  0.0  1036  836  ??  I    11:15AM   0:00.00 cron: running job (cron)
cacti      1571  0.0  0.0  2016  836  ??  I    11:10AM   0:00.00 /usr/local/bin/rrdtool -
cacti      1570  0.0  0.0   628  428  ??  I    11:10AM   0:00.00 sh -c /usr/local/bin/rrdtool -
cacti      1560  0.0  1.4 55592 53208  ??  S    11:10AM   0:01.17 /usr/local/bin/php /usr/local/cacti/poller.php
cacti      1558  0.0  0.0   628  428  ??  Is   11:10AM   0:00.00 /bin/sh -c /usr/local/bin/php /usr/local/cacti/poller.php >/dev/nu
root       1556  0.0  0.0  1036  836  ??  I    11:10AM   0:00.00 cron: running job (cron)
cacti      1534  0.0  0.0  2016  836  ??  I    11:05AM   0:00.00 /usr/local/bin/rrdtool -
cacti      1533  0.0  0.0   628  428  ??  I    11:05AM   0:00.00 sh -c /usr/local/bin/rrdtool -
cacti      1523  0.0  2.4 94896 92604  ??  S    11:05AM   0:02.06 /usr/local/bin/php /usr/local/cacti/poller.php
cacti      1521  0.0  0.0   628  428  ??  Is   11:05AM   0:00.00 /bin/sh -c /usr/local/bin/php /usr/local/cacti/poller.php >/dev/nu
root       1519  0.0  0.0  1036  836  ??  I    11:05AM   0:00.00 cron: running job (cron)
cacti      1492  0.0  0.0  2016  836  ??  I    11:00AM   0:00.00 /usr/local/bin/rrdtool -
cacti      1491  0.0  0.0   628  428  ??  I    11:00AM   0:00.00 sh -c /usr/local/bin/rrdtool -
cacti      1480  0.0  3.4 136600 133840  ??  S    11:00AM   0:03.08 /usr/local/bin/php /usr/local/cacti/poller.php
cacti      1477  0.0  0.0   628  428  ??  Is   11:00AM   0:00.00 /bin/sh -c /usr/local/bin/php /usr/local/cacti/poller.php >/dev/nu
root       1474  0.0  0.0  1036  836  ??  I    11:00AM   0:00.00 cron: running job (cron)
cacti      1430  0.0  0.0  2016  836  ??  I    10:55AM   0:00.00 /usr/local/bin/rrdtool -
cacti      1429  0.0  0.0   628  428  ??  I    10:55AM   0:00.00 sh -c /usr/local/bin/rrdtool -
cacti      1419  0.0  4.4 175252 172828  ??  S    10:55AM   0:03.92 /usr/local/bin/php /usr/local/cacti/poller.php
cacti      1417  0.0  0.0   628  428  ??  Is   10:55AM   0:00.00 /bin/sh -c /usr/local/bin/php /usr/local/cacti/poller.php >/dev/nu
root       1415  0.0  0.0  1036  836  ??  I    10:55AM   0:00.00 cron: running job (cron)
cacti      1395  0.0  0.0  2016  836  ??  I    10:50AM   0:00.00 /usr/local/bin/rrdtool -
cacti      1394  0.0  0.0   628  428  ??  I    10:50AM   0:00.00 sh -c /usr/local/bin/rrdtool -
cacti      1384  0.0  5.7 223676 221492  ??  S    10:50AM   0:05.16 /usr/local/bin/php /usr/local/cacti/poller.php
cacti      1382  0.0  0.0   628  428  ??  Is   10:50AM   0:00.00 /bin/sh -c /usr/local/bin/php /usr/local/cacti/poller.php >/dev/nu
root       1380  0.0  0.0  1036  836  ??  I    10:50AM   0:00.00 cron: running job (cron)
cacti      1360  0.0  0.0  2016  836  ??  I    10:45AM   0:00.00 /usr/local/bin/rrdtool -
cacti      1359  0.0  0.0   628  428  ??  I    10:45AM   0:00.00 sh -c /usr/local/bin/rrdtool -
cacti      1349  0.0  6.9 271204 269092  ??  S    10:45AM   0:06.21 /usr/local/bin/php /usr/local/cacti/poller.php
cacti      1347  0.0  0.0   628  428  ??  Is   10:45AM   0:00.00 /bin/sh -c /usr/local/bin/php /usr/local/cacti/poller.php >/dev/nu
root       1345  0.0  0.0  1036  836  ??  I    10:45AM   0:00.00 cron: running job (cron)
cacti      1321  0.0  0.0  2016  836  ??  I    10:40AM   0:00.00 /usr/local/bin/rrdtool -
cacti      1320  0.0  0.0   628  428  ??  I    10:40AM   0:00.00 sh -c /usr/local/bin/rrdtool -
cacti      1310  0.0  8.4 329148 326104  ??  S    10:40AM   0:07.49 /usr/local/bin/php /usr/local/cacti/poller.php
cacti      1308  0.0  0.0   628  428  ??  Is   10:40AM   0:00.00 /bin/sh -c /usr/local/bin/php /usr/local/cacti/poller.php >/dev/nu
root       1306  0.0  0.0  1036  836  ??  I    10:40AM   0:00.00 cron: running job (cron)
cacti      1277  0.0  0.0  2016  836  ??  I    10:35AM   0:00.00 /usr/local/bin/rrdtool -
cacti      1276  0.0  0.0   628  428  ??  I    10:35AM   0:00.00 sh -c /usr/local/bin/rrdtool -
cacti      1266  0.0 10.0 392808 389940  ??  S    10:35AM   0:09.17 /usr/local/bin/php /usr/local/cacti/poller.php
cacti      1264  0.0  0.0   628  428  ??  Is   10:35AM   0:00.00 /bin/sh -c /usr/local/bin/php /usr/local/cacti/poller.php >/dev/nu
root       1262  0.0  0.0  1036  836  ??  I    10:35AM   0:00.00 cron: running job (cron)
cacti      1250  0.0  0.0  2016  836  ??  I    10:30AM   0:00.00 /usr/local/bin/rrdtool -
cacti      1249  0.0  0.0   628  428  ??  I    10:30AM   0:00.00 sh -c /usr/local/bin/rrdtool -
cacti      1235  0.0 11.8 462216 460096  ??  S    10:30AM   0:10.63 /usr/local/bin/php /usr/local/cacti/poller.php
cacti      1233  0.0  0.0   628  428  ??  Is   10:30AM   0:00.00 /bin/sh -c /usr/local/bin/php /usr/local/cacti/poller.php >/dev/nu
root       1231  0.0  0.0  1036  836  ??  I    10:30AM   0:00.00 cron: running job (cron)

[gandalf]

Did not yet check the log, cause your addendum is important. Please check all crontabs (e.g. /etc/crontab, /etc/cron.d/cacti and crontabs of users root and cactiuser) for poller entries. There must be exactly ONE. If there are more, this will definitively fail
Reinhard

[itallushrt]

There is only ONE process being called in cron.

[gandalf]

The polling seems to run fine (from your log). But there's not a single rrdtool update statement. Please have a look at http://forums.cacti.net/viewtopic.php?t=12037. And you may su to your cacti user and run the poller from command line to see possible rrdtool update statements.
Reinhard

[itallushrt]

That did it. The table in my instance had over 3.8 million objects.


mysql> select count(*) from poller_output;
+----------+
| count(*) |
+----------+
| 3859945 |


Deleted them and now I'm graphing. UGLY.

LVM .. THANKS for sticking with me on this and helping me reach a solution. You're truly a valuable asset to the OSS community.