rrdtool update not running

[tdkevin]

Hello,

I have recently done a migration of my cacti server to a new machine. Everything has migrated fine, except now I notice my RRD graphs are no longer updating.

I have followed the troubleshooting documentation and everything (permissions, mysql, etc) checked out EXCEPT i cannot see any "rrdtool update" statements, as one of the steps on the troubleshooting documentation says to check the cacti.log for that :

Code: Select all

# cat cacti.log | grep -i rrdtool | grep -i update
#

So if I look in the poller cache for the paritcular RRD graph not updating (all of them are not updating), I can see that data is being pulled, but no data is being input in to the graphs :

Code: Select all

Data Source Debug

/usr/bin/rrdtool create \
/usr/local/cacti-0.8.7g/rra/web2-1_cpu_378.rrd \
--step 300  \
DS:cpu:GAUGE:600:0:100 \
RRA:AVERAGE:0.5:1:600 \
RRA:AVERAGE:0.5:6:700 \
RRA:AVERAGE:0.5:24:775 \
RRA:AVERAGE:0.5:288:797 \
RRA:MAX:0.5:1:600 \
RRA:MAX:0.5:6:700 \
RRA:MAX:0.5:24:775 \
RRA:MAX:0.5:288:797 \

Does anyone have suggestions? The troubleshooting documentation at docs.cacti.net don't mention what you should do next if rrdtool isnt updating , after checking the cacti.log file. Not sure where to go from here, so any help would be greatly appreciated!

Thanks

[tdkevin]

I should note that all the graphs have NAN values.

[tdkevin]

Here's an example of a blank graph :

Code: Select all

RRDTool Command:

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title='TOPSMEDIA1 - CPU Usage' \
--rigid \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label='percent' \
--slope-mode \
--font TITLE:12: \
--font AXIS:8: \
--font LEGEND:10: \
--font UNIT:8: \
DEF:a="/usr/local/cacti-0.8.7g/rra/topsmedia1_cpu_system_705.rrd":cpu_system:AVERAGE \
DEF:b="/usr/local/cacti-0.8.7g/rra/topsmedia1_cpu_user_706.rrd":cpu_user:AVERAGE \
DEF:c="/usr/local/cacti-0.8.7g/rra/topsmedia1_cpu_nice_704.rrd":cpu_nice:AVERAGE \
CDEF:cdefbc=TIME,1296755696,GT,a,a,UN,0,a,IF,IF,TIME,1296755696,GT,b,b,UN,0,b,IF,IF,TIME,1296755696,GT,c,c,UN,0,c,IF,IF,+,+ \
AREA:a#FF0000FF:"System"  \
GPRINT:a:LAST:"Current\:%8.2lf %s"  \
GPRINT:a:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:a:MAX:"Maximum\:%8.2lf %s\n"  \
AREA:b#0000FFFF:"User":STACK \
GPRINT:b:LAST:"  Current\:%8.2lf %s"  \
GPRINT:b:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:b:MAX:"Maximum\:%8.2lf %s\n"  \
AREA:c#00FF00FF:"Nice":STACK \
GPRINT:c:LAST:"  Current\:%8.2lf %s"  \
GPRINT:c:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:c:MAX:"Maximum\:%8.2lf %s\n"  \
LINE1:cdefbc#000000FF:"Total"  \
GPRINT:cdefbc:LAST:" Current\:%8.2lf %s"  \
GPRINT:cdefbc:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:cdefbc:MAX:"Maximum\:%8.2lf %s" 

RRDTool Says:

OK

[gandalf]

When migrating, did paths change?
R.

[tdkevin]

not at all -- i booted the new server into finnix and rsync'd the entire server over -- everything else works except the graphs within cacti.

i just tried moving the rrd files and seeing if it creates new ones, but it does not. I verified all the paths are valid in the cacti settings.

[gandalf]

Nevertheless, please run a rebuild poller cache. Verify, taht "System Utilities -> View Poller cache" shows correct paths and commands
R.

[tdkevin]

Yes -- Following the documentation I did do that. I rebuilt the poller cache twice. For one of the records in the poller cache you can see its trying to create the graphs with proper data :

Code: Select all

/usr/bin/rrdtool create \
/usr/local/cacti-0.8.7g/rra/topsweb2-1_cpu_378.rrd \
--step 300  \
DS:cpu:GAUGE:600:0:100 \
RRA:AVERAGE:0.5:1:600 \
RRA:AVERAGE:0.5:6:700 \
RRA:AVERAGE:0.5:24:775 \
RRA:AVERAGE:0.5:288:797 \
RRA:MAX:0.5:1:600 \
RRA:MAX:0.5:6:700 \
RRA:MAX:0.5:24:775 \
RRA:MAX:0.5:288:797 \

But that graph still shows "nan" ...

[tdkevin]

if theres something else i can check or verify or run manually, i'm open to anything at this point. i've done the debugging howto and couldn't find a root cause to this problem.

[gandalf]

Let me sum up what I did understand:

- the poller itself is running
- it creates entries in cacti.log
- data is polled from the targets and seen in that log file
- you did redirect the poller's output by changing the crontab to ... > /var/www/html/cacti/log/poller.log 2>&1
- log/poller.log does show entries? But no "rrdtool" entries???
- are rrd files updated or not?

What happens when you run "php -q poller.php --force" as cactiuser?
R.

[tdkevin]

- the poller itself is running

Yes.

- it creates entries in cacti.log

Yes

- data is polled from the targets and seen in that log file

Yes

- you did redirect the poller's output by changing the crontab to ... > /var/www/html/cacti/log/poller.log 2>&1
- log/poller.log does show entries? But no "rrdtool" entries???

I did that , there are entries such as the following in the log file :

Code: Select all

OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
ERROR: Not enough arguments
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
ERROR: Not enough arguments
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
OK u:0.00 s:0.03 r:4.16
ERROR: Not enough arguments

You can view the entire poller.log here : http://pastebin.com/cFCAw3gJ Please let me know if you see anything weird happening.

- are rrd files updated or not?

No they are not being updated. I have chmodded them 777 to rule out permissions.

What happens when you run "php -q poller.php --force" as cactiuser?

See the pastebin of the poller.log output.


If you want I can post the cacti.log file for you to review, but its rather large and I'm not sure if its necessary.

[gandalf]

Thanks. Those "ok" messages are output by rrdtool commands. So there are such commands. They will be printed to the log only when using higher debugging levels as given by the debugging guide. So I really require access to such a log (a single run would be ok).
R.

[tdkevin]

I had to cut the cacti.log in half because pastebin said it was too large


cacti.log : http://pastebin.com/gx60nD97

poller.log : http://pastebin.com/hz7F7edQ



Both are now with debug / developer logging enabled. Please let me know if you need anything else.


Thanks

[tdkevin]

I was able to resolve this issue. The cron poller entry was pointing to /cacti/lib/poller.php , but i changed it to /cacti/poller.php

The two files are different and I'm not sure why it exists in two locations. I am seeing graphs populate again!

This whole process was somewhat confusing, but Gandalf I thank you for offering your help. Going through this process has enlightened me towards more in-depth troubleshooting for cacti in general.

Thanks again!

[gandalf]

The lib/poller.php is a library file that defines some functions used by other modules
R.

[samirs2k]

Dear Gandalf,

Even I am facing somewhat similar problem. My 1) My Device was not showing up 2)RRD files were not getting created 3) No graphs were generated. After going through forums I tried triggering the Poller.php manually. I could see 1) Device showing up 2) RRD files created 3) Empty graph getting generated. I concluded the poller is not running as per the CRON schedule. I tried finding out a solution but all in vain. Can you please provide me a solution where poller does its job properly as defined in CRON (5Minutes).

If I keep on manually ( Through commandline) triggering poller.php the graph does generate. I dont understand why its not getting automated.