Log shows RRD Updates, but RRD Not Updated

[Fred]

I just upgraded to 0.8.6j from 0.8.6h. After doing so, a good portion of my graphs have stopped updating. It appears that it's the graphs that were added later that aren't being updated.

... And here's where it gets weird.

Looking at the logs, I see the script execute successfully, generate proper output, and update the RRD. See:

Code: Select all

02/02/2007 01:25:14 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/www/localhost/htdocs/cacti/rra/router_voip_gateway_chanidle_3043.
rrd --template chanidle:chanprop:chanbusy:chanrsrv:chanrsrt:chanmaint 1170440709:84:0:33:3:0:0

Note that the log shows this update at 1:25pm. Looking at the RRD file, however:

Code: Select all

fdamstra@monitor02 ~/cacti/log $ sudo -u apache /usr/bin/rrdtool dump /var/www/localhost/htdocs/cacti/rra/router_voip_gateway_chanidle_3043.rrd | grep "lastupdate"
        <lastupdate> 1170438914 </lastupdate> <!-- 2007-02-02 12:55:14 EST -->

The last update was at 12:55pm. It's pretty clear the RRD file did not actually get updated.

Running the rrdtool update by hand, as the same user who's running cmd.php, works fine:

Code: Select all

fdamstra@monitor02 ~/cacti/log $ sudo -u apache /usr/bin/rrdtool update /var/www/localhost/htdocs/cacti/rra/router_voip_gateway_chanidle_3043.rrd --template chanidle:chanprop:chanbusy:chanrsrv:chanrsrt:chanmaint 1170440709:84:0:33:3:0:0
fdamstra@monitor02 ~/cacti/log $ echo $?
0

And, after running it by hand, the RRD file is actually updated.

I'm baffled.

[Fred]

FWIW, I've tried:
(1) Clearing the poller cache. I really don't get what this does, but I tried it. No change.
(2) Verifying that poller_output is empty after cmd.php executes. It is.
(3) Checking the "SYSTEM" line... the same number of data sources and RRDs are processed as before the upgrade.
(4) Verified that all patches are installed (including the poller_output_remainder patch).
(5) Checked permissions (in fact, did a 'chown -R' to the rra and log directories, just to be safe).
(6) Adding a new device. This device is not graphed, and the rrd is not even created. (again, running the command manually as the same user works fine)

I've turned on debug logging, but I don't spot anything unusual.

Edit to add:
I had to resort to backup. Too many of my graphs were affected. As I was looking at the differences between the poller.php files, I noticed that in my old (0.8.6h) version, the line:

Code: Select all

$rrdtool_pipe = rrd_init();

is commented out.

It was commented out because I was having a very similar problem when i upgraded to 0.8.6h, and followed the fix in this thread (commenting out rrd_init is hidden somewhere in the middle, but it solved my problem, and all was happy, until this upgrade).

So, reading more about this rrd_init function, I still have no idea why it causes problems. Looking at my path's configuration, though, my "RRDTool Default Font Path" is blank. I'm running rrdtool 1.2.15, so I'd have to think the fix mentioned in that other thread has been applied to my binaries.

Any thoughts or suggestions on why this line might cause problems?

[TheWitness]

Fred,

not to extend you pain any more, I would still like to get this fixed. Ok, so please post the follwoing:

OS and Version

Also, if Linux:

rpm -qa | grep libc

Then,

1) Upgrade to rrdtool 1.2.18 and see if it does the same thing
2) If so, run the following from the command line and look for errors in STDOUT/ERR: "cd <path_cacti>; php poller.php"

Post your output if it get's errors.

TheWitness

[Fred]

This is Gentoo. The install is pretty old, but it's been kept up to date.

libc is version 2.3.5-r3. rrdtool is 1.2.15-r3.

I ran poller.php by hand, but did not spot any errors, though there is a lot of information there. I'll have to wait for a maintenance period and try the upgrade again. After I do so, I'll run it and keep stdout, stderr, and the log for you.

Looking back, I'm pretty certain this relates to the 'rrdtool_init' line. Is it possible that the lack of a value in 'RRDTool Default Font Path' would cause this function call to fail? I've filled it in for good measure.

[TheWitness]

Unless you are graphing, it should not Fred.

Larry

[Fred]

Well, a month's gone by, and this month I had an extended maintenance period to do some major system updates... Basically bringing my gentoo system fully up-to-date, rather than just doing the security patches that I usually do. There's also a new kernel on the box.

Anyhow, after doing so, this problem disappeared. So I'm afraid the issue was either the font path for rrdtool, or it was something else entirely that was cleaned up by the system updates.

Wish I could give you more insight, TheWitness, but I have none to give.

[TheWitness]

NP.

TheWitness

[hoganw]

I'm experiencing this same problem, but it seems to be random. Graphs for servers on my local LAN update fine, but remote hosts are sketchy. Setting debug output, I see rrdtool update being called, but the .rrd files are not updated. If I copy and paste the rrdtool update and run it as the user running cacti then the .rrd file updates fine.

My rrdtool is a bit aged as is the cacti install, so first I suppose I'll try upgrading rrdtool and see what happens.

If anyone else has any insight into this problem, please let me know. Thanks!

[hoganw]

I'm experiencing this same problem, but it seems to be random. Graphs for servers on my local LAN update fine, but remote hosts are sketchy. Setting debug output, I see rrdtool update being called, but the .rrd files are not updated. If I copy and paste the rrdtool update and run it as the user running cacti then the .rrd file updates fine.

My rrdtool is a bit aged as is the cacti install, so first I suppose I'll try upgrading rrdtool and see what happens.

If anyone else has any insight into this problem, please let me know. Thanks!

After upgrading rrdtool to 1.2.18 all of my flakey/empty graphs have started updating correctly across the board. Sweet!

[Fred]

Hmm... Seems I never updated this.

I have always, since 2005, needed to comment out the 'rrd_init' and 'rrd_close' lines in poller.php. If I do not do this, graphs created after a certain date (and I have not had luck figuring out which graph this is), will not update. After commenting out these lines in poller.php, all my graphs start working again. I need to repeat this process every time I upgrade. I've always blamed it on some eccentricity of that Gentoo server.

However, the powers that be rewarded me with an upgraded server for cacti. So I put a fresh install of Ubuntu 6.06 LTS Server on it, and installed cacti from scratch. (I did not use the repository, as the version in there is quite outdated)

Once cacti was working correctly with a fresh install (it was working fine), I moved my database, scripts, resources, and rrd's across. Once I did this, the problem reappeared on the new server. Commenting out 'rrd_init' and 'rrd_close' fixed the issue once again.

At this point, I think there must be a setting in my cacti install, some corruption in the database, or a problem with one of my scripts that is causing the poller to stop working after a certain point.

The logs are large, and there are lots of lines (before and after the "fix") that contain:
"WARNING: Result from SCRIPT not valid. Partial result: ...". I believe these are the result of data sources set up with unresponsive hosts (which generally return "U" in place of a number).

Capturing STDERR from poller.php results in a number of messages with "ERROR: Not enough arguments", and occasionally "write error: broken pipe". These seem to happen whether or not the rrd_init/rrd_close lines are commented out.

The fix works, but I know The_Witness showed some interest in getting it resolved. However, he alluded that removing these lines might result in a speed penalty, and we'd like to get as much out of this machine as we can.

(Though, in other news, our server upgrade brought our 5+ minute polling process down to 122seconds:

03/18/2008 07:32:04 AM - SYSTEM STATS: Time:122.4952 Method:spine Processes:4 Threads:32 Hosts:310 HostsPerProcess:78 DataSources:7778 RRDsProcessed:3274

[Fred]

I have always, since 2005, needed to comment out the 'rrd_init' and 'rrd_close' lines in poller.php. If I do not do this, graphs created after a certain date will not update. After commenting out these lines in poller.php, all my graphs start working again. I need to repeat this process every time I upgrade.

I just upgraded from 0.8.7b to 0.8.7d, and it looks like this has been fixed. More accurately, I no longer need to comment out these lines, but a new "error" has found its way into my logs. I'm guessing somebody else was experiencing the problem and a workaround was found.

Everything graphs successfully, but in my log, even with logging set to low, shows:

Code: Select all

02/17/2009 01:15:39 PM - CMDPHP: Poller[0] ERROR: Detected RRDtool Crash attempting to perform write.

This message occurs approximately 2,930 times per polling cycle. The poller succeeds, with:

Code: Select all

02/17/2009:01:15:39 PM - SYSTEM STATS: Time:51.5949 Method:spine Processes:4 Threads:32 Hosts:320 HostsPerProcess:80 DataSources:10097 RRDsProcessed:4479

Looking at the code, it looks like this error is generated in lib/rrd.php in rrdtool_execute(). Appears to me that the "fix" for these problems was to close and reopen the pipe to rrd, and that permanent errors are solved if the open takes more than 4 attempts. It's clear from more advanced logging that the first (re)open works.

Curious if this is anything I should be concerned about, or if anybody knows why rrdtool is dying.

This is pretty much a stock Ubuntu 8.04 LTS Server Edition (x64) running rrdtool 1.2.19 (from repository). Now running cacti 0.8.7d and spine 0.8.7c. The problem with rrdtool_init started when I was running Gentoo. Database and scripts have all been moved across to the new server, and the problem followed.

[gandalf]

The huge pile of error messages is clearly not good. But rrdtool dying each time is the root cause. Please upgrade rrdtool, as 1.2.19 is quite old. 1.2.26 or 1.2.30 should be fine
Reinhard

[cooldude919]

I just upgraded cacti to .8.7d and i am having the same error now.

05/07/2009 01:36:14 PM - CMDPHP: Poller[0] ERROR: Detected RRDtool Crash attempting to perform write

I tried updating rrdtool to 1.2.30 but the error is still there. Most of the time everything still seems to complete ok, but sometimes the poller will randomly take more than 300 seconds and fail to finish a polling session.

Any ideas?

[TheWitness]

Are your running PHP5. There is a fix that works specifically for PHP5 but not PHP4. In PHP5, you can use the modified lib/rrd.php to properly restart rrdtool when it crashes. This is primary an RRDtool issue, but what the heck

TheWitness

[cooldude919]

Are your running PHP5. There is a fix that works specifically for PHP5 but not PHP4. In PHP5, you can use the modified lib/rrd.php to properly restart rrdtool when it crashes. This is primary an RRDtool issue, but what the heck

TheWitness

Thanks for the response.

I am running php 5.1.6.

It seems i am getting one of those error messages for each rrd that is updated. so my log is full of them every polling session, but it does seem that it retrys again for each one and completes ok because my rrd files are getting updated.

Would this fix still apply to me? Where do i get it from?