RRD Files Stopped Updating for No Reason

[TheWitness]

Hi,

There have been a few reports of experienced users who have had their RRD files stop for no aparent reason. There appears to be something to this and we have identified a work around until more information is available for a fix. Please read the post below if you are having this problem.

TheWitness

http://forums.cacti.net/viewtopic.php?t ... sc&start=0

[TheWitness]

Ok, I have received a patch from the RRDtool Author. I need someone to test with RRDtool 1.2.10. Please download the 1.2.10 source and then in the "src" directory, place the following 2 files and remake.

Let's see if that fixes the issue. Please post responses.

TheWitness

EDIT: Originally uploaded dated files.

[lucky]

can't download it!

[fmangeant]

Hi
you should RRDtool 1.2.11, released last week : http://lists.ee.ethz.ch/rrd-users/msg10074.html

Code: Select all

New Features
------------

* Graph generation is much faster with this release. An MRTG type
  graph is about twice as fast, and SmokePing graph about 6 times
  as fast.

* RRDtool in interactive mode does now only quit when it really has
  to. In all other cases it complains and waits for new input.

Bugfixes
--------

* When stacking known on unknown data, the graph does
  not go haywire anymore.

* rrdtool graph with --no-gridfit and --only-graph does now produce
  sensible output

* configure works now with gcc-4.0

* changing just the font and not the size of the font as described
  in the man page now actually works. -- Havard Eidnes

* the makefiles now are less GNUish and work with Solaris standard
  make too. -- Stan Sinyagin

* make install locations more 'standard'  -- Stan Sinyagin

* bottom align the x-axis labels, this helps with languages where
  not every weekday name has the same 'height'

[starkie]

guys,

has anyone tried the files,

I have tried rrdtool-1.2.11 but my graphs do not show up. (They do but the thumbnails do not show. The rest of them do)

Does any one have the poller.php file with commented out stuff??

Thanks in advance

[BSOD2600]

Something like the problem I'm having here? http://forums.cacti.net/viewtopic.php?t=8762

[fletch]

I created a script Data source and can see it returning the 2 values in the log with no problem.

But the poller is not making the rrdtool create call - so the rrd file is never created -

This is a copy of a working Datasource (graphing temperatures in different datacenters) - I have compared the working to this one and they are the same...

I upgraded to rrdtool-1.2.11 - no difference.

Any idea why cacti would not make the rrdtool create call?

Thanks,
Fletch

[TheWitness]

Did you apply the patches at http://www.cacti.net/downloads/patches/ ... re-patched

TheWitness

[fletch]

I'm on 0.8.6d and I'm applied those patches.

I'm under the impression e and f were security only updates

Where is the code that iterates through the rrd's to be created/updated - I'm going to debug this when I get a chance...

Is it poller?
rrd.php?

I want to understand why the code is not calling rrdtool create.

Thanks,
Fletch.

[TheWitness]

Fletch, do you have AIM, MSIM or YIM? Can we work together?

TheWitness

[chemical]

The same problem applies here. Needed to move cacti 0.6.8 from and old hardware to a new one.

Enabled register_globals in php.ini to get 0.6.8 working on SuSE 9.3

(SuSE 9.3 uses php 4.3.10 and rrdtool 1.0.49)

Upgraded to 0.8.0 and fixed all my scripts to enable output with datasource name if more than one argument, and if the output is one argument, no change was needed (why is this???).
Nevertheless, copied all my old *.rrd into the rra/ dir and verified that the cacti user was able to write.

0.8.0 was working fine for a day, updated all graphs, web interface worked, nice.

The next day (today) I wanted to do the second step and update to 0.8.6f.
Did the same procedure like yesterday execpt that 0.8.6f doesn't want to update from a fresh cacti.sql import. Moved 0.8 database into the new one.
Upgrading worked. Checked rights again.

Now I'm suffering the same problems. No new graphs are created (if I delete a rrd file) and no graphs are updated. The rrd files are not even touched, tho all scripts work fine, respond with values and no errors are logged by cmd.php

I read this thread and applied all "pre-patches". No change.
I updated from rrdtool 1.0.49 to 1.2.11. No change.
I commented out the rrd_init and rrd_close. No change.
I cleared the poller cache. No change.
Checked a few data sources for duplicate Source names. None found.

I wondered where you guys get those CACTI2RRD messages if you set your logging style to debug. My debug output looks like this:

Code: Select all

08/16/2005 02:45:50 PM - CMDPHP: Poller[0] Host[0] DS[78] CMD: /bin/sh /var/sapcheck/check_oracle_tablespace.sh PSAPCLUD, output: Total:17253328 Allocated:16470176
08/16/2005 02:45:53 PM - CMDPHP: Poller[0] Host[0] DS[79] CMD: /bin/sh /var/sapcheck/check_oracle_tablespace.sh PSAPCLUI, output: Total:1979840 Allocated:1251560
08/16/2005 02:45:56 PM - CMDPHP: Poller[0] Host[0] DS[80] CMD: /bin/sh /var/sapcheck/check_oracle_tablespace.sh PSAPDDICD, output: Total:256000 Allocated:60888
08/16/2005 02:45:59 PM - CMDPHP: Poller[0] Host[0] DS[81] CMD: /bin/sh /var/sapcheck/check_oracle_tablespace.sh PSAPDDICI, output: Total:262144 Allocated:9976
08/16/2005 02:45:59 PM - CMDPHP: Poller[0] Host[0] DS[82] CMD: /var/sapcheck/diskspace.pl mahjong /, output: total:101005312 used:69844992
08/16/2005 02:46:05 PM - CMDPHP: Poller[0] Host[0] DS[83] CMD: /var/sapcheck/spool_p06.pl, output: 24863
08/16/2005 02:46:08 PM - CMDPHP: Poller[0] Host[0] DS[84] CMD: /bin/sh /var/sapcheck/check_oracle_tablespace.sh PSAPDOCUD, output: Total:307200 Allocated:68416
08/16/2005 02:46:08 PM - CMDPHP: Poller[0] Host[0] DS[85] CMD: /var/sapcheck/temp_check.pl un38, output: 20.4
08/16/2005 02:46:08 PM - CMDPHP: Poller[0] Host[0] DS[86] CMD: /var/sapcheck/spool_a06.pl, output: 1808

Found that log_verbosity is checked in table settings from rrd.php. I don't have such an entry there, and nothing changes if I set it manually to 5.

For me it looks like rrdtool isn't even called. But I'm not wizard enough to get through this php stuff in the smallest time.


I'm helpless now. Ideas? Any IRC links where I can get in touch?

Thanks
Roland

[TheWitness]

Have you insured that your cron is now running poller.php instead of cmd.php? If you've been running cmd.php for a while, you may have to "truncate table poller_output" to resolve the issue once poller.php is scheduled.

TheWitness

[chemical]

Oh.

Thanks. Looks like I missed that fact. Seems to work now!

[fletch]

I added some debug log statements to lib/poller.php
because I noticed its not logging the

Parsed MULTI output field for this new data source

and its this if statement that is failing:

if (isset($rrd_field_names{$matches[1]})) {

08/15/2005 04:58:16 PM - POLLER: Poller[0] FBC isset failed on Arrayvent Array
08/15/2005 04:58:16 PM - POLLER: Poller[0] FBC isset failed on Arrayrack1 Array

and the data is being returned:

08/15/2005 04:58:15 PM - CMDPHP: Poller[0] Host[25] CMD: /fs/web/irt-web/htdocs/irt-cacti3/scripts/getX126temp, output: vent:66
.31 rack1:67.32

So why is that IF condition failing?

continue the debug...

[fletch]

I commented out the poller_output table truncate in poller.php to debug and I notice that the problem DataSource is the only one sticking around after every polling interval:

+------------------------+---------------------+---------------+----------------------------------------------------------------+-------+
| output | time | local_data_id | rrd_path | rrd_na|
+------------------------+---------------------+---------------+----------------------------------------------------------------+-------+
| vent:66.20 rack1:66.99 | 2005-08-16 15:22:00 | 748 | /fs/web/irt-web/htdocs/irt-cacti3/rra/irtdevstage_vent_748.rrd | |
| vent:66.09 rack1:66.99 | 2005-08-16 15:17:00 | 748 | /fs/web/irt-web/htdocs/irt-cacti3/rra/irtdevstage_vent_748.rrd | |
| vent:66.20 rack1:66.99 | 2005-08-16 15:27:00 | 748 | /fs/web/irt-web/htdocs/irt-cacti3/rra/irtdevstage_vent_748.rrd | |
| vent:66.09 rack1:67.10 | 2005-08-16 15:32:00 | 748 | /fs/web/irt-web/htdocs/irt-cacti3/rra/irtdevstage_vent_748.rrd | |
+------------------------+---------------------+---------------+----------------------------------------------------------------+-------+

Does this point at a Data template problem ?

Thanks,
Fletch.