RRD Files Stopped Updating for No Reason

[TheWitness]

Here is a new version for testing.

Larry

[hoser]

Here's the debug from one data source:

Data Source Debug

/usr/bin/rrdtool create \
/var/www/cacti/rra/txsps0dist_traffic_in_1272.rrd \
--step 300 \
DS:traffic_in:COUNTER:600:0:100000000 \
DS:traffic_out:COUNTER:600:0:100000000 \
RRA:AVERAGE:0.5:1:600 \
RRA:AVERAGE:0.5:6:700 \
RRA:AVERAGE:0.5:24:775 \
RRA:AVERAGE:0.5:288:797 \
RRA:MAX:0.5:1:600 \
RRA:MAX:0.5:6:700 \
RRA:MAX:0.5:24:775 \
RRA:MAX:0.5:288:797 \

Here is another:

/usr/bin/rrdtool create \
/var/www/cacti/rra/pgrbtxbro1790_bwd_pd_mobile_data_traffic_in_1291.rrd \
--step 300 \
DS:traffic_in:COUNTER:600:0:100000000 \
DS:traffic_out:COUNTER:600:0:100000000 \
RRA:AVERAGE:0.5:1:600 \
RRA:AVERAGE:0.5:6:700 \
RRA:AVERAGE:0.5:24:775 \
RRA:AVERAGE:0.5:288:797 \
RRA:MAX:0.5:1:600 \
RRA:MAX:0.5:6:700 \
RRA:MAX:0.5:24:775 \
RRA:MAX:0.5:288:797 \

[TheWitness]

Does the latest file I provided fix the problem.

TheWitness

[TheWitness]

One other thing, for that Data Template, can you please select the "LAST" RRA and then see what happens. I am concerned that the "LAST" RRA is not found although I am not sure that would actually be a problem.

TheWitness

[TheWitness]

Also, please look for the item in your poller cache. If you don't find it, we have a start. I have tested (Win32) and the rrd is created just fine.

TheWitness

[hoser]

I have tried the latest file and it has not fixe the issue. I am not sure what you mean by changing the data template to use the LAST RRA. The last 'active' RRD file is *.1294 - the RRD files in question have been created, but then they only get written to maybe once per hour.

As far as the last active rrd file, it's showing up in the poller cache:

Data Source: PGRB-TX-BRO-93 Goldstar EMS - Traffic - 10.0.x.x - 1
RRD: /var/www/cacti/rra/pgrbtxbro93_goldstar_ems_traffic_in_1294.rrd
Action: 0, OID: .1.3.6.1.2.1.2.2.1.10.1 (Host: 10.0.x.x, Community: public)

But, for whatever reason, the file just doesn't get written to except periodically:

-rw-r--r-- 1 cacti cacti 94660 Sep 16 11:26 pgrbtxbro93_goldstar_ems_traffic_in_1294.rrd

It should have been written to at 11:30, 11:35, and lastly 11:40, but no dice.

[TheWitness]

Cactid or cmd.php? If Cactid, what version?

TheWitness

[hoser]

Cactid - version 0.8.6.d

[hoser]

Seem to have found the problem. Seems one of our system admins did a bunch of updates a couple of days ago and updated rrdtool. That's right about the time we added a few new devices and they didn't graph...

The upgraded version is rrdtool-1.0.50-1.1. That version seems to be broken in that although it can update already existing rrd files, it doesn't seem to be able to properly update rrd files created by this version.

I have backed off to rrdtool-1.0.49-2.1 and things are working for the new device graphs. Apparently the older version can update rrd files created by the next higher version.

I recommend anyone using the 0.8.6.f version of Cacti *not* use rrdtool-1.0.50-1.1.

[TheWitness]

Those system admins do it to you all the time. I was getting nervous.

TheWitness

[eshine]

My Cacti is working fine with some DS.
Don't have more details, but after some new inserts, I'm facing the folowing problem:
- some RRDs are not being updated (and was working a day ago) for a specific host template.
- this host template are based in XML data templates.
- there are other hosts using the above host template and data template and working fine.
- new devices based on this host template do not create RRD files.

Info:
- Cacti 0.86e
- Cactid 0.86e
- Linux OS Slack 10.1
- RRDTool 1.2.11
- MySQL 4.1.11
- PHP 4.3.11

Some log parsing...
---
Host 1: working fine
cactiuser@phoenix:/usr/local/cacti$ grep "\[180\]" log/cacti.log.debug
09/23/2005 11:45:04 AM - CACTID: Poller[0] Host[180] SNMP Result: Host responded to SNMP
09/23/2005 11:45:04 AM - CACTID: Poller[0] Host[180] RECACHE: Processing 1 items in the auto reindex cache for '10.43.9.8'
09/23/2005 11:45:05 AM - CACTID: Poller[0] Host[180] DS[4527] SNMP: v2: 10.43.9.8, dsname: BurstErrorRate, oid: .1.3.6.1.4.1.4325.4.3.5.8.0, value: 0.00
09/23/2005 11:45:07 AM - CACTID: Poller[0] Host[180] DS[6007] SNMP: v2: 10.43.9.8, dsname: SuRxPower, oid: .1.3.6.1.4.1.4325.4.2.14.0, value: -79
09/23/2005 11:45:07 AM - CACTID: Poller[0] Host[180] DS[7267] SNMP: v2: 10.43.9.8, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.4, value: 553934825
09/23/2005 11:45:07 AM - CACTID: Poller[0] Host[180] DS[7267] SNMP: v2: 10.43.9.8, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.4, value: 197242584
09/23/2005 11:45:07 AM - CACTID: Poller[0] Host[180] DEBUG: The POPEN returned the following File Descriptor 18
09/23/2005 11:45:08 AM - CACTID: Poller[0] Host[180] DS[2849] SCRIPT: perl /usr/local/cacti-0.8.6e/scripts/ping.pl 10.43.9.8, output: 72.9
09/23/2005 11:45:08 AM - CACTID: Poller[0] Host[180] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
---
Host 2: missing traffic_in/out polling
cactiuser@phoenix:/usr/local/cacti$ grep "\[192\]" log/cacti.log.debug
09/23/2005 11:45:07 AM - CACTID: Poller[0] Host[192] SNMP Result: Host responded to SNMP
09/23/2005 11:45:07 AM - CACTID: Poller[0] Host[192] RECACHE: Processing 1 items in the auto reindex cache for '10.43.5.10'
09/23/2005 11:45:08 AM - CACTID: Poller[0] Host[192] DS[4528] SNMP: v2: 10.43.5.10, dsname: BurstErrorRate, oid: .1.3.6.1.4.1.4325.4.3.5.8.0, value: 0.05
09/23/2005 11:45:10 AM - CACTID: Poller[0] Host[192] DS[6009] SNMP: v2: 10.43.5.10, dsname: SuRxPower, oid: .1.3.6.1.4.1.4325.4.2.14.0, value: -86
09/23/2005 11:45:10 AM - CACTID: Poller[0] Host[192] DEBUG: The POPEN returned the following File Descriptor 26
09/23/2005 11:45:10 AM - CACTID: Poller[0] Host[192] DS[2911] SCRIPT: perl /usr/local/cacti-0.8.6e/scripts/ping.pl 10.43.5.10, output: 34.4
09/23/2005 11:45:10 AM - CACTID: Poller[0] Host[192] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
---
Host 3: polling SuRxPower, but didn't accept STRING data
cactiuser@phoenix:/usr/local/cacti$ grep "\[1111\]" log/cacti.log.debug
09/23/2005 11:45:05 AM - CACTID: Poller[0] Host[47] DS[1111] SNMP: v2: 200.188.215.36, dsname: huge_buff_m, oid: .1.3.6.1.4.1.9.2.1.67.0, value: 4
09/23/2005 11:45:17 AM - CACTID: Poller[0] Host[1111] SNMP Result: Host responded to SNMP
09/23/2005 11:45:17 AM - CACTID: Poller[0] Host[1111] RECACHE: Processing 1 items in the auto reindex cache for '10.48.9.5'
09/23/2005 11:45:18 AM - CACTID: Poller[0] Host[1111] DS[7268] SNMP: v2: 10.48.9.5, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.1, value: 1581920721
09/23/2005 11:45:18 AM - CACTID: Poller[0] Host[1111] DS[7268] SNMP: v2: 10.48.9.5, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.1, value: 82887051
09/23/2005 11:45:19 AM - CACTID: Poller[0] Host[1111] DS[7263] SNMP: v2: 10.48.9.5, dsname: BurstErrorRate, oid: .1.3.6.1.4.1.4325.4.3.5.8.0, value: 0.00
09/23/2005 11:45:19 AM - CACTID: Poller[0] Host[1111] DEBUG: The POPEN returned the following File Descriptor 20
09/23/2005 11:45:19 AM - CACTID: Poller[0] Host[1111] DS[7266] SCRIPT: perl /usr/local/cacti-0.8.6e/scripts/ping.pl 10.48.9.5, output: 96.8
09/23/2005 11:45:20 AM - CACTID: Poller[0] Host[1111] DS[7264] ERROR: SNMP timeout detected [300 milliseconds], ignoring host '10.48.9.5'
09/23/2005 11:45:20 AM - CACTID: Poller[0] Host[1111] DS[7264] SNMP: v2: 10.48.9.5, dsname: SuRxPower, oid: .1.3.6.1.4.1.4325.4.2.14.0, value: U
09/23/2005 11:45:20 AM - CACTID: Poller[0] Host[1111] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function

Here is the snmpget of SuRxPower:
cactiuser@phoenix:/usr/local/cacti$ snmpget -v 2c -c
<community> 10.48.9.5 .1.3.6.1.4.1.4325.4.2.14.0
SNMPv2-SMI::enterprises.4325.4.2.14.0 = STRING: "-70,NA"
---EOT

Any tips? If possible, I want to restore these devices without recreate them

Full debug log file is attached for more info.

TIA for your help.

[TheWitness]

Edgar,

You must check your poller cache for the entries, and then verify via "medium" that they are being updated or not. Then, run rrdtool "fetch" to get the "last" entries. Also, make sure your min and max value limits in the rrdfile have not been exceeded.

Larry

[eshine]

Larry,

After read your post I've had a look in the poller cache and I saw a strange behaviour:
1) After login, went to "Console", "System Utilities".
2) Cacti shows me the attached image (before_click)
3) I've point my mouse over "View Poller Cache" and click it. After this, Cacti return a blank screen and without some options in left frame. See attached file "after_click".

Additional info:
- Web server: Apache 1.3.33 (Slack 10.1 package).

[eshine]

Wow... looking in the DB, I'm wondering if makes sense print all poller cache in a page:

mysql> select count(*) from poller_item ;
+----------+
| count(*) |
+----------+
| 3836 |
+----------+

I've tried "Clear Poller Cache"... unsuccessfully.

[TheWitness]

Edgar,

I suspect a problem with T-Hold.

Larry