RRD Files Stopped Updating for No Reason

[brammator]

i was on cacti 0.8.6d, cactid 0.8.6c, rrdtool 1.0 (debian sarge stable) and got bug described in subject of this thread (poller_output table grows, .rrd files not updated)

so i upgraded my system to debian testing: cacti 0.8.6h-2, cactid 0.8.6g-2, rrdtool 1.2.11-0.5

all upgrade scenaries was 'succesful' and 'found'

now all i get in my log is

03/01/2006 01:40:01 PM - CMDPHP: Poller[0] DEBUG: SQL Cell: "select count(*) from poller_item"
03/01/2006 01:40:01 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select id from host where disabled = '' order by id"
03/01/2006 01:40:01 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "replace into settings (name,value) values ('path_webroot','/usr/share/cacti/site')"
03/01/2006 01:40:01 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "truncate table poller_time"
03/01/2006 01:40:01 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/sbin/cactid, ARGS: 0 259]
03/01/2006 01:40:02 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "replace into settings (name,value) values ('date',NOW())"
03/01/2006 01:40:02 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_id,end_time from poller_time where poller_id = 0"
03/01/2006 01:40:02 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name)"


and no SNMP queries at all. all settings and data/host/etc configs seems normal.

well, how to repair my broken cacti site? any suggestions? thanks.

[gandalf]

Please run cactid from cli; best if DEBUG mode set. It would be enough to run it for a single host or two, e.g. ./cactid 0 1 .Please post your findings
Reinhard

[brammator]

Resolved by setting php.ini/memory_limit to 600M and three manual runs poller.php from command line (292 seconds internal limit). Too many data was stored in poller_output.

Also, there is a bug in debian testing distribution: via upgrade, conf-files in /etc/cacti/ was replaced by emptied ones (all settings was like password=''), so i mistyped "localhost" in cactid.conf 1-(

Sadly, there was no error messages about "can't connect to sql" in /var/log/cacti/*

[Grogman]

Hi, I'm having the problem listed in this post. I've the following installed:

Cacti 0.8.6h
Cactid 0.8.6g
RRDTools 1.0.49-2

I have a lot of graphs working fine, but a week ago, all the graphs I create don't graph. I have looked this post, and I have seen that my poller_output table is not empty. I saw in the post a solution consisting in truncating the table manually. I have done so, and each time I truncate the poller_output table, all the graphs work, including the new ones. The problem is that after 2 or 3 polling cycles, the new graphs stop working again, and if I look in the poller_output table, it's not empty again.

I supose I'm missing something else that must be done apart of truncating the poller_output, in order to prevent further population of the table.

Can anybody help me?

Thanks to everybody

[TheWitness]

Two things could be happening:

1) PHP is running out of memory, please check and increase that value in php.ini

2) RRDtool is crashing. In that case, you will get a timeout in the poller with runtime exceeding 292 seconds. You should be able to watch the running processes and see if this is happening.

Please respond with your findings.

TheWitness

[Grogman]

1) I readed that PHP memory settings could create problems, and I changed the setting to 600 MB (maybe to much )

2) If I run cactid from CLI, it finishes in 30 seconds...

Could this problem be related with some scripts returning "bad" values?

Thanks for your help!

[TheWitness]

Not typically. Please watch your RRDtool process while the poller is running and verify that it completes without issues (aka poller_output table is empty after a polling cycle has finished). Also, review that you don't get any timeout's in the poller > 292 seconds. 128MBytes is more than enough, although I set to 256MBytes at my old work.

TheWitness

[Grogman]

Ok, it appears to be two separate issues malfunctioning in my cacti system. The first one is related to rrd files not being created sometimes, when I create new graphs. In that situation, poller_output is empty after the polling.

The second is that sometimes, that rrd files are created a lot of time later after creating the graph, but the graph is empty, and I only get NaN values. In that situations, poller_output isn't empty after the polling.

Now, old graps wich I created some time ago work fine, but I cannot create new graphs...

The polling takes around 30 seconds to finish.

Thanks a lot!

[Grogman]

Hi, for some reason, now I'm able to create graphs, and they appear to be working correctly. Nevertheless, poller_out is not empty, and it's growing with each graph I create, so I suppose data is being lost, and with time, poller_output will turn really big...

For some time, I believed that it was happening only with scripts, but snmp requests are also affected...


Thanks!

[TheWitness]

Dump your database and e-mail to me. Make sure it has some of those entries in it.

Thanks,

TheWitness