cacti 0.8.6i graphing intermittendly

[ns]

Ever since I added a lot of graphs, I noticed that cacti will only graph intermittently (see attached) for some of my graphs. Any idea why this would be?

Here's the info your looking for:
cacti/cactid 0.8.6i
FreeBSD 6.1
PHP 5.2.0

php.ini:
memory_limit = 80M

end of cacti.log:

Code: Select all

11/18/2006 02:24:06 PM - SYSTEM STATS: Time:120.9854 Method:cactid Processes:1 Threads:1 Hosts:136 HostsPerProcess:136 DataSources:26941 RRDsProcessed:11774
11/18/2006 02:24:06 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "replace into settings (name,value) values ('stats_poller','Time:120.9854 Method:cactid Processes:1 Threads:1 Hosts:136 HostsPerProcess:136 DataSources:26941 RRDsProcessed:11774')"
11/18/2006 02:24:06 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "truncate table poller_output"
11/18/2006 02:24:06 PM - CMDPHP: Poller[0] DEBUG: SQL Cell: "select count(*) from poller_command"

Keep in mind that although it says "RRDsProcessed:11774" there are only 9959 RRD files in /rrd. This is up from ~3000 files 10 hours ago. Like I said earlier, it's only updating some of the RRD's intermittendly.

I also noticed that poller_output had around ~20k entries in it when I wrote this response. I've cleaned it out but it didn't help. For whatever reason, cacti isn't always cleaning it out (although it usually does).

Let me know what other info you need. I'll be checking the forums often.
Thanks.

[TheWitness]

NS, please look for a patch at the following location. There was a problem introduced by the poller in 0.8.6i.

TheWitness

http://www.cacti.net/download_patches.php

[Jeb]

ns, is the patch working for you ?
I have the same problem and the patch seems to not work for me..

[TheWitness]

Jeb,

Did you do what I told you to?

TheWitness

[Jeb]

ns, after patching you will have to change "limit 10000" by "$limit" on line 240 of lib/poller.php for the patch to work

[TheWitness]

The lib/poller.php in SVN Branch 0.8.6 now also has the updated file.

TheWitness

[ns]

Thanks for responding so fast. I patched the files and changed line 240 on lib/poller.php from

limit 10000
to
$limit

I'm going to let the poller run overnight, and I'll let you know what happens.

[ns]

Doesn't look like it worked.

Any other ideas?

[TheWitness]

You likely have 2 cron's running.

TheWitness

[Jeb]

ns, take a look to the logs and see if you get the
SYSTEM STATS: Time:105.7410 Method:cactid Processes:2 Threads:5 Hosts:325 HostsPerProcess:163 DataSources:26539 RRDsProcessed:13763
line.
If you don't get it, it's because poller.php take too much time to update rrd and exit before the end. (i got this problem..)

In this case you may try my patch at http://bugs.cacti.net/view.php?id=868
(in note 0002071)

[ns]

First off, it looks like this person is having the exact same problem as I am:
http://forums.cacti.net/viewtopic.php?t ... hlight=rrd

But to answer your questions:

Only one cron running:

Code: Select all

# ps -A | grep cron
531  ??  Is     0:01.25 /usr/sbin/cron -s
19305  p1  R+     0:00.00 grep cron

Here's the last couple lines from the log:

Code: Select all

11/20/2006 12:56:07 PM - CMDPHP: Poller[0] DEBUG: SQL Row: "select  data_template_rrd.data_source_name,  data_template_data.name  from (data_template_rrd,data_template_data)  where data_template_rrd.local_data_id=data_template_data.local_data_id  and data_template_rrd.id=19719"

11/20/2006 12:56:07 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/bin/rrdtool create ****cacti/rra/****9549.rrd   --step 300   DS:cisco_free_mem:GAUGE:600:0:4600000000  RRA:AVERAGE:0.5:1:600  RRA:AVERAGE:0.5:2:600  RRA:AVERAGE:0.5:6:700  RRA:AVERAGE:0.5:12:700  RRA:AVERAGE:0.5:24:775  RRA:AVERAGE:0.5:48:775  RRA:AVERAGE:0.5:288:797  RRA:AVERAGE:0.5:576:797  RRA:MIN:0.5:1:600  RRA:MIN:0.5:2:600  RRA:MIN:0.5:6:700  RRA:MIN:0.5:12:700  RRA:MIN:0.5:24:775  RRA:MIN:0.5:48:775  RRA:MIN:0.5:288:797  RRA:MIN:0.5:576:797  RRA:MAX:0.5:1:600  RRA:MAX:0.5:2:600  RRA:MAX:0.5:6:700  RRA:MAX:0.5:12:700  RRA:MAX:0.5:24:775  RRA:MAX:0.5:48:775  RRA:MAX:0.5:288:797  RRA:MAX:0.5:576:797  RRA:LAST:0.5:1:600  RRA:LAST:0.5:2:600  RRA:LAST:0.5:6:700  RRA:LAST:0.5:12:700  RRA:LAST:0.5:24:775  RRA:LAST:0.5:48:775  RRA:LAST:0.5:288:797  RRA:LAST:0.5:576:797
11/20/2006 12:56:07 PM - POLLER: Poller[0] CACTI2RRD: /usr/local/bin/rrdtool update ****cacti/rra/****9549.rrd --template cisco_free_mem N:U
11/20/2006 12:56:07 PM - SYSTEM STATS: Time:66.1401 Method:cactid Processes:1 Threads:5 Hosts:136 HostsPerProcess:136 DataSources:26941 RRDsProcessed:11760
11/20/2006 12:56:07 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "replace into settings (name,value) values ('stats_poller','Time:66.1401 Method:cactid Processes:1 Threads:5 Hosts:136 HostsPerProcess:136 DataSources:26941 RRDsProcessed:11760')"
11/20/2006 12:56:07 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "truncate table poller_output"
11/20/2006 12:56:07 PM - CMDPHP: Poller[0] DEBUG: SQL Cell: "select count(*) from poller_command"

So it definitely finishes up polling in time. Should I be seeing something in the log like the "OK: 4 5 1" or whatever it is when rrdtool says it created the graph?

[gandalf]

Please read my "NaN Debugging" howto, linked in my signature. Please feel free to comment or extend it.
Reinhard

[ns]

I read the NaN guide before I initially posted, and I'm afraid it doesn't solve the problem. I've been following this thread which seems to be having the same trouble as I am:
http://forums.cacti.net/viewtopic.php?t ... hlight=rrd

The suggestion there is to revert back to cacti 8.6h, but I would rather use 8.6i . If anyone has any suggestions on how to make cacti work with 8.6i over 10,000 Data Sources please let me know, although It's sounding to me like its a bug in cacti 0.8.6i.

[TheWitness]

I think you are correct. However, I need to you verify by backing out only the following 2 files:

<path_cacti>/poller.php
<path_cacti>/lib/poller.php

Report back your findings.

TheWitness

[ns]

Just to be clear, by backing out you want me to try using

<path_cacti>/poller.php
<path_cacti>/lib/poller.php

from 0.8.6h and leave the rest from 0.8.6i right? I'm assuming this will be compatible with my current 0.8.6i cacti database (I can't remember if there was any differences in the database structure from 8.6h->8.6i).