After upgrade cacti/cactid - poller not processing all RRDs

[aleto]

What is your poller interval? What is your cron interval?

TheWitness

Hi,

Thanks for your reply.

My cron interval is 5 mins, my polling interval is 1 min

HTHH,

aleto

[mgeni]

I have similar problem, any Solutions?

[Tomeq]

I have similar problem, any Solutions?

We had/and now have similar problem. It looks like that there is too much i/o's when updating the rrd files.

We had the problem, but switched to a more modern hardware and the problem was gone. For some time, until much more data sources and rrds arrived. Now, the server is writing rrds for 3-4 minutes (it is Xeon, with good and fast RAID set up) one minute more and the problem will back.


Greets,
Tomek

[yardus9]

I am having a similar problem with spine.

I have 2 cacti servers, both CentOS. I recently upgraded one from 0.86i with cactid to 0.87 with spine0.87b without a problem.
However, after I upgraded my second server, when I switch from cactid to spine (by changing the path), I get this problem where some of my RRDs fail to update. This second server has the plugin architecture installed, the first one did not. I have changed the LIMIT in lib/poller.php to 50000, and my php.ini memory lilmit is 64MB.
I rebuilt the poller cache, did not seem to help.

Here is a look at my cacti stats, polling looks normal, however, the number of updated RRDs is short by about 1500 when I use the spine poller:

01/11/2008 04:05:46 PM - SYSTEM STATS: Time:43.3462 Method:spine Processes:4 Threads:10 Hosts:441 HostsPerProcess:111 DataSources:13378 RRDsProcessed:8328
01/11/2008 04:06:02 PM - RECACHE STATS: RecacheTime:15.0207 HostsRecached:2
01/11/2008 04:10:41 PM - SYSTEM STATS: Time:39.0059 Method:spine Processes:4 Threads:10 Hosts:441 HostsPerProcess:111 DataSources:13378 RRDsProcessed:8328
01/11/2008 04:10:55 PM - RECACHE STATS: RecacheTime:13.7780 HostsRecached:3
01/11/2008 04:19:59 PM - SYSTEM STATS: Time:295.2987 Method:spine Processes:4 Threads:10 Hosts:441 HostsPerProcess:111 DataSources:13378 RRDsProcessed:6721
01/11/2008 04:20:17 PM - RECACHE STATS: RecacheTime:18.1796 HostsRecached:2
01/11/2008 04:24:59 PM - SYSTEM STATS: Time:295.2255 Method:spine Processes:4 Threads:10 Hosts:441 HostsPerProcess:111 DataSources:13377 RRDsProcessed:6721
01/11/2008 04:25:37 PM - SYSTEM STATS: Time:33.4695 Method:spine Processes:4 Threads:10 Hosts:441 HostsPerProcess:111 DataSources:13378 RRDsProcessed:8317
01/11/2008 04:25:49 PM - RECACHE STATS: RecacheTime:11.8979 HostsRecached:2


The poll seems to fail on the more recently added hosts. When i run spine on one of these hosts from the command line, it runs fine.

Any ideas??
Would this patch be applicable on this version:
http://www.cacti.net/downloads/patches/ ... nder.patch

[khufure]

Not sure if it applies to all but I was able to resolve a similar problem using http://forums.cacti.net/viewtopic.php?p=124130#124130.

Regards,
P

[yardus9]

I tried both the suggestions from that Post, neither seems to have solved my problem.

Anybody else have any ideas?

I ran spine from the command line via strace. Can't really make heads or tails of the results. Seems to be cruising along, then tons of this:
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
gettimeofday({1201806333, 641847}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL)

[leonk]

I tried both the suggestions from that Post, neither seems to have solved my problem.

Anybody else have any ideas?

I ran spine from the command line via strace. Can't really make heads or tails of the results. Seems to be cruising along, then tons of this:
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
gettimeofday({1201806333, 641847}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL) = 0
nanosleep({0, 400000}, NULL)

ditto... Having the same problem.. Have tried all fixes mentioned.

[yardus9]

I just built a new box from scratch.
CentOS5
cacti-0.8.7b
spine0.8.7b
mysql-5.0.22
php-5.1.6

Imported my database, and am still getting the same problem, over half my RRDs are not getting updated.

03/13/2008 06:49:56 PM - SYSTEM STATS: Time:295.0487 Method:spine Processes:4 Threads:10 Hosts:461 HostsPerProcess:116 DataSources:14676 RRDsProcessed:5967
03/13/2008 06:49:59 PM - RECACHE STATS: RecacheTime:3.1101 HostsRecached:1

I am getting a ton of these errors in the DEBUG log:
03/13/2008 06:42:57 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_id,end_time from poller_time where poller_id=0"
03/13/2008 06:42:57 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name) LIMIT 10000"


I had the plugin architecture installed in a previous version while using this same database. Could that be causing a problem? Is there something in the database that I can strip out?

Any other ideas?

[davromaniak]

I just built a new box from scratch.
CentOS5
cacti-0.8.7b
spine0.8.7b
mysql-5.0.22
php-5.1.6

Imported my database, and am still getting the same problem, over half my RRDs are not getting updated.

03/13/2008 06:49:56 PM - SYSTEM STATS: Time:295.0487 Method:spine Processes:4 Threads:10 Hosts:461 HostsPerProcess:116 DataSources:14676 RRDsProcessed:5967
03/13/2008 06:49:59 PM - RECACHE STATS: RecacheTime:3.1101 HostsRecached:1

I am getting a ton of these errors in the DEBUG log:
03/13/2008 06:42:57 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_id,end_time from poller_time where poller_id=0"
03/13/2008 06:42:57 PM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name) LIMIT 10000"


I had the plugin architecture installed in a previous version while using this same database. Could that be causing a problem? Is there something in the database that I can strip out?

Any other ideas?

I have the same problem.

I'm using cacti 0.8.7a and spine 0.8.7a on a Debian Etch, with PHP 5.2.0 and MySQL 5.0.32.

The SNMP requests are working very well, but only 452 out of 3006 RRD files are updated.

The RRD update worked when cacti was not updated, and worked under 0.8.6j version.

Regards.

[matguitar]

Hi,
I have the same problem: the gathering of the snmp data works perfect when collecting with cacti (0.8.7b) itself. But when switching to spine (0.8.7c) only a few results are being updated to the RRD DB so the graphs are looking quite ragged.
I'm running 64 bit Redhat 2.6.18-53.el5 #1 SMP x86_64 GNU/Linux
Please solve this problem!

[gandalf]

Hi,
I have the same problem: the gathering of the snmp data works perfect when collecting with cacti (0.8.7b) itself. But when switching to spine (0.8.7c) only a few results are being updated to the RRD DB so the graphs are looking quite ragged.
I'm running 64 bit Redhat 2.6.18-53.el5 #1 SMP x86_64 GNU/Linux
Please solve this problem!

Hey matguitar! You're not really far away from me. We should have a beer instead of discussing boring IT stuff, then, hehe.
For me, spine 087c is broken. So I use 087b ATM. cmd.php is no option for you? Larry will address known problems, but this may still take some days
Reinhard

[faman]

Deleting all UCD 10 FS graphs actually helped me. Times went from constant timeouts after 50s from the poller, with the assosciated db insert errors, to sub 10s poller runs against 50 or so hosts in one of our networks. I did the same in another network with 87 hosts and poller times went from timout to about 20s per run.

These errors started happening when I went from 0.8.6j/cactid to cacti 0.8.7b and spine 0.8.7a. Under 0.8.6 I was using a 1 minute poll interval and didn't have issues.

[Aiyaya]

I tried using Cactid 0.8.6i with my Cacti 0.8.6h installation and I do not experience the problem. Looks like it's just limited to Cacti 0.8.6i.

[gandalf]

086 should no longer be used, by no means
R.