Cacti dying out

[jkukis]

We recently setup a new server and moved over our Cacti data to a fresh install of Cacti version 0.8.7d, RRDtool 1.3, NET-SNMP version: 5.3.2.2 , Spine .8.7c, PHP Version 5.1.6
Cacti was installed from a package, not from source.

Previously we were running Cacti .8.6.

We are experiencing the following errors, and I have been able to resolve them after going through past forum posts .

07/22/2009 03:56:07 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
07/22/2009 03:56:07 PM - SPINE: Poller[0] FATAL: Connection Failed, Error:'1129', Message:'Host 'servername' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'' (Spine thread)

POLLER: Poller[0] WARNING: Cron is out of sync with the Poller Interval! The Poller Interval is '300' seconds, with a maximum of a '300' second Cron, but 325 seconds have passed since the last poll!

and some messages about SNMP timeouts for some(seemingly random) hosts.

*I have checked that there is only one CRON entry(I've tried adding the --force to the cron entry based on a forum post I found but it seems to make no difference):
*/5 * * * * cacti php /var/www/cacti/poller.php --force &>/dev/null

*My Poller and Cron intervals are both set to 5 minutes, I have also rebuilt the Poller cache.

*My Poller settings are:
Concurrent Processes: 2
Max Threads: 25
Script Timeout: 25
Max OID:10 ( Reduced this from 25 originally, might have reduced the frequency of our timeouts, but has not removed them)


Does anyone have any suggestions on what to try, how to trace this down better? I've turned on debugging, and didn't see anything significant, although when debugging was on Cacti was running OK.

Here are the last 5 run stats from the log:
07/24/2009 11:07:13 AM - SYSTEM STATS: Time:130.1447 Method:spine Processes:2 Threads:25 Hosts:197 HostsPerProcess:99 DataSources:22667 RRDsProcessed:11873
07/24/2009 11:01:58 AM - SYSTEM STATS: Time:114.8550 Method:spine Processes:2 Threads:25 Hosts:197 HostsPerProcess:99 DataSources:22667 RRDsProcessed:11873
07/24/2009 10:55:57 AM - SYSTEM STATS: Time:54.7781 Method:spine Processes:2 Threads:25 Hosts:197 HostsPerProcess:99 DataSources:22667 RRDsProcessed:11873
07/24/2009 10:50:49 AM - SYSTEM STATS: Time:46.9483 Method:spine Processes:2 Threads:25 Hosts:197 HostsPerProcess:99 DataSources:22667 RRDsProcessed:11873
07/24/2009 10:45:39 AM - SYSTEM STATS: Time:36.9977 Method:spine Processes:2 Threads:25 Hosts:197 HostsPerProcess:99 DataSources:22667 RRDsProcessed:11873

[BSOD2600]

Cacti and spine 0.8.7e are now the latest.

Have you increased the number of connections for MySQL?
I thought mysql had some additional debugging that could be turned on, so you could see what type of connection errors it's getting.

[jkukis]

MySQL connections were upped to 500. I'll try the MySQL debugging and see if we can catch it failing.