All of a sudden, two environments out of 6 large installs of cacti have resulted in this error :
Code: Select all
06/01/2009 03:00:00 PM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
06/01/2009 03:00:00 PM - SYSTEM STATS: Time:299.2690 Method:spine Processes:1 Threads:15 Hosts:784 HostsPerProcess:784 DataSources:15783 RRDsProcessed:13255
06/01/2009 03:00:01 PM - SPINE: Poller[0] ERROR: Spine Timed Out While Processing Hosts Internal
06/01/2009 03:00:23 PM - SYSTEM STATS: Time:22.5015 Method:spine Processes:1 Threads:15 Hosts:784 HostsPerProcess:784 DataSources:15783 RRDsProcessed:11755
06/01/2009 03:05:01 PM - SPINE: Poller[0] ERROR: Spine Timed Out While Processing Hosts Internal
06/01/2009 03:05:02 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 2620, Data Sources: traffic_in(DS[11461]), traffic_out(DS[11461]), traffic_in(DS[11462]), traffic_out(DS[11462]), traffic_in(DS[11463]), traffic_out(DS[11463]), traffic_in(DS[11464]), traffic_out(DS[11464]), traffic_in(DS[11465]), traffic_out(DS[11465]), traffic_in(DS[11466]), traffic_out(DS[11466]), traffic_in(DS[11467]), traffic_out(DS[11467]), traffic_in(DS[11468]), traffic_out(DS[11468]), traffic_in(DS[11469]), traffic_out(DS[11469]), traffic_in(DS[11470]), traffic_out(DS[11470]), traffic_in(DS[11471]), Additional Issues Remain. Only showing first 20
Code: Select all
mysql> select count(*) from poller_output;
+----------+
| count(*) |
+----------+
| 0 |
+----------+
1 row in set (0.00 sec)
# php -q poller_output_empty.php
There were 0, RRD updates made this pass
Nothing has changed system configuration wise, and all other environments are exactly identical and are working fine (and are actually 3x larger as well).
Seems as if its the "double poller issue" yet the poller isn't actually being called twice, which I've tripled checked.
Thoughts?
Code: Select all
Technical Support
General Information
Date Mon, 01 Jun 2009 15:02:08 -0400
Cacti Version 0.8.7d
Cacti OS unix
SNMP Version NET-SNMP version: 5.3.1
RRDTool Version RRDTool 1.3.x
Hosts 808
Graphs 6687
Data Sources Script/Command: 26
SNMP: 11289
SNMP Query: 2426
Total: 13741
Poller Information
Interval 300
Type spine
Items Action[0]: 15757
Action[1]: 26
Total: 15783
Concurrent Processes 1
Max Threads 15
PHP Servers 1
Script Timeout 25
Max OID 10
Last Run Statistics Time:21.8129 Method:spine Processes:1 Threads:15 Hosts:784 HostsPerProcess:784 DataSources:15783 RRDsProcessed:11811
PHP Information
PHP Version 5.1.6
PHP OS Linux
PHP uname Linux r21119u04 2.6.18-92.1.22.el5 #1 SMP Tue Dec 16 11:57:43 EST 2008 x86_64
PHP SNMP Installed
max_execution_time 30
memory_limit 850M
spine 0.8.7c
Seems as if the poller finishes, then doesn't exit properly?
Code: Select all
06/01/2009 03:10:19 PM - SPINE: Poller[0] Host[770] DS[12347] SNMP: v1: host1, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.25, value: 2936684301
06/01/2009 03:10:19 PM - SPINE: Poller[0] Host[770] DS[12347] SNMP: v1: host1, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.25, value: 2503128661
06/01/2009 03:10:22 PM - SYSTEM STATS: Time:21.2385 Method:spine Processes:1 Threads:15 Hosts:784 HostsPerProcess:784 DataSources:15783 RRDsProcessed:11083
Code: Select all
apache 1752 90.2 0.0 416208 23524 ? Sl 15:10 0:39 /usr/local/spine/bin/spine 0 792
# strace -p 1752
Process 1752 attached - interrupt to quit
restart_syscall(<... resuming interrupted call ...>) = 0
nanosleep({0, 5000000}, NULL) = 0
nanosleep({0, 5000000}, NULL) = 0
nanosleep({0, 5000000}, NULL) = 0
nanosleep({0, 5000000}, NULL) = 0
nanosleep({0, 5000000}, NULL) = 0
nanosleep({0, 5000000}, NULL) = 0
nanosleep({0, 5000000}, NULL) = 0
