PHP Script Server timing out

[ARC1450]

I've got Cacti running on a Win2K8 box here at work, and within the past 6 months I'd say, I have been getting issues like this repeated in my log every 5 minutes:

Code: Select all

02/28/2012 10:05:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 21, Data Sources: traffic_in(DS[18630]), traffic_out(DS[18630]), traffic_in(DS[18631]), traffic_out(DS[18631]), traffic_in(DS[18632]), traffic_out(DS[18632]), traffic_in(DS[18633]), traffic_out(DS[18633]), ms_dhcp_free(DS[24835]), ms_dhcp_inuse(DS[24836]), ms_dhcp_free(DS[24837]), ms_dhcp_inuse(DS[24838]), ms_dhcp_free(DS[24839]), ms_dhcp_inuse(DS[24840]), ms_dhcp_free(DS[24841]), ms_dhcp_inuse(DS[24842]), users(DS[18294]), proc(DS[18295]), hdd_used(DS[18297]), hdd_total(DS[18297]), cpu(DS[18299]), Additional Issues Remain. Only showing first 20
02/28/2012 10:05:01 AM - POLLER: Poller[0] WARNING: There are '1' detected as overrunning a polling process, please investigate
02/28/2012 10:05:00 AM - SYSTEM STATS: Time:24.1687 Method:spine Processes:1 Threads:16 Hosts:493 HostsPerProcess:493 DataSources:48567 RRDsProcessed:0
02/28/2012 10:05:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/28/2012 10:04:24 AM - SYSTEM THOLD STATS: Time:9.8775 Tholds:2246 DownHosts:0
02/28/2012 10:04:14 AM - SYSTEM STATS: Time:49.0149 Method:spine Processes:1 Threads:16 Hosts:493 HostsPerProcess:493 DataSources:48567 RRDsProcessed:0
02/28/2012 10:03:59 AM - SPINE: Poller[0] ERROR: SS[0] PHP Script Server communications lost. Restarting PHP Script Server
02/28/2012 10:03:43 AM - SPINE: Poller[0] ERROR: SS[6] PHP Script Server communications lost. Restarting PHP Script Server
02/28/2012 10:03:12 AM - SYSTEM THOLD STATS: Time:9.9722 Tholds:2247 DownHosts:0
02/28/2012 10:03:02 AM - SYSTEM STATS: Time:46.6911 Method:spine Processes:1 Threads:16 Hosts:493 HostsPerProcess:493 DataSources:48567 RRDsProcessed:0
02/28/2012 10:02:01 AM - SYSTEM THOLD STATS: Time:10.2606 Tholds:2240 DownHosts:0
02/28/2012 10:01:51 AM - SYSTEM STATS: Time:46.0985 Method:spine Processes:1 Threads:16 Hosts:493 HostsPerProcess:493 DataSources:48567 RRDsProcessed:0
02/28/2012 10:00:52 AM - SYSTEM THOLD STATS: Time:3.9445 Tholds:982 DownHosts:0
02/28/2012 10:00:48 AM - SYSTEM STATS: Time:47.3869 Method:spine Processes:1 Threads:16 Hosts:493 HostsPerProcess:493 DataSources:48567 RRDsProcessed:0
02/28/2012 10:00:30 AM - SYSTEM THOLD STATS: Time:5.9562 Tholds:1382 DownHosts:0
02/28/2012 10:00:24 AM - SYSTEM STATS: Time:27.4136 Method:spine Processes:1 Threads:16 Hosts:493 HostsPerProcess:493 DataSources:48561 RRDsProcessed:0
02/28/2012 10:00:24 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/28/2012 10:00:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 21, Data Sources: traffic_out(DS[4897]), traffic_in(DS[4897]), traffic_out(DS[4898]), traffic_in(DS[4898]), traffic_out(DS[4899]), traffic_in(DS[4899]), traffic_out(DS[4900]), traffic_in(DS[4900]), traffic_out(DS[4901]), traffic_in(DS[4901]), traffic_out(DS[4902]), traffic_in(DS[4902]), traffic_out(DS[4903]), traffic_in(DS[4903]), traffic_out(DS[4904]), traffic_in(DS[4904]), traffic_out(DS[4905]), traffic_in(DS[4905]), traffic_out(DS[4906]), MIA(DS[26892]), UnSyncRTs(DS[26892]), Additional Issues Remain. Only showing first 20
02/28/2012 10:00:01 AM - POLLER: Poller[0] WARNING: There are '1' detected as overrunning a polling process, please investigate 

I have no clue how to hunt down why the script server is timing out. I have ran spine with '-V=3 -S -R' and isolated hosts that time out (all Windows boxes) and cranked down their timeouts to 1000ms. I have upped the connections allowed in my MySQL from 1000 to 4000 (due to having ~1200 Script Server scripts running).

Any thoughts?

[BSOD2600]

What versions of everything you running?

The Windows boxes which were timing out... most likely due to the SNMP disk query. It's a known inherent 'flaw' with the snmp agent. I bet those graphs are now missing data.

[ARC1450]

Code: Select all

Date 	Tue, 28 Feb 2012 13:16:01 -0500
Cacti Version 	0.8.7i
Cacti OS 	win32
SNMP Version 	NET-SNMP version: 5.5
RRDTool Version 	RRDTool 1.2.x
Hosts 	522
Graphs 	25245
Data Sources 	Script/Command: 5
SNMP: 542
SNMP Query: 24558
Script - Script Server (PHP): 264
Script Query - Script Server: 905
Total: 26274
Poller Information
Interval 	60
Type 	SPINE 0.8.7i Copyright 2002-2011 by The Cacti Group
Items 	Action[0]: 48333
Action[2]: 1188
Total: 49521
Concurrent Processes 	1
Max Threads 	16
PHP Servers 	10
Script Timeout 	45
Max OID 	20
Last Run Statistics 	Time:47.1283 Method:spine Processes:1 Threads:16 Hosts:493 HostsPerProcess:493 DataSources:48557 RRDsProcessed:0
PHP Information
PHP Version 	5.2.14
PHP OS 	WINNT
PHP uname 	Windows NT CACTI 6.1 build 7601
PHP SNMP 	Installed
max_execution_time 	45
memory_limit 	3G

And they never timed out. . .they just took up to 9 seconds to respond. That said, I checked the disk graphs and they were there and filled in.

Now that you mention it, though, I did try and get HMIB working to no avail. Maybe I'll try and replace the old scripts that you were supposed to redo with HMIB.

[edit]Which reminds me. . .is the default for Host CPU and Host Disk to be an indexed SNMP query?

[BSOD2600]

With HMIB, make sure you follow the setup instructions. I didn't, went off to work on other projects, and week later noticed my disk/cpu graphs were empty, doh!

Default install:
SNMP - Get Processor Information = <path_cacti>/resource/script_server/host_cpu.xml
SNMP - Get Mounted Partitions = <path_cacti>/resource/script_server/host_disk.xml

...unless you're using the pure snmp implementation Gandalf implements in the documentation site.

[ARC1450]

With HMIB, make sure you follow the setup instructions. I didn't, went off to work on other projects, and week later noticed my disk/cpu graphs were empty, doh!

Default install:
SNMP - Get Processor Information = <path_cacti>/resource/script_server/host_cpu.xml
SNMP - Get Mounted Partitions = <path_cacti>/resource/script_server/host_disk.xml

...unless you're using the pure snmp implementation Gandalf implements in the documentation site.

Yeah, I thought I followed the setup instructions, but I got the same results. . .blank graphs. Granted, I tried to switch over existing servers, so it might only work with new stuff.

Not sure.

[ARC1450]

By the way, this got resolved by wiping out all the HOST data sources that used the Script Server.

The moment I take out most of the script server stuff, polling drops by 10 seconds or more. Is there any way to avoid this? Right now, my Chassis Switchport Usage uses the script server to query the MySQL server to pull capacity usage of my access switches, and then there is the SNMP Get Processor Script Server script. Is there an SNMP version of the CPU info (yes, I'm checking and searching for one).

Or, if anyone knows of a way to speed up the Script Server, that would be awesome.

[BSOD2600]

TheWitness, BorisL, etc -- those who run really big installations will probably have some good ideas for you to explore. I'm not much use in this department :-/