Cacti poller sometimes gives errors

[hid3nax]

Hello everyone.

After realising that I can't see all the traffic for my gigabit interfaces, I decided to install the Fix64bit plugin (http://docs.cacti.net/userplugin:fix64bit) and deleted the old network interface graphs. I created new ones under In/Out Bytes (64-bit Counters) template.

After doing that, *sometimes* errors appear in cacti.log and poller fails to gather statistics. E.g.:

01/12/2012 01:17:09 PM - SYSTEM STATS: Time:7.5310 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 01:12:09 PM - SYSTEM STATS: Time:7.6040 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 01:12:02 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 3, Data Sources: hdd_total(DS[60]), hdd_used(DS[60]), traffic_out(DS[183])
01/12/2012 01:07:09 PM - SYSTEM STATS: Time:7.7546 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:157
01/12/2012 01:07:09 PM - CMDPHP: Poller[0] Host[4] DS[61] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 01:07:09 PM - CMDPHP: Poller[0] Host[4] DS[61] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 01:07:09 PM - CMDPHP: Poller[0] Host[4] DS[60] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 01:07:09 PM - CMDPHP: Poller[0] Host[4] DS[60] WARNING: Result from SERVER not valid. Partial Result: 01/12/2012 01:07:08
01/12/2012 01:07:08 PM - PHPSVR: Poller[0] Maximum runtime of 300 seconds exceeded for the Script Server. Exiting.
01/12/2012 01:02:09 PM - SYSTEM STATS: Time:7.6214 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 12:57:18 PM - SYSTEM STATS: Time:17.5617 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 12:57:00 PM - POLLER: Poller[0] WARNING: There are '1' detected as overrunning a polling process, please investigate
01/12/2012 12:57:00 PM - SYSTEM STATS: Time:298.1449 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:131
01/12/2012 12:57:00 PM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
01/12/2012 12:47:10 PM - SYSTEM STATS: Time:9.4233 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145

01/12/2012 12:42:10 PM - SYSTEM STATS: Time:8.9640 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 12:37:09 PM - SYSTEM STATS: Time:7.8778 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 12:37:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 21, Data Sources: (DS[121]), users(DS[122]), cpu_user(DS[126]), load_1min(DS[127]), load_15min(DS[128]), load_5min(DS[129]), uptime(DS[130]), ping(DS[131]), hdd_total(DS[132]), hdd_used(DS[132]), (DS[137]), users(DS[138]), proc(DS[139]), cpu_nice(DS[140]), cpu_system(DS[141]), cpu_user(DS[142]), load_1min(DS[143]), load_15min(DS[144]), load_5min(DS[145]), uptime(DS[146]), ping(DS[147]), Additional Issues Remain. Only showing first 20
01/12/2012 12:32:57 PM - CMDPHP: Poller[0] Host[9] DS[170] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 12:32:57 PM - CMDPHP: Poller[0] Host[9] DS[170] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 12:32:57 PM - CMDPHP: Poller[0] Host[9] DS[149] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 12:32:57 PM - CMDPHP: Poller[0] Host[9] DS[149] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 12:32:57 PM - CMDPHP: Poller[0] Host[9] DS[148] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 12:32:57 PM - CMDPHP: Poller[0] Host[9] DS[148] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 12:32:57 PM - CMDPHP: Poller[0] Host[9] DS[137] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 12:32:57 PM - CMDPHP: Poller[0] Host[8] DS[132] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 12:32:57 PM - CMDPHP: Poller[0] Host[8] DS[132] WARNING: Result from SERVER not valid. Partial Result: 01/12/2012 12:32:57
01/12/2012 12:32:57 PM - PHPSVR: Poller[0] Maximum runtime of 300 seconds exceeded for the Script Server. Exiting.
01/12/2012 12:32:09 PM - SYSTEM STATS: Time:7.6315 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 12:27:16 PM - SYSTEM STATS: Time:15.8311 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 12:27:16 PM - CMDPHP: Poller[0] Host[8] DS[131] WARNING: Result from CMD not valid. Partial Result: U
01/12/2012 12:27:00 PM - POLLER: Poller[0] WARNING: There are '1' detected as overrunning a polling process, please investigate
01/12/2012 12:27:00 PM - SYSTEM STATS: Time:298.4964 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:115
01/12/2012 12:27:00 PM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
01/12/2012 12:17:09 PM - SYSTEM STATS: Time:7.6038 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 12:12:09 PM - SYSTEM STATS: Time:7.5066 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 12:07:09 PM - SYSTEM STATS: Time:7.4746 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 12:02:10 PM - SYSTEM STATS: Time:7.8736 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 11:57:10 AM - SYSTEM STATS: Time:8.2079 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 11:52:10 AM - SYSTEM STATS: Time:7.8537 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 11:47:12 AM - SYSTEM STATS: Time:10.1127 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145

01/12/2012 11:07:09 AM - SYSTEM STATS: Time:7.6931 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 11:02:16 AM - SYSTEM STATS: Time:15.4300 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 11:02:16 AM - CMDPHP: Poller[0] Host[8] DS[131] WARNING: Result from CMD not valid. Partial Result: U
01/12/2012 11:02:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 21, Data Sources: (DS[121]), users(DS[122]), cpu_user(DS[126]), load_1min(DS[127]), load_15min(DS[128]), load_5min(DS[129]), uptime(DS[130]), ping(DS[131]), hdd_total(DS[132]), hdd_used(DS[132]), (DS[137]), users(DS[138]), proc(DS[139]), cpu_nice(DS[140]), cpu_system(DS[141]), cpu_user(DS[142]), load_1min(DS[143]), load_15min(DS[144]), load_5min(DS[145]), uptime(DS[146]), ping(DS[147]), Additional Issues Remain. Only showing first 20
01/12/2012 10:57:57 AM - CMDPHP: Poller[0] Host[9] DS[170] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:57:57 AM - CMDPHP: Poller[0] Host[9] DS[170] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:57:57 AM - CMDPHP: Poller[0] Host[9] DS[149] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:57:57 AM - CMDPHP: Poller[0] Host[9] DS[149] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:57:57 AM - CMDPHP: Poller[0] Host[9] DS[148] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:57:57 AM - CMDPHP: Poller[0] Host[9] DS[148] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:57:57 AM - CMDPHP: Poller[0] Host[9] DS[137] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:57:57 AM - CMDPHP: Poller[0] Host[8] DS[132] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:57:57 AM - CMDPHP: Poller[0] Host[8] DS[132] WARNING: Result from SERVER not valid. Partial Result: 01/12/2012 10:57:56
01/12/2012 10:57:56 AM - PHPSVR: Poller[0] Maximum runtime of 300 seconds exceeded for the Script Server. Exiting.
01/12/2012 10:57:09 AM - SYSTEM STATS: Time:7.5764 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 10:57:02 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 21, Data Sources: (DS[83]), cpu_nice(DS[86]), cpu_system(DS[87]), cpu_user(DS[88]), load_1min(DS[89]), load_15min(DS[90]), load_5min(DS[91]), uptime(DS[92]), ping(DS[93]), (DS[94]), (DS[95]), hdd_total(DS[96]), hdd_used(DS[96]), hdd_total(DS[97]), hdd_used(DS[97]), hdd_total(DS[98]), hdd_used(DS[98]), hdd_total(DS[99]), hdd_used(DS[99]), hdd_total(DS[100]), hdd_used(DS[100]), Additional Issues Remain. Only showing first 20
01/12/2012 10:52:57 AM - CMDPHP: Poller[0] Host[6] DS[102] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:52:57 AM - CMDPHP: Poller[0] Host[6] DS[102] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:52:57 AM - CMDPHP: Poller[0] Host[6] DS[101] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:52:57 AM - CMDPHP: Poller[0] Host[6] DS[101] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:52:57 AM - CMDPHP: Poller[0] Host[6] DS[100] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:52:57 AM - CMDPHP: Poller[0] Host[6] DS[100] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:52:57 AM - CMDPHP: Poller[0] Host[6] DS[99] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:52:57 AM - CMDPHP: Poller[0] Host[6] DS[99] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:52:57 AM - CMDPHP: Poller[0] Host[6] DS[98] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:52:57 AM - CMDPHP: Poller[0] Host[6] DS[98] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:52:57 AM - CMDPHP: Poller[0] Host[6] DS[97] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:52:57 AM - CMDPHP: Poller[0] Host[6] DS[97] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:52:57 AM - CMDPHP: Poller[0] Host[6] DS[96] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:52:57 AM - CMDPHP: Poller[0] Host[6] DS[96] WARNING: Result from SERVER not valid. Partial Result: 01/12/2012 10:52:56
01/12/2012 10:52:56 AM - PHPSVR: Poller[0] Maximum runtime of 300 seconds exceeded for the Script Server. Exiting.
01/12/2012 10:52:08 AM - SYSTEM STATS: Time:7.5916 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 10:52:00 AM - POLLER: Poller[0] WARNING: There are '1' detected as overrunning a polling process, please investigate
01/12/2012 10:52:00 AM - SYSTEM STATS: Time:298.4931 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:115
01/12/2012 10:52:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
01/12/2012 10:47:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 2, Data Sources: hdd_total(DS[79]), hdd_used(DS[79])
01/12/2012 10:47:01 AM - POLLER: Poller[0] WARNING: There are '1' detected as overrunning a polling process, please investigate
01/12/2012 10:47:01 AM - SYSTEM STATS: Time:298.8302 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:172
01/12/2012 10:47:01 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
01/12/2012 10:42:57 AM - CMDPHP: Poller[0] Host[5] DS[81] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:42:57 AM - CMDPHP: Poller[0] Host[5] DS[81] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:42:57 AM - CMDPHP: Poller[0] Host[5] DS[80] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:42:57 AM - CMDPHP: Poller[0] Host[5] DS[80] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:42:57 AM - CMDPHP: Poller[0] Host[5] DS[79] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:42:57 AM - CMDPHP: Poller[0] Host[5] DS[79] WARNING: Result from SERVER not valid. Partial Result: 01/12/2012 10:42:56
01/12/2012 10:42:56 AM - CMDPHP: Poller[0] Host[9] DS[170] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:42:56 AM - CMDPHP: Poller[0] Host[9] DS[170] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:42:56 AM - CMDPHP: Poller[0] Host[9] DS[149] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:42:56 AM - CMDPHP: Poller[0] Host[9] DS[149] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:42:56 AM - CMDPHP: Poller[0] Host[9] DS[148] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:42:56 AM - CMDPHP: Poller[0] Host[9] DS[148] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:42:56 AM - CMDPHP: Poller[0] Host[9] DS[137] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:42:56 AM - CMDPHP: Poller[0] Host[8] DS[132] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:42:56 AM - CMDPHP: Poller[0] Host[8] DS[132] WARNING: Result from SERVER not valid. Partial Result: 01/12/2012 10:42:56
01/12/2012 10:42:56 AM - PHPSVR: Poller[0] Maximum runtime of 300 seconds exceeded for the Script Server. Exiting.
01/12/2012 10:42:56 AM - PHPSVR: Poller[0] Maximum runtime of 300 seconds exceeded for the Script Server. Exiting.
01/12/2012 10:42:02 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 21, Data Sources: (DS[66]), users(DS[67]), proc(DS[68]), cpu_nice(DS[69]), cpu_system(DS[70]), cpu_user(DS[71]), load_1min(DS[72]), load_15min(DS[73]), load_5min(DS[74]), uptime(DS[75]), ping(DS[76]), (DS[77]), (DS[78]), hdd_total(DS[79]), hdd_used(DS[79]), hdd_total(DS[80]), hdd_used(DS[80]), hdd_total(DS[81]), hdd_used(DS[81]), traffic_in(DS[181]), traffic_out(DS[181]), Additional Issues Remain. Only showing first 20
01/12/2012 10:38:25 AM - AUTH LOGIN: User 'admin' Authenticated
01/12/2012 10:37:57 AM - CMDPHP: Poller[0] Host[5] DS[81] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:37:57 AM - CMDPHP: Poller[0] Host[5] DS[81] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:37:57 AM - CMDPHP: Poller[0] Host[5] DS[80] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:37:57 AM - CMDPHP: Poller[0] Host[5] DS[80] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:37:57 AM - CMDPHP: Poller[0] Host[5] DS[79] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:37:57 AM - CMDPHP: Poller[0] Host[5] DS[79] WARNING: Result from SERVER not valid. Partial Result: 01/12/2012 10:37:56
01/12/2012 10:37:56 AM - PHPSVR: Poller[0] Maximum runtime of 300 seconds exceeded for the Script Server. Exiting.
01/12/2012 10:37:08 AM - SYSTEM STATS: Time:7.7351 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 10:37:00 AM - POLLER: Poller[0] WARNING: There are '2' detected as overrunning a polling process, please investigate
01/12/2012 10:37:00 AM - SYSTEM STATS: Time:298.4902 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:98
01/12/2012 10:37:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
01/12/2012 10:32:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 3, Data Sources: hdd_total(DS[60]), hdd_used(DS[60]), traffic_out(DS[183])
01/12/2012 10:32:01 AM - POLLER: Poller[0] WARNING: There are '1' detected as overrunning a polling process, please investigate
01/12/2012 10:32:01 AM - SYSTEM STATS: Time:298.7086 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:140
01/12/2012 10:32:01 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
01/12/2012 10:27:09 AM - CMDPHP: Poller[0] Host[4] DS[61] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:27:09 AM - CMDPHP: Poller[0] Host[4] DS[61] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:27:09 AM - CMDPHP: Poller[0] Host[4] DS[60] WARNING: Result from SERVER not valid. Partial Result: U
01/12/2012 10:27:09 AM - CMDPHP: Poller[0] Host[4] DS[60] WARNING: Result from SERVER not valid. Partial Result: 01/12/2012 10:27:07
01/12/2012 10:27:07 AM - PHPSVR: Poller[0] Maximum runtime of 300 seconds exceeded for the Script Server. Exiting.
01/12/2012 10:22:10 AM - SYSTEM STATS: Time:8.1722 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 10:17:09 AM - SYSTEM STATS: Time:7.7907 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 10:17:01 AM - POLLER: Poller[0] WARNING: There are '1' detected as overrunning a polling process, please investigate
01/12/2012 10:17:01 AM - SYSTEM STATS: Time:298.9470 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:131
01/12/2012 10:17:01 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
01/12/2012 10:07:09 AM - SYSTEM STATS: Time:7.6331 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 10:02:10 AM - SYSTEM STATS: Time:8.1464 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 09:57:09 AM - SYSTEM STATS: Time:7.6785 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 09:52:11 AM - SYSTEM STATS: Time:9.8419 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 09:47:10 AM - SYSTEM STATS: Time:7.7980 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 09:42:09 AM - SYSTEM STATS: Time:7.7389 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 09:37:09 AM - SYSTEM STATS: Time:7.4584 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 09:32:10 AM - SYSTEM STATS: Time:7.9261 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 09:27:09 AM - SYSTEM STATS: Time:7.6677 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 09:22:09 AM - SYSTEM STATS: Time:7.7310 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/12/2012 09:17:15 AM - SYSTEM STATS: Time:13.8097 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145

There is no exact pattern when the error happens. This is quite annoying since it results in heavy CPU and MySQL usage on the server for the unsuccessful polling interval. Any ideas what is the cause of the problem and how to make it go away?


The Cacti (0.8.7i with PIA) is running on Debian Squeeze, PHP version is 5.3.3-7+squeeze3, MySQL version is 5.1.49-3. Fix64bit version is 0.3. If any mor einformation is needed, please let me know.

Thanks!

[hid3nax]

Okay, I uninstalled and removed the Fix64bit plugin but the issue is still there. Any ideas?

[hid3nax]

Now this morning even more errors started to appear:

01/13/2012 12:37:09 PM - SYSTEM STATS: Time:7.2899 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/13/2012 12:32:09 PM - SYSTEM STATS: Time:7.1215 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/13/2012 12:32:02 PM - POLLER: Poller[0] WARNING: Cron is out of sync with the Poller Interval! The Poller Interval is '300' seconds, with a maximum of a '300' second Cron, but 301 seconds have passed since the last poll!
01/13/2012 12:27:09 PM - SYSTEM STATS: Time:7.5157 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/13/2012 12:22:08 PM - SYSTEM STATS: Time:7.3631 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/13/2012 12:17:09 PM - SYSTEM STATS: Time:7.2932 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/13/2012 12:12:09 PM - SYSTEM STATS: Time:7.4286 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/13/2012 12:12:02 PM - POLLER: Poller[0] WARNING: Cron is out of sync with the Poller Interval! The Poller Interval is '300' seconds, with a maximum of a '300' second Cron, but 301 seconds have passed since the last poll!
01/13/2012 12:07:09 PM - SYSTEM STATS: Time:7.5176 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/13/2012 12:02:09 PM - SYSTEM STATS: Time:7.4039 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/13/2012 11:57:14 AM - SYSTEM STATS: Time:12.2125 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/13/2012 11:57:02 AM - POLLER: Poller[0] WARNING: Cron is out of sync with the Poller Interval! The Poller Interval is '300' seconds, with a maximum of a '300' second Cron, but 301 seconds have passed since the last poll!
01/13/2012 11:52:09 AM - SYSTEM STATS: Time:7.4510 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/13/2012 11:47:09 AM - SYSTEM STATS: Time:7.4785 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/13/2012 11:42:13 AM - SYSTEM STATS: Time:11.3986 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/13/2012 11:37:09 AM - SYSTEM STATS: Time:7.0740 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/13/2012 11:37:02 AM - POLLER: Poller[0] WARNING: Cron is out of sync with the Poller Interval! The Poller Interval is '300' seconds, with a maximum of a '300' second Cron, but 301 seconds have passed since the last poll!
01/13/2012 11:32:09 AM - SYSTEM STATS: Time:7.5597 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/13/2012 11:27:09 AM - SYSTEM STATS: Time:7.2597 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145
01/13/2012 11:22:08 AM - SYSTEM STATS: Time:7.2817 Method:cmd.php Processes:10 Threads:N/A Hosts:10 HostsPerProcess:1 DataSources:185 RRDsProcessed:145

What the hell is going on?

[hid3nax]

After digging logs for some time I found that:

01/14/2012 02:47:08 PM - CMDPHP: Poller[0] Host[4] DS[61] WARNING: Result from SERVER not valid. Partial Result: U
01/14/2012 02:47:08 PM - CMDPHP: Poller[0] Host[4] DS[61] WARNING: Result from SERVER not valid. Partial Result: U
01/14/2012 02:47:08 PM - CMDPHP: Poller[0] Host[4] DS[60] WARNING: Result from SERVER not valid. Partial Result: U
01/14/2012 02:47:08 PM - CMDPHP: Poller[0] Host[4] DS[60] WARNING: Result from SERVER not valid. Partial Result: 01/14/2012 02:47:06
01/14/2012 02:47:06 PM - PHPSVR: Poller[0] Maximum runtime of 300 seconds exceeded for the Script Server. Exiting.

"Maximum runtime of 300 seconds exceeded for the Script Server." is probably for the ss_host.disk.php script, since the WARNINGS are always about disk space.

Is there anything I can do to troubleshoot/debug the script?

Someone PLEASE help!

[gandalf]

Please see Settings -> Poller and post a screenshot
R.

[hid3nax]

gandalf,

Thank you for response!

Here are the screenshots:

[gandalf]

After digging logs for some time I found that:

01/14/2012 02:47:08 PM - CMDPHP: Poller[0] Host[4] DS[61] WARNING: Result from SERVER not valid. Partial Result: U
01/14/2012 02:47:08 PM - CMDPHP: Poller[0] Host[4] DS[61] WARNING: Result from SERVER not valid. Partial Result: U
01/14/2012 02:47:08 PM - CMDPHP: Poller[0] Host[4] DS[60] WARNING: Result from SERVER not valid. Partial Result: U
01/14/2012 02:47:08 PM - CMDPHP: Poller[0] Host[4] DS[60] WARNING: Result from SERVER not valid. Partial Result: 01/14/2012 02:47:06
01/14/2012 02:47:06 PM - PHPSVR: Poller[0] Maximum runtime of 300 seconds exceeded for the Script Server. Exiting.

"Maximum runtime of 300 seconds exceeded for the Script Server." is probably for the ss_host.disk.php script, since the WARNINGS are always about disk space.

Is there anything I can do to troubleshoot/debug the script?

Someone PLEASE help!

Are you polling a windows box? They are known to respond quite slow, so please increase timeout value for that host. In case same error message apperas for that very host each time, please verify, that this host responds to the OID for disk space.
R

[gandalf]

gandalf,

Thank you for response!

Those look fine. So you're doing a 5 minute polling, using cmd.php. But the time in the log are not on 5 minute boundaries. So I'm interested in seeing you crontab entry for cacti
R.

[hid3nax]

After digging logs for some time I found that:

01/14/2012 02:47:08 PM - CMDPHP: Poller[0] Host[4] DS[61] WARNING: Result from SERVER not valid. Partial Result: U
01/14/2012 02:47:08 PM - CMDPHP: Poller[0] Host[4] DS[61] WARNING: Result from SERVER not valid. Partial Result: U
01/14/2012 02:47:08 PM - CMDPHP: Poller[0] Host[4] DS[60] WARNING: Result from SERVER not valid. Partial Result: U
01/14/2012 02:47:08 PM - CMDPHP: Poller[0] Host[4] DS[60] WARNING: Result from SERVER not valid. Partial Result: 01/14/2012 02:47:06
01/14/2012 02:47:06 PM - PHPSVR: Poller[0] Maximum runtime of 300 seconds exceeded for the Script Server. Exiting.

"Maximum runtime of 300 seconds exceeded for the Script Server." is probably for the ss_host.disk.php script, since the WARNINGS are always about disk space.

Is there anything I can do to troubleshoot/debug the script?

Someone PLEASE help!

Are you polling a windows box? They are known to respond quite slow, so please increase timeout value for that host. In case same error message apperas for that very host each time, please verify, that this host responds to the OID for disk space.
R

Unfortunately, no. This is Ubuntu box. The error appears randomly, no pattern. I have ran logging to debug level and got the command. In shell I tried running `while true; do <the command with ss_host_disk>; done` and it never reported an error during this test. So I think host responds okay.

When this error happens, I can see some php poller processes stuck in `ps aux`. I'll try to post the output next time I catch it.

[hid3nax]

gandalf,

Thank you for response!

Those look fine. So you're doing a 5 minute polling, using cmd.php. But the time in the log are not on 5 minute boundaries. So I'm interested in seeing you crontab entry for cacti
R.

Since other servers may be also doing something on 5 minute basis, I decided not to load things up and have chosen a different time for polling. This is how my crontab looks like:

Code: Select all

2-57/5 *        * * *   www-data        php /var/www/stats/poller.php > /dev/null 2>&1

[hid3nax]

Okay, looks like I caught when the process hangs. This is what related processes I see in ps aux:

Code: Select all

www-data 31721  0.0  0.1   4432  1164 ?        Ss   18:02   0:00 /bin/sh -c php /var/www/stats/poller.php > /dev/null 2>&1
www-data 31722 54.3  1.3  42740 14144 ?        RN   18:02   1:42 php /var/www/stats/poller.php
www-data 31753  0.2  1.3  42212 13804 ?        SN   18:02   0:00 /usr/bin/php -q /var/www/stats/cmd.php 12 12
www-data 31755  0.0  0.1  10256  1804 ?        SN   18:02   0:00 /usr/bin/rrdtool -
www-data 31786  0.2  1.2  41188 12772 ?        SN   18:02   0:00 /usr/bin/php -q /var/www/stats/script_server.php cmd

[hid3nax]

caught one more:

Code: Select all

www-data 11503  0.1  1.3  42212 13788 ?        SN   12:37   0:00 /usr/bin/php -q /var/www/stats/cmd.php 4 4
www-data 11526  0.1  1.3  42212 13796 ?        SN   12:37   0:00 /usr/bin/php -q /var/www/stats/cmd.php 12 12
www-data 11535  0.0  1.2  41188 12776 ?        SN   12:37   0:00 /usr/bin/php -q /var/www/stats/script_server.php cmd
www-data 11565  0.0  1.2  41188 12776 ?        SN   12:37   0:00 /usr/bin/php -q /var/www/stats/script_server.php cmd

Any ideas what's wrong?

[gandalf]

Seems to hang on host 4 and 12, respectively. Are there any unusual scripts running for those?
Please pay attention to timeouts! Sometimes, no explicit timeouts are used and or cacti host specific timeouts differ from default timeouts
R.

[hid3nax]

Seems to hang on host 4 and 12, respectively. Are there any unusual scripts running for those?
Please pay attention to timeouts! Sometimes, no explicit timeouts are used and or cacti host specific timeouts differ from default timeouts
R.

Unfortunately, this is not limited to those.

Code: Select all

www-data  5138  0.0  1.3  42212 13840 ?        SN   22:52   0:00 /usr/bin/php -q /var/www/stats/cmd.php 10 11
www-data  5173  0.0  1.2  41444 12916 ?        SN   22:52   0:00 /usr/bin/php -q /var/www/stats/script_server.php cmd
www-data  5360  0.0  0.1   4432  1156 ?        Ss   22:57   0:00 /bin/sh -c nice -n 5 php /var/www/stats/poller.php > /dev/null 2>&1
www-data  5361 75.1  1.3  42740 13972 ?        RN   22:57   3:13 php /var/www/stats/poller.php
www-data  5395  0.1  1.3  42212 13812 ?        SN   22:57   0:00 /usr/bin/php -q /var/www/stats/cmd.php 13 13
www-data  5397  0.0  0.1  10256  1780 ?        SN   22:57   0:00 /usr/bin/rrdtool -
www-data  5427  0.1  1.2  41444 12844 ?        SN   22:57   0:00 /usr/bin/php -q /var/www/stats/script_server.php cmd

All the hosts have pretty much same, default templates. Just disk space, cpu usage and network interfaces are being monitored. Also, load average, host ping, logged in users but I doubt these can cause problems.

[gandalf]

gandalf,

Thank you for response!

Here are the screenshots:

Please reduce number of processes to 2 and retry
R.