we have about 130 devices (2300 datasources) in monitoring.
when some devices are down, graphs of another devices have white holes too (see attachment). How to configure cacti accurately?
cacti fine tuning
Moderators: Developers, Moderators
cacti fine tuning
- Attachments
-
- sc1.JPG (197.42 KiB) Viewed 9289 times
-
TheWitness
- Developer
- Posts: 17007
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
Need to see your cacti.log from that time period please.
TheWitness
TheWitness
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
-
TheWitness
- Developer
- Posts: 17007
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
That really did not help. Low log level is much mor informative for this type of issue, and do not attach images of the log. Its better to cu and past into the note.
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
i changed threads number to 10. In result many errors in log:cigamit wrote:Your poller is running over.
Spine: 1 Process 1 Thread
Crank those threads up to 10-15 or so and see how it does (under Settings).
02/19/2009 11:05:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 11:00:34 AM - SYSTEM STATS: Time:33.6712 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4197 RRDsProcessed:2105
02/19/2009 11:00:09 AM - SPINE: Poller[0] Host[102] Hostname[10.16.2.254] ERROR: HOST EVENT: Host is DOWN Message: ICMP: Ping timed out
02/19/2009 10:59:24 AM - SYSTEM STATS: Time:22.2166 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4193 RRDsProcessed:2122
02/19/2009 10:58:37 AM - SYSTEM STATS: Time:35.2275 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4193 RRDsProcessed:2110
02/19/2009 10:57:37 AM - SYSTEM STATS: Time:35.3897 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4193 RRDsProcessed:2135
02/19/2009 10:57:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 26, Data Sources: users(DS[6]), proc(DS[7]), hdd_free(DS[14]), hdd_used(DS[14]), traffic_in(DS[22]), traffic_out(DS[22]), traffic_in(DS[27]), traffic_out(DS[27]), traffic_in(DS[30]), traffic_out(DS[30]), traffic_in(DS[35]), traffic_out(DS[35]), traffic_in(DS[40]), traffic_out(DS[40]), traffic_in(DS[45]), traffic_out(DS[45]), traffic_in(DS[85]), traffic_out(DS[85]), traffic_in(DS[1430]), traffic_out(DS[1430]), traffic_in(DS[1431]), Additional Issues Remain. Only showing first 20
02/19/2009 10:57:00 AM - SYSTEM STATS: Time:0.1297 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4200 RRDsProcessed:0
02/19/2009 10:57:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:56:50 AM - SYSTEM STATS: Time:48.4232 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4193 RRDsProcessed:3005
02/19/2009 10:56:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 2477, Data Sources: mem_swap(DS[4]), traffic_in(DS[18]), traffic_out(DS[18]), traffic_in(DS[20]), traffic_out(DS[20]), traffic_in(DS[25]), traffic_out(DS[25]), 5min_cpu(DS[29]), traffic_in(DS[33]), traffic_out(DS[33]), traffic_in(DS[38]), traffic_out(DS[38]), traffic_in(DS[51]), traffic_out(DS[51]), traffic_in(DS[58]), traffic_out(DS[58]), traffic_in(DS[63]), traffic_out(DS[63]), snmp_oid(DS[66]), traffic_in(DS[71]), traffic_out(DS[71]), Additional Issues Remain. Only showing first 20
02/19/2009 10:56:00 AM - SYSTEM STATS: Time:0.1272 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4200 RRDsProcessed:0
02/19/2009 10:56:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:56:00 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 8, Data Sources: (DS[2276]), (DS[2277]), sys_temperature(DS[2278]), (DS[2279]), cpu_nice(DS[2280]), load_5min(DS[2285]), traffic_in(DS[2286]), traffic_out(DS[2286])
02/19/2009 10:55:37 AM - SYSTEM STATS: Time:35.5321 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4193 RRDsProcessed:3012
02/19/2009 10:55:04 AM - SPINE: Poller[0] ERROR: SQL Failed! Error:'1062', Message:'Duplicate entry '2298-traffic_out-2009-02-19 10:55:04' for key 1', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (2297,'5min_cpu','2009-02-19 10:55:04','25'),(2298,'traffic_out','2009-02-19 10:55:04','2390571507'),(2298,'traffic_in','2009-02-19 10:55:04','3176070661'),(2299,'traffic_out','2009-02-19 10:55:04','2848156154'),(2299,'traffic_in','2009-02-19 10:55:04','3723975665'),(2300,'traffic_out','2009-02-19 10:55:04','1032345723'),(2300,'traffic_in','2009-02-19 10:55:04','1596763351'),(2301,'traffic_out','2009-02-19 10:55:04','2306708721'),(2301,'traffic_in','2009-02-19 10:55:04','3176072770'),(2302,'traffic_out','2009-02-19 10:55:04','3334185899'),(2302,'traffic_in','2009-02-19 10:55:04','2932061848'),(2303,'traffic_out','2009-02-19 10:55:04','604414267'),(2303,'traffic_in','2009-02-19 10:55:04','791903077')'
02/19/2009 10:55:04 AM - SPINE: Poller[0] ERROR: SQL Failed! Error:'1062', Message:'Duplicate entry '2274-traffic_in-2009-02-19 10:55:04' for key 1', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (2274,'traffic_in','2009-02-19 10:55:04','4167502481'),(2274,'traffic_out','2009-02-19 10:55:04','4252844496'),(2275,'traffic_in','2009-02-19 10:55:04','1280515672'),(2275,'traffic_out','2009-02-19 10:55:04','1203764692')'
02/19/2009 10:55:04 AM - SPINE: Poller[0] ERROR: SQL Failed! Error:'1062', Message:'Duplicate entry '2270-traffic_out-2009-02-19 10:55:04' for key 1', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (2270,'traffic_out','2009-02-19 10:55:04','1423105982'),(2270,'traffic_in','2009-02-19 10:55:04','624990466'),(2271,'traffic_out','2009-02-19 10:55:04','731626284'),(2271,'traffic_in','2009-02-19 10:55:04','2477186952'),(2272,'traffic_in','2009-02-19 10:55:04','1428145589'),(2272,'traffic_out','2009-02-19 10:55:04','624541263'),(2273,'traffic_in','2009-02-19 10:55:04','6871222'),(2273,'traffic_out','2009-02-19 10:55:04','5196333')'
02/19/2009 10:55:04 AM - SPINE: Poller[0] ERROR: SQL Failed! Error:'1062', Message:'Duplicate entry '2266-traffic_in-2009-02-19 10:55:04' for key 1', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (2266,'traffic_in','2009-02-19 10:55:04','1321849238'),(2266,'traffic_out','2009-02-19 10:55:04','550410249'),(2267,'traffic_in','2009-02-19 10:55:04','644334664'),(2267,'traffic_out','2009-02-19 10:55:04','1421847472'),(2268,'traffic_out','2009-02-19 10:55:04','1296943225'),(2268,'traffic_in','2009-02-19 10:55:04','545362214'),(2269,'traffic_out','2009-02-19 10:55:04','5048779'),(2269,'traffic_in','2009-02-19 10:55:04','6298306')'
02/19/2009 10:55:04 AM - SPINE: Poller[0] ERROR: SQL Failed! Error:'1062', Message:'Duplicate entry '2256-traffic_in-2009-02-19 10:55:04' for key 1', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (2256,'traffic_in','2009-02-19 10:55:04','3752120588'),(2256,'traffic_out','2009-02-19 10:55:04','2342154614'),(2257,'traffic_in','2009-02-19 10:55:04','3151572016'),(2257,'traffic_out','2009-02-19 10:55:04','18347707')'
02/19/2009 10:55:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 2424, Data Sources: mem_buffers(DS[3]), traffic_in(DS[23]), traffic_out(DS[23]), 5min_cpu(DS[28]), traffic_in(DS[31]), traffic_out(DS[31]), traffic_in(DS[36]), traffic_out(DS[36]), traffic_in(DS[41]), traffic_out(DS[41]), 5min_cpu(DS[43]), traffic_in(DS[46]), traffic_out(DS[46]), 5min_cpu(DS[52]), traffic_in(DS[54]), traffic_out(DS[54]), traffic_in(DS[61]), traffic_out(DS[61]), snmp_oid(DS[66]), traffic_in(DS[71]), traffic_out(DS[71]), Additional Issues Remain. Only showing first 20
02/19/2009 10:55:00 AM - SYSTEM STATS: Time:0.1281 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4200 RRDsProcessed:0
02/19/2009 10:55:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:55:00 AM - SYSTEM STATS: Time:238.3950 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4200 RRDsProcessed:8
02/19/2009 10:55:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:50:30 AM - SYSTEM STATS: Time:28.2842 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4200 RRDsProcessed:2134
02/19/2009 10:49:23 AM - SYSTEM STATS: Time:22.3322 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4192 RRDsProcessed:2131
02/19/2009 10:48:39 AM - SYSTEM STATS: Time:38.4015 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4192 RRDsProcessed:2106
02/19/2009 10:47:38 AM - SYSTEM STATS: Time:37.2630 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4192 RRDsProcessed:2130
02/19/2009 10:47:00 AM - SYSTEM STATS: Time:59.0118 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4192 RRDsProcessed:2728
02/19/2009 10:46:59 AM - SYSTEM STATS: Time:0.1268 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4197 RRDsProcessed:0
02/19/2009 10:46:59 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:46:59 AM - SPINE: Poller[0] ERROR: Spine Timed Out While Processing Hosts Internal
02/19/2009 10:46:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 3044, Data Sources: mem_swap(DS[4]), traffic_in(DS[18]), traffic_out(DS[18]), traffic_in(DS[21]), traffic_out(DS[21]), traffic_in(DS[26]), traffic_out(DS[26]), 5min_cpu(DS[29]), traffic_in(DS[33]), traffic_out(DS[33]), traffic_in(DS[38]), traffic_out(DS[38]), traffic_in(DS[45]), traffic_out(DS[45]), traffic_in(DS[53]), traffic_out(DS[53]), traffic_in(DS[60]), traffic_out(DS[60]), traffic_in(DS[65]), traffic_out(DS[65]), snmp_oid(DS[66]), Additional Issues Remain. Only showing first 20
02/19/2009 10:45:59 AM - SYSTEM STATS: Time:0.1230 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4197 RRDsProcessed:0
02/19/2009 10:45:59 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:45:29 AM - SYSTEM STATS: Time:28.0255 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4192 RRDsProcessed:2127
02/19/2009 10:45:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: mem_buffers(DS[3])
02/19/2009 10:44:59 AM - SYSTEM STATS: Time:0.1490 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4197 RRDsProcessed:0
02/19/2009 10:44:59 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:44:59 AM - SYSTEM STATS: Time:238.8708 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4197 RRDsProcessed:3
02/19/2009 10:44:59 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:40:27 AM - SYSTEM STATS: Time:27.1300 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4197 RRDsProcessed:2107
02/19/2009 10:40:01 AM - SPINE: Poller[0] Host[66] Hostnanme[10.5.2.254] NOTICE: HOST EVENT: Host Returned from DOWN State
02/19/2009 10:40:00 AM - SYSTEM STATS: Time:58.3849 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4193 RRDsProcessed:1669
02/19/2009 10:40:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:38:44 AM - SYSTEM STATS: Time:42.3985 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4193 RRDsProcessed:2137
02/19/2009 10:37:35 AM - SYSTEM STATS: Time:33.3762 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4193 RRDsProcessed:2133
02/19/2009 10:36:54 AM - SYSTEM STATS: Time:52.7154 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4193 RRDsProcessed:1016
02/19/2009 10:36:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 2240, Data Sources: (DS[5]), traffic_in(DS[24]), traffic_out(DS[24]), traffic_in(DS[31]), traffic_out(DS[31]), traffic_in(DS[36]), traffic_out(DS[36]), traffic_in(DS[41]), traffic_out(DS[41]), traffic_in(DS[51]), traffic_out(DS[51]), traffic_in(DS[58]), traffic_out(DS[58]), traffic_in(DS[63]), traffic_out(DS[63]), snmp_oid(DS[66]), traffic_in(DS[71]), traffic_out(DS[71]), traffic_in(DS[72]), traffic_out(DS[72]), traffic_in(DS[73]), Additional Issues Remain. Only showing first 20
02/19/2009 10:36:00 AM - SYSTEM STATS: Time:0.1255 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4197 RRDsProcessed:0
02/19/2009 10:36:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:35:39 AM - SYSTEM STATS: Time:37.3014 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4193 RRDsProcessed:3039
02/19/2009 10:35:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 2239, Data Sources: traffic_in(DS[18]), traffic_out(DS[18]), traffic_in(DS[22]), traffic_out(DS[22]), traffic_in(DS[27]), traffic_out(DS[27]), traffic_in(DS[34]), traffic_out(DS[34]), traffic_in(DS[39]), traffic_out(DS[39]), 5min_cpu(DS[43]), traffic_in(DS[46]), traffic_out(DS[46]), 5min_cpu(DS[52]), traffic_in(DS[54]), traffic_out(DS[54]), traffic_in(DS[61]), traffic_out(DS[61]), snmp_oid(DS[66]), traffic_in(DS[71]), traffic_out(DS[71]), Additional Issues Remain. Only showing first 20
02/19/2009 10:35:00 AM - SYSTEM STATS: Time:0.1252 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4197 RRDsProcessed:0
02/19/2009 10:35:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:35:00 AM - SYSTEM STATS: Time:179.1512 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4197 RRDsProcessed:1
02/19/2009 10:35:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:32:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 143, Data Sources: users(DS[6]), proc(DS[7]), hdd_free(DS[14]), hdd_used(DS[14]), traffic_in(DS[21]), traffic_out(DS[21]), traffic_in(DS[26]), traffic_out(DS[26]), 5min_cpu(DS[29]), traffic_in(DS[33]), traffic_out(DS[33]), traffic_in(DS[38]), traffic_out(DS[38]), traffic_in(DS[45]), traffic_out(DS[45]), traffic_in(DS[60]), traffic_out(DS[60]), traffic_in(DS[65]), traffic_out(DS[65]), snmp_oid(DS[66]), traffic_in(DS[71]), Additional Issues Remain. Only showing first 20
02/19/2009 10:32:00 AM - SYSTEM STATS: Time:0.1306 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4201 RRDsProcessed:0
02/19/2009 10:32:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:32:00 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 8, Data Sources: (DS[2276]), (DS[2277]), sys_temperature(DS[2278]), (DS[2279]), cpu_nice(DS[2280]), load_5min(DS[2285]), traffic_in(DS[2286]), traffic_out(DS[2286])
02/19/2009 10:31:42 AM - SYSTEM STATS: Time:40.8183 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4197 RRDsProcessed:3012
02/19/2009 10:31:04 AM - SPINE: Poller[0] Host[66] Hostname[10.5.2.254] ERROR: HOST EVENT: Host is DOWN Message: ICMP: Ping timed out
02/19/2009 10:31:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 2420, Data Sources: mem_swap(DS[4]), traffic_in(DS[24]), traffic_out(DS[24]), traffic_in(DS[31]), traffic_out(DS[31]), traffic_in(DS[36]), traffic_out(DS[36]), traffic_in(DS[41]), traffic_out(DS[41]), traffic_in(DS[51]), traffic_out(DS[51]), traffic_in(DS[58]), traffic_out(DS[58]), traffic_in(DS[63]), traffic_out(DS[63]), snmp_oid(DS[66]), traffic_in(DS[71]), traffic_out(DS[71]), traffic_in(DS[72]), traffic_out(DS[72]), traffic_in(DS[73]), Additional Issues Remain. Only showing first 20
02/19/2009 10:31:00 AM - SYSTEM STATS: Time:0.1270 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4201 RRDsProcessed:0
02/19/2009 10:31:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:31:00 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 7, Data Sources: (DS[2276]), (DS[2277]), sys_temperature(DS[2278]), (DS[2279]), cpu_user(DS[2282]), traffic_in(DS[2286]), traffic_out(DS[2286])
02/19/2009 10:30:39 AM - SYSTEM STATS: Time:37.9777 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4197 RRDsProcessed:3029
02/19/2009 10:30:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 2426, Data Sources: mem_buffers(DS[3]), traffic_in(DS[18]), traffic_out(DS[18]), traffic_in(DS[22]), traffic_out(DS[22]), traffic_in(DS[27]), traffic_out(DS[27]), traffic_in(DS[34]), traffic_out(DS[34]), traffic_in(DS[39]), traffic_out(DS[39]), 5min_cpu(DS[43]), traffic_in(DS[46]), traffic_out(DS[46]), traffic_in(DS[55]), traffic_out(DS[55]), traffic_in(DS[61]), traffic_out(DS[61]), snmp_oid(DS[66]), traffic_in(DS[71]), traffic_out(DS[71]), Additional Issues Remain. Only showing first 20
02/19/2009 10:30:00 AM - SYSTEM STATS: Time:0.1288 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4201 RRDsProcessed:0
02/19/2009 10:30:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:30:00 AM - SYSTEM STATS: Time:238.8482 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4201 RRDsProcessed:1084
02/19/2009 10:30:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:27:00 AM - SYSTEM STATS: Time:0.1268 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4199 RRDsProcessed:0
02/19/2009 10:27:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:27:00 AM - SPINE: Poller[0] ERROR: Spine Timed Out While Processing Hosts Internal
02/19/2009 10:26:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 2354, Data Sources: mem_swap(DS[4]), traffic_in(DS[20]), traffic_out(DS[20]), traffic_in(DS[25]), traffic_out(DS[25]), traffic_in(DS[53]), traffic_out(DS[53]), traffic_in(DS[60]), traffic_out(DS[60]), traffic_in(DS[65]), traffic_out(DS[65]), snmp_oid(DS[66]), traffic_in(DS[71]), traffic_out(DS[71]), traffic_in(DS[72]), traffic_out(DS[72]), traffic_in(DS[73]), traffic_out(DS[73]), traffic_in(DS[74]), traffic_out(DS[74]), traffic_in(DS[75]), Additional Issues Remain. Only showing first 20
02/19/2009 10:26:00 AM - SYSTEM STATS: Time:0.1266 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4199 RRDsProcessed:0
02/19/2009 10:26:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:25:33 AM - SYSTEM STATS: Time:32.2019 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4201 RRDsProcessed:2136
02/19/2009 10:25:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1, Data Sources: mem_buffers(DS[3])
02/19/2009 10:25:00 AM - SYSTEM STATS: Time:0.1327 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4199 RRDsProcessed:0
02/19/2009 10:25:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:25:00 AM - SYSTEM STATS: Time:239.0042 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4199 RRDsProcessed:3
02/19/2009 10:25:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:20:35 AM - SYSTEM STATS: Time:34.5514 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4199 RRDsProcessed:2134
02/19/2009 10:19:11 AM - SYSTEM STATS: Time:10.6128 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4202 RRDsProcessed:2130
02/19/2009 10:18:12 AM - SYSTEM STATS: Time:11.7928 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4202 RRDsProcessed:2138
02/19/2009 10:17:23 AM - SYSTEM STATS: Time:22.9901 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4202 RRDsProcessed:2102
02/19/2009 10:16:36 AM - SYSTEM STATS: Time:36.2753 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4202 RRDsProcessed:2139
02/19/2009 10:15:02 AM - SYSTEM STATS: Time:2.2388 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4202 RRDsProcessed:8
02/19/2009 10:15:00 AM - SYSTEM STATS: Time:10.5094 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4192 RRDsProcessed:2127
02/19/2009 10:15:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:14:07 AM - SYSTEM STATS: Time:17.3485 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4192 RRDsProcessed:2129
02/19/2009 10:13:27 AM - SYSTEM STATS: Time:37.2556 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4192 RRDsProcessed:4236
02/19/2009 10:13:00 AM - SYSTEM STATS: Time:0.1308 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:0
02/19/2009 10:13:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:12:49 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 8, Data Sources: (DS[2276]), (DS[2277]), sys_temperature(DS[2278]), (DS[2279]), cpu_nice(DS[2280]), load_5min(DS[2285]), traffic_in(DS[2286]), traffic_out(DS[2286])
02/19/2009 10:12:26 AM - SYSTEM STATS: Time:36.6836 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4192 RRDsProcessed:4263
02/19/2009 10:12:00 AM - SYSTEM STATS: Time:0.1304 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:0
02/19/2009 10:12:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:11:49 AM - SYSTEM STATS: Time:108.0970 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4192 RRDsProcessed:2129
02/19/2009 10:11:00 AM - SYSTEM STATS: Time:0.1933 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:0
02/19/2009 10:11:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:10:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 4, Data Sources: users(DS[6]), proc(DS[7]), hdd_free(DS[14]), hdd_used(DS[14])
02/19/2009 10:10:00 AM - SYSTEM STATS: Time:0.1300 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:0
02/19/2009 10:10:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:10:00 AM - SYSTEM STATS: Time:298.6453 Method:spine Processes:1 Threads:10 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:1259
02/19/2009 10:10:00 AM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
02/19/2009 10:04:38 AM - SYSTEM STATS: Time:36.5296 Method:spine Processes:1 Threads:1 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:2134
02/19/2009 10:03:36 AM - SYSTEM STATS: Time:35.1259 Method:spine Processes:1 Threads:1 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:2130
02/19/2009 10:02:42 AM - SYSTEM STATS: Time:41.1350 Method:spine Processes:1 Threads:1 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:2129
02/19/2009 10:01:54 AM - SYSTEM STATS: Time:53.1534 Method:spine Processes:1 Threads:1 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:2133
02/19/2009 10:00:59 AM - SYSTEM STATS: Time:57.5885 Method:spine Processes:1 Threads:1 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:2134
02/19/2009 09:59:39 AM - SYSTEM STATS: Time:37.6283 Method:spine Processes:1 Threads:1 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:2133
02/19/2009 09:58:33 AM - SYSTEM STATS: Time:31.4644 Method:spine Processes:1 Threads:1 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:2135
02/19/2009 09:57:48 AM - SYSTEM STATS: Time:46.6646 Method:spine Processes:1 Threads:1 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:2128
02/19/2009 09:56:54 AM - SYSTEM STATS: Time:52.5167 Method:spine Processes:1 Threads:1 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:2131
02/19/2009 09:56:19 AM - SPINE: Poller[0] Host[92] Hostnanme[10.8.2.254] NOTICE: HOST EVENT: Host Returned from DOWN State
02/19/2009 09:55:52 AM - SYSTEM STATS: Time:50.2357 Method:spine Processes:1 Threads:1 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:2134
02/19/2009 09:54:40 AM - SYSTEM STATS: Time:38.8277 Method:spine Processes:1 Threads:1 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:2137
02/19/2009 09:53:50 AM - SYSTEM STATS: Time:48.6678 Method:spine Processes:1 Threads:1 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:2133
02/19/2009 09:52:38 AM - SYSTEM STATS: Time:36.3645 Method:spine Processes:1 Threads:1 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:2129
02/19/2009 09:52:20 AM - SPINE: Poller[0] Host[92] Hostname[10.8.2.254] ERROR: HOST EVENT: Host is DOWN Message: ICMP: Ping timed out
02/19/2009 09:51:53 AM - SYSTEM STATS: Time:51.5002 Method:spine Processes:1 Threads:1 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:2131
02/19/2009 09:50:43 AM - SYSTEM STATS: Time:41.4641 Method:spine Processes:1 Threads:1 Hosts:124 HostsPerProcess:124 DataSources:4198 RRDsProcessed:2134
-
TheWitness
- Developer
- Posts: 17007
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
What version of spine?
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
-
TheWitness
- Developer
- Posts: 17007
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
What platform and are you running scripts?
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
-
Howie
- Cacti Guru User
- Posts: 5508
- Joined: Thu Sep 16, 2004 5:53 am
- Location: United Kingdom
- Contact:
I have similar kinds of problems (poller starts overrunning when more than a few devices are down) since moving from 32bit FreeBSD to 64bit CentOS on faster hardware. Luckily the new hardware is fast enough that cmd.php is usable. Cacti 0.8.7b and Spine 0.8.7c, 15 threads, 5 second timeout on scripts and 10 script server processes (most of my polling is SNMP though).
Weathermap 0.98a is out! & QuickTree 1.0. Superlinks is over there now (and built-in to Cacti 1.x).
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
Some Other Cacti tweaks, including strip-graphs, icons and snmp/netflow stuff.
(Let me know if you have UK DevOps or Network Ops opportunities, too!)
Debian 4.0 / 2.6.18-4TheWitness wrote:What platform and are you running scripts?
P4 3.0Ghz, RAM 2Gb
Poller settings:
Poller Type - spine
Poller Interval - every minute
Cron Interval - every 5 mins
Maximum Concurrent Poller Processes - 1
Balance Process Load - enabled
Maximum Threads per Process - 1
Number of PHP Script Servers - 1
Script and Script Server Timeout Value - 80
The Maximum SNMP OID's Per SNMP Get Request - 60
Bad devices seems to effect the timeout of scripts to other servers, most likely just unfortunate to be put in the same thread. It's like the timeout (1) value is totally ignored.
(1)
Script and Script Server Timeout Value
The maximum time that Cacti will wait on a script to complete. This timeout value is in seconds
(1)
Script and Script Server Timeout Value
The maximum time that Cacti will wait on a script to complete. This timeout value is in seconds
-
TheWitness
- Developer
- Posts: 17007
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
So, the timeout value is not working here for script server and you are running the latest spine?khufure wrote:Bad devices seems to effect the timeout of scripts to other servers, most likely just unfortunate to be put in the same thread. It's like the timeout (1) value is totally ignored.
(1)
Script and Script Server Timeout Value
The maximum time that Cacti will wait on a script to complete. This timeout value is in seconds
TheWitness
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
-
TheWitness
- Developer
- Posts: 17007
- Joined: Tue May 14, 2002 5:08 pm
- Location: MI, USA
- Contact:
An 80 second script/script server timeout, when your poller interval is 60 seconds is ridiculous. Make it 2/3 seconds if you have that many hosts.Pengu1n wrote:Debian 4.0 / 2.6.18-4TheWitness wrote:What platform and are you running scripts?
P4 3.0Ghz, RAM 2Gb
Poller settings:
Poller Type - spine
Poller Interval - every minute
Cron Interval - every 5 mins
Maximum Concurrent Poller Processes - 1
Balance Process Load - enabled
Maximum Threads per Process - 1
Number of PHP Script Servers - 1
Script and Script Server Timeout Value - 80
The Maximum SNMP OID's Per SNMP Get Request - 60
TheWitness
True understanding begins only when we realize how little we truly understand...
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Life is an adventure, let yours begin with Cacti!
Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages
For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
cacti-spine-0.8.7c.tar.gz from Jan 23TheWitness wrote:So, the timeout value is not working here for script server and you are running the latest spine?khufure wrote:Bad devices seems to effect the timeout of scripts to other servers, most likely just unfortunate to be put in the same thread. It's like the timeout (1) value is totally ignored.
(1)
Script and Script Server Timeout Value
The maximum time that Cacti will wait on a script to complete. This timeout value is in seconds
TheWitness
I don't have this strange problem on 2 bigger servers, though. Is there a later version?
Thanks!
Who is online
Users browsing this forum: No registered users and 0 guests