[SOLVED] WARNING: Poller Output Table not Empty

[aKeLa]

idem

[gsaray101]

I dont think it is solved it either. I have these versions:

cacti: 0.8.7d
pa: 2.4
boost: 2.4
thold: 0.4.1
spine: 0.8.7e

I still see poller ouput table not empty errors very often:

Code: Select all

7/10/2009 01:36:44 PM - SYSTEM STATS: Time:44.1192 Method:spine Processes:1 Thr
ads:8 Hosts:316 HostsPerProcess:316 DataSources:29009 RRDsProcessed:0
7/10/2009 01:36:49 PM - SYSTEM THOLD STATS: Time:4.8557 Tholds:1532 Hosts:0
7/10/2009 01:37:03 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empt
.  Issues Found: 38, Data Sources: discards_in(DS[44128]), discards_out(DS[4412
]), errors_out(DS[44128]), AvgDiskReadQueueLen(DS[44677]), AvgDiskSecPerRead(DS
44677]), AvgDiskWriteQueueLe(DS[44677]), DiskReadsPerSec(DS[44677]), DiskTransf
rsPerSec(DS[44677]), DiskWritesPerSec(DS[44677]), PercentDiskReadTime(DS[44677]
, PercentDiskWriteTim(DS[44677]), AvgDiskReadQueueLen(DS[44678]), AvgDiskSecPer
ead(DS[44678]), AvgDiskSecPerWrite(DS[44678]), AvgDiskWriteQueueLe(DS[44678]),
iskReadsPerSec(DS[44678]), DiskWritesPerSec(DS[44678]), PercentDiskReadTime(DS[
4678]), PercentDiskWriteTim(DS[44678]), AvgDiskReadQueueLen(DS[44679]), AvgDisk
ecPerRead(DS[44679]), Additional Issues Remain.  Only showing first 20
7/10/2009 01:37:36 PM - SPINE: Poller[0] Host[1285] DS[44327] WARNING: SNMP tim
out detected [2000 ms], ignoring host 'ip'
7/10/2009 01:37:36 PM - SPINE: Poller[0] Host[1285] DS[44328] WARNING: SNMP tim

[BSOD2600]

cacti 0.8.7e and pia 2.5 are the latest. Please upgrade. There is no db change either, so it's painless.

[Jason.S]

I just updated from Spine 0.8.7a to 0.7.8e and I am now getting

Code: Select all

07/22/2009 08:59:03 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 1000, Data Sources: alv_eth_in(DS[265]), alv_eth_out(DS[266]), alv_radio_in(DS[267]), alv_radio_out(DS[268]), rx_frames_via_radio(DS[269]), tx_frames_to_radio(DS[270]), alv_radio_in(DS[271]), alv_radio_out(DS[272]), ba_rssi_value(DS[273]), retrans_frames(DS[274]), tx_frames_to_radio(DS[275]), rx_frames_via_eth(DS[276]), rx_frames_via_radio(DS[277]), retrans_frames(DS[278]), tx_frames_to_eth(DS[279]), tx_frames_to_radio(DS[280]), int_discard_packets(DS[281]), ack_crc(DS[282]), ack_time_out(DS[283]), errors_csl(DS[284]), eod(DS[285]), Additional Issues Remain. Only showing first 20

Code: Select all

07/22/2009 08:37:03 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 660, Data Sources: alv_eth_in(DS[580]), alv_eth_out(DS[581]), alv_radio_in(DS[582]), alv_radio_out(DS[583]), rx_frames_via_radio(DS[584]), tx_frames_to_radio(DS[585]), alv_radio_in(DS[586]), alv_radio_out(DS[587]), ba_rssi_value(DS[588]), retrans_frames(DS[589]), tx_frames_to_radio(DS[590]), rx_frames_via_eth(DS[591]), rx_frames_via_radio(DS[592]), retrans_frames(DS[593]), tx_frames_to_eth(DS[594]), tx_frames_to_radio(DS[595]), int_discard_packets(DS[596]), ack_crc(DS[597]), ack_time_out(DS[598]), errors_csl(DS[599]), eod(DS[600]), Additional Issues Remain. Only showing first 20

I am not sure how to fix this....... thank you for your help


Jason

[gsaray101]

I am occasionally seeing this problem on the cacti log and whenever I see this error, I see gaps on some graphs. My poller is 1 minute poller, should I change 298 to 60 on this line in poller.php?

define("MAX_POLLER_RUNTIME", 298);

please advice and I appreciate the help very much.

[gsaray101]

BSOD2600:

how does cacti 0.7.8e and pa 2.5 address this issue?

thanks,

hasan

[thardie]

I'm running 0.8.7e and get this problem occasionally. No idea why. I'll get the poller table not empty error for 1 host, but there will be a gap in almost all my graphs for that time interval. It happens around 4-5 time a day, and sometimes can span several consecutive intervals.

[TheWitness]

What poller? It mean that the poler is overrunning.

TheWitness

[Vertigo]

I'm getting a similar issue, running 0.8.7e.
But rather than gaps some graphs just aren't updating

06/01/2010 10:08:32 AM - SYSTEM STATS: Time:31.7265 Method:spine Processes:1 Threads:5 Hosts:30 HostsPerProcess:30 DataSources:514 RRDsProcessed:280
06/01/2010 10:08:01 AM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 61, Data Sources: traffic_in(DS[12]), traffic_out(DS[13]), traffic_in(DS[18]), traffic_out(DS[19]), traffic_in(DS[23]), traffic_out(DS[24]), traffic_in(DS[28]), traffic_out(DS[29]), traffic_in(DS[50]), traffic_in(DS[55]), traffic_in(DS[60]), traffic_out(DS[61]), traffic_in(DS[65]), traffic_out(DS[66]), traffic_in(DS[70]), traffic_out(DS[71]), traffic_in(DS[75]), traffic_in(DS[79]), traffic_out(DS[80]), traffic_in(DS[84]), traffic_in(DS[88]), Additional Issues Remain. Only showing first 20

[TheWitness]

I am thinking that might be a related to a bug in MySQL. Can you see about a MySQL update for your system.

Also, you may want to test by changing your poller_output table from MEMORY to MyISAM if you are unable to upgrade MySQL.

If you poller_output table is not MEMORY, then ignore. If it is, provide the "show create table poller_output" and your MySQL version.

TheWitness

[Vertigo]

Just upgraded from 5.1.40 to 5.1.47 and still getting the same issue. Table is already set to MyISAM -

Code: Select all

CREATE TABLE `poller_output` (
  `local_data_id` mediumint(8) unsigned NOT NULL DEFAULT '0',
  `rrd_name` varchar(19) NOT NULL DEFAULT '',
  `time` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
  `output` text NOT NULL,
  PRIMARY KEY (`local_data_id`,`rrd_name`,`time`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

[TheWitness]

Increase your snmp timeouts for the devices and let's see if that has any impact. After that, reduce the max oids for the device, if you are using spine.

TheWitness

[Vertigo]

Changed the timeouts and that didn't seem to do anything, but after reducing the max oids it came good. Thanks!
Any ideas why it might of suddenly done this?

[TheWitness]

Some devices get too busy to respond to bulkget's. You might want to make sure you are using snmpv2 and not snmpv1 too. Could be that you have a new monitoring system watching OID's too. That would effectively double the load on the device.

TheWitness

[ftorres]

Same problem here... didn't see the solution or maybe I read to fast... Can anyone give the specific link?