[SOLVED] WARNING: Poller Output Table not Empty

[bernd]

I had a similar problem with cacti 0.8.7g.
In my case only hexadecimal datasources where affected (Hex-String:)

poller.php inserts them as '00cd:ef:00...' into the poller_output table but lib/poller.php doesn't recognize those values as hexadecimal.
The solution for me was to add ':' into functions.php:is_hexadecimal() (line 903) to the allowed characters.
But I'm facing the next problem now, the resulting values are way of. PB instead of MB. Looking into it...

[deepsys]

Hello,

i know its offtopic, but this is exactly my problem:

But I'm facing the next problem now, the resulting values are way of. PB instead of MB. Looking into it...

Could you resolved it?

Thanks!

[bernd]

Hi deepsys,

I'm not sure if I really solved it, yet. The datasource in question is a 64bit counter which is reported as octet(Hex) string. So far, so good. Cacti can interpret those values.

But the device (Brocade SAN/FC switch) doesn't reset those counters at the 64bit limit as one would expect but wrap's them earlier at a random(?) value.
To have cacti recognize this correclty I had to set max. value in the data template. I set it to interface speed*poller interval now the graphs do look fine. But I watch this the next days just to be sure

As for the originial problem, could anyone confirm if adding the colon(:) to is_hexadecimal() help? I would prefer to have this confirmend before registering it as a bug

regards,
bernd

[deepsys]

Hi Bernd,

i have found a solution, take a look at this post: http://forums.cacti.net/viewtopic.php?f ... 37#p215337

And yes, i can confirm that adding the colon(:) to is_hexadecimal() helps. Without it, nothing works.
But this is only the first step, you have to correct another bug: http://bugs.cacti.net/view.php?id=1876

Then it works !!

[gandalf]

Seems that those fixed are available in current 087h code.
R.

[madaxe]

Hi Everyone,

Sorry for bumping thread again but i have the same problem.

Last night our cacti (cacti-0.8.7i) stopped updating graphs without any change on config. I've checked and tried everything in this thread and couldn't find a proper solution. Then I've updated my installation to 0.8.8a but still the same.

We're always getting " POLLER: Poller[0] WARNING: Poller Output Table not Empty." error in logs and then there is no update in graphs.

Our setup is running on a Debian Squeeze 64bit virtual machine with fastcgi and nginx.

poller and cron are working every 5 min.

And here are the errors (i've removed succesful OK lines)

Code: Select all

05/09/2012 05:40:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty.  Issues Found: 184, Data Sources: traffic_in(DS[16]), traffic_out(DS[16]), traffic_in(DS[17]), traffic_out(DS[17]), traffic_in(DS[18]), traffic_out(DS[18]), traffic_in(DS[19]), traffic_out(DS[19]), traffic_in(DS[20]), traffic_out(DS[20]), traffic_in(DS[21]), traffic_out(DS[21]), traffic_in(DS[22]), traffic_out(DS[22]), traffic_in(DS[23]), traffic_out(DS[23]), traffic_in(DS[24]), traffic_out(DS[24]), traffic_in(DS[25]), traffic_out(DS[25]), Additional Issues Remain.  Only showing first 20
05/09/2012 05:40:01 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty.  Issues Found: 184, Data Sources: traffic_in(DS[16]), traffic_out(DS[16]), traffic_in(DS[17]), traffic_out(DS[17]), traffic_in(DS[18]), traffic_out(DS[18]), traffic_in(DS[19]), traffic_out(DS[19]), traffic_in(DS[20]), traffic_out(DS[20]), traffic_in(DS[21]), traffic_out(DS[21]), traffic_in(DS[22]), traffic_out(DS[22]), traffic_in(DS[23]), traffic_out(DS[23]), traffic_in(DS[24]), traffic_out(DS[24]), traffic_in(DS[25]), traffic_out(DS[25]), Additional Issues Remain.  Only showing first 20
05/09/2012 05:40:05 PM - CMDPHP: Poller[0] Host[6] DS[102] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:05 PM - CMDPHP: Poller[0] Host[7] DS[131] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:05 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.52.22', and OID:'.1.3.6.1.4.1.2021.4.13.0'
05/09/2012 05:40:05 PM - CMDPHP: Poller[0] Host[8] DS[159] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:05 PM - CMDPHP: Poller[0] Host[9] DS[187] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:05 PM - CMDPHP: Poller[0] Host[10] DS[216] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:05 PM - CMDPHP: Poller[0] Host[11] DS[245] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:05 PM - CMDPHP: Poller[0] Host[12] DS[274] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:06 PM - CMDPHP: Poller[0] Host[13] DS[302] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:06 PM - CMDPHP: Poller[0] Host[14] DS[330] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:06 PM - CMDPHP: Poller[0] Host[15] DS[359] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:06 PM - CMDPHP: Poller[0] Host[16] DS[388] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:06 PM - CMDPHP: Poller[0] Host[18] DS[416] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:06 PM - CMDPHP: Poller[0] Host[19] DS[444] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:06 PM - CMDPHP: Poller[0] Host[20] DS[472] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:06 PM - CMDPHP: Poller[0] Host[21] DS[500] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:06 PM - CMDPHP: Poller[0] Host[22] DS[528] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:06 PM - CMDPHP: Poller[0] Host[23] DS[556] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[24] DS[584] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.82.22', and OID:'.1.3.6.1.4.1.2021.4.13.0'
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[25] DS[612] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.82.23', and OID:'.1.3.6.1.4.1.2021.4.13.0'
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[26] DS[640] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.82.24', and OID:'.1.3.6.1.4.1.2021.4.13.0'
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[27] DS[668] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[28] DS[696] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[29] DS[724] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[30] DS[752] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[31] DS[780] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[32] DS[808] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[33] DS[830] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[33] DS[827] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[33] DS[829] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[33] DS[828] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[33] DS[826] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[33] DS[831] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[33] DS[837] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[33] DS[836] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[33] DS[835] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[33] DS[834] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[33] DS[833] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:07 PM - CMDPHP: Poller[0] Host[33] DS[832] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:10 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.29', and OID:'.1.3.6.1.2.1.1.3.0'
05/09/2012 05:40:13 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.30', and OID:'.1.3.6.1.2.1.1.3.0'
05/09/2012 05:40:16 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.31', and OID:'.1.3.6.1.2.1.1.3.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.25', and OID:'.1.3.6.1.2.1.1.3.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.112.20', and OID:'.1.3.6.1.4.1.2021.4.13.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[39] DS[1068] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.112.21', and OID:'.1.3.6.1.4.1.2021.4.13.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[40] DS[1097] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.112.22', and OID:'.1.3.6.1.4.1.2021.4.13.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[41] DS[1126] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.112.23', and OID:'.1.3.6.1.4.1.2021.4.13.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[42] DS[1155] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.112.24', and OID:'.1.3.6.1.4.1.2021.4.13.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[43] DS[1184] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.24', and OID:'.1.3.6.1.4.1.2021.4.13.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[44] DS[1213] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.27', and OID:'.1.3.6.1.4.1.2021.4.13.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[45] DS[1244] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.9', and OID:'.1.3.6.1.4.1.2021.11.53.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[46] DS[1265] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.9', and OID:'.1.3.6.1.4.1.2021.11.51.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[46] DS[1266] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.9', and OID:'.1.3.6.1.4.1.2021.11.52.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[46] DS[1267] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.9', and OID:'.1.3.6.1.4.1.2021.11.50.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[46] DS[1268] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.9', and OID:'.1.3.6.1.4.1.2021.10.1.3.3'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[46] DS[1269] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.9', and OID:'.1.3.6.1.4.1.2021.10.1.3.1'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[46] DS[1270] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.9', and OID:'.1.3.6.1.4.1.2021.10.1.3.2'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[46] DS[1271] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.9', and OID:'.1.3.6.1.4.1.2021.4.6.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[46] DS[1272] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.9', and OID:'.1.3.6.1.4.1.2021.4.14.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[46] DS[1273] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.9', and OID:'.1.3.6.1.4.1.2021.4.15.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[46] DS[1274] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.9', and OID:'.1.3.6.1.4.1.2021.4.13.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[46] DS[1275] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.9', and OID:'.1.3.6.1.4.1.2021.4.5.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[46] DS[1276] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.102.25', and OID:'.1.3.6.1.4.1.2021.4.13.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[47] DS[1305] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.102.26', and OID:'.1.3.6.1.4.1.2021.4.13.0'
05/09/2012 05:40:20 PM - CMDPHP: Poller[0] Host[48] DS[1334] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:21 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.102.27', and OID:'.1.3.6.1.4.1.2021.4.13.0'
05/09/2012 05:40:21 PM - CMDPHP: Poller[0] Host[49] DS[1363] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:21 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.102.28', and OID:'.1.3.6.1.4.1.2021.4.13.0'
05/09/2012 05:40:21 PM - CMDPHP: Poller[0] Host[50] DS[1392] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:21 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.xxx.xxx.20', and OID:'.1.3.6.1.4.1.2021.4.13.0'
05/09/2012 05:40:21 PM - CMDPHP: Poller[0] Host[52] DS[1421] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:21 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.72.22', and OID:'.1.3.6.1.4.1.2021.4.13.0'
05/09/2012 05:40:21 PM - CMDPHP: Poller[0] Host[53] DS[1450] WARNING: Result from SNMP not valid.  Partial Result: U
05/09/2012 05:40:24 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.62.25', and OID:'.1.3.6.1.2.1.1.3.0'
05/09/2012 05:40:27 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.62.26', and OID:'.1.3.6.1.2.1.1.3.0'
05/09/2012 05:40:30 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.62.27', and OID:'.1.3.6.1.2.1.1.3.0'
05/09/2012 05:40:33 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.62.28', and OID:'.1.3.6.1.2.1.1.3.0'
05/09/2012 05:40:36 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.62.29', and OID:'.1.3.6.1.2.1.1.3.0'
05/09/2012 05:40:39 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.62.30', and OID:'.1.3.6.1.2.1.1.3.0'
05/09/2012 05:40:42 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'192.xxx.62.31', and OID:'.1.3.6.1.2.1.1.3.0'
05/09/2012 05:40:42 PM - SYSTEM STATS: Time:40.4713 Method:cmd.php Processes:1 Threads:N/A Hosts:61 HostsPerProcess:61 DataSources:2151 RRDsProcessed:1225
05/09/2012 05:40:42 PM - SYSTEM STATS: Time:40.4713 Method:cmd.php Processes:1 Threads:N/A Hosts:61 HostsPerProcess:61 DataSources:2151 RRDsProcessed:1225

Where should I start to debug?

Any recommendations?

[gandalf]

The "real" issues are the "Result: U" thingies. See 2nd link of my sig to debug, please
R.

[kjay]

I know this is a kinda dead thread, however thought I'd share my experience with the "poller table not empty" issue.

I found in my case that the poller_item table had one of the rows for a given data source that was being polled at a different time to all the others.
For example:
mysql> select rrd_name, rrd_step from poller_item where local_data_id = 5498;
+---------------------+----------+
| rrd_name | rrd_step |
+---------------------+----------+
| ALARM_PCT | 15 |
| ALARM_THRESH_BW | 15 |
| CONFIGURED_RATIO | 15 |
| MAX_PROV_BW_UP | 15 |
| PROVISIONED_BW | 15 |
| PROV_CONTENTION | 15 |
| PROV_CONTENTION_PCT | 0 |
| RATE_LIMIT | 15 |
+---------------------+----------+

When I manually updated the rrd_step value for PROV_CONTENTION_PCT the rrd file got updated correctly again.
However this issue kept reappearing, no matter if I removed/added graphs, restarted everything, etc.
My solution was:

- export the data query, data template, graph template from cacti
- delete all the exported items from cacti
- re-import everything

This fixed the issue for me. Hope it helps someone else, cheers