WARNING: Poller Output Table not Empty

[imro]

I have "Poller Output Table not Empty" warnings in my log file and the graphs associated with those data sources are empty. I have read some other posts about this problem, but they seem to be a different issue. They are either related to spine (which I am not using), affect all graphs, poller interval being set to 1 and cron to 5 and/or poller taking long time to finish. I do not experience any of them. In my situation all of the graphs are for disk/memory usage on windows hosts. My cron as well as poller interval are set to 1 minute.

The problem, in my opinion, is that the data template for "Host MIB - Hard Drive Space" is composite of three different data source items and is set to poll only every 5 minutes. Now when poller cache is generated and rrd_next_step is assigned to the three different data sources they get different values. So they get polled in different poller cycles. Since all the data required for given data template is not available within one cycle, it does not get written to rrd.

[imro]

I can confirm that syncing the rrd_next_step for all data sources of particular rrd, by manually updating poller_item table, fixes the problem.

But another problem has surfaced. For some rras based on "Host MIB - Hard Drive Space" the poller item table is not correctly populated. It misses one of the 3 data sources.

[imro]

OK, so after few hours of code reading I have found the problem: a bad template!

Through out this process I have found at lease three caveats of Cacti that allowed this to happen:

1. Cacti does not take care of inconsistencies in built-in templates between versions. "Host MIB - Hard Drive Space" is a built-in template, and in the course of version changes has been tweaked a little by dropping the hrStoragePercent data source item. This would be OK if the upgrade process took care of it, but at least in my case it has not. Data template still had this item, but it got deleted out of host_disk.xml.

2. The way poller cache rebuild process keeps sure that all items from a single template are in sync, when it comes to rrd_next_step, can be thrown off by bad template. It happens because api_poller_get_rrd_next_step is build on assumption that all poller items come in order. Which generally works, until there is a bad template.

3. Poller cache rebuilt process does not properly take care of missing fields in SNMP or script query xml files. In lib/utility.php the $script_path is not cleared after each $output and therefore even if a field is missing for particular $output, old $script_path is used to call api_poller_cache_item_add.

Anyway, thanks everybody for help. I could not have done it without you.

[imro]

Master's my friend!

[imro]

You don't have any friends. Nobody likes you!

[imro]

Not listening, I'm not listening.