Hello,
I've already read many discussions about the "Poller Output Table not Empty" issue, but not able to fix it yet.
I explain my case (I run cacti 1.2.26 with spine 1.2.26, 1 minute collection):
Two days ago I create a new data input method named "Varnish Aggregated Stats", type: "Script/Command" and command:
/usr/bin/php /data/logger_scripts/get_varnish_aggregated_stats.php --last-minute --cacti
Such script runs in few milliseconds and output 40 metrics, one per line, with format key:value
After that I created a data source template named "Varnish Aggregated Stats" to map the 40 metrics grabbed from the script to internal (and shorter) keys.
After that I created 10 graph templates using the metrics above, and 13 data sources were created (later I deleted the 3 data sources not associated to any graph, probably caused by my mistakes during creation of the templates).
After that I added the graphs to two devices. All good, but there are several holes in my graphs, and I noticed the classic error "Poller Output Table not Empty" in my logs:
2024/10/28 08:59:21 - SYSTEM MAINT STATS: Time:0.01
2024/10/28 09:00:01 - POLLER: Poller[1] PID[1741500] WARNING: Poller Output Table not Empty. Issues: 5, DS[57346, 57336, 57335, 57342, 57337]
2024/10/28 09:00:01 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024/10/28 09:00:21 - SYSTEM STATS: Time:19.9534 Method:spine Processes:2 Threads:60 Hosts:19 HostsPerProcess:10 DataSources:1212 RRDsProcessed:1161
2024/10/28 09:00:22 - SYSTEM MAINT STATS: Time:0.01
2024/10/28 09:01:01 - POLLER: Poller[1] PID[1745334] WARNING: Poller Output Table not Empty. Issues: 4, DS[57346, 57342, 57336, 57334]
2024/10/28 09:01:01 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024/10/28 09:01:21 - SYSTEM STATS: Time:19.9652 Method:spine Processes:2 Threads:60 Hosts:19 HostsPerProcess:10 DataSources:1212 RRDsProcessed:1159
2024/10/28 09:01:21 - SYSTEM MAINT STATS: Time:0.01
2024/10/28 09:02:01 - POLLER: Poller[1] PID[1748925] WARNING: Poller Output Table not Empty. Issues: 6, DS[57342, 57337, 57336, 57335, 57334, 57346]
2024/10/28 09:02:01 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024/10/28 09:02:21 - SYSTEM STATS: Time:19.8194 Method:spine Processes:2 Threads:60 Hosts:19 HostsPerProcess:10 DataSources:1212 RRDsProcessed:1163
2024/10/28 09:02:21 - SYSTEM MAINT STATS: Time:0.00
2024/10/28 09:03:01 - POLLER: Poller[1] PID[1752525] WARNING: Poller Output Table not Empty. Issues: 2, DS[57336, 57335]
2024/10/28 09:03:01 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024/10/28 09:03:21 - SYSTEM STATS: Time:19.5803 Method:spine Processes:2 Threads:60 Hosts:19 HostsPerProcess:10 DataSources:1212 RRDsProcessed:1117
2024/10/28 09:03:21 - SYSTEM MAINT STATS: Time:0.00
Before removing the 3 unused data sources the errors were 2-3 more, on average.
You can notice that the data sources with errors are quite random. On my server I noticed that the script is properly called 10 times per minute (despite it outputs 10 times the same data - but I know this is by design, despite not good)
I've tried to remove on of the RRD on disk (prod-logger_an_2xx_g_ms_57335.rrd), it was recreated later, but it still has the same random issues.
I tried to edit the device and click on Save, but no luck.
I tried to unflag the "Balance Process Load" for the poller, but nothing changes.
I tried to rebuild the poller cache for one of the two devices affected, but no luck.
The RRDfile Checker doesn't report any issue.
I don't use Boost.
This error happens only with the last 10 graphs I created, my logs were all clean and green before.
I spent already 5-6 hours on it, but without luck. Can somebody try to give some advice?
Poller Output Table not Empty (another)
Moderators: Developers, Moderators
Re: Poller Output Table not Empty (another)
Look for a php error/warning in the cacti log about your script get_varnish_aggregated_stats.php
If you remove all graphs and data sources based on get_varnish_aggregated_stats.php, your "Poller Output Table not Empty." issue will disappear?
If you remove all graphs and data sources based on get_varnish_aggregated_stats.php, your "Poller Output Table not Empty." issue will disappear?
Let the Cacti grow!
Re: Poller Output Table not Empty (another)
Hello Macan,
thank you for your feedback.
Tonight I fixed it, but please let me know if this was the real reason.
I changed the output format of my script get_varnish_aggregated_stats.php from one key:value per line, to all key:value on the same line, space separated, and with a final carriage return.
I already tested this case in the past but didn't work. The only change I did was to use only one echo in my script (to print the full output all at once), instead of an echo for each key:value.
It seems that spine was fetching partial output from my script, without waiting the end of file. Is it possible?
I've now another problem after upgrading spine to a newer version, but I think is related to https://forums.cacti.net/viewtopic.php?t=63098 (under investigation)
thank you for your feedback.
Tonight I fixed it, but please let me know if this was the real reason.
I changed the output format of my script get_varnish_aggregated_stats.php from one key:value per line, to all key:value on the same line, space separated, and with a final carriage return.
I already tested this case in the past but didn't work. The only change I did was to use only one echo in my script (to print the full output all at once), instead of an echo for each key:value.
It seems that spine was fetching partial output from my script, without waiting the end of file. Is it possible?
I've now another problem after upgrading spine to a newer version, but I think is related to https://forums.cacti.net/viewtopic.php?t=63098 (under investigation)
Re: Poller Output Table not Empty (another)
it was thing what I saw in your post but I forget comment it. Sorry.
Your fix is correct. Poller expect all results on one line.
Your fix is correct. Poller expect all results on one line.
Let the Cacti grow!
Re: Poller Output Table not Empty (another)
thank you! I hope to be able to solve the other issue; otherwise I'll open a new post.
I found out that, by mistake, I was using an older spine (1.2.18) and not 1.2.26. After doing the "make install" the new executable was put in /usr/local/spine/bin/spine while in cacti config I was using /usr/local/spine/spine (without bin).
The error seems related to the parsing of only one python file (working with 1.2.18, but not with 1.2.28 - I just installed the most recent version too). Anyway, better to not go off-topic here
I found out that, by mistake, I was using an older spine (1.2.18) and not 1.2.26. After doing the "make install" the new executable was put in /usr/local/spine/bin/spine while in cacti config I was using /usr/local/spine/spine (without bin).
The error seems related to the parsing of only one python file (working with 1.2.18, but not with 1.2.28 - I just installed the most recent version too). Anyway, better to not go off-topic here
Who is online
Users browsing this forum: No registered users and 1 guest