Poller Output Table not Empty (another)

[black]

Hello,
I've already read many discussions about the "Poller Output Table not Empty" issue, but not able to fix it yet.

I explain my case (I run cacti 1.2.26 with spine 1.2.26, 1 minute collection):

Two days ago I create a new data input method named "Varnish Aggregated Stats", type: "Script/Command" and command:
/usr/bin/php /data/logger_scripts/get_varnish_aggregated_stats.php --last-minute --cacti
Such script runs in few milliseconds and output 40 metrics, one per line, with format key:value

After that I created a data source template named "Varnish Aggregated Stats" to map the 40 metrics grabbed from the script to internal (and shorter) keys.

After that I created 10 graph templates using the metrics above, and 13 data sources were created (later I deleted the 3 data sources not associated to any graph, probably caused by my mistakes during creation of the templates).

After that I added the graphs to two devices. All good, but there are several holes in my graphs, and I noticed the classic error "Poller Output Table not Empty" in my logs:

2024/10/28 08:59:21 - SYSTEM MAINT STATS: Time:0.01
2024/10/28 09:00:01 - POLLER: Poller[1] PID[1741500] WARNING: Poller Output Table not Empty. Issues: 5, DS[57346, 57336, 57335, 57342, 57337]
2024/10/28 09:00:01 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024/10/28 09:00:21 - SYSTEM STATS: Time:19.9534 Method:spine Processes:2 Threads:60 Hosts:19 HostsPerProcess:10 DataSources:1212 RRDsProcessed:1161
2024/10/28 09:00:22 - SYSTEM MAINT STATS: Time:0.01
2024/10/28 09:01:01 - POLLER: Poller[1] PID[1745334] WARNING: Poller Output Table not Empty. Issues: 4, DS[57346, 57342, 57336, 57334]
2024/10/28 09:01:01 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024/10/28 09:01:21 - SYSTEM STATS: Time:19.9652 Method:spine Processes:2 Threads:60 Hosts:19 HostsPerProcess:10 DataSources:1212 RRDsProcessed:1159
2024/10/28 09:01:21 - SYSTEM MAINT STATS: Time:0.01
2024/10/28 09:02:01 - POLLER: Poller[1] PID[1748925] WARNING: Poller Output Table not Empty. Issues: 6, DS[57342, 57337, 57336, 57335, 57334, 57346]
2024/10/28 09:02:01 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024/10/28 09:02:21 - SYSTEM STATS: Time:19.8194 Method:spine Processes:2 Threads:60 Hosts:19 HostsPerProcess:10 DataSources:1212 RRDsProcessed:1163
2024/10/28 09:02:21 - SYSTEM MAINT STATS: Time:0.00
2024/10/28 09:03:01 - POLLER: Poller[1] PID[1752525] WARNING: Poller Output Table not Empty. Issues: 2, DS[57336, 57335]
2024/10/28 09:03:01 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024/10/28 09:03:21 - SYSTEM STATS: Time:19.5803 Method:spine Processes:2 Threads:60 Hosts:19 HostsPerProcess:10 DataSources:1212 RRDsProcessed:1117
2024/10/28 09:03:21 - SYSTEM MAINT STATS: Time:0.00

Before removing the 3 unused data sources the errors were 2-3 more, on average.

You can notice that the data sources with errors are quite random. On my server I noticed that the script is properly called 10 times per minute (despite it outputs 10 times the same data - but I know this is by design, despite not good)

I've tried to remove on of the RRD on disk (prod-logger_an_2xx_g_ms_57335.rrd), it was recreated later, but it still has the same random issues.
I tried to edit the device and click on Save, but no luck.
I tried to unflag the "Balance Process Load" for the poller, but nothing changes.
I tried to rebuild the poller cache for one of the two devices affected, but no luck.
The RRDfile Checker doesn't report any issue.
I don't use Boost.

This error happens only with the last 10 graphs I created, my logs were all clean and green before.

I spent already 5-6 hours on it, but without luck. Can somebody try to give some advice?

[macan]

Look for a php error/warning in the cacti log about your script get_varnish_aggregated_stats.php

If you remove all graphs and data sources based on get_varnish_aggregated_stats.php, your "Poller Output Table not Empty." issue will disappear?

[black]

Hello Macan,
thank you for your feedback.

Tonight I fixed it, but please let me know if this was the real reason.

I changed the output format of my script get_varnish_aggregated_stats.php from one key:value per line, to all key:value on the same line, space separated, and with a final carriage return.

I already tested this case in the past but didn't work. The only change I did was to use only one echo in my script (to print the full output all at once), instead of an echo for each key:value.
It seems that spine was fetching partial output from my script, without waiting the end of file. Is it possible?

I've now another problem after upgrading spine to a newer version, but I think is related to https://forums.cacti.net/viewtopic.php?t=63098 (under investigation)

[macan]

it was thing what I saw in your post but I forget comment it. Sorry.

Your fix is correct. Poller expect all results on one line.

[black]

thank you! I hope to be able to solve the other issue; otherwise I'll open a new post.

I found out that, by mistake, I was using an older spine (1.2.18) and not 1.2.26. After doing the "make install" the new executable was put in /usr/local/spine/bin/spine while in cacti config I was using /usr/local/spine/spine (without bin).

The error seems related to the parsing of only one python file (working with 1.2.18, but not with 1.2.28 - I just installed the most recent version too). Anyway, better to not go off-topic here