Poller Output Table not Empty

[trayhons]

I am running Alma Linux 8 and I was running Cacti 1.2.16. I ran "yum update" on this server and via the EPEL repository, my cacti package upgraded to version 1.2.19. I decided to check the cacti logs (which I haven't done in a while) to see if there are any issues post upgrade. I am seeing the following warning in the cacti.log file.

2021/11/18 14:15:55 - SYSTEM STATS: Time:53.4435 Method:cmd.php Processes:1 Threads:1 Hosts:222 HostsPerProcess:222 DataSources:3704 RRDsProcessed:2536
2021/11/18 14:15:01 - POLLER: Poller[1] PID[267154] WARNING: Poller Output Table not Empty. Issues: 2, DS[4795, 4795]

I have about a month's history of cacti logs and I verified that this issue was occurring prior to the upgrade to version 1.2.19. Since this warning shows up in all of my logs (going back one month), I don't know when this started occurring.

The result of this warning is that the corresponding RRD file is not being updated which is evident when you look at the corresponding graph.

I've tried several things to resolve this issue.

1) I went to the device management page in the UI and clicked "Save" as I read this will repopulate the poller cache for this device. No change in behavior after this attempt.
2) I deleted the graph and recreated it. The new graph is working but another graph is listed in the Poller Output Table not Empty warning. I repeated deleting the graph/DS in the warning with the new graph working but an older graph would stop working and be listed in the warning.
3) I found another instance where if your poller is taking longer then the polling interval that this issue can occur. My polling interval is 5 minutes and my poller takes less than 60 seconds to complete.
4) I increased the polling processes to 2. This did decrease the polling time but not significantly. This did not have any affect on the situation. I changed it back to 1.
5) I went to cacti.net to verify that I was on the latest version of Cacti. I see that version 1.2.19 is the current version available.

At present, I have deleted and recreated the graphs/DS to the point where the graph/DS that is associated with the warning is a non-critical graph. This is acceptable until this is resolved.

If needed, I can supply logs and debugging. Please let me know what further information I can provide to resolve this issue. Thanks!

[trayhons]

I have continued to look into this further. What I have found is that when the data for data store 4795 is stored in the poller_output table, the timestamp for the corresponding rrd_name records were not the same. This particular data store has two rrd_names. When the check "make sure each .rrd file has complete data" at line 525 of lib/poller.php, the $rrd_update_array element for this data store was being "unset" because it was only finding one rrd_name per time stamp. However, the correct data was in the poller_output table just that the time stamps were a couple seconds apart.

This lead me to wonder and investigate how this data is populated into the poller_output table. I found the INSERT in cmd.php. The INSERT was using CURRENT_TIMESTAMP() to populate the "time" field of this table. With the two rrd_names being a couple seconds apart, that tells me that it took this script a few seconds between polling the first rrd_name and the second rrd_name. It's possible that other data stores for this host were in between the two rrd_names, but I didn't track that down.

I implemented a fix that keeps the time the same for all data stores and rrd_names of a host in cmd.php. I can share my code changes if anyone is interested.

All of my data stores are updating now which correlates to updated graphs for my users. In between posting my original issue on November 18 and now, I've had several other data stores show up in the 'Poller Output Table not Empty' warning. Since implementing my changes, I haven't received any warnings.

[KevinNoall]

I'm getting something similar, so if you could post your method to fix I would appreciate

[trayhons]

I am currently running version 1.2.25 and my custom code was overwritten during an upgrade since Nov 2021. I am currently running the stock code base. Sorry, I don't have the custom code anymore.

[TheWitness]

First, make sure you are running spine. Add more threads if the poller runs too long. Second, increase the error reporting to get poller collector errors. There is an option for this under logging or poller settings to see detailed poller errors. That'll clue you in to what is broken.