Polls for 30 seconds then nothing until runtime exceeded

[mandragor]

This is on cacti 1.1.38 with spine running on Ubuntu 16.04. I have migrated an old 0.8.8b cacti-installation running on old hardware that also had "gaps" in its graphs, hoping that it
was due to lack of resources. I suspect spine is being blocked by a difficult host, but since one symptom is lack of logging I am unsure how to proceed..

If I set loglevel to debug I will see something like this
2018/09/07 20:55:30 - SPINE: Poller[1] SPINE: The Final Value of Threads is 0
2018/09/07 20:55:30 - SPINE: Poller[1] DEBUG: Thread Cleanup Complete
2018/09/07 20:55:30 - SPINE: Poller[1] DEBUG: PHP Script Server Pipes Closed
2018/09/07 20:55:30 - SPINE: Poller[1] DEBUG: Allocated Variable Memory Freed
2018/09/07 20:55:30 - SPINE: Poller[1] DEBUG: MYSQL Free & Close Completed
2018/09/07 20:55:30 - SPINE: Poller[1] DEBUG: Net-SNMP Close Completed
2018/09/07 20:55:30 - SPINE: Poller[1] Time: 28.1343 s, Threads: 20, Devices: 204
2018/09/07 21:00:00 - POLLER: Poller[1] Maximum runtime of 298 seconds exceeded. Exiting.

The runtime varies from 20-30 seconds before logging stops until the "runtime exceeded"-error.

I've attached the General Information and Poller Information from the Technical Support page if that helps.

[netniV]

What are your poller and spine, threads/processes settings?

[mandragor]

Maximum Concurrent Poller Processes 4
Maximum Threads per Process 20
Number of PHP Script Servers 10
Script and Script Server Timeout Value 10
The Maximum SNMP OIDs Per SNMP Get Request 10

These settings worked fine for years without any changes and on the new VM there is no shortage of CPU or memory, so
that's not it. And the gaps don't happen on all graphs, so I would assume that something is blocking one or more of the
threads from completing its polling cycle?

[netniV]

Update your logging to medium, you will be able to see the results returned from each device. This may be quite a lot but essentially, you are looking for any returning 'U' and then the log entries that relate to it.

[mandragor]

Thank you for your suggestion, I got it sorted now but I'm unsure what really did it..

I found about 30 devices returning U, but I couldn't see anything in the logs that told me why.
I tried increasing snmp_timeout, but it didn't make any visible difference.
I disabled all the "U"-returning devices, deleting some duplicate devices I found in the process and on some devices re-ran Verbose Query.
I added the disabled devices, a few at a time, and expected the error to re-appear once I added the "right" device - but even after adding all devices
the polling cycle completes as it should in under 40 seconds and no more gaps in the raph.

[netniV]

It is likely due to a device that you've removed then. Devices that return 'U' for a status can do so for several reasons, one being a timeout, one being that the device simple does not support the OID being queried, one being that the SNMP auth parameters are wrong, etc.