Spine Timed Out While Waiting for Threads to End

[ocolin]

Hello!

I am a bit new to Cacti and this is my first forum post. I am having an issue with Cacti which results in blank sections on graphs. When I check the cacti log files I see the following error periodically, which I suspect must be related:

017-08-24 12:06:04 - SPINE: Poller[1] ERROR: Spine Timed Out While Waiting for Threads to End

I am using spine and have been playing with as many variations of settings as I can. It seems no matter how many processes and threads I add the error does not go away. I eventually get the following errors if Increase the threads and processes too ghih:

SPINE: Poller[1] FATAL: Connection Failed, Error:'1040', Message:'Too many connections' [0, Success] (Spine thread)

I have yet to find any settings that put any strain on the CPU or run out of RAM. The last error happens even with MySQL set to 2000 max connections. So I know my issue isn't CPU load or memory, but the bottleneck is mySQL connections. Can I jsut keep cranking up the max connections in MySQL? 2000 seems higher than anyone I have seen set theirs in my searches on the forum. Perhaps there are other MySQL settings involved?

Information is needed for anyone to help me diagnose the problem and I am not sure what information to post. So here is what I can think of to post so far:

Server settings:
rrdtool 1.3.8
cacti 1.1.16
spine 1.1.12
MySQL 5.5.57
Centos 6.9
PHP 5.6.31
Devices: 958
Graphs: 9833
RAM: 4G
MySQL max connections = 2000

Poller settings:
Poller Type: spine
Poller Interval: every 10 seconds
Cron Interval: 5 min
Maximum Concurrent Poller Processes: 7
Balance Process Load: checked
Disable increasing OID Check: Unchecked
Invalid Data Logging: Detailed
Maximum Threads per Process: 20
Number of PHP Script Servers: 6
Script and Script Server Timeout Value: 25
The Maximum SNMP OIDs Per SNMP Get Request: 7

The poller settings started out much lower and I worked them up, then backed them down to get rid of the too many connections errors. I would love some recommendations and happy to prove mode information if requested. Thanks!

[ocolin]

While I have not resolved the error in the log file, I have resolved the dropouts in the graphs by changing the poller interval from 10 seconds to 1 minute. I suspect that since I am only checking every 5 minutes I might be able to change the poller time to 5 minutes, but still not 100% about it. Looking at it now, 10 seconds sound absurd, but that's what it was set to by default.

[Osiris]

How many devices?

[Hipska]

I have this message also in my log file. Graphs have no gaps.

2017/10/13 11:35:01 - POLLER: Poller[1] Maximum runtime of 298 seconds exceeded. Exiting.
2017/10/13 11:35:01 - SYSTEM STATS: Time:54.4122 Method:spine Processes:1 Threads:50 Hosts:268 HostsPerProcess:268 DataSources:23524 RRDsProcessed:0
2017/10/13 11:35:02 - POLLER: Poller[1] WARNING: There are '1' detected as overrunning a polling process, please investigate
2017-10-13 11:35:07 - SPINE: Poller[1] ERROR: Spine Timed Out While Waiting for Threads to End

[yohannnnnnn]

Hello,

Do you find an issue ?

[Atreides]

Hello,
I've had the same problem with spine dying, and the issue was that the path to spine.conf was missing in cacti settings. Check there(usually /etc/spine/spine.conf or /usr/share/spine/spine.conf).