Issues with SPINE?

[Solari]

Hi all,

Windows: Cacti 0.8.8a

After a while, I seem to get a "Stuck" Spine.exe that's hammering the CPUs. This doesn't seem to be normal behaviour as it only happens after a day or two of everything running fine. Eventually I get another "stuck" spine.exe so the CPU is constantly maxed and I start getting gaps in my graphs until I restart everything/reboot the box/kill the spine processes. Killing the rogue process(es) doesn't seem to affect anything as far as I can see.

Could this be something to do with the poller settings or a spine bug? I'm not running a massive deployment (~1000 graphs) and it's running on a Windows HyperV VM, Windows Server 2008 R2, 2xIntel Xeon X5670@2.93GHz and 4GB RAM. We are polling every minute due to the requirement for high granularity. Perhaps this has something to do with it.

I see these messages in the log, nothing untoward:

05/09/2013 01:09:12 AM - SYSTEM STATS: Time:12.1740 Method:spine Processes:2 Threads:4 Hosts:170 HostsPerProcess:85 DataSources:2508 RRDsProcessed:1088

I have the odd SNMP timeout where a host has become unreachable but that's about it.

I'd be really grateful if anyone has any ideas.

Many thanks in advance!

Date Thu, 09 May 2013 01:22:40 -0700
Cacti Version 0.8.8a
Cacti OS win32
SNMP Version NET-SNMP version: 5.5
RRDTool Version RRDTool 1.4.x
Hosts 169
Graphs 971
Data Sources Script/Command: 5
SNMP: 300
SNMP Query: 893
Script Query - Script Server: 3
Total: 1201
Poller Information
Interval 60
Type SPINE 0.8.8a Copyright 2002-2012 by The Cacti Group
Items Action[0]: 2744
Action[2]: 6
Total: 2750
Concurrent Processes 2
Max Threads 4
PHP Servers 1
Script Timeout 25
Max OID 10
Last Run Statistics Time:12.0994 Method:spine Processes:2 Threads:4 Hosts:170 HostsPerProcess:85 DataSources:2501 RRDsProcessed:1094
PHP Information
PHP Version 5.3.17
PHP OS WINNT
PHP uname Windows NT XXXXXXXX 6.1 build 7601 (Windows Server 2008 R2 Standard Edition Service Pack 1) i586
PHP SNMP Installed
max_execution_time 30
memory_limit 128M

[BSOD2600]

Running any scripts/templates which don't come with the stock install? Typically if a rogue script doesnt have time out logic built in, this can cause spine.exe to hang and fail to terminate.

Using a tool like Sysinternals Process Explorer, investigate the problematic spine.exe process the next time it occurs. Whats the commandline arguments? Whats the callstack? Could try turning up the cacti logging level, but that might be a bit of a needle in the haystack to find the source.

[Solari]

All the polling I'm doing is via SNMP as far as I remember but I will double check. I'd have expected anything hanging to have just stuck around and not used up so much CPU, it's as if it's in a loop of some sort. I did clear out some problematic graphs as reported by the poller in the logs (no/bad snmp responses) so maybe that will help.

I'll see if I can get SysInternals loaded on there, I'll need permission for that but don't think it'll be a problem. Unfortunately there doesn't appear to be a separate spine log as far as I can see. Logging levels may work, I'll give it a go as I should be able to see where to look from where the graphs start going blank (seeing as it's a minute poll rate it shouldn't be too hard to find), thanks! Unfortunately it takes about a day or so usually until the problem occurs.

I'll report back with what I find. Much appreciated!

[Solari]

Strangely, this issue hasn't returned since I spent a bit of time clearing out some errors (output not empty) on some datasources.

Thanks for the suggestions, I'll continue to monitor.