Creating certain thresholds causes Cacti poller to fail

[skatter]

So I have created a number of thresholds, and I've noticed that when certain thresholds are enabled thold no longer runs, and my poller interval, using spine, is not respected.

I am using Cacti 0.8.7g, Spine 0.8.7g, and thold 0.4.4.

As I stated, certain thresholds cause thold to break entirely, which in turn also breaks spine. If I'm watching cacti.log, I don't see thold running, and my poller interval (set to 1 minute) only runs at the cron interval (5 minutes). If I disable the offending threshold, spine starts polling every 1 minute again, populating graphs and working as expected. If I enable other thresholds, which don't cause this problem, everything works as expected.

What makes a threshold an offender is still a mystery to me, as I have some traffic_in/traffic_out thresholds that break thold, while other traffic_in/traffic_out thresholds work normally.

Is there a way to enable debug level logging for thold to find out what it is in the offending threshold that is causing the problem?

[skatter]

Problem solved, this was caused when using 64-bit counters on data sources that do not support it (Serial interfaces on Cisco routers, in this case). When I configure thresholds on these data sources it throws everything in Cacti out of whack. I re-configured the data sources with 32-bit counters and they work fine with, and without, thresholds.

[skatter]

I spoke too soon. While changing the data sources from 64 -> 32 bit did seem to resolve the matter temporarily, it just made it more intermittent.

Here's an excerpt from my cacti.log, showing a window of 16 hours or so where Cacti stopped processing thold information:

Code: Select all

08/25/2011 05:32:14 PM - SYSTEM THOLD STATS: Time:0.3030 Tholds:38 Hosts:0
08/25/2011 05:33:13 PM - SYSTEM THOLD STATS: Time:0.2957 Tholds:38 Hosts:0
08/25/2011 05:34:14 PM - SYSTEM THOLD STATS: Time:0.3106 Tholds:38 Hosts:0
08/25/2011 05:35:11 PM - SYSTEM THOLD STATS: Time:0.3224 Tholds:38 Hosts:0
<-- GAP -->
08/26/2011 09:50:11 AM - SYSTEM THOLD STATS: Time:0.3058 Tholds:38 Hosts:0
08/26/2011 09:51:11 AM - SYSTEM THOLD STATS: Time:0.3136 Tholds:38 Hosts:0
08/26/2011 09:52:12 AM - SYSTEM THOLD STATS: Time:0.3126 Tholds:38 Hosts:0
08/26/2011 09:53:13 AM - SYSTEM THOLD STATS: Time:0.3066 Tholds:38 Hosts:0

At the same time that thold stops working, my poller continues to run at the cron interval, but the poller interval of 1 minute is not respected:

Code: Select all

08/25/2011 05:34:13 PM - SYSTEM STATS: Time:11.7612 Method:spine Processes:8 Threads:10 Hosts:70 HostsPerProcess:9 DataSources:1221 RRDsProcessed:703
08/25/2011 05:35:11 PM - SYSTEM STATS: Time:11.2514 Method:spine Processes:8 Threads:10 Hosts:70 HostsPerProcess:9 DataSources:1221 RRDsProcessed:703
08/25/2011 05:36:11 PM - SYSTEM STATS: Time:10.4466 Method:spine Processes:8 Threads:10 Hosts:70 HostsPerProcess:9 DataSources:1221 RRDsProcessed:703
<-- poller interval no longer respected & runs every 5 minutes -->
08/25/2011 05:40:09 PM - SYSTEM STATS: Time:8.9724 Method:spine Processes:8 Threads:10 Hosts:70 HostsPerProcess:9 DataSources:1221 RRDsProcessed:703
08/25/2011 05:45:11 PM - SYSTEM STATS: Time:10.7492 Method:spine Processes:8 Threads:10 Hosts:70 HostsPerProcess:9 DataSources:1221 RRDsProcessed:703
08/25/2011 05:50:10 PM - SYSTEM STATS: Time:9.9515 Method:spine Processes:8 Threads:10 Hosts:70 HostsPerProcess:9 DataSources:1221 RRDsProcessed:703
08/25/2011 05:55:09 PM - SYSTEM STATS: Time:8.3860 Method:spine Processes:8 Threads:10 Hosts:70 HostsPerProcess:9 DataSources:1221 RRDsProcessed:703
08/25/2011 06:00:09 PM - SYSTEM STATS: Time:9.5674 Method:spine Processes:8 Threads:10 Hosts:70 HostsPerProcess:9 DataSources:1221 RRDsProcessed:703
08/25/2011 06:05:12 PM - SYSTEM STATS: Time:12.1527 Method:spine Processes:8 Threads:10 Hosts:70 HostsPerProcess:9 DataSources:1221 RRDsProcessed:703

Both events happen at the same time, and normal operation resumed after thold started running again the next day. Unfortunately I have no graph data during this time, and am forced to disable thold entirely.

Who supports thold that would be able to provide guidance on what could be causing this?

[TheWitness]

I would have needed to see your apache error log, and your /var/log/messages during that time period.

[skatter]

I can duplicate this at any time; just did a moment ago. I have a number of thresholds in the disabled state, and as soon as I enable them this problem occurs.

What data would you like me to collect while in a non-functional state that would assist in diagnosing the problem?

FYI - I have updated thold to 0.4.4-45 and the problem persists.

[TheWitness]

This problem does not exist in the forthcoming 0.4.6 release.

TheWitness

[skatter]

This problem does not exist in the forthcoming 0.4.6 release.

TheWitness

That is good news. Is there a workaround while 0.4.6 is pending release?