intelligent spike killer

[arg0n]

The FAQ says this

One of my devices rebooted and now I have a huge spike on my graph!

This occurs because the reboot causes SNMP's counters to reset, which can cause a rather large spike on the graph when RRDTool tries to determine the change between the new small counter value and the large previous value. One way to combat this issue is to specify realistic maximum values for your data sources. RRDTool will ignore any value that is larger than the maximum value.


Can Cacti be smart about this situation and realise that the SNMP value is lower. A counter data source should always be increasing... Cacti should realise that the counter has been reset. does this make sense ?

[TheWitness]

This is a perplexing issue in that it is almost impossible to eliminate the spike due to the nature of the counter rrd format. Will continue to think about it until such time as I have more time to think about it. Until then, there are a few utilities on this web site that provide spike killing utilities.

Good luck!!

TheWitness

[bulek]

Cacti should realise that the counter has been reset. does this make sense ?

It does not . Counter because of its finite nature can overflow - means that just after you read very high value you can read value just above 0 in the next polling cycle. RRDTool tries to handle such situation and write to rrd file correct value. As a side effect it causes spikes in unexpected counter reset case. As you can see Cacti can be only as smart as you are when providing correct maximum value (as you mentioned in your post).

- Piotr

[Jasper]

This is a perplexing issue in that it is almost impossible to eliminate the spike due to the nature of the counter rrd format. Will continue to think about it until such time as I have more time to think about it. Until then, there are a few utilities on this web site that provide spike killing utilities.

Good luck!!

TheWitness

Actually it's easy to stop it happening, at least for SNMP counters:

When you grab an oid value, grab sysUpTime at the same time.

if sysUpTime < your sample interval then you know the device you are sampling was reset since you last sampled it.

In that case don't just enter the value you sampled into the rrd, enter NaN and then the value instead.


Since there is now a NaN entry in the rrd rrdtool won't try to interpolate the pre and post crash records, there will be a gap in the graphs, but you won't get spikes anymore.

(this won't help if there is a way to reset the snmp stats for an interface independent of rebooting the box you are sampling, but it's against the snmp specs to do that).

This method has been used in cricket for ages, I patched cacti-0.8.5a, but had to use the machine i was developeing on for something else so I havn't had time to port the changes to 0.8.6. (the changes in 0.8.6 make it harder to implement).

I really should of got back to this problem ages ago (i don't use cacti at the moment so it's not an issue for me), here's the patch for 0.8.5a:

http://pointless.net/~jasper/cacti-no-spikes.patch

have fun!

[sejtam]

So checking against system uptime sounds like the ticket to prevent spikes.

However, in mycase almost all input is from scripts, so an SNMP sysUptime does not exist.

I can have my script return the sysuptime as part of the data input method.

How then can I make cacti utilize that part? Is there a patch which allows me to specify a data item which denotes the sysuptime?

[TheWitness]

Jasper,

It's easier to fix that you think. Since we track if the host requires recache, before polling, all you need to do is set a binary value and enter NaN or should it be "U" for that sample for all Counter type data.

TheWitness

[mgb]

Jasper,

It's easier to fix that you think. Since we track if the host requires recache, before polling, all you need to do is set a binary value and enter NaN or should it be "U" for that sample for all Counter type data.

TheWitness

Is this something for the new release?

Michael

[TheWitness]

Yup. Just got to keep from forgetting it.

TheWitness

[TheWitness]

I might just patch now that I think about it.

[Atle]

I have found that in some cases it is better to use DERIVE rather than COUNTER fwiw. Instead of a spike, you will get a tiny gap in your graph.

[TheWitness]

Try this:

http://forums.cacti.net/viewtopic.php?t=6899

TheWitness