Holt-Winters functionality (forecasting) with Cacti

[scruboy01]

Has anyone made a template for the HW data? I'm having some issues getting the CDEFs correct and with some of the graph template.

[Jem]

Hi to all,

This functionnality looks awesome, but I would like to check that I well understand the use of it.

What I'm looking for is to have a forecast on my links to check that my links bandwith is sufficient, I think this would be better with the image bellow:



In fact, what I would love to see is a graph like this, My real datas, and then the blue line which is the prediction using the real datas, can Holt-Winters works like this ?

Thanks for your answers.

[koaps]

The easiest way to deal with the CDEF's for the confidence bounds is to hardcode it to the DEFs for the graphs you want.

For example, if you only have one DS to graph, bps in my case, it will be DEF a, and you will have HW and Dev predict as DEF's b and c.

DEF:a="/Local.rrd":bps:AVERAGE \
DEF:b="/Local.rrd":bps:HWPREDICT \
DEF:c="/Local.rrd":bps:DEVPREDICT \

You can now easily make a CDEF = b,c,2,*,+ for upper and b,c,2,*,- for lower bounds.

I attached an image of the CDEF, and my Graph template layout and the Upper template, it uses the Upper CDEF.

If you have more than one DS to graph, I would try to group things to make it easier for instance:

DEF:a="/Local.rrd":bps_in:AVERAGE \
DEF:b="/Local.rrd":bps_in:HWPREDICT \
DEF:c="/Local.rrd":bps_in:DEVPREDICT \

DEF:d="/Local.rrd":bps_out:AVERAGE \
DEF:e="/Local.rrd":bps_out:HWPREDICT \
DEF:f="/Local.rrd":bps_out:DEVPREDICT \

You would then have to make Upper_In and Upper_Out using the correct DEF letters in the math for the CDEF's.

I really hope Cacti gets TICK support at some point, it would be very useful.

[koaps]

I worked out tick support.

Check this post out:

http://forums.cacti.net/viewtopic.php?t=29963

[Christian]

Hi,

I got it working recently, but then I resized the SEASONAL rra in order to save 1 week of data. Now the prediction values are all NaN, except for the SEASONAL ones.
Do I have to wait one week until the data shows up in my graphs or is something simply not working like it should?

cacti poller log output:

Code: Select all

03/12/2009 09:35:01 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /srv/www/htdocs/cacti/rra/als-hwkl-de01_traffic_in_360.rrd --template traffic_in:traffic_out 1236846901:1353753669475:734534484581
03/12/2009 09:35:01 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /srv/www/htdocs/cacti/rra/als-hwkl-de01_traffic_in_361.rrd --template traffic_in:traffic_out 1236846901:844633059:96368053 

rrdtool fetch output:

Code: Select all

>rrdtool fetch als-hwkl-de01_traffic_in_361.rrd DEVPREDICT -s -15min
                     traffic_in         traffic_out

1236847800: nan nan
1236848100: nan nan
1236848400: nan nan
1236848700: nan nan
>rrdtool fetch als-hwkl-de01_traffic_in_361.rrd DEVSEASONAL -s -15min
                     traffic_in         traffic_out

1236847800: nan nan
1236848100: nan nan
1236848400: nan nan
1236848700: nan nan
>rrdtool fetch als-hwkl-de01_traffic_in_361.rrd HWPREDICT -s -15min
                     traffic_in         traffic_out

1236847800: nan nan
1236848100: nan nan
1236848400: nan nan
1236848700: nan nan
> rrdtool fetch als-hwkl-de01_traffic_in_361.rrd SEASONAL -s -15min
                     traffic_in         traffic_out

1236847800: 3.8900287401e+05 1.3175177146e+05
1236848100: 2.8396709241e+05 9.2689563032e+04
1236848400: 3.2382529023e+05 8.9881003473e+04
1236848700: nan nan

rrdtool info:

Code: Select all

> rrdtool info als-hwkl-de01_traffic_in_361.rrd                    filename = "als-hwkl-de01_traffic_in_361.rrd"
rrd_version = "0003"
step = 300
last_update = 1236848701
ds[traffic_in].type = "COUNTER"
ds[traffic_in].minimal_heartbeat = 600
ds[traffic_in].min = NaN
ds[traffic_in].max = 3.4010000000e+07
ds[traffic_in].last_ds = "1479377451"
ds[traffic_in].value = 4.1099257667e+05
ds[traffic_in].unknown_sec = 0
ds[traffic_out].type = "COUNTER"
ds[traffic_out].minimal_heartbeat = 600
ds[traffic_out].min = NaN
ds[traffic_out].max = 3.4010000000e+07
ds[traffic_out].last_ds = "317955184"
ds[traffic_out].value = 7.9626486667e+04
ds[traffic_out].unknown_sec = 0
rra[0].cf = "AVERAGE"
rra[0].rows = 500
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[0].cdp_prep[1].value = NaN
rra[0].cdp_prep[1].unknown_datapoints = 0
rra[1].cf = "AVERAGE"
rra[1].rows = 600
rra[1].pdp_per_row = 1
rra[1].xff = 5.0000000000e-01
rra[1].cdp_prep[0].value = NaN
rra[1].cdp_prep[0].unknown_datapoints = 0
rra[1].cdp_prep[1].value = NaN
rra[1].cdp_prep[1].unknown_datapoints = 0
rra[2].cf = "AVERAGE"
rra[2].rows = 700
rra[2].pdp_per_row = 6
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = 4.1073030466e+05
rra[2].cdp_prep[0].unknown_datapoints = 0
rra[2].cdp_prep[1].value = 7.9688130267e+04
rra[2].cdp_prep[1].unknown_datapoints = 0
rra[3].cf = "AVERAGE"
rra[3].rows = 775
rra[3].pdp_per_row = 24
rra[3].xff = 5.0000000000e-01
rra[3].cdp_prep[0].value = 4.4733808467e+06
rra[3].cdp_prep[0].unknown_datapoints = 0
rra[3].cdp_prep[1].value = 2.4366267847e+06
rra[3].cdp_prep[1].unknown_datapoints = 0
rra[4].cf = "AVERAGE"
rra[4].rows = 797
rra[4].pdp_per_row = 288
rra[4].xff = 5.0000000000e-01
rra[4].cdp_prep[0].value = 1.2907689381e+07
rra[4].cdp_prep[0].unknown_datapoints = 0
rra[4].cdp_prep[1].value = 8.9514376007e+06
rra[4].cdp_prep[1].unknown_datapoints = 0
rra[5].cf = "MAX"
rra[5].rows = 500
rra[5].pdp_per_row = 1
rra[5].xff = 5.0000000000e-01
rra[5].cdp_prep[0].value = NaN
rra[5].cdp_prep[0].unknown_datapoints = 0
rra[5].cdp_prep[1].value = NaN
rra[5].cdp_prep[1].unknown_datapoints = 0
rra[6].cf = "MAX"
rra[6].rows = 600
rra[6].pdp_per_row = 1
rra[6].xff = 5.0000000000e-01
rra[6].cdp_prep[0].value = NaN
rra[6].cdp_prep[0].unknown_datapoints = 0
rra[6].cdp_prep[1].value = NaN
rra[6].cdp_prep[1].unknown_datapoints = 0
rra[7].cf = "MAX"
rra[7].rows = 700
rra[7].pdp_per_row = 6
rra[7].xff = 5.0000000000e-01
rra[7].cdp_prep[0].value = 4.1073030466e+05
rra[7].cdp_prep[0].unknown_datapoints = 0
rra[7].cdp_prep[1].value = 9.8128086807e+04
rra[7].cdp_prep[1].unknown_datapoints = 0
rra[8].cf = "MAX"
rra[8].rows = 775
rra[8].pdp_per_row = 24
rra[8].xff = 5.0000000000e-01
rra[8].cdp_prep[0].value = 4.1171964275e+05
rra[8].cdp_prep[0].unknown_datapoints = 0
rra[8].cdp_prep[1].value = 9.8237296167e+05
rra[8].cdp_prep[1].unknown_datapoints = 0
rra[9].cf = "MAX"
rra[9].rows = 797
rra[9].pdp_per_row = 288
rra[9].xff = 5.0000000000e-01
rra[9].cdp_prep[0].value = 8.8556089256e+05
rra[9].cdp_prep[0].unknown_datapoints = 0
rra[9].cdp_prep[1].value = 1.4279442042e+06
rra[9].cdp_prep[1].unknown_datapoints = 0
rra[10].cf = "HWPREDICT"
rra[10].rows = 4032
rra[10].pdp_per_row = 1
rra[10].alpha = 1.0000000000e-01
rra[10].beta = 3.5000000000e-03
rra[10].cdp_prep[0].intercept = 8.3500166667e+03
rra[10].cdp_prep[0].slope = 0.0000000000e+00
rra[10].cdp_prep[0].NaN_count = 1
rra[10].cdp_prep[1].intercept = 8.2470833333e+03
rra[10].cdp_prep[1].slope = 0.0000000000e+00
rra[10].cdp_prep[1].NaN_count = 1
rra[11].cf = "SEASONAL"
rra[11].rows = 2016
rra[11].pdp_per_row = 1
rra[11].gamma = 1.0000000000e-01
rra[11].cdp_prep[0].seasonal = NaN
rra[11].cdp_prep[1].seasonal = NaN
rra[12].cf = "DEVSEASONAL"
rra[12].rows = 2016
rra[12].pdp_per_row = 1
rra[12].gamma = 1.0000000000e-01
rra[12].cdp_prep[0].deviation = NaN
rra[12].cdp_prep[1].deviation = NaN
rra[13].cf = "DEVPREDICT"
rra[13].rows = 4032
rra[13].pdp_per_row = 1
rra[14].cf = "FAILURES"
rra[14].rows = 4032
rra[14].pdp_per_row = 1
rra[14].delta_pos = 2.0000000000e+00
rra[14].delta_neg = 2.0000000000e+00
rra[14].failure_threshold = 6
rra[14].window_length = 9
rra[14].cdp_prep[0].history = "000000000"
rra[14].cdp_prep[1].history = "000000000"

[TheWitness]

Just so that we can track the patches, can you guys post them at the bugs site? Now that this is out there, we have to be sensitive when moving forward with Cacti.

TheWitness

[psyber]

Will any of the Holt-Winters features make it into 8.8?

[gandalf]

Will any of the Holt-Winters features make it into 8.8?

If you provide some code, why not? It is not _planned_ for 088. That does not mean much, at least it means you can't be sure.
Reinhard

[timi]

so is it possible to use holt-walters function with cacti ?

[psyber]

It's possible but certainly not for the faint of heart.
You really need to know what you are doing with rrdtool and you need to have a sound grasp of the algorithm and what its limitations are. If your intent on using Cacti you'll have to modify a few things to be able to create the graph templates and CDEFS and such. Some of the features I wanted to make use of weren't in the current stable version of rrdtool so I had to go bleeding edge with an SVN snapshot. So if your not the type to RTFM, RTFC, and get your hands really dirty I can't recommend this path.

Currently I'm doing HW as an externally created and updated RRD and using cacti just for the graphing and navigation, which as most will tell you is an enormous pain (I have ~1000 data sources set up this way).

Its still very much a slow moving learning experience for me. Some of my "experiments" take a year to produce enough data just to get the initial prediction values I really want and if I've miscalculated any of my values will require a good bit of tweaking with rrdtune.

For anyone really interested in doing the HW stuff, try out my calculator http://forums.cacti.net/about14810.html

Info on the PREDICT patch :
http://n2.nabble.com/Patch-Prediction-C ... 37164.html

[jez]

Apart from people hacking this in to their own Cacti installations, has anyone made any progress on creating patches to Cacti to make a framework for this? Perhaps a plug-in? Perhaps patches to the core to enable a plug-in to do this? I'd really like to get this going, even if it means doing it myself. I've done HW in the past with MRTG, so I kind-of know what I'm doing, at least some of the time, and I've done a bit of PHP coding in the past. But I'm fairly new to Cacti so I don't really know the best way to splice this all in.

It seems to me that there are two goals people are trying to achieve with this:
1) aberrant behaviour detection (detecting when the current values are away from the predicted value by more than a predicted deviation)
2) prediction (showing what a the graph is likely to look like in the future)

Predicting the future seems more difficult in that you would end up with timestamps in the future, and I really don't know how RRD would handle this. Regardless, I'm only hoping for aberrant behaviour detection, with those pretty yellow tick marks. The end result for me is an easy way to scan through all my graphs and see which ones need closer scrutiny.

Assuming I have the time and smarts to splice this into Cacti, can anyone suggest the ideal way to get the graphing done? I'm imagining a checkbox on the graph template to turn on Holt-Winters aberrant behaviour detection. Or perhaps it should be turned on for any RRD file that has the appropriate CFs defined (but I dunno how to look at an RRD file within rrd.php, can anyone help?).

If enabled, the code that runs rrdgraph to display a graph (in lib/rrd.php) would automatically add the necessary DEFs, CDEFs and LINEs to display the bounds. The various settings would be tunable by some configuration option(s) where you set a template string, and this would allow selection of colours and line type, perhaps to show an area confidence interval rather than two lines. Initially, defaults would be fine.

Setting appropriate HW parameters within the RRD file would initially be done using the hw_recreate script. Later, it could be defined somewhere within Cacti.

Things get tricky when there are two or more sets of values on a graph, or when some of them are calculated. For the moment I'm going to assume that HW will be applied only to one data set on a graph.

[gandalf]

I was trying to add this to 088, the next release with substantial new features.
But it turns out that the data model is quite complicated (many parameter interdependencies).
Please feel free to suggest changes based on SVN/main for 088 or, better, post code contributions.
I will happily integrate them into core in case I'm ok with them.
R.

[psyber]

My needs lean more towards prediction (esp as it relates to storage and power growth) I think adding that functionality to the current 8.7 is achievable with PREDICT and LSLINT/LSLSLOPE but a pile of work none the less and I wasn't willing to graft the VDEF support into 8.7 without a substantial block of free time, also I saw somewhere VDEF support slated for 8.8 and I'm not one to reinvent the wheel esp when we have such talented wheel designers in cacti already. I could only hope to create a Fred Flintstone wheel where your creating Firestone high performance racing tires. I haven't looked at the 8.8 SVN yet (its been on my TODO for several months) time to spin up a vm and check it out I guess. Is it functional at this point?

@jez,
I feel your pain.
The prediction (PREDICT and LSLINT/LSLSLOPE) stuff IMHO is the low hanging fruit here the algorithms are simple operators on existing DSs and as such do not require tweaks at the RRA level to work (so they will work with any existing file out of the box provided your version of RRDtool supports them. the ABD stuff requires the RRD to be rewritten with the extra DS's and to be set with appropriate values for alpha and beta and then later (in most cases) tweaked. Also you need to know a lot about the data and how it behaves and the expected cycles (seasons) you have. In many cases you don't know this at creation time so there would have to be a mechanism to tune the RRD at some point after initial creation to get meaningful results.

I don't seem to have much time on my hands as of late we are down to a half dozen people (about half to a third of what we had a year or 2 ago) but what "free" time I do have I'd be willing to lend to helping to work out the core framework for getting this into 8.8. I think if we can get cacti to a point that it can be extended in these areas we can possibly leverage the community more for bringing the actual functionality into play. Plus that sets the stage for easily adding other features as RRDtool evolves.

I probably won't be much help on the PHP side of things (I'm a butcher at best) but I can help out with the DB, logic flow and RRDtool side of things. If you have any specific tasks you can throw at me please please please do so. If left to my own devices I will get side tracked on who knows what.

[gandalf]

My needs lean more towards prediction (esp as it relates to storage and power growth) I think adding that functionality to the current 8.7 is achievable with PREDICT and LSLINT/LSLSLOPE but a pile of work none the less and I wasn't willing to graft the VDEF support into 8.7 without a substantial block of free time, also I saw somewhere VDEF support slated for 8.8 and I'm not one to reinvent the wheel esp when we have such talented wheel designers in cacti already. I could only hope to create a Fred Flintstone wheel where your creating Firestone high performance racing tires. I haven't looked at the 8.8 SVN yet (its been on my TODO for several months) time to spin up a vm and check it out I guess. Is it functional at this point?

Functional? It was, at some point in time. Due to 087g, I did not visit it recently. Full VDEF support is in and MOST rrdtool features as well. In fact, Holt-Winters is the most substantial one that's still missing. I was heading at full support but dropped it due to the complexity of parameters (and the option to dynamically change them if prediction results are bad).
If you have some experience using Holt-Winters, perhaps we should team up. E.g. if there are certain use-cases that don't require the full magic, I'll give it a go.

@jez,
I feel your pain.
The prediction (PREDICT and LSLINT/LSLSLOPE) stuff IMHO is the low hanging fruit here the algorithms are simple operators on existing DSs and as such do not require tweaks at the RRA level to work (so they will work with any existing file out of the box provided your version of RRDtool supports them.

Would this be a good start?

the ABD stuff requires the RRD to be rewritten with the extra DS's and to be set with appropriate values for alpha and beta and then later (in most cases) tweaked. Also you need to know a lot about the data and how it behaves and the expected cycles (seasons) you have. In many cases you don't know this at creation time so there would have to be a mechanism to tune the RRD at some point after initial creation to get meaningful results.

That was my fear. To be honest, I coded some features to tweak existing rrd files (e.g. add a new data source, which is not available with native rrdtool). So it may happen somehow, but perhaps not now.

I don't seem to have much time on my hands as of late we are down to a half dozen people (about half to a third of what we had a year or 2 ago) but what "free" time I do have I'd be willing to lend to helping to work out the core framework for getting this into 8.8. I think if we can get cacti to a point that it can be extended in these areas we can possibly leverage the community more for bringing the actual functionality into play. Plus that sets the stage for easily adding other features as RRDtool evolves.

Deal!

I probably won't be much help on the PHP side of things (I'm a butcher at best) but I can help out with the DB, logic flow and RRDtool side of things. If you have any specific tasks you can throw at me please please please do so. If left to my own devices I will get side tracked on who knows what.

Examples of "low hanging fruits" and logic flow would be fine. And, at the end, some skilled documentation of a guy "who knows".
R.

[psyber]

Agreed, I have a presentation middle of next week I need to prep for. As soon as I get that off the hot list I'll check out 8.8 and see whats cooking. I know PREDICT works with very little tweaking.

If there's VDEF in 8.7g I need to update (still running 8.7d) I'm a little reluctant though, I can't remember what all I did to get the functionality I have and I know there are some management types that would be unhappy if I broke what I have now.