[HOWTO] 0.8.8 and 1 minute polling + File Size Calculator!

[Voiper99]

Three hours in to 2013 and here I am on the Cacti forums

This thread was inspired by the [HOWTO] 0.8.7 and 1 minute polling page, and in particular, tekbot's fantastic post.

Before I begin, I must repeat tekbot's suggestion:

Also, send the developers some love (and / or money). As mentioned in the 0.8.7 release notes, they all have Day Jobs and contribute to Cacti because they WANT TO.

Cacti is an excellent, free product, so I feel the least we can do as users is donate to those who have dedicated so much of their free time in to giving us this software.

Now, let's begin!

Why the new thread?

There are two reasons for this new thread. The first reason is because I am going to create an updated version of gandalf's excellent tutorial and thought it would be better suited in a new thread rather than tacked on to an existing thread.

The second reason is because I have created a calculator that takes all the hassle out of creating new polling intervals (e.g one minute polling). Rather than tack it on to an existing thread, putting it in a new thread will allow me to track and fix issues, feature requests, etc.



What exactly does the calculator do?

Whether you have configured One Minute Polling before, or you are looking to implement it, you will most probably be familiar with terms such as Data Source Steps, Heartbeats, RRA Steps, Rows and Timespans. Calculating these can be quite difficult (especially for those who are just starting out), and, if you have no idea what they all mean, it can make enabling One Minute Polling very challenging - however, this is where the Calculator comes in. By answering four simple questions the calculator will tell you which values you need to enter in to each of the above mentioned fields and will also estimate the amount of space you will need per Data Source.

Note: Although this calculator was built to simplify the One Minute Polling process, I do urge all readers to understand the process fully. Doing so will ensure that you get the most out of this feature.

Two Part Calculator

The calculator is broken in to two parts. The "Forward" calculator and the "Reverse" calculator. Everything written in the previous section applies to the "Forward" calculator. The "Reverse" calculator allows you to turn "Steps", "Rows" and "Timespans" in to easily understandable information, without having to perform any manual calculations.

Reverse Calculator Examples

When you first install Cacti, all Data Templates have a Step value of 300 and the below RRAs are preconfigured:

Code: Select all

Name                       Steps    Rows       Timespan
Hourly (1 Minute Average)    1 	   500 	    14400 
Daily (5 Minute Average)     1 	   600 	    86400
Monthly (2 Hour Average)    24 	   775      2678400
Weekly (30 Minute Average)   6 	   700       604800
Yearly (1 Day Average)     288 	   797     33053184

Using the Calculator, simply plug in the above details and it will translate the information to an easily understandable format.

Let's dissect a few of the above mentioned RRAs:

Code: Select all

Name                       Steps    Rows       Timespan
Daily (5 Minute Average)     1 	   600 	    86400

The calculator tells us that the Daily (5 Minute Average) does the following:

• * It keeps data for 2 days before overwriting it.

• * It averages data over a 5 minute period.

• * It displays a graph that spans 1 day.

OK, time for the next one:

Code: Select all

Name                       Steps    Rows       Timespan
Monthly (2 Hour Average)    24 	   775      2678400

The calculator tells us that the Monthly (2 Hour Average) does the following:

• * It keeps data for just over 2 months before overwriting it.
  * It averages data over a 2 hour period.
  * It displays a graph that spans 1 month.

And finally:

Code: Select all

Name                       Steps    Rows       Timespan
Yearly (1 Day Average)     288 	   797     33053184

The calculator tells us that the Yearly (1 Day Average) does the following:

• * It keeps data for just over 2years before overwriting it.
  * It averages data over a 1 day period.
  * It displays a graph that spans just over a 1 year.

Forward Calculator Examples

Now that we've seen how the Reverse Calculator works, it should make understanding the Forward Calculator easier.

Below is a list of the One Minute RRAs I wanted to create for myself:

• * Daily (1 Minute Average) - Display a 1 day graph that polls hosts every minute and does not consolidate the data. The data is to be overwritten after 1 week.
  * Weekly (5 Minute Average) - Display a 1 week graph that polls hosts every minute and consolidates the data every 5 minutes. The data is to be overwritten after one month.
  * Monthly (30 Minute Average) - Display a 1 month graph that polls hosts every minute and consolidates the data every 30 minutes. The data is to be overwritten after three months.

So let's get started.

The below example shows how the Daily (1 Minute Average) RRA was configured:



The below example shows how the Weekly (5 Minute Average) RRA was configured:



The below example shows how the Monthly (30 Minute Average) RRA was configured:



Why the long data retention?

A common question I have seen is, "why would you have a longer data retention period than the timespan of the graph?". For example, as mentioned above, one of the RRAs that I wanted was:

• * Daily (1 Minute Average) - Display a 1 day graph that polls hosts every minute and does not consolidate the data. The data is to be overwritten after 1 week.

It may sound silly to store 7 days worth of data and only display 1 day's worth of graph. However, there is method behind this madness.

Using Cacti's "Graph Filters", I can look at any time frame in the past week and view the graphs at a one minute polled interval. If I had not done this, Cacti would have no choice but to display the graphs in my "next best" RRA configuration, which was the Weekly (5 Minute Average) graph. This is not ideal because the data is averaged over 5 minutes, therefore it will effectively have five times less detail than my Daily (1 Minute Average) graphs.

An alternative to Cacti's "Graph Filters" option is its "Zoom Graph" option. Cacti automatically displays graphs using the most detailed graph possible. For example, if a time period is covered by both the Daily (1 Minute Average) and Weekly (5 Minute Average) RRAs, it will present the graph using the Daily (1 Minute Average) RRA.

As per the graphs below, I can zoom in to my "Yearly (1 Day Average)" to the point where the data will be displayed using the Daily (1 Minute Average) RRA.

Fig 1: Zooming in to the "Yearly (1 Day Average)" graph:



Fig 2: After the first zoom, the graph is drawn using the Weekly (5 Minute Average) graph:



Note: The reason why the graph drawing skipped the Monthly (2 Hour Average) RRA is because it had the specified time frame covered by the Weekly (5 Minute Average) RRA, and, as mentioned above, Cacti will always use the RRA which has the most detail in it. If I were to choose a date that was over 1 month ago, the data would not be available in the Weekly (5 Minute Average) RRA (because it only stores data for one month before it is overwritten) and therefore Cacti would use the Monthly (2 Hour Average) RRA instead.

Fig 3: Finally, after zooming in a third time, the graph is drawn using the Daily (1 Minute Average) RRA. You can confirm this is true by counting the amount of times the graphs data changes between two time frames. For example, you can see that the "Inbound" and "Outbound" lines change ten times (once per minute) between 10:50am and 11:am:



HOWTO: One Minute Polling

(Work In Progress).

That's All Folks!

I will update this thread again in the near future to include the above mentioned HOWTO: One Minute Polling section.

Thank you for reading my lengthy post, and I wish you all a Happy New Year!

Please let me know if you find any bugs or have any feature requests.

[hurgh]

Hi,

Any update to this?
I would like to see the calculator and the How-To.

Regards

-Hurgh-

[Voiper99]

Hi,

Any update to this?
I would like to see the calculator and the How-To.

Regards

-Hurgh-

Sorry about the delay. I have finally completed and uploaded it. The download link can be found in the bottom of my first post of this thread.

[Tolaris]

Excellent work, Voiper99, and hugely useful. Thank you.

[Voiper99]

Not a problem at all, I'm glad I could help

[Voiper99]

Thanks to phalek, I have now been able to attach the spreadsheet to my post above I have for that reason removed the external download link.

[Tolaris]

Most RRD files have multiple RRAs, and multiple time spans. For instance, my "Interface Traffic" RRDs have:

Hourly (1 Minute Average) 1 500 14400
Daily (5 Minute Average) 1 600 86400
Weekly (30 Minute Average) 6 700 604800
Monthly (2 Hour Average) 24 775 2678400
Yearly (1 Day Average) 288 797 33053184

Each of those RRAs has four consolidation functions: AVERAGE, MIN, MAX, and LAST.

That's 5 RRAs, 4 consolidations, and two data sources (bits in and out). However, your file size calculator assumes only one RRA. By hand, I can do 5 calculations, add them up, and remove the 4 extra headers. However, it shouldn't be too hard to add this to your calculator. Are you interested in doing it?

[Voiper99]

Thanks for the feedback. I'd be more than happy to update the tool but I'm not too sure if I understand what you're saying.

Most RRD files have multiple RRAs, and multiple time spans. For instance, my "Interface Traffic" RRDs have:

Hourly (1 Minute Average) 1 500 14400
Daily (5 Minute Average) 1 600 86400
Weekly (30 Minute Average) 6 700 604800
Monthly (2 Hour Average) 24 775 2678400
Yearly (1 Day Average) 288 797 33053184

Agreed, though each one of the above RRAs will result in a different file size due to their polling intervals and data retention times, so you would need to input their details separately to find out how big each individual RRA file is going to be.

Each of those RRAs has four consolidation functions: AVERAGE, MIN, MAX, and LAST.

That's 5 RRAs, 4 consolidations, and two data sources (bits in and out).

My tool allows you to specify how many consolidations and data sources you're using. However, as mentioned above (and please correct me if I'm wrong), as each RRA's file size is going to be different depending on how much detail you want stored and how long you want the data stored for, the tool requires you to input the details of each RRA separately. For example, if you enter "Yearly" RRA's data in to the tool, that information cannot be used to determine the "Hourly" RRA's file size.

[Tolaris]

Agreed, though each one of the above RRAs will result in a different file size due to their polling intervals and data retention times, so you would need to input their details separately to find out how big each individual RRA file is going to be.

Thanks for the help. You are correct that each RRA would differ. The problem is nearly all RRD files in Cacti have multiple RRAs in them. One for each span of time (Hourly, Daily, ...), for instance.

Could I calculate each RRA and add them up to calculate the entire RRD file total? IE- which of the following is true?

1. An RRD file has no header, just a series of concatenated RRAs with headers and data.
2. The RRD file has just one header, and a series of RRAs with only data.
3. The RRD file has a header itself, and a series of concatenated RRAs each with headers and data.

Is that more clear?

[Feren]

Hello, the screenshots of using the Forward Calculator have gone missing so I'm not entirely sure I'm tracking on the right path as I use it. Could somebody share them (if they still have them)? Or repost them to this thread from their own calculation? Voiper99, perhaps?

[Voiper99]

Hi Feren, sorry for the late reply. Are the images still missing? They load fine for me.

[phalek]

Hi

There's a small issue on the Reverse tab, which calculates the "years of data to keep" wrong.

The C10 column on the Reverse tab needs to be change from H8 to H9. Otherwise it will display the same number as for the "months" part.