[HOWTO] 0.8.7 and 1 minute polling

[moggie]

Hi tekbot,
First of all, thanks for your great post!, it helped me allot to configure my 1min interval polling.
I still have one question regarding Rows calculation.

I understand the math behind Rows calculation, but I did not undersand why you choose longer periods than the period represented by Timespan value.

For example: In your 1min poll / 24 hours view, why did you choose Rows value of 4320 which is equal to 3 days if due to the Timespan value only 24hours will be displayed in the graph? my common sense would be to choose Rows value of 1440 in this case. what am I missing?

thanks
Yuval

Likewise, I believe I understand everything except this last bit... I'm not sure of the reasoning behind keeping extra rows beyond the timespan values Cacti will use when rendering graphs (in most cases double the number of rows needed to represent the specified timespan).

For example, with step adjusted for 1 minute polling, the default RRAs might look something like this:

Code: Select all

Hourly (1 Minute Average)         1       500      14400
Daily (5 Minute Average)          6       600      86400
Weekly (30 Minute Average)        30      700      604800
Monthly (2 Hour Average)          120     775      2678400
Yearly (1 Day Average)            1440    797      33053184 

Taking the Daily (5 Minute Average) as an example. The timespan specifies that when this graph is displayed, it will show 1 day's worth of data; so in theory, the number of rows needed to represent this would be 288 (the number of 5 minute intervals in one day) - but the RRA is storing 600 rows of data (giving it a history of over two days).

My question is, what is the reasoning behind Cacti storing twice the data needed (and tekbot 3 times or more) when the timespan setting means this superfluous data wont be used to render the standard graphs?

Could it be that this extra data is used when custom graph view settings are used. For example, if I ask Cacti to show me graphs spanning the past two days instead of one, will it use this extra data being stored when generating this custom graph for me?


If so, it got me thinking about this a bit further... Again using the RRAs above as an example; what if I asked Cacti to show me custom graphs spanning the past 4 days (exceeding the available information contained in the Hourly or Daily RRAs).

Does Cacti use the more accurate Hourly RRA data to render the first 8 hours or so, the Daily RRA data to render up until the past 2 days, and then fill in the rest using the less accurate Weekly (30 Minute Average) RRA? Or does it instead use the RRA data from the Weekly RRA because that's the only one that contains enough data spanning the requested timespan?

Thank you in advanced for your time and consideration.

Edit: I think this from the Cacti manual kind of answers my question: "The data retrieval function of RRDtool automatically selects the archive with the highest resolution that still covers the requested timespan. This mechanism is also used by RRDtool's graphing subsystem."

[TheWitness]

In order to do 1 minute polling, your Data Template's that are going to be polled at one minute, need to have their STEP values changed to 60 from 300. However, you must also understand the relationship between the step and the RRA settings so that you don't get graphs that end after a few months.

Cacti is installed to be run at 5 minute polling. Lot's of customers poll cacti at 1 minutes or less for critical hardware monitoring. But keep in mind, this is an advanced subject. I wish it were easier, but for the experienced RRDtool user, it's not so bad.

Finally, after your Data Templates have been altered, you will have to delete your impacted RRDfiles, and repopulate the poller cache.

TheWitness

[lupetalo]

Is there any chance for someone to actually post step by step transform from fresh installed cacti at 5 min to 1 min, including all needed things. It is really hard to read 7 pages of discussions. Or insert in first post.

[TheWitness]

There are a number both in the How-To section of the forum, and in the documentation. Please use Google.

[nonothekid]

Is there any chance for someone to actually post step by step transform from fresh installed cacti at 5 min to 1 min, including all needed things. It is really hard to read 7 pages of discussions. Or insert in first post.

hi lupetalo, i hope this link would help you..

http://dikehtonon.blogspot.com/2011/11/ ... spine.html

compiling all information, I've search from cacti.net and other blogs, I'm now able to understand how to make it work.

[daveynihil]

Fantastic guide by Tekbot. Followed this on a fresh 8.8a install and have 1 minute polling working perfectly on bits per second graphs on cisco devices.

There's just one small question now.

Data Templates = When I got the box, the graphs for the actual Cacti Server had already been setup. (mysql etc)

After changing poller (not cron) from 5 mins to 1 min, I expected these graphs to start acting odd. They have not. Do I need to change the step and heartbeat on ALL templates that are active/i want to use in the future? Should I just then delete and re-create these graphs?

Also, if My poller interval is now 1 minute, I assume all hosts will poll on all graphs setup every 1 minute, and I can't mix and match 1 minute & 5 minute polling per device? Graphs obviously I could make a 5 minute average RRA template, but the actual SNMP query would still hit the box every minute? We'd still store the data points every minute also?

[classen]

My question is, what is the reasoning behind Cacti storing twice the data needed (and tekbot 3 times or more) when the timespan setting means this superfluous data wont be used to render the standard graphs?

My understanding is that this allows you to display the higher resolution graphs further back in time.

So if you're looking at the last 1 hour you can "shift left" several hours and still get the highest resolution information.

[classen]

Here's my actual RRA settings. I wanted 1 minute averages on all and I added a 3 year graph.

Code: Select all

Name                                Steps          Rows          Timespan
Hourly  (1 Minute Average)           1              300           14400
Daily   (1 Minute Average)           1              1500          86400
Weekly  (1 Minute Average)           1              11000         604800
Monthly (1 Minute Average)           1              45000         2678400
Yearly  (1 Minute Average)           1              526000        31536000
3 Years (1 Minute Average)           1              1576800       94608000

I hope this helps someone. Thanks.

This seems wrong.

My understanding is that if you want more 1 minute data stored then you need to add more rows to your Hourly (1 min avg) RRA.

Your current Hourly (1 minute average) RRA would hold 5 hours of 1 minute data, and your 3 Year RRA will hold 3 years of 1 minute data. But your 3 year graph will only be 500 pixels wide so you have way more information than you will be able to see. If you really want 3 years of 1 minute data then I propose something along these lines:

Code: Select all

Name                                Steps          Rows          Timespan
Hourly  (1 Minute Average)           1            1576800           14400
Daily   (5 Minute Average)           5               4320           86400
Weekly  (20 Minute Average)          20              4320          604800
Monthly (60 Minute Average)          60              4320         2678400
Yearly  (1 day Average)              1440            1095        31536000

This would result in RRDs that hold:
3 years of 1 minute data
15 days of 5 minute averaged data
60 days of 20 minute averaged data
180 days of 60 minute averaged data
1095 days (3 years) of 1 day (1440 minute) averaged data


or if you want 3 years of data for each consolidation level then something like this:

Code: Select all

Name                                Steps          Rows              Timespan
Hourly  (1 Minute Average)           1            1576800(3y)           14400
Daily   (5 Minute Average)           5             315360(3y)           86400
Weekly  (20 Minute Average)          20             78840(3y)          604800
Monthly (60 Minute Average)          60             26280(3y)         2678400
Yearly  (1440 Minute Average)        1440            1095(3y)        31536000

However, this will require ~122Mb for a simple Interface RRD such as made from teh command below.

Code: Select all

rrdtool create \
test.rrd \
--step 60  \
DS:traffic_in:COUNTER:120:0:1000000000 \
DS:traffic_out:COUNTER:120:0:1000000000 \
RRA:AVERAGE:0.5:1:1576800 \
RRA:AVERAGE:0.5:5:315360 \
RRA:AVERAGE:0.5:20:78840 \
RRA:AVERAGE:0.5:60:26280 \
RRA:AVERAGE:0.5:1440:1095 \
RRA:MIN:0.5:1:1576800 \
RRA:MIN:0.5:5:315360 \
RRA:MIN:0.5:20:78840 \
RRA:MIN:0.5:60:26280 \
RRA:MIN:0.5:1440:1095 \
RRA:MAX:0.5:1:1576800 \
RRA:MAX:0.5:5:315360 \
RRA:MAX:0.5:20:78840 \
RRA:MAX:0.5:60:26280 \
RRA:MAX:0.5:1440:1095 \
RRA:LAST:0.5:1:1576800 \
RRA:LAST:0.5:5:315360 \
RRA:LAST:0.5:20:78840 \
RRA:LAST:0.5:60:26280 \
RRA:LAST:0.5:1440:1095

[Voiper99]

There's just one small question now.

Data Templates = When I got the box, the graphs for the actual Cacti Server had already been setup. (mysql etc)

After changing poller (not cron) from 5 mins to 1 min, I expected these graphs to start acting odd. They have not. Do I need to change the step and heartbeat on ALL templates that are active/i want to use in the future? Should I just then delete and re-create these graphs?

Also, if My poller interval is now 1 minute, I assume all hosts will poll on all graphs setup every 1 minute, and I can't mix and match 1 minute & 5 minute polling per device? Graphs obviously I could make a 5 minute average RRA template, but the actual SNMP query would still hit the box every minute? We'd still store the data points every minute also?

In regards to the part I "bolded" above, I assumed this too. However, my graphs remained in tact just as yours did. After studying the whole process for a while I realised that it all comes down to the Data Template's Step value. It determines the time at which the hosts will be polled, regardless of whether you set your poller time to 1 minute or 5 minutes.

Because all of your original Data Templates have a Step value of 300 and heartbeat value of 600, the corresponding hosts will be polled every 5 minutes even after the poller has been changed to run every one minute.

The same is true for your newly created one minute Data Templates. As your Step value is 60 and your heartbeat value is 120, those hosts will be updated every minute.

In other words, so long as you've got the correct Data Template Step and Heartbeat values configured, your One Minute and Five Minute graphs will work perfectly well in harmony.

My question is, what is the reasoning behind Cacti storing twice the data needed (and tekbot 3 times or more) when the timespan setting means this superfluous data wont be used to render the standard graphs?

My understanding is that this allows you to display the higher resolution graphs further back in time.

So if you're looking at the last 1 hour you can "shift left" several hours and still get the highest resolution information.

This is correct.

[Voiper99]

Here's my actual RRA settings. I wanted 1 minute averages on all and I added a 3 year graph.

Code: Select all

Name                                Steps          Rows          Timespan
Hourly  (1 Minute Average)           1              300           14400
Daily   (1 Minute Average)           1              1500          86400
Weekly  (1 Minute Average)           1              11000         604800
Monthly (1 Minute Average)           1              45000         2678400
Yearly  (1 Minute Average)           1              526000        31536000
3 Years (1 Minute Average)           1              1576800       94608000

I hope this helps someone. Thanks.

This seems wrong.

My understanding is that if you want more 1 minute data stored then you need to add more rows to your Hourly (1 min avg) RRA.

The part I have bolded above is correct.

Your current Hourly (1 minute average) RRA would hold 5 hours of 1 minute data

This is also correct.

and your 3 Year RRA will hold 3 years of 1 minute data.

Correct.

But your 3 year graph will only be 500 pixels wide so you have way more information than you will be able to see.

This is not quite true. Yes, the graph will display three years worth of data in the a 500 pixel picture, however, if he uses Cacti's zoom option he will be able to view the 1 minute data without any averaging/consolidation. Therefore it will be as crystal clear as the day the data was captured three years prior.

If you really want 3 years of 1 minute data then I propose something along these lines:

Code: Select all

Name                                Steps          Rows          Timespan
Hourly  (1 Minute Average)           1            1576800           14400
Daily   (5 Minute Average)           5               4320           86400
Weekly  (20 Minute Average)          20              4320          604800
Monthly (60 Minute Average)          60              4320         2678400
Yearly  (1 day Average)              1440            1095        31536000

This would result in RRDs that hold:
3 years of 1 minute data
15 days of 5 minute averaged data
60 days of 20 minute averaged data
180 days of 60 minute averaged data
1095 days (3 years) of 1 day (1440 minute) averaged data


or if you want 3 years of data for each consolidation level then something like this:

Code: Select all

Name                                Steps          Rows              Timespan
Hourly  (1 Minute Average)           1            1576800(3y)           14400
Daily   (5 Minute Average)           5             315360(3y)           86400
Weekly  (20 Minute Average)          20             78840(3y)          604800
Monthly (60 Minute Average)          60             26280(3y)         2678400
Yearly  (1440 Minute Average)        1440            1095(3y)        31536000

Both of these options would work, but would not provide the granular level of data that is provided with 1 Step, as was originally planned.

However, this will require ~122Mb for a simple Interface RRD such as made from teh command below.

Code: Select all

rrdtool create \
test.rrd \
--step 60  \
DS:traffic_in:COUNTER:120:0:1000000000 \
DS:traffic_out:COUNTER:120:0:1000000000 \
RRA:AVERAGE:0.5:1:1576800 \
RRA:AVERAGE:0.5:5:315360 \
RRA:AVERAGE:0.5:20:78840 \
RRA:AVERAGE:0.5:60:26280 \
RRA:AVERAGE:0.5:1440:1095 \
RRA:MIN:0.5:1:1576800 \
RRA:MIN:0.5:5:315360 \
RRA:MIN:0.5:20:78840 \
RRA:MIN:0.5:60:26280 \
RRA:MIN:0.5:1440:1095 \
RRA:MAX:0.5:1:1576800 \
RRA:MAX:0.5:5:315360 \
RRA:MAX:0.5:20:78840 \
RRA:MAX:0.5:60:26280 \
RRA:MAX:0.5:1440:1095 \
RRA:LAST:0.5:1:1576800 \
RRA:LAST:0.5:5:315360 \
RRA:LAST:0.5:20:78840 \
RRA:LAST:0.5:60:26280 \
RRA:LAST:0.5:1440:1095

This is really interesting. I'd understanding how different STEP and ROW values affect the size of an RRD file. Can you please explain to me how the above works?

[Voiper99]

By the way, I have started a new 1 minute polling thread over here. I have written a tutorial in the thread and have also created a calculator which can be used to take all of the manual work out of calculating Steps, Heatbeats, Rows, Timespans, etc.

[gandalf]

By the way, I have started a new 1 minute polling thread over here. I have written a tutorial in the thread and have also created a calculator which can be used to take all of the manual work out of calculating Steps, Heatbeats, Rows, Timespans, etc.

Great job! Thx
R.