Change default RRA rows

[eschoeller]

I have an installation with mixed 5 minute / 1 minute polling but with
all the original default RRA settings. My 5 minute and 1 minute graphs
are working fine, but I'd like to keep my high resolution data longer,
and also add a 5 year RRA.

I found this doc:

http://docs.cacti.net/node/393

But I'm still a bit confused by what steps I should actually take to
accomplish this task.

Here is what I *think* I'm supposed to do:

1. Stop my poller
2. backup the DB and rrd files
3. change the rows on my RRA definitions to reflect the retention I want
4. Add a 5 year RRA
5. use resize.pl in some way to "grow" my existing RRD files
6. move the new RRDs into place
7. Add the 5 year RRA to any data sources I want to watch for 5 years
8. restart the poller
9. cross my fingers

Running:
Cacti 0.8.7b Spine Plugin Arch 2 Aggregate 0.64

[rony]

That is pretty much what you need to do.

I would suggest, if you have the resources, you try this on a test system first.

[gandalf]

I have an installation with mixed 5 minute / 1 minute polling but with
all the original default RRA settings. My 5 minute and 1 minute graphs
are working fine, but I'd like to keep my high resolution data longer,
and also add a 5 year RRA.

I found this doc:

http://docs.cacti.net/node/393

But I'm still a bit confused by what steps I should actually take to
accomplish this task.

Here is what I *think* I'm supposed to do:

1. Stop my poller
2. backup the DB and rrd files
3. change the rows on my RRA definitions to reflect the retention I want
4. Add a 5 year RRA
5. use resize.pl in some way to "grow" my existing RRD files
6. move the new RRDs into place
7. Add the 5 year RRA to any data sources I want to watch for 5 years
8. restart the poller
9. cross my fingers

Running:
Cacti 0.8.7b Spine Plugin Arch 2 Aggregate 0.64

That's a quite complete description of the tasks. There's only one oddity: Step 7 is NOT doable with rrdtool. You can't add a new RRA to existing rrd files. It is possible to resize existing rra's, but no way to add a new one.
I'm not sure, if there's a contribution to rrdtool to add such a functionality
Reinhard

[eschoeller]

Right. So If I create the new RRA I can only use it for new data sources and RRDs, correct?

Could I increase the rows of the 1 year RRA to essentially turn it into a 5 year RRA and rename it? Then, I guess I wouldn't have a 1 year RRA only a 5 year ...

[gandalf]

Right. So If I create the new RRA I can only use it for new data sources and RRDs, correct?

Yes

Could I increase the rows of the 1 year RRA to essentially turn it into a 5 year RRA and rename it? Then, I guess I wouldn't have a 1 year RRA only a 5 year ...

Yes, but an rra has no name. The data source has a name and you may change it using rrdtool tune
Reinhard

[eschoeller]

Can the number of steps for a given RRA be changed, or will that affect the existing RRD files?

[gandalf]

Sure, See my description and script to do so at the first link of my sig; rrdtool chapter
Reinhard

[eschoeller]

Hmm, I read through that document again (http://docs.cacti.net/node/393) and I only see how to resize RRA rows, not the steps. Could you elaborate?

I tried resizing the rows of the Default RRAs, and it seemed to work, partially. Now I seem to have gaps in my graphs, but I think the data is still there. At certain zoom levels the gaps appear, and after zooming further they disappear. I am suspecting the "timespan" could be the culprit?

In my test environment I made the following changes:

Code: Select all

RRA                           Steps  Rows    Timespan
Hourly (1 Minute Average)   	 1    2500    14400 
Daily (5 Minute Average)    	 1 	2600 	86400
Weekly (30 Minute Average) 	  6 	2700 	604800
Monthly (2 Hour Average) 	   24 	2775 	2678400
Yearly (1 Day Average) 	     288   2797 	33053184 

So, as a test, I simply increased the row size of all the RRA's by 2000. And ran this:

Code: Select all

perl /usr/local/sbin/resize.pl -g 2000 -f "/usr/local/cacti-dev/rra/*.rrd" -s 600 -g 2000 -o "/usr/local/cacti-dev/rra/"

perl /usr/local/sbin/resize.pl -g 2000 -f "/usr/local/cacti-dev/rra/*.rrd" -s 500 -g 2000 -o "/usr/local/cacti-dev/rra/"

perl /usr/local/sbin/resize.pl -g 2000 -f "/usr/local/cacti-dev/rra/*.rrd" -s 700 -g 2000 -o "/usr/local/cacti-dev/rra/"

perl /usr/local/sbin/resize.pl -g 2000 -f "/usr/local/cacti-dev/rra/*.rrd" -s 775 -g 2000 -o "/usr/local/cacti-dev/rra/"

perl /usr/local/sbin/resize.pl -g 2000 -f "/usr/local/cacti-dev/rra/*.rrd" -s 797 -g 2000 -o "/usr/local/cacti-dev/rra/"

Where did I go wrong?

[gandalf]

Code: Select all

RRA                           Steps  Rows    Timespan
Hourly (1 Minute Average)   	 1    2500    14400 
Daily (5 Minute Average)    	 1 	2600 	86400

Bad. Do NOT use two different RRAs with same step size. You may define both RRAs, but for a selected data source make sure to use only ONE of both.
Reinhard

[eschoeller]

Hmmm... So in cacti 0.8.7b the "Hourly (1 Minute Average)" RRA was added. Are you saying that for all data sources using step 60 heartbeat 120 (1 minute polling) I should have selected the Hourly RRA and DESELECTED the Daily RRA?

I didn't do that. I have tons of data sources using both! What's going to happen?

[gandalf]

We posted an "advice" for this issue months ago.
To be honest, I did not try to create such a broken rrd file. Perhaps it will work; but no promise made. For me, results are undetermined.
It is required to closely match RRA definitions and step sizes and polling intervals and stuff. Simply selecting all RRAs for the data source definition is not supposed to work.
That's why I do not support this stuff, personally. I suppose I would know how to do so, but it requires great care and a good rrdtool understanding. There has been an approriate thread describing all/most of the issues to take care of.
Reinhard

[fuzzyducky]

Hey guys,

Sorry to reopen such an old thread, but the posts by eschoeller essentially echo what I'm seeing in my own cacti installation.

We're on cacti 0.8.7g as shipped with Debian stable, and back in the day I created an Hourly/1 Minute RRA with a step value of 1 so we could get a 1 minute resoloution on our graphs - I also however did not remove the Daily/5 Minute RRA from any of my graphs either.

All has appeared to be running fine, however today I was looking at increacing the historical resoloution of our graphs and found this thread!

This is what I have:

• Hourly (1 Minute Average) 1 500 14400
  Daily (5 Minute Average) 1 600 86400
  Weekly (30 Minute Average) 6 700 604800
  Monthly (2 Hour Average) 24 775 2678400
  Yearly (1 Day Average) 288 797 33053184

.... and I was looking at first resizing the Daily 5 minute Average's steps to 1440 and resizing my rrd files (figuring that'd give me the resoloution I want). However I now read above that, I've done something terribly wrong!

I'm guessing it wouldn't suffice to change the step on the 5 minute rra? And I can't remove it from my existing graphs?

Just wondering on the best course of action bearing in mind I'd like to have more accurate historical graphs - I don't want to butcher it up any more than it appears I already have!

Thanks

[eschoeller]

No apology necessary for digging up an old thread, I do it all the time. RRA stuff can be *really* tricky. I have been battling with it for years.

If you haven't read these posts, it's a good start:
http://forums.cacti.net/viewtopic.php?f=21&t=29360
http://forums.cacti.net/viewtopic.php?f ... &start=135

There is another post out there that really hit home for me, and I used it as the basis for our newest set of RRAs in our production environment. I can't find it now, but I will keep looking for it.

Knowing exactly what is going to happen in your environment given the changes that you have made is very difficult to do. You need an incredibly good understanding of rrdtool to really make these types of changes successfully. This is compounded upon the fact that the changes you make may appear to function correctly on Day 1, but on Day 365 they might not. This is what happened to me back in 2008. I think I tried messing around with resizing things and couldn't make it work. We made the decision to ditch all of our historical data (mostly 2 years, but it was a growing environment) and start over with new RRAs. Everything appeared to be fine, but after 3-4 months we realized that we were losing data off the back end of the RRDs. I had somehow screwed up the RRAs and my retention was only a couple months. So, we had to ditch everything again and start over with new RRAs.

The moral of the story is that sometimes you can fix this stuff live - but you generally need to be an rrdtool ninja, and accept certain risks that your data might not be there in months or years to come. The only way I ever felt comfortable was starting over. Another example, an admin developed a test template in our dev environment about 2 years ago. He defined ALL RRAs for the data templates because he didn't know what he was doing. It wasn't a big deal because it was dev. But there were issues with the template so it never rolled into production. I left it running, because we still needed the data. Well, I looked back a few months ago and after a year and a half, the data just stops. Something in the overlapping RRA definitions simply blew up one day and it stopped accepting new data. This *could* happen to you. We have another instance where we have several data templates suffering from the same fate of having all RRAs defined ... and they are working just fine - but I know they could disappear one day. Simply changing the data templates now would break all the existing data sources that rely upon it (I think). In that case, my plan is to export the affected RRDs to xml and then blow them all away, re-create with the correct RRAs and then try an XML import ... knowing that this might not work and I'd lose all that data.

My one recommendation to *anyone* building Cacti for the first time is to look closely at the RRAs. 90% of folks these days will eventually want more retention than the default settings provide. This is because hardware has dramatically improved over the years (as it does) but the Cacti defaults have not changed. People have massively larger I/O and disk capacity now than they did 12 years ago. It's unfortunate that there isn't a larger DISCLAIMER somewhere really nudging people to look at this before they start Cacti up. Once you start, it's a lot harder to go back.

Here is our current production RRA defintions:

Code: Select all

1 Minute Poll - 1 Minute Average - Hourly View - 7 Day Retention        1       10080   3600    
HiRes 1 Minute Poll - 1 Minute Average - 365 Day Retention              1       525600  86400
5 Minute Poll - 5 Minute Average - Daily View - 60 Day Retention        1       17280   86400   
1 Minute Poll - 5 Minute Average - Daily View - 60 Day Retention        5       17280   86400   
1 Minute Poll - 15 Minute Average - Weekly View - 180 Day Retention     15      17280   604800  
5 Minute Poll - 15 Minute Average - Weekly View - 180 Day Retention     3       17280   604800  
HiRes 1 Minute Poll - 10 Minute Average - 10 Year Retention             10      525600  604800
1 Minute Poll - 30 Minute Average - Monthly View - 1 Year Retention     30      17520   2678400
5 Minute Poll - 30 Minute Average - Monthly View - 1 Year Retention     6       17520   2678400
1 Minute Poll - 2 Hour Average - Yearly View - 7 Year Retention         120     30660   31536000
5 Minute Poll - 2 Hour Average - Yearly View - 7 Year Retention         24      30660   31536000

I also wrote an audit script. It uses the settings of our configuration and parses through all available data templates to make sure they conform to the correct RRAs, step and heartbeat values. Here is an example output of it:

Code: Select all

Data Template 'Unix - Ping Host': Mis-matched RRA Poller Intervals
Data Template 'Unix - Ping Host': Excessive(9/5) RRAs defined for Poller Interval '1 Minute Poll'
Data Template 'LeftHand Volume Usage': Mis-matched RRA Poller Intervals
Data Template 'LeftHand Volume Usage': Excessive(9/5) RRAs defined for Poller Interval '1 Minute Poll'
Data Template 'APC InRow Group Setpoints(groupStptCool)': step / heartbeat ratio check failure (300/120) Should be: 300/600
Data Template 'APC InRow Group Setpoints(groupStptCool)': Incorrect heartbeat value(120) should be 600
Data Template 'APC InRow Unit Fan Run Hours(unitFanRun3)': step / heartbeat ratio check failure (300/120) Should be: 300/600
Data Template 'APC InRow Unit Fan Run Hours(unitFanRun2)': step / heartbeat ratio check failure (300/120) Should be: 300/600
Data Template 'APC InRow Unit Fan Run Hours(unitFanRun3)': Incorrect heartbeat value(120) should be 600
Data Template 'APC InRow Unit Fan Run Hours(unitFanRun2)': Incorrect heartbeat value(120) should be 600

I won't help you much now, other than report all the problems you currently have. Moving forward, if you run it out of cron for instance, it would help make sure nobody messes things up before it's too late. I meant to release the code, but never got around to it. It will not work out of the box for you. It's got static constants and code structure that is dependent upon our specific installation. As I state several times in the comments, too much automation defeats the overall purpose of the application. It's a watchdog application.

Anyway, sorry for the rambling on the subject. I hope some of this was helpful!

[eschoeller]

Here is that other forum post I was looking for:

http://forums.cacti.net/about23885-0-asc-15.html

tekbot really nails it.

And of course, the online RRA calculator is always useful:

http://rrdtools.appspot.com/

[jorhett]

The moral of the story is that sometimes you can fix this stuff live - but you generally need to be an rrdtool ninja, and accept certain risks that your data might not be there in months or years to come.

Oh, bosh. Sorry for responding to an old thread but I want to prevent others from believing this kind of nonsense. Yes, it takes a while to understand RRD in general but there's no ninja and there's no chance of losing data. The kind of issues eschoeller described above was a failure to do elementary (in every sense of the word) math. It's very simple. Multiply your sample size in seconds by the rows in the RRA then divide by days (86400) and you'll know exactly how much data you are saving in that sample size.

Example:
If you have an RRA storing 5 minute averages (= 900 seconds) and you have 1000 rows of them, then you will have 900 * 1000 / 86400 = 10.4 days worth of 5 minute averages available.

Likewise, do the reverse to figure out how many rows you should have. If you really want a year's worth of 5 minute averages:
86400 seconds in a day * 366 days in a leap year / 900 seconds in 5 minutes = 35136 rows in your RRA

There is absolutely nothing difficult about specifying RRA sizes. The logic around using the right RRA for a report when multiple RRAs have data for the same time range is a bit tricker, but that's non-destructive and you can test until you get it right.