[HOWTO] RESIZE existing RRAs of existing RRDs

[gandalf]

Please find current version at http://docs.cacti.net


Preface
This tutorial is based on rrdtool, written by Tobias Oetiker. It is written by Reinhard Scheck to help you figuring out, how to resize an existing RRA of a given rrd file. This tutorial is based on Howto RESIZE existing RRAs of existing RRDs and extends this to some kind. Hope, this one will work even for windows users.

Be warned!
BACKUP ALL YOUR RRDs!
There's a good chance, that you will destroy all of your rrd files. I'm not joking! At the time of writing, rrdtool 1.2.12 is stable. Pay attention to older rrdtool-1.2.x version as they contain a bug when resizing rrd files created by rrdtool-1.0.x (see above reference for more).
SO BACKUP ALL YOUR RRDs! And check for sufficient file space!
As always: Use this information at your own risk.

Here we go!
As an attachment to the forum entry you will find a perl script resize.pl. It is necessary, to customize the /path/to/the/rrd/binary, e.g /usr/bin/rrdtool.

Help!
Put resize.pl wherever you want. There's no need to put it into the rrd working directory. But you will need some scratch space here for all rrds to be resized (due to the way rrdtool resize works). The user that runs this script must have

• write permissions to the current directory used for scratch
• read permissions on the original rrds to be resized
• write permissions to the target directory to store the resized rrds in

The script does not care about space provided. To get help, simply type

Code: Select all

perl resize.pl -h

you will receive

Code: Select all

resize.pl Version 0.43 - resize an existing rrd
        Usage:    resize.pl
                                -f <filemask>
                                -r <rra> | -s <actual row size>
                                -o <output dir>
                                -g <growth>
                                -i
                           [-d <debug>]
        Requires: Getopt::Std, File::Basename, File::stat, File::Copy, File::KGlob, RRDp
        Author:   Reinhard Scheck
        Date:     2006-01-15
        Options:
                  -f, filemask of the source rrds
                  -r, rra to be changed (first rra denotes as -r 0)
                  -s, take only rra's with exactly that actual row size
                  -o, output directory for resized rrds
                  -g, growth (number of data points to be ADDED to those already defined)
                  -i, invoke rrdtool info instead of resizing
                  -d, debug level (0=standard, 1=function trace, 2=verbose)
                  -h, usage and options (this help)

        -s or -r must be given. -s will override -r option
        No parameter validation done. Hope you know what you're going to do!

Dry run
You may want to have a look at your rrds before resizing them. Specially for the required parameter -r (denoting the rra to be resized), you will want to have a look at those rras, that are defined in the rrd in question. Example (linefeeds only for ease of reading):

Code: Select all

perl resize.pl -f "/var/www/html/cacti/rra/localhost_uptime_57.rrd" /
            -r 0 /
            -o /var/www/html/cacti/rra/resized/ 
            -g 8000 /
            -i

will result in:

Code: Select all

-- RRDTOOL INFO localhost_uptime_57.rrd ...
ds[uptime].type = "GAUGE"
rra[0].cf = "AVERAGE"
rra[0].rows = 600
rra[1].cf = "AVERAGE"
rra[1].rows = 700
rra[2].cf = "AVERAGE"
rra[2].rows = 775
rra[3].cf = "AVERAGE"
rra[3].rows = 797
rra[4].cf = "MIN"
rra[4].rows = 600
rra[5].cf = "MIN"
rra[5].rows = 700
rra[6].cf = "MIN"
rra[6].rows = 775
rra[7].cf = "MIN"
rra[7].rows = 797
rra[8].cf = "MAX"
rra[8].rows = 600
rra[9].cf = "MAX"
rra[9].rows = 700
rra[10].cf = "MAX"
rra[10].rows = 775
rra[11].cf = "MAX"
rra[11].rows = 797
rra[12].cf = "LAST"
rra[12].rows = 600
rra[13].cf = "LAST"
rra[13].rows = 700
rra[14].cf = "LAST"
rra[14].rows = 775
rra[15].cf = "LAST"
rra[15].rows = 797

You may notice

• a single data source (uptime)
• four consolidation functions (AVERAGE, MIN, MAX, LAST)
• four rra's for each of the consolidation functions

Of course, you may also enter a partly qualified dataset name. But it makes sense to take only those rrd's, that belong to the same datasource (e.g. with the same rrd file structure).

Resizing a single RRA of a single RRD
For ease of use, you may simply omit the trailing parameter -i. But pay attention to the parameter -r! In this example, only the first RRA of the consolidation function AVERAGE shall be resized. It depends on your needs, whether this will result in a correct RRD!

Code: Select all

perl resize.pl -f "/var/www/html/cacti/rra/localhost_uptime_57.rrd" /
            -r 0 /
            -o /var/www/html/cacti/rra/resized/ 
            -g 8000

The output will look like:

Code: Select all

-- RRDTOOL RESIZE localhost_uptime_57.rrd RRA (0) growing 8000.. (95328).. RRA#0.. (159328).. Done.

The first parenthesis contain the file size before resizing, the second one after resizing.

Resizing multiple RRA of a single RRD
Simply enter all RRAs to be resized in quotes:

Code: Select all

perl resize.pl -f "/var/www/html/cacti/rra/localhost_uptime_57.rrd" /
            -r \u201c0 4 8 12\u201d /
            -o /var/www/html/cacti/rra/resized/ 
            -g 8000

to result in

Code: Select all

-- RRDTOOL RESIZE localhost_uptime_57.rrd RRA (0 4 8 12) growing 8000.. (95328).. RRA#0#4#8#12.. (351328).. Done.

Resizing multiple RRAs of multiple RRDs
Please enter all RRAs to be resized in quotes and partly qualify alll RRDs:

Code: Select all

perl resize.pl -f "/var/www/html/cacti/rra/*_uptime_*.rrd" /
            -r \u201c0 4 8 12\u201d /
            -o /var/www/html/cacti/rra/resized/ 
            -g 8000

to result in

Code: Select all

-- RRDTOOL RESIZE router_uptime_59.rrd RRA (0 4 8 12) growing 8000.. (95328).. RRA#0#4#8#12.. (351328).. Done.
-- RRDTOOL RESIZE gandalf_uptime_58.rrd RRA (0 4 8 12) growing 8000.. (95328).. RRA#0#4#8#12.. (351328).. Done.
-- RRDTOOL RESIZE localhost_uptime_57.rrd RRA (0 4 8 12) growing 8000.. (95328).. RRA#0#4#8#12.. (351328).. Done.

Resizing all RRAs of a given row size
This is a new feature of this version. Use the parameter -s to specify the rowsize of the rra's you want to change. This parameter overrides the -r parameter, cause all relevant rra's will be calculated from the current rrd definition. This is useful if you're working on a list of files with different rrd structure (e.g. different Data Templates)

Code: Select all

perl resize.pl -g 8000 -f "/var/www/html/workspace/branch/rra/gandalf*.rrd" /
            -s 600
            -o /var/www/html/cacti/rra/resized/ 
            -g 8000

to result in

Code: Select all

... removed ...
-- RRDTOOL RESIZE gandalf_cpu_system_9.rrd RRA (0 4 ) growing 8000..47836.. RRA#0#4..175840.. Done.
-- RRDTOOL RESIZE gandalf_cpu_user_10.rrd RRA (0 4 ) growing 8000..47836.. RRA#0#4..175840.. Done.
-- RRDTOOL RESIZE gandalf_errors_in_18.rrd RRA (0 4 ) growing 8000..188308.. RRA#0#4..700312.. Done.
... removed ...
-- RRDTOOL RESIZE gandalf_unicast_in_20.rrd RRA (0 4 ) growing 8000..94660.. RRA#0#4..350664.. Done.
... removed ...
-- RRDTOOL RESIZE gandalf_uptime_58.rrd RRA (0 4 8 12 ) growing 8000..95328.. RRA#0#4#8#12..351328.. Done.
-- RRDTOOL RESIZE gandalf_users_89.rrd RRA (0 4 8 12 ) growing 8000..95328.. RRA#0#4#8#12..351328.. Done.
user time: 0.34 system time: 1.93 real time: 7.16

Please notice the last line of output, which reports the rrdtool runtime. If -s is given so that no rowsize of any rra will match, the corresponding rrd file is skipped:

Code: Select all

perl resize.pl -g 8000 -f "/var/www/html/workspace/branch/rra/gandalf*.rrd" -s 601 -o new-resized
user time: 0.01 system time: 0.02 real time: 0.23

Something to keep in mind
Be warned!
You may even enter -o to resolve to the current RRD directory. This will result in overwriting your existing RRDs. YOU DON'T WANT TO DO THAT. Always look at the output after resizing. Try to generate graphs from them. Verify, that everything runs fine.
BACKUP YOUR ORIGINAL RRDs.
I appreciate any feedback to improve this document.
Reinhard Scheck aka lvm

Edit (Jan, 21th 2006): As I noticed recently, the module File::KGlob isn't really necessary. So if you get trouble with installing this really old module, simply delete this one. I checked this with some 1000 rrds without any problem

[evilzardoz]

Once I have resized my RRAs, how do I tell cacti not to consolidate?

Also, I can't seem to find RRDp.pm anywhere.

[gandalf]

Once I have resized my RRAs, how do I tell cacti not to consolidate?

You can't. First, consolidation is a feature of rrdtool, not cacti. Second, as long as you keep rra's with steps>1, this will ask rrdtool to do consolidation. There's nothing bad in doing so. This is a very efficient way of keeping the important part of data. See my first few signatures to get an idea how to interpret MAXIMUM or AVERAGE values after consolidation.

Also, I can't seem to find RRDp.pm anywhere.

This is part of the rrdtool rpms related to perl. When compiling on your own, there are some switches to set for perl modules to be generated.
Reinhard

[evilzardoz]

lvm - thanks for the help so far. We want to keep more detailed information for historical reasons which is why consolidation, while a good idea, isn't quite what we're looking for.

I've found some extra packages for RRD-perl stuff and they look like what I might need so I'll give those a shot.

[gandalf]

Of course you can "turn off consolidation" by only defining a single rra, consisting of LOTS of data points. That would be HUGE,of course. But for some very few data templates, I did this. Took me SOME discussions with those interpreting the results; this is a bit "funny". Try this, and you'll knwo what I mean. This effect will surely pertain to all other solutions (what do you think comes out, when you try to graph some 1000 data points when there are only about 500 pixels on the screen ...). Perhaps this would be a solution for intervals of 1h or so ...
Reinhard

[evilzardoz]

I still want the daily, monthly, yearly graphs.. but I don't want to loose granularity over time i.e. if I want to see what our network looked like three years ago.

Is this a bad idea?

[gandalf]

I still want the daily, monthly, yearly graphs.. but I don't want to loose granularity over time i.e. if I want to see what our network looked like three years ago.

Is this a bad idea?

Not bad; not yet solved only
Some ideas but no solution
Reinhard

[gandalf]

Please define 4 new RRAs as shown (Xff is always 0.5) in picture 1. In your Data Templates, you will reference them as shown in picture 2 when creating the Data Source.
Afterwards, when the Data Sources are created (not earlier) you may change this to settings like picture 3. Now, when graphing, the commonly known 4 timespan graphs are shown.
If you add all 4 RRAs when creating the data sources, you'll get an rrd file with 4 rra's while you only want to have one. So follow the instructions carefully (perhaps a little code patch would supress generating weird rra definitions like those I mentioned).
In fact, I'd prefer to seperate the definition of graphing timespans from rra definitions. In "reality", the timespans to be defined for cacti's rra are not needed for rrdtool.
Reinhard

[time]

I only just noticed today, but some of my data sources maintain 5 minute resolution without me having done anything. It is seemingly random in that I have used the same graph and data template multiple times and in some instances it keeps all the data while for other devices it hasn't. See a couple of graphs as an example, these are taken today (28/02/2006) by zooming in on my yearly graph - notice the dates and you can see it is still data per 5 minutes.

I'm not too sure where to poke around and what to look for to work out why this is happening either, but I've checked the RRD file sizes and the ones with 5 minute resolution are ~1.5M or 3M versus ~140K. This seems to have happened for about 1500 of my 3200 data sources. The time these data sources were created isn't of any significance either by the looks of it as some of the ones that are maintaining resolution are fairly old.

Tim

[gandalf]

The size is a good indication for those rrd's who got wrong. If you're using pure cacti, I'd swear there is ABSOLUTELY NO way to change rrd definitions if the file has once been created.
So it must have occured at generation time OR was changed manually afterwards.
You may check your RRA definitions to see, whether there is actually any RRA defining 5 min intervals for infinity. Another source may be importing tenplates. In each import you'll find rra definitions that will overwrite your settings. cacti always reports "updated" whether it really changed existing rra definitions or not. To be SURE, you would have to check rra definitions EACH time after importing. No joke, this is a very useful check; you should really do this (well, it already occured to me and I claim to be a somewhat experienced cacti user).
And if you have this unwanted settings for lots of rrds, this will surely slow cactis poller performance down. Changing this is near to magic ...
Perhaps dumping, changing XML and reloading would do ...
Reinhard

[time]

I haven't changed anything manually, so perhaps things got changed with template imports as you suggested. What RRA definitions should I be checking exactly? I've looked at the rra definitions for the graphs above that have 5 minute data and they all have the Daily, Weekly, Monthly, Yearly RRAs associated with them.

It is not an unwanted setting, I actually prefer to keep all the data there as it allows me to look at old data with 5 minute accuracy. The reason I posted in this thread was because I thought maybe something happened with my installation that inadvertently caused some of my graphs to maintain 5 minute accuracy which is what a few people seem to want also.

It also doesn't seem to slow the poller down at all, my poller stats are:

03/02/2006 08:55:32 AM - SYSTEM STATS: Time:30.3416 Method:cactid Processes:1 Threads:6 Hosts:172 HostsPerProcess:172 DataSources:5819 RRDsProcessed:3072

This is running on a P4 2.8G, 512 M ram and it's got 40 Gig hard disk (of which only 6 or 7 Gig is being used currently) so I have plenty of space to store the larger rrd files.

[warenet]

Reinhard,

I have a question about your insructions below:

Please define 4 new RRAs as shown (Xff is always 0.5) in picture 1. In your Data Templates, you will reference them as shown in picture 2 when creating the Data Source.
Afterwards, when the Data Sources are created (not earlier) you may change this to settings like picture 3. Now, when graphing, the commonly known 4 timespan graphs are shown.
If you add all 4 RRAs when creating the data sources, you'll get an rrd file with 4 rra's while you only want to have one. So follow the instructions carefully (perhaps a little code patch would supress generating weird rra definitions like those I mentioned).
In fact, I'd prefer to seperate the definition of graphing timespans from rra definitions. In "reality", the timespans to be defined for cacti's rra are not needed for rrdtool.

If I want to switch all my customer bandwidth graphs to 1 year, 5 minute accuracy and I create the "ZLVM" RRAs, will I have to do something special each time I want to create a new graph?

Specifically do I have to go in and disable all the RRAs but the "yearly" one?

I want to make this easy for my network engineers to handle when they turn up a new customer.

Thx!

[warenet]

Code: Select all

./resize.pl -f "/var/www/html/cacti/rra.bak/ana_traffic_out_795.rrd" -s 600 -g 114600 -o /var/www/html/cacti/rra.bak/resized/ -i

-- RRDTOOL INFO ana_traffic_out_795.rrd ...
open2: exec of /usr/bin/rrdtool - failed at /usr/local/rrdtool-1.2.12/lib/perl/5.8.5/RRDp.pm line 120

Then it just hangs...

[gandalf]

If I want to switch all my customer bandwidth graphs to 1 year, 5 minute accuracy and I create the "ZLVM" RRAs, will I have to do something special each time I want to create a new graph?

Yes, these RRAs are faked. AFAIK, they will break rrd file generation (but I don't remember exactly) or at least produce some unreasonable rrd. My suggestion is a REALLY NASTY WORKAROUND, no more.
Reinhard

[gandalf]

Code: Select all

./resize.pl -f "/var/www/html/cacti/rra.bak/ana_traffic_out_795.rrd" -s 600 -g 114600 -o /var/www/html/cacti/rra.bak/resized/ -i

-- RRDTOOL INFO ana_traffic_out_795.rrd ...
open2: exec of /usr/bin/rrdtool - failed at /usr/local/rrdtool-1.2.12/lib/perl/5.8.5/RRDp.pm line 120

Then it just hangs...

Please change the script

Code: Select all

# --- initialization ----------------------------------------------------------
my $rrd_binary  = "/usr/bin/rrdtool";   # path to rrd binary, to be customized! <<<<<<<<<<<<<<<<<<<<

as stated in my first post
Reinhard