CLI script to mass tune RRDs

[lints]

Have you ever created a data template without maximum or minimum values and then made a graph based on it only to discover after some time (and many created graphs later), that you should have!

Maybe you were diligent and DID put maximum or minimum values, but they were wrong!
Maybe you downloaded a template pack and the min/max values were set, but too high/low for your environment!

In the past you had 2 options after updating the data template to match your desired settings:

1. Find all the RRDs associated with that template and manually run 'rrdtool tune' to change the min/max values
2. Delete all your data sources and graphs based on this template and re-create them, loosing all your historical data.

Well not any more!!!

I present for use by the Cacti Community the RRD mass tuning Command-line tool!

Put this file in your 'cli' directory, change the extention to 'php' and tune away your data source blues.

tune_data_source_rrds.txt

CLI script

(10.13 KiB) Downloaded 1079 times

This script takes a data template ID and optionally a host/device ID and passes rrdtool whatever options you specify, allowing you to mass tune all the RRDs for a particular data template.

For example, I created a template for SQL Server's table scan stats some time ago, but since then our servers have undergone a massive increase in load thanks to SharePoint. Naturally, we have a large number of SQL Servers and therefore a large number of data sources based on this data template.

If I were to try and update all these data sources, I'd have to go in to each one and find the full path to the RRD file, copy it to my command prompt window and type out the command manually. This isn't so bad if you only have 2-3 graphs of a given type, but what if you had 100 or 1000! it could take AGES to do and you might miss one.

With this new script, you can just run:

Code: Select all

php -q tune_data_source_rrds.php --data-template-id=123 -- -a mssql_index_srchs:400000

This will tune ALL data sources based on this data template to update the 'mssql_index_srchs' item to have a maximum value of 400000. It's also possible to use any of the arguments that you'd normally pass to 'rrdtool tune'.

I've tried this on my server and it seems to work great for my needs, but I'd like a few people out in the community to try it on their environments and see what kind of results they get.

If you have questions, comments or recommendations, I'd love to hear them.

Thanks!

[noname]

Great works! :)

One proposal: It's better to set 'true' for the 2nd argument of get_data_source_path().
('extended_paths' config option is for "Structured RRA Path", so it's not the same as 'expand_paths' in that function)

At line 152:
$data_sources[$ds['local_data_id']]['rrd_path'] = get_data_source_path($ds['local_data_id'], TRUE);

Anyway, this tool worked fine in my environment. (Cacti 0.8.7g + Official patches)

When checked:

Code: Select all

# php -q tune_data_source_rrds.php --host-id=11 --data-template-id=41
Test Switch - Traffic - Fa0/4 (Host: 11)
        RRD Path: /var/www/cacti-0.8.7g/rra/test_swich_traffic_in_534.rrd
        RRD Definition:
        DS[traffic_in] typ: COUNTER     hbt: 600        min: 0.0000     max: 100000000.0000
        DS[traffic_out] typ: COUNTER    hbt: 600        min: 0.0000     max: 10000000.0000

Test Switch - Traffic - Fa0/2 (Host: 11)
        RRD Path: /var/www/cacti-0.8.7g/rra/test_swich_traffic_in_532.rrd
        RRD Definition:
        DS[traffic_in] typ: COUNTER     hbt: 600        min: 0.0000     max: 100000000.0000
        DS[traffic_out] typ: COUNTER    hbt: 600        min: 0.0000     max: 10000000.0000

......

When changed parameters:

Code: Select all

# php -q tune_data_source_rrds.php --host-id=11 --data-template-id=41 -- --minimum traffic_in:99 --maximum traffic_in:999999999

Test Switch - Traffic - Fa0/4 (Host: 11)
        RRD Path: /var/www/cacti-0.8.7g/rra/test_swich_traffic_in_534.rrd
        RRD definition before tune:
                DS[traffic_in] typ: COUNTER     hbt: 600        min: 0.0000     max: 100000000.0000
                DS[traffic_out] typ: COUNTER    hbt: 600        min: 0.0000     max: 10000000.0000

        RRD definition after tune:
                DS[traffic_in] typ: COUNTER     hbt: 600        min: 99.0000    max: 999999999.0000
                DS[traffic_out] typ: COUNTER    hbt: 600        min: 0.0000     max: 10000000.0000

Test Switch - Traffic - Fa0/2 (Host: 11)
        RRD Path: /var/www/cacti-0.8.7g/rra/test_swich_traffic_in_532.rrd
        RRD definition before tune:
                DS[traffic_in] typ: COUNTER     hbt: 600        min: 0.0000     max: 100000000.0000
                DS[traffic_out] typ: COUNTER    hbt: 600        min: 0.0000     max: 10000000.0000

        RRD definition after tune:
                DS[traffic_in] typ: COUNTER     hbt: 600        min: 99.0000    max: 999999999.0000
                DS[traffic_out] typ: COUNTER    hbt: 600        min: 0.0000     max: 10000000.0000
......

Result:

# rrdtool info /var/www/cacti-0.8.7g/rra/test_swich_traffic_in_534.rrd
filename = "/var/www/cacti-0.8.7g/rra/test_swich_traffic_in_534.rrd"
rrd_version = "0003"
step = 300
last_update = 1314864012
ds[traffic_in].type = "COUNTER"
ds[traffic_in].minimal_heartbeat = 600
ds[traffic_in].min = 9.9000000000e+01
ds[traffic_in].max = 9.9999999900e+08
ds[traffic_in].last_ds = "0"
ds[traffic_in].value = 0.0000000000e+00
ds[traffic_in].unknown_sec = 0
ds[traffic_out].type = "COUNTER"
ds[traffic_out].minimal_heartbeat = 600
ds[traffic_out].min = 0.0000000000e+00
ds[traffic_out].max = 1.0000000000e+07
ds[traffic_out].last_ds = "0"
ds[traffic_out].value = 0.0000000000e+00
ds[traffic_out].unknown_sec = 0
...

Awesome! ^^

[lints]

One proposal: It's better to set 'true' for the 2nd argument of get_data_source_path().
('extended_paths' config option is for "Structured RRA Path", so it's not the same as 'expand_paths' in that function)

$data_sources[$ds['local_data_id']]['rrd_path'] = get_data_source_path($ds['local_data_id'], TRUE);

You're absolutely right. I think I mis-read the argument to 'get_data_source_path'; extended ~= expanded.

I've updated the script and re-attached it to the first post.

[noname]

Thanks :)
It worked fine as before.

[tosage]

It's work fine too on my installation.

An enhancement

an parameter --data-source-id=xx for modify only the data source

Thanks a lot for the code and your time lints

[prosodie]

Very useful script .

It's work fine too on my installation.

[fingaz]

I'm very sorry for bumping an old post; but will this also work if the Data Template is set to "Use Per-Data Source Value (Ignore this Value)"- ie:



Would i be correct in thinking that if i untick that box, save it, and then use this tool, *all* of the associated graphs will be modified to this value, and then i can use this tool as well?
I'm also assuming that the script won't modify the actual data template itself, so i will need to amend that first, prior to running rrdtune right?

Thanks in advance for any help!

Edit: Probably a bad example for the image above, but it displays the selection box at least- i'm assuming that i can ignore this Data Template, and just update the associated "traffic_in" and "traffic_out" Data Templates instead which this refers to. By all means, let me know if i'm wrong there- but the question is whether unticking that box, and setting a value, updates/overwrites all existing settings. Thanks

[tosage]

Sorry to dig up this post, changing this value does not modifiy the value in the rrd.
If you modifiy this value, only new datasources so rrd created wil have this value.
The PHP script modify the value in the rdd and it's do this job very well

[gandalf]

Not diving too much into this tool (thanks for the contribution anyway). But I'd like to add, that at http://docs.cacti.net, we published a "similar" toy. It was designed for resizing only, to be honest. It would have to be extended for more use cases.
But please also know, that RRD::Tweak, a perl extension, has been published some months ago.
Depending on your use case, you now may have different options to solve them ...
R.

[tosage]

Hello gandalf, what "toy" ?

[gandalf]

A resize perl script which uses rrdtool tune as well and allows for flexible filtering on specific rrd files
R.