[HOWTO] Use externally updated rrd files + BONUS TRACK

[gandalf]

This document is not updated any more. Last version, see http://docs.cacti.net/manual:087:8_rrdt ... dated_rrds
Preface
Due to a proposal of BSOD2600 I try to compile a HowTo for Using externally updated rrd files. Cacti itself provides a very flexible and fast data gatherer, but situations may occur, where this is not what you want. Assuming, that there already is some rrd file filled by other means, question arises how to use this with cacti's flexible rrdtool front-end administration facilities.
Sometimes people call this feature Importing external rrds to cacti. But what I'm going to explain is not an automated function. It will require some manual interaction.
Of course, the webserver must have at least read access to the required rrd file(s). For sake of easiness, I'll assume the file to be located in cacti's default ./rra/ directory. In my examples, this file is called example.rrd.

As always: Use this on your own risk

Table of Contents
Chapter I: Get rrdtool info
Chapter II: Create the Data Template
Chapter III. Create the Graph Template
Chapter IV: Prepare the Host
Chapter V: Creating the Data Sources
Chapter VI: Create the new Graph
Bonus Track: Updating RRD Files from Remote

Reinhard

[gandalf]

Chapter I: Get rrdtool info
First task is to get information about the data sources used in the rrd file. This is done using rrdtool info <rrd file>:

Code: Select all

rrdtool info ./rra/external.rrd
filename = "/var/www/html/cacti/rra/external.rrd"
rrd_version = "0003"
step = 300
last_update = 1140957748
ds[external_ds1].type = "GAUGE"
ds[external_ds1].minimal_heartbeat = 600
ds[external_ds1].min = 0.0000000000e+00
ds[external_ds1].max = 5.0000000000e+02
ds[external_ds1].last_ds = "UNKN"
ds[external_ds1].value = 4.4510209500e+02
ds[external_ds1].unknown_sec = 0
ds[external_ds2].type = "GAUGE"
ds[external_ds2].minimal_heartbeat = 600
ds[external_ds2].min = 0.0000000000e+00
ds[external_ds2].max = 5.0000000000e+02
ds[external_ds2].last_ds = "UNKN"
ds[external_ds2].value = 7.4183682500e+02
ds[external_ds2].unknown_sec = 0
ds[external_ds3].type = "GAUGE"
ds[external_ds3].minimal_heartbeat = 600
ds[external_ds3].min = 0.0000000000e+00
ds[external_ds3].max = 5.0000000000e+02
ds[external_ds3].last_ds = "UNKN"
ds[external_ds3].value = 1.0385715550e+03
ds[external_ds3].unknown_sec = 0
rra[0].cf = "AVERAGE"
rra[0].rows = 600
rra[0].pdp_per_row = 1
(... more to follow ...)

From the ds[.....] statements, the names of the ds' are taken. In this case the data sources are named

• external_ds1,
• external_ds2,
• external_ds3

respectively. While this is not that a meaningful name, it should show you the principles when dealing with multi-ds rrds. Alongside with this, it is good to know the ds[...].type, ds[...].min and ds[...].max for correct definition of the data sources. In this case, all data sources are of type GAUGE. This will not affect data gathering nor graphing, but to me it seems to be advantageous to create the correct data sources.
All other paramaters (step, heartbeat, xff, rra[..].rows) are assumed to be standard settings.

[gandalf]

Chapter II: Create the Data Template
The Data Template will tell cacti, how the data is stored within the rrd file. This is the way to tell cacti about all data sources and their properties. The purpose is twofold:

• tell cacti, how to create the correct rrd file with all parameters
• store the name of the data sources for later use with Graph Templates

While the first goal is not needed in this context, the second one is crucial.
So lets define a new Data Template. Goto Data Templates and Add a new one:




Fill in the usual header data:




If you want to associate this to a certain host, you may use |host_description| as a placeholder as usual. As this external.rrd file is updated externally, you must set the Data Input Method to None.
Select the Associated RRA's as they will define the Detailed Graph Views (usually Daily, Weekly, Monthly, Yearly). And you'll have to uncheck the Data Source Active checkbox. This will prevent cacti from actually gathering data for this Data Template. Now add the first Data Source:





Internal Data Source Name
You must use the existing data source name of the rrd as retrieved in Chapter I. In this example, use external_ds1.

Minimum Value
Fill in the ds[...].min value from rrdtool info above. In this case, use 0. This is not really needed, but for sake of consistency I recommend this.

Maximum Value
Fill in the ds[...].max value from rrdtool info above. In this case, use 500. This is not really needed, but for sake of consistency I recommend this.

Data Source Type
Fill in the ds[...].type value from rrdtool info above. In this case, use GAUGE. This is not really needed, but for sake of consistency I recommend this.

Heartbeat
Fill in the ds[...].minimal_heartbeat value from rrdtool info above. In this case, use 600. This is not really needed, but for sake of consistency I recommend this.

Now Create to see:




Repeat this for all other data sources of the example.rrd (use the New function of Data Source Items):





You will have noticed, that no Custom Data is given as defined by the Data Input Method set to NONE. You'll see the result as:




Now you're done.

Reinhard

[gandalf]

Chapter III. Create the Graph Template
The Graph template will tell cacti, what Data Sources should be shown on the Graph. This is very straightforward; simply use the Data Sources as defined in Chapter II and use all the Graph magic cacti provides. So Add a new Graph Template




and fill in the usual header data:




Now add the first Graph Item:




and fill in the usual Data:




Use the Legend Option as a time saver:




And add all other Graph Items like that:







to end up in





Of course you will use more meaningful input for Text Format. That's all for now.

Reinhard

[gandalf]

Chapter IV: Prepare the Host
Now its time for the Host to be prepared. You have the choice between two different approaches

• the Host already exists (perhaps you're polling some other data from this host) and the status is up
• the Host does not yet exist in cacti's tables and shall never been polled for other data by cacti's own poller

The first approach does not need any additional changes to the Devices list.
The second approch will be more common. You will need a Host entry in the Devices list even for this host. So we will create kind of a dummy entry. Please goto the Devices list and Add a new one:




Fill in Description and Name as usual. To deactivate all checks, please check Disable Host and leave SNMP Community empty.



Create




Please proceed to the next chapter now

Reinhard

[gandalf]

Chapter V: Creating the Data Sources
Normally, you would create the Data Sources and Graphs automagically using Create Graphs for this Host. But using this approach, cacti would enumerate and generate the needed rrd file(s) on its own. That's not wanted now.
Our task is: make cacti generate all Data Sources stuff but accept our external.rrd file name. This is accomplished by adding the Data Sources manually from the Data Sources list. In this case, there's no difference whether you're going to use an already existing host or the kind of host we generated in the previous chapter.




Now select the Data Template we generated in Chapter II:




and Create. You will be prompted to fill in the full path to your external.rrd file. If this resides in cacti's default ./rra directory, you may use <path_rra> for this. Remember, that the web server must have at least read access to that file.:




The result is shown in the next image




Reinhard

[gandalf]

Chapter VI: Create the new Graph
Now we're nearly done! The last Step now!
Let's create a new Graph from Graph Management now.




Select the Selected Graph Template be defined in Chapter III:




and Create. Now select all needed Data Source [...] and Save:




The result is shown like:




Please select this Graph again and Turn on Graph Debug Mode to see




Isn't it nice, this Graph

have fun, use cacti
Reinhard

[gandalf]

Bonus Track: Updating RRD Files from Remote
It is known, that rrd files are architecture-dependant. So, if you're updating external rrd files and graphing them via cacti, there's a need to make cacti access these rrd files, e.g. by nfs. This will not succeed when mixing architectures.
And nfs may not be the best choice for all cases. But with rrdtool 1.2.x there's a new feature, rrd server. Pasted from rrdtool homepage:

RRD Server

If you want to create a RRD-Server, you must choose a TCP/IP Service number and add them to /etc/services like this:

rrdsrv 13900/tcp # RRD server

Attention: the TCP port 13900 isn't officially registered for rrdsrv. You can use any unused port in your services file, but the server and the client system must use the same port, of course.

With this configuration you can add RRDtool as meta-server to /etc/inetd.conf. For example:

rrdsrv stream tcp nowait root /opt/rrd/bin/rrdtool rrdtool - /var/rrd

Don't forget to create the database directory /var/rrd and reinitialize your inetd.

If all was setup correctly, you can access the server with perl sockets, tools like netcat, or in a quick interactive test by using 'telnet localhost rrdsrv'.

NOTE: that there is no authentication with this feature! Do not setup such a port unless you are sure what you are doing.

For my local setup (RHEL 4.0), I had to modify this a bit. My /etc/services

Code: Select all

rrdsrv         13900/tcp                       # RRD server

is standard. And I put

Code: Select all

# default: off
# description: RRDTool as a service
service rrdsrv
{
        disable         = no
        socket_type     = stream
        protocol        = tcp
        wait            = no
        user            = cactiuser
        server          = /usr/bin/rrdtool
        server_args     = - /var/www/html/cacti/rra
}

as /etc/xinetd.d/rrdsrv. Then xinetd is updated to start rrdsrv.
user should be set to the user defined in include/config.php.
server_args should be set to cacti's rra directory.
server contains the full path to rrdtool's binary. To verify this, try

Code: Select all

telnet localhost 13900
info external.rrd

assuming, that the rrd file "external.rrd" used in this howto is located in the ./rra directory.
Now its time for some remote script to use this new feature. As an example, see

Code: Select all

#!/usr/bin/perl
use IO::Socket;

my $host = shift @ARGV;
my $port = shift @ARGV;
my $rrd  = shift @ARGV;

my $socket = IO::Socket::INET->new(PeerAddr=> $host,
                                PeerPort=> $port,
                                Proto=> 'tcp',
                                Type=> SOCK_STREAM)
                                or die "Can't talk to $host at $port";

my $_cmd = "update " . $rrd . " N:" . int(rand(10)) . ":" . int(rand(10)) . ":" . int(rand(10));
print $socket $_cmd . "\n";
close $socket;

Of course,

Code: Select all

my $_cmd = "update " . $rrd . " N:" . int(rand(10)) . ":" . int(rand(10)) . ":" . int(rand(10));

is only an example prepared for the "external.rrd" of our example. To use this for updating your own rrd files, this must fit to the data source definitions of your special rrd file. In our example, I put

Code: Select all

*/5 * * * *     cactiuser       /usr/bin/perl /var/www/html/cacti/scripts/rrd-remote-update.pl localhost 13900 external.rrd

into crontab to perform regular updates.

Disadvantages
This handling has the great disadvantage, that you must configure the rrd file name to each single updating script. This rrd file name on the remote system must match the one on cacti's host. There's a good chance to mess things up when used for lots of rrds. But for some few files this may be appropriate.

Reinhard

[colchaodemola]

Hi , i am trying to import a rrd file in to cacti but i am having a lot of troubles ...

see this :

# rrdtool info imq1-tc.rrd
filename = "imq1-tc.rrd"
rrd_version = "0003"
step = 10
last_update = 1171814910
ds[101-].type = "DERIVE"
ds[101-].minimal_heartbeat = 60
ds[101-].min = 0.0000000000e+00
ds[101-].max = NaN
ds[101-].last_ds = "0"
ds[101-].value = 0.0000000000e+00
ds[101-].unknown_sec = 0
ds[102-].type = "DERIVE"
ds[102-].minimal_heartbeat = 60
ds[102-].min = 0.0000000000e+00
ds[102-].max = NaN
ds[102-].last_ds = "0"
ds[102-].value = 0.0000000000e+00
ds[102-].unknown_sec = 0
ds[103-].type = "DERIVE"
ds[103-].minimal_heartbeat = 60
ds[103-].min = 0.0000000000e+00
ds[103-].max = NaN
ds[103-].last_ds = "0"
ds[103-].value = 0.0000000000e+00
ds[103-].unknown_sec = 0
ds[104-].type = "DERIVE"
ds[104-].minimal_heartbeat = 60
ds[104-].min = 0.0000000000e+00
ds[104-].max = NaN
ds[104-].last_ds = "0"
ds[104-].value = 0.0000000000e+00
ds[104-].unknown_sec = 0
ds[105-].type = "DERIVE"
ds[105-].minimal_heartbeat = 60
ds[105-].min = 0.0000000000e+00
ds[105-].max = NaN
ds[105-].last_ds = "0"
ds[105-].value = 0.0000000000e+00
ds[105-].unknown_sec = 0
ds[106-].type = "DERIVE"
ds[106-].minimal_heartbeat = 60
ds[106-].min = 0.0000000000e+00
ds[106-].max = NaN
ds[106-].last_ds = "0"
ds[106-].value = 0.0000000000e+00
ds[106-].unknown_sec = 0
ds[191-].type = "DERIVE"
ds[191-].minimal_heartbeat = 60
ds[191-].min = 0.0000000000e+00
ds[191-].max = NaN
ds[191-].last_ds = "1702761427"
ds[191-].value = 3.1231833343e+02
ds[191-].unknown_sec = 0
ds[192-].type = "DERIVE"
ds[192-].minimal_heartbeat = 60
ds[192-].min = 0.0000000000e+00
ds[192-].max = NaN
ds[192-].last_ds = "0"
ds[192-].value = 0.0000000000e+00
ds[192-].unknown_sec = 0
ds[193-].type = "DERIVE"
ds[193-].minimal_heartbeat = 60
ds[193-].min = 0.0000000000e+00
ds[193-].max = NaN
ds[193-].last_ds = "43731333"
ds[193-].value = 0.0000000000e+00
ds[193-].unknown_sec = 0
ds[194-].type = "DERIVE"
ds[194-].minimal_heartbeat = 60
ds[194-].min = 0.0000000000e+00
ds[194-].max = NaN
ds[194-].last_ds = "402684"
ds[194-].value = 0.0000000000e+00
ds[194-].unknown_sec = 0
rra[0].cf = "AVERAGE"
rra[0].rows = 8640
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[0].cdp_prep[1].value = NaN
rra[0].cdp_prep[1].unknown_datapoints = 0
rra[0].cdp_prep[2].value = NaN
rra[0].cdp_prep[2].unknown_datapoints = 0
rra[0].cdp_prep[3].value = NaN
rra[0].cdp_prep[3].unknown_datapoints = 0
rra[0].cdp_prep[4].value = NaN
rra[0].cdp_prep[4].unknown_datapoints = 0
rra[0].cdp_prep[5].value = NaN
rra[0].cdp_prep[5].unknown_datapoints = 0
rra[0].cdp_prep[6].value = NaN
rra[0].cdp_prep[6].unknown_datapoints = 0
rra[0].cdp_prep[7].value = NaN
rra[0].cdp_prep[7].unknown_datapoints = 0
rra[0].cdp_prep[8].value = NaN
rra[0].cdp_prep[8].unknown_datapoints = 0
rra[0].cdp_prep[9].value = NaN
rra[0].cdp_prep[9].unknown_datapoints = 0


data source names ends with a `-` and cacti shows a error when i try create data template using those names.

So i tried to select use per data source name but after all the passes when i try to build the graphs i get :


RRDTool Command:

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title="Localhost - Traffic_Control_GRAPH_Template" \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label="" \
--slope-mode \
GPRINT::LAST:"Current\:%8.2lf %s" \
GPRINT::AVERAGE:"Average\:%8.2lf %s" \
GPRINT::MAX:"Maximum\:%8.2lf %s\n" \
GPRINT::LAST:"Current\:%8.2lf %s" \
GPRINT::AVERAGE:"Average\:%8.2lf %s" \
GPRINT::MAX:"Maximum\:%8.2lf %s\n"

RRDTool Says:

ERROR: Could not parse line 'GPRINT::LAST:Current\:%8.2lf %s'





Any help ???

[gandalf]

Hmm, I'm not sure when it comes to "-" in ds names. Perhaps this screws everything up. All I can say now, is that the graph is definitively broken. But let's please start at your Data Template (a screenshot would do).
Reinhard

[colchaodemola]

OK.
Starting by data template ...

[colchaodemola]

after graph template ...

[colchaodemola]

anything else ?

[gandalf]

OK.
Starting by data template ...

Ok, as you said; cacti is not accepting the "-" sign. From current rrdtool man pages (man rrdcreate), I've found

DS:ds-name:DST:dst arguments
...
ds-name is the name you will use to reference this particular data source from an RRD. A ds-name
must be 1 to 19 characters long in the characters [a-zA-Z0-9_].

This states, that a "-" sign is not allowed (although in your case, the rrd files accepted this char). So I'm afraid: the solution requires to change the ds names (this is supported by "rrdtool tune ..."). It will require to change your data sampling script(s) as well.
Reinhard

[colchaodemola]

If you know perl could you give me a help ?

I am trying to change the script but i always get some syntax error ...

The script is here : http://edseek.com/~jasonb/code/polltc-1.05.tar.gz

It would be a big help , it is a very small script ...