Questions about making a custom data input method.

[cignul9]

Okay, my bosses want Cacti to graph the latency between each hop in a traceroute from host A to host B. I've written a perl script that I monkey moded from the other perl scripts that come with Cacti. Here it is:

$>cat ./traceroute.pl

Code: Select all

#!/usr/bin/perl

@trace = `traceroute -m 10 -w 2 -q 1 -n $ARGV[0] | sed -l0 -e '1{d}' -e 's/\*/0\.0\.0\.0\ \ 0\.000\ ms/g'`;
$hopcount = @trace;
for ($i = 0; $i < $hopcount; $i ++){
        $hop = $i + 1;
        @trace[$i] =~ s/( |)(.*[0-9])(.* )(.*[0-9]\..*[0-9]\..*[0-9]\..*[0-9])(.* )(.*[0-9])(\.[0-9][0-9][0-9] ms)//;
        print "hop$hop:$6 ";
}

This script will run a traceroute (10 hop count max, but you can change it if you need it to do the full 30) and record the latency (truncated to the nearest whole millisecond) for each hop and assign it to a label (variable?) of "hop1" for the first, "hop2" for the second, etc. The output of a trace to a host using this script looks like this:

$>./traceroute.pl 158.58.14.190
hop1:0 hop2:0 hop3:228 hop4:0 hop5:0 hop6:0 hop7:0 hop8:0 hop9:0 hop10:0

One caveat is timeouts. If a hop times out (after 2 seconds) it will record the hop as 0 milliseconds which should work for the graph I want to make.

My plan is to use this as a script for a data source and graph an AREA for each hop. That way we should be able to see the total latency with a breakdown for each hop up to the last one.

I've read the documentation and looked at the sample scripts and have some unresolved questions that I thought I'd ask here. After I got the script working I believe I need to make an associated Data Input Method in Cacti. Then I was going to make a Data Template and associate the two, and finally a Graph Template referencing the Data Template.

Some questions/concerns I have are:

• 1. I'm not sure how Cacti pipes my script output into an rrd database file, specifically, assuming the rrd database is created okay, I can't see how to associate the "hop1" "hop2", etc. lables the script generates into Output Fields in my Data Input Method(or whatever it is you associate these with). Does Cacti just assign the first to the first, second to second, etc?
  
  2. I can't find documentation about this, but it looks like:
  
  One script/query output item --> One Data Input Method Output Field --> One Data Template Data Source Item --> one or more Graph Template Items
  
  I hope this makes sense I'm trying to show associative relationships as I understand them, so is this correct?
  
  3.There are various other features in different screens that I don't understand and can't find in the manual, such as "Heartbeat" and "Step" in the Data Template. I'm curious, and would be happy to leave them on the default settings, but I would like to know that the default isn't going to undermine my efforts to make the graph I'm describing.

Again, any help you can offer is appreciated, and thanks for a great graphing tool.

[raX]

1. I'm not sure how Cacti pipes my script output into an rrd database file, specifically, assuming the rrd database is created okay, I can't see how to associate the "hop1" "hop2", etc. lables the script generates into Output Fields in my Data Input Method(or whatever it is you associate these with). Does Cacti just assign the first to the first, second to second, etc?

Cacti will be able to find your output data as long as your output string is predictable (yours looks correct). When you create a new "Data Input Method", you must define each output field (hop 1-10). They must be named exactly how they are shown in your script's output. This is how Cacti keeps track of this data.

2. I can't find documentation about this, but it looks like:

One script/query output item --> One Data Input Method Output Field --> One Data Template Data Source Item --> one or more Graph Template Items

I hope this makes sense I'm trying to show associative relationships as I understand them, so is this correct?

This is correct. You will have to create at least one data source associated with your new data input method. This data source will have 10 items, one for each output field.

3.There are various other features in different screens that I don't understand and can't find in the manual, such as "Heartbeat" and "Step" in the Data Template. I'm curious, and would be happy to leave them on the default settings, but I would like to know that the default isn't going to undermine my efforts to make the graph I'm describing.[/list]Again, any help you can offer is appreciated, and thanks for a great graphing tool.

All of these settings that you name come straight from RRDTool. I could explain them, but I think the RRDTool manual does a better job.

-Ian

[sayasif]

Very good Script.. and will look even better with area stacked graph in RRD !!!

Please move it to scripts and Add ons so other ppl can also use it

thanx

[cignul9]

Thanks for the RRDTool manual link. I know this sounds silly, but it only recently occurred to me that Cacti is really just a program that simplifies using RRDTool for graphing by providing a php interface. It also provides a very good list of commonly-used data gathering methods and presents them well with some pre-configured graph templates. So before you get Cacti, it's wise to know how RRDTool works first. I've spent some time learning about RRDTool and how it works and have a better understanding of things now.

I'm using version 0.81 of Cacti, and my quest for the traceroute graph continues. I've run into what I think are bugs in Cacti that I'd like to report here. The first is easy to work around, and the last two don't seem to have any work around solution at all. Here they are:

1. The Graph Template page changes the Template name to match a new Graph Item Input name when you create a new Graph Item Input. I can consistently duplicate this problem on any Graph Template I choose and similar problems may occur in other pages.

Steps to duplicate: Click Graph Templates and then click any template in the list. Click Add for Graph Item Inputs. Type a name and click Save (all you need to do is type the name, but you can play with the other stuff if you want). When you are returned to the main Graph Template page, notice the name of the template is now the name of the Graph Item Input you just created. Click the red X button next to the input you created and notice the Graph Template name changes back to the original.

2. Again in the Graph Template, I can't successfully create a Graph Template Item of type GPRINT. This problem may exist for other types, and I know I have been able to successfully use AREA.

Steps to duplicate: select any Graph Template and Add a Graph Template Item. Select an appropriate Data Source and choose type GPRINT. Give it a name and choose Save. Associate the Graph Template with a Polling Host and check the graph. You should see that the graph is broken. Check the Source link and you'll see the GPRINT line. It looks something like this for the GPRINT line:

Code: Select all

GPRINT::AVERAGE:"Gateway%8.2lf %s"

The back-to-back colons should have a letter between them that references a data source. If you check the Graph Template in Cacti, it shows you've done that, but it doesn't seem to reflect that when it does the rrdtool graph command.

3. This one is the doozy and I have to explain a lot so here we go. As you know, with the traceroute graph I need to create a Data Input Method with as many Output Fields as I have hops that I want to trace. I have also created a Data Template with corresponding Data Source Items for each hop. Next I created a Graph Template with Graph Items of type AREA for each hop (I haven't done a legend for the graph yet, just the lines. I can't do the legend because of bug 2 and because I just want to get the graph working and then I'll pretty it up later). For kicks and giggles I added two Graph Item Inputs for each of my ten AREA Graph Items (20 total). One for the line Color and one for the Data Source for each hop. (Note: Graph Item 1 is Hop10, Graph Item 2 is Hop9, etc. because the farther hops will have higher latency and I want the lower latency graph lines to be drawn last and therefore visible in front of the higher ones so you can see which hops are jacking up the milliseconds.)

Okay, now I sicked my graph template on a polling host and waited a while for it to gather data. The polling host I was tracing was four hops away from the Cacti box. The first two hops were next to zero milliseconds so they shouldn't have shown up on the graph, the third hop jumped to 200 ms and the last hop was averaging around ten milliseconds more than the third. Now the graph isn't labeled yet, but Hop1 was yellow, Hop2 was a sunny orange color, Hop3 was redish-orange color, and Hop4 was red.

The graph I was looking at was showing Hop1 (yellow) hovering at 200 ms with Hop4 (red) just above that. I looked at the source for the graph and noticed the data source for Hop1 was in the third position and the data source for Hop3 was in the first position. I blamed myself. I went to the Console and clicked Data Sources and fished out the data source for the graph (I have a big list now). I double-checked each of the Hops and made sure they were pointing to the right Internal Data Source Name. Then I double-checked the Data Template. Each of the Data Source Items was pointing to the right Output Field. I rechecked all the Output fields in the Data Input Method. Everything looked right. I did a dump of the rrd database for the graph and saw that Hop3's data was indeed being dumped into the field for Hop1. I decided that my script had a problem and redesigned it:

Code: Select all

#!/usr/bin/perl

@trace = `traceroute -m 9 -w 2 -q 1 -n $ARGV[0] | sed -l0 -e '1{d}' -e 's/\*/0\.0\.0\.0\ \ 0\.000\ ms/g'`;
$hopcount = @trace;
for ($i = 0; $i < $hopcount; $i ++){
        $hop = $i + 1;
        @trace[$i] =~ s/( |)(.*[0-9])(.* )(.*[0-9]\..*[0-9]\..*[0-9]\..*[0-9])(.* )(.*[0-9])(\.[0-9][0-9][0-9] ms)//;
        print "hop0$hop:$6 ";
}

I omitted the tenth hop (in case the collection mechanism was getting confused between hop1 and hop10) and renamed my labels hop01, hop02, etc. instead of hop1, hop2, etc. I realize the perl code is messy; I'll fix that after I get it working.

I scrapped my Data Input Method, Data Template, and Graph Template and started over. This time I was careful to make sure I made no mistakes with naming and to create them in order. When I was done I again picked a Polling Host (this time a different one, but again with only four hops). I noticed a clue when I created the Data Source this time. Because the Graph Template has Graph Item Inputs of type Data Source for each hop I was able to see the order in which the data sources should be listed (from Hop09 to Hop01, remember I need them listed in reverse order to graph properly) but instead I saw two sets of numbers transposed again so they appeared from the top down in this order: 8, 9, 7, 6, 5, 3, 4, 2, 1

I attempted to correct this by manually inputting the correct order for the four hops that were out of place. When I waited for the data to collect and viewed the graph I was disappointed again. Here's the source for the graph:

Code: Select all

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--title="Host 158.58.14.189 - Hop Latency" \
--base=1000 \
--height=150 \
--width=800 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label="Milliseconds" \
DEF:a="/srv/www/htdocs/rra/host_158_58_14_189_hop09_103.rrd":hop08:AVERAGE \
DEF:b="/srv/www/htdocs/rra/host_158_58_14_189_hop09_103.rrd":hop09:AVERAGE \
DEF:c="/srv/www/htdocs/rra/host_158_58_14_189_hop09_103.rrd":hop07:AVERAGE \
DEF:d="/srv/www/htdocs/rra/host_158_58_14_189_hop09_103.rrd":hop06:AVERAGE \
DEF:e="/srv/www/htdocs/rra/host_158_58_14_189_hop09_103.rrd":hop05:AVERAGE \
DEF:f="/srv/www/htdocs/rra/host_158_58_14_189_hop09_103.rrd":hop03:AVERAGE \
DEF:g="/srv/www/htdocs/rra/host_158_58_14_189_hop09_103.rrd":hop04:AVERAGE \
DEF:h="/srv/www/htdocs/rra/host_158_58_14_189_hop09_103.rrd":hop02:AVERAGE \
DEF:i="/srv/www/htdocs/rra/host_158_58_14_189_hop09_103.rrd":hop01:AVERAGE \
AREA:a#55009D:""  \
AREA:b#005199:""  \
AREA:c#00A348:""  \
AREA:d#00FF00:""  \
AREA:e#FFF200:""  \
AREA:f#FFAB00:""  \
AREA:g#FF7D00:""  \
AREA:h#FF5700:""  \
AREA:i#FF0000:""

The DEF lines show that the two sets of data sources are still being transposed. I've tested the traceroute.pl script to see if it's listing the hops out of order and it's not. I'm not sure what else to try at this point. I hope you can fix this or help me out because I'm really stuck.

Thanks,

[cignul9]

I've made some changes to the perl script. I added documentation and decided to make it so it supports all 30 hops that a standard traceroute in Unix will do. It shouldn't hurt the data collection and graphing if you only want to use the first few hops.

Just remember that to use all 30 you'd have to make a data input method with 30 output fields, a data template with 30 data source items, and a graph template with 30 graph items not to mention all the graph item inputs you might want to include for each hop!

Code: Select all

#!/usr/bin/perl
#Copywri...aw hell, just take it.
#Run a traceroute and clean up the output with sed so it can be parsed by the loop,
#i.e.  delete the first line, replace asterisks (timeouts) with "0.0.0.0  0.000 ms",
#and replace the space at the beginning of the first nine lines with "0"

@trace = `traceroute -w 2 -q 1 -n $ARGV[0] | sed -l0 -e '1{d}' -e 's/\*/0\.0\.0\.0\ \ 0\.000\ ms/g' -e 's/^\ /0/g'`;

#Number of hops in the trace

$hopcount = @trace;

#Run a loop for each hop (or line) in the trace

for ($i = 0; $i < $hopcount; $i ++){

#Parse each line in the trace output and assign variables to each identifiable segment

        @trace[$i] =~ s/(.*[0-9])(.* )(.*[0-9]\..*[0-9]\..*[0-9]\..*[0-9])(.* )(.*[0-9])(\.[0-9][0-9][0-9] ms)//;

#Concatentate and print "hop" + the first segment in the line (line number) + the fifth
#segment in the line (number of milliseconds)

        print "hop$1:$5 ";
}

[cignul9]

I found I could edit the Host-specific instance of the graph in Graph Management and change the hop order for bug 3. I'm gathering data now to see how it looks...looks okay so far. That's one down.

I also found I could add a GPRINT Input Item of type Data Source to my Graph Template. WHen I did that I checked Graph Management again and it did show that no Data Source was associated with the GPRINT, so I added the right one. That's two down. It would be nice if when you add the GPRINT it would automagically take care of the existing graphs the way the other types do.

[raX]

Thanks for the RRDTool manual link. I know this sounds silly, but it only recently occurred to me that Cacti is really just a program that simplifies using RRDTool for graphing by providing a php interface. It also provides a very good list of commonly-used data gathering methods and presents them well with some pre-configured graph templates. So before you get Cacti, it's wise to know how RRDTool works first. I've spent some time learning about RRDTool and how it works and have a better understanding of things now.

Cacti was definitely built to be a compliment to RRDTool's graph/data storage features. The 0.6.x releases provided RRDTool's features in PHP-based web interface while 0.8.x adds even more to that.

1. The Graph Template page changes the Template name to match a new Graph Item Input name when you create a new Graph Item Input. I can consistently duplicate this problem on any Graph Template I choose and similar problems may occur in other pages.

This was a strange code ordering bug on my part. It has been fixed in my local copy/CVS and will make it into version 0.8.2.

2. Again in the Graph Template, I can't successfully create a Graph Template Item of type GPRINT. This problem may exist for other types, and I know I have been able to successfully use AREA.

The problem is not that you cannot add a new GPRINT item to your graph template, but that the selected data source does not propagate out to attached graphs. I am well aware of this problem, but haven't decided how I am going to fix it yet. The data sources that you choose for graph templates are different from the data sources list for the graph management page and therefore I cannot simply push out this data.

3. This one is the doozy and I have to explain a lot so here we go. As you know, with the traceroute graph I need to create a Data Input Method with as many Output Fields as I have hops that I want to trace. I have also created a Data Template with corresponding Data Source Items for each hop. Next I created a Graph Template with Graph Items of type AREA for each hop (I haven't done a legend for the graph yet, just the lines. I can't do the legend because of bug 2 and because I just want to get the graph working and then I'll pretty it up later). For kicks and giggles I added two Graph Item Inputs for each of my ten AREA Graph Items (20 total). One for the line Color and one for the Data Source for each hop. (Note: Graph Item 1 is Hop10, Graph Item 2 is Hop9, etc. because the farther hops will have higher latency and I want the lower latency graph lines to be drawn last and therefore visible in front of the higher ones so you can see which hops are jacking up the milliseconds.)

Ok, I read through this and I think I understand the problem that you are having. One problem is that I am not sure where to even start debugging this in my code. I guess I need to experience the problem first hand to get a feel for where the problem lies. I will be out of town until Monday, but I would be more than happy to debug further on your installation. I understand that providing a "steps to reproduce" for this particular problem would be difficult.

-Ian

[cignul9]

I've got some more information about the problem I called Bug 3. I've decided to expand the traceroute capability to graph all 30 possible hops. After I got it all setup I added the template to one of my many Polling Hosts. There's a screen called "Create 1 Graph from Host Template" that you get when you create the graph for the first time for a given Polling Host. In that dialog I was seeing the problem where the Hops were out of order, but I realized that they were being listed twice: once in the Graph Template section and once in the Data Template section. The Graph Template list was in order; the Data Template list was wack (in the Data Template I checked the box "Use Per-Data Source Value" for the Internal Data Source Name for each of my Data Source Items). I was manually correcting the order of the Internal Data Sources in the Data Template section of that screen, which I think caused all the problems I was having.

If I create a new Data Source without changing any of those Internal Data Source names, the graphs are drawn properly. I went back to my Data Template and cleared that check box for the Internal Data Source name for all the Data Source Items so they wouldn't appear at all on the Graph Template part of that dialog. I'd originally done that because I mistakenly guessed that changing the Internal Data Source Name value was how you re-associated a Data Source with a graph line in the graph. Anyway, looking at the Data Input Method I created for my traceroute graph, I see a list of thirty output fields (one for each hop), but they're not listed in order. I'm wondering if that has something to do with the out-of-order list I was seeing when I create a new Data Source with my Graph Template. They're numbered as followed from the top down even though I entered them in order from one to thirty: hop10, hop09, hop08, hop07, hop06, hop05, hop04, hop03, hop02, hop01, hop11, hop12...hop30. This dialog has a column called Field Order which appears to be some vestigial organ from an earlier version of the program, or perhaps a work in progress and not yet implemented. I'm wondering whether tinkering with the order of the list, which I currently cannot do without perhaps hacking the SQL database would have fixed the problem.

Here are some additional observations, minor bugs and feature requests:

• 1. How do you alter the time domain for the Daily graphs? I would think this is something you change on the Graph Template dialog or maybe the Available RAA's screen under the corresponding timeframe (Daily), but the explanation for the Step field in Cacti is ambiguous to me and doesn't seem to correspond to the --step switch for rrdgraph. There is a value for –start in the source for the rrdgraph command that indicates the graph for daily is 24 hours (86400 seconds). I would have expected to find that somewhere so I could change it if I wanted.
  
  2. Can we get a new interface for positioning objects in a list besides those up and down arrows? How about a field on each row that allows you to type or select a position number from a list? Or how about a check-box or radio button on each line with a pick list of options for the selected line(s) like “Insert New Item Above Here” or “Cut” and “Paste” and “Delete”, etc. Also, how about a left/right arrow for hierarchical lists like the Graph Tree Items list?
  
  In my Graph Template, I’ve got a list of 125 Graph Template Items and moving them one position at a time is excruciating. Especially since updating the Graph Template with changes you make to the order doesn’t necessarily update your existing graphs. I began noticing problems with my existing graphs when I deleted some Graph Template Items and checked my graphs that were using it. I wasn’t able to fix the problems they were having without deleting and re-adding them. Oh well. Now I’ve realized that I’d like to re-arrange my Legend into two columns of fifteen hops and you can imagine how nightmarish it’s going to be with just those up and down links.
  
  3. On the Data Template screen can you redo the interface so that many tabs don't shoot the width of the dialog way out to the right? Perhaps you could make the tabs wrap or something. Also clicking Save after you create a new Data Source Item dumps you back at the list of Data Source Templates, which sucks if you're adding thirty Data Source Items. You have to re-select the Data Template you want every time you add an Item.