graphing multiple oid's and combinging from several machines

[tomasperez]

What I am wanting to do is this:
I have 10 machines running an app, I have setup a script on each box to do some data gathering. What I want to do is graph each box seperately, and then on one graph, add up all the values across the boxes and show it on one. So basically, each server has oid of x.x.x.x, and returns value 10 on each server, I want to add up the values on each server and show it on one graph as a combined value, instead of 10 seperate values. What is the best way to accomplish this?

[gandalf]

First, please create a graph for each single target as usual.
Then, please goto Graph Management and Add. Fill in the data on the first page and Create. Now find the Section named Graph Items and add each item, referring to one of your target hosts, as usual.
Reinhard

[tomasperez]

I may have been unclear and left out a few details:
Here is an example:
For each box, I want to graph 3 OID's:
1 - number of messages in
2 - number of messages out
3 - number of failed messages
I want all 3 of these on 1 graph.

Then I want one graph that combines all the boxes and graphs the above variables as a "product" view. So basically each of the 10 servers is one piece of the product. The oid is the same on each server, so would a template be best to use?
I want all 3 of the above on 1 product graph.

[gandalf]

I may have been unclear and left out a few details:
Here is an example:
For each box, I want to graph 3 OID's:
1 - number of messages in
2 - number of messages out
3 - number of failed messages
I want all 3 of these on 1 graph.

Please make 3 copies of the SNMP - Generic OID Template. Replace the OID found within with one of the three above. Then, please create a new Graph Template. As Graph Items, you may use the three just created data sources. Now apply this new Graph template to each of those 10 hosts at the Associated Graph Templates section. Select Create Graphs for this Host[{b], check the according row and create. This procedure is describe in the second to last link of my signature (among others).

Then I want one graph that combines all the boxes and graphs the above variables as a "product" view. So basically each of the 10 servers is one piece of the product. The oid is the same on each server, so would a template be best to use?
I want all 3 of the above on 1 product graph.

Having build the above Graphs for each host, please proceed as given in my last post. You may choose, whether to graph each of the 10 items as an AREA/STACK (this will give nice graphs) or whether to graph only the sum of all 10 items.
Reinhard

[tomasperez]

Ok, I have done like you stated, it appears to be collecting data at this point, but the graphs are not showing up. Debug info:


/usr/local/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title="Product Server Stats" \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label="" \
--slope-mode \
DEF:a="/usr/local/www/apache22/data/cacti/rra/hostname_snmp_oid_2149.rrd":snmp_oid:AVERAGE \
DEF:b="/usr/local/www/apache22/data/cacti/rra/hostname_snmp_oid_2150.rrd":snmp_oid:AVERAGE \
DEF:c="/usr/local/www/apache22/data/cacti/rra/hostname_snmp_oid_2151.rrd":snmp_oid:AVERAGE \
:a#FF0000:"Incoming Messages":STACK \
:b#EACC00:"Outgoing Messages":STACK \
:c#96E78A:"Incoming SMTP Messages":STACK
RRDTool Says:


ERROR: Could not make sense out of ':a#FF0000:Incoming Messages:STACK'


I saw you talking about this issue in another post, but I never saw a resolution. Is this a bug in the code, or an issue with the way I created the graph template?

[gandalf]

There's sth wrong with your Graph Template. This should look like

Code: Select all

DEF:a="/var/www/html/workspace/branch/rra/localhost_mem_buffers_3.rrd":mem_buffers:AVERAGE \
DEF:b="/var/www/html/workspace/branch/rra/localhost_mem_swap_4.rrd":mem_swap:AVERAGE \
AREA:a#FF4105:"Free"  \
AREA:b#FFC73B:"Swap":STACK \

Perhaps you've forgottten to define the FIRST item to be graphed as an AREA instead of a STACK (all subsequent must be STACKed, then)
Reinhard

[tomasperez]

Awesome, that fixed it. It make sense that the first has to be area, didnt really think about it. Thanks for your quick responses, still working on that named from another post that you were helping me out on. Thanks for your help man.

[tomasperez]

I may have spoken too soon. I see the graphs now, but there is nothing in the graph. I have verified via snmpget that the values are increasing on the remote system, but the values are not graphing. What troubleshooting can I do to see why nothing is graphing?

[gandalf]

First, make sure you've got the correct data source type. Please post the outcome of an snmpwalk against those OIDs.
If there's sth like COUNTER or GAUGE, excatly that is needed, respectively. For STRINGS, it's not that obvious. COUNTER are "always increasing" numbers, whereas GAUGEs respresent the "actual" numeric values of sth.
If you feel the need to change the Data Template(s), please don't forget to "rrdtool tune" the already existing rrd files.
Else, change Settings->Logging Level to DEBUG for ONE poller's run at have a look at log/cacti.log for the OIDs you've polled
Reinhard

[tomasperez]

04/14/2006 03:11:01 PM - CMDPHP: Poller[0] Host[20] DS[2191] SNMP: v2: hostname.mydomain.com, dsname: snmp_oid, oid: .1
.3.6.1.4.1.17548.100.2.1.4, output: 1066
04/14/2006 03:16:09 PM - CMDPHP: Poller[0] Host[20] DS[2191] SNMP: v2: hostname.mydomain.com, dsname: snmp_oid, oid: .1
.3.6.1.4.1.17548.100.2.1.4, output: 1073
04/14/2006 03:21:01 PM - CMDPHP: Poller[0] Host[20] DS[2191] SNMP: v2: hostname.mydomain.com, dsname: snmp_oid, oid: .1
.3.6.1.4.1.17548.100.2.1.4, output: 1073
04/14/2006 03:35:57 PM - CMDPHP: Poller[0] Host[20] DS[2191] SNMP: v2: hostname.mydomain.com, dsname: snmp_oid, oid: .1
.3.6.1.4.1.17548.100.2.1.4, output: 1083

[gandalf]

First, make sure you've got the correct data source type. Please post the outcome of an snmpwalk against those OIDs.
If there's sth like COUNTER or GAUGE, excatly that is needed, respectively. For STRINGS, it's not that obvious. COUNTER are "always increasing" numbers, whereas GAUGEs respresent the "actual" numeric values of sth.
If you feel the need to change the Data Template(s), please don't forget to "rrdtool tune" the already existing rrd files.

Sorry, but the results of the above are missing ...
Reinhard

[tomasperez]

Ok, I did some playing around with it an have figured out how to create a graph template and applied that to the 10 servers. So at this point, I am ready to graph all the machines on 1 graph. Another words, for oid 1.1.1.1, I get 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, for each machine respectively. I want to put those values as a "product" graph. Do I need to create another graph template?

[gandalf]

You may extend the existing one. Add a new Graph Item, data source set to NONE. Choose a COLOR and a LINEx. Apply the CDEF Total All Data Sources. You may even add a LEGEND for this (shortcut: Add a New Item and select Graph Item Type = LEGEND)
Reinhard

[tomasperez]

I saw that, but is not what I am looking for. What I have graphing now is all 10 servers on individual graphs. Now I want an overall view on another seperate graph that totals all the oid categories seperate:
ex.
OID Meaning Value
1 # of errors 10
2 # of successes 1000
3 # of rejects 50


So each server is being graphed individually now for those items. Now the 10 servers graphing those 3 items, I want a "product" graph that will add up oid 1, 2, and 3 on each server and the "product" graph will have 3 lines that will represent a total of the 10 servers.

[gandalf]

Ok, lets go this way. The easiest task would be adding a new Graph Template for errors, rejects and successes seperately. Putting all this in one graph will require some huge CDEFs, but will work of course.
First: Add all "basic" data sources to the new Graph Template
Then, add a new Graph Item, data source = NONE, LINEx, CDEF= Total all Data Sources
This will automatically add up all data sources previously selected

Other way:
Add first "basic" data source as an AREA, COLOR e.g. light yellow
Add next "basic" data source as a STACK, COLOR: darker yellow
Perform second step for each subsequent data source, changing COLOR slightly from step to step. This will make up some nice graphs. And stacking will automagically produce the SUM (but this will not be shown in the legend: You may add the "Total All Data Sources" Item as a LINEx, COLOR black and add a LEGEND, if you want to)

Reinhard