Combining multple CDEF values from different data sources?

[brentonw]

I have created some graphs using multiple rrd files basically they take the values from 3+ devices for :

* reported CPU Load
and
* CPU cores

I have CDEF's working to calculate the total number of CPU cores.

I have a CDEF combining all the CPU load's reported in the graph.

All of which works fine in the one graph.

However, how can I use the resulting data from each CDEF and use in another (ie - divide value of CDEF for total load by value of CDEF for number of cpu cores reported for example).

My problem is that the graph creation has been scripted, and the menberships of devices are expected to change - so the entry of the graphs is not going to be stagnant - ie I can't just create a CDEF that will not allow for the altering of the data sources it is using as changes to the number of devices, cores, load reports etc will change the required CDEF, however I can easily add in each of the individual CDEF's for the data sources using standard definitions, it's the combining of the two data sources CDEF's that is causing me grief.

(Basically this is to group a set of servers and get the total load across the group as well as the total cpu cores available and then divide them to give a standard measure we can use across all groups).

Thanks,

Brenton

[gandalf]

I prehaps down't understand completely. But I see a chance, that http://docs.cacti.net/manual:087:6_refe ... ata_source may help, especially the "COUNT ..." stuff.
R.

[brentonw]

Thanks for that - I read through the manual and that page to get me started on this path...

What I have is:

CDEF1: Count of Similar Data Sources (Don't Include Duplicates) - associated with "Data Source 1"

CDEF2: Similar Data Sources (Don't Include Duplicates) - associated with 'Data Source 2"

[Both of the above work fine]

I then want to create a third CDEF:

CDEF3: CDEF1 / CDEF2


This works BUT cdef3 requires a data source association. However the association then flows to CDEF1 and CDEF2 calculations in CDEF3. So the resultant figure is not what I'm looking for.

What I want is:

CDEF3 = CDEF1 (use data source 1) / CDEF2 (using data source 2)


Does that make more sense?

The actual definitions are:

CDEF 1 =
1) SIMILAR_DATA_SOURCES_NODUPS
2) Custom String: 100
3) Operator: *

CDEF 2:
1) COUNT_SIMILAR_DS_NODUPS

CDEF 3:
1) CDEF1
2) CDEF2
3) Operator: /


I appreciate any assistance you can offer!

Thanks again

[gandalf]

Sounds fine. But I don't see the problem. Please post the list of graph template items along with the associated CDEFs
R.

[brentonw]

The graph has been created by direct interaction with the sql database.

The data sources associated with the graph are two types:

Data Source [HrProcessorLoad]
and
ucd/net - Load Average - 1 Minute

The result is to have a data source for multiple CPU cores per machine and a single Load average value. What I am doing with the grouped graph is providing an overall load value on multiple cluster groups - so the graph itself has on average - 3 x ucd load average data sources, and 8-12 cpu core load reports.

The cdef definitions are:

CDEF1:
Item #1 Special Data Source: SIMILAR_DATA_SOURCES_NODUPS
Item #2 Custom String: 100
Item #3 Operator: *


CDEF2:
Item #1 Special Data Source: COUNT_SIMILAR_DS_NODUPS


CDEF3:
Item #1 Another CDEF: Total Similar Data Source (no dups), Multiply by 100
Item #2 Another CDEF: CDEF - Count Similar Items - No Duplicates
Item #3 Operator: /


It works fine if I associate a data source (item_task_id) in the database to this graph.

OUTPUT FROM DEBUG OF GRAPH IS:

CDEF:cdefbh=TIME,1281685074,GT,e,e,UN,0,e,IF,IF,TIME,1281685074,GT,j,j,UN,0,j,IF,IF,TIME,1281685074,GT,bd,bd,UN,0,bd,IF,IF,+,+,100,*,TIME,1281685074,GT,1,e,UN,0,1,IF,IF,TIME,1281685074,GT,1,j,UN,0,1,IF,IF,TIME,1281685074,GT,1,bd,UN,0,1,IF,IF,+,+,/ \


The only issue is that it uses the single datasource for BOTH cdef's and hence gives me a result that is not what I am looking for.

However if no datasource is applied to the final cdef then the datasource has a null value and doesn't work (as in the CDEF's that it refers to have no data source associated with them so they don't work - thus the whole calculation fails:)

OUTPUT FROM GRAPH DEBUG:

CDEF:cdefbh=,100,*,,/ \


Hope this helps.

If there's a better way to do it then I'm all ears.

Thanks,

Brenton

[gandalf]

All this sounds reasonable. Assigning a datasource is indeed required to make all those SIMILAR_DATA_SOURCE magic happen.
As you do not have a graph template, please post the graph definition, best is to add the complete rrdtool graph statement as well
R.

[brentonw]

Hello again,

So - whilst it is not that helpful to you, I created a template through the console that I could use to try and help the process - it doesn't actually create the graph as it ends up but it will show the data sources I am using. .

However - once created I the template I applied to a device so the graph was in use then added all the appropriate items directly in SQL.

The sql that was added to the graph:

DATA SOURCE ADDITION:

insert into graph_templates_item values("","","0","1812","0","5487","69","FF","4","0","1","Processor Use","","","2","1");
insert into graph_templates_item values("","","0","1812","0","5488","69","FF","4","0","1","Processor Use","","","2","1");
insert into graph_templates_item values("","","0","1812","0","5489","69","FF","4","0","1","Processor Use","","","2","1");
insert into graph_templates_item values("","","0","1812","0","5490","69","FF","4","0","1","Processor Use","","","2","1");
insert into graph_templates_item values("","","0","1812","0","3800","15","FF","7","0","1","Load Average","","","2","1");
insert into graph_templates_item values("","","0","1812","0","5499","69","FF","4","0","1","Processor Use","","","2","1");
insert into graph_templates_item values("","","0","1812","0","5500","69","FF","4","0","1","Processor Use","","","2","1");
insert into graph_templates_item values("","","0","1812","0","5501","69","FF","4","0","1","Processor Use","","","2","1");
insert into graph_templates_item values("","","0","1812","0","5502","69","FF","4","0","1","Processor Use","","","2","1");
insert into graph_templates_item values("","","0","1812","0","3151","15","FF","7","0","1","Load Average","","","2","1");
insert into graph_templates_item values("","","0","1812","0","5517","69","FF","4","0","1","Processor Use","","","2","1");
insert into graph_templates_item values("","","0","1812","0","5518","69","FF","4","0","1","Processor Use","","","2","1");
insert into graph_templates_item values("","","0","1812","0","5519","69","FF","4","0","1","Processor Use","","","2","1");
insert into graph_templates_item values("","","0","1812","0","5520","69","FF","4","0","1","Processor Use","","","2","1");
insert into graph_templates_item values("","","0","1812","0","3305","15","FF","7","0","1","Load Average","","","2","1");

(this applies the same load and cpu use to the graph from three hosts).

CACLULATED FIELDS ADDITION:
insert into graph_templates_item values("","","0","1812","0","3305","44","FF","9","22","4","CDEF LOAD TOTAL*100","","","2","2");
insert into graph_templates_item values("","","0","1812","0","5487","44","FF","9","18","4","CDEF Num cores","","","2","2");
insert into graph_templates_item values("","","0","1812","0","3305","44","FF","9","23","4","Incorrect calculation","","","2","3");



THE RESULTING GRAPH STATEMENT:

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-14400 \
--end=-300 \
--title='Group Load - Template Design' \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label= \
--slope-mode \
--font TITLE:12:/usr/share/rrdtool/fonts/DejaVuSansMono-Roman.ttf \
--font AXIS:6:/usr/share/rrdtool/fonts/DejaVuSansMono-Roman.ttf \
--font LEGEND:8:/usr/share/rrdtool/fonts/DejaVuSansMono-Roman.ttf \
--font UNIT:6:/usr/share/rrdtool/fonts/DejaVuSansMono-Roman.ttf \
DEF:a="/nicky/cacti/rra/nttarthur_hrprocessorload_1879.rrd":HrProcessorLoad:AVERAGE \
DEF:b="/nicky/cacti/rra/nttursula_hrprocessorload_1898.rrd":HrProcessorLoad:AVERAGE \
DEF:c="/nicky/cacti/rra/nttrobin_hrprocessorload_1865.rrd":HrProcessorLoad:AVERAGE \
DEF:d="/nicky/cacti/rra/nttrobin_hrprocessorload_1868.rrd":HrProcessorLoad:AVERAGE \
DEF:e="/nicky/cacti/rra/nttarthur_hrprocessorload_1877.rrd":HrProcessorLoad:AVERAGE \
DEF:f="/nicky/cacti/rra/nttarthur_hrprocessorload_1878.rrd":HrProcessorLoad:AVERAGE \
DEF:g="/nicky/cacti/rra/nttrobin_load_1min_1267.rrd":load_1min:AVERAGE \
DEF:h="/nicky/cacti/rra/nttursula_load_1min_1081.rrd":load_1min:AVERAGE \
DEF:i="/nicky/cacti/rra/nttursula_hrprocessorload_1896.rrd":HrProcessorLoad:AVERAGE \
DEF:j="/nicky/cacti/rra/nttrobin_hrprocessorload_1867.rrd":HrProcessorLoad:AVERAGE \
DEF:ba="/nicky/cacti/rra/nttarthur_hrprocessorload_1880.rrd":HrProcessorLoad:AVERAGE \
DEF:bb="/nicky/cacti/rra/nttrobin_hrprocessorload_1866.rrd":HrProcessorLoad:AVERAGE \
DEF:bc="/nicky/cacti/rra/nttursula_hrprocessorload_1897.rrd":HrProcessorLoad:AVERAGE \
DEF:bd="/nicky/cacti/rra/nttarthur_load_1min_1030.rrd":load_1min:AVERAGE \
DEF:be="/nicky/cacti/rra/nttursula_hrprocessorload_1895.rrd":HrProcessorLoad:AVERAGE \
CDEF:cdefbf=TIME,1281921546,GT,g,g,UN,0,g,IF,IF,TIME,1281921546,GT,h,h,UN,0,h,IF,IF,TIME,1281921546,GT,bd,bd,UN,0,bd,IF,IF,+,+,100,* \
CDEF:cdefbg=TIME,1281921546,GT,1,a,UN,0,1,IF,IF,TIME,1281921546,GT,1,b,UN,0,1,IF,IF,TIME,1281921546,GT,1,c,UN,0,1,IF,IF,TIME,1281921546,GT,1,d,UN,0,1,IF,IF,TIME,1281921546,GT,1,e,UN,0,1,IF,IF,TIME,1281921546,GT,1,f,UN,0,1,IF,IF,TIME,1281921546,GT,1,i,UN,0,1,IF,IF,TIME,1281921546,GT,1,j,UN,0,1,IF,IF,TIME,1281921546,GT,1,ba,UN,0,1,IF,IF,TIME,1281921546,GT,1,bb,UN,0,1,IF,IF,TIME,1281921546,GT,1,bc,UN,0,1,IF,IF,TIME,1281921546,GT,1,be,UN,0,1,IF,IF,+,+,+,+,+,+,+,+,+,+,+ \
CDEF:cdefbh=TIME,1281921546,GT,g,g,UN,0,g,IF,IF,TIME,1281921546,GT,h,h,UN,0,h,IF,IF,TIME,1281921546,GT,bd,bd,UN,0,bd,IF,IF,+,+,100,*,TIME,1281921546,GT,1,g,UN,0,1,IF,IF,TIME,1281921546,GT,1,h,UN,0,1,IF,IF,TIME,1281921546,GT,1,bd,UN,0,1,IF,IF,+,+,/ \
LINE1:a#00004DFF:"Processor Use" \
LINE1:b#00004DFF:"Processor Use" \
LINE1:c#00004DFF:"Processor Use" \
LINE1:d#00004DFF:"Processor Use" \
LINE1:e#00004DFF:"Processor Use" \
LINE1:f#00004DFF:"Processor Use" \
AREA:g#EACC00FF:"Load Average" \
AREA:h#EACC00FF:"Load Average" \
LINE1:i#00004DFF:"Processor Use" \
LINE1:j#00004DFF:"Processor Use" \
LINE1:ba#00004DFF:"Processor Use" \
LINE1:bb#00004DFF:"Processor Use" \
LINE1:bc#00004DFF:"Processor Use" \
AREA:bd#EACC00FF:"Load Average" \
LINE1:be#00004DFF:"Processor Use" \
GPRINT:cdefbf:LAST:"CDEF LOAD TOTAL*100%8.2lf %s" \
GPRINT:cdefbg:LAST:"CDEF Num cores%8.2lf %s" \
GPRINT:cdefbh:LAST:"Incorrect calculation%8.2lf %s"


As you can see on the attached export of the graph. The calculated field is not using the correct data - it is using the load average - for the entire calculation.... the data used is 357 / 3 (all cdef's using load data) where I'm looking to get: 357 / 12 (load / cpu core instances).


Thanks,

Brenton

[gandalf]

This is not of much use. The graph template has nothing in common with the graph itself. And from the graph statement I can only guess how the CDEFs look like and to which graph items they are applied.
At least I need the _exact_ definition of ALL CDEFs used by your graph and the info to which exact graph item they are applied to help
R.

[brentonw]

CDEF 1 (CDEF: Total Similar Data Source (no dups), Multiply by 100)
1) SIMILAR_DATA_SOURCES_NODUPS
2) Custom String: 100
3) Operator: *

CDEF 2: (CDEF - Count Similar Items - No Duplicates)
1) COUNT_SIMILAR_DS_NODUPS

CDEF 3:
Item #1 Another CDEF: Total Similar Data Source (no dups), Multiply by 100
Item #2 Another CDEF: CDEF - Count Similar Items - No Duplicates Move Down
Item #3 Operator: /

Is how the CDEF's show in the console

The SQL insert relating to CDEF:

CDEF1:
insert into graph_templates_item values("","","0","1812","0","3305","44","FF","9","22","4","CDEF LOAD TOTAL*100","","","2","2");

CDEF2:
insert into graph_templates_item values("","","0","1812","0","5487","44","FF","9","18","4","CDEF Num cores","","","2","2");

CDEF3
insert into graph_templates_item values("","","0","1812","0","3305","44","FF","9","23","4","Incorrect calculation","","","2","3");

The task_item_id for CDEF 1 and 3 refers to the ucd/net load average.
The task_item for CDEF 2 refers to HrProcessorLoad

I think it might be easier if I simply create another data source and combine the figures in that step of the process - so the graph works with a single type of data - which seems to be the sticking point. If I make the data source to query a device and have the device normalise the load before reporting then that will result in the same figures that I'm trying to achieve. It adds another "monitor" and graph to the system - but it would appear at the moment that mixing and matching sources is not easy when you want to calculate on multiple data sources and types.

Thanks,

Brenton

[gandalf]

At least it's not easy the current way. I'm not prepared to help much on non-templated graphs because it's hard to reproduce.
And yes, that task does not count among the easiest exercises in cacti
R.

[brentonw]

Completely understand.

Thanks heaps for taking a look anyway. I spent a little time last night putting together the required script and it looks like it's coming together well - basically uses snmp query anyway from a perl script and manipulates the output to be appropriate for cacti... so this will be the way I move forward.

Again - appreciate the time and everything that you put into it.

Regards,

Brenton