Automate aggregate

[matej_v]

Autom8 plugin currently does an excellent job of adding newly created graphs to Cacti trees. But I would like to also add new graphs Data sources to a predefined Aggregate graph, based on similar rules as one can define in Autom8.

Is there currently such a feature or is someone working on something similar? Based on Aggregate templates in latest Aggregate plugin such a thing should not be too hard.

[gandalf]

To be honest, I did not make it to the aggregate plugins. That feature was provided by others (and is still not documented, sigh). But yes, solving it should go that way.
R.

[matej_v]

Ok, here is what I'll try to do:

• Verify that the api_* functions in aggregate work and document them. If needed add more api functions.
• In autom8 detect if aggregate is also installed. If it is a new item is shown in Console side menu - "Aggregate rules".
• Aggregate rules are evaluated on graph creation - similar to Tree Rules.
• An aggregate rule uses an predefined aggregate template to create aggregate graphs and a graph template to filter new graphs for consideration.
• An aggregate rule resolves, based on data from new graph, to an aggregate graph. If this aggregate graph does not yet exist, it is created.
• To not complicate things too much an aggregate graph will be created even if it will contain only one graph.

Use case: I'm graphing ports for multiple customers. Each customer can have multiple ports - for those I want to automatically create an aggregate graph. A new customer graph is created and an aggregate rule is evaluated. The customer ID is extracted from ifAlias and is used to search for appropriate aggregate graph. If no match is found a new aggregate graph is created. If a match is found the new graph is added to aggregate graphs items. This way I do not need a new aggregate rule for each new customer - aggregate creation can be completely automated.

Would this make sense?

Which autom8 sources from SVN should I base my changes on? I'd need this to work in 0.8.7i.

[gandalf]

You should use aggregate/tags/v0.75 in that case.
But we had several huge patches applied to aggregate/trunk. So I definitively need new code in patch format
R.

[matej_v]

I've posted the patch to autom8 in bugtracker under #2171.

A short example to present aggregate rules:

I have traffic graphs of each of my customer's interfaces. A customer my have more than one interface. I wish to display all traffic of each customer on one graph. But I am too lazy to create aggregate graphs by hand Enter autom8 Aggregate Rules.

First I create an Aggregate Template for traffic graphs named "SIX - Member Total Traffic". (Aggregate Template creation is out of scope of this example.)

Then I define an Aggregate Rule that uses "SIX - Member Total Traffic" template. Image linked to avoid stretching thread.

This rules defines the title pattern of the aggregate graphs that will be created: "SIX - Member Total Traffic - ${1}". Any ${N} parts will be substituted from matches in rule items (explained later). Traffic graphs that share the same Aggregate Title will be aggregated together.

I want to filter and match data on ifAlias value so I also select the data query "SNMP - Interface Statistics".

Now to select eligible graphs. I only want to consider graphs that belong to hosts wind names beginning with "ltest". For each of these matching graphs their Data Query data is also retrieved (ifName, ifSpeed, ifAlias...).

ifAlias of eligible graphs will need to match the regex:

Code: Select all

/.*SIX.*\((AS[0-9]+)\).*/

If you look carefully the part "AS[0-9]+" is in parenthesis. As this is a first match it will replace ${1} variable in Aggregate Graph Title Pattern. Matches are numbered left to right in the same item and top to bottom among multiple items.

As you can see a total of 5 aggregate graphs could be created from 12 matching graphs. Any new graphs created will be evaluated against this rule. It the graph matches it will be placed in appropriate aggregate.

For member1 (AS1111) interfaces FA0/33, Fa0/38 and Fa0/42 would be aggregated on the same aggregate graph:

autom8_as1111_aggregate.png (32.71 KiB) Viewed 6887 times

Hope anybody else finds this patch useful.

[seansullivan]

What's the status of these additions? I am working on automating as much as our cacti stuff as possible. Like when a new machine is kickstarted it will hit cacti to add itself with the appropriate host template and all the necessary graphs then into the tree.

One thing I wasn't sure how to solve is automating the recreation of existing agg graphs when a new server is added to a pool and this looks like it might be a good start. Judging by the bug report I don't think this stuff has been merged into the main branch yet. Is that correct, and if so is there any estimate on when that might happen?

[matej_v]

No, my patches have not been merged (yet). Until now there was no feedback from other users, so the devs probably haven't considered a priority. If you want to try this patch, here is how to get started:

Get the latest Aggregate plugin from SVN trunk, Autom8 plugin can be 0.35, I think. Then apply patches in bug reports: #2171 for autom8 and #2169 for aggregate.

Let me know if you need any help. Feedback for bugs or improvements is also welcome.

[jeanm]

I am interested in this feature also.

Using automate v0.35 only fails the patch on 3 trunks at setup.php:

Code: Select all

Hunk #1 succeeded at 47.
Hunk #2 succeeded at 61.
Hunk #3 succeeded at 87.
Hunk #4 FAILED at 105.
Hunk #5 FAILED at 165.
Hunk #6 succeeded at 173 (offset -8 lines).
Hunk #7 FAILED at 201.
Hunk #8 succeeded at 219 (offset -8 lines).
Hunk #9 succeeded at 312 (offset -3 lines).
Hunk #10 succeeded at 454 (offset -3 lines).
Hunk #11 succeeded at 710 (offset -3 lines).
Hunk #12 succeeded at 838 (offset -3 lines).
Hunk #13 succeeded at 1005 (offset -3 lines).
Hunk #14 succeeded at 1044 (offset -3 lines).

105 is related to not finding any "api_plugin_register_realm" call in plugin_autom8_install function. 165 is related to realm stuff again.

I've tried to use the version from the SVN (1.0? rev 2087) but I got more hunks fails... (8 in 2 files instead of 3 hunks in 1 file).

I've a question, considering that it changes the setup.php the autom8 plugin must be reinstalled right?

I'll see what I can do meanwhile...

JF. Mousinho

[matej_v]

Sorry, I said to use autom8 0.35, but really should have written 0.36. Though 0.36 is not published on docs.cacti.net for some reason. In any case even 0.36 it too old for my patch. I really recommend you start with main branch. I just tried to use my patch on main@rev 2076 and it applies cleanly.

So do:

Code: Select all

cd <cacti_root>/plugins/
mv autom8 <somewhere_save>
svn checkout svn://svn.cacti.net/cacti_plugins/autom8/branches/main/ autom8
patch -p1 < autom8-aggregate_rules-v2.patch

I did a version bump to 0.37 in the patch so I *think* Cacti will take care of upgrade procedure by itself. You should see autom8 0.37 under Plugin Management.

Please, let me know if this procedure works for you.

EDIT: Uploaded an updated patch to bug tracker.

[jeanm]

OK, the patch on autom8 went OK now. The one now failing is aggregate.

Using r2079 fails. Shall I use the r2069 you used in the patch?

Code: Select all

r2079 | gandalf | 2012-02-23 21:18:53 +0000 (Thu, 23 Feb 2012) | 1 line
create a main branch for 087 support

Perhaps the new 087 is coming out soon?

Thanks for your help.

JF. Mousinho

[matej_v]

I believe you tried to use branch main. Main is missing a bunch more patches for autom8 - aggregate integration to work. So please use aggregate trunk. Currently it is still at rev 2069 so the patch from bug #2169 will apply.

[jeanm]

OK. It seems to be working now. The problem is that I'm being able to do what I wanted, not sure if I'm following the docs correctly. It seems that autom8 docs refer mainly to the creation of aggregate graphs for a specific host. What I want to automate is the creation of an aggregate graph (totals only) that comes from several hosts. I can do this manually but I'm not sure if I can do it using autom8...

Any advice is welcome. Thanks.

[matej_v]

Ok, I'll try to explain:

• I assume you created an Aggregate Template for your aggregate graph already.
• Create an Aggregate Rule and choose an Aggregate Template from previous step.
• (optional) Select Data Query - this allows you to use variables from data query when determining which graph goes to which Aggregate Graph
• Create appropriate "Rule Items => Eligible Objects". This will filter which graphs are considered for aggregation. You could add two rules, that will select graphs from hostA OR hostB:

  Code: Select all

  Item     Sequence    Operation     Field                 Operator     Pattern
  Item#1   1                         host.description      matches      hostA
  Item#2   2           OR            host.description      matches      hostA
  

• Now you can "group" graphs selected in previous point into one or multiple Aggregate Graphs. You do this via "Rule Items => Create Aggregate Graph". These rules allow you to also filter by Data Query variables such as Interface Name, Interface Description... (You can't do this in the first step because the underlying SQL queries are quite slow with a large number of graphs).
  
  If you want just one Aggregate Graph from this rule, AND you filtered all the graphs you want with "Rule Items => Eligible Objects", you still need to and a dummy rule here, so graphs will show up when you click "Show Eligible Objects." Something like this should work:

  Code: Select all

  Item     Sequence      Field Name          Search Pattern
  Item#1   1             host.description    .*
  

  If you want only one Aggregate Graph from this Aggregate Rule, but you could not filter them exactly with "Rule Items => Eligible Objects", create one or more rules under "Rule Items => Create Aggregate Graph". Keep in mind that the Matching Pattern field is a Regular Expression. Keep tuning your rules until only desired graphs show up under "Matching Graphs".
  
  If you want to create multiple Aggregate Graphs from this Aggregate Rule you need to find a value that all the graphs that should be in one Aggregate Graph have in common. Most likely this will be some substring of ifAlias for traffic graphs. Example - I have graphs for the following interfaces:

  Code: Select all

  Host      ifName   ifAlias
  routerA   Gi1/1    custID:1234
  routerA   Gi1/2    custID:1234
  routerA   Gi1/3    custID:5678
  routerB   Gi1/1    custID:1234
  routerB   Gi1/2    custID:5678
  routerB   Gi1/3    custID:9999
  

  I want all the graphs that share the same custID to go in one Aggregate Graph. So in the end I'll have 3 Aggregate Graphs:

  Code: Select all

  Aggregate for custID:1234
          member graph: routerA   Gi1/1
          member graph: routerA   Gi1/2
          member graph: routerB   Gi1/1
  
  Aggregate for custID:5678
          member graph: routerA   Gi1/3
          member graph: routerB   Gi1/2
          member graph: routerB   Gi1/3
  
  Aggregate for custID:9999
          member graph: routerB   Gi1/3
  

  To do this create this kind of rule:

  Code: Select all

  Item    Sequence     Field Name             Search Pattern
  Item#1  1            data_query.ifAlias     custID:([0-9]+)
  

  And enter this into "Aggregate Graph Title Pattern":

  Code: Select all

  Aggregate for custID:${1}
  

  String ${1} will be replaced by the value matched in parenthesis in the rule. If you look into Matching Graphs, column Aggregate Graph, you'll see this. All the graphs that share the same value here, will be put in the same Aggregate.
• Finally go to "Graph Management", check all graphs to aggregate and select "Apply Autom8 Rules for Graph(s)". If all goes well, you should have your aggregate graphs created.

When you create any new graphs (manually or via Autom8 Graph Rules), Aggregate Rules are also evaluated. If a graph matches a rule it will be added to an appropriate Aggregate Graph.

I hope this explanation helped. If not, try to describe what you'd like to do in more detail and I'll tell you what to do.

[jeanm]

Thanks alot for your detailed explanation. The problem was the additional rule ".*" that I was not adding.

In fact autom8 seems to be working now for what I wanted but a new problem problem appeared... =/

Aggregate now has some trouble doing the totals, even if I try to do an aggregate manually. The aggregate graph config is attached. The debug output is:

Code: Select all

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=0 \
--end=0 \
--title='Aggregate' \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label='Clients' \
--slope-mode \
--font TITLE:10: \
--font AXIS:7: \
--font LEGEND:7: \
--font UNIT:7: \
DEF:a="/usr/share/cacti/site/rra/107/412.rrd":eduroamguest:AVERAGE \
DEF:b="/usr/share/cacti/site/rra/107/412.rrd":eduroam:AVERAGE \
DEF:c="/usr/share/cacti/site/rra/106/408.rrd":eduroamguest:AVERAGE \
DEF:d="/usr/share/cacti/site/rra/106/408.rrd":eduroam:AVERAGE \
DEF:e="/usr/share/cacti/site/rra/108/416.rrd":eduroamguest:AVERAGE \
DEF:f="/usr/share/cacti/site/rra/108/416.rrd":eduroam:AVERAGE \
DEF:g="/usr/share/cacti/site/rra/44/176.rrd":eduroamguest:AVERAGE \
DEF:h="/usr/share/cacti/site/rra/44/176.rrd":eduroam:AVERAGE \
CDEF:cdefa=a,0,* \
CDEF:cdefb=b,0,* \
CDEF:cdefd=c,0,* \
CDEF:cdefe=d,0,* \
CDEF:cdefg=e,0,* \
CDEF:cdefh=f,0,* \
CDEF:cdefj=g,0,* \
CDEF:cdefba=h,0,* \
CDEF:cdefbc= \
CDEF:cdefbg= \
CDEF:cdefca= \
AREA:cdefa#00BF47FF:""  \
AREA:cdefb#00FF00FF:"":STACK \
LINE1:cdefb#00694AFF:""  \
AREA:cdefd#00BF47FF:""  \
AREA:cdefe#00FF00FF:"":STACK \
LINE1:cdefe#00694AFF:""  \
AREA:cdefg#00BF47FF:""  \
AREA:cdefh#00FF00FF:"":STACK \
LINE1:cdefh#00694AFF:""  \
AREA:cdefj#00BF47FF:""  \
AREA:cdefba#00FF00FF:"":STACK \
LINE1:cdefba#00694AFF:""  \
AREA:cdefbc#00BF47FF:"eduroam-guest"  \
GPRINT:cdefbc:LAST:"Curr\:%8.0lf"  \
GPRINT:cdefbc:AVERAGE:"Avg\:%8.0lf"  \
GPRINT:cdefbc:MAX:"Max\:%8.0lf\n"  \
AREA:cdefbg#00FF00FF:"eduroam":STACK \
GPRINT:cdefbg:LAST:"      Curr\:%8.0lf"  \
GPRINT:cdefbg:AVERAGE:"Avg\:%8.0lf"  \
GPRINT:cdefbg:MAX:"Max\:%8.0lf\n"  \
LINE1:cdefca#00694AFF:"Total users"  \
GPRINT:cdefca:LAST:"  Curr\:%8.0lf"  \
GPRINT:cdefca:AVERAGE:"Avg\:%8.0lf"  \
GPRINT:cdefca:MAX:"Max\:%8.0lf\n" 

As you can see, the aggregate_simular defs are empty. Can you try use this type of graphs (aggregate with totals only) and see if it works for you?

Thanks for your help.

[matej_v]

The problem was the additional rule ".*" that I was not adding.

Yeah, I'll have to fix this.

As you can see, the aggregate_simular defs are empty. Can you try use this type of graphs (aggregate with totals only) and see if it works for you?

I assume you have "Total Users" items in your original graphs and its values are calculated in the fly (not stored in RRD). It is possible Aggregate does not support this yet. Please check Skip and uncheck Total for last four items if this fixes it.

I might take a look at the code, but I fear it is quite a complex problem.