Overall explanation of architecture

[trasko]

I feel I just don't get it. I have been a sys admin for 8 years now, I'm an experienced coder/scripter (B.S. in Computer Science no less) and for the life of me I just don't get Cacti. In the past 4 years or so I've installed and fiddled with Cacti 4 times. The last time I did it about a year ago, I ended up writing all my own scripts to get data into .rrd files and other scripts to rsync them around and even more scripts to periodically generate the graphs and html to view them... I did all that because I couldn't figure out Cacti, despite knowing that Cacti is *exactly* what I need.

So here I am again. A new job one year after my last attempt. Install was mostly painless, but I'm running into all the same old problems. I *know* cacti is what I need. I want to help our main IT guy by throwing up some data for our networking equipment and perhaps an environmental sensor or two. Nothing complex.

Is it me or is Cacti very difficult to understand? I feel like I get the ideas in theory, but in practice nothing I do seems to function as expected. I understand the concept of a "device" and have gotten as far as to have all that lined up. I don't mean to say I can't get graphs to show up but organizationally nothing makes sense to me. I lack the conceptual framework to put everything in. So after creating devices, what do I do?

Create a data source? A data query? I'm going to have several of the same type of switch, so I figure I should make a data template...? If I associate graph templates or data queries with a host template, what does that REALLY do? Graphs don't actually get created automatically for me, right? So what do these templates do?

I apologize for the first-time poster rant you are getting. I know there are too many questions to really answer... How about this: Can someone point me to a guide that explains what all the terminology means? Or a guide that shows how to setup a fairly complex environment the "proper" way -- using templates. One that uses built-in templates, custom-made templates, no templates and explains the reasons why one would choose one over the other in different cases. Highly technical is fine. I need explanations more than I need a step-by-step "click here" tutorial.

Thank you for any help. It's a thorn in my side that I know cacti is exactly what I need but I have been unable to get it work for me after all this time.

[nebj00la]

Don't get too frustrated. There's plenty of documentation out there for all the different functions of Cacti.

One particular section that will most likely get things going is "Creating the Graphs", which can be found in the official documentation.

Documentation: http://www.cacti.net/downloads/docs/html/

Creating the Graphs: http://www.cacti.net/downloads/docs/htm ... raphs.html

[trasko]

I appreciate the reply. To put it succinctly, I feel the official documentation sucks. I hardly understand what each of the different things are (graph templates, data queries, data templates, etc) much less how to use them and how they interact with one another. Cacti is free -- I get that and it seems like it works for a lot of people, it just doesn't make sense to me.

How about this: Could someone provide me a translation between what you do in the GUI and what happens on the backend? I have created many .rrd files by hand. I have created/generated graphs by hand(script). I have used SNMP Perl modules to pull SNMP data after looking up OIDs for what I need on the net. I feel like I should understand cacti better given that I know all the underlying systems.

Here is a series of questions whose answers would help me understand cacti better. They relate to the backend vs. the GUI:

1a) When I use rrdtool to create an .rrd file, I provide the expected min/max values, the data type, etc and the different granularity levels with which it should be stored. I assume this translates in to "data source", correct?

1b) When using cacti, does one normally add in data sources by hand manually or does one usually create or use an existing template of some sort?

2) To get the data to put into a .rrd file, I usually used scripts but SNMP is good, too.
2a) Is this what "data queries" are?

2b) What is the difference between a "data query" and "data input methods"?

2c) Data templates? What are these? Are they templating "data queries", "data input methods", "data sources" or all or some of the above? I don't understand this abstraction.

3) When I create rrd files manually, I know when the .rrd is created and I know when I run the update cmd to add data to the .rrd. I understand that the "poller" runs every 5 minutes (by default), but I don't understand when the .rrd files are made. I can imagine if you make a data source manually, then that makes a .rrd file, but I get the feeling templates are usually used instead of manually creating data sources. If I am right, when does the .rrd file get created? When you apply a device/data/graph template to a device?

An alternative to the above questions might be to just provide an explanation of a cacti object/function in terms of the backend. Some things I'd like to know are:

Device:
--------
There is no translation, I assume. People usually just name their .rrd files or put them in directories to organize them. Nothing about .rrd files cares where the data comes from. (Correct me if I'm wrong)

Host Template:
------------------
????? Are we using "device" and "host" interchangeably? I can see how abstracting away similar characteristics is helpful, but it seems most of the common "likeness" is due to similar ways to get data or graph data -- not anything inherit to the host itself. The relationship to this template to a device to all the other different templates is one of the things that really puts my mind into a spin.

Data Source:
-------------
????

Data Query:
-------------
???? The script/SNMP get cmd that gets the date you want to graph???

Data Input Methods:
-----------------------
????

Graph / Graph Templates / Graph Management:
------------------------------
I think I understand these fine. The graph is the "rrdtool graph" command that is run to generate the graph you want to see. The graph template just serves to abstract away options that get repeated a lot and allows for mass changes of similar graphs by just changing the template. There can be an unlimited number of graphs for a single host or even a single set of data -- you can change colors, change the scale, make a different chart type to create a new and different graph. I like the ability to organize them into a tree. This is part of cacti that I understand and know is very useful.

Anything Else?
-----------------
??? I'm sure I've left stuff off. How does all of this inter-relate with each other and with the manual process of creating rrd files??


Again, thanks for your help. I feel like this is the time I'm going to be successful with Cacti. I just need to get past some conceptual barrier and then I'll be good from there on out.

[irfan199]

Here is a series of questions whose answers would help me understand cacti better. They relate to the backend vs. the GUI:

1a) When I use rrdtool to create an .rrd file, I provide the expected min/max values, the data type, etc and the different granularity levels with which it should be stored. I assume this translates in to "data source", correct?

Correct

1b) When using cacti, does one normally add in data sources by hand manually or does one usually create or use an existing template of some sort?

Usually using data template. It defines your RRD's structure.

2) To get the data to put into a .rrd file, I usually used scripts but SNMP is good, too.

It is related in the data template in section ''Data Input Method". This defines, how the rrd filled with data. Using what kind of input method. Is it using SNMP or through script (perl, etc) or Script Server (PHP). It refers to "Collection Method" (Data Input Method) template. CMIIW.

2a) Is this what "data queries" are?

I think Data queries is like an SQL query but retreive with SNMP or other method to get the list of ex: interface in a router.

2b) What is the difference between a "data query" and "data input methods"?

Data query is like what i have explained above. Data input method is how to retrieve the value that will be stored in rrd. It's like the snmpget command with oid, ex: snmpget -c public -v2c 127.0.0.1 .1.2.3.4.5
when you execute it you'll get some value and this value is stored on RRD.

2c) Data templates? What are these? Are they templating "data queries", "data input methods", "data sources" or all or some of the above? I don't understand this abstraction.

It's just define the RRD structure and what the data input methods that the template used to get data.

3) When I create rrd files manually, I know when the .rrd is created and I know when I run the update cmd to add data to the .rrd. I understand that the "poller" runs every 5 minutes (by default), but I don't understand when the .rrd files are made. I can imagine if you make a data source manually, then that makes a .rrd file, but I get the feeling templates are usually used instead of manually creating data sources. If I am right, when does the .rrd file get created? When you apply a device/data/graph template to a device?

The rrd is created in the next polling cycle. Usually when you create graphs, it automatically create data sources. This will become informations stored in database. When the poller runs, it reads the rrd structure and creates the rrd according to it.

The graph template defines how you view the RRD. So, you can have many kinds of graph from single data source (RRD). Ex: you can change the color or the graph type (line, area, stack) without recreating the RRD's.

An alternative to the above questions might be to just provide an explanation of a cacti object/function in terms of the backend. Some things I'd like to know are:

Device:
--------
There is no translation, I assume. People usually just name their .rrd files or put them in directories to organize them. Nothing about .rrd files cares where the data comes from. (Correct me if I'm wrong)

Device can be a router, a switch, a server or any device that has snmp enabled or can get the data through script or any means.

Host Template:
------------------
????? Are we using "device" and "host" interchangeably? I can see how abstracting away similar characteristics is helpful, but it seems most of the common "likeness" is due to similar ways to get data or graph data -- not anything inherit to the host itself. The relationship to this template to a device to all the other different templates is one of the things that really puts my mind into a spin.

It's just a template that groups many graph template specific to the device type. For example, If I want to monitor a Cisco Router, I'd pick Cisco Router host template when I create the Device. It will add graph templates that specific to Cisco Router like, CPU monitor, SNMP Interface data query, etc.

Data Source:
-------------
????

Data Query:
-------------
???? The script/SNMP get cmd that gets the date you want to graph???

Data Input Methods:
-----------------------
????

Explained above.

Graph / Graph Templates / Graph Management:
------------------------------
I think I understand these fine. The graph is the "rrdtool graph" command that is run to generate the graph you want to see. The graph template just serves to abstract away options that get repeated a lot and allows for mass changes of similar graphs by just changing the template. There can be an unlimited number of graphs for a single host or even a single set of data -- you can change colors, change the scale, make a different chart type to create a new and different graph. I like the ability to organize them into a tree. This is part of cacti that I understand and know is very useful.

Correct

Anything Else?
-----------------
??? I'm sure I've left stuff off. How does all of this inter-relate with each other and with the manual process of creating rrd files??

Again, thanks for your help. I feel like this is the time I'm going to be successful with Cacti. I just need to get past some conceptual barrier and then I'll be good from there on out.

Here are the steps that I usually do to monitor device in cacti:
1. Create device. (Pick correct Host Template or just use Generic SNMP enabled Host.
2. Edit the device and click "Create Graphs for This Host"
3. Pick the graph template I want to create. Click Create button.
4. Fill in the parameters that is asked. Click Create.
5. Cacti will save the information in database.
6. Wait for cacti poller to run at least 2 times. The RRD will be created and the graph can be viewed.

FYI, I'm a newbie too. So, it's only my understanding. Forgive me if it turns incorrect.

Regards.

[gandalf]

Some remarks of "The Root Cause of all Documentation Problems"

I felt just like you, 3 years ago. That's why I started documenting.

To be honest, after some time you don not understand "newbie" problems well enough. That's not intentionally, but I can't overcome it.

My offer: Please refer to exact documentation links and tell us what is hard to understand or missing and I'll add it. Or simply change it on your own, it's a wiki (create a new userid at http://bugs.cacti.net, we use the same userids on both sites).

I tried to create a chapter on "How it works in principle". But this may still be hard to understand. What worries you exactly?

Reinhard

[ahpook]

For what it's worth, I'm in exactly the same place as the original poster. I consider myself pretty well-versed in the generalities of computing and the specifics of graphing interesting stuff with RRD, yet cacti gives me fits trying to understand how the different abstract items are supposed to fit together when I'm starting a new input type from scratch.

Here is a very concrete example that is currently driving me insane. I write a script that returns some custom info, for new GAUGE style data sources. The script queries a host and outputs:

Code: Select all

fs1_max:100 fs1_avg:50 fs2_max:200 fs2_avg:150 total_max:300 total_avg:100

There could be up to forty or so "fs??_max" and avg items. I'd like for each of the filesystems and the totals to be graphed separately, with average and max on the same graph.

Can (and SHOULD) I use an indexed Data Query to match the repeating patterns of fs[1-10] and total?

If not, do I have to list out each of the possible output fields separately in the Data Input Method screen for the script? (Is that scriptable in any way?)

When I create Data Templates to map each of the output fields to a RRS DS, should I put them ALL as Data Source Items into one Data Template? One Data Template per fs and a separate one for Totals? Why?

Similarly, when I make Graph Templates for these, it seems like I need to make a different template for each end-product graph (for fs1, fs2, etc and for totals) which would seem to defeat the benefit of templating. Am I missing something?

If I *don't* make an indexing xml file by setting my script up as "Get Script Data (Indexed)" in the Data Query screen, should adding my Data Source template to a Device through the "Data Source List" screen cause the query to start being run for that device?

It seems like (though I can't find explicit reference to this) there needs to be a 1:1 relationship of instantiated queries to data templates, because according to the "Apply Graph Template to Host" section, adding a line to "Associated Graph Templates" on a device's screen will "create an entry to the poller_table to instruct cacti to gather data on each polling cycle". This suggests if I have two data templates for two different kinds of graphs, the query will be run twice -- is that true?

For each step I am puzzled over above, the docs seem to focus on screenshots with "go here click this" type instructions instead of describing how the code actually uses each of the kinds of items. The examples that are out there are either (a) too trivial to answer these kind of questions (one ds, one line, one graph), or (b) use SNMP which has a lot of builtin support that is not available to scripts. So it ends up being very hard to adapt the examples in a meaningful way.

To give praise where it is due, the "cacti to rrdtool translation" table is incredibly useful! That is a wonderful example of the right level of detail we need. (Sadly I didn't see it until I started writing this post, probably because google indexes the old static HTML docs above the wiki.)

Thanks for any help and thanks for letting me vent. In fact the process of writing this post helped clarify some things for me.

[Hyperlord]

Hey hey,

I saw a nice MS Visio-style diagramm somewhere that explains how the different components interact with each other. I printed it to A3 and "implemented" it "into" my desk

I'll see if I find that again!
Edit: Found it.

As for me: It took quite a time until I "got" Cacti - and a lot of debugging and troubleshooting, too.

brgds

[ahpook]

Yes, thanks, that graph is somewhat helpful. It could stand to be cleaned up and made part of the documentation. Isn't the fact that it appears to be the only design/architecture diagram of Cacti in the world somewhat troubling?

I found the source of at least some of my problems. I got to the point where I added my query to a particular Device by going to the Data Source List and adding my custom Data Template. This correctly caused a RRD to be created for the host and a poller entry added to run the script and populate the RRD with the output fields. But then I tried to use the "Create graphs for this host" and/or "Associated Graph Templates" links on the host page, which caused multiple Poller entries to be added for the host, each with its own separate RRD! (Why?)

What I had to do instead was go to the Graph Management tab and add a new graph from *that* screen, not the Device's screen. Then I was able to select my Graph Template and associate it with a host, which caused a form to come up that let me associate the Graph Item Field with a RRD DS from that device's RRD. Success! Mysterious, undocumented (AFAICT) and illogical, but success none-the-less.

I will attempt to redo the diagram and document this out for other people's edification.

Are the forum logins supposed to work on the docs wiki? Or is there a separate authentication system there?

[Howie]

(you need to register on bugs.cacti.net, and that login is used for the wiki)

Ages ago, I started writing a plugin called 'Explain', that added a new icon alongside each graph and tried to explain which graph templates, data templates, hosts etc were used in it's construction.

Part of the reason for writing that plugin is that I still don't understand fully how all these things fit together either, and solving problems like:
* Why does that title not reflect the template I thought it was using?
* Why is that graph hitting an invisible maximum, and which rrd file is the data coming from?
etc, require a good understanding of how the graph is made.

I returned to writing the plugin again yesterday, so I could understand why some of my graphs are misbehaving. Hopefully I'll have enough that it is useful to others soon - but it's also hard to *write* the plugin without the information!

[gandalf]

Are the forum logins supposed to work on the docs wiki? Or is there a separate authentication system there?

The id's from http://bugs.cacti.net are valid for http://docs.cacti.net.
Unfortunately, we have no integration (yet) with forums.
In case anybody knows how to integrate mantis and phpBB userids, please feel free to chime in
R.