Graphing Counts of Specific Processes

[mrballcb]

**NOTICE** The attached files do not work, only bit parts seem to work. Do not use them as is.

I'm a cacti newbie, and as I configure it to use existing snmp sources (currently graphed using mrtg), things are slowly starting to make sense. I am currently trying to mimic one graph we have in mrtg, but I'm having difficulty making it work in cacti. I'm trying to graph the number of a specific process. This is useful both for detecting non-standard load patterns and for forensics (for example, seeing a spike in apache or sendmail processes can indicate something wrong on the backend with dns or ldap, etc). In the snmpd.conf, it would be "proc sendmail" for the sendmail process.

Using the attached xml files, for a specific host I'm able to:
1. Add the Data Query "ucd/net - Get Monitored Processes" (net-snmp_proc.xml) which shows the list of processes that are in the remote host snmpd.conf.
2. Looking at the Data Query Verbose output, it has no issues obtaining the info. On a host with 6 processes being monitored, the status shows "Success [12 Items, 6 Rows]"
3. Click Create Graphs for this Host.
4. Select one or more of these process items that appear in the Processes Data Query box and click Create. I selected "sendmail" for the purposes of this example.
5. Cacti reports "Created graph: admin51 - Processes - sendmail". **BUT** no rrd files are actually created. This is the problem I'm trying to figure out, why doesn't it create the beginning rrd files?
6. Naturally the graphs don't update since there's no rrd file.

Can anybody see what's wrong with my configuration?

It won't let me attach my graph template, so if that's needed, I can provide it out of band.

Thanks!

[gandalf]

I'm sorry, but I can't reproduce. Can you tell me how you made prTable available to snmpd? I don't have that part of the HOST MIB.
R.

[mrballcb]

I'm sorry, but I can't reproduce. Can you tell me how you made prTable available to snmpd? I don't have that part of the HOST MIB.
R.

Hmmm, can I take that to mean that you imported these xml files and added the data query and created the graph and it actually created the rrd files? Or am I reading too much into the first sentence?

For your followup question, I assume that you're asking how snmp on my cacti host knows about prTable? I don't reference prTable in cacti by name anywhere, instead the root OIDs for each table are specified in the net-snmp_proc.xml file attached in the original post (which BTW, I modeled after the net-snmp_disk.xml that comes with cacti).

To answer your question directly, this is where prTable is defined in my net-snmp install (CentOS 5.5 box) :

Code: Select all

[root@status cacti]# grep ^prTable /usr/share/snmp/mibs/*
/usr/share/snmp/mibs/UCD-SNMP-MIB.txt:prTable OBJECT-TYPE
[root@status cacti]# rpm -qf /usr/share/snmp/mibs/UCD-SNMP-MIB.txt
net-snmp-5.3.2.2-9.el5_5.1.x86_64

If you're asking how snmp on the host I'm polling knows about prTable, it's the same:

Code: Select all

admin51 ~ # grep ^prTable /usr/share/snmp/mibs/*
/usr/share/snmp/mibs/UCD-SNMP-MIB.txt:prTable OBJECT-TYPE

Interestingly enough, my Ubuntu 10.10 box does not know about prTable nor any of the tables defined in UCD-SNMP-MIB. If you're using Ubuntu 10.10 maybe that's why it's unknown? I did an snmpwalk with debug output and I see it parsing the UCD-SNMP-MIB file but not finding any tables.

In the following capture, ignore security and access settings because for now I've got it open to any inside host while testing to make sure I'm not stepping on my own feet. Once I get the graph working I'll tighten it back up.

Here is the beginning of an snmpwalk (first 12 lines shown) of the prTable done from my cacti box:

Code: Select all

[root@status cacti]# snmpwalk -v1 -cCommunity admin51 prTable | head -n 12
UCD-SNMP-MIB::prIndex.1 = INTEGER: 1
UCD-SNMP-MIB::prIndex.2 = INTEGER: 2
UCD-SNMP-MIB::prIndex.3 = INTEGER: 3
UCD-SNMP-MIB::prIndex.4 = INTEGER: 4
UCD-SNMP-MIB::prIndex.5 = INTEGER: 5
UCD-SNMP-MIB::prIndex.6 = INTEGER: 6
UCD-SNMP-MIB::prNames.1 = STRING: cron
UCD-SNMP-MIB::prNames.2 = STRING: sendmail
UCD-SNMP-MIB::prNames.3 = STRING: slapd
UCD-SNMP-MIB::prNames.4 = STRING: slurpd
UCD-SNMP-MIB::prNames.5 = STRING: named
UCD-SNMP-MIB::prNames.6 = STRING: rbldnsd
[root@status cacti]# snmpwalk -v1 -cCommunity admin51 prTable -On | head -n 12
.1.3.6.1.4.1.2021.2.1.1.1 = INTEGER: 1
.1.3.6.1.4.1.2021.2.1.1.2 = INTEGER: 2
.1.3.6.1.4.1.2021.2.1.1.3 = INTEGER: 3
.1.3.6.1.4.1.2021.2.1.1.4 = INTEGER: 4
.1.3.6.1.4.1.2021.2.1.1.5 = INTEGER: 5
.1.3.6.1.4.1.2021.2.1.1.6 = INTEGER: 6
.1.3.6.1.4.1.2021.2.1.2.1 = STRING: cron
.1.3.6.1.4.1.2021.2.1.2.2 = STRING: sendmail
.1.3.6.1.4.1.2021.2.1.2.3 = STRING: slapd
.1.3.6.1.4.1.2021.2.1.2.4 = STRING: slurpd
.1.3.6.1.4.1.2021.2.1.2.5 = STRING: named
.1.3.6.1.4.1.2021.2.1.2.6 = STRING: rbldnsd

The actual field with counts is .1.3.6.1.4.1.2021.2.1.5.$INDEX.

Here is the snmp.conf from the "admin51" host that I am polling:

Code: Select all

rocommunity  Community
proc  cron
proc  sendmail
proc  slapd
proc  slurpd
proc  named
proc  rbldnsd
disk  /
disk  /disk1
load  12 12 12
syslocation  "El Segundo"
syscontact  spongadmin@localhost
# sysservices 0*1 + 0*2 + 1*4 + 1*8 + 1*64
sysservices 76

I hope this clarifies where I'm getting the data and what I'm trying to do with it. Please do not hesitate to ask for more info. I'm comfortable with snmp, but just learning cacti. I'm trying to be the guy requiring handholding.

I've pastebin'ed the graph template I mentioned earlier: http://pastebin.ca/2004852 (Maybe I used a reserved filename? The export process named it cacti_graph_template_ucdnet_-_processes.xml.)

Thanks Gandalf!

[mrballcb]

Using the attached xml files, for a specific host I'm able to:
1. Add the Data Query "ucd/net - Get Monitored Processes" (net-snmp_proc.xml) which shows the list of processes that are in the remote host snmpd.conf.
2. Looking at the Data Query Verbose output, it has no issues obtaining the info. On a host with 6 processes being monitored, the status shows "Success [12 Items, 6 Rows]"

If you look carefully at the index_order of the net-snmp_proc.xml file I attached, you can see that I had:

Code: Select all

<index_order>dskDevice:dskIndex</index_order>

which is left over from when I was converting net-snmp_disk.xml. I changed it to match the fields in that xml file:

Code: Select all

<index_order>prName:prIndex</index_order>

That fixed EVERYTHING! Yay!

There was also one fundamental misunderstanding on my part. I didn't realize that the rrd files didn't get created the instant I clicked on the create graph selection. It was a moment of clarity when I realized it was created when the poller ran.

Thank you for the feedback. I was oh-so-close, but having to struggle through it was a good learning experience.

Regards... Todd

[gandalf]

I was worrying about that entry, but did not expect it to fail the whole query. So I learnt as much as you
R.

[mrballcb]

I was worrying about that entry, but did not expect it to fail the whole query. So I learnt as much as you
R.

To be precise: the query which retrieved the snmp index of processes and names functioned correctly with those improper index_order, however when it came time to create graphs, it could not properly tie together the data sources and graph inputs.

There is one symptom that could possibly allow you to spot this in the future and look like a freaking genius: I clicked create graphs, then selected one of the processes to add. It said that it created the graph (which I now know means that it put info in the database for the poller to use, which will create the graph when it next runs). However, when the create graphs page reloaded, that process was not grayed out, it was still selectable. Knowing what I know now, the fact that item was still selectable was indicating an error.

Again, thanks for all your help. I'm gonna package the templates into a tarball and upload it to leave a functional template package for anybody who stumbles upon this.

I saw there is a wiki for user contributed templates. Should I create an account there and upload it? Or do I need to get sponsored by someone within the development team? Thus far I have some exim, sendmail, and these per-process-counting templates that are at least useful to me.

Regards... Todd

[gandalf]

I saw there is a wiki for user contributed templates. Should I create an account there and upload it?

Yes. It's the same userid that works with our mantis at http://bugs.cacti.net. Unfortunately, we can't share id's with phpBB (at least I don't know of such a feature; but I'm not the phpBB guy)
R.