cpu graphs for solaris 10 box

[ade]

I am new to cacti and here is the issue I have:

I am adding a solaris 10 box to cacti to graph CPU, memory usage and established connections. However, I am not able to get the CPU and memory graphs. I only get the established connections graph. The cacti server is ver. 5.0.9 and the solaris 10 box has net-snmp ver. 5.0.9 installed on it that came with the OS. SNMP seems to be working fine since I am able to do snmpget from the cacti server to the solaris 10 box.

[cacti_server]$ snmpget -v 1 -c public sol_10 sysUpTime.0
SNMPv2-MIB::sysUpTime.0 = Timeticks: (261330381) 30 days, 5:55:03.81

what do I need to do to get the CPU and memory graphs?

Thanks in advance.

[bbice]

Are you sure it's net-snmp that is running and not the Solaris snmp agent? If you do ps -ef |grep snmp, do you see the net-snmp snmpd daemon or the Solaris one?

I don't have the MIB OID for net-snmp's Raw CPU stats memorized (sorry - grin) but what do you get back if you do snmpget or snmpwalk on those MIB OIDs? (you can get the right OIDs to use by looking at the data templates for ucd/net snmp cpu stats)

Lessee... The only other thing I can think of that might be wrong is that net-snmp may be configured to limit what branches of the MIB tree can be read. The configuration can be quite complicated now allowing you to tell net-snmp that a given community string has read access to only certain portions of the MIB tree, read/write to other portions, and no access to still other portions. Add settings specific to which IP a query comes from and it can get still more complicated.

As I recall, you can comment out a lot of the fancy authentication rules in snmpd.conf and just replace them (at least for testing) with something simple like:
rocommunity public
(or whatever community string you're using -- you ARE using something other than just "public", right? - wink)

Brent

[ade]

snmpd is running on the solaris 10 server and it is net-snmp:

sol_10> ps -ef| grep snmpd
root 876 1 0 Nov 28 ? 0:42 /usr/sfw/sbin/snmpd
sol_10> /usr/sfw/sbin/snmpd -v
NET-SNMP version: 5.0.9
Web: http://www.net-snmp.org/
Email: net-snmp-coders@lists.sourceforge.net

Also, when I do snmpget from the cacti server, I get a response

$ snmpget -v 1 -c public sol_10 sysUpTime.0
SNMPv2-MIB::sysUpTime.0 = Timeticks: (261330381) 30 days, 5:55:03.81

But when I do this:

$ snmpget -v 1 -c public sol_10 memory
Error in packet
Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: UCD-SNMP-MIB::memory

$ snmpget -v 1 -c public sol_10 cpu
Error in packet
Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: UCD-SNMP-MIB::ssCpuRawInterrupt

Does that show you where my pbm lies? Thanks.

[bbice]

sAlso, when I do snmpget from the cacti server, I get a response

$ snmpget -v 1 -c public sol_10 sysUpTime.0
SNMPv2-MIB::sysUpTime.0 = Timeticks: (261330381) 30 days, 5:55:03.81

But when I do this:

$ snmpget -v 1 -c public sol_10 memory
Error in packet
Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: UCD-SNMP-MIB::memory

$ snmpget -v 1 -c public sol_10 cpu
Error in packet
Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: UCD-SNMP-MIB::ssCpuRawInterrupt

Ok. It could be a permissions problem -- net-snmp may be configured to allow anyone to query mib2 (sysUpTime is part of the mib2 branch) but only allow localhost to query the ucd-snmp private branch of the mib tree (which is where all the ssCpuRaw stuff is). It might also be that your snmpget command is failing because your snmp client can't figure out what numeric mib oid "ssCpuRawInterrupt" is.

So... When in doubt, use the numeric OID. Try this:
snmpget -v 1 -c public sol_10 .1.3.6.1.4.1.2021.11.53.0
or
snmpwalk -v 1 -c public sol_10 .1.3.6.1.4.1.2021.11
(note the leading period before the long numeric oid - it matters). BTW, you really shouldn't use underscores in hostnames either -- as I recall, the DNS RFCs don't allow for underscores, even though some DNS servers let you use them.

Ok, back to the problem... So if you get replies from the two above snmp client commands, then the cacti server can send queries and get replies on stuff and it's not a problem with your snmp daemon config. If you still get snmp errors, then take a close look at your snmpd.conf file -- my guess would then be that the permissions on snmpd.conf are set wrong.

The simplest snmpd.conf sorta setup would be to comment out all the lines about com2sec, group, view, and access and instead have this line:
rocommunity public (which would mean allow any machine using the public community string to query any part of the mib tree).

Brent

[ade]

Brent,

I get replies on the commands you suggested I run:

cacti_server# snmpget -v 1 -c public sol_10 .1.3.6.1.4.1.2021.11.53.0
UCD-SNMP-MIB::ssCpuRawIdle.0 = Counter32: 173491630

cacti_server# snmpwalk -v 1 -c public sol_10 .1.3.6.1.4.1.2021.11
UCD-SNMP-MIB::ssIndex.0 = INTEGER: 1
UCD-SNMP-MIB::ssErrorName.0 = STRING: systemStats
UCD-SNMP-MIB::ssSwapIn.0 = INTEGER: 0
UCD-SNMP-MIB::ssSwapOut.0 = INTEGER: 0
UCD-SNMP-MIB::ssIOSent.0 = INTEGER: 0
UCD-SNMP-MIB::ssIOReceive.0 = INTEGER: 0
UCD-SNMP-MIB::ssSysInterrupts.0 = INTEGER: 892
UCD-SNMP-MIB::ssSysContext.0 = INTEGER: 742
UCD-SNMP-MIB::ssCpuUser.0 = INTEGER: 9
UCD-SNMP-MIB::ssCpuSystem.0 = INTEGER: 2
UCD-SNMP-MIB::ssCpuIdle.0 = INTEGER: 89
UCD-SNMP-MIB::ssCpuRawUser.0 = Counter32: 133409225
UCD-SNMP-MIB::ssCpuRawSystem.0 = Counter32: 7182310
UCD-SNMP-MIB::ssCpuRawIdle.0 = Counter32: 173495076
UCD-SNMP-MIB::ssCpuRawWait.0 = Counter32: 0
UCD-SNMP-MIB::ssCpuRawKernel.0 = Counter32: 7182310
UCD-SNMP-MIB::ssIORawSent.0 = Counter32: 705917
UCD-SNMP-MIB::ssIORawReceived.0 = Counter32: 4694
UCD-SNMP-MIB::ssRawInterrupts.0 = Counter32: 1543913307
UCD-SNMP-MIB::ssRawContexts.0 = Counter32: 1877325644

But still no memory and cpu graphs.
Thanks

[bbice]

Ok, that confirms that snmp from the cacti machine is working fine, and I assume when you created the host in cacti that you used the same "public" community string. So...

One thing I've seen is that the CPU graphs by default assume a maximum value you'll ever get is 100%. On Solaris, I'd see 200% for a 2 processor machine, 400% for a 4 processor machine, etc. So when cacti queries it and gets a number bigger than what the data template says is the max, it ignores the value as invalid.

A quick way to test this supposition is to edit the data sources for the cpu data. Select the "DataSources" link, then pull down "Select a host" and select your solaris machine, then in the search field to the right of the pull-down enter "cpu" (without the quotes) and hit enter.
That ought to result in the data sources for your cpu graph for your solaris machine. Open each one, and set the maximum value to something appropriate for your machine (I typically use something like Num_procs*100+10 -- 210 for a 2 processor machine, 410 for a 4 processor machine, etc). Do this for every cpu-related data source for the solaris machine, then rename or remove the RRD files for these same data sources. That'll force cacti to recreate 'em with the new maximums you just specified.

If that makes the cpu graphs start working for solaris, then you could alter the data templates so that they ignore the fixed default of 100 and have a "per-data-source maximum". Then when you create a new cpu graph, cacti will prompt you for the maximums for the cpu data sources and you can enter in whatever is appropriate for whatever machine/s you're creating graphs for.

Another tool I've found useful for troubleshooting stuff like this is rrdtool. For instance, you can see what maximum is specified in the RRD file by doing:
rrdtool dump some-file.rrd |more

The RRD files are dumped as XML files that are pretty easy to read 'n figure out. The maximums for each datasource in the RRD file are listed in the first page or two. For instance:
rrdtool dump overlord_cpu_system_303.rrd |more
<!-- Round Robin Database Dump --><rrd> <version> 0003 </version>
<step> 300 </step> <!-- Seconds -->
<lastupdate> 1167936602 </lastupdate> <!-- 2007-01-04 10:50:02 PST -->

<ds>
<name> cpu_system </name>
<type> COUNTER </type>
<minimal_heartbeat> 600 </minimal_heartbeat>
<min> 0.0000000000e+00 </min>
<max> 8.2000000000e+02 </max>

<!-- PDP Status -->
<last_ds> 40621267 </last_ds>
<value> 3.8886666667e+01 </value>
<unknown_sec> 0 </unknown_sec>
</ds>
[...]
So you can see that for the maximum value for the cpu_system datasource in this rrd file I've got 820 set (overlord is an 8 processor sgi altix box). You can also see what the last counter it got was (the last_ds number). Further down in the output would be all the values for the data source that had been captured.

I only mention this because when making my own graphs, I've sometimes had broken graphs because one data source was messed up and rrdtool can be useful to see which RRDs are getting data and which aren't (as well as what data is in there).

[ade]

Brent,

Thank you again. First of all I dont see where you change the 'max' value on cacti. When I click on the data source, cpu1 - idle, all I see is the data template (cpu1-idle), host and the and the data source path. Where do i change the max?


I ran the rrd tool and what I get is:

<rrd>
<version> 0001 </version>
<step> 300 </step> <!-- Seconds -->
<lastupdate> 1167951602 </lastupdate> <!-- 2007-01-04 18:00:02 EST -->

<ds>
<name> cpu1_idle </name>
<type> GAUGE </type>
<minimal_heartbeat> 600 </minimal_heartbeat>
<min> 0.0000000000e+00 </min>
<max> NaN </max>

<!-- PDP Status -->
<last_ds> UNKN </last_ds>
<value> 0.0000000000e+00 </value>
<unknown_sec> 2 </unknown_sec>
</ds>

<!-- Round Robin Archives -->
<rra>
<cf> AVERAGE </cf>
<pdp_per_row> 1 </pdp_per_row> <!-- 300 seconds -->
<xff> 5.0000000000e-01 </xff>

<cdp_prep>
<ds><value> NaN </value> <unknown_datapoints> 0 </unknown_datapoints></ds>
</cdp_prep>
<database>

'Nan' is what is on the field for the max value. That is also what I see under the graphs 'Nan'. Thank you for your time on this.

[bbice]

Brent,

Thank you again. First of all I dont see where you change the 'max' value on cacti. When I click on the data source, cpu1 - idle, all I see is the data template (cpu1-idle), host and the and the data source path. Where do i change the max?

(smacks forehead) My mistake. Before you can see the max value on the data sources, you have to edit the data templates and check the box for "per data-source value' next to the maximum.

click on "Data Templates" then in the search field enter "cpu" and you should see all the ucd/net cpu data templates. For each one you'd find the Maximum and right beneath it on the left should be the checkbox to make the maximum a per data-source value. THEN you should be able to edit the individual data sources, set the maximums, then remove or rename the RRD file so cacti will recreate it.

I ran the rrd tool and what I get is:
(snip)
<max> NaN </max>

'Nan' is what is on the field for the max value. That is also what I see under the graphs 'Nan'. Thank you for your time on this.

Hmm. That NaN does seem odd. Hopefully, when we get your maximums sorted out it'll do the right thing. Then you can do the same thing with the memory graphs -- use snmpget/snmpwalk to confirm the numeric OIDs in the memory data templates work, and if they do, use rrdtool to dump the memory RRD files and see what the min/max values are set to and compare that with what you're seeing from snmpwalk and snmpget. I think the memory data templates are all gauges instead of counters. So if snmpget or snmpwalk is showing values between the min/max values shown in rrdtool, they should be ok. If the min/max on those are set to NaN also, then.... Check the min/max values on the templates and make sure they make sense, then remove the memory RRD files and see if they get recreated the same way.

[ade]

Brent,

Yup..that was the problem. It took me a while because I had to re-do the whole built again on another test box, re-produce the problem and try again. Now I have all my graphs. I still have some issue with it but I will play with it till I get all working.Thank you very much for your help.

-ade

[nahun]

I had problems with CPU graphs and other with Solaris 10, but using this template everything worked just fine. Go to the list of scripts and templates and search for Solaris. You'll get a few.