Linux: Next Generation CPU Graph - ANY number of CPU Cores

[mikaelf]

Hi!

This is the Next Generation Linux CPU Graph:

Next Generation CPU Graph!

cacti_cpu_graph_fridh.png (47.83 KiB) Viewed 24428 times

As a bonus you get this I/O graph as well:

Simple added bonus - an system I/O graph

cacti_systemstats_io.png (38.14 KiB) Viewed 24428 times

Compare this to the original:

Original Cacti graph

cacti_cpu_graph_original.png (29.09 KiB) Viewed 24428 times

Data query: (install in resource/snmp_queries)

systemstats.xml

This version uses a data query.

(3.21 KiB) Downloaded 3415 times

Templates: (Import in Cacti)

cacti_data_query_fridh_-_ucdnet_-_systemstats.xml

systemstats template and dependencies

(61.52 KiB) Downloaded 3029 times

The host above is a 4 core server, performing a 16 thread mysqldump import.

Q: How did you get the core estimation as well as the graph levelled at 100%?
A: Some CDEF magic... I spent the major part of a good night getting this to work and I almost lost my sanity in the process but I made it... I was very determined and very sick of the standard Cacti graph over the past few years.

Q: Seriously, no need to specify or choose between different graph templates depending on how many cores a server have?
A: NO!!!! It's almost like magic! But if you try to figure the CDEF out you might go bananas.

No documentation, no nothing... atleast not yet. But I sure know I'm alot happier with these than the standard cacti CPU graph which hasn't received much love for the past few years.

INSTALLATION
1. copy systemstats.xml* to resource/snmp_queries/systemstats.xml
2. import cacti_data_query_fridh_-_ucdnet_-_systemstats.xml*
3. Assign data query to a host or a host template.
4. Create new graphs.

(All needed files are attached to this post).


The templates should work on all hosts, no matter how many cpus.
To actually get decent graphs with the existing templates you had to create individual templates that either made it clear how many CPUs each machine had, or create individual graph templates for 2, 4, 8 and 16 core machines with CDEFs dividing all values by #of CPUs.

The graph should peak at 100% no matter how many logical CPUs.
Example: Previous graphs peak at 800% for an 8-core, and you would not know if 100% on a graph is a 1-core server at its peak or an 8-core server at a mere 12.5% CPU usage.

Present as much information as possible in the graph, without cluttering it.
This means, include User, System, Wait etc.

The number of logical CPUs should be deductible from viewing the graph.

So far, I think I met all these goals, and I hope this post can be updated to contain a final working version eventually, but right now there's a problem with repeatable behaviour on creating graphs in different cacti installations. The workaround is: If you have issues, you can fix the CDEF once and then all graphs in that particular Cacti installation will work.



BUGS
Basically, there seems to be an issue (i.e. functional difference) in the few Cacti installations I've tried.
On Cacti installation 1 the graphs ended up with 7 DEFs. (only the AVERAGE one)
On Cacti installation 2 the graphs ended up with 14 DEFs. (AVERAGE, MAX)
On Cacti installation 3 the graphs ended up with 28 DEFs. (AVERAGE, LAST, MIN, MAX)

What this means is that the CDEF will fail to work properly, and you have to adjust the graph template before adding it to hosts.

THE PROBLEM IN MORE DETAIL:
I have a CDEF that gives me the total CPU Usage as follows:
cdef=a,b,c,d,e,f,+,+,+,+,+,ALL_DATA_SOURCES_NODUPS,100,/,/

Obviously this relies on DEF a, b, c, d, e, f being the correct ones in the EXACT correct place, which they're not in these cases...

The original Version 0.8.7e Cacti where I exported the templates; graph debug shows the following DEFs:

Code: Select all

DEF:a="/usr/local/share/cacti-0.8.7e/rra/web22_sscpurawsystem_260.rrd":ssCpuRawSystem:AVERAGE \
DEF:b="/usr/local/share/cacti-0.8.7e/rra/web22_sscpurawsystem_260.rrd":ssCpuRawUser:AVERAGE \
DEF:c="/usr/local/share/cacti-0.8.7e/rra/web22_sscpurawsystem_260.rrd":ssCpuRawNice:AVERAGE \
DEF:d="/usr/local/share/cacti-0.8.7e/rra/web22_sscpurawsystem_260.rrd":ssCpuRawInterrupt:AVERAGE \
DEF:e="/usr/local/share/cacti-0.8.7e/rra/web22_sscpurawsystem_260.rrd":ssCpuRawWait:AVERAGE \
DEF:f="/usr/local/share/cacti-0.8.7e/rra/web22_sscpurawsystem_260.rrd":ssCpuRawSoftIRQ:AVERAGE \
DEF:g="/usr/local/share/cacti-0.8.7e/rra/web22_sscpurawsystem_260.rrd":ssCpuRawIdle:AVERAGE \

Then, I imported to a Version 0.8.7b Cacti (after regexping the hash_NNXXXX version value in the exported xml), and the graph debug shows the following:

Code: Select all

DEF:a="/var/lib/cacti/rra/snmplocal_sscpurawnice_145.rrd":ssCpuRawSystem:AVERAGE \
DEF:b="/var/lib/cacti/rra/snmplocal_sscpurawnice_145.rrd":ssCpuRawSystem:MAX \
DEF:c="/var/lib/cacti/rra/snmplocal_sscpurawnice_145.rrd":ssCpuRawUser:AVERAGE \
DEF:d="/var/lib/cacti/rra/snmplocal_sscpurawnice_145.rrd":ssCpuRawUser:MAX \
DEF:e="/var/lib/cacti/rra/snmplocal_sscpurawnice_145.rrd":ssCpuRawNice:AVERAGE \
DEF:f="/var/lib/cacti/rra/snmplocal_sscpurawnice_145.rrd":ssCpuRawNice:MAX \
DEF:g="/var/lib/cacti/rra/snmplocal_sscpurawnice_145.rrd":ssCpuRawInterrupt:AVERAGE \
DEF:h="/var/lib/cacti/rra/snmplocal_sscpurawnice_145.rrd":ssCpuRawInterrupt:MAX \
DEF:i="/var/lib/cacti/rra/snmplocal_sscpurawnice_145.rrd":ssCpuRawWait:AVERAGE \
DEF:j="/var/lib/cacti/rra/snmplocal_sscpurawnice_145.rrd":ssCpuRawWait:MAX \
DEF:ba="/var/lib/cacti/rra/snmplocal_sscpurawnice_145.rrd":ssCpuRawSoftIRQ:AVERAGE \
DEF:bb="/var/lib/cacti/rra/snmplocal_sscpurawnice_145.rrd":ssCpuRawSoftIRQ:MAX \
DEF:bc="/var/lib/cacti/rra/snmplocal_sscpurawnice_145.rrd":ssCpuRawIdle:AVERAGE \
DEF:bd="/var/lib/cacti/rra/snmplocal_sscpurawnice_145.rrd":ssCpuRawIdle:MAX \

Obviously, I had to change the CDEF to read:

Code: Select all

cdef=a,c,e,g,i,ba,+,+,+,+,+,ALL_DATA_SOURCES_NODUPS,100,/,/

And finally, I imported to a 0.8.7b-based Appliance version and got even more DEFs this time!!

Code: Select all

DEF:a="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawSystem:AVERAGE \
DEF:b="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawSystem:LAST \
DEF:c="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawSystem:MIN \
DEF:d="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawSystem:MAX \
DEF:e="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawUser:AVERAGE \
DEF:f="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawUser:LAST \
DEF:g="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawUser:MIN \
DEF:h="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawUser:MAX \
DEF:i="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawNice:AVERAGE \
DEF:j="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawNice:LAST \
DEF:ba="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawNice:MIN \
DEF:bb="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawNice:MAX \
DEF:bc="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawInterrupt:AVERAGE \
DEF:bd="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawInterrupt:LAST \
DEF:be="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawInterrupt:MIN \
DEF:bf="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawInterrupt:MAX \
DEF:bg="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawWait:AVERAGE \
DEF:bh="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawWait:LAST \
DEF:bi="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawWait:MIN \
DEF:bj="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawWait:MAX \
DEF:ca="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawSoftIRQ:AVERAGE \
DEF:cb="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawSoftIRQ:LAST \
DEF:cc="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawSoftIRQ:MIN \
DEF:cd="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawSoftIRQ:MAX \
DEF:ce="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawIdle:AVERAGE \
DEF:cf="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawIdle:LAST \
DEF:cg="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawIdle:MIN \
DEF:ch="/opt/statistics/rra/labdb01_sscpurawsystem_229.rrd":ssCpuRawIdle:MAX \

The good news is that you only need to do this fix once, once you've fixed it, you don't need to touch it again on that Cacti installation.
Maybe this issue does not occur for you at all.

I'm happy for any feedback on this issue or general feedback on whether or not the graphs work for you.

[brad1970]

I'm just getting started with cacti/RRDtool and this is exactly the kind of thing I'm looking to do. Unfortunately I can see that there is a pretty steep learning curve, so I can't help yet. But I plan to use this template as a learning tool and will let you know how it works for me (once I get it working).

Thanks!
-- Brad

[clockwork]

This works out of the box on smaller systems, however when used with larger systems (say 64 core/vcpu) the math no longer holds and the usage maxes out at 100 even when the cpu's are idle.

[c3226026]

Hello,

Thank for your works, but actually, with cacti 0.8.7g for RHEL I've nan for the two graphs.

How can I debut it ?

[mgothard]

Ha! I sure wish I would have looked here before going through this crap myself.

Anyway, I have attached my template with full dependencies. No Data Source here (though I considered creating one,) just templates.

linux_cpu_full_usage_with_depen.zip

cd/net - CPU percent Usage graph template with full dependencies

(5.32 KiB) Downloaded 1123 times

I ran into the same CDEF issue for a while before I sat down, did the math and rewrote an equivalent function that reads/works better with reverse polish notation. I believe my single CDEF works in all cases. Props to fmangeant @ http://forums.cacti.net/viewtopic.php?t=15412 for first turning me onto this whole idea.


12/15 Update: the CDEF I'm using is:

Code: Select all

cdef=CURRENT_DATA_SOURCE,100,*,ALL_DATA_SOURCES_DUPS,/

This is functionally equivalent to

Code: Select all

cdef=a,b,c,d,e,f,+,+,+,+,+,ALL_DATA_SOURCES_NODUPS,100,/,/

but without the headache.

What you are mathematically doing is: ((a+b+c+d+e+f)*100)/ALL_DATA_SOURCES_NODUPS
What I'm mathematically doing is: (CURRENT_DATA_SOURCE*100)/ALL_DATA_SOURCES_DUPS

This works because I'm applying the CDEF Function to each item in the Graph Template. The a+b+etc. addition is happening on graph (via STACK Graph Item Type) and not needlessly calculated separately. This way I don't need to keep track of the Cacti version behavior differences. The "duplicates" vs. "no duplicates" difference I'm a little unsure about (Cacti documentation is hazy on this), but I get more consistent results using "duplicates."

graph_image.png (38.45 KiB) Viewed 22697 times

[catchvjay]

Hi,

I have 12 Core 2 CPU on my server. So in total there are 24 logical CPUs on the server. On this server this template is not working properly. It is not generating correct graph. CPU utilization is very low on the server but graph shows 70% cpu utilization. I am attaching screen shot of the graph. Can anyone suggest solution for this?


Regards,
Vijay

[chrism01]

@mikaelf; I really like the look of those graphs.
Unfortunately I have a problem; can't seem to get any CPU utilisation methods to work.
Tried built-in Host-Mib method, tried built-in ucd/net method, tried yours....

I have an 8 way system

Code: Select all

grep -c ^processor /proc/cpuinfo 
8

snmpwalk -v1 -c public 127.0.0.1 .1.3.6.1.2.1.25.3

HOST-RESOURCES-MIB::hrDeviceDescr.768 = STRING: GenuineIntel: Intel(R) Xeon(R) CPU    X5460  @ 3.16GHz
HOST-RESOURCES-MIB::hrDeviceDescr.769 = STRING: GenuineIntel: Intel(R) Xeon(R) CPU    X5460  @ 3.16GHz
HOST-RESOURCES-MIB::hrDeviceDescr.770 = STRING: GenuineIntel: Intel(R) Xeon(R) CPU    X5460  @ 3.16GHz
HOST-RESOURCES-MIB::hrDeviceDescr.771 = STRING: GenuineIntel: Intel(R) Xeon(R) CPU    X5460  @ 3.16GHz
HOST-RESOURCES-MIB::hrDeviceDescr.772 = STRING: GenuineIntel: Intel(R) Xeon(R) CPU    X5460  @ 3.16GHz
HOST-RESOURCES-MIB::hrDeviceDescr.773 = STRING: GenuineIntel: Intel(R) Xeon(R) CPU    X5460  @ 3.16GHz
HOST-RESOURCES-MIB::hrDeviceDescr.774 = STRING: GenuineIntel: Intel(R) Xeon(R) CPU    X5460  @ 3.16GHz
HOST-RESOURCES-MIB::hrDeviceDescr.775 = STRING: GenuineIntel: Intel(R) Xeon(R) CPU    X5460  @ 3.16GHz

Installed your stuff as per instructions, got

Code: Select all

+ Running data query [10].
+ Found type = '3' [SNMP Query].
+ Found data query XML file at '/var/www/html/cacti/resource/snmp_queries/systemstats.xml'
+ XML file parsed ok.
+ <oid_num_indexes> missing in XML file, 'Index Count Changed' emulated by counting oid_index entries
+ Executing SNMP walk for list of indexes @ '.1.3.6.1.4.1.2021.11.50' Index Count: 0
+ No SNMP data returned

for verbose query.

Can you (or anyone please) help?

Cheers
Chris

[chrism01]

Ok, cancel prev qn; discovered another tool had added a passwd to the ro community. Reset (for testing) to public & after a certain amt of fiddling, got the graphs working

However, after re-reading the orig post, I'm not clear how many DEFs (for CPU) I should have; currently I only see

Code: Select all

DEF:a="/var/www/html/cacti/rra/xxx_sscpurawsystem_32.rrd":ssCpuRawSystem:AVERAGE \
DEF:b="/var/www/html/cacti/rra/xxx_sscpurawsystem_32.rrd":ssCpuRawUser:AVERAGE \
DEF:c="/var/www/html/cacti/rra/xxx_sscpurawsystem_32.rrd":ssCpuRawNice:AVERAGE \
DEF:d="/var/www/html/cacti/rra/xxx_sscpurawsystem_32.rrd":ssCpuRawInterrupt:AVERAGE \
DEF:e="/var/www/html/cacti/rra/xxx_sscpurawsystem_32.rrd":ssCpuRawWait:AVERAGE \
DEF:f="/var/www/html/cacti/rra/xxx_sscpurawsystem_32.rrd":ssCpuRawSoftIRQ:AVERAGE \
DEF:g="/var/www/html/cacti/rra/xxx_sscpurawsystem_32.rrd":ssCpuRawIdle:AVERAGE \

QN: is this right or wrong; if wrong exactly how do I fix it?

Cheers
Chris

[hid3nax]

Does this template work with Cacti 0.8.7i? Where can I get the latest version if it does?

[gandalf]

Does this template work with Cacti 0.8.7i? Where can I get the latest version if it does?

In general, templates will work in higher releases. Of course, they won't make use of new features, in general
R.

[hid3nax]

Thanks.
I've tried the template and it seems to work okay on some hosts. However, on others (linux boxes) it displays CPU count as 1 (instead of 2). Utilization displayed is also double than the real. The boxes affected by this run 3.1.9 kernel. Any fixes for that?

[stormonts]

Since switching over to your template, I have started getting gaps in my CPU graphs (wasn't getting them before with the other template I was using in Cacti). Is there a setting I should change in regards to these hosts to fix the issue? SNMP timeout or maximum OIDs per request?

Also. we have one 32 bit Linux host that seems more prone to the gaps.

[edward1234]

Hi Everyone,

Bump for catchvjay's question. I'm having the exact same issue with dual Intel Xeons with 6 cores giving me 24 threads. My graph looks exactly like his wrt the values being inaccurate and the CPU count being 0. Fridh's graph works for my other boxes (thanks), which have dual Xeons with 4 cores (16 threads). Maybe we're exceeding some threshold?

Cacti version is 0.8.7b

I verified the OIDs are correct for systemstats.xml and verified via Graph Management with debugging turned on.

Thanks

[b0fh]

Template looks nice, but same problem here with two Xeon 6 core CPUs: User/system times are wrong, idle is "nan".
Did anyone fix this or know any other nice scalable Linux-Multi-CPU template?

Edit: The template seems to work when most of the cores are under load. "calculated logical CPUs" and usage/idle percent are shown correctly. It seems to be a problem when most of the available cores are not loaded in the given interval.

[classen]

This is a great script and template. Thank you so much for this contribution.

I have installed this script/template on 6 hosts so far. Five are working great. One has a problem with the Idle percentage showing up as NaN so the graph is pegged at 100%. This particular machine has 24 cores. Could that be causing a problem?

Here is the output of top:

Code: Select all

top - 16:09:53 up 140 days, 56 min, 12 users,  load average: 0.06, 0.04, 0.03
Tasks: 475 total,   1 running, 474 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.7%us,  0.0%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu8  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu9  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu10 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu11 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu12 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu13 :  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu14 :  0.0%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu15 :  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu16 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu17 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu18 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu19 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu20 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu21 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu22 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu23 :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  32915592k total, 24861164k used,  8054428k free,   240056k buffers
Swap:  2097144k total,        0k used,  2097144k free, 23949304k cached