Extended system stats graphing- vmstat, iostat, mpstat (sar)

[phil25ls]

Hi,

I have a problem with sarparse.pl script and cacti, the sar graph not update to the poller but it's ok manually.

cacti version : 0.8.7g
my system user : www-data

By Sarparse.pl script user www-data :

$ perl sarparse.pl localhost
procs:0,00 cswchs:58,00 CPU:all user:0,00 nice:0,00 system:0,00 iowait:0,00 steal:0,00 idle:100,00 pgpgins:0,00 pgpgouts:0,00 faults:0,00 majflts:0,00 pgfrees:0,00 pgscanks:0,00 pgscands:0,00 pgsteals:0,00 vmeff:0,00 kbmemfree:7173912 kbmemused:1007460 memused:12,31 kbbuffers:137612 kbcached:502320 kbswpfree:2095096 kbswpused:0 swpused:0,00 kbswpcad:0 dentunusd:38210 file_sz:896 inode_sz:35042 pty_sz:34 totsck:0 tcpsck:0 udpsck:0 rawsck:0 ip_frag:0 tcp_tw:0 runq_sz:0 plist_sz:234 ldavg_1:0,00 ldavg_5:0,00 ldavg_15:0,00

By cacti poller with cron "www-data php /usr/share/cacti/site/poller.php >/dev/null 2>/var/log/cacti/poller-error.log":

02/23/2011 11:50:06 AM - CMDPHP: Poller[0] Host[2] DS[45] WARNING: Result from CMD not valid. Partial Result: U
02/23/2011 11:50:06 AM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/localhost_dentunusd_44.rrd --template 1298458201:U
02/23/2011 11:50:05 AM - CMDPHP: Poller[0] Host[2] DS[44] CMD: perl /usr/share/cacti/site/scripts/sarparse.pl 127.0.0.1, output: U

By manually poller with user www-data:

/23/2011 12:04:58 PM - CMDPHP: Poller[0] Host[2] DS[45] CMD: perl /usr/share/cacti/site/scripts/sarparse.pl 127.0.0.1, output: procs:0,00 cswchs:90,28 CPU:all user:0,31 nice:0,00 system:0,00 iowait:0,00 steal:0,00 idle:99,69 pgpgins:0,00 pgpgouts:0,00 faults:0,00 majflts:0,00 pgfrees:0,00 pgscanks:0,00 pgscands:0,00 pgsteals:0,00 vmeff:0,00 kbmemfree:7141540 kbmemused:1039832 memused:12,71 kbbuffers:138076 kbcached:502684 kbswpfree:2095096 kbswpused:0 swpused:0,00 kbswpcad:0 dentunusd:38422 file_sz:1152 inode_sz:35281 pty_sz:34 totsck:0 tcpsck:0 udpsck:0 rawsck:0 ip_frag:0 tcp_tw:0 runq_sz:0 plist_sz:238 ldavg_1:0,20 ldavg_5:0,05 ldavg_15:0,01
02/23/2011 12:04:58 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/lib/cacti/rra/localhost_dentunusd_44.rrd --template dentunusd 1298459093:38422
02/23/2011 12:04:58 PM - POLLER: Poller[0] Parsed MULTI output field 'dentunusd:38422' [map dentunusd->dentunusd]

I need some help

Thanks

[stucky101]

ahhdem

I see your point about stacking graphs but I just get around that by using line graphs instead of areas. Solves the problem of overlapping just fine.
Unfortunately, I discovered a new problem and that's a real downer. Turns out some graphs only retain data for 3 days instead of 7. The 2 affected are the memory and load average ones.
The hourly and daily graphs are fine but starting at the weekly I'm missing data.
As if the rrd is only defined for 3 days when the graph needs 7.
That's a real bummer ! BTW I run RHEL5.5 sysstat-7.0.2-3.el5.
Are you saying I need the bleeding edge sysstat to fix this problem as well ?
The attached graph shows sarparse running since January but never maintaining more than 3 day's worth of graphs.
You also mentioned some bugs. Is there a new version of sarparse I can try ?

[noname]

On this point only,

Turns out some graphs only retain data for 3 days instead of 7. The 2 affected are the memory and load average ones.
The hourly and daily graphs are fine but starting at the weekly I'm missing data.
As if the rrd is only defined for 3 days when the graph needs 7.

Probably you're using 1-minute polling, but your RRA settings remains default 5-minute polling.
(See 'Console -> Data Sources -> RRAs')

default_RRAs.png (12.82 KiB) Viewed 5917 times

With default 'Weekly' RRA, Cacti (RRDtool) stores 700 rows in RRD file.
It assumed that "1 row" equals to "1 value which is consolidated for 30min" (=5min * 6steps).
Under the default 5-minute polling, data will be stored for about 2 weeks:
700 * 5min * 6steps = 350hours (= 2 weeks)

So if you changed polling interval to 1-min, Cacti regards this RRA as follows:
700 * 1min * 6steps = 70hours (= 2.9days)

NOTE: 'Timespan' (e.g. Weekly : 604800sec = 7days) is just a default time range when drawing graphs, not stored in RRD file.
It can be changed dynamically, like as 'Presets:' pull-down menu in 'graphs' tab.

***

You can modify these RRA settings for 1-minute polling,
but the changes (maybe) only affects to graph which is newly added after this changes.

If you want to work with existing RRD as well without losing data,
you should extend number of rows in RRA by using "rrdtool resize" command.
- RRDtool - rrdresize

Or see this document:
- manual:087:8_rrdtool.04_resize - Cacti Docs

// Sorry my English

[zx1]

don't work
1. Request Lifetime
2. Queue Length

RRDTool Command:

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title=' hw.xxx.com::sda1 - Request Lifetime ' \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label='ms/req' \
--slope-mode \
--font TITLE:12: \
--font AXIS:8: \
--font LEGEND:10: \
--font UNIT:8: \
DEF:a="/var/www/cacti/rra/hw_btsc_lv_rsecs_313.rrd":svctm:LAST \
DEF:b="/var/www/cacti/rra/hw_btsc_lv_rsecs_313.rrd":await:LAST \
DEF:c="/var/www/cacti/rra/hw_btsc_lv_rsecs_313.rrd":util:LAST \
AREA:a#7EE600FF:"Service Time" \
GPRINTLAST:"Current\:%8.2lf%s" \
GPRINTAVERAGE:"Average\:%8.2lf%s" \
GPRINTMAX:"Maximum\:%8.2lf%s\n" \
AREA:b#FF4105FF:"Wait time":STACK \
GPRINTLAST:" Current\:%8.2lf%s" \
GPRINTAVERAGE:"Average\:%8.2lf%s" \
GPRINTMAX:"Maximum\:%8.2lf%s\n" \
LINE1:c#000000FF:"CPU%"

RRDTool Says:

ERROR: the RRD does not contain an RRA matching the chosen CF





RRDTool Command:

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title='hw.xxx.com::sdb - Queue Length ' \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label='requests' \
--slope-mode \
--font TITLE:12: \
--font AXIS:8: \
--font LEGEND:10: \
--font UNIT:8: \
DEF:a="/var/www/cacti/rra/hw_btsc_lv_rsecs_308.rrd":avgqu_sz:LAST \
AREA:a#FF0000FF:"Waiting Requests\:" \
GPRINTLAST:"Current\:%8.2lf%s" \
GPRINTAVERAGE:"Average\:%8.2lf%s" \
GPRINTMAX:"Maximum\:%8.2lf%s\n"

RRDTool Says:

ERROR: the RRD does not contain an RRA matching the chosen CF

[Daniell]

I have also imported the uptime template posted in here and received this error.
Any idea how to solve this?

[noname]

RRDTool Says:
ERROR: the RRD does not contain an RRA matching the chosen CF

See this: http://forums.cacti.net/viewtopic.php?p=215185#p215185

[eschoeller]

I have found a problem between using spine and cmd.php. with spine:

Code: Select all

04/28/2012 11:40:38 AM - SPINE: Poller[0] Host[299] TH[1] DS[9859] SCRIPT: perl /usr/local/cacti-0.8.7g-dev/scripts/sarparse.pl host.domain.edu, output: 0
04/28/2012 11:40:39 AM - SPINE: Poller[0] Host[299] TH[1] DS[9860] SCRIPT: perl /usr/local/cacti-0.8.7g-dev/scripts/sarparse.pl host.domain.edu, output: 0
04/28/2012 11:40:40 AM - SPINE: Poller[0] Host[299] TH[1] DS[9861] SCRIPT: perl /usr/local/cacti-0.8.7g-dev/scripts/sarparse.pl host.domain.edu, output: 0

with cmd.php

Code: Select all

04/28/2012 11:41:43 AM - CMDPHP: Poller[0] Host[299] DS[9865] CMD: perl /usr/local/cacti-0.8.7g-dev/scripts/sarparse.pl host.domain.edu, output: CPU:all user:0.26 nice:0.00 system:0.26 iowait:0.00 steal:0.00 idle:99.48 procs:0.00 cswchs:213.13 pgpgins:0.00 pgpgouts:0.00 faults:38.38 majflts:0.00 pgfrees:76.77 pgscanks:0.00 pgscands:0.00 pgsteals:0.00 vmeff:0.00 kbmemfree:3454948 kbmemused:462752 memused:11.81 kbbuffers:19688 kbcached:218576 kbswpused:685080 swpused:8.55 dentunusd:7207 file_sz:1696 inode_sz:15627 pty_sz:8 runq_sz:0 plist_sz:274 ldavg_1:0.00 ldavg_5:0.00 ldavg_15:1.06 totsck:177 tcpsck:19 udpsck:16 rawsck:0 ip_frag:0 tcp_tw:14
04/28/2012 11:41:44 AM - CMDPHP: Poller[0] Host[299] DS[9866] CMD: perl /usr/local/cacti-0.8.7g-dev/scripts/sarparse.pl host.domain.edu, output: CPU:all user:0.26 nice:0.00 system:0.26 iowait:0.00 steal:0.00 idle:99.47 procs:0.00 cswchs:168.32 pgpgins:0.00 pgpgouts:0.00 faults:37.62 majflts:0.00 pgfrees:73.27 pgscanks:0.00 pgscands:0.00 pgsteals:0.00 vmeff:0.00 kbmemfree:3455072 kbmemused:462628 memused:11.81 kbbuffers:19696 kbcached:218572 kbswpused:685080 swpused:8.55 dentunusd:7207 file_sz:1696 inode_sz:15627 pty_sz:8 runq_sz:1 plist_sz:274 ldavg_1:0.00 ldavg_5:0.00 ldavg_15:1.05 totsck:177 tcpsck:19 udpsck:16 rawsck:0 ip_frag:0 tcp_tw:14
04/28/2012 11:41:45 AM - CMDPHP: Poller[0] Host[299] DS[9867] CMD: perl /usr/local/cacti-0.8.7g-dev/scripts/sarparse.pl host.domain.edu, output: CPU:all user:0.27 nice:0.00 system:0.27 iowait:0.00 steal:0.00 idle:99.46 procs:0.00 cswchs:169.31 pgpgins:0.00 pgpgouts:0.00 faults:38.61 majflts:0.00 pgfrees:74.26 pgscanks:0.00 pgscands:0.00 pgsteals:0.00 vmeff:0.00 kbmemfree:3455188 kbmemused:462512 memused:11.81 kbbuffers:19696 kbcached:218576 kbswpused:685080 swpused:8.55 dentunusd:7207 file_sz:1664 inode_sz:15627 pty_sz:8 runq_sz:0 plist_sz:274 ldavg_1:0.00 ldavg_5:0.00 ldavg_15:1.05 totsck:177 tcpsck:19 udpsck:16 rawsck:0 ip_frag:0 tcp_tw:14

This appears to be a problem with how the script is outputting data. There is a trailing whitespace and spine just can't handle that. Quite silly to be honest, but read:

http://forums.cacti.net/viewtopic.php?f=21&t=42742

The way to fix this is to slightly change sarparse.pl to suppress that last trailing space on the output. I had to reformat the code so I could better understand it, and then I simply added a if/else clause at the end to eliminate that last space. I could have just blindly chopped the last character from the variable but that seemed too drastic. Here is the code, please note it has our nagios_plugin_path specified which will be different than yours!

Code: Select all

#!/usr/bin/perl -w

########################################################
#
#  f:  sarparse.pl v.01
#  a:  adam backer
#  e:  backcountry.com
# 
#  summary:
#    fetches some sar data via nrpe, parses it and produces cacti-friendly output
#
#       command[sar]=sar -Brcquwv -n SOCK 1 1|grep Average
#
# 20120428: bugs added by Eric Schoeller, University of Colorado Boulder to resolve
#           polling issue with spine.

$ip=$ARGV[0];
$nagios_plugin_path="/usr/local/nagios/libexec/";


if (!$ip){ # !~ /\d+/) {
   print "Usage:  sarparse.pl <ip>\n";
   exit 3;
   }

else {
   @input=`$nagios_plugin_path/check_nrpe -H $ip -c sar`;
   #@input=`ssh $ip "sar -Brcquwv -n SOCK 1 1|grep Average"`;
   $i=0;
   $f=0;
   $v=0;
   foreach $line (@input) {
      @line = split(' ', $line);
      $fc=$#line;
         for ($x=1; $x<=$fc; $x++) {
            if ($x == $fc)  { 
               $end = 1;
            } 
            else { 
               $end = 0; 
            }
            if ($i == 0) {
               $line[$x]=~s/\///;
               $line[$x]=~s/\%//;
               $line[$x]=~s/-/_/;
               $line[$x]=~s/nr/sz/;
               $line[$x]=~s/commit/swpused/;
               $fields[$f]=$line[$x];
               #print "settings fields $f to $line[$x]  -  ";
               $f++;
               if ($end) {
                  $i=1;
               }
               next;
            }
            if ($i == 1) {
               $vals[$v]=$line[$x];
               #print "setting vals $v to $line[$x]\n";
               $v++;
            if ($end) {
               $i=0;
            }
            next;
         }
      }
   }
   $output='';
   for ($x=0; $x<=$#fields; $x++) {
      # if there is still more data to come, add a space after current metric
      if ($x < $#fields) {
         $output=$output."$fields[$x]:$vals[$x] ";
      }
      # don't add a trailing space at the end of $output. Spine will yell at you.
      else {
         $output=$output."$fields[$x]:$vals[$x]";
      }
   }
   # For good measure to be spine-friendly, but probably is ineffective.
   chomp($output);
   print "$output";
}

[eschoeller]

I also adapted this template to essentially run sarparse.pl on the client side and export this data over SNMP via 'extend'. I know ahhdem was not in favor of SNMP, but some things have changed since 2007 - the extend directive is kinda awesome. The cacti poller simply runs an snmpget command which pulls in the output already formatted correctly. This apparently offloads a considerable amount of work from the Cacti server to the individual servers being polled. In my test environment my poller runtime went from ~13s to ~7s which represents a 46% increase in efficiency. I guess it all depends on where your horsepower (or bottlenecks) are ... on the Cacti server or your monitored servers. Generally speaking there isn't a whole lot of funding lying around for Cacti servers ........ !

If folks are interested in some of the changes I've made just let me know.

[justjoel]

If folks are interested in some of the changes I've made just let me know.

I'd like to see the changes you made. I was considering doing something similar here.

[Thalamus]

Sounds like a good idea. I would be very interested in your changes.

[JJX]

I dont know if the documentation was updated elsewhere but this one http://forums.cacti.net/viewtopic.php?p=138627#p138627 has a few erros

page 7:
wrong: command[sarparse_cpustat]=sar -P ALL 1 1 |grep Averge |grep -v all
correct: command[sarparse_cpustat]=sar -P ALL 1 1 |grep Average |grep -v all

command[sar]=sar -Brcquwv -n SOCK 1 1|grep Average

The command "command[sar]=sar -Brcquwv -n SOCK 1 1|grep Average" doesnt seem to work on debian or redhat
It should be "command[sar]=sar -BrCquwv -n SOCK 1 1|grep Average" ?
The output

Code: Select all

Average:        CPU     %user     %nice   %system   %iowait    %steal     %idle
Average:        all      0.00      0.00      0.30      0.00      0.00     99.70
Average:       proc/s   cswch/s
Average:         0.00     22.00
Average:     pgpgin/s pgpgout/s   fault/s  majflt/s  pgfree/s pgscank/s pgscand/s pgsteal/s    %vmeff
Average:         0.00      0.00     34.00      0.00     45.00      0.00      0.00      0.00      0.00
Average:    kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
Average:       182576   1891908     91.20    241992   1480028    380568      7.05
Average:    dentunusd   file-nr  inode-nr    pty-nr
Average:        28058       576     17953       264
Average:      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
Average:            0       137      0.27      0.31      0.18
Average:       totsck    tcpsck    udpsck    rawsck   ip-frag    tcp-tw
Average:           74         6         4         0         0         0

is what expected from the command?

[AshleyW]

Code: Select all

./sarparse.pl IP
Name "main::nagios_plugin_path" used only once: possible typo at ./sarparse.pl line 16.
The authenticity of host 'IP (IP)' can't be established.
RSA key fingerprint is ...
Are you sure you want to continue connecting (yes/no)? y
Please type 'yes' or 'no': yes
Warning: Permanently added 'IP' (RSA) to the list of known hosts.
Address IP maps to IP, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
root@IP's password:
stdin: is not a tty
CPU:all user:0.00 nice:0.00 system:0.13 iowait:0.00 steal:0.00 idle:99.87 procs:0.00 cswchs:428.57 pgpgins:0.00 pgpgouts:8.16 faults:0.00 majflts:0.00 pgfrees:0.00 pgscanks:0.00 pgscands:0.00 pgsteals:0.00 vmeff:0.00 kbmemfree:6711932 kbmemused:1676676 memused:19.99 kbbuffers:0 kbcached:1101840 kbswpused:0 swpused:0.00 dentunusd:3132104 file_sz:1280 inode_sz:2234535 pty_sz:21186 runq_sz:0 plist_sz:208 ldavg_1:0.22 ldavg_5:0.18 ldavg_15:0.11 totsck:3744 tcpsck:19 udpsck:1 rawsck:0 ip_frag:0 tcp_tw:262 [

Does this mean I need to setup keys for keyless logins?

I also made this edit:

Code: Select all

#@input=`$nagios_plugin_path/check_nrpe -H $ip -c sar`;
@input=`ssh $ip "sar -Brquwv -n SOCK 1 1|grep Average"`

[JJX]

If you have NRPE configured why do you use ssh?
NRPE can do the job easily