95th percentile problem

kenny29 · Post by **kenny29** » Wed Mar 10, 2010 4:29 am

Hello,

Sorry for my english...

Since I have updated my Cacti version (0.8.7d -> 0.8.7e), there is a problem with new graphs using the template name : In/out Bits with 95th Percentile.

Indeed if I create a new graph with this template, the graph is never diplayed and seems doesn't work, but if I change it (In/Out bits for example) it's OK.

This is strange because all others graphs created above and using In/out Bits with 95th Percentile are still working well...

Someone have an idea ?

Thanks.

sini · Post by **sini** » Tue Apr 20, 2010 9:18 am

And what can you see when graph debug mode is on?

DervMan · Post by **DervMan** » Thu Apr 29, 2010 9:22 am

Like you've I've recently upgraded to the latest version and I'm seeing the same problem. If I add a host and use 95th percentile the graphs don't get created. Using the same host and switch port but use bits per second I get graphs.

Here's some debug from a switch port we're monitoring using 95th percentile

Code: Select all

04/29/2010 03:18:01 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/www/cacti/rra/ar01-wiho-01_traffic_in_5810.rrd --template traffic_out:traffic_in 1272550680:147613975:3326921589
04/29/2010 03:18:00 PM - CMDPHP: Poller[0] Host[340] DS[5810] SNMP: v1: 10.32.251.254, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.11, output: 147613975

All the graphs that have been previously created with 95th percentile continue to work correctly.

Here's what I'm seeing in the graph debug.

Code: Select all

RRDTool Command:
/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title="AS01-BBond-00 - Traffic - Gi0/10" \
--rigid \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label="bits per second" \
--slope-mode \
--font TITLE:12: \
--font AXIS:8: \
--font LEGEND:10: \
--font UNIT:8: \
DEF:a="/var/www/cacti/rra/as01-bbond-00_traffic_in_7613.rrd":traffic_in:AVERAGE \
DEF:b="/var/www/cacti/rra/as01-bbond-00_traffic_in_7613.rrd":traffic_out:AVERAGE \
CDEF:cdefa=a,8,* \
CDEF:cdeff=b,8,* \
CDEF:cdefbe=a,UN,INF,UNKN,IF \
CDEF:cdefbh=TIME,1272550885,GT,a,a,UN,0,a,IF,IF,TIME,1272550885,GT,b,b,UN,0,b,IF,IF,+ \
AREA:cdefa#00CF00FF:"Inbound"  \
GPRINT:cdefa:LAST:" Current\:%8.2lf%s"  \
GPRINT:cdefa:AVERAGE:"Average\:%8.2lf%s"  \
GPRINT:cdefa:MAX:"Maximum\:%8.2lf%s\n"  \
COMMENT:"Transfer\:  0 bytes"  \
LINE1:cdeff#002A97FF:"Outbound"  \
GPRINT:cdeff:LAST:"Current\:%8.2lf%s"  \
HRULE:|95\:bits\:0\:current|#00FF00FF:"95%\:"  \
COMMENT:"|95\:bits\:6\:current| mbit\n"  \
GPRINT:cdeff:AVERAGE:"Average\:%8.2lf%s"  \
GPRINT:cdeff:MAX:"Maximum\:%8.2lf%s\n"  \
COMMENT:" \n" \
HRULE:0#FF0000FF:"95th Percentile"  \
COMMENT:"(0 mbit in+out)"  \
AREA:cdefbe#8F9286FF:""  \
COMMENT:"Transfer\: 0 bytes"  \
HRULE:|95\:bits\:0\:current|#0000FFFF:"95%\:"  \
COMMENT:"0 mbit"  \
COMMENT:"Total Transfer\: 0 bytes\n"

And RRDTool says: Nothing!!!

Here's the .RRD

Code: Select all

-rw-r--r--  1 root root  254920 Apr 29 15:15 as01-bbond-00_traffic_in_7613.rrd

Interestingly if I use graph management to change from 95th percentile to bits per second the graph immediately appears albeit with no data.

Post by **gandalf** » Thu Apr 29, 2010 4:10 pm

Please execute the rrdtool command from cli. Post output to /dev/null, but catch the error messages
R.

DervMan · Post by **DervMan** » Fri Apr 30, 2010 2:32 am

Not sure what parameters would want along with the rrdtool command so here's one for starters!

Code: Select all

[britsg@s-csswan rra]$ sudo /usr/bin/rrdtool info /var/www/cacti/rra/as01-bbond-00_traffic_in_7606.rrd
Password:
filename = "/var/www/cacti/rra/as01-bbond-00_traffic_in_7606.rrd"
rrd_version = "0003"
step = 300
last_update = 1272612332
ds[traffic_in].type = "COUNTER"
ds[traffic_in].minimal_heartbeat = 600
ds[traffic_in].min = 0.0000000000e+00
ds[traffic_in].max = 1.0000000000e+09
ds[traffic_in].last_ds = "1073157302"
ds[traffic_in].value = 1.8218875762e+06
ds[traffic_in].unknown_sec = 0
ds[traffic_out].type = "COUNTER"
ds[traffic_out].minimal_heartbeat = 600
ds[traffic_out].min = 0.0000000000e+00
ds[traffic_out].max = 1.0000000000e+09
ds[traffic_out].last_ds = "3065791010"
ds[traffic_out].value = 5.0608304106e+06
ds[traffic_out].unknown_sec = 0
rra[0].cf = "AVERAGE"
rra[0].rows = 500
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[0].cdp_prep[1].value = NaN
rra[0].cdp_prep[1].unknown_datapoints = 0
rra[1].cf = "AVERAGE"
rra[1].rows = 500
rra[1].pdp_per_row = 1
rra[1].xff = 5.0000000000e-01
rra[1].cdp_prep[0].value = NaN
rra[1].cdp_prep[0].unknown_datapoints = 0
rra[1].cdp_prep[1].value = NaN
rra[1].cdp_prep[1].unknown_datapoints = 0
rra[2].cf = "AVERAGE"
rra[2].rows = 500
rra[2].pdp_per_row = 1
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = NaN
rra[2].cdp_prep[0].unknown_datapoints = 0
rra[2].cdp_prep[1].value = NaN
rra[2].cdp_prep[1].unknown_datapoints = 0
rra[3].cf = "AVERAGE"
rra[3].rows = 500
rra[3].pdp_per_row = 1
rra[3].xff = 5.0000000000e-01
rra[3].cdp_prep[0].value = NaN
rra[3].cdp_prep[0].unknown_datapoints = 0
rra[3].cdp_prep[1].value = NaN
rra[3].cdp_prep[1].unknown_datapoints = 0
rra[4].cf = "AVERAGE"
rra[4].rows = 600
rra[4].pdp_per_row = 1
rra[4].xff = 5.0000000000e-01
rra[4].cdp_prep[0].value = NaN
rra[4].cdp_prep[0].unknown_datapoints = 0
rra[4].cdp_prep[1].value = NaN
rra[4].cdp_prep[1].unknown_datapoints = 0
rra[5].cf = "AVERAGE"
rra[5].rows = 700
rra[5].pdp_per_row = 6
rra[5].xff = 5.0000000000e-01
rra[5].cdp_prep[0].value = 1.4141515961e+05
rra[5].cdp_prep[0].unknown_datapoints = 0
rra[5].cdp_prep[1].value = 9.1739819078e+05
rra[5].cdp_prep[1].unknown_datapoints = 0
rra[6].cf = "AVERAGE"
rra[6].rows = 775
rra[6].pdp_per_row = 24
rra[6].xff = 5.0000000000e-01
rra[6].cdp_prep[0].value = 1.2597869629e+06
rra[6].cdp_prep[0].unknown_datapoints = 0
rra[6].cdp_prep[1].value = 3.8909882740e+06
rra[6].cdp_prep[1].unknown_datapoints = 0
rra[7].cf = "AVERAGE"
rra[7].rows = 797
rra[7].pdp_per_row = 288
rra[7].xff = 5.0000000000e-01
rra[7].cdp_prep[0].value = 1.2845780815e+06
rra[7].cdp_prep[0].unknown_datapoints = 0
rra[7].cdp_prep[1].value = 3.9860877248e+06
rra[7].cdp_prep[1].unknown_datapoints = 0
rra[8].cf = "MIN"
rra[8].rows = 600
rra[8].pdp_per_row = 1
rra[8].xff = 5.0000000000e-01
rra[8].cdp_prep[0].value = NaN
rra[8].cdp_prep[0].unknown_datapoints = 0
rra[8].cdp_prep[1].value = NaN
rra[8].cdp_prep[1].unknown_datapoints = 0
rra[9].cf = "MIN"
rra[9].rows = 700
rra[9].pdp_per_row = 6
rra[9].xff = 5.0000000000e-01
rra[9].cdp_prep[0].value = 1.6006892530e+04
rra[9].cdp_prep[0].unknown_datapoints = 0
rra[9].cdp_prep[1].value = 1.4362017566e+05
rra[9].cdp_prep[1].unknown_datapoints = 0
rra[10].cf = "MIN"
rra[10].rows = 775
rra[10].pdp_per_row = 24
rra[10].xff = 5.0000000000e-01
rra[10].cdp_prep[0].value = 1.7651236624e+02
rra[10].cdp_prep[0].unknown_datapoints = 0
rra[10].cdp_prep[1].value = 1.2684370448e+03
rra[10].cdp_prep[1].unknown_datapoints = 0
rra[11].cf = "MIN"
rra[11].rows = 797
rra[11].pdp_per_row = 288
rra[11].xff = 5.0000000000e-01
rra[11].cdp_prep[0].value = 1.5372584322e+02
rra[11].cdp_prep[0].unknown_datapoints = 0
rra[11].cdp_prep[1].value = 9.9265913463e+02
rra[11].cdp_prep[1].unknown_datapoints = 0
rra[12].cf = "MAX"
rra[12].rows = 500
rra[12].pdp_per_row = 1
rra[12].xff = 5.0000000000e-01
rra[12].cdp_prep[0].value = NaN
rra[12].cdp_prep[0].unknown_datapoints = 0
rra[12].cdp_prep[1].value = NaN
rra[12].cdp_prep[1].unknown_datapoints = 0
rra[13].cf = "MAX"
rra[13].rows = 500
rra[13].pdp_per_row = 1
rra[13].xff = 5.0000000000e-01
rra[13].cdp_prep[0].value = NaN
rra[13].cdp_prep[0].unknown_datapoints = 0
rra[13].cdp_prep[1].value = NaN
rra[13].cdp_prep[1].unknown_datapoints = 0
rra[14].cf = "MAX"
rra[14].rows = 500
rra[14].pdp_per_row = 1
rra[14].xff = 5.0000000000e-01
rra[14].cdp_prep[0].value = NaN
rra[14].cdp_prep[0].unknown_datapoints = 0
rra[14].cdp_prep[1].value = NaN
rra[14].cdp_prep[1].unknown_datapoints = 0
rra[15].cf = "MAX"
rra[15].rows = 500
rra[15].pdp_per_row = 1
rra[15].xff = 5.0000000000e-01
rra[15].cdp_prep[0].value = NaN
rra[15].cdp_prep[0].unknown_datapoints = 0
rra[15].cdp_prep[1].value = NaN
rra[15].cdp_prep[1].unknown_datapoints = 0
rra[16].cf = "MAX"
rra[16].rows = 600
rra[16].pdp_per_row = 1
rra[16].xff = 5.0000000000e-01
rra[16].cdp_prep[0].value = NaN
rra[16].cdp_prep[0].unknown_datapoints = 0
rra[16].cdp_prep[1].value = NaN
rra[16].cdp_prep[1].unknown_datapoints = 0
rra[17].cf = "MAX"
rra[17].rows = 700
rra[17].pdp_per_row = 6
rra[17].xff = 5.0000000000e-01
rra[17].cdp_prep[0].value = 1.4086676772e+05
rra[17].cdp_prep[0].unknown_datapoints = 0
rra[17].cdp_prep[1].value = 2.3579767021e+05
rra[17].cdp_prep[1].unknown_datapoints = 0
rra[18].cf = "MAX"
rra[18].rows = 775
rra[18].pdp_per_row = 24
rra[18].xff = 5.0000000000e-01
rra[18].cdp_prep[0].value = 7.7510068657e+05
rra[18].cdp_prep[0].unknown_datapoints = 0
rra[18].cdp_prep[1].value = 9.1721134753e+05
rra[18].cdp_prep[1].unknown_datapoints = 0
rra[19].cf = "MAX"
rra[19].rows = 797
rra[19].pdp_per_row = 288
rra[19].xff = 5.0000000000e-01
rra[19].cdp_prep[0].value = 7.7510068657e+05
rra[19].cdp_prep[0].unknown_datapoints = 0
rra[19].cdp_prep[1].value = 9.1721134753e+05
rra[19].cdp_prep[1].unknown_datapoints = 0
rra[20].cf = "LAST"
rra[20].rows = 600
rra[20].pdp_per_row = 1
rra[20].xff = 5.0000000000e-01
rra[20].cdp_prep[0].value = NaN
rra[20].cdp_prep[0].unknown_datapoints = 0
rra[20].cdp_prep[1].value = NaN
rra[20].cdp_prep[1].unknown_datapoints = 0
rra[21].cf = "LAST"
rra[21].rows = 700
rra[21].pdp_per_row = 6
rra[21].xff = 5.0000000000e-01
rra[21].cdp_prep[0].value = 5.3867609010e+04
rra[21].cdp_prep[0].unknown_datapoints = 0
rra[21].cdp_prep[1].value = 1.6283548919e+05
rra[21].cdp_prep[1].unknown_datapoints = 0
rra[22].cf = "LAST"
rra[22].rows = 775
rra[22].pdp_per_row = 24
rra[22].xff = 5.0000000000e-01
rra[22].cdp_prep[0].value = 5.3867609010e+04
rra[22].cdp_prep[0].unknown_datapoints = 0
rra[22].cdp_prep[1].value = 1.6283548919e+05
rra[22].cdp_prep[1].unknown_datapoints = 0
rra[23].cf = "LAST"
rra[23].rows = 797
rra[23].pdp_per_row = 288
rra[23].xff = 5.0000000000e-01
rra[23].cdp_prep[0].value = 5.3867609010e+04
rra[23].cdp_prep[0].unknown_datapoints = 0
rra[23].cdp_prep[1].value = 1.6283548919e+05
rra[23].cdp_prep[1].unknown_datapoints = 0

Post by **gandalf** » Fri Apr 30, 2010 3:29 pm

gandalf wrote:Please execute the rrdtool command from cli. Post output to /dev/null, but catch the error messages
R.

I was referring to the "rrdtool graph" statement you've posted before. The rrdtool info does not help here
R.

DervMan · Post by **DervMan** » Tue May 04, 2010 2:23 am

Any suggestions for what to try next?

I like using the 95th percentile so don't want to use bits per second instead.....

Post by **gandalf** » Tue May 04, 2010 2:39 pm

DervMan wrote:Any suggestions for what to try next?

I like using the 95th percentile so don't want to use bits per second instead.....

I'm still waiting for the results of a "rrdtool graph" run from cli. I'm hoping this will throw an error which I need to see
R.

DervMan · Post by **DervMan** » Wed May 05, 2010 2:34 am

Here's the rrdtool output.

$/usr/bin/rrdtool graph - \
> --imgformat=PNG \
> --start=-86400 \
> --end=-300 \
> --title="AS01-BBond-00 - Traffic - Gi0/3" \
> --rigid \
> --base=1000 \
> --height=120 \
> --width=500 \
> --alt-autoscale-max \
> --lower-limit=0 \
> --vertical-label="bits per second" \
> --slope-mode \
> --font TITLE:12: \
> --font AXIS:8: \
> --font LEGEND:10: \
> --font UNIT:8: \
> DEF:a="/var/www/cacti/rra/as01-bbond-00_traffic_in_7606.rrd":traffic_in:AVERAGE \
> DEF:b="/var/www/cacti/rra/as01-bbond-00_traffic_in_7606.rrd":traffic_out:AVERAGE \
> CDEF:cdefa=a,8,* \
> CDEF:cdeff=b,8,* \
> CDEF:cdefbe=a,UN,INF,UNKN,IF \
> CDEF:cdefbh=TIME,1273044152,GT,a,a,UN,0,a,IF,IF,TIME,1273044152,GT,b,b,UN,0,b,IF,IF,+ \
> AREA:cdefa#00CF00FF:"Inbound" \
> GPRINT:cdefa:LAST:" Current\:%8.2lf%s" \
> GPRINT:cdefa:AVERAGE:"Average\:%8.2lf%s" \
> GPRINT:cdefa:MAX:"Maximum\:%8.2lf%s\n" \
> COMMENT:"Transfer\: 2.38 GB" \
> LINE1:cdeff#002A97FF:"Outbound" \
> GPRINT:cdeff:LAST:"Current\:%8.2lf%s" \
> HRULE:|95\:bits\:0\:current|#00FF00FF:"95%\:" \
> COMMENT:"|95\:bits\:6\:current| mbit\n" \
> GPRINT:cdeff:AVERAGE:"Average\:%8.2lf%s" \
> GPRINT:cdeff:MAX:"Maximum\:%8.2lf%s\n" \
> COMMENT:" \n" \
> HRULE:4533291.68#FF0000FF:"95th Percentile" \
> COMMENT:"(4.53 mbit in+out)" \
> AREA:cdefbe#8F9286FF:"" \
> COMMENT:"Transfer\: 11.83 GB" \
> HRULE:|95\:bits\:0\:current|#0000FFFF:"95%\:" \
> COMMENT:"5.36 mbit" \
> COMMENT:"Total Transfer\: 14.21 GB\n"
-bash: 95:bits:0:current: command not found
-bash: COMMENT:|95\:bits\:6\:current| mbit\n: command not found
-bash: 95:bits:0:current: command not found
-bash: COMMENT:5.36 mbit: command not found
ERROR: Expected some arguments after 'HRULE:'

Post by **gandalf** » Wed May 05, 2010 5:50 am

Ok, that helps.
The HRULE statement is broken. Cacti supports some custom calculations found between "|...|". They are not executed as can be seen from the graph statement and the error message.
This happens on new graphs only? You are surely running other graphs using the same 95th percentile Graph Template successfully?
Did you change any Cacti code after upgrade?
R.

DervMan · Post by **DervMan** » Wed May 05, 2010 6:00 am

Graphs I setup before the upgrade to Version 0.8.7e worked fine. The vast majority of devices I monitor are Cisco switches for which we use 95th percentile so there's literally hundreds of graphs that continue to work.

I have doubled checked the old 95th percentile graphs are still working.

I haven't knowingly changed any code as I don't have the ability, although I did (re)install the thold plugin.

Post by **gandalf** » Wed May 05, 2010 6:04 am

Which version of rrdtool are you running (rrdtool --version)?
Which version is configured under "Settings -> RRDTool Utility Version"?

Please review the 95th percentile Graph Template. Select the HRule Graph Item and post a screenshot of this one.

R.

DervMan · Post by **DervMan** » Wed May 05, 2010 6:32 am

$ rrdtool --version
RRDtool 1.2.23 Copyright 1997-2007 by Tobias Oetiker <tobi@oetiker.ch>
Compiled Jun 7 2007 00:19:10

"settings" - RRDTool 1.2.x

There are a few references to HRule, the one I've attached seems most relevant!

DervMan · Post by **DervMan** » Wed May 05, 2010 7:34 am

This doesn't look right to me.....

DervMan · Post by **DervMan** » Tue May 11, 2010 5:08 am

Any suggestions?

95th percentile problem

95th percentile problem

Who is online