Graphs Spinning Out of Control???

[windexh8er]

Anybody know of any good reason why a graph will suddenly spin out of control with crazy spikes? I've tried using killspike as well as rebuilding the poller cache to no avail... Obviously this link is not right, it was running fine for about 2 weeks and then I get this (see below).



It's annoying because the only way I can figure out how to fix it is to delete the RRD and start over. But then I lose any historical data. I think my version is 0.8.7a. I am using the CactiEz 0.3 install which I plan to upgrade to the latest of spine and Cacti, but would like to figure this out first.

TIA!

[gandalf]

Please post "rrdtool info" of the rrd file used.
Reinhard

[windexh8er]

You mean this?

-----------------------------------------------------------------------

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-14400 \
--end=-60 \
--title="XXXXXXXXXXXXX_CE_01 (xxx.xxx.xxx.xxx) - Ckt Util - Se1/0" \
--base=1000 \
--height=120 \
--width=550 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label="Percent" \
--slope-mode \
--font TITLE:10: \
--font AXIS:8: \
--font LEGEND:8: \
--font UNIT:8: \
DEF:a="/var/www/html/rra/XXXXX_ce_01_XXX_traffic_in_128.rrd":traffic_in:LAST \
DEF:b="/var/www/html/rra/XXXXX_ce_01_XXX_traffic_in_128.rrd":traffic_in:AVERAGE \
CDEF:cdefa=a,43324000,/,100,*,8,* \
CDEF:cdefd=b,8,* \
CDEF:cdefg=a,8,* \
CDEF:cdefi=b,43324000,/,100,*,8,* \
AREA:cdefa#002A8F:"" \
COMMENT:"CIRCUIT SPEED\: 43324000 bits per second\n" \
COMMENT:"INBOUND TRAFFIC RATE" \
GPRINT:cdefd:AVERAGE:" Avg\:%8.2lf%s" \
GPRINT:cdefd:MIN:"Min\:%8.2lf%s" \
GPRINT:cdefd:MAX:"Max\:%8.2lf%s" \
GPRINT:cdefg:LAST:"Current\:%8.2lf%s\n" \
COMMENT:"INBOUND TRAFFIC %" \
GPRINT:cdefi:AVERAGE:" Avg\:(%5.2lf %%)" \
GPRINT:cdefi:MIN:"Min\:(%5.2lf %%)" \
GPRINT:cdefi:MAX:"Max\:(%5.2lf %%)" \
GPRINT:cdefa:LAST:"Current\:(%5.2lf %%)\n"

[windexh8er]

I know the SNMP that I'm pulling from the router is correct because I pull the same data to build an In/Out bps graph with 95th percentile. And those are working fine for the same interface. It's really odd. I have a total of 30 graphs using this template (utilization in percent) and randomly one or two will do this out of the blue.

[windexh8er]

*bump*

Anybody?

[BSOD2600]

You mean this?

No, he said rrdtool info, you posted rrdtool graph.

Code: Select all

rrdtool info XXXXX_ce_01_XXX_traffic_in_128.rrd

Have you changed the cacti logging level to medium, then watched what data is returned for this device?

[windexh8er]

No, I haven't changed the logging yet. Here's what was originally requested though, thanks for the info:

filename = "/var/www/html/rra/****traffic_in_127.rrd"
rrd_version = "0003"
step = 60
last_update = 1205258190
ds[traffic_out].type = "COUNTER"
ds[traffic_out].minimal_heartbeat = 120
ds[traffic_out].min = 0.0000000000e+00
ds[traffic_out].max = 4.3324000000e+07
ds[traffic_out].last_ds = "1683561377"
ds[traffic_out].value = 3.3634891500e+07
ds[traffic_out].unknown_sec = 0
ds[traffic_in].type = "COUNTER"
ds[traffic_in].minimal_heartbeat = 120
ds[traffic_in].min = 0.0000000000e+00
ds[traffic_in].max = 4.3324000000e+07
ds[traffic_in].last_ds = "U"
ds[traffic_in].value = NaN
ds[traffic_in].unknown_sec = 30
rra[0].cf = "AVERAGE"
rra[0].rows = 10080
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[0].cdp_prep[1].value = NaN
rra[0].cdp_prep[1].unknown_datapoints = 0
rra[1].cf = "AVERAGE"
rra[1].rows = 25920
rra[1].pdp_per_row = 1
rra[1].xff = 5.0000000000e-01
rra[1].cdp_prep[0].value = NaN
rra[1].cdp_prep[0].unknown_datapoints = 0
rra[1].cdp_prep[1].value = NaN
rra[1].cdp_prep[1].unknown_datapoints = 0
rra[2].cf = "AVERAGE"
rra[2].rows = 2250
rra[2].pdp_per_row = 24
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = 9.0853986943e+07
rra[2].cdp_prep[0].unknown_datapoints = 0
rra[2].cdp_prep[1].value = NaN
rra[2].cdp_prep[1].unknown_datapoints = 20
rra[3].cf = "AVERAGE"
rra[3].rows = 9000
rra[3].pdp_per_row = 6
rra[3].xff = 5.0000000000e-01
rra[3].cdp_prep[0].value = 2.2010602610e+06
rra[3].cdp_prep[0].unknown_datapoints = 0
rra[3].cdp_prep[1].value = NaN
rra[3].cdp_prep[1].unknown_datapoints = 2
rra[4].cf = "AVERAGE"
rra[4].rows = 797
rra[4].pdp_per_row = 288
rra[4].xff = 5.0000000000e-01
rra[4].cdp_prep[0].value = 1.3571059488e+09
rra[4].cdp_prep[0].unknown_datapoints = 3
rra[4].cdp_prep[1].value = NaN
rra[4].cdp_prep[1].unknown_datapoints = 212
rra[5].cf = "MIN"
rra[5].rows = 10080
rra[5].pdp_per_row = 1
rra[5].xff = 5.0000000000e-01
rra[5].cdp_prep[0].value = NaN
rra[5].cdp_prep[0].unknown_datapoints = 0
rra[5].cdp_prep[1].value = NaN
rra[5].cdp_prep[1].unknown_datapoints = 0
rra[6].cf = "MIN"
rra[6].rows = 25920
rra[6].pdp_per_row = 1
rra[6].xff = 5.0000000000e-01
rra[6].cdp_prep[0].value = NaN
rra[6].cdp_prep[0].unknown_datapoints = 0
rra[6].cdp_prep[1].value = NaN
rra[6].cdp_prep[1].unknown_datapoints = 0
rra[7].cf = "MIN"
rra[7].rows = 2250
rra[7].pdp_per_row = 24
rra[7].xff = 5.0000000000e-01
rra[7].cdp_prep[0].value = 7.2194931148e+05
rra[7].cdp_prep[0].unknown_datapoints = 0
rra[7].cdp_prep[1].value = NaN
rra[7].cdp_prep[1].unknown_datapoints = 20
rra[8].cf = "MIN"
rra[8].rows = 9000
rra[8].pdp_per_row = 6
rra[8].xff = 5.0000000000e-01
rra[8].cdp_prep[0].value = 1.0983934314e+06
rra[8].cdp_prep[0].unknown_datapoints = 0
rra[8].cdp_prep[1].value = NaN
rra[8].cdp_prep[1].unknown_datapoints = 2
rra[9].cf = "MIN"
rra[9].rows = 797
rra[9].pdp_per_row = 288
rra[9].xff = 5.0000000000e-01
rra[9].cdp_prep[0].value = 4.0821838450e+05
rra[9].cdp_prep[0].unknown_datapoints = 3
rra[9].cdp_prep[1].value = NaN
rra[9].cdp_prep[1].unknown_datapoints = 212
rra[10].cf = "MAX"
rra[10].rows = 10080
rra[10].pdp_per_row = 1
rra[10].xff = 5.0000000000e-01
rra[10].cdp_prep[0].value = NaN
rra[10].cdp_prep[0].unknown_datapoints = 0
rra[10].cdp_prep[1].value = NaN
rra[10].cdp_prep[1].unknown_datapoints = 0
rra[11].cf = "MAX"
rra[11].rows = 25920
rra[11].pdp_per_row = 1
rra[11].xff = 5.0000000000e-01
rra[11].cdp_prep[0].value = NaN
rra[11].cdp_prep[0].unknown_datapoints = 0
rra[11].cdp_prep[1].value = NaN
rra[11].cdp_prep[1].unknown_datapoints = 0
rra[12].cf = "MAX"
rra[12].rows = 2250
rra[12].pdp_per_row = 24
rra[12].xff = 5.0000000000e-01
rra[12].cdp_prep[0].value = 1.8037682633e+07
rra[12].cdp_prep[0].unknown_datapoints = 0
rra[12].cdp_prep[1].value = NaN
rra[12].cdp_prep[1].unknown_datapoints = 20
rra[13].cf = "MAX"
rra[13].rows = 9000
rra[13].pdp_per_row = 6
rra[13].xff = 5.0000000000e-01
rra[13].cdp_prep[0].value = 1.1026668295e+06
rra[13].cdp_prep[0].unknown_datapoints = 0
rra[13].cdp_prep[1].value = NaN
rra[13].cdp_prep[1].unknown_datapoints = 2
rra[14].cf = "MAX"
rra[14].rows = 797
rra[14].pdp_per_row = 288
rra[14].xff = 5.0000000000e-01
rra[14].cdp_prep[0].value = 3.8442768750e+07
rra[14].cdp_prep[0].unknown_datapoints = 3
rra[14].cdp_prep[1].value = NaN
rra[14].cdp_prep[1].unknown_datapoints = 212
rra[15].cf = "LAST"
rra[15].rows = 10080
rra[15].pdp_per_row = 1
rra[15].xff = 5.0000000000e-01
rra[15].cdp_prep[0].value = NaN
rra[15].cdp_prep[0].unknown_datapoints = 0
rra[15].cdp_prep[1].value = NaN
rra[15].cdp_prep[1].unknown_datapoints = 0
rra[16].cf = "LAST"
rra[16].rows = 25920
rra[16].pdp_per_row = 1
rra[16].xff = 5.0000000000e-01
rra[16].cdp_prep[0].value = NaN
rra[16].cdp_prep[0].unknown_datapoints = 0
rra[16].cdp_prep[1].value = NaN
rra[16].cdp_prep[1].unknown_datapoints = 0
rra[17].cf = "LAST"
rra[17].rows = 9000
rra[17].pdp_per_row = 6
rra[17].xff = 5.0000000000e-01
rra[17].cdp_prep[0].value = 1.1026668295e+06
rra[17].cdp_prep[0].unknown_datapoints = 0
rra[17].cdp_prep[1].value = NaN
rra[17].cdp_prep[1].unknown_datapoints = 2
rra[18].cf = "LAST"
rra[18].rows = 2250
rra[18].pdp_per_row = 24
rra[18].xff = 5.0000000000e-01
rra[18].cdp_prep[0].value = 1.1026668295e+06
rra[18].cdp_prep[0].unknown_datapoints = 0
rra[18].cdp_prep[1].value = NaN
rra[18].cdp_prep[1].unknown_datapoints = 20
rra[19].cf = "LAST"
rra[19].rows = 797
rra[19].pdp_per_row = 288
rra[19].xff = 5.0000000000e-01
rra[19].cdp_prep[0].value = 1.1026668295e+06
rra[19].cdp_prep[0].unknown_datapoints = 3
rra[19].cdp_prep[1].value = NaN
rra[19].cdp_prep[1].unknown_datapoints = 212

[windexh8er]

*bump*

Anyone?

TIA!!!

[gandalf]

CDEF:cdefa=a,43324000,/,100,*,8,* \
CDEF:cdefd=b,8,* \
CDEF:cdefg=a,8,* \
CDEF:cdefi=b,43324000,/,100,*,8,* \

Why do you multiply the % Value by 8? That does not hit your primary problem, but worried me (I always wonder if % grow above 100%)
Reinhard

[gandalf]

ds[traffic_out].max = 4.3324000000e+07
ds[traffic_in].max = 4.3324000000e+07

This one is fine. Was aiming at MAX clipping ...
Reinhard

[gandalf]

rra[0].cf = "AVERAGE"
rra[0].rows = 10080
rra[0].pdp_per_row = 1

rra[1].cf = "AVERAGE"
rra[1].rows = 25920
rra[1].pdp_per_row = 1

rra[2].cf = "AVERAGE"
rra[2].rows = 2250
rra[2].pdp_per_row = 24

rra[3].cf = "AVERAGE"
rra[3].rows = 9000
rra[3].pdp_per_row = 6

rra[4].cf = "AVERAGE"
rra[4].rows = 797
rra[4].pdp_per_row = 288

rra[5].cf = "MIN"
rra[5].rows = 10080
rra[5].pdp_per_row = 1

rra[6].cf = "MIN"
rra[6].rows = 25920
rra[6].pdp_per_row = 1

rra[7].cf = "MIN"
rra[7].rows = 2250
rra[7].pdp_per_row = 24

rra[8].cf = "MIN"
rra[8].rows = 9000
rra[8].pdp_per_row = 6

rra[9].cf = "MIN"
rra[9].rows = 797
rra[9].pdp_per_row = 288

rra[10].cf = "MAX"
rra[10].rows = 10080
rra[10].pdp_per_row = 1

rra[11].cf = "MAX"
rra[11].rows = 25920
rra[11].pdp_per_row = 1

rra[12].cf = "MAX"
rra[12].rows = 2250
rra[12].pdp_per_row = 24

rra[13].cf = "MAX"
rra[13].rows = 9000
rra[13].pdp_per_row = 6

rra[14].cf = "MAX"
rra[14].rows = 797
rra[14].pdp_per_row = 288

rra[15].cf = "LAST"
rra[15].rows = 10080
rra[15].pdp_per_row = 1

rra[16].cf = "LAST"
rra[16].rows = 25920
rra[16].pdp_per_row = 1

rra[17].cf = "LAST"
rra[17].rows = 9000
rra[17].pdp_per_row = 6

rra[18].cf = "LAST"
rra[18].rows = 2250
rra[18].pdp_per_row = 24

rra[19].cf = "LAST"
rra[19].rows = 797
rra[19].pdp_per_row = 288

This one's weird. Definitively non-standard. Please post RRA table settings and describe, what you're aiming at. Not sure, if this one is related.
Did you already try second link of my sig?
Reinhard

[windexh8er]

RRA settings from the 'Data Sources'? I guess I'm unclear of what will help you help me.

What I'm trying to do is figure out why the other graphs are working (on similar links, same configuration) but then out of the blue one will start to spin out of control.

That and I have no idea what you meant by 'the second link of my sig'. I'm assuming it's something to do with your signature, but I don't see one in any of the posts. (EDIT: Now I see them, they were gone last time I looked)

Thanks for your help, hopefully I can get you the information to help figure this out!

[gandalf]

Select "Data Sources" from nav tree. Then select "RRAs" and post the whole table.
The Debug link should help you finding out, what data was retrieved by cacti and how it updated the rrd files
Reinhard

[windexh8er]

Name Steps Rows Timespan**
Hourly (1 Minute Average) 1 10080 14400
Daily (5 Minute Average) 1 25920 86400
Weekly (30 Minute Average) 6 9000 604800
Monthly (2 Hour Average) 24 2250 2592000
Yearly (1 Day Average) 288 797 33053184

[windexh8er]

So I looked through the debugging and I still have no idea why the problem is ocurring. Grrr... It's only one out of 20 graphs right now. I don't want to delete it and spin a new one because I feel like it'll happen again.

Any more thoughts gandalf?

TIA!