[SOLVED] Graphs not displaying all data - rrd oddities

[becketts]

Hi all, i have a curly one here that i've been working on, and times running a little short as some of our network admins need this in an working asap.

I've done some searches to see if i can find something similar but so far only infor regarding people using counter instead of gauge and that resulting in data stored in last_ds instead of value. i have no idea if that is relevant or not but i can confirm that my settings have always been set to gauge.

Exact details are below, if need be i can post copies of rrd's etc upon request.

Problem: Graphs to show cisco ace context stats are.not being generated with the correct numbers.
to be more specific. Each graph is meant to be made up of the following Limit_max, Current, Peak, denied and limit_min.
All ace graphs are not populating one or more of the above values. For this example i'll list the details for "FRONTEND-syslog rate" as its input values are currently static (nice quiet source for testing. the values coming from the ace module are as follows
limit_max:0 limit_min:45000 Peak:300 denied:0 Current:1
The graph is missing values for Limit_min and Peak.

Evidence:
Poller cache shows correct commands and commands run on cli returns the correct values.
Cacti log file shows commands being run and returning correct values.
Cacti log with debug turned on shows the following for the rrd update...

Code: Select all

rrdtool update cacti-0.8.7e/rra/682/19092.rrd --template limit_max:limit_min:peak:denied:current 1256850930:0:45000:300:0:1

when run, this command returns no errors.

rrdtool lastupdate returns as follows...

Code: Select all

rrdtool lastupdate cacti-0.8.7e/rra/682/19092.rrd 
 limit_max limit_min peak denied current

1256872837: 0 45000 300 0 0

rrdtool info reports the following, please note that i tihnk this is kinda interesting... the correct data for Peak is in last_ds, but not in value. Current on the other hand, shows results in last_ds AND value (this value tends to be 0-3 under normal circumstances)

Code: Select all

rrdtool info /opt/cacti-0.8.7e/rra/682/19092.rrd 
filename = "/opt/cacti-0.8.7e/rra/682/19092.rrd"
rrd_version = "0003"
step = 300
last_update = 1256873133
ds[limit_max].type = "GAUGE"
ds[limit_max].minimal_heartbeat = 600
ds[limit_max].min = 0.0000000000e+00
ds[limit_max].max = NaN
ds[limit_max].last_ds = "0"
ds[limit_max].value = 0.0000000000e+00
ds[limit_max].unknown_sec = 0
ds[limit_min].type = "GAUGE"
ds[limit_min].minimal_heartbeat = 600
ds[limit_min].min = 0.0000000000e+00
ds[limit_min].max = 1.0000000000e+02
ds[limit_min].last_ds = "45000"
ds[limit_min].value = NaN
ds[limit_min].unknown_sec = 33
ds[peak].type = "GAUGE"
ds[peak].minimal_heartbeat = 600
ds[peak].min = 0.0000000000e+00
ds[peak].max = 1.0000000000e+02
ds[peak].last_ds = "300"
ds[peak].value = NaN
ds[peak].unknown_sec = 33
ds[denied].type = "GAUGE"
ds[denied].minimal_heartbeat = 600
ds[denied].min = 0.0000000000e+00
ds[denied].max = 1.0000000000e+02
ds[denied].last_ds = "0"
ds[denied].value = 0.0000000000e+00
ds[denied].unknown_sec = 0
ds[current].type = "GAUGE"
ds[current].minimal_heartbeat = 600
ds[current].min = 0.0000000000e+00
ds[current].max = 1.0000000000e+02
ds[current].last_ds = "1"
ds[current].value = 3.3000000000e+01
ds[current].unknown_sec = 0

graph debug code is as follows.

Code: Select all

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title="NIGSHSACE2000C - Cisco Ace - CONTEXT-FRONTEND-syslog rate" \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label="" \
--slope-mode \
--font TITLE:12: \
--font AXIS:8: \
--font LEGEND:8: \
--font UNIT:8: \
DEF:a="/opt/cacti-0.8.7e/rra/682/19092.rrd":limit_max:LAST \
DEF:b="/opt/cacti-0.8.7e/rra/682/19092.rrd":peak:LAST \
DEF:c="/opt/cacti-0.8.7e/rra/682/19092.rrd":current:LAST \
DEF:d="/opt/cacti-0.8.7e/rra/682/19092.rrd":denied:LAST \
DEF:e="/opt/cacti-0.8.7e/rra/682/19092.rrd":limit_min:LAST \
AREA:a#C0C0C0FF:"Max Limit"  \
GPRINT:a:LAST:"Current\:%8.2lf %s\n"  \
LINE1:b#FF0000FF:"Peak"  \
GPRINT:b:LAST:"     Current\:%8.2lf %s"  \
GPRINT:b:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:b:MAX:"Maximum\:%8.2lf %s\n"  \
LINE1:c#00FF00FF:"Current"  \
GPRINT:c:LAST:"  Current\:%8.2lf %s"  \
GPRINT:c:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:c:MAX:"Maximum\:%8.2lf %s\n"  \
LINE1:d#EA8F00FF:"Denied"  \
GPRINT:d:LAST:"   Current\:%8.2lf %s"  \
GPRINT:d:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:d:MAX:"Maximum\:%8.2lf %s\n"  \
LINE1:e#4444FFFF:"Min Limit"  \
GPRINT:e:LAST:"Current\:%8.2lf %s\n" 

Technical stats below for completeness.

Code: Select all

General Information
Date 	Fri, 30 Oct 2009 14:32:56 +1100
Cacti Version 	0.8.7e
Cacti OS 	unix
SNMP Version 	NET-SNMP version: 5.3.2.2
RRDTool Version 	RRDTool 1.2.x
Hosts 	333
Graphs 	4812
Data Sources 	Script/Command: 343
SNMP: 5338
SNMP Query: 992
Script Query: 168
Script - Script Server (PHP): 5
Script Query - Script Server: 1206
Total: 8052
Poller Information
Interval 	300
Type 	spine
Items 	Action[0]: 7301
Action[1]: 1015
Action[2]: 2349
Total: 10665
Concurrent Processes 	1
Max Threads 	10
PHP Servers 	1
Script Timeout 	25
Max OID 	10
Last Run Statistics 	Time:79.0515 Method:spine Processes:1 Threads:10 Hosts:326 HostsPerProcess:326 DataSources:10665 RRDsProcessed:7455
PHP Information
PHP Version 	5.1.6
PHP OS 	Linux
PHP uname 	Linux monitor02.ni.news.com.au 2.6.18-128.4.1.el5PAE #1 SMP Tue Aug 4 20:58:34 EDT 2009 i686
PHP SNMP 	Installed
max_execution_time 	30
memory_limit 	256M

[gandalf]

the values coming from the ace module are as follows
limit_max:0 limit_min:45000 Peak:300 denied:0 Current:1
The graph is missing values for Limit_min and Peak.

Code: Select all

rrdtool update cacti-0.8.7e/rra/682/19092.rrd --template limit_max:limit_min:peak:denied:current 1256850930:0:45000:300:0:1

when run, this command returns no errors.

rrdtool lastupdate returns as follows...

Code: Select all

rrdtool lastupdate cacti-0.8.7e/rra/682/19092.rrd 
 limit_max limit_min peak denied current

1256872837: 0 45000 300 0 0

So lastupdate gets the "current" value wrong?
What does rrdtool fetch show?
R.

[becketts]

the values coming from the ace module are as follows
limit_max:0 limit_min:45000 Peak:300 denied:0 Current:1
The graph is missing values for Limit_min and Peak.

Code: Select all

rrdtool update cacti-0.8.7e/rra/682/19092.rrd --template limit_max:limit_min:peak:denied:current 1256850930:0:45000:300:0:1

when run, this command returns no errors.

rrdtool lastupdate returns as follows...

Code: Select all

rrdtool lastupdate cacti-0.8.7e/rra/682/19092.rrd 
 limit_max limit_min peak denied current

1256872837: 0 45000 300 0 0

So lastupdate gets the "current" value wrong?
What does rrdtool fetch show?
R.

Sorry no, my bad on this the lastupdate came from a subsequent run.
The main problem here is the peak value (300), it appears in the rrd and is retrieved from the script correctly, however the graphing command failed to render it. instead it appears as "0"

[becketts]

ok here some more info and stuff to play with.
Firstly, rrdtool fetch results in the following.....

Code: Select all

rrdtool fetch -s 1257110000 /opt/cacti/rra/682/19092.rrd AVERAGE
                      limit_max           limit_min                peak              denied             current

1257110100: 0.0000000000e+00 nan nan 0.0000000000e+00 8.9333333333e-01
1257110400: 0.0000000000e+00 nan nan 0.0000000000e+00 1.0000000000e+00
1257110700: 0.0000000000e+00 nan nan 0.0000000000e+00 1.1000000000e-01

Also attached is a dump and copy of the resulting graph.

So to recap. All data is generated from the scripts ok. Data is entered into rrd via what looks like the correct command, however all attempts to build a graph using this data fail as key DS's come up wrong, PEAK is the prime example as it should be 300 in all examples, however it comes up as nan or 0 in many graphs.

The only hint i have is that there is nothing stored in the value column of the rrd instead it looks like its stored in last_ds

[becketts]

I'm an idiot... my maximum limits were wrong... set to 100 by default so anything north of that wouldnt store and hence wouldnt render