Inaccurate graphs without consolidation

[mete]

Hi all

First of all, thank for the nice documentation and work.
I read a few times in the documentation, but unfortunately I can't figure out, why this behaviour happens.

My RRAs look like this:

Hourly (1 Minute Average) 1 500 14400
Daily (5 Minute Average) 1 600 86400
Weekly (5 Minute Average) 1 2016 604800
Monthly (5 Minute Average) 1 8928 2678400
Yearly (5 Minute Average) 1 105120 31536000
3 Year (5 Minute Average) 1 315360 94608000
10 Year (10 Minute Average) 2 525600 315360000

In all RRAs I've enabled AVERAGE, MIN, MAX, LAST.
Now lets look at a weekly graph. Can someone tell me, why the two Lines (one with consolidation LAST, one with consolidation average) are showing different values?

graph_image.png (13.25 KiB) Viewed 673 times

Code: Select all

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start='1474476963' \
--end='1475081763' \
--title='TEST GRAPH 333' \
--base='1000' \
--height='120' \
--width='500' \
--alt-autoscale-max \
--lower-limit='100' \
--y-grid='50:1' \
COMMENT:"From 2016/09/21 18\:56\:03 To 2016/09/28 18\:56\:03\c" \
COMMENT:"  \n" \
--vertical-label='' \
--slope-mode \
--font TITLE:10: \
--font AXIS:7: \
--font LEGEND:8: \
--font UNIT:7: \
DEF:a='/usr/share/cacti/site/rra/RPI_REG_ina219_current_18.rrd':'ina219_current':LAST \
DEF:b='/usr/share/cacti/site/rra/RPI_REG_ina219_current_18.rrd':'ina219_current':AVERAGE \
LINE1:a#FF3932FF:'LAST'  \
LINE1:b#0000FFFF:'AVERAGE' 

As I understood, consolidation is only needed for RRAs with steps greater 1 (as my 10 Year RRA).

But, the daily graph is showing the same values for average and last:

graph_image (1).png (24.31 KiB) Viewed 671 times

Code: Select all

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start='1474995930' \
--end='1475082330' \
--title='TEST GRAPH 333' \
--base='1000' \
--height='120' \
--width='500' \
--alt-autoscale-max \
--lower-limit='100' \
--y-grid='50:1' \
COMMENT:"From 2016/09/27 19\:05\:30 To 2016/09/28 19\:05\:30\c" \
COMMENT:"  \n" \
--vertical-label='' \
--slope-mode \
--font TITLE:10: \
--font AXIS:7: \
--font LEGEND:8: \
--font UNIT:7: \
DEF:a='/usr/share/cacti/site/rra/RPI_REG_ina219_current_18.rrd':'ina219_current':LAST \
DEF:b='/usr/share/cacti/site/rra/RPI_REG_ina219_current_18.rrd':'ina219_current':AVERAGE \
LINE1:a#FF3932FF:'LAST'  \
LINE1:b#0000FFFF:'AVERAGE' 

So my question is:
Is it possible to disable the consolidation in cacti complety, or for selected RRAs?

Thank you very much
Best, Mete

[cigamit]

All consolidation is done by RRDTool itself. RRDTool does consolidation based upon various things, the RRAs you select are only part of it. Even if you setup a RRA perfectly so that you think no consolidation should be done, the problem comes down to is there even "pixel space" to show every data point that you requested. For example, say you only have a single RRA that stores data at 1 minute intervals for 1 year. If you only have 300 pixels of graph display, yet to display a weekly graph, you have 7 days of data at 1 minute intervals (60 * 24 * 7) = 10080 data points. So it has to consolidate ~33 data points into 1 pixel. Depending on the consolidation function you are displaying (LAST or AVERAGE). AVERAGE does an average of those 33 data points and displays it. LAST will show the last data point in that set (so data point #33).

That is why some people choose to use MAX instead of AVERAGE or LAST. That can have its own issues though depending on the graph (example, CPU usage) and can then lead into the whole AVERAGE of MAX scenario, etc... You then also have to take into account other RRAs. So if RRDTool deems that the timespan you selected better fits your weekly RRA, then may be getting an AVERAGE of an AVERAGE if it needs to consolidate multiple datapoints from that RRA into a single pixel (not in your case since your RRAs are nonstandard).

This may explain it a little better than I can.
http://rrdtool.vandenbogaerdt.nl/process.php
http://rrdtool.vandenbogaerdt.nl/min-avg-max.php

I think the way you are trying to do RRAs is defeating the entire purpose of consolidation and creating huge disk I/O and CPU load (yet it will still do consolidation because of the pixel issue, it just has to do it on the fly with more data points!!!). Basically all the RRAs are than redundant of the last one. Since you are polling at 1 minute intervals, it is basically telling it that each polling to store fresh data for every single RRA. This means you are doing 7 writes every time to that RRD (which adds some I/O) and the RRD is going to be extremely huge. The only real thing different between the RRAs in your setup is the MRGT view which displays the different time intervals. The main problem is any graph that doesn't display exactly the same data points as pixel is going to be consolidating A LOT of data on the fly which means a lot of disk IO to read that much data, and a lot of CPU to consolidate the data on the fly so it can display it. For instance, your 10 Year graph will consolidate 525,600 data points on the fly to fit within the specified pixels.

Also, your RRAs seem to be setup incorrectly for 1 minute polling. If your running with a step of 60 for a data template, your Daily graph is only going to be storing 6 hours worth of information (that 10 year only has 2 years of data). You should have a separate set of RRAs for 1 minute polling (60s step data template) vs 5 minute polling. We kinda fixed this in Cacti 1.0, as now the RRAs are in a Profile Group and the Step (polling period) is specified in there instead of the data template.

[mete]

Thank you very much, I got the point.