Strange behavior for LAST

[Damon]

Hello, I've tried looking for this bug on the forums and in the bug tracker but haven't had much luck -- mostly because I'm not sure how to describe it, or if it's even a bug (maybe I'm doing something wrong and don't know it, etc.). I have a traffic aggregation graphs which total up traffic from multiple sources into a single graph. While there are different variations of it, they all have this similar behavior:

In the console area of Cacti, on the Graph Management page, if I select the graph it displays the LAST values correctly, as the most recent value on the graph. In the presentation/graph area of Cacti, however, these graphs display with 0.00 for both inbound and outbound. Sometimes but not always, there is also a sharp downward drop on the graphs associated with this -- which is never present in the console view. I have attached images saved at more or less the same time, one from the console and one from the graphing area. This is the same graph presumably being generated with the same settings/information. I am also providing the debug output from the graph management page.

Code: Select all

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-300 \
--title="Aggregate External Traffic" \
--base=1000 \
--height=120 \
--width=500 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label="" \
--slope-mode \
DEF:a="[path removed].rrd":traffic_in:AVERAGE \
DEF:b="[path removed].rrd":traffic_in:AVERAGE \
DEF:c="[path removed].rrd":traffic_in:AVERAGE \
DEF:d="[path removed].rrd":traffic_in:AVERAGE \
DEF:e="[path removed].rrd":traffic_in:AVERAGE \
DEF:f="[path removed].rrd":traffic_in:AVERAGE \
DEF:g="[path removed].rrd":traffic_in:AVERAGE \
DEF:h="[path removed].rrd":traffic_in:AVERAGE \
DEF:i="[path removed].rrd":traffic_in:AVERAGE \
DEF:j="[path removed].rrd":traffic_in:AVERAGE \
DEF:ba="[path removed].rrd":traffic_in:AVERAGE \
DEF:bb="[path removed].rrd":traffic_in:AVERAGE \
DEF:bc="[path removed].rrd":traffic_in:AVERAGE \
DEF:bd="[path removed].rrd":traffic_in:AVERAGE \
DEF:be="[path removed].rrd":traffic_out:AVERAGE \
DEF:bf="[path removed].rrd":traffic_out:AVERAGE \
DEF:bg="[path removed].rrd":traffic_out:AVERAGE \
DEF:bh="[path removed].rrd":traffic_out:AVERAGE \
DEF:bi="[path removed].rrd":traffic_out:AVERAGE \
DEF:bj="[path removed].rrd":traffic_out:AVERAGE \
DEF:ca="[path removed].rrd":traffic_out:AVERAGE \
DEF:cb="[path removed].rrd":traffic_out:AVERAGE \
DEF:cc="[path removed].rrd":traffic_out:AVERAGE \
DEF:cd="[path removed].rrd":traffic_out:AVERAGE \
DEF:ce="[path removed].rrd":traffic_out:AVERAGE \
DEF:cf="[path removed].rrd":traffic_out:AVERAGE \
DEF:cg="[path removed].rrd":traffic_out:AVERAGE \
DEF:ch="[path removed].rrd":traffic_out:AVERAGE \
CDEF:cdefbe=a,UN,0,a,IF,b,UN,0,b,IF,+,c,UN,0,c,IF,+,d,UN,0,d,IF,+,
-> e,UN,0,e,IF,+,f,UN,0,f,IF,+,g,UN,0,g,IF,+,h,UN,0,h,IF,+,i,UN,0,i,IF,+,
-> j,UN,0,j,IF,+,ba,UN,0,ba,IF,+,bb,UN,0,bb,IF,+,bc,UN,0,bc,IF,+,
-> bd,UN,0,bd,IF,+,8,* \
CDEF:cdefdc=be,UN,0,be,IF,bf,UN,0,bf,IF,+,bg,UN,0,bg,IF,+,
-> bh,UN,0,bh,IF,+,bi,UN,0,bi,IF,+,bj,UN,0,bj,IF,+,ca,UN,0,ca,IF,+,
-> cb,UN,0,cb,IF,+,cc,UN,0,cc,IF,+,cd,UN,0,cd,IF,+,ce,UN,0,ce,IF,+,
-> cf,UN,0,cf,IF,+,cg,UN,0,cg,IF,+,ch,UN,0,ch,IF,+,8,* \
AREA:a:""  \
AREA:b:""  \
AREA:c:""  \
AREA:d:""  \
AREA:e:""  \
AREA:f:""  \
AREA:g:""  \
AREA:h:""  \
AREA:i:""  \
AREA:j:""  \
AREA:ba:""  \
AREA:bb:""  \
AREA:bc:""  \
AREA:bd:""  \
AREA:cdefbe#7CB3F1:"Inbound"  \
GPRINT:cdefbe:LAST:"Current\:%8.2lf %s"  \
GPRINT:cdefbe:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:cdefbe:MAX:"Maximum\:%8.2lf %s\n"  \
LINE1:be:""  \
LINE1:bf:""  \
LINE1:bg:""  \
LINE1:bh:""  \
LINE1:bi:""  \
LINE1:bj:""  \
LINE1:ca:""  \
LINE1:cb:""  \
LINE1:cc:""  \
LINE1:cd:""  \
LINE1:ce:""  \
LINE1:cf:""  \
LINE1:cg:""  \
LINE1:ch:""  \
LINE1:cdefdc#0D006A:"Outbound"  \
GPRINT:cdefdc:LAST:"Current\:%8.2lf %s"  \
GPRINT:cdefdc:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:cdefdc:MAX:"Maximum\:%8.2lf %s\n"

I upgraded to Cacti 0.8.6i today (from 0.8.6h) as well as to Cactid 0.8.6i (from Cactid 0.8.6g) and to RRDTool 1.2.15 (from RRDTool 1.2.13) but this issue continues to show up (present before and after the upgrades with no change). I'm reasonably sure it has to be an issue/variation in how Cacti utilizes RRDTool to generate the graph between the different views.

Edit: Broke CDEF lines to stop horizontal spanning of the page...

[gandalf]

There's an error with you CDEF, I suppose. Lately, we had a very similar problem. Please pay attention, that the timespan used for both graphs differ slightly. From console, I suppose end time is set to -300 and from GRAPHS, I suppose it's set to 0
Reinhard

[Damon]

I don't think the graphs in the management area specify a timespan -- only the graphs in the graph view do (visible from the URL used for the images). I'm not sure what would be the error with the CDEFs -- it doesn't seem to appear without the IF statements meant to remove invalid values. Does the LAST function go back to the most recent actual data if there isn't any current data present? The CDEF is likely substituting in zeroes for the end of the graph where no data has been collected, but it would seem more aesthetic if the graph didn't continue past the point it had data. The CDEF itself is for aesthetics also -- the summation of the various values fails if even a single value is missing, resulting in breaks in the graph (hence the substitution of zero for unknown values).

[gandalf]

... Does the LAST function go back to the most recent actual data if there isn't any current data present?...

No. Please copy that update statement from graph management and vary the --end timestamps. Replace the first - sign by a valid png file name to hold the graph. So you may try to verify what you're doing.

hth
Reinhard

[Damon]

Sorry, you lost me... could you be more specific?

[gandalf]

You see your very LOOONG code snippet in your first post? I was referring to that. Cacti simply streams this to an rrdtool pipe to perform the magic. You may do so from command line manually; and this is exactly what I was referring to. But replace

Code: Select all

/usr/bin/rrdtool graph - \ 

by

Code: Select all

/usr/bin/rrdtool graph /tmp/rubbish.png \ 

leaving the rest of it as is. Best way is to copy all of this, paste it into some shell file to re-execute as you change --end or your cdef until all is fine
Reinhard

[Damon]

Sorry if I seem obstinate, but I'm not sure what that would accomplish -- the command came from the debugging in the graph management view and that graph is fine, I don't think it needs adjustment so long as that's the code used to generate the graph there. The problem graph is the one in the graph view: it can display correctly, but only if I manually adjust the timestamps in the URL to it back in time. I'm unable to determine a specific timeframe that this works at, but 300 seconds seems to always work, i.e.

from...
graph_image.php?graph_start=1167171274&graph_end=1167257674

to:
graph_image.php?graph_start=1167170974&graph_end=1167257374

I think the CDEF could be made to work more reliably if it would, instead of giving back 0, it gave UN if (and only if) all of the values are unknown, but I'm not sure how to get it to do that -- I'm reasonably sure that's the issue now, but RPN expressions aren't entirely my forte. Any ideas?

Edit: This is the logic of my RPN expressions as I understand it. Looks like I exchanged two problems for a new one (from normal summation to summation with the IF statements):

a,b,+
a = 2, b = 3, RESULT: 5
a = 2, b = U, RESULT: U <-- This is bad...
a = U, b = 3, RESULT: U <-- This is bad...
a = U, b = U, RESULT: U

a,UN,0,a,IF,b,UN,0,b,IF,+
a = 2, b = 3, RESULT: 5
a = 2, b = U, RESULT: 2
a = U, b = 3, RESULT: 3
a = U, b = U, RESULT: 0 <-- Now this is bad instead...

I thought perhaps this might work, but I get an RPN underflow error with Cacti when using it:

a,UN,0,a,IF,b,UN,0,b,IF,+,0,UN,1,IF,*
[RPN underflow error, does not evaluate]

I see why it doesn't work -- it no longer includes what was in the stack (replaces it with 1, then attempts to multiply 1 with nothing, hence the underflow). How would you back reference the stack in the IF statement? i.e. if(STACK = 0) then(UN) else(STACK) -- but stack isn't a standalone variable.

[gandalf]

Sorry if I seem obstinate, but I'm not sure what that would accomplish -- the command came from the debugging in the graph management view and that graph is fine, I don't think it needs adjustment so long as that's the code used to generate the graph there. The problem graph is the one in the graph view: it can display correctly, but only if I manually adjust the timestamps in the URL to it back in time. I'm unable to determine a specific timeframe that this works at, but 300 seconds seems to always work, i.e.

from...
graph_image.php?graph_start=1167171274&graph_end=1167257674

to:
graph_image.php?graph_start=1167170974&graph_end=1167257374

Yep, you're on the right way. The only difference betwenn graph management and graph tree is that the first one has an offset of -300 for --end! And your CDEF needs this offset to get rid of the NaN/UNKNOWN stuff.
To handle the stack, please use PUSH and POP
Reinhard

[Damon]

I can't seem to find any references on how to use PUSH within an RPN expression and POP only clears the stack -- doesn't seem like that would be helpful with the small amount of information I've been able to find about how it works. Incidentally I tried this:

a,UN,0,a,IF,b,UN,0,b,IF,+,0,UN,1,IF,a,UN,0,a,IF,b,UN,0,b,IF,+,*

I expected it to sum the values with my IF statements, if the total result was zero, to push UN onto the stack, otherwise push 1, then sum the values again, multiplying them by the UN or 1 at the end. Instead my graphs just show 0 across the entire timespan (RRDTool doesn't show any error with the CDEF and I'm not sure why it didn't work).

[gandalf]

a,UN,0,a,IF,b,UN,0,b,IF,+
a = 2, b = 3, RESULT: 5
a = 2, b = U, RESULT: 2
a = U, b = 3, RESULT: 3
a = U, b = U, RESULT: 0 <-- Now this is bad instead...

Let's try it the other way round. What are you aiming at? What result do you want in the above case? If all you need is summation, why not trying cacti's builtin variables for bandwitdh summation, see http://www.cacti.net/downloads/docs/htm ... _VARIABLES or for the CDEF the ALL_DATA_SOURCES_NODUPS (do not remember the notation exactly)
Reinhard

[Damon]

I believe the following is the result I'm looking for:

a = 2, b = 3, RESULT: 5
a = 2, b = U, RESULT: 2
a = U, b = 3, RESULT: 3
a = U, b = U, RESULT: U

But I'm mostly interested in having functional LAST values in the graph. Cacti's built-in summation functions never worked out for me, so I just wrote the CDEF's myself to do what I wanted and got it working (with the gaps for unknowns). Now I've gotten rid of the gaps in the graphs and picked up this strange problem with LAST.

I would presume from the name of the Cacti function that it would sum all of the data sources to the graph, there are two sets of sources in these graphs (inbound and outbound) which need to be summed independently of one another.

[gandalf]

Just try to ALL_DATA_SOURCES_NODUPS,8,* as a new graph item to your graph template, e.g. graphed as a LINE1. Then visit Graph Management and select DEBUG. You'll see a nice CDEF that will serve as a start to create your own. I can't try it at time of writing, my test system stayed at work ...
Reinhard

[Damon]

I was correct, it does sum inbound and outbound together (undesirable effect). Cacti seems to use the timestamps as a means to avoid the unknowns at the end of the graph. This is the excerpt from the CDEF handling one variable:

TIME,1167335137,GT,a,a,UN,0,a,IF,IF

I'm aware that TIME pulls the current time, but I'm not sure where the timestamp came from (probably inserted that time of generation by Cacti).

[gandalf]

Next try: use SIMILAR_DATA_SOURCES_NODUPS? Never tried this on my own; please report your findings.
I'm currently on the run checking your CDEF problem, but as I told already, I do not have access to my system at time of writing. And from memory, this is really a task!
Reinhard

[gandalf]

Now I've found the reference I've talked about earlier. The solution may not be exactly what you're searching for, but see http://forums.cacti.net/viewtopic.php?t=18524
Reinhard