RESOLVED: graphs aren't matching rrd LAST, not even close

[windsor]

I set up cacti with the snmp_oid data template to monitor the ASA for the number of VPN (AnyConnect) connections.

Everything works well on the data collection side:
OID: 1.3.6.1.4.1.9.9.392.1.3.35.0
Data Source Type: GAUGE (which matches what "snmpget" returns -- Gauge32)
Maximum Value: U

On the Data Source, kick on "Data Source Info Mode" and the "Last DS" value matches what snmpget says and is +/- 1 from "show vpn-sessiondb summary" tells me in CLI, which is "close enough" for my work here.

After a day of cranking away, that number also matches the output of: rrdtool lastupdate /path/to/cacti/rra/mydevice_snmp_oid_555.rrd

So we know the rrd has the right information.

This morning, the graph looked pretty flat, didn't have the usual early morning rise.

I checked CLI, 395
I checked snmpget, 395
I checked rra, 395
graph: 113 and change for "Current" (at the time)

The graph is slowly rising, but still nowhere near the real count for "right now."



(Yes, I verified it's the same RRD file. )

[Dummy]

Can you share the Debug - Information of this graph?
Maybe this will give more background information for troubleshooting.

[windsor]

Sure, I'll collect it all again tomorrow morning when usage ramps up.

[windsor]

Here's the current graph:


snmpget on the CLI returns 378

Data source information confirms this:


data source debug output:

Code: Select all

Data Source Debug

/usr/bin/rrdtool create \
/srv/www/cacti/rra/_________snmp_oid_387.rrd \
--start -900 --step 300  \
DS:snmp_oid:GAUGE:600:0:U \
RRA:AVERAGE:0.5:1:600 \
RRA:AVERAGE:0.5:6:700 \
RRA:AVERAGE:0.5:24:775 \
RRA:AVERAGE:0.5:288:797 \
RRA:MIN:0.5:1:600 \
RRA:MIN:0.5:6:700 \
RRA:MIN:0.5:24:775 \
RRA:MIN:0.5:288:797 \
RRA:MAX:0.5:1:600 \
RRA:MAX:0.5:6:700 \
RRA:MAX:0.5:24:775 \
RRA:MAX:0.5:288:797 \
RRA:LAST:0.5:1:600 \
RRA:LAST:0.5:6:700 \
RRA:LAST:0.5:24:775 \
RRA:LAST:0.5:288:797 \

graph debug output:

Code: Select all

RRDtool Command:

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start='-86400' \
--end='-300' \
--pango-markup  \
--title='________ - VPN Clients' \
--vertical-label='VPN clients' \
--slope-mode \
--base=1000 \
--height=200 \
--width=700 \
--tabwidth '40' \
--alt-autoscale-max \
--lower-limit='0' \
COMMENT:"From 2020/03/22 14\:20\:00 To 2020/03/23 14\:15\:00\c" \
COMMENT:"  \n" \
--color BACK#F3F3F3 \
--color CANVAS#FDFDFD \
--color SHADEA#CBCBCB \
--color SHADEB#999999 \
--color FONT#000000 \
--color AXIS#2C4D43 \
--color ARROW#2C4D43 \
--color FRAME#2C4D43 \
--border 1 --font TITLE:11:'Arial' \
--font AXIS:8:'Arial' \
--font LEGEND:8:'Courier' \
--font UNIT:8:'Arial' \
--font WATERMARK:6:'Arial' \
--slope-mode \
--watermark 'Generated by Cacti®' \
DEF:a='/srv/www/cacti/rra/_________snmp_oid_387.rrd':'snmp_oid':MAX \
AREA:a#00BD27FF:'users'  \
GPRINT:a:LAST:'Current\:%8.2lf %s'  \
GPRINT:a:AVERAGE:'Average\:%8.2lf %s'  \
GPRINT:a:MAX:'Maximum\:%8.2lf %s\n'

RRDtool Says:

OK

[Dummy]

Hmm, looks all fine. One thing is confusing to me:
I understand, in your first post, that you are capturing your data inside "rrdtool lastupdate /path/to/cacti/rra/mydevice_snmp_oid_555.rrd"
Your graph, in your last post, was retrieving his data out of "... /srv/www/cacti/rra/_________snmp_oid_387.rrd':'snmp_oid ... "
If this is just a misunderstanding by me just ignore my comment.

So, if you get the correct values by snmpwalk then cacti should graph the values correct. Your DataSource is assigned to the same device on which you compare the data with snmpwalk?

[windsor]

good catch, let me check on that.

[windsor]

Oh, I think the original '555' was me trying to abstract the data before posting it. I don't have a *555.rrd, highest I have is something_406.rrd.

[Dummy]

And the device, which is assigned to your DataSource, is the same on which you are running snmpwalk manually?

[windsor]

And the device, which is assigned to your DataSource, is the same on which you are running snmpwalk manually?

Certainly. Output from the data source matches the manual snmpget.

[windsor]

What's interesting is that it appears to be time-shifted. For example, right now:

snmp and "DS" value: 273

graph Current: 434

I made sure the "Preset" wasn't messed up as well, using "Last Day" or even "Last Hour", still wrong.

[windsor]

I ran a while loop to do the "snmp get" every five minutes and just let it hum overnight and during the day.

Here's the graph and the output for comparison.


One point to easily identify on the graph, it crosses 400-count right at 2:30p (14:30).



The text output shows that happening at about 10:30a (yellow box).

That's a 4hr lag in the graph display.

Coincidentally, EDT is UTC-4.

Sounds like cacti/rrdtool believes the information in the rra is UTC instead of EDT.

How do we check that?

[windsor]

found this: viewtopic.php?t=47931

Adjusting my php.ini (both of them) and restarting apache2.

[windsor]

Yup, that was the solution.

needed correct timezone in both of the php.ini files.

FWIW, the OS is SLES 15, this was a "greenfield" install using the latest stable version at the time (1.2.7), installed via zypper.

[Dummy]

Great. Thank you for sharing your solution