How can I prevent an RRD source from being smoothened?

[marcmerlin]

I have a problem. I'm using a data source as a bit field to keep track of fan or HVAC status in my house.
bit 1: hvac heat, bit 2: hvac cool, bit 7: computer fan closet on.

So my raw data looks like 129, 128, 2, 0 and so forth.

Unfortunately when I go from 128 to 0, rrdtool tries to be helpful and smoothens my data, so I end up with intermediate values like 66.13:
http://graphs.merlins.org/graphs/g.php? ... nt_size=12

Can I tell cacti/rrdtool to jump from 128 to 0 and not smoothen values?

For now, I get garbage like this:
https://graphs.merlins.org/graphs/g.php ... view_type=
(the lines in the graphs are due to the induced error above).

If you're curious, I use this CDEF for each bit I want to display:
Custom String: CURRENT_DATA_SOURCE,8,/,FLOOR,2,%,1,EQ,INF,UNKN,IF

When things actually work (here it's bit 7, 128 so I don't get the average rouding bug), here's what it looks like:
https://graphs.merlins.org/graphs/g.php ... nt_size=12

Marc

[gandalf]

There are several issues to discuss.

1. Why do you report different data items as bits of a number? This is really not a good approach with rrdtool due to different "consolidation issues". I really recommend splitting them at the source.

2. Still "smoothing effects" may occur and are part of rrdtool's feature list. See e.g. http://docs.cacti.net/manual:087:8_rrdt ... solidation if not already done

3. To discuss your issue, it is important to know which polling interval was used and which rrdtool parameters were defined on the rrd file (e.g. STEP and HEARTBEAT). Please use "rrdtool info" to fetch data

This may take some days to discuss, please be prepared
R.

[marcmerlin]

There are several issues to discuss.

1. Why do you report different data items as bits of a number? This is really not a good approach with rrdtool due to different "consolidation issues". I really recommend splitting them at the source.

2. Still "smoothing effects" may occur and are part of rrdtool's feature list. See e.g. http://docs.cacti.net/manual:087:8_rrdt ... solidation if not already done

3. To discuss your issue, it is important to know which polling interval was used and which rrdtool parameters were defined on the rrd file (e.g. STEP and HEARTBEAT). Please use "rrdtool info" to fetch data

This may take some days to discuss, please be prepared
R.

1) I'm using data bits because my RRD file is already 2GB due to the amount of data sources I have, and how long I keep the data, and I can't afford to lose 64 bits for every one bit of status data I need to keep track

2) Yes, I know rrdtool smoothens data as a feature, I was looking at how to turn it off if it's possible

3) My polling interval is 1mn

filename = "housetemp_17.rrd"
rrd_version = "0003"
step = 60
last_update = 1281902944
header_size = 32124
(...)
ds[Full_HVAC_Status].index = 43
ds[Full_HVAC_Status].type = "GAUGE"
ds[Full_HVAC_Status].minimal_heartbeat = 600
ds[Full_HVAC_Status].min = 0.0000000000e+00
ds[Full_HVAC_Status].max = 9.2233720369e+17
ds[Full_HVAC_Status].last_ds = "0"
ds[Full_HVAC_Status].value = 0.0000000000e+00
ds[Full_HVAC_Status].unknown_sec = 0

Marc

[gandalf]

1) I'm using data bits because my RRD file is already 2GB due to the amount of data sources I have, and how long I keep the data, and I can't afford to lose 64 bits for every one bit of status data I need to keep track

Why do you put all data source items into a single rrd file? It would be more flexible to split them up. And then, 2 GB is not an issue.
And more; due to some "Problems" when using 1 min polling, please post complete rrdtool info output. I've seen many of them holding superfluous duplication of data.

2) Yes, I know rrdtool smoothens data as a feature, I was looking at how to turn it off if it's possible

I fear that you will have to deal with that. You can't switch it off, verbosely. There's no such option in rrdtool

3) My polling interval is 1mn

filename = "housetemp_17.rrd"
rrd_version = "0003"
step = 60
last_update = 1281902944
header_size = 32124
(...)
ds[Full_HVAC_Status].minimal_heartbeat = 600

IMHO, it makes no sense to have HEARTBEAT = 10*step. In most cases 1.5 to 2.5 is ok. If more time than "heartbeat" passes by between two rrdtool updates, rrdtool consideres the data to be NaN (= missing data)
R.

[drewshin99]

unfortunately, it looks as if with all "GAUGE" data sources, we have to live with the fact that rrdtool, automatically smooths this out. reading over at the rrdtool mailing list, tobi doesnt think this is an issue.

the thinking is if instead of say 300 seconds between update 1 and update 2, if instead 315 seconds elapses, rrdtool calculates a 5% (15 divided by 300) difference and adjusts your output by that amount. I can understand doing this for traffic/error stats as you're trying to get a rate per second....but I dont understand why we want this for "GAUGE" type data as this type of data is not time dependent....I'm seeing this for my Ping graphs, CPU graphs, memory graphs, they're all a little off, or RRDTOOL will split the results into two data points (for instance, in my ping graphs, instead of showing 20% loss, it will show 19% loss in one point, and 1% loss on the point after that).

the only way to fix this is to somehow flag all gauge data sources and tell cacti , "if this is a gauge data source, adjust the timestamp for the value to the nearest 300 second interval"....otherwise rrdtool will keep adjusting the values itself...hope that made sense and is accurate. is this something we can request in a feature request? or is it too difficult to accomplish? thx

[marcmerlin]

1) I'm using data bits because my RRD file is already 2GB due to the amount of data sources I have, and how long I keep the data, and I can't afford to lose 64 bits for every one bit of status data I need to keep track

Why do you put all data source items into a single rrd file? It would be more flexible to split them up. And then, 2 GB is not an issue.
And more; due to some "Problems" when using 1 min polling, please post complete rrdtool info output. I've seen many of them holding superfluous duplication of data.

2) Yes, I know rrdtool smoothens data as a feature, I was looking at how to turn it off if it's possible

I fear that you will have to deal with that. You can't switch it off, verbosely. There's no such option in rrdtool

3) My polling interval is 1mn

filename = "housetemp_17.rrd"
rrd_version = "0003"
step = 60
last_update = 1281902944
header_size = 32124
(...)
ds[Full_HVAC_Status].minimal_heartbeat = 600

IMHO, it makes no sense to have HEARTBEAT = 10*step. In most cases 1.5 to 2.5 is ok. If more time than "heartbeat" passes by between two rrdtool updates, rrdtool consideres the data to be NaN (= missing data)
R.

3) I like heartbeat to be 10mn for my gauge data because sometimes I stop data gathering for a few minutes, my temps don't change up and down a whole lot and I'd rather have a straight line than a hole.

2) argh, no way to turn it off, darn Thanks.

1) Sure, here's the full paste, I avoided it so far because it's very big (34KB).
Here's a link:
http://marc.merlins.org/tmp/rrdtool.info
And yes, I could split the file if I _really_ had to, but it really makes my management of that data more work and something I wanted to avoid.
That said, if I can't play the neat bitfield trick I thought would work, I might be out of luck

Thanks,
Marc

Thanks,
Marc

[marcmerlin]

Well, I had a chat on the rrdtool mailing list where they confirmed that they don't support storing data in GAUGE fields, but thinking about it I realized that I can just discard any data that is not an integer since I use that DS as a bitfield, and it will work 99.999x% of the time.

So, to extract bit #3, instead of
CURRENT_DATA_SOURCE,8,/,FLOOR,2,%,1,EQ,INF,UNKN,IF
I now use
CURRENT_DATA_SOURCE,DUP,FLOOR,EQ,CURRENT_DATA_SOURCE,8,/,FLOOR,2,%,1,EQ,INF,UNKN,IF,UNKN,IF

which first discards any data that is not integer. Yes, this means I could miss a one sample long data flip in my input but thankfully my data to track (dampers, thermostats) don't flip on an off all the time and losing one data sample or two is acceptable in my case.

[gandalf]

I followed your thread there and am happy that my understanding was quite ok and I did not tell you rubbish. I assume your current approach is the best way to do it
R.