After migration, some graphs not updating correctly

[ryaner]

I've decided to split our cacti install and to start I simply backed up and migrated everything to another host. Almost all of the graphs have been working fine however a few of them are stuck with NaN values in the rrdtool.

I've stepped through the logs in debug and everything is polling correctly and calling rrdtool correctly, it just isn't accepting the values.

Using the cacti poller stats as an example

The update call to rrd

Code: Select all

10/11/2012 01:45:02 PM - POLLER: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/www/localhost/htdocs/cacti/rra/63/3601.rrd --template Time:Processes:Threads:Hosts:HostsPerProcess:DataSources:RRDsProcessed 1349959502:39.6111:6:100:78:13:1634:1363

The rrd info output. Min/Max values look correct

Code: Select all

rrdtool info /var/www/localhost/htdocs/cacti/rra/63/3601.rrd | grep -i 'min\|max'
ds[RRDsProcessed].minimal_heartbeat = 120
ds[RRDsProcessed].min = 0.0000000000e+00
ds[RRDsProcessed].max = NaN
ds[DataSources].minimal_heartbeat = 120
ds[DataSources].min = 0.0000000000e+00
ds[DataSources].max = NaN
ds[HostsPerProcess].minimal_heartbeat = 120
ds[HostsPerProcess].min = 0.0000000000e+00
ds[HostsPerProcess].max = NaN
ds[Hosts].minimal_heartbeat = 120
ds[Hosts].min = 0.0000000000e+00
ds[Hosts].max = NaN
ds[Threads].minimal_heartbeat = 120
ds[Threads].min = 0.0000000000e+00
ds[Threads].max = 1.0000000000e+02
ds[Processes].minimal_heartbeat = 120
ds[Processes].min = 0.0000000000e+00
ds[Processes].max = 1.0000000000e+02
ds[Time].minimal_heartbeat = 120
ds[Time].min = 0.0000000000e+00
ds[Time].max = 3.0000000000e+02

Some averages showing the nan problems

Code: Select all

cacti4 [VE] cacti # rrdtool fetch /var/www/localhost/htdocs/cacti/rra/63/3601.rrd AVERAGE | tail
1349962560: -nan -nan -nan -nan -nan -nan -nan
1349962920: -nan -nan -nan -nan -nan -nan -nan
1349963280: -nan -nan -nan -nan -nan -nan -nan
1349963640: -nan -nan -nan -nan -nan -nan -nan
1349964000: -nan -nan -nan -nan -nan -nan -nan
1349964360: -nan -nan -nan -nan -nan -nan -nan
1349964720: -nan -nan -nan -nan -nan -nan -nan
1349965080: -nan -nan -nan -nan -nan -nan -nan
1349965440: -nan -nan -nan -nan -nan -nan -nan
1349965800: -nan -nan -nan -nan -nan -nan -nan

I'm not sure where else to look and would appreciate a push in the right direction.

[ryaner]

I've traced this back to the heartbeat in the rrdfiles. We had the original poller running at 1 minute intervals while the new host will only go at 5 minutes. 10k data sources and 1 minute intervals weren't getting along.

Code: Select all

rrdtool info /var/www/localhost/htdocs/cacti/rra/63/3601.rrd | grep heartbeat
ds[RRDsProcessed].minimal_heartbeat = 120
ds[DataSources].minimal_heartbeat = 120
ds[HostsPerProcess].minimal_heartbeat = 120
ds[Hosts].minimal_heartbeat = 120
ds[Threads].minimal_heartbeat = 120
ds[Processes].minimal_heartbeat = 120
ds[Time].minimal_heartbeat = 120

Fixed with

Code: Select all

# rrdtool tune /var/www/localhost/htdocs/cacti/rra/63/3601.rrd -h RRDsProcessed:600
# rrdtool tune /var/www/localhost/htdocs/cacti/rra/63/3601.rrd -h DataSources:600
# rrdtool tune /var/www/localhost/htdocs/cacti/rra/63/3601.rrd -h HostsPerProcess:600
# rrdtool tune /var/www/localhost/htdocs/cacti/rra/63/3601.rrd -h Hosts:600
# rrdtool tune /var/www/localhost/htdocs/cacti/rra/63/3601.rrd -h Threads:600
# rrdtool tune /var/www/localhost/htdocs/cacti/rra/63/3601.rrd -h Processes:600
# rrdtool tune /var/www/localhost/htdocs/cacti/rra/63/3601.rrd -h Time:600

My question now is should updating the Heartbeat setting inside of cacti not go an tune the rrdfile?

[gandalf]

You should reflect those changes to the data template. But please make sure, that you won't be beaten by an issue with the "step" ...
R.

[ryaner]

I did switch the heartbeat in the data_templates of the affected graphs but still had to manually tune things. Manually tuning got the graphs back working.

I've switched the step setting as well, thanks for the reminder. Do you know how to update the rrd files or is that even needed? Nothing about switching the step appears on the man pages of rrdtune.

[gandalf]

And, again, here is the problem:

1. Unfortunately, NO changes to any data template are propagated to any existing rrd file
2. Very unfortunately, only few parameters of existing rrd files CAN be changed AFTER having created those files

This results in
1. There are SOME parameters, which, when encountering a change require existing rrd files (of this data template) to be DELETED, thus LOOSING ALL EXISTING DATA. So, be carefully when planning exact values of those parameters. The most important of those is the STEP value (this relates to Cacti polling interval)
2. All other parameter changes currently must be propagated MANUALLY to EXISTING rrd files. The command to perform such changes is "rrdtool tune".

BTB: recently, a new "RRDtweak" Utility has been posted to rrd contrib. This one seems to leverage SOME of the restrictions (e.g. deleting and adding data source items)

R.

[BRow]

So I'd rather thread necro than post a new thread since this relates heavily to my current situation--

I inherited a cacti install that was running every minute. At one point after reading a lot of how-to's and everyone referencing the cron running @ 5min, I switched to a 5min run. However, in the process I started getting blank graphs and "NaN" values from rrdinfo. After seeing this thread, I decided to reset the cron to run every minute again in hopes that it would fix my blank graphs. That didn't work. Thinking that setting the cron to 5min changed some values, I went through the web gui and reset the templates -> Data templates to use a 60 step and 120 heartbeat as they were before. As of now I have not updated the rrd files themselves via rrdtool because I worry that I'll lose the data. So, to be completely clear, would changing the step value in the web front-end all the existing data or is it the update to the rrd files themselves that would cause the issue? Would it be possible to salvage the old rrd data somehow and maybe move it on to another cacti instance? Thanks in advance for any help/advice.

[gandalf]

So, to be completely clear, would changing the step value in the web front-end all the existing data or is it the update to the rrd files themselves that would cause the issue?

Neither of the two will do. You can't change the STEP after rrd file has been created. This value is used for normalization within rrd. I only did not check the last utilities that have been published to the rrd site
R.