NaN graphs and missing poller items from cache

[bleearg]

I am having trouble finding the source of a problem with some graphs not updating and showing NaN. I have probably 40 devices in Cacti and the three devices having issues are different from the rest in that that they use a different data profile. I am on 1.1.38 and spine.

I've gone through all the troubleshooting steps in the docs for identifying NaN issues, to no avail. One step indicates you should check the poller cache to see if the devices are missing, which they are, but it offers no suggestions on what to do if that's the case.

I've tried rebuilding my poller cache, both via the web and the CLI and I still cannot get the devices to show up in the poller cache. I'm not sure what happened here, as these three devices had been working for several months, when suddenly at 2AM one day, two of the devices stopped updating. When I started looking at the problem and running through the troubleshooting steps outlined in the docs, suddenly the third device stopped updating, without me having changed anything besides rebuilding the poller cache. I believe the issue has to do with the different data profile, but I'm not sure what to do to fix it. The data profile has the exact same Data Source profiles as the default, but there is one additional that defines a 10-Minute Average:

Data profile

data_profile.PNG (23.05 KiB) Viewed 1410 times

Can someone please point me in the right direction on how to get these three devices to start polling again? Don't ask me where I got the numbers for the steps/rows in the 'Semi-weekly' data source profile. I copied them from another Cacti installation with the same profile, which has been working for several years.

EDIT: I found the following when using the Data Source Info selection in the web GUI:

Code: Select all

Number of rows for Cacti RRA id '2' should be '700'
Number of rows for Cacti RRA id '3' should be '775'
Number of rows for Cacti RRA id '4' should be '797'
Number of rows for Cacti RRA id '7' should be '700'
Number of rows for Cacti RRA id '8' should be '775'
Number of rows for Cacti RRA id '9' should be '797'
Number of rows for Cacti RRA id '12' should be '700'
Number of rows for Cacti RRA id '13' should be '775'
Number of rows for Cacti RRA id '14' should be '797'
Number of rows for Cacti RRA id '17' should be '700'
Number of rows for Cacti RRA id '18' should be '775'
Number of rows for Cacti RRA id '19' should be '797'
/bin/rrdtool resize /var/www/html/cacti/rra/r1_traffic_in_339.rrd 2 SHRINK 2800
rename /var/www/html/cacti/rra/resize.rrd to /var/www/html/cacti/rra/r1_traffic_in_339.rrd

I am pretty sure this is why this is happening. I'm not versed at all in rrdtool - can someone give me an idea what I'll need to do to get this fixed?

[Osiris]

Resize leaves Nan's in the new rows, which makes the graphs break for a while. There have been those that have written utilities to insert good data, but rrdtool does not.

[bleearg]

I'm not concerned about the missing data at this point. I just want Cacti to actually start polling the devices, as I believe I fixed the issues with the rrd files. Rebuilding the poller cache doesn't seem to work. I am able to create new graphs on the same hosts and they get added into the cache, but for some reason, these just will not. How can I get Cacti to start polling? Debug shows nothing because the devices are not being polled. If I try rebuilding the cache for just the singular devices via the CLI, I get no errors.

[Osiris]

You might want to 'resave' them as it will populate the poller cache if it's not already. This is kind of hard to solve over the forums, but basically there is a column in the poller_item tabled called rrd_next_step and when it goes negative, the data source will poll again. It decrements by the poller interval. So, if you are polling every 60 seconds, you will see the value slowly creep towards zero. Sometimes people set the heartbeat too low. Say for example, you poller interval is 300 seconds, but your heartbeat is set to 60 seconds. Those graphs will never show data.

Under the Data Sources page, there is an Info debug level that will show you what the RRDfiles look like, but some of this is hand viewing the poller cache and in rare cases (like a bug in Cacti) reviewing the rrd_next_step value for accuracy.

I hope this helps. There is a lot to review. One more thing, in Cacti 1.2, there is a troubleshooting pick and under that Data Debug that you can enable for the Data Sources in question.

[bleearg]

I've tried everything I can think of to "re-save" them. I've disabled and re-enabled the hosts, changed settings in the individual graph pages and changed them back again. I've been trying to find how the poller cache is populated, but I'm not a coder and can't really figure it out from the PHP files. I am about to just delete the graphs, re-add them and use the old (fixed) RRD files over top of the new ones it creates.

Thanks for the info about the poller_item column. Unfortunately, these items are not even in the database at all, so there is no field to attempt to decrement. I can't create new graphs using the same template without deleting the existing ones first, so it does know that graphs were created. It just is not putting them into the poller_items table when rebuilding.

I did try to update to 1.2 the other day, but all my graphs stopped updating after doing so and nothing I was doing was working to get them back online. I got frustrated and just restored to 1.1.38.

[Osiris]

Okay, there was an issue found in 1.2.0. If you are running 1.2.0, do the following:

UPDATE host SET deleted="" WHERE deleted=NULL;
ALTER TABLE host MODIFY COLUMN deleted CHAR(2) NOT NULL default "";