0.8.8h upgrade to 1.0.4 some graphs not updating

[allywilson]

After the upgrade, we have graphs for certain devices no longer updating.

Things I've tried to fix this:

Upgrade SPINE from 0.8.8h to 1.0.4
Clear poller cache
Change RRD from Simple to Advanced mode.
Enable debug for affected devices, and system wide too (not seeing errors).
Change poller from SPINE to cmd.php.
Chmod'ed all RRDs and directory to 777.
Chown'ed all RRDs and directory to cacti user and group.
Installed Boost (since disabled).
Enabled RRDfile Auto Clean
Made sure timezone info is correct at OS, PHP and MariaDB.
Changed poller and cron intervals from 5 mins to 1 min (alternating and at the same time), reverted back to 5 mins.
Increased number of threads from 30 to 60, reverted.
Increased number of poller processes from 4 to 6.
Increased number of vCPUs from 2 to 4.
Script and server timeout value from 25 to 180 (and in php.ini).
Manually running SPINE and poller from command line.

I'm not seeing errors. The RRD files are being updated, but there's nothing in the graphs in the web interface.

If I move an RRD to something.rrd.old, a new file is created with permissions 644 and the graph starts to be populated.
If I modify a different RRD file that's not graphing to 644, it still doesn't start graphing.
If I move the .old file back to the original file name and ensure 644 permissions - it doesn't work.

CentOS 7.3
Cacti 1.0.4
SPINE 1.0.4
RRDTool 1.4.8
MariaDB 5.5.52-MariaDB
PHP 5.4.16

Has anyone else come across this? I've been trawling through the forums, and it seems others have/had similar issues after upgrade, but their resolution steps never seem to work for me...

[allywilson]

OK, so found the issue and the answer.

After the update, I had rebooted the VM (it's on Hyper-V, but has Time Synch guest tools disabled - we have NTP configured, so thought it enough).

When it came back online, it thought it was Apr 20th, and I didn't notice that at first.

Corrected the time.

But, it had updated 811 graphs out of 6308 by the time I noticed.

Code: Select all

rrdtool info graphname.rrd | grep last_update

shows the unix timestamp, and just by chance I noticed that it was in the future.

Tried to update graphs with 'rrdtool update N:1:2:3' or whatever, and it refused to update them due to the timestamp.

From everything I've read online, rrdtool will not allow that to be overwritten. So people export to XML, then back again, etc.

Luckily, stumbled upon this nice one-liner: https://stackoverflow.com/a/11645890

Threw that into a script called 'rrd-time-correction.sh'

Exported all rrd filenames to a new file:

Code: Select all

ls >> test.txt

Created another script:

Code: Select all

#!/bin/bash
while read p; do
  lasttime=`rrdtool info $p | grep last_update | awk '{print $3}'`
   if [ $lasttime -gt `date +%s` ]
   then
     echo $p,$lasttime
     cp $p $p.bkp
     ( exec /usr/share/cacti/rra/rrd-time-correction.sh $p )
   fi
done <test.txt

So it will echo the RRD name and unix timestamp of every RRD with a last_update timestamp in the future, backup the RRD file, then run the first script (which will modify the last_update time to right now)

It took 15 minutes to do 811 graphs for me.