Some graphs don't populate - migration from 1.2.14 to 1.2.28

[slademan]

After migrating mysql database from 5.7 to 8.3 , cacti from 1.2.14 to 1.2.28 , some graphs don't populate, while other do. I synced all rrd files. When I run check with data source troubleshooter, I get red x marked under status and RRD match (it determines if RRD file matches data source template).
How to debug this in more detail?

[macan]

Management -> Data sources -> choose problematic DS -> Turn on data sourcer debug and info mode
show screenshot

[slademan]

Here is example of one data source that doesn't work.

[macan]

Are the non-functional graphs different or are they all the same template?

Try add device again, create graph and show DS detail from new device after few poller cycles.

[slademan]

Graphs come from different templates. Some graphs from one template work, while others don't. Only thing that is in common is that graph which doesn't work it doesn't work on any device.

I deleted device, add it again, but it seems to me that problem is the same. Graphs that didn't work still don't work, while others continue to work.

Here is new print screen of example graph "CPU Usage".

[macan]

please try data source troubleshooter for newly added device. The result is the same?

[slademan]

Unfortunately, result is same again. It shows issue under "RRD Match" field.
I also noticed that rrd file for CPU Usage graph is changing timestamp (when poller runs), but size is always the same. I don't know does that information helps at all.

[slademan]

It runs with php-fpm 8.3 version, but I also tested older php-fpm 7.4 version. There was no change, it works in same way.

[macan]

RRD Match says - Determines if the RRD file matches the Data Source Template so it seems that template is different. Show screenshot of Templates-> data source -> problematic DS

[slademan]

X CPU Usage DT template:

[TheWitness]

Why this Happened
This is a likely a very old install (or the original database). In older versions of Cacti, it was very easy to create an RRDfile with both the 1 minute and 5 minute RRA's to be selected.

This would result in what is essentially damaged RRDfile in today's Cacti. In a new Cacti install, something like this is not possible thankfully.

Recommended Fix - Stage 1
What I would recommend would be to do the following:

• Setup a new Cacti that includes your damaged database.

• Copy the RRDfiles to this new setup and ensure everything is collecting properly.

• Create a new 1 Minute Data Source Profile that matches your requirements.

• Once everything is collecting properly, update ALL your Data Templates to use the new Data Source Profile. You can do this from the GUI.

• Update your Data Sources to use this profile. You can not do this from the GUI, but have to do this from the MariaDB/MySQL prompt. Assuming a Poller Interval of 1 minute, and a heartbeat of 5, you would run the following two queries.
  

  Code: Select all

  UPDATE data_template_data SET data_source_profile_id=<newid>, rrd_step=60;
  UPDATE data_template_rrd SET rrd_heartbeat=300;
  

• Once this is done, Repopulate your Poller Cache. You can do this from Utilities.

Recommended Fix - Stage 2
Now is where it becomes interesting. Let's assume your install is in /var/www/html/cacti. Then assuming that, you need to do the following. This is in bash.

Non Structured Paths

Code: Select all

cd /var/www/html/cacti/rra
mkdir /tmp/rra
mv /var/www/html/cacti/rra/*.rrd /tmp/rra

Structured Paths

Code: Select all

cd /var/www/html/cacti/rra
mkdir /tmp/rra
mv * /tmp/rra

Then wait for the poller to re-create all the RRDfiles in /var/www/html/cacti/rra using the new Data Source profile.

Once you confirm that, you need to run the following (in bash again).

Non Structured Paths

Code: Select all

cd /var/www/html/cacti/rra
for file in $(ls -1 *.rrd);do
   php -q /var/www/html/cacti/cli/splice_rrd.php \
      --oldrrd=/tmp/rra/$file \
      --newrrd=/var/www/html/cacti/rra/$file \
      --finrrd=/var/www/html/cacti/rra/$file
   chown apache:apache $file
done

Structured Paths

Code: Select all

cd /var/www/html/cacti/rra
for file in $(find . -name \*.rrd -print);do
   php -q /var/www/html/cacti/cli/splice_rrd.php \
     --oldrrd=/tmp/rra/$file \ 
     --newrrd=/var/www/html/cacti/rra/$file \
     --finrrd=/var/www/html/cacti/rra/$file
   chown apache:apache $file
done

Why the Second Install?
The reason that I asked you to create a new instance is not that you need to keep this setup, but so that you don't break production. Once you have done this correctly on the test instance, you can perform the same thing on your production instance.

Once that is done and done properly, you can attempt to perform a splicing operation in production. Of course after all is said and done, make sure you remove the files in /tmp/rra.

Edit!!!
I found an error in PHP 8 when running the splice command. Fixing that this morning.

Edit of Edit!!!
The fix is in the 1.2.x branch in GitHub now. It's not a service impacting issue if you are running an older version of PHP. Also, note that the conversion takes about 3 seconds per RRDfile. So, if you have thousands of RRDfiles, this will take a while.

For those who are comfortable with HPC
I've use a batch scheduler (LSF) in the past to speed things up by 300x by using 300 scheduling slot's to do the conversions in parallel. To do this, you would need an NFS location or to use SCP, but SCP does have issues with having too many concurrent sessions at once. So, it's better to write to NFS, and then to have a process that finds the files in NFS and copies them to the local storage as the process goes. It's an advanced subject. So, likely just take it slowly unless you have the chops to do it using a batch system.

Instructions for Batch Savvy

• Copy the splice_rrd.php script to NFS or to the compute hosts.

• The current code attempts to source the cli_check.php script so that has to be commented out

• The current code calls the function read_config_option() that does not exist. So, replace that with the path to rrdtool

• Create a conversion script similar to the following, and place it in NFS somewhere, or copy to /tmp on the compute hosts which is the path assumed for this example. We will call the file spliceme.sh:
  

  Code: Select all

  #!/bin/sh
  # --------------------------------------------------------
  # This script assumes that the run-as user is apache and the
  # apache user has an authorized_key file that allows passwordless
  # login to/from the execution hosts.
  #
  # It also assumes that the compute hosts have both RRDfile and
  # php installed on the compute host and that that php install has
  # the sql-lite module installed.
  # --------------------------------------------------------
  
  CACTIHOST="hostname" # replace with your hostname
  
  oldrrd=$1
  newrrd=$2
  
  # Copy the files locally.
  scp $CACTIHOST:$oldrrd /tmp/$oldrrd
  scp $CACTIHOST:$newrrd /tmp/$newrrd
  
  php -q /tmp/splice_rrd.php --oldrrd=/tmp/$oldrrd --newrrd=/tmp/$newrrd --finrrd=/tmp/$newrrd
  
  if [ $? -eq 0 ]; then
     echo "NOTE: File $newfile properly spliced"
     scp /tmp/$newrrd $CACTIHOST:$newrrd
  else
     echo "ERROR: Unable to splice $newrrd"
  fi
  /bin/rm -f /tmp/$oldrrd /tmp/$newrrd
  

• Move the files as you would have otherwise done in the non-batch process
  Non Structured Paths

  Code: Select all

  cd /var/www/html/cacti/rra
  mkdir /tmp/rra
  mv /var/www/html/cacti/rra/*.rrd /tmp/rra
  

  Structured Paths

  Code: Select all

  cd /var/www/html/cacti/rra
  mkdir /tmp/rra
  mv * /tmp/rra
  

• Submit a bunch of jobs to splice the files.

  Code: Select all

  cd /var/www/html/cacti/rra
  for file in $(find . -name \*.rrd -print);do
     bsub -q normal /tmp/spliceme.sh /tmp/rra/$file /var/www/html/cacti/rra/$file
  done
  

[slademan]

I get stuck on this step:
UPDATE data_template_data SET data_source_profile_id=<newid>, rrd_step=60;

I am not sure which value is this <newid>

I have 2 data sources profiles - Upgrade Profile 1 and Upgrade Profile 2. I will now create Upgrade Profile 3 with 1 minute setup. How can I find which newid value belongs to this new profile? Is it 3 ?

[macan]

I did similar thing last year and I wrote small howto. I don't remember exact procedure but maybe you will find answer here
https://github.com/Cacti/documentation/ ... o-1-min.md

[slademan]

I did similar thing last year and I wrote small howto. I don't remember exact procedure but maybe you will find answer here
https://github.com/Cacti/documentation/ ... o-1-min.md

I found it, it's in URL when you open Data Source Profile.

[slademan]

I tried both instructions (from TheWitness in this thread and from https://github.com/Cacti/documentation/ ... o-1-min.md ), but it didn't resolved issue. Graphs that didn't work, still don't work, others that worked are still ok.
I am also getting new error in cacti log: FATAL: Spine Encountered a Segmentation Fault

All data sources and templates are moved from 1 and 2 to new Data Source Profile 3.
I don't understand these "Data Source Profile RRAs" , they are made same as on Profile 2. Profile 1 has all of them except "Hourly" RRA.