[SOLVED] rrdclean not cleaning.... or archiving

[classen]

The title post pretty much summarizes by problem.

I select all unused rrd files, then either delete, archive, or backup.
In the rrd window all the rrd files disappear, but if I open a terminal and look in the rra/backup or rra/archive folder nothing is there
If I rescan then all the rrd files show up again.
The archive and backup folders are owned by the same user that is running the cacti poller
I'm at a bit of a loss. There is no output in the cacti.log that would help debug this issue either.

[noname]

From http://docs.cacti.net/plugin:rrdclean,

2. file delete is done by the poller
- This avoids permission errors or security issues.
- The same instance that creates rrd files, the poller, will be used for deletion as well.

You'd better to wait for next polling.

NOTE: Sometimes deleting seems to fail for some reason. If so, re-try a few times.

[classen]

Well I've tried deleting, archiving, and/or backing up and then waited several polling cycles and yet all unused rrd's remain.

probably a permission issue somewhere?

[noname]

Hmm, when I tried, I can see following lines in cacti.log.
(no need to change "Poller Logging Level" in Settings)

Archive:

Code: Select all

08/19/2011 03:16:15 PM - RRDCLEAN: Poller[0] RRDClean is now running on 1 items
08/19/2011 03:16:15 PM - RRDCLEAN: Poller[0] Moved: xxxx.rrd to: /var/www/cacti/rra/archive/
08/19/2011 03:16:15 PM - RRDCLEAN: Poller[0] delete from plugin_rrdclean_action: xxxx.rrd
08/19/2011 03:16:15 PM - RRDCLEAN: Poller[0] RRDClean has finished 1 items

Backup:

Code: Select all

08/19/2011 03:22:41 PM - RRDCLEAN: Poller[0] RRDClean is now running on 1 items
08/19/2011 03:22:42 PM - RRDCLEAN: Poller[0] Copied: xxxx.rrd to: /var/www/cacti/rra/backup/
08/19/2011 03:22:42 PM - RRDCLEAN: Poller[0] RRDClean has finished 1 items

Delete:

Code: Select all

08/19/2011 03:26:14 PM - RRDCLEAN: Poller[0] RRDClean is now running on 1 items
08/19/2011 03:26:14 PM - RRDCLEAN: Poller[0] Deleted: xxxx.rrd
08/19/2011 03:26:14 PM - RRDCLEAN: Poller[0] RRDClean has finished 1 items

>> probably a permission issue somewhere?

Maybe..?

Please perform this, and post the output.
% ls -ld rra/
% ls -l rra/xxxx.rrd (one of the files you want to delete)

[classen]

Do you think that the red files must be user executable (e.g. 755)?

Code: Select all

$ ls -ld rra/
drwxr-xr-x 80 webadmin apache 8192 Aug 15 21:45 rra/

$ ls -ld rra/41
drwxr-xr-x 2 webadmin apache 4096 Aug  5 17:36 rra/41

$ ls -ld rra/41/697.rrd
-rw-r--r-- 1 webadmin apache 386312 Sep 29  2010 rra/41/697.rrd

I went ahead and changed permissions to 755 and rrdclean is still unable to remove rrd files.

[classen]

I'm now suspicious that there is a misconfiguration in my MySQL database.

there is a function do_rrd in the rrdcleaner.php file that moves doomed rrd files into the plugin_rrdclean_action table. This appears to be working just fine. I select a single file, delete it and it shows up in the plugin_rrdclean_action table, but it just stays there. cacti/poller/spine never touches it.

[noname]

Do you think that the red files must be user executable (e.g. 755)?

Code: Select all

$ ls -ld rra/
drwxr-xr-x 80 webadmin apache 8192 Aug 15 21:45 rra/

$ ls -ld rra/41
drwxr-xr-x 2 webadmin apache 4096 Aug  5 17:36 rra/41

$ ls -ld rra/41/697.rrd
-rw-r--r-- 1 webadmin apache 386312 Sep 29  2010 rra/41/697.rrd

I went ahead and changed permissions to 755 and rrdclean is still unable to remove rrd files.

Hmm, I think these permissions are not so strange...
My rrdclean can delete files that have the same permission as the above.
(And, it's not necessary to add executable permission for RRD files.)

I'm now suspicious that there is a misconfiguration in my MySQL database.

there is a function do_rrd in the rrdcleaner.php file that moves doomed rrd files into the plugin_rrdclean_action table. This appears to be working just fine. I select a single file, delete it and it shows up in the plugin_rrdclean_action table, but it just stays there. cacti/poller/spine never touches it.

Then, in your PHP error log, is there anything which regarding Cacti?

[classen]

My php error logging was turned off.
I turned it on tried to delete a few rrd files. I see some warnings in the php-error.log file, but none appear to be related to rrdclean or any of my other malfunctioning plugins.

Code: Select all

[31-Aug-2011 12:48:07] PHP Warning:  Missing argument 4 for rrdtool_execute(), called in /home/www/cacti-0.8.7g/plugins/thold/thold_functions.php on line 918 and defined in /home/www/cacti-0.8.7g/lib/rrd.php on line 61
[31-Aug-2011 12:48:07] PHP Notice:  Undefined variable: rrdtool_pipe in /home/www/cacti-0.8.7g/lib/rrd.php on line 95
[31-Aug-2011 12:48:07] PHP Warning:  Missing argument 4 for rrdtool_execute(), called in /home/www/cacti-0.8.7g/plugins/thold/thold_functions.php on line 918 and defined in /home/www/cacti-0.8.7g/lib/rrd.php on line 61
[31-Aug-2011 12:48:07] PHP Notice:  Undefined variable: rrdtool_pipe in /home/www/cacti-0.8.7g/lib/rrd.php on line 95
[31-Aug-2011 12:48:07] PHP Warning:  Division by zero in /home/www/sibyls/cacti-0.8.7g/plugins/thold/thold_functions.php on line 896

[classen]

It turns out there was a CentOS distributed version of php residing in /usr/bin

I had installed my hand-rolled php into /usr/local/bin

Unfortunately php appears to be hard coded into various scripts throughtout the cacti and plugin files.

I deleted /usr/bin/php and made a symbolic link to my good version of php

ln -s /usr/local/bin/php /usr/bin/php

now all my plugins are working! yeah!

perhaps these are some of the culprits?

Code: Select all

$ grep -r "/usr/bin/php" cli/*
cli/add_data_query.php:#!/usr/bin/php -q
cli/add_device.php:#!/usr/bin/php -q
cli/add_graphs.php:#!/usr/bin/php -q
cli/add_graph_template.php:#!/usr/bin/php -q
cli/add_perms.php:#!/usr/bin/php -q
cli/add_tree.php:#!/usr/bin/php -q

and

Code: Select all

$ grep -r "/usr/bin/php" plugins/*
plugins/mactrack/poller_mactrack.php:#!/usr/bin/php -q
plugins/nectar/poller_nectar.php:#!/usr/bin/php -q
plugins/syslog/template/cacti_graph_template_localhost_syslog_messages.xml:             <input_string>/usr/bin/php -q <path_cacti>/plugins/syslog/syslog_counter.php</input_string>
plugins/weathermap/docs/pages/cacti-plugin.html:*/5 * * * *   cactiuser  /usr/bin/php   /your/cacti/path/plugins/weathermap/weathermap-cacti-rebuild.php
plugins/weathermap/weathermap:#!/usr/bin/php

[noname]

Ah, is it the same situation as this topic you mentioned?
- http://forums.cacti.net/viewtopic.php?f=14&t=43840

Sorry I'm not sure in detail, but it might be the culprits if there were differences between 2 PHPs. (e.g. installed modules, include_path, etc.)

[TheWitness]

The shbang's are a red herring. Might be due to structured paths. I never tested it on this.

[noname]

The shbang's are a red herring. Might be due to structured paths. I never tested it on this.

Hmm, I'm not sure whether shebang ("#!/usr/local/bin/php" or "#!/usr/bin/php") is 'trap',
but rrdclean plugin worked also with "Structured RRA Path".

Before cleaning:

Code: Select all

% ls -ld rra/
drwxr-xr-x 4 cactiuser wwwuser 1024 Oct  6 13:25 rra/

% ls -lR rra/
rra/:
drwxr-xr-x 2 cactiuser cactiuser 1024 Oct  6 13:40 1/
drwxr-xr-x 2 cactiuser cactiuser  512 Oct  6 13:25 3/

rra/1:
<snip>
-rw-r--r-- 1 cactiuser cactiuser 111392 Oct  6 13:30 68.rrd  <-- should be deleted
-rw-r--r-- 1 cactiuser cactiuser 111392 Oct  6 13:30 69.rrd  <-- should be deleted
-rw-r--r-- 1 cactiuser cactiuser 111392 Oct  6 13:25 70.rrd  <-- should be deleted
-rw-r--r-- 1 cactiuser cactiuser 111392 Oct  6 13:25 71.rrd  <-- should be deleted
-rw-r--r-- 1 cactiuser cactiuser 111392 Oct  6 13:25 72.rrd  <-- should be deleted

rra/3:
-rw-r--r-- 1 cactiuser cactiuser  47992 Oct  6 13:40 73.rrd
-rw-r--r-- 1 cactiuser cactiuser 111392 Oct  6 13:40 74.rrd
-rw-r--r-- 1 cactiuser cactiuser 111392 Oct  6 13:40 75.rrd
-rw-r--r-- 1 cactiuser cactiuser 111392 Oct  6 13:40 76.rrd
-rw-r--r-- 1 cactiuser cactiuser 111392 Oct  6 13:40 77.rrd
-rw-r--r-- 1 cactiuser cactiuser 111392 Oct  6 13:40 78.rrd
-rw-r--r-- 1 cactiuser cactiuser 111392 Oct  6 13:40 79.rrd
-rw-r--r-- 1 cactiuser cactiuser 111392 Oct  6 13:40 80.rrd
-rw-r--r-- 1 cactiuser cactiuser 111392 Oct  6 13:40 81.rrd
-rw-r--r-- 1 cactiuser cactiuser 111392 Oct  6 13:40 82.rrd

After cleaning:
(using Cacti 0.8.7g + PIA 2.9 + rrdclean 0.41)

10/06/2011 01:45:19 PM - SYSTEM STATS: Time:18.0071 Method:cmd.php Processes:1 Threads:N/A Hosts:3 HostsPerProcess:3 DataSources:59 RRDsProcessed:39
10/06/2011 01:45:19 PM - SYSTEM THOLD STATS: Time:0.0017 Tholds:0 Hosts:0
10/06/2011 01:45:20 PM - RRDCLEAN: Poller[0] RRDClean is now running on 29 items
<snip>
10/06/2011 01:45:20 PM - RRDCLEAN: Poller[0] Deleted: 1/68.rrd
10/06/2011 01:45:20 PM - RRDCLEAN: Poller[0] Deleted: 1/69.rrd
10/06/2011 01:45:20 PM - RRDCLEAN: Poller[0] Deleted: 1/70.rrd
10/06/2011 01:45:20 PM - RRDCLEAN: Poller[0] Deleted: 1/71.rrd
10/06/2011 01:45:20 PM - RRDCLEAN: Poller[0] Deleted: 1/72.rrd
10/06/2011 01:45:20 PM - RRDCLEAN: Poller[0] RRDClean has finished 29 items


% ls rra/1/72.rrd
ls: cannot access rra/1/72.rrd: No such file or directory

Successfully deleted.

[TheWitness]

Most of the cacti scripts are launched using 'php -q <scriptname>' to prevent issues with shbang paths. Maybe rrdclean is an exception. I'll have to chastise the author.

[noname]

>> I'll have to chastise the author.

Oh, that's too bad...

[gandalf]

Most of the cacti scripts are launched using 'php -q <scriptname>' to prevent issues with shbang paths. Maybe rrdclean is an exception. I'll have to chastise the author.

The cleaning is called by poller_bottom hook, which, to my knowledge, uses "php -q". So I don't know how I should react ...
R.