Constant Recaching Loop

[rdege]

This morning, I discovered that there are 2 hosts that are constantly being re-cached. Is this normal, for a host to be recached this frequently? I thought this was abnormal, and decided to run php -q poller_reindex_hosts.php -id=148 (after disabling the poller first). However, this didn't appear to resolve the problem, and the hosts are still being recached on every run.


05/28/2010 10:40:26 AM - PCOMMAND: Poller[0] Host[148] RECACHE: Re-cache successful.
05/28/2010 10:40:26 AM - PCOMMAND: Poller[0] Host[148] RECACHE: Re-cache for Host, data query #6
05/28/2010 10:40:26 AM - PCOMMAND: Poller[0] Host[148] WARNING: Recache Event Detected for Host



05/28/2010 10:35:25 AM - PCOMMAND: Poller[0] Host[148] RECACHE: Re-cache successful.
05/28/2010 10:35:24 AM - PCOMMAND: Poller[0] Host[148] RECACHE: Re-cache for Host, data query #6
05/28/2010 10:35:24 AM - PCOMMAND: Poller[0] Host[148] WARNING: Recache Event Detected for Host


05/28/2010 10:30:26 AM - PCOMMAND: Poller[0] Host[148] RECACHE: Re-cache successful.
05/28/2010 10:30:26 AM - PCOMMAND: Poller[0] Host[148] RECACHE: Re-cache for Host, data query #6
05/28/2010 10:30:26 AM - PCOMMAND: Poller[0] Host[148] WARNING: Recache Event Detected for Host

Any insight is appreciated.

-Rob

[gandalf]

This is which version of cacti?
R.

[rdege]

Oh geez. I can't believe I omitted that from the post. Here ya go!

Cacti Version - 0.8.7e
Plugin Architecture - 2.6
Poller Type - Cactid v
Server Info - Linux 2.6.18.8
Web Server - Apache/2.0.52 (Oracle)
PHP - 5.2.5
PHP Extensions - xmlwriter, libxml, dom, xmlreader, xml, wddx, tokenizer, sysvshm, sysvsem, session, pcre, SimpleXML, SPL, PDO, sockets, snmp, SQLite, shmop, standard, Reflection, pspell, posix, pdo_sqlite, json, iconv, hash, gmp, gettext, gd, ftp, filter, exif, dba, date, curl, ctype, calendar, bz2, bcmath, zlib, openssl, apache2filter, ldap, mysql
MySQL - 5.0.18
RRDTool - 1.2.15
SNMP - 5.1.2
Plugins

• Update Checker (update - v0.4)
  Global Plugin Settings (settings - v0.5)
  Thresholds (thold - v0.4.1)
  Device Monitoring (monitor - v0.8.2)
  Uptime (uptime - v0.3)
  Host Info (hostinfo - v0.2)
  Login Page Mod (loginmod - v1.0)

[gandalf]

Please tell us, which data query is #6 in your system. Default setup will have "Unix - Get Mounted Partitions" for that.
Then, please edit host 148 and tell us, which reindex method has been chosen for your data query #6
R.

[rdege]

Data Query #6 is Unix - Get Mounted Partitions.

The re-index method for Unix - Get Mounted Partitions on host 148 is Index Count Changed.

[gandalf]

Please press the green circle next to that data query once. Then, please verify if things have changed (I fear not, but it's worth a try)
R.

[rdege]

Sadly, it didn't resolve the problem. I still get the recaching messages:

05/28/2010 03:30:24 PM - PCOMMAND: Poller[0] Host[148] RECACHE: Re-cache successful.
05/28/2010 03:30:24 PM - PCOMMAND: Poller[0] Host[148] RECACHE: Re-cache for Host, data query #6
05/28/2010 03:30:24 PM - PCOMMAND: Poller[0] Host[148] WARNING: Recache Event Detected for Host

[gandalf]

Please replace the script by the one I've attached. Again, press the green circle. This should resolve the issue.
You know, that this script will only work on localhost, not on remotes machines?
R.

[rdege]

Host[148] is a remote host and not a localhost. Just for fun, I tried your modified script anyways, and just as you had stated, it did not resolve the problem.

[gandalf]

But again: that script executes "df" on localhost. Even though the host it is assigned to is a remote host, the script will always report "df" results from localhost. The results will make no sense for a remote host.
Use "net/snmp ..." data queries instead for remote hosts.
The Data Query/Scripts will be renamed in 088 to avoid confusion.

But for sake of debugging the script, please run following SQL

Code: Select all

SELECT *
FROM `poller_reindex` WHERE host_id = 148 AND data_query_id=6

and post results.
R.

[rdege]

Hi Gandalf,

Here's the output you requested:

Code: Select all

mysql> SELECT *
    -> FROM `poller_reindex` WHERE host_id = 148 AND data_query_id=6;
+---------+---------------+--------+----+--------------+------------------------------------------------------------------------------+
| host_id | data_query_id | action | op | assert_value | arg1                                                                         |
+---------+---------------+--------+----+--------------+------------------------------------------------------------------------------+
|     148 |             6 |      1 | =  | 3            | perl /var/www/html/cacti-0.8.7e/scripts/query_unix_partitions.pl num_indexes |
+---------+---------------+--------+----+--------------+------------------------------------------------------------------------------+
1 row in set (0.00 sec)

[gandalf]

Please download the latest code of that script from SVN/0.8.7
I fixed the bug some days ago.
R.

[rdege]

Per your instructions, I replaced the script, but it did not resolve the problem. It is possible that I checked out the wrong file. Just to confirm, I ran svn checkout svn://svn.cacti.net/cacti, and used the file cacti/cacti/branches/0.8.7/scripts/query_unix_partitions.pl.

Code: Select all

06/02/2010 11:40:19 AM - PCOMMAND: Poller[0] Host[148] RECACHE: Re-cache successful.
06/02/2010 11:40:19 AM - PCOMMAND: Poller[0] Host[148] RECACHE: Re-cache for Host, data query #6
06/02/2010 11:40:19 AM - PCOMMAND: Poller[0] Host[148] WARNING: Recache Event Detected for Host
06/02/2010 11:40:14 AM - SPINE: Poller[0] Host[148] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function 

[TheWitness]

If you use the index count changed function, it will always have to determine the number of indexes. The question becomes, how does it ask the script that question.

Somehow the fog is clearing on this issue now.

TheWitness

[gandalf]

Please run

Code: Select all

perl /var/www/html/cacti-0.8.7e/scripts/query_unix_partitions.pl num_indexes

with your new script. What does it return?
R.