Big problem - What's going on!

[BSOD2600]

You can verify the database is fine by running: mysqlcheck --auto-repair --databases cacti.

If you manually invoke the poller (with logging set to debugging), does anything run?

[Cain]

I have set the logging to debugging and run the command php poller.php from within the cacti directory it does update the rrd files. This is exciting, but what does it mean for the regular process? Thanks!

Matthew

[TheWitness]

You run poller.php from the command line and get what error?

Larry

Here is a handy database repair tool that I wrote. Depending on where you place it, you may have to change the initial include line.

Code: Select all

<?php
/*
 ex: set tabstop=4 shiftwidth=4 autoindent:
 +-------------------------------------------------------------------------+
 | Copyright (C) 2006 Platform Computing, Inc.                             |
 | Portions Copyright (C) 2004-2006 The Cacti Group                        |
 |                                                                         |
 | This program is free software; you can redistribute it and/or           |
 | modify it under the terms of the GNU Lesser General Public              |
 | License as published by the Free Software Foundation; either            |
 | version 2.1 of the License, or (at your option) any later version.      |
 |                                                                         |
 | This program is distributed in the hope that it will be useful,         |
 | but WITHOUT ANY WARRANTY; without even the implied warranty of          |
 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the           |
 | GNU Lesser General Public License for more details.                     |
 |                                                                         |
 | You should have received a copy of the GNU Lesser General Public        |
 | License along with this library; if not, write to the Free Software     |
 | Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA           |
 | 02110-1301, USA                                                         |
 +-------------------------------------------------------------------------+
 | - Platform - http://www.platform.com/                                   |
 | - Cacti - http://www.cacti.net/                                         |
 +-------------------------------------------------------------------------+
*/

/* do NOT run this script through a web browser */
if (!isset($_SERVER["argv"][0]) || isset($_SERVER['REQUEST_METHOD'])  || isset($_SERVER['REMOTE_ADDR'])) {
	die("<br><strong>This script is only meant to run at the command line.</strong>");
}

$no_http_headers = true;

include(dirname(__FILE__) . "/../../include/config.php");

/* process calling arguments */
$parms = $_SERVER["argv"];
array_shift($parms);

global $debug;

$debug = FALSE;
$form  = "";

foreach($parms as $parameter) {
	@list($arg, $value) = @explode("=", $parameter);

	switch ($arg) {
	case "-d":
		$debug = TRUE;
		break;
	case "-h":
		display_help();
		exit;
	case "-form":
		$form = " USE_FRM";
		break;
	case "-v":
	case "-V":
		display_help();
		exit;
	case "--version":
		display_help();
		exit;
	case "--help":
		display_help();
		exit;
	default:
		print "ERROR: Invalid Parameter " . $parameter . "\n\n";
		display_help();
		exit;
	}
}
echo "Repairing All Cacti Database Tables\n";

$tables = db_fetch_assoc("SHOW TABLES FROM " . $database_default);

if (sizeof($tables)) {
	foreach($tables AS $table) {
		echo "Repairing Table -> '" . $table['Tables_in_' . $database_default] . "'";
		$status = db_execute("REPAIR TABLE " . $table['Tables_in_' . $database_default] . $form);
		echo ($status == 0 ? " Failed" : " Successful") . "\n";
	}
}

/*	display_help - displays the usage of the function */
function display_help () {
	print "Grid Database Repair Tool, Copyright 2007 - Platform Computing, Inc.\n\n";
	print "usage: datebase_repair.php [-d] [-h] [--form] [--help] [-v] [-V] [--version]\n\n";
	print "-form            - Force rebuilding the indexes from the database creation syntax\n";
	print "-d               - Display verbose output during execution\n";
	print "-v -V --version  - Display this help message\n";
	print "-h --help        - display this help message\n";
}
?>

[TheWitness]

What about the following:

select count(*) from poller_output;

TheWitness

[Cain]

When I run the php poller.php manually it seems to work fine. Here is the end of the output when I run it manually.

06/13/2007 12:54:02 AM - SYSTEM STATS: Time:8.1024 Method:cmd.php Processes:1 Threads:N/A Hosts:26 HostsPerProcess:26 DataSources:2475 RRDsProcessed:475
06/13/2007 12:54:02 AM - CMDPHP: Poller[0] DEBUG: SQL Exec: "replace into settings (name,value) values ('stats_poller','Time:8.1024 Method:cmd.php Processes:1 Threads:N/A Hosts:26 HostsPerProcess:26 DataSources:2475 RRDsProcessed:475')"
06/13/2007 12:54:02 AM - CMDPHP: Poller[0] DEBUG: SQL Exec: "truncate table poller_output"
06/13/2007 12:54:02 AM - CMDPHP: Poller[0] DEBUG: SQL Cell: "select count(*) from poller_command"

I have not been able to find any error when the cron job runs. This is a bit deeper into Cacti than I have dug before, so I apologize if my answers are brief. The one thing I did notice was that the poller_outpud.MYD file int the mysql cacti directory had gotten up to 100 Megs, which seemed rather large.

Matthew

[Cain]

This is what I get from the mysql command:

mysql> select count(*) from poller_output;
+----------+
| count(*) |
+----------+
| 0 |
+----------+
1 row in set (0.00 sec)

[mcutting]

Cain - have you tried to rebuild the poller cache, and try the polling again ?

[Cain]

I had the same issue a while back as this, and it was actually a corrupt database. I dropped the existing one, and restored from the backup from the night before. After doing this, the RRA's started updating again.

Due to the late hour last night I got turned around with some of the output I posted. In the end it seems that I had the same problem as mcutting, my database had become corrupt. I had tried moving the now deemed corrupt database to a new server and still had the same issue of the rrd files not updating, though the poller seemed to functioning properly. I ended up restoring my last backup from a couple weeks ago on the new server. Everything is now working well off the backup database. I believe I got my windows reversed when I posted the results and posted from the server that is now working though I referenced it as from the server that was bad.

A couple of notes: The usual size of my cacti database is less than 10 Megabytes. The one that I now believe is corrupt was around 175MB with these two files looking strangely large:

-rw-rw---- 1 mysql mysql 54M Jun 13 00:43 poller_output.MYI
-rw-rw---- 1 mysql mysql 107M Jun 13 00:43 poller_output.MYD

We did have some network problems last week and I feel that caused Cacti to not be able to properly poll all the devices. I am concluding that is what lead to the database corruption and am resolved to backup my cacti database much more frequently.

I greatly appreicate everyone's help over the last two days. I was about to go crazy trying to get it back up and working.

Matthew

[mcutting]

Cain - My database gets backed up daily using a simple batch script I wrote paired with a scheduled task.

It's not rocket science, and simply uses the mysqldump command to create a backup to a location of my choice. Using this method, I was able to restore to a database that was only 3 hours out of date.

[Cain]

Cain - My database gets backed up daily using a simple batch script I wrote paired with a scheduled task.

It's not rocket science, and simply uses the mysqldump command to create a backup to a location of my choice. Using this method, I was able to restore to a database that was only 3 hours out of date.

I am going to start doing the same with this command run from a cronjob/shell script:

mysqldump -u root -p cacti > cacti_export_YYYYMMDD.sql

or something similar. Thanks again for your help.

Matthew