HA you've got to hide your data away

[oxo-oxo]

Let's do a thought experiment...

A new Cacti user set's up a standard cacti solution on some hardware.
The hardware is good, and the number of data elements is low.
Everythings is working except for the nagging feeling of no HA.

So, another piece of hardware is obtained, and is placed in another machine room.
One implements MySQL replication and then ask on the Cacti forum how to replicate the rrd data...

There are several suggestions, some (all) require a specific OS.

What is missing?
- a standard cacti method to replicate RRD data that is not specific to an OS

[oxo-oxo]

OK, lets do a what if...

If the first server is the "active" server, then it collects data update mysql and creates the local rrd files.

The second server is the "passive" server, it just reads the mysql data and creates the local rrd files.

This is a solution that does not require specific OS support or mounting of filesystems or syncing of files from one system to another
- we just do mysql replication and have "something" that keeps an eye on the second servers mysql and creates/updates rrd

If the first server dies, "do something" to get the second server running as the primary server (the server that was the primary server, is repaired and brought up as the secondary server)

[oxo-oxo]

But then one begins to have doubts, maybe one needs a replication to a off site
- no problem, add another server and create a slave replication of the primary/secondary server set: the slave also creates the rrd files locally without haveing to find out how to add filesystem sync of the main rrd files.


But one is still not happy: the secondary server isn't doing anything but replicate data in case there is a failure of the primary server

-- but that is another question/problem

[oxo-oxo]

Now, what is this "something" that creates/updates the rrdfiles ...

Well, it must be some form of poller.php (http://svn.cacti.net/viewvc/cacti/branc ... iew=markup)
- but with an option --r<ead only mysql>

On the primary, there is a poller.php running under cron, do we want this on the secondary: probably not as we may have timing issues.
So we need a poller.php that sleeps a minute and checks if there has been a new completed poller run.
Then it goes thru the data available and creates/updates rrds

How does mysql replication with creation/updating of rrd compare with file or sector updates?
- who knows ....

How does one trigger a primary -> secondary failover?
- who knows (but by the secondary poller could have logic, if no updates for 30 min -> the primary is dead, trigger a read/write cron install. The IP address for the Cacti has already been moved to secondary by "standard" HA of tcpip ...)

[oxo-oxo]

A quick check of poller.php shows the first write : truncate
- so the -r option needs to stop these.

Also, identification of the last poller run in the mysql table should be done.

Code: Select all

/* retreive the last time the poller ran */
if ($poller_id == 0) {
    $poller_lastrun = read_config_option('poller_lastrun');
}else{
    $poller_lastrun = read_config_option('poller_lastrun_$poller_id');
}

- how would multiple pollers interact on the secondary: well, the secondary might just accept all poller id's ...?

But the code is ... a bit long ... for the start of poller.php
- it does many things and it would need to be sliced down in some functions to clearly devide r contra rw poller run, otherwise for many if opt r statements...

Anyway, in the code executes either spine or cmd.php : which we don't want...

And finally, the rrd creation/update starts:

Code: Select all

        /* open a pipe to rrdtool for writing */
        $rrdtool_pipe = rrd_init();

        $rrds_processed = 0;
        while (1) {

So, hackable ...

Gotta check that log writes are to filesystem and not to mysql
- so how does one replicate logs if this is true.

OK: logs to a file would have been nice to a table, but never mind....

Code: Select all

/* cacti_log - logs a string to Cacti's log file or optionally to the browser
   @arg $string - the string to append to the log file
   @arg $output - (bool) whether to output the log line to the browser using print() or not
   @arg $environ - (string) tell's from where the script was called from */
function cacti_log($string, $output = false, $environ = "CMDPHP") {

[oxo-oxo]

Code: Select all

Index: poller.php
===================================================================
--- poller.php	(revision 5180)
+++ poller.php	(working copy)
@@ -56,10 +56,14 @@
 		$debug = TRUE;
 
 		break;
-	case "--force":
+	case "--force":			
 		$force = TRUE;
 
 		break;
+	case "--rox":			/* support for HA rrd creating from mysql, ro */
+		$rox = TRUE;
+
+	break;
 	case "--poller":
 		$poller_id = $value;
 
@@ -77,111 +81,117 @@
 	}
 }
 }
+/* no writes in this section */
+	api_plugin_hook('poller_top');	/* what did that just do */
 
-api_plugin_hook('poller_top');
+	/* record the start time */
+	list($micro,$seconds) = split(" ", microtime());
+	$poller_start         = $seconds + $micro;
+	$overhead_time = 0;
 
-/* record the start time */
-list($micro,$seconds) = split(" ", microtime());
-$poller_start         = $seconds + $micro;
-$overhead_time = 0;
+	/* get number of polling items from the database */
+	$poller_interval = read_config_option("poller_interval");
 
-/* get number of polling items from the database */
-$poller_interval = read_config_option("poller_interval");
+	/* retreive the last time the poller ran */
+	if ($poller_id == 0) {
+		$poller_lastrun = read_config_option('poller_lastrun');
+	}else{
+		$poller_lastrun = read_config_option('poller_lastrun_$poller_id');
+	}
 
-/* retreive the last time the poller ran */
-if ($poller_id == 0) {
-	$poller_lastrun = read_config_option('poller_lastrun');
-}else{
-	$poller_lastrun = read_config_option('poller_lastrun_$poller_id');
-}
+	/* get the current cron interval from the database */
+	$cron_interval = read_config_option("cron_interval");
 
-/* get the current cron interval from the database */
-$cron_interval = read_config_option("cron_interval");
+	if ($cron_interval != 60) {
+		$cron_interval = 300;
+	}
 
-if ($cron_interval != 60) {
-	$cron_interval = 300;
-}
+	/* see if the user wishes to use process leveling */
+	$process_leveling = read_config_option("process_leveling");
 
-/* see if the user wishes to use process leveling */
-$process_leveling = read_config_option("process_leveling");
+	/* retreive the number of concurrent process settings */
+	$concurrent_processes = read_config_option("concurrent_processes");
 
-/* retreive the number of concurrent process settings */
-$concurrent_processes = read_config_option("concurrent_processes");
+	/* assume a scheduled task of either 60 or 300 seconds */
+	if (isset($poller_interval)) {
+		$num_polling_items = db_fetch_cell("SELECT COUNT(*) FROM poller_item WHERE rrd_next_step<=0" . ($poller_id == 0 ? "" : " AND poller_id=$poller_id "));
+		$items_perhost     = array_rekey(db_fetch_assoc("SELECT host_id, COUNT(*) AS data_sources
+								FROM poller_item
+								WHERE rrd_next_step<=0 " .
+								($poller_id == 0 ? "" : "AND poller_id=$poller_id ") . "
+								GROUP BY host_id
+								ORDER BY host_id"), "host_id", "data_sources");
+		$poller_runs       = $cron_interval / $poller_interval;
 
-/* assume a scheduled task of either 60 or 300 seconds */
-if (isset($poller_interval)) {
-	$num_polling_items = db_fetch_cell("SELECT COUNT(*) FROM poller_item WHERE rrd_next_step<=0" . ($poller_id == 0 ? "" : " AND poller_id=$poller_id "));
-	$items_perhost     = array_rekey(db_fetch_assoc("SELECT host_id, COUNT(*) AS data_sources
-							FROM poller_item
-							WHERE rrd_next_step<=0 " .
-							($poller_id == 0 ? "" : "AND poller_id=$poller_id ") . "
-							GROUP BY host_id
-							ORDER BY host_id"), "host_id", "data_sources");
-	$poller_runs       = $cron_interval / $poller_interval;
+		define("MAX_POLLER_RUNTIME", $poller_runs * $poller_interval - 2);
+	}else{
+		$num_polling_items = db_fetch_cell("SELECT COUNT(*) FROM poller_item" . ($poller_id == 0 ? "" : " WHERE poller_id=$poller_id "));
+		$items_perhost     = array_rekey(db_fetch_assoc("SELECT host_id, COUNT(*) AS data_sources
+								FROM poller_item " .
+								($poller_id == 0 ? "" : "WHERE poller_id=$poller_id ") . "
+								GROUP BY host_id
+								ORDER BY host_id"), "host_id", "data_sources");
+		$poller_runs       = 1;
 
-	define("MAX_POLLER_RUNTIME", $poller_runs * $poller_interval - 2);
-}else{
-	$num_polling_items = db_fetch_cell("SELECT COUNT(*) FROM poller_item" . ($poller_id == 0 ? "" : " WHERE poller_id=$poller_id "));
-	$items_perhost     = array_rekey(db_fetch_assoc("SELECT host_id, COUNT(*) AS data_sources
-							FROM poller_item " .
-							($poller_id == 0 ? "" : "WHERE poller_id=$poller_id ") . "
-							GROUP BY host_id
-							ORDER BY host_id"), "host_id", "data_sources");
-	$poller_runs       = 1;
+		define("MAX_POLLER_RUNTIME", 298);
+	}
 
-	define("MAX_POLLER_RUNTIME", 298);
-}
+	if (sizeof($items_perhost)) {
+		$items_per_process   = floor($num_polling_items / $concurrent_processes);
+	}else{
+		$process_leveling    = "off";
+	}
 
-if (sizeof($items_perhost)) {
-	$items_per_process   = floor($num_polling_items / $concurrent_processes);
-}else{
-	$process_leveling    = "off";
-}
+	/* some text formatting for platform specific vocabulary */
+	if (CACTI_SERVER_OS == "unix") {
+		$task_type = "Cron";
+	}else{
+		$task_type = "Scheduled Task";
+	}
+/* end of ro */
 
-/* some text formatting for platform specific vocabulary */
-if (CACTI_SERVER_OS == "unix") {
-	$task_type = "Cron";
-}else{
-	$task_type = "Scheduled Task";
-}
+/* basically no writes except the logs ...*/	
+	if (read_config_option('log_verbosity') >= POLLER_VERBOSITY_MEDIUM) {
+		$poller_seconds_sincerun = "never";
+		if (isset($poller_lastrun)) {
+			$poller_seconds_sincerun = $seconds - $poller_lastrun;
+		}
+		cacti_log("NOTE: Poller ID: '$poller_id', Poller Int: '$poller_interval', $task_type Int: '$cron_interval', Time Since Last: '$poller_seconds_sincerun', Max Runtime '" . MAX_POLLER_RUNTIME. "', Poller Runs: '$poller_runs'", TRUE, "POLLER");;
+	}
 
-if (read_config_option('log_verbosity') >= POLLER_VERBOSITY_MEDIUM) {
-	$poller_seconds_sincerun = "never";
-	if (isset($poller_lastrun)) {
-		$poller_seconds_sincerun = $seconds - $poller_lastrun;
+	/* our cron can run at either 1 or 5 minute intervals */
+	if ($poller_interval <= 60) {
+		$min_period = "60";
+	}else{
+		$min_period = "300";
 	}
 
-	cacti_log("NOTE: Poller ID: '$poller_id', Poller Int: '$poller_interval', $task_type Int: '$cron_interval', Time Since Last: '$poller_seconds_sincerun', Max Runtime '" . MAX_POLLER_RUNTIME. "', Poller Runs: '$poller_runs'", TRUE, "POLLER");;
-}
-
-/* our cron can run at either 1 or 5 minute intervals */
-if ($poller_interval <= 60) {
-	$min_period = "60";
-}else{
-	$min_period = "300";
-}
-
-/* get to see if we are polling faster than reported by the settings, if so, exit */
-if ((isset($poller_lastrun) && isset($poller_interval) && $poller_lastrun > 0) && (!$force)) {
-	/* give the user some flexibility to run a little moe often */
-	if ((($seconds - $poller_lastrun)*1.3) < MAX_POLLER_RUNTIME) {
-		if (read_config_option('log_verbosity') >= POLLER_VERBOSITY_MEDIUM) {
-			cacti_log("NOTE: $task_type is configured to run too often!  The Poller ID: '$poller_id', Poller Int: '$poller_interval' seconds, with a minimum $task_type period of '$min_period' seconds, but only " . ($seconds - $poller_lastrun) . ' seconds have passed since the poller last ran.', true, 'POLLER');
+	/* get to see if we are polling faster than reported by the settings, if so, exit */
+	if ((isset($poller_lastrun) && isset($poller_interval) && $poller_lastrun > 0) && (!$force)) {
+		/* give the user some flexibility to run a little moe often */
+		if ((($seconds - $poller_lastrun)*1.3) < MAX_POLLER_RUNTIME) {
+			if (read_config_option('log_verbosity') >= POLLER_VERBOSITY_MEDIUM) {
+				cacti_log("NOTE: $task_type is configured to run too often!  The Poller ID: '$poller_id', Poller Int: '$poller_interval' seconds, with a minimum $task_type period of '$min_period' seconds, but only " . ($seconds - $poller_lastrun) . ' seconds have passed since the poller last ran.', true, 'POLLER');
+			}
+			exit;
 		}
-		exit;
 	}
-}
 
-/* check to see whether we have the poller interval set lower than the poller is actually ran, if so, issue a warning */
-if ((($seconds - $poller_lastrun - 5) > MAX_POLLER_RUNTIME) && ($poller_lastrun > 0)) {
-	cacti_log("WARNING: $task_type is out of sync with the Poller Interval!  The Poller ID: '$poller_id', Poller Int: '$poller_interval' seconds, with a maximum of a '300' second $task_type, but " . ($seconds - $poller_lastrun) . ' seconds have passed since the last poll!', true, 'POLLER');
-}
+	/* check to see whether we have the poller interval set lower than the poller is actually ran, if so, issue a warning */
+	if ((($seconds - $poller_lastrun - 5) > MAX_POLLER_RUNTIME) && ($poller_lastrun > 0)) {
+		cacti_log("WARNING: $task_type is out of sync with the Poller Interval!  The Poller ID: '$poller_id', Poller Int: '$poller_interval' seconds, with a maximum of a '300' second $task_type, but " . ($seconds - $poller_lastrun) . ' seconds have passed since the last poll!', true, 'POLLER');
+	}
+/* end of ro */
 
-if ($poller_id == 0) {
-	db_execute("REPLACE INTO settings (name,value) VALUES ('poller_lastrun'," . $seconds . ')');
-}else{
-	db_execute("REPLACE INTO settings (name,value) VALUES ('poller_lastrun_$poller_id'," . $seconds . ')');
+/* the first "real" write */
+if (!isset($rox)) {
+	if ($poller_id == 0) {
+		db_execute("REPLACE INTO settings (name,value) VALUES ('poller_lastrun'," . $seconds . ')');
+	}else{
+		db_execute("REPLACE INTO settings (name,value) VALUES ('poller_lastrun_$poller_id'," . $seconds . ')');
+	}
 }
+/* end of rw */
 
 /* let PHP only run 1 second longer than the max runtime, plus the poller needs lot's of memory */
 ini_set("max_execution_time", MAX_POLLER_RUNTIME + 1);
@@ -190,6 +200,7 @@
 $poller_runs_completed = 0;
 $poller_items_total    = 0;
 
+/* start of a whole lot of code: truncates replace and updates are of coure rw... */
 while ($poller_runs_completed < $poller_runs) {
 	/* record the start time for this loop */
 	list($micro,$seconds) = split(" ", microtime());
@@ -214,16 +225,20 @@
 	$last_host      = 0;
 
 	/* update web paths for the poller */
+if (!isset($rox)) {
 	if ($poller_id == 0) {
 		db_execute("REPLACE INTO settings (name,value) VALUES ('path_webroot','" . addslashes((CACTI_SERVER_OS == "win32") ? strtr(strtolower(substr(dirname(__FILE__), 0, 1)) . substr(dirname(__FILE__), 1),"\\", "/") : dirname(__FILE__)) . "')");
 	}
+}
 
 	/* obtain some defaults from the database */
 	$poller      = read_config_option("poller_type");
 	$max_threads = read_config_option("max_threads");
 
+if (!isset($rox)) {
 	/* initialize poller_time and poller_output tables, check poller_output for issues */
 	db_execute("TRUNCATE TABLE poller_time");
+}
 
 	$issues = db_fetch_assoc("SELECT local_data_id, rrd_name FROM poller_output" . ($poller_id == 0 ? "" : " WHERE poller_id=$poller_id "));
 	if (sizeof($issues)) {
@@ -312,7 +327,7 @@
 			}
 
 			$host_count ++;
-
+if (!isset($rox)) {
 			if ($change_proc) {
 				exec_background($command_string, "$extra_args --first=$first_host --last=$last_host");
 				usleep(100000);
@@ -324,8 +339,10 @@
 
 				$process_number++;
 			} /* end change_process */
+}
 		} /* end for each */
 
+if (!isset($rox)) {
 		/* launch the last process */
 		if ($host_count > 1) {
 			$last_host = $item["id"];
@@ -335,16 +352,19 @@
 
 			$process_number++;
 		}
-
+}
+if (!isset($rox)) {
 		/* insert the current date/time for graphs */
 		if ($poller_id == 0) {
 			db_execute("REPLACE INTO settings (name,value) VALUES ('date',NOW())");
 		}
-
+}
 		if ($poller == "1") {
 			$max_threads = "N/A";
 		}
 
+/* the start of the intresting code for ro with only log write and cacti stats*/
+/* inputs to this area of code ins $poller_id */
 		/* open a pipe to rrdtool for writing */
 		$rrdtool_pipe = rrd_init();
 
@@ -358,7 +378,7 @@
 				log_cacti_stats($loop_start, $method, $concurrent_processes, $max_threads,
 					sizeof($polling_hosts), $hosts_per_process, $num_polling_items, $rrds_processed);
 
-				break;
+				break;			/* the "normal" exit point for this infinite while */ 
 			}else {
 				if (read_config_option("log_verbosity") >= POLLER_VERBOSITY_MEDIUM) {
 					print "Waiting on " . ($process_number - sizeof($polling_items)) . "/$process_number pollers.\n";
@@ -373,7 +393,7 @@
 					log_cacti_stats($loop_start, $method, $concurrent_processes, $max_threads,
 						sizeof($polling_hosts), $hosts_per_process, $num_polling_items, $rrds_processed);
 
-					break;
+					break;		/* a bad exit from this while loop */
 				}else{
 					sleep(1);
 				}
@@ -381,7 +401,10 @@
 		}
 
 		rrd_close($rrdtool_pipe);
+/* end of intresting code for ro */
 
+if (!isset($rox)) {
+/* probably not needed for ro to end*/
 		/* process poller commands */
 		if (db_fetch_cell("SELECT COUNT(*) FROM poller_command" . ($poller_id == 0 ? "" : " WHERE poller_id=$poller_id ")) > 0) {
 			$command_string = read_config_option("path_php_binary");
@@ -414,7 +437,9 @@
 	}else if (read_config_option('log_verbosity') >= POLLER_VERBOSITY_MEDIUM) {
 		cacti_log("NOTE: There are no items in your poller for this polling cycle!", TRUE, "POLLER");
 	}
+}
 
+
 	$poller_runs_completed++;
 
 	/* record the start time for this loop */
@@ -449,6 +474,7 @@
 		cacti_log("WARNING: Cacti Polling Cycle Exceeded Poller Interval by " . $loop_end-$loop_start-$poller_interval . " seconds", TRUE, "POLLER");
 	}
 }
+/* end of a lot of code ... */
 
 function log_cacti_stats($loop_start, $method, $concurrent_processes, $max_threads, $num_hosts,
 	$hosts_per_process, $num_polling_items, $rrds_processed) {

[oxo-oxo]

Does anyone understand this ?

[oxo-oxo]

Some thoughts:

install 2 identical cacti with fancy HA features
- which algorithm deceides which one is primary (also coordinating with HA on the network card(s) )

primary fails -> secondary new primary -> old primary recovers to secondary role instead of taking over primary role.

primary fails and is then restored with mysql in syn with the new primary
- how does one repair the old primary's rrd files

the main core, a pair of mysql's, would need to keep an eye on their other machine as part of a deceision to "elect" a primary or deceide that the primary is down (the case of temporary network connectivity problems gives an added challenge)
- the other machine is identified as being a machine that is both a master and a slave. Other mysql servers as slaves are not the partner in a HA core (what happens if one makes a 3 server core)

[streaker69]

Maybe I'm a little confused by all this, but wouldn't setting up a Linux cluster pretty much do everything you're asking about?

[mgb]

So if i understand this right. The idea is to have HA build into cacti instead off depening on the OS.

I think this is a good idea. I have setup cacti on both windows and linux systems and setting up a good HA solution is cumbersome to say the least.

Michael

[oxo-oxo]

It is important to understand this post shows how to ensure that RRD data is available using only MySQL replication, and "something" that creates/updates RRD files on a machine that is not "active".

That is, it does not create HA by using Cacti alone, but addresses how to ensure that all data is available on a machine that can takeover Cacti if the primary machine goes down without resorting to copying/syncronising files by some OS specific manner.

Thanks for the feedback

[oxo-oxo]

The next step is "someone" creates a 2 cacti installation with circular mysql replication and continues to hack the poller.php code so there is a Proof of Concept
- unless someone says the idea is a waste of time ...

[oxo-oxo]

As a side note ...

Of course, if a high availability is needed now, one could have 2 cacti's polling the same data sources (for example, network devices).

However, one would need to keep the 2 cacti's identical with the overhead that one would extra management work so as to have HA
- ie click 2 places instead of one for all actions required ...

I am not recommending it, just naming it.

And of course, using some technique to copy RRD files from the primary to the secondary is OK and will work now
- it's just not "nice" as the data is already in the MySQL stream so one might as well use that instead of parrallel data streams of (almost) the same content ...