Boost Plugin v1.6 Released

[TheWitness]

All,

I have released V1.3 of the boost plugin. This plugin address two issues with processing RRD files, adds additional logging for troubleshooting problems, and allows you to increase performance by tweaking string sizes for varying system types.

If you use the attached archive and are on Linux/UNIX, make sure your run the following command: "unzip -a boost-v1.3.zip" in order to convert your files to Linux/UNIX compatible files.

Here is the ChangeLog:

----[ Changelog
--- SVN ---

--- 1.0 ---
Initial release

--- 1.1 ---
bug: Fix issues with Multiprocess RRDupdating of RRD Files

--- 1.2 ---
bug: Added boost_server.php and poller_boost.php to the no session array

bug: Made slight change to rrd-update functions to accomodate for abarant
output in the poller_output_boost table. This would cause the boost
plugin to loose data if improperly formatted data made it into the table.

--- 1.3 ---
bug: Fixed an issue where multi part responses with a "0" would return "U"

feature: Added better logging functions to help in debugging

feature: Allow you to increase the MySQL Insert string length to improve performance

feature: Allow you to increase the RRDtool update string length to improve performance

Regards,

TheWitness

[TheWitness]

All,

Working with rcaston this weekend, I have corrected a pretty significant bug in Boost. Thanks to rcaston for brining this to my attention. The latest boost release is now v1.4.

Regards,

TheWitness

[vdragosh]

BUG : when a rrd file has more DataSouces then DataSources used, setup.php puts an aditional ":" after template list

From logs :
07/12/2007 10:37:10 AM - BOOST: Poller[0] CACTI2RRD: /usr/bin/rrdtool update /var/www/cacti.******.ro/htdocs/rra/sediu_free_pages_2909.rrd --template pool_size:free_pages:database_pages:modified_pages: 1184225107465:557:0 1184225407465:557:0 1184225707465:557:0

It is weird that i get no errors in logs. I do not have time to find out why.
If i run that line i get this error :

Code: Select all

# /usr/bin/rrdtool update /var/www/cacti.******.ro/htdocs/rra/sediu_free_pages_2909.rrd --template pool_size:free_pages:database_pages:modified_pages: 1184225107:1024:465:557:0 1184225407:1024:465:557:0 1184225707:1024:465:557:0
ERROR: unknown DS name ''

original code in setup.php:

Code: Select all

for ($i=0; $i<count($values); $i++) {
        if (preg_match("/^([a-zA-Z0-9_\.-]+):([eE0-9\+\.-]+)$/", $values[$i], $matches)) {
                if (isset($rrd_field_names{$matches[1]})) {
                        if (read_config_option("log_verbosity") == POLLER_VERBOSITY_DEBUG) {
                                cacti_log("Parsed MULTI output field '" . $matches[0] . "' [map " . $matches[1] . "->" . $rrd_field_names{$matches[1]} . "]" , false, "BOOST");
                        }

                        if (!$multi_vals_set) {
                                $rrd_tmpl .= $rrd_field_names{$matches[1]};
                        }

                        if ((!is_numeric($matches[2])) && ($matches[2] != 0)) {
                                $outbuf .= ":U";
                        }else{
                                $outbuf .= ":" . $matches[2];
                        }

                        if (!$multi_vals_set) {
                                if ($i+1 < count($values)) {
                                        $rrd_tmpl .= ":";
                                }
                        }
                }
        }
}

modified code in setup.php:

Code: Select all

$first_tmpl=1;
for ($i=0; $i<count($values); $i++) {
        if (preg_match("/^([a-zA-Z0-9_\.-]+):([eE0-9\+\.-]+)$/", $values[$i], $matches)) {
                if (isset($rrd_field_names{$matches[1]})) {
                        if (read_config_option("log_verbosity") == POLLER_VERBOSITY_DEBUG) {
                                cacti_log("Parsed MULTI output field '" . $matches[0] . "' [map " . $matches[1] . "->" . $rrd_field_names{$matches[1]} . "]" , false, "BOOST");
                        }

                        if (!$multi_vals_set) {
                                if (!$first_tmpl) $rrd_tmpl .= ":";
                                $rrd_tmpl .= $rrd_field_names{$matches[1]};
                                $first_tmpl=0;
                        }

                        if ((!is_numeric($matches[2])) && ($matches[2] != 0)) {
                                $outbuf .= ":U";
                        }else{
                                $outbuf .= ":" . $matches[2];
                        }
                }
        }
}

or you can use the patch :

Code: Select all

# diff -Naur setup.php setup.php.old
--- setup.php   2007-07-12 12:24:30.000000000 +0300
+++ setup.php.old       2007-07-12 10:59:48.000000000 +0300
@@ -795,7 +795,6 @@
                                        $rrd_tmpl = "";
                                }

-                               $first_tmpl=1;
                                for ($i=0; $i<count($values); $i++) {
                                        if (preg_match("/^([a-zA-Z0-9_\.-]+):([eE0-9\+\.-]+)$/", $values[$i], $matches)) {
                                                if (isset($rrd_field_names{$matches[1]})) {
@@ -804,9 +803,7 @@
                                                        }

                                                        if (!$multi_vals_set) {
-                                                               if (!$first_tmpl) $rrd_tmpl .= ":";
                                                                $rrd_tmpl .= $rrd_field_names{$matches[1]};
-                                                               $first_tmpl=0;
                                                        }

                                                        if ((!is_numeric($matches[2])) && ($matches[2] != 0)) {
@@ -814,6 +811,12 @@
                                                        }else{
                                                                $outbuf .= ":" . $matches[2];
                                                        }
+
+                                                       if (!$multi_vals_set) {
+                                                               if ($i+1 < count($values)) {
+                                                                       $rrd_tmpl .= ":";
+                                                               }
+                                                       }
                                                }
                                        }
                                }

PS: sorry for my bad english.

[TheWitness]

Thanks for the fix.

Larry

[leonardo_gyn]

Did i miss something ?!?! Are you sure this patch applies on v1.4 ??? I search my setup.php from unmodified v1.4 and couldnt find the instance first_tmpl which the patch refeers.

[vdragosh]

Yes, you missed something
first_tmpl is added in the patch .
If you want to find the for loop just go to line 798 or search the string "i<count"

I think you mixed up original code with modified code

[leonardo_gyn]

That's it ...... you diffed NEWFILE OLDFILE ..... i usually use to diff OLDFILE NEWFILE. I would have done:

diff -Naur setup.php.old setup.php

That would make easy to see what's added and what's removed .... your diff is reversed, i didnt realize that.

OK guys .... sorry for that

[TheWitness]

I will be releasing v1.5 sometime in the next 72 hours. It addresses this issue and another when attempting to FORCE a poller_output_boost dump that in some circumstances would not execute.

Both changes are in SVN is you are in a hurry.

TheWitness

[leonardo_gyn]

Hi TheWitness ..... i think i found another interesting situation here in v1.4.

I have 'On Demand RRD Updating' enabled and 'How Often Should Boost Update All RRD's' set to 4 hours. So, RRDs will be written to disk every 4 hours.

My last BOOST updated happened about 4:10 PM

Code: Select all

07/14/2007 04:10:38 PM - SYSTEM BOOST STATS: Time:7.0927 RRDUpdates:7275

So, next one would happen about 8PM and some minutes according to my settings ....

At about 7PM, i created a new graph, which uses 3 datasources. It's a normal Linux Load graph, one datasource for 1 minute, other 5 minutes and last one 15 min.

Watching logs, i can see those datasources are being correctly collected and parsed:

07/14/2007 07:05:05 PM - CACTID: Poller[0] Host[16] DS[522] SNMP: v1: 192.168.1.7, dsname: load_1min, oid: .1.3.6.1.4.1.2021.10.1.3.1, value: 0.37
07/14/2007 07:05:05 PM - CACTID: Poller[0] Host[16] DS[524] SNMP: v1: 192.168.1.7, dsname: load_5min, oid: .1.3.6.1.4.1.2021.10.1.3.2, value: 0.30
07/14/2007 07:05:05 PM - CACTID: Poller[0] Host[16] DS[523] SNMP: v1: 192.168.1.7, dsname: load_15min, oid: .1.3.6.1.4.1.2021.10.1.3.3, value: 0.20

07/14/2007 07:15:05 PM - CACTID: Poller[0] Host[16] DS[522] SNMP: v1: 192.168.1.7, dsname: load_1min, oid: .1.3.6.1.4.1.2021.10.1.3.1, value: 0.13
07/14/2007 07:15:05 PM - CACTID: Poller[0] Host[16] DS[524] SNMP: v1: 192.168.1.7, dsname: load_5min, oid: .1.3.6.1.4.1.2021.10.1.3.2, value: 0.20
07/14/2007 07:15:05 PM - CACTID: Poller[0] Host[16] DS[523] SNMP: v1: 192.168.1.7, dsname: load_15min, oid: .1.3.6.1.4.1.2021.10.1.3.3, value: 0.18

07/14/2007 07:55:06 PM - CACTID: Poller[0] Host[16] DS[522] SNMP: v1: 192.168.1.7, dsname: load_1min, oid: .1.3.6.1.4.1.2021.10.1.3.1, value: 0.14
07/14/2007 07:55:06 PM - CACTID: Poller[0] Host[16] DS[524] SNMP: v1: 192.168.1.7, dsname: load_5min, oid: .1.3.6.1.4.1.2021.10.1.3.2, value: 0.17
07/14/2007 07:55:06 PM - CACTID: Poller[0] Host[16] DS[523] SNMP: v1: 192.168.1.7, dsname: load_15min, oid: .1.3.6.1.4.1.2021.10.1.3.3, value: 0.17

(last one before Boost RRD Update happens)
07/14/2007 08:15:07 PM - CACTID: Poller[0] Host[16] DS[522] SNMP: v1: 192.168.1.7, dsname: load_1min, oid: .1.3.6.1.4.1.2021.10.1.3.1, value: 0.06
07/14/2007 08:15:07 PM - CACTID: Poller[0] Host[16] DS[524] SNMP: v1: 192.168.1.7, dsname: load_5min, oid: .1.3.6.1.4.1.2021.10.1.3.2, value: 0.12
07/14/2007 08:15:07 PM - CACTID: Poller[0] Host[16] DS[523] SNMP: v1: 192.168.1.7, dsname: load_15min, oid: .1.3.6.1.4.1.2021.10.1.3.3, value: 0.15

after 20h15 pooling process, Boost RRD Update happens !

I see on the logs the RRD being created:

Code: Select all

07/14/2007 08:15:34 PM - BOOST: Poller[0] CACTI2RRD: /usr/local/rrdtool-1.2.23/bin/rrdtool create /home/httpd/html/cacti2/rra/[b]mail_gateway_sp_load_1min_522.rrd[/b]   --start 1184449805 --step 300   DS:load_1min:GAUGE:600:0:500  RRA:AVERAGE:0.5:1:600  RRA:AVERAGE:0.5:6:700  RRA:AVERAGE:0.5:24:775  RRA:AVERAGE:0.5:1:115200  RRA:MIN:0.5:1:600  RRA:MIN:0.5:6:700  RRA:MIN:0.5:24:775  RRA:MIN:0.5:1:115200  RRA:MAX:0.5:1:600  RRA:MAX:0.5:6:700  RRA:MAX:0.5:24:775  RRA:MAX:0.5:1:115200  RRA:LAST:0.5:1:600  RRA:LAST:0.5:6:700  RRA:LAST:0.5:24:775  RRA:LAST:0.5:1:115200

07/14/2007 08:15:35 PM - BOOST: Poller[0] CACTI2RRD: /usr/local/rrdtool-1.2.23/bin/rrdtool create /home/httpd/html/cacti2/rra/mail_gateway_sp_load_5min_524.rrd   --start 1184449805 --step 300   DS:load_5min:GAUGE:600:0:500  RRA:AVERAGE:0.5:1:600  RRA:AVERAGE:0.5:6:700  RRA:AVERAGE:0.5:24:775  RRA:AVERAGE:0.5:1:115200  RRA:MIN:0.5:1:600  RRA:MIN:0.5:6:700  RRA:MIN:0.5:24:775  RRA:MIN:0.5:1:115200  RRA:MAX:0.5:1:600  RRA:MAX:0.5:6:700  RRA:MAX:0.5:24:775  RRA:MAX:0.5:1:115200  RRA:LAST:0.5:1:600  RRA:LAST:0.5:6:700  RRA:LAST:0.5:24:775  RRA:LAST:0.5:1:115200  

07/14/2007 08:15:35 PM - BOOST: Poller[0] CACTI2RRD: /usr/local/rrdtool-1.2.23/bin/rrdtool create /home/httpd/html/cacti2/rra/mail_gateway_sp_load_15min_523.rrd   --start 1184449805 --step 300   DS:load_15min:GAUGE:600:0:500  RRA:AVERAGE:0.5:1:600  RRA:AVERAGE:0.5:6:700  RRA:AVERAGE:0.5:24:775  RRA:AVERAGE:0.5:1:115200  RRA:MIN:0.5:1:600  RRA:MIN:0.5:6:700  RRA:MIN:0.5:24:775  RRA:MIN:0.5:1:115200  RRA:MAX:0.5:1:600  RRA:MAX:0.5:6:700  RRA:MAX:0.5:24:775  RRA:MAX:0.5:1:115200  RRA:LAST:0.5:1:600  RRA:LAST:0.5:6:700  RRA:LAST:0.5:24:775  RRA:LAST:0.5:1:115200  

But there's no 'rrdtool update' at all !!! In the above case, i lost about 1 hour of data that was correctly collected, correctly parsed, correctly stored on poller_output_boost table but NOT correctly updated on the RRD file. Next updates didn't updated the lost data as well.

I made another test ..... created the graph, let the poller successfully run for 4-5 times and watched the graph through cacti interface, which forces the updating process. Nothing shows on the logs, even with DEBUG, but i know CREATE happens because the RRD file was created on the disk. But again, with no values, it was just created and apparently data was lost. After the initial CREATE, tough, things happens as expected. A force update, through a graph watching, would issue the expected UPDATEs and graph shows with no problems.

So, as i observed, people using boost can loose some hours of collected values, if they create the graph and didn't force the RRD create process.

This is completly dependant on boost settings. A '10 minutes' on 'How Often....' would made the problem almost dissapear. It would still exist, but data lost would be 1-2 pooling process only. On the other side, with 6 hours, people could potentially lost great amount of data.

Not a MAJOR bug indeed. But loosing collected data is always a bad idea

[TheWitness]

Ok, well. I have found the following to be true. Here are some details:

1) If you do not have enough PHP memory boost may not run well at all
2) Certain versions of PHP do not play well with boost. If you don't have up2date or the equivalent for your OS, be very cautious. I saw one instance this week where the system would start to update rrd's and then PHP would just end. No explanation, no log nothing. That is a problem, but not linked to the code.
3) If you run memory tables and you allow them to get too big, bad things happen.

I could go on an on with additional scnearios, but the point is that, per my initial statement in the release notes.

1) Always have a test system.
2) Always test before introducing something like boost.
3) If you listen, you forget, if you see, you remember, if you do, you will understand. By doing, you will understand boost. So, to keep the pain low. Do in test.

Larry

[leonardo_gyn]

Yeah i understand completly the importance of testing boost before deploying in producting, because it's a huge change on cacti way of working.

In fact, these strange things i found (this last one and the zero one), i found them in my test environment !!! I have not deployed boost yet. I'm trying to exaust all situations i can try and i will have in real production systems. The problem is that i think several users will a little before deploying boost, which may lead to really undesired problems to them. Because of these several tests i'm doing, luckly some bugs came up, we can fix them and avoid further people from having problems.

About memory and PHP versions, specially PHP version, it would be very nice if we could have some specific version or specific vendor which has problems with boost. In my case (PHP 5.1.6 hand compiled on Fedora Core 5), i didn't realized ANY 'incorrect' behavior but the zero bug and this last one, which i consider a bug as well.

Anyway TheWitness, boost is an awesome cacti addon. Sure it's not needed by all users. Small and medium cacti environment, in my opinion, would live just nice without it. I'm studying it but for sure i'll not use it on all cactis i'm running.

How about saying something about that on RELEASE and stuff .... that this is intended for BIG environments and advantages would exist, but wouldnt be noticed on small/medium cacti environments ??? It would be interesting to give some numbers to people base their decision on as well. Since when you would consider a cacti environment a big one ? How many devices, how many data sources, things like that. What do you think ?

[TheWitness]

I know of some Cacti environments in excess of 3400 hosts. Only possible with my new Cactid. 3400+ hosts in 38 seconds. Turning in now...

TheWitness

[leonardo_gyn]

Woooooow, 3400+ hosts is really something

[TheWitness]

Boost v1.5 Released. There are a few fixes/enhancements.

TheWitness

[TheWitness]

All,

Boost v1.6 release with some major performance changes. Read the ChangeLog for more details.

Regards,

TheWitness