Script + Cactid + Boost = No Graph Updates

[disirk]

Hey folks,

I have a simple perl script that telnets into a 7206VXR (and ESR) and collects the output from "show caller summary" and parses it into connected PPPoE, PPPoA, and PPPoVPDN. The script outputs all 3 data points

Code: Select all

#!/usr/bin/perl

use Net::Telnet;

my $ip = $ARGV[0];
my $user = "user";
my $password = "password";

my ($pppoe,$pppoa,$pppovpdn) = &Telnet($ip, $user, $password);
print "pppoa:$pppoa pppoe:$pppoe pppovpdn:$pppovpdn";
exit;

The data is being gathered and input into poller_output_boost as shown below:

Code: Select all

mysql> SELECT *
    -> FROM `poller_output_boost`
    -> WHERE output LIKE '%pppoa%';
+---------------+----------+---------------------+-----------------------------------+
| local_data_id | rrd_name | time                | output                            |
+---------------+----------+---------------------+-----------------------------------+
|         92229 |          | 2007-07-31 22:00:32 | pppoa:0 pppoe:90 pppovpdn:0       | 
|         92230 |          | 2007-07-31 22:00:46 | pppoa:554 pppoe:4928 pppovpdn:0   | 
|         92245 |          | 2007-07-31 21:55:39 | pppoa:0 pppoe:0 pppovpdn:0        | 
|         92223 |          | 2007-07-31 22:20:28 | pppoa:0 pppoe:0 pppovpdn:0        | 
|         92235 |          | 2007-07-31 22:20:05 | pppoa:821 pppoe:2463 pppovpdn:0   | 
|         92229 |          | 2007-07-31 21:55:37 | pppoa:0 pppoe:90 pppovpdn:0       | 
|         92221 |          | 2007-07-31 22:20:27 | pppoa:0 pppoe:6184 pppovpdn:0     | 
|         92223 |          | 2007-07-31 22:00:29 | pppoa:0 pppoe:0 pppovpdn:0        | 
|         92224 |          | 2007-07-31 21:55:23 | pppoa:828 pppoe:198 pppovpdn:0    | 
|         92222 |          | 2007-07-31 22:20:27 | pppoa:0 pppoe:1433 pppovpdn:0     | 
|         92235 |          | 2007-07-31 22:15:06 | pppoa:820 pppoe:2471 pppovpdn:0   | 
|         92228 |          | 2007-07-31 21:55:21 | pppoa:0 pppoe:2 pppovpdn:0        | 
|         92233 |          | 2007-07-31 22:10:17 | pppoa:0 pppoe:11960 pppovpdn:0    | 

However, it rarely makes it into the RRD's.



Some times it works, other times it doesn't....any ideas? And yes...I have verified that the info for the graph shown is consistently entered into the DB.

[TheWitness]

Did you update to boost v1.5?

Larry

[disirk]

Yes sir!

[disirk]

Guess I should have elaborated...before the 1.5 update, I was getting nothing at all.

The little data you see, and that I get on random intervals is from 1.5

[TheWitness]

Hmm, what happens when you force a poller_boost.php run? Does it end properly or does php simply croak? I have found that some versions of PHP do not work acceptably. Also, can you tell me if your Database is on the same system as the poller?

Larry

[TheWitness]

In addition, run the following from mysql:

show create table poller_output_boost;

Post here.

Larry

[TheWitness]

Turning in...

You need to watch the boost process status and see that it actually ends. If it does not end properly, then, we need to understand what is causing it to hang up and then stop altogether. If it takes more than 5 minutes to complete a boost dump, then the number of rows remaining in the poller_output_boost table will remain non-zero.

So, watch from the Web Site while you run:

php -q poller_boost.php -f -d

TheWitness

[disirk]

Ok lets see,

Code: Select all

CREATE TABLE `poller_output_boost` (
  `local_data_id` mediumint(8) unsigned NOT NULL default '0',
  `rrd_name` varchar(19) NOT NULL default '',
  `time` datetime NOT NULL default '0000-00-00 00:00:00',
  `output` varchar(512) NOT NULL,
  PRIMARY KEY  (`local_data_id`,`rrd_name`,`time`)
) ENGINE=MEMORY DEFAULT CHARSET=latin1 | 

Poller runs on one machine, I have disabled the "do_hook" in the poller.php that executes the poller_boost, and poller_boost is called via cron on a more robust machine...this machine also stores all the rrd's on a local drive which in turn exports those via NFS to the web server.

poller_boost appears to end just fine...

OK u:2.62 s:10.11 r:1260.73
OK u:2.62 s:10.11 r:1260.76
OK u:2.62 s:10.11 r:1260.78
08/01/2007 03:13:20 AM - SYSTEM BOOST STATS: Time:1303.4053 RRDUpdates:199968

Granted, its a little slow...but I am trying to remedy that with 15k RPM HD's in the near future.


Thanks

[TheWitness]

At 199k updates, how many poller intervals is this? How many RRD files? What about "du ." from the RRA folder?

Larry

[disirk]

Unfortunately, its the equivalent of 2 poller intervals. Although, in the time it is taking to run the poller has polled 4 times.

OUTPUT FROM du .
du .
2 ./.snap
9172068 .

RRD Files: 38574

08/01/2007 11:14:31 AM - SYSTEM BOOST STATS: Time:1231.1425 RRDUpdates:599470
08/01/2007 11:35:49 AM - SYSTEM BOOST STATS: Time:1249.0237 RRDUpdates:399959
08/01/2007 11:58:00 AM - SYSTEM BOOST STATS: Time:1320.2910 RRDUpdates:449731
08/01/2007 12:19:43 PM - SYSTEM BOOST STATS: Time:1242.4155 RRDUpdates:599393
08/01/2007 12:40:03 PM - SYSTEM BOOST STATS: Time:1203.1547 RRDUpdates:299983
08/01/2007 01:02:38 PM - SYSTEM BOOST STATS: Time:1297.6406 RRDUpdates:399928
08/01/2007 01:23:08 PM - SYSTEM BOOST STATS: Time:1207.9326 RRDUpdates:349966
08/01/2007 01:44:05 PM - SYSTEM BOOST STATS: Time:1204.8281 RRDUpdates:499908

Current Boost Status
Boost On Demand Updating: Running
Total Data Sources: 103212
Total Boost Records: 1168215

Those are the last few Boost stats...its all over the place, so I'm thinking about moving the SQL to the same machine so I am not updating over a wire. Also, I'm thinking that perhaps my mysql server may be taxed (still need to look into this). Do you know of anyone who is polling 100k+ datasources every 5 minutes and then using boost to update the RRD's? If so I'd like to talk to them about their setup and any tweaks they may have made to overcome so of them performance issues I am seeing.

-Dave