Boost 2.5.1 Performance

[Deviloper]

Hi,
having continues performance trouble I added a few lines of code for debugging into the boost_plugin:

####################################
function log_boost_statistics($rrd_updates) {
global $start;
global $rt;
global $ups; # added

/* take time and log performance data */
list($micro,$seconds) = split(" ", microtime());
$end = $seconds + $micro;
$rt = round($end-$start,4); # added
$ups = $rrd_updates / $rt; # added

$cacti_stats = sprintf(
"Time:%01.4f " .
"RRDUpdates:%s ".
"Updates per Second:%s", # added
round($end-$start,4),
$rrd_updates,
$ups); # added
#############################################

Now I have the average updates per second value.
And I noticed that the performance of boost largly depends on the current db table size of the output table. Having a maximum performance around 50 000 - 60 000 entries (rows) the performance dramatically decrease when more than 120 000 are buffered.

Output from my two servers:
Server 1:
02/08/2010 05:38:18 PM - SYSTEM BOOST STATS: Time:42.6188 RRDUpdates:55372 Updates per Second:1299.2388335664

02/08/2010 05:37:30 PM - SYSTEM BOOST STATS: Time:114.2694 RRDUpdates:93794 Updates per Second:820.81467129433

02/08/2010 05:37:24 PM - SYSTEM STATS: Time:142.0007 Method:spine Processes:10 Threads:100 Hosts:131 HostsPerProcess:14 DataSources:149166 RRDsProcessed:0

Server 2:
02/08/2010 05:42:56 PM - SYSTEM BOOST STATS: Time:130.0987 RRDUpdates:101896 Updates per Second:783.22073933098

02/08/2010 05:40:55 PM - SYSTEM STATS: Time:53.7142 Method:spine Processes:10 Threads:100 Hosts:158 HostsPerProcess:16 DataSources:148024 RRDsProcessed:0

02/08/2010 05:40:40 PM - SYSTEM BOOST STATS: Time:33.9636 RRDUpdates:15081 Updates per Second:444.03420132141

02/08/2010 05:38:06 PM - SYSTEM BOOST STATS: Time:82.3847 RRDUpdates:55534 Updates per Second:674.0814738659

02/08/2010 05:36:38 PM - SYSTEM BOOST STATS: Time:85.0914 RRDUpdates:61443 Updates per Second:722.0823726017

02/08/2010 05:35:55 PM - SYSTEM STATS: Time:53.2083 Method:spine Processes:10 Threads:100 Hosts:158 HostsPerProcess:16 DataSources:148024 RRDsProcessed:0


#############################################
Boost Status Server 1.
Boost On Demand Updating: Idle
Total Data Sources: 149166
Boost Storage Statistics
Database Engine: MEMORY
Current Boost Table Size: 32 MBytes
Avg Bytes/Record: 97 Bytes
Max Allowed Boost Table Size: 1 GBytes
Estimated Maximum Records: 11842740 Records
Previous Runtime
Last Start Time: 2010-02-08 17:54:51
Last Run Duration: 0 seconds (0% of update frequency)
RRD Updates: 0
Peak Poller Memory: 8.01 MBytes
Max Poller Memory Allowed: 3072 MBytes
Run Time Configuration
Update Frequency: 30 Minutes
Next Start Time: 2010-02-08 18:24:51
Maximum Records: 500000 Records
Maximum Allowed Runtime: 1 Hour
Boost Server Details
Server Config Status: Enabled
Multiprocess Server: Multiple Process
Update Timeout: 2 Seconds
Server/Port: localhost@9050
Authorized Update Web Servers: 127.0.0.1
RRDtool Binary Used: /usr/bin/rrdupdate
Image Caching
Image Cacing Status: Disabled
Cache Directory: /apache2/logs/cacticache
Cached Files: 0 Files
Cached Files Size: 0 Bytes
############################################

Boost status Server 2.:
Boost On Demand Updating: Running
Total Data Sources: 148024
Total Boost Records: 4176
Boost Storage Statistics
Database Engine: MEMORY
Current Boost Table Size: 12 MBytes
Avg Bytes/Record: 97 Bytes
Max Record Length: 14 Bytes
Max Allowed Boost Table Size: 1 GBytes
Estimated Maximum Records: 11842740 Records
Previous Runtime
Last Start Time: 2010-02-08 17:51:29
Last Run Duration: 1 minutes 11 seconds (4% of update frequency)
RRD Updates: 46208Updates
Peak Poller Memory: 49.09 MBytes
Max Poller Memory Allowed: 3072 MBytes
Run Time Configuration
Update Frequency: 30 Minutes
Next Start Time: 2010-02-08 18:20:12
Maximum Records: 500000 Records
Maximum Allowed Runtime: 1 Hour
Boost Server Details
Server Config Status: Enabled
Multiprocess Server: Multiple Process
Update Timeout: 2 Seconds
Server/Port: localhost@9050
Authorized Update Web Servers: 127.0.0.1
RRDtool Binary Used: /usr/bin/rrdupdate
Image Caching
Image Cacing Status: Disabled
Cache Directory: /apache2/logs/cacticache
Cached Files: 0 Files
Cached Files Size: 0 Bytes
############################################

Forgett about the last runtime values. I am running boost permanent in the background to update the rrds, so polling and updating is separated into two processes.

[TheWitness]

First of all, for 150k data sources, I would not update the boost records any more frequently than once every two hours, not 30 minutes.

Second, you might want to convert some of your tables to Innodb, but not before you have it tuned Innodb properly though. The default Innodb settings are not adequate.

I know of a few users that have >400k Data Sources. One of which does not update Boost until 9M records are in the boost table, and that update takes ~13 minutes to execute.

TheWitness