Boost performance ?

[msylvain]

Hello !

I just set up 2 days ago the boost plugin, and hopping a decrease of the load average of the machine. I'm a little disapointed, because i see now that the load average and the cpu load is higher than before boost. The only benefice are on disk i/o and on polling time.
I attached 4 graphs to this post to illustrate it.

Here is my Boost status :

Code: Select all

Current Boost Status
Boost On Demand Updating:	Idle
Total Data Sources:	37135
Total Boost Records:	442481

Boost Storage Statistics
Database Engine:	MyISAM
Current Boost Table Size:	94 MBytes
Avg Bytes/Record:	223 Bytes
Max Allowed Boost Table Size:	Unlimited
Estimated Maximum Records:	Unlimited Records

Runtime Statistics
Last Start Time:	2010-03-24 15:08:50
Last Run Duration:	3 minutes 54 seconds (7% of update frequency)
RRD Updates:	441878
Peak Poller Memory:	30.87 MBytes
Detailed Runtime Timers:	RRDUpdates:441878 TotalTime:234.2244 range_local_data_id:5.01 rcaston_add:9.35 get_records:11.33 results_cycle:156.64 rrd_path:22.01 rrd_template:43.87 rrd_lastupdate:6.53 rrd_field_names:17.95 rrdupdate:38.58 delete:40.3
Max Poller Memory Allowed:	512 MBytes

Run Time Configuration
Update Frequency:	1 Hour
Next Start Time:	2010-03-24 16:08:50
Maximum Records:	1000000 Records
Maximum Allowed Runtime:	20 Minutes

Boost Server Details
Server Config Status:	Enabled
Multiprocess Server:	Multiple Process
Update Timeout:	5 Seconds
Server/Port:	localhost@9050
Authorized Update Web Servers:	127.0.0.1
RRDtool Binary Used:	/opt/rrdtool/bin/rrdupdate

Image Caching
Image Cacing Status:	Enabled
Cache Directory:	/var/www/cacti_ng/boostimgcache/
Cached Files:	11242 Files
Cached Files Size:	274 MBytes

Here are general stats on my cacti :

Code: Select all

03/24/2010 03:13:49 PM - SYSTEM THOLD STATS: Time:0.0054 Tholds:0 Hosts:0
03/24/2010 03:13:49 PM - SYSTEM STATS: Time:47.8314 Method:spine Processes:1 Threads:16 Hosts:295 HostsPerProcess:295 DataSources:37135 RRDsProcessed:0 

rrd files are updated once per hour, but i plan to set it to 2 hours, and my max_heap_table_size is set to 256 Mo.

For now, only 2 tables are Memory tables : poller_output and poller_output_boost_processes; the others are MyISAM.

Can someone tell me where i'm wrong with my boost setup, or if boost is just not for me ?

regards,

sylvain

[TheWitness]

poller_output_boost is the main table that should be memory. However, you must make sure that:

1) your output column width is optimal (aka as small as it can be)
2) You have enough memory to handle memory tables
3) Your max_heap_table_size is large enough to handle the number of records you choose to cache.
4) Don't flush boost too often. I typically size it so that you can flush about every 4 hours.

TheWitness

[msylvain]

Thanks for your so fast answer !

1) your output column width is optimal (aka as small as it can be)

I'm still tuning it, and i just realized that for now, i set up the output field to be a varchar(190), which is not enought, since i'm using an apache script server polling returning more than 300 characters

2) You have enough memory to handle memory tables

Well, My server has 2Go of RAM.
2Go ought to be enough for anybody (tm)

3) Your max_heap_table_size is large enough to handle the number of records you choose to cache.

here is the size of my poller_output table :

Code: Select all

+---------------+-----------------------+------+-----+---------------------+-------+
| Field         | Type                  | Null | Key | Default             | Extra |
+---------------+-----------------------+------+-----+---------------------+-------+
| local_data_id | mediumint(8) unsigned | NO   | PRI | 0                   |       |3 octets
| rrd_name      | varchar(19)           | NO   | PRI |                     |       |20 octets
| time          | datetime              | NO   | PRI | 0000-00-00 00:00:00 |       |8 octets
| output        | varchar(190)          | NO   |     |                     |       |191 octets
+---------------+-----------------------+------+-----+---------------------+-------+

and here is how i calculate my max_heap_table_size :

total : 3+20+8+191 = 222 octets, lets say 256 octets


for 37k datasources, lets say 40k, during 2 hours, that is to say 2*12 pollings, i think i need
256 * 40000 * 2 * 12 = 245760000

256Mo seems to be a large enought choice. Maybe i'll try 512Mo for 4 hours between updates

Tell me if i'm wrong.

[TheWitness]

Your right on track.

TheWitness