Boost Question regarding update frequency

[yousillygoose]

Hey all,

So we poll our servers every minute. We have a handful of cacti servers monitoring a very large number of servers.

We have been running boost for a long time. Recently (Friday) we started having an issue with one of our cacti servers and are still trying to debug it. On Friday, the poller_output_boost table became corrupt and we had to drop and recreate it. Our graphs were gapping pretty bad when we realized this was happening (our nightly maintenance script had failed).

After looking into the server for a while it looks like boost is taking 118% of its update frequency in order to finish it's job. Here is some of our boost information.

Current Boost Status
Boost On Demand Updating: Running
Total Data Sources: 49773
Total Boost Records: 1602916
Boost Storage Statistics
Database Engine: MyISAM
Current Boost Table Size: 69 MBytes
Avg Bytes/Record: 41 Bytes
Max Allowed Boost Table Size: Unlimited
Estimated Maximum Records: Unlimited Records
Runtime Statistics
Last Start Time: 2012-01-26 9:03:59
Last Run Duration: 35 minutes 28 seconds (118% of update frequency)
RRD Updates: 1419821
Peak Poller Memory: 87.79 MBytes
Detailed Runtime Timers: RRDUpdates:1419821 TotalTime:2128.4364 range_local_data_id:10.08 rcaston_add:24.53 get_records:24.7 results_cycle:1988.52 rrd_path:40.17 rrd_template:39.74 rrd_lastupdate:27.25 rrd_field_names:27.32 rrdupdate:1812.53 delete:67.99
Max Poller Memory Allowed: 1024 MBytes
Run Time Configuration
Update Frequency: 30 Minutes
Next Start Time: 2012-01-26 9:27:59
Maximum Records: 1000000 Records
Maximum Allowed Runtime: 40 Minutes


You can see a few things here. First, we run boost every 30 minutes. Second, boost took 35 min to run (longer than the actual recurring interval). Every 30 minutes we gather around 1.5 million data points on this server. Boost takes slightly longer than 30 minutes to process this.

My question for you is what happens to the data boost processes? It seems that boost takes 35 minutes to process 30 minutes of data. To me, that says that every half hour we will be falling another 5 minutes of data behind. If this is the case, what are the negative repercussions?

We are running cacti 0.8.7d on windows. Boost version: 4.3. This setup has been working for a long time for us and we are a bit confused as to why our problems came out of the blue on Friday.

Thanks!

[lightenup]

I'm nearly in the same boat as you:

01/28/2012 11:30:52 AM - SYSTEM THOLD STATS: Time:8.6377 Tholds:3932 DownHosts:0
01/28/2012 11:30:43 AM - SYSTEM STATS: Time:24.2763 Method:spine Processes:4 Threads:15 Hosts:598 HostsPerProcess:150 DataSources:108207 RRDsProcessed:0
...
01/28/2012 05:46:29 AM - SYSTEM BOOST STATS: Time:964.6900 RRDUpdates:1082391
01/28/2012 04:49:11 AM - SYSTEM BOOST STATS: Time:526.4200 RRDUpdates:1082389
01/28/2012 03:59:06 AM - SYSTEM BOOST STATS: Time:521.9700 RRDUpdates:1082385
01/28/2012 03:17:13 AM - SYSTEM BOOST STATS: Time:1004.5000 RRDUpdates:1082389
01/28/2012 02:22:31 AM - SYSTEM BOOST STATS: Time:725.8500 RRDUpdates:1082382
01/28/2012 01:49:18 AM - SYSTEM BOOST STATS: Time:1732.7700 RRDUpdates:1079499

At times you can see I peak upwards to 28 minutes, but my fastest time is around 8 minutes. I have my Cacti installation running on a 3gz quad core HP DL380, w/10gb of ram and fiber channel storage (not really sure how its configured on the EMC). I will tell you that I run my boost updates every hour though.

[gandalf]

I have my Cacti installation running on a 3gz quad core HP DL380, w/10gb of ram and fiber channel storage (not really sure how its configured on the EMC). I will tell you that I run my boost updates every hour though.

This is quite similar to our setup (FC SAN). It is quite important, how busy the SAN is during updates. E.g. on weekends, BOOST flush is _really_ fast compared to th erest of the week. Please ask your SAN admin
R.

[yousillygoose]

Our storage isn't on SAN- it's local raided storage. I believe it was decided that our SAN was too expensive to use as a solution. Any insight on my first post?

[gandalf]

Moving to plugins forum; don't know a solution to this
R.