Boost plugin: lockf'ed continious poller_boost.php

[BorisL]

I want to implement new style of poller_boost.php.

It tries to run after each poll cycle without any time checks, but checks for locked file (so it will know whether another boost process is running already). Because of the lock, it does not need to honor runtime limit at all - lock will automatically and optimally (by disk usage) scheldule rrd updates.

Any thoughts?

[TheWitness]

Boris,

What I am recommending until Boost 2.0 is to rename poller_boost.php to some other name, say poller_boost_force.php. The schedule the poller_boost_force.php -force in cron on the hour or two hour depending on how many RRD's you have.

TheWitness

[BorisL]

until Boost 2.0

I checked "2.0" version from SVN out. It can overrun easily when >10000 datasources are set in configuration. Now I'm playing with poller_boost.php rewriting it to continious scheme, it is pretty simple.

The schedule the poller_boost_force.php -force in cron on the hour or two hour depending on how many RRD's you have.

This sooner or later will produce situation as in this bug splitting one datasource by timestamp.

[TheWitness]

Continuous is a bad idea, I am pretty sure. There is a reason for that, when you understand you will likely stop what you are doing.

TheWitness

[BorisL]

Hmm, last two weeks I was doing this thing: playing with boost parameters to make update of 200k+ datasource items be robust when updating 18Gb of rrds: start: 1hour, max runtime:40min. But it will start overrunning when I add 100k datasource items more simply because it will stays in idle for 20 minutes!
MySQL plays as an bottleneck when you have 3M records even in memory table. May be that poller_output_boost partitioning is a cure, but I am not sure.

So continious run is a only cure for me: it will evenly split load.

[BorisL]

I wrote it.
It works pretty well.

[BorisL]

Let me explain meaning of 'continious' boost running.
In fact my patch disgards maximum run time.
"How Often Should Boost Update All RRD's" and "Maximum Records" are honored as usual.

[TheWitness]

Ah, ok, then that's different. You are simply "Daemonizing" the process. This is fine. I was considering getting rid of the "Maximum" runtime. It's a pain, and has caused some anyway. The reason for it was to attempt to detect a runon (segfault) condition. With windows and Unix, we simply need to track pids.

TheWitness

[BorisL]

Ok, then I can prepare another patch for substituting curren poller_output.php with mine.

[BorisL]

Here it is.

[BorisL]

Another bunch of patches for SVN boost.

Per-graph rrd update is malfunctioning without this patch!
This patch totally eliminates overruns and endless cycling caused by stale data in poller_output_boost table: it checks (and skip stale data) rrd last update time before build update string for updating it.

[TheWitness]

All who read this please note that TRUNK is under heavy development now. It is NOT stable.

TheWitness

[TheWitness]

Boost 2.0 is official. Please Check it Out Here:

http://forums.cacti.net/viewtopic.php?p=150337#150337

Regards,

TheWitness