boost_server.php forgets config

aldobert · Post by **aldobert** » Thu Feb 07, 2008 8:59 am

I'm trying to set up and test a kind of Cacti cluster. My live setup contains Cacti 0.8.7a+plugininfra, 27k datasources, dual dualcore opterons, 8G mem, SAS disk array and so on. Main problem is iowait during rrd updates.

In first phase I separated mysql, but impact to main server load was next to nothing.

Next, I made a test environment where Boost_server, boost_poller and RRD files are on a separate server and sharing RRD directory to main (web+poller) server via NFS. This setup seems very good. Test env contains 2k RRD-s, boost poller runs by crontab in every 15 min and run duration is less than 2 seconds. But boost_server on demand update gets broken after about 20 minutes of run. It starts to reject connections from main server:

Code: Select all

2008:02:07 12:55:29 - Host Connected '192.168.25.120'
02/07/2008 01:36:07 PM - BOOST SERVER: Poller[0] WARNING: Host '192.168.25.120' Not Authorized.

I made some debugging and added variable dumps into this authorization code and found that boost_server.php just misses configuration value for boost_server_clients.

$config before:

Code: Select all

array (
  'url_path' => '/cacti_test/',
  'cacti_server_os' => 'unix',
  'php_snmp_support' => true,
  'base_path' => '/home/cacti/cacti_test',
  'library_path' => '/home/cacti/cacti_test/lib',
  'include_path' => '/home/cacti/cacti_test/include',
  'rra_path' => '/home/cacti/cacti_test/rra',
  'boost_version' => '1.8',
  'config_options_array' =>
  array (
    'availability_method' => '2',
    'ping_method' => '2',
    'ping_port' => '23',
    'ping_timeout' => '1000',
    'ping_retries' => '0',
    'snmp_ver' => '1',
    'snmp_community' => '',
    'snmp_username' => '',
    'snmp_password' => '',
    'snmp_auth_protocol' => 'MD5',
    'snmp_priv_passphrase' => '',
    'snmp_priv_protocol' => 'DES',
    'snmp_port' => '161',
    'snmp_timeout' => '500',
    'max_get_size' => '50',
    'boost_server_effective_user' => 'cacti',
    'boost_server_listen_port' => '9051',
    'path_rrdtool' => '/usr/bin/rrdtool',
    'boost_path_rrdupdate' => '/usr/bin/rrdupdate',
    'path_php_binary' => '/usr/bin/php',
    'path_cactilog' => '/home/cacti/cacti_test/log/cacti.log',
    'poller_verbosity' => NULL,
    'log_verbosity' => '5',
    'boost_server_multiprocess' => '1',
    'boost_server_clients' => '192.168.25.120',
    'log_destination' => '1',
  ),
  'cacti_version' => '0.8.7a',
)

$config after:

Code: Select all

array (
  'url_path' => '/cacti_test/',
  'cacti_server_os' => 'unix',
  'php_snmp_support' => true,
  'base_path' => '/home/cacti/cacti_test',
  'library_path' => '/home/cacti/cacti_test/lib',
  'include_path' => '/home/cacti/cacti_test/include',
  'rra_path' => '/home/cacti/cacti_test/rra',
  'boost_version' => '1.8',
  'config_options_array' =>
  array (
    'availability_method' => '2',
    'ping_method' => '2',
    'ping_port' => '23',
    'ping_timeout' => '1000',
    'ping_retries' => '0',
    'snmp_ver' => '1',
    'snmp_community' => '',
    'snmp_username' => '',
    'snmp_password' => '',
    'snmp_auth_protocol' => 'MD5',
    'snmp_priv_passphrase' => '',
    'snmp_priv_protocol' => 'DES',
    'snmp_port' => '161',
    'snmp_timeout' => '500',
    'max_get_size' => '50',
    'boost_server_effective_user' => 'cacti',
    'boost_server_listen_port' => '9051',
    'path_rrdtool' => '/usr/bin/rrdtool',
    'boost_path_rrdupdate' => '/usr/bin/rrdupdate',
    'path_php_binary' => '/usr/bin/php',
    'path_cactilog' => '/home/cacti/cacti_test/log/cacti.log',
    'poller_verbosity' => NULL,
    'log_verbosity' => '5',
    'boost_server_multiprocess' => '1',
    'boost_server_clients' => '127.0.0.1',
    'log_destination' => '1',
  ),
  'cacti_version' => '0.8.7a',
)

And another bad thing is that the main server loses also the data from poller_output_boost after unsuccessful rrd update against boost_server.

I can handle this issue by some custom hacks but it is good to know if someone else had done similar setup and problems too

aldobert · Post by **aldobert** » Fri Feb 08, 2008 10:35 am

From other logs I found that boost_server goes really mad after that and also found the root cause - mysql connection broke by idle timeout. I had mysql variable wait_timeout set to 900 seconds.

I suggest to add automatic reconnect to boost server because when it's used only for on demand rrd updates, it may get timeout for any timeout value if users do not use cacti wui that long time.

Some examples what happened after timeout:

boost_server.log (boost_server.php -d -f > $LOG):

Code: Select all

2008:02:08 12:05:51 - Host Connected '192.168.25.120'
2008:02:08 12:05:51 - Host Validated '192.168.25.120'
2008:02:08 12:05:51 - RRD Command 'update /home/cacti/cacti_test/rra/cactibe_rb_traf_tx_27529.rrd --template rb_traf_rx:rb_traf_tx 1202472316:6
4956675963:59680022319'
02/08/2008 12:05:51 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: plugins/boost/boost_rrdupdate.php 5
18 /usr/bin/rrdupdate /home/cacti/cacti_test/rra/cactibe_rb_traf_tx_27529.rrd --template rb_traf_rx:rb_traf_tx 1202472316:64956675963:596800223
19]
02/08/2008 12:06:01 PM - BOOST SERVER: Poller[0] WARNING: Broken connection detected '192.168.25.120'

cacti.log@boost_server:

Code: Select all

02/08/2008 12:05:51 PM - CMDPHP: Poller[0] ERROR: SQL Row Failed!, Error:'2006', SQL:"select value from settings where name='poller_verbosity'"
02/08/2008 12:05:51 PM - BOOST: Poller[0] PROGERR: ERRNO:'2' TYPE:'Warning' MESSAGE:'pclose(): 57 is not a valid stream resource' IN FILE:'/hom
e/cacti/cacti_test/lib/rrd.php' LINE NO:'57'
... and thousands of such rows until disk gets full

and cacti.log@frontend server:

Code: Select all

02/08/2008 12:06:01 PM - BOOST: Poller[0] ERROR: Timeout detected.  Boost server is down.  Contact support immediately!!
02/08/2008 12:06:01 PM - BOOST: Poller[0] WARNING: RRD Update Warning 'FAILED' for Local Data ID '27529'

02/08/2008 12:06:01 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "INSERT INTO `rcaston_1202472351` (local_data_id, rrd_name, time) VALUES ('27529',
'rb_traf_rx', '2008-02-08 12:05:16'), ('27529', 'rb_traf_tx', '2008-02-08 12:05:16') ON DUPLICATE KEY UPDATE local_data_id=VALUES(local_data_id
)"
02/08/2008 12:06:01 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "DELETE FROM `poller_output_boost`  USING `poller_output_boost`, `rcaston_120247235
1`  WHERE rcaston_1202472351.local_data_id=poller_output_boost.local_data_id  AND rcaston_1202472351.rrd_name=poller_output_boost.rrd_name  AND
 rcaston_1202472351.time=poller_output_boost.time"
02/08/2008 12:06:01 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "DROP table `rcaston_1202472351`"
02/08/2008 12:06:01 PM - BOOST: Poller[0] PROGERR: ERRNO:'2' TYPE:'Warning' MESSAGE:'socket_write(): 55 is not a valid Socket resource' IN FILE
:'/home/cacti/cacti_test/plugins/boost/setup.php' LINE NO:'1193'
02/08/2008 12:06:01 PM - BOOST: Poller[0] PROGERR: ERRNO:'2' TYPE:'Warning' MESSAGE:'socket_close(): 55 is not a valid Socket resource' IN FILE
:'/home/cacti/cacti_test/plugins/boost/setup.php' LINE NO:'1194'

Post by **TheWitness** » Fri Feb 08, 2008 7:09 pm

Holly crap, nasty dependencies. This is one of my "projects" to get this thing working again.

TheWitness

pooreiple · Post by **pooreiple** » Tue Feb 02, 2010 2:43 pm

how do i get a hb server up global?
i know how to do it lan but had just got a new internet connection so i can host a global server..
any 1 can make ore know a guide to help me
thanks

Post by **TheWitness** » Tue Feb 02, 2010 3:53 pm

pooreiple wrote:how do i get a hb server up global?
i know how to do it lan but had just got a new internet connection so i can host a global server..
any 1 can make ore know a guide to help me
thanks

Your post makes no sense.

TheWitness

simonjones80 · Post by **simonjones80** » Tue Sep 21, 2010 5:21 am

Hi,

Thanks for the great work on boost! It really has changed my life!

I have also had this issue and found it disturbing to find a gap in my graphs.
I think boost should only remove data from the poller_output_boost table once the rrd file has been successfully updated. I guess a rrd update is currently considered a success even if the update is a nan, which it shouldn't be as the data is correct in the DB, but the client is not authorized. If the client it not authorized it should not perform any processing at all. Disconnect before fetching any data maybe?

Just a quick note on the reconnect check you do in boost server. You are probably way ahead of me but it may be a better idea to use something like http://php.net/manual/en/function.mysql-ping.php.

Thanks again for the great work on boost. It is really appreciated!
All the best!

Post by **TheWitness** » Tue Sep 21, 2010 8:04 am

Boost 4.2 already does this.

kanada · Post by **kanada** » Wed Nov 10, 2010 4:55 pm

Unfortunately i still don't see this, and got an error when processing 'On Demand'.

PM - BOOST SERVER: Poller[0] WARNING: Broken connection detected '127.0.0.1'

MySQL 5.1.46, PHP 5.3.3, Cacti 0.8.7g && Spine && Boost 4.2

What's wrong?

Alex

c3226026 · Post by **c3226026** » Fri Mar 04, 2011 9:49 am

kanada wrote:Unfortunately i still don't see this, and got an error when processing 'On Demand'.
PM - BOOST SERVER: Poller[0] WARNING: Broken connection detected '127.0.0.1'
MySQL 5.1.46, PHP 5.3.3, Cacti 0.8.7g && Spine && Boost 4.2

What's wrong?

Alex

Hi,

I've also this with :

MySQL 5.1.49, PHP 5.3.3-7, Cacti 0.8.7g && Spine && Boost 4.3.

I've "solve" this by change Boost Server Timeout 2s -> 20s

And I've never this alarm and no gap on graph.

But I've some alarms when I want to see graphs.

Post by **TheWitness** » Sat Oct 01, 2011 12:47 pm

Use boost 5.0

Cacti

boost_server.php forgets config

boost_server.php forgets config

Fri Feb 08 2008 12 09 pm

Re: Fri Feb 08 2008 12 09 pm

Re: boost_server.php forgets config

Re: boost_server.php forgets config

Re: boost_server.php forgets config

Who is online