Poller overruns after 1.2.23 upgrade

[TheWitness]

The remotes don't need a lot so those are sized appropriately in my opinion.

[TheWitness]

Honestly the remotes should maybe be like one process ten threads really cuz that gets you 10 hosts at a time and I imagine they don't have more than 10 hosts per remote

[TheWitness]

In the newer spine binaries we are doing thread pools so that's going to exhaust your main servers total available processes or Max underscore connections

[KevinNoall]

I will work to get the main poller more resources, but why would it change so dramatically between minor versions, and having downgraded to the older working spine v1.2.22? effectively my install is now down/offline. i'll have to downgrade the entire install to get working again.

[TheWitness]

The change in spine was fairly significant not minor as you indicate. So what it does now is it allocates all threads before it even starts handing them out to devices so really you need to minimize your thread use especially with as many data collectors as you have or get a bigger database server.

MySQL does not do so well with several hundred or if not a thousand open connections with so few CPU cores.

You definitely need to either reduce the thread count or go back to an older spine or both with such a small database server.

[KevinNoall]

First off let me remember myself and thank you greatly for everything you and the cacti team do. In addition thank you for your prompt responses to this problem, it really is greatly appreciated.

I now have all remote pollers disabled, and i have no devices at all in the main poller scope (effectively there are no enabled devices at all), am on spine v1.2.22, all plugins disabled, and the main poller continues to overrun.

i've requested additional resources for the main poller VM but that may take a bit of time.

[TheWitness]

Sorry about the spine thing. That was me. I support a one poller Cacti install with tens of thousands of hosts and 2+ million graphs. It presents a lot of challenges without using a connection pool.

The database server alone has 120 threads and there are three web servers behind a load balancer. Pretty impressive setup.

[TheWitness]

Having everything disabled is likely a negative test case which might be generating those errors. Better enable a few hosts.

[KevinNoall]

enabled a few hosts and it did not seem to change anything

[KevinNoall]

Also still getting boost crash messages every minute

[TheWitness]

Has to be a negative test case. Suggest you open a ticket and post your cacti.log in the ticket.

[KevinNoall]

should that be in cacti or spine?

[TheWitness]

Cacti.

[jkrejci]

Piggybacking on this thread...

I am experiencing the same issue with the same error message. New install so I can blow away any data, settings, etc. I am looking to deploy a completely fresh install of multiple pollers to replace a single server that is overwhelmed and is out of date.

My current new setup is one main poller and one remote poller. I have 3 devices in cacti, currently all configured on the primary poller. I am sure my config is now far from ideal as I have butchered a lot of settings on this road I have traveled to figure out what is wrong.

OS: Ubuntu 22.04.2 LTS
Cacti version 1.2.24
Spine version 1.2.24

Excluding cacti/spine, current apt packages used for dependencies: mariadb, rrdtool, apache, php, etc.

When I run the graph in real-time mode, the pop out graph data does populate properly.

Code: Select all

2023-03-21 17:59:59 - BOOST ERROR: Failed to retrieve any rows from archive tables
2023-03-21 17:59:59 - SYSTEM MAINT STATS: Time:0.01
2023-03-21 17:59:59 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2023-03-21 17:59:59 - BOOST SVR WARNING: Detected Poller Boost Overrun, Possible Boost Poller Crash
2023-03-21 17:59:59 - SYSTEM STATS: Time:298.5004 Method:spine Processes:1 Threads:20 Hosts:3 HostsPerProcess:3 DataSources:732 RRDsProcessed:0
2023-03-21 17:59:59 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2023-03-21 17:59:59 - POLLER: Poller[1] PID[1730] Maximum runtime of 298 seconds exceeded. Exiting.
2023-03-21 17:55:01 - POLLER: Poller[1] PID[1713] STATS: Poller ID 2 fully Replicated
2023-03-21 17:55:00 - SYSTEM MAINT STATS: Time:0.02

All of the boost tables are empty.

Code: Select all

MariaDB [cacti]> show tables like '%boost%';
+-------------------------------------+
| Tables_in_cacti (%boost%)           |
+-------------------------------------+
| poller_output_boost                 |
| poller_output_boost_arch_1679432701 |
| poller_output_boost_arch_1679433001 |
| poller_output_boost_arch_1679433901 |
| poller_output_boost_arch_1679434970 |
| poller_output_boost_arch_1679435692 |
| poller_output_boost_arch_1679436299 |
| poller_output_boost_arch_1679437500 |
| poller_output_boost_arch_1679438700 |
| poller_output_boost_arch_1679438999 |
| poller_output_boost_arch_1679439300 |
| poller_output_boost_arch_1679439599 |
| poller_output_boost_arch_1679439900 |
| poller_output_boost_local_data_ids  |
| poller_output_boost_processes       |
+-------------------------------------+
15 rows in set (0.001 sec)

MariaDB [cacti]> describe poller_output_boost;
+---------------+------------------+------+-----+---------------------+-------+
| Field         | Type             | Null | Key | Default             | Extra |
+---------------+------------------+------+-----+---------------------+-------+
| local_data_id | int(10) unsigned | NO   | PRI | 0                   |       |
| rrd_name      | varchar(19)      | NO   | PRI |                     |       |
| time          | timestamp        | NO   | PRI | 0000-00-00 00:00:00 |       |
| output        | varchar(512)     | NO   |     | NULL                |       |
+---------------+------------------+------+-----+---------------------+-------+
4 rows in set (0.001 sec)

MariaDB [cacti]> select * from poller_output_boost;
Empty set (0.000 sec)

MariaDB [cacti]> select * from poller_output_boost_arch_1679439900;
Empty set (0.000 sec)

MariaDB [cacti]> select * from poller_output_boost_local_data_ids;
Empty set (0.000 sec)

MariaDB [cacti]> select * from poller_output_boost_processes;
Empty set (0.000 sec)

MariaDB [cacti]>

If I can supply any helpful or relevant details to aid in understanding what is going, please let me know. I would appreciate any guidance. If this is not a bug somewhere, then I am guessing I have just missed some mundane detail and perhaps just wiping everything I have and starting over would be best for me but if the OP and I are experiencing the same issue, maybe I can provide some useful additional info.

[TheWitness]

Can you pull the latest poller_boost.php and lib/boost.php from the 1.2.x branch. You should manually remove the archive files for the time being. Let us know if that fixes it. Also, don't enable boost unless until you have some devices added and graphs created. I know that's not an entirely good answer, but should get you back to where you are.