Cacti not spawning enough poller processes

[paulrobertsjr]

We've got a medium-sized setup that was just upgraded from 1.x to 1.2.x via EPEL, and I am trying to determine if this is a bug or a failure of some kind related to the upgrade (the installation is fairly new so blowing it away and rebuilding from scratch will not end the world).

Since the upgrade, the Device Threads setting (under Configuration->Device Defaults) seems to have become the hard limit for every type of poller. The most copies of cmd.php that I can catch running are six. Switching over to Spine, again... six total. This despite the fact that I've configured Spine to max at 8, and cmd.php should be running to 64. The box itself has plenty of capacity for this (four virtual cores on a host with 16 available) and load doesn't even reach 1.

Because of this the machine struggles to check even the 100 hosts it's been told to track, because it takes longer than a minute to run all those tests when it'll only run a small number of them at a time. The majority of them don't even incur a host up check, because the bulk of the tests are simply collecting interface availability metrics, and that test takes a maximum of 8 seconds. After refactoring that down from 12 seconds, the run generally completes in about 50-55 seconds.

Any suggestions? I've seen quite a bit of verbiage in the transients over the configuration items about things being done differently with 1.2.x, and I suspect it might be an oversight that the interface will only allow you to set a maximum of six device threads.

[netniV]

These poller settings are now on the individual settings of the data collectors with the defaults being just for when new one are created.

[paulrobertsjr]

Yeah, I understand that, but despite what's under the other tabs (specifically, the Poller tab sporting a value of 64 for default data collector processes as well as 32 threads for Spine-specific settings) neither Spine nor cmd.php seem interested in spawning more than six instances at once. Either the database schema upgrade failed in a weird way, or both spine and cmd.php are looking at the wrong ceiling limit, because a few seconds after the top of the minute I see this with cmd.php:

Code: Select all

apache    2007     1  4 12:37 ?        00:00:00 /bin/php -q /usr/share/cacti/cmd.php --poller=1 --first=15 --last=532 --mibs
apache    2019     1  3 12:37 ?        00:00:00 /bin/php -q /usr/share/cacti/cmd.php --poller=1 --first=533 --last=549 --mibs
apache    2031     1  3 12:37 ?        00:00:00 /bin/php -q /usr/share/cacti/cmd.php --poller=1 --first=550 --last=566 --mibs
apache    2041     1  3 12:37 ?        00:00:00 /bin/php -q /usr/share/cacti/cmd.php --poller=1 --first=567 --last=583 --mibs
apache    2053     1  3 12:37 ?        00:00:00 /bin/php -q /usr/share/cacti/cmd.php --poller=1 --first=584 --last=600 --mibs
apache    2063     1  3 12:37 ?        00:00:00 /bin/php -q /usr/share/cacti/cmd.php --poller=1 --first=601 --last=616 --mibs

...and flipping the switch to using Spine, I see this:

Code: Select all

apache    3667     1  0 12:39 ?        00:00:00 /usr/bin/spine -C /etc/spine.conf --poller 1 --first 15 --last 532 --mibs
apache    3683     1  0 12:39 ?        00:00:00 /usr/bin/spine -C /etc/spine.conf --poller 1 --first 533 --last 549 --mibs
apache    3700     1  0 12:39 ?        00:00:00 /usr/bin/spine -C /etc/spine.conf --poller 1 --first 550 --last 566 --mibs
apache    3716     1  0 12:39 ?        00:00:00 /usr/bin/spine -C /etc/spine.conf --poller 1 --first 567 --last 583 --mibs
apache    3727     1  0 12:39 ?        00:00:00 /usr/bin/spine -C /etc/spine.conf --poller 1 --first 584 --last 600 --mibs
apache    3747     1  0 12:39 ?        00:00:00 /usr/bin/spine -C /etc/spine.conf --poller 1 --first 601 --last 616 --mibs

Load average (somewhat predictably) isn't even hitting single digits, and only after some refactoring do these checks (the majority of which are pretty simple ping -A -c 5 -w 8 hostname tests) all manage to complete in under a minute for per-minute polling. After the upgrade to 1.2.2-1.el7 only about 2/3 of the RRDs were actually seeing updates because the tests just didn't get run. Everything I'm looking at seems to indicate that the things are just doing a maximum of six tasks simultaneously. More baffling, the logs are showing different figures still...

2019-03-18 12:47:59 - SYSTEM STATS: Time:56.8681 Method:spine Processes:10 Threads:1 Hosts:100 HostsPerProcess:10 DataSources:177 RRDsProcessed:172

2019-03-18 12:28:57 - SYSTEM STATS: Time:55.4773 Method:cmd.php Processes:10 Threads:N/A Hosts:100 HostsPerProcess:10 DataSources:177 RRDsProcessed:172

At no point am I seeing ten simultaneous anythings, and as far as I know they should be reporting 32 threads, not 1, and a bit more than just ten processes. So, something ain't right... and it's not a problem to blow away the db, but I figured at the very least I should leave a trail of bread crumbs for anyone else who might run face-first into the same issue.

[ign]

10 is the number of processes you have configured in 'Data collection -> Data Collector'. If you select spine you can select the number of threads there too. Try 3 processes (low as recommended) and 10 threads. top -H will do to show the 30 threads

[netniV]

You don't have installs in different folders which are different versions? some peopel do that and forget to update their cron task.

[paulrobertsjr]

The installation was via packages, and the upgrade was also via packages, and both packages appear to have been sticking everything in the same place (and otherwise some custom scripts would have fallen right off).

Also, cmd.php ran enough processes to be useful before the upgrade. If something was using stale configuration files that would be one thing, but it would at least be using the previous version's settings.

[netniV]

Code: Select all

select id, name, processes, threads from poller;
select * from settings where name like '%threa%' or name like '%proces%';

[paulrobertsjr]

After wiping the thing and repopulating it (I also took a moment to spin up a brand new maria instance, so... zero chance of stale database information), there's probably something really knackered in the EPEL packages for 1.2.2, because I am pretty sure this output means exactly what it looks like it means.

Code: Select all

MariaDB [cacti]> select id, name, processes, threads from poller;
+----+-------------+-----------+---------+
| id | name        | processes | threads |
+----+-------------+-----------+---------+
|  1 | Main Poller |         1 |       1 |
+----+-------------+-----------+---------+
1 row in set (0.00 sec)

MariaDB [cacti]> select * from settings where name like '%threa%' or name like '%proces%';
+---------------------------------------------+-------------------------------+
| name                                        | value                         |
+---------------------------------------------+-------------------------------+
| concurrent_processes                        | 32                            |
| device_threads                              | 5                             |
| install_table_automation_processes          | automation_processes          |
| install_table_poller_output_boost_processes | poller_output_boost_processes |
| max_threads                                 | 32                            |
| process_leveling                            | on                            |
+---------------------------------------------+-------------------------------+
6 rows in set (0.00 sec)

[paulrobertsjr]

...and after poking higher numbers under threads/processes, almost like magic, things run more than just one at a time.

This doesn't appear to be a problem that's limited to the EPEL packages, either. Nothing in the web GUI appears to affect these settings on an entirely separate instance I run at home, on Slackware, where everything was built up from source tarballs.

[netniV]

The default values (in the settings table) will not affect existing collectors. Only those in the pollers table affect that. I updated my test poller from 1 / 1 to 2 / 3, spine kicked in 3 threads so the functionality is working.

[paulrobertsjr]

It seems like there's somewhat of a glaring interface problem here, since most people doing a fresh install are going to expect the settings for the pollers to be under Configuration->Settings->Poller, or at least for those to at least have some impact on operation. Perhaps it's time to consider using a negative value or some other obvious canary to make a poller grab the defaults instead of staying set to the 1/1 they inherited when they were instantiated. Especially considering that the majority of Cacti installations are probably still using a single poller, having its configuration in an entirely separate section of the interface is likely to be a source of continued confusion.

[netniV]

if you look at the help for the settings configuration, it clearly states that they are for default values only. When upgrading existing systems, the defaults are transferred to the poller table for any existing pollers.