Boost not updating

[mp3merlin]

Hi,

I'm currently testing the distributed pollers of Cacti 1 on a test infrastructure.

I have a Cacti 1.1.16 test install with about 10000 devices which has been working fine for about 3 weeks.
All polling is done from a single remote poller (in the same LAN).
I decided to add about 2000 devices using the add_device.php CLI script. After the successful add, boost isn't running anymore.
I can still run a poller_boost.php --force to update the RRDs.

In my cacti.log (on the main server) I have some messages like this :

2017-08-30 17:50:09 - SYSTEM STATS: Time:8.2158 Method:spine Processes:4 Threads:50 Hosts:0 HostsPerProcess:0 DataSources:0 RRDsProcessed:0
2017-08-30 17:53:10 - ERROR PHP ERROR: Maximum execution time of 299 seconds exceeded in file: /var/www/cacti/lib/database.php on line: 374
2017-08-30 17:53:10 - CMDPHP PHP ERROR Backtrace: (CactiShutdownHandler)(/lib/functions.php: 4432 cacti_debug_backtrace)
2017-08-30 17:55:04 - SYSTEM STATS: Time:3.1443 Method:spine Processes:4 Threads:50 Hosts:0 HostsPerProcess:0 DataSources:0 RRDsProcessed:0
2017-08-30 17:57:32 - ERROR PHP ERROR: Maximum execution time of 299 seconds exceeded in file: /var/www/cacti/lib/snmpagent.php on line: 0
2017-08-30 17:57:32 - CMDPHP PHP ERROR Backtrace: (CactiShutdownHandler)(/lib/functions.php: 4432 cacti_debug_backtrace)
2017-08-30 18:00:12 - SYSTEM STATS: Time:10.4571 Method:spine Processes:4 Threads:50 Hosts:0 HostsPerProcess:0 DataSources:0 RRDsProcessed:0
2017-08-30 18:03:18 - ERROR PHP ERROR: Maximum execution time of 299 seconds exceeded in file: /var/www/cacti/lib/snmpagent.php on line: 823
2017-08-30 18:03:18 - CMDPHP PHP ERROR Backtrace: (CactiShutdownHandler)(/lib/functions.php: 4432 cacti_debug_backtrace)
2017-08-30 18:05:09 - SYSTEM STATS: Time:7.7355 Method:spine Processes:4 Threads:50 Hosts:0 HostsPerProcess:0 DataSources:0 RRDsProcessed:0
2017-08-30 18:06:53 - ERROR PHP ERROR: Maximum execution time of 299 seconds exceeded in file: /var/www/cacti/lib/snmpagent.php on line: 449
2017-08-30 18:06:53 - CMDPHP PHP ERROR Backtrace: (CactiShutdownHandler)(/lib/functions.php: 4432 cacti_debug_backtrace)

No problems on the poller :

2017-08-30 18:16:51 - SYSTEM STATS: Time:107.8277 Method:spine Processes:4 Threads:50 Hosts:12387 HostsPerProcess:3097 DataSources:116298 RRDsProcessed:0

I activated the slow query log on my mariadb server :

# mysqldumpslow /var/log/mysql.slow.log

Reading mysql slow query log from /var/log/mysql.slow.log
Count: 294 Time=18.51s (5442s) Lock=0.00s (0s) Rows_sent=1.0 (294), Rows_examined=88606191.8 (26050220386), Rows_affected=0.0 (0), cactiuser[cactiuser]@localhost
SELECT count(*) FROM poller_output_boost_arch_1504509016

Count: 463 Time=5.85s (2709s) Lock=0.00s (0s) Rows_sent=1.0 (463), Rows_examined=27815985.8 (12878801408), Rows_affected=0.0 (0), cactiuser[cactiuser]@localhost
SELECT count(*) FROM poller_output_boost_arch_1504172552

Count: 958 Time=1.25s (1198s) Lock=0.00s (0s) Rows_sent=10.0 (9580), Rows_examined=1448706.5 (1387860838), Rows_affected=0.0 (0), cactiuser[cactiuser]@integsr-t241
SELECT DISTINCT field_name FROM host_snmp_cache AS hsc WHERE snmp_query_id= 'S'

Is there anything that could explain my problem ?

[Osiris]

Looks like one of the sidebar processes. Maybe snmp agent updates. Update to 1.1.21 and see if it makes a difference.

[mp3merlin]

Thanks, I'll try that.

[mp3merlin]

I still have the problem after an upgrade of both servers to 1.1.21

[Osiris]

Open a performance bug on github.com