Cacti System Warning for Cron is out of sync

[gdsotirov]

I'm polling hosts every minute and I'm having such warnings generated frequently from time to time:

Code: Select all

WARNING: Cron is out of sync with the Poller Interval for poller id 1! The Poller Interval is 60 seconds, with a maximum of a 60 seconds, but 121 seconds have passed since the last poll!

So clearly a poll is skipped. Looking at the log, I could find the following:

Code: Select all

2020/06/25 01:39:01 - POLLER: Poller[1] WARNING: Cron is out of sync with the Poller Interval!  The Poller Interval is '60' seconds, with a maximum of a '60' second Cron, but 120.3 seconds have passed since the last poll!
2020/06/25 01:39:01 - MAILER INFO: Mail successfully sent via PHP Mail() Function from 'Administrator <cacti@host.net>', to 'Administrator <root@host.net>', cc '', and took 0.08 seconds, Subject 'Cacti System Warning'
2020/06/25 01:39:03 - POLLER: Poller[1] WARNING: Invalid Response(s), Errors[1] Device[51] Thread[1] DS[963]
2020/06/25 01:39:05 - POLLER: Poller[1] WARNING: Invalid Response(s), Errors[1] Device[54] Thread[1] DS[982]
2020/06/25 01:39:14 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.25.2.3.1.5.1'
2020/06/25 01:39:15 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.25.2.3.1.6.1'
2020/06/25 01:39:16 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.25.2.3.1.5.2'
2020/06/25 01:39:17 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.25.2.3.1.6.2'
2020/06/25 01:39:42 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.25.2.3.1.5.4'
2020/06/25 01:39:43 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.25.2.3.1.6.4'
2020/06/25 01:39:44 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.25.2.3.1.5.5'
2020/06/25 01:39:45 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.25.2.3.1.6.5'
2020/06/25 01:39:45 - POLLER: Poller[1] WARNING: Invalid Response(s), Errors[5] Device[32] Thread[1] DS[511, 512, 513, 516, 754]
2020/06/25 01:39:45 - SYSTEM STATS: Time:44.4740 Method:cmd.php Processes:6 Threads:0 Hosts:39 HostsPerProcess:7 DataSources:880 RRDsProcessed:569
2020/06/25 01:39:47 - SYSTEM THOLD STATS: Time:0.5052 Tholds:144 TotalDevices:39 DownDevices:4 NewDownDevices:0
2020/06/25 01:39:47 - SYSTEM MONITOR STATS: Time:0.06 Reboots:0 DownDevices:0 Notifications:0 Purges:0
2020/06/25 01:40:03 - POLLER: Poller[1] WARNING: Invalid Response(s), Errors[1] Device[51] Thread[1] DS[963]
2020/06/25 01:40:04 - POLLER: Poller[1] WARNING: Invalid Response(s), Errors[1] Device[54] Thread[1] DS[982]
2020/06/25 01:40:06 - CMDPHP Device[32] NOTICE: HOST EVENT: Device Returned FROM DOWN State:
2020/06/25 01:40:14 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.25.2.3.1.5.1'
2020/06/25 01:40:15 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.25.2.3.1.6.1'
2020/06/25 01:40:16 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.25.2.3.1.5.2'
2020/06/25 01:40:17 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.25.2.3.1.6.2'
2020/06/25 01:40:42 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.25.2.3.1.5.4'
2020/06/25 01:40:43 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.25.2.3.1.6.4'
2020/06/25 01:40:44 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.25.2.3.1.5.5'
2020/06/25 01:40:45 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.25.2.3.1.6.5'
2020/06/25 01:40:45 - POLLER: Poller[1] WARNING: Invalid Response(s), Errors[5] Device[32] Thread[1] DS[511, 512, 513, 516, 754]
2020/06/25 01:40:45 - SYSTEM STATS: Time:44.4579 Method:cmd.php Processes:6 Threads:0 Hosts:39 HostsPerProcess:7 DataSources:880 RRDsProcessed:569
2020/06/25 01:40:48 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.1.1.0'
2020/06/25 01:40:50 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.1.3.0'
2020/06/25 01:40:52 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.1.5.0'
2020/06/25 01:40:54 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.1.6.0'
2020/06/25 01:40:56 - CMDPHP WARNING: SNMP Error:'snmp2_get(): No response from 192.168.0.5', Device:'dvrv', OID:'.1.3.6.1.2.1.1.4.0'
2020/06/25 01:40:56 - MAILER INFO: Mail successfully sent via PHP Mail() Function from 'Administrator <cacti@host.net>', to 'root@host.net', cc '', and took 0.12 seconds, Subject 'Device Notice: 25.5-DVRViewer (dvrv) returned from DOWN state'
2020/06/25 01:40:56 - SYSTEM THOLD STATS: Time:10.5883 Tholds:145 TotalDevices:39 DownDevices:4 NewDownDevices:0
2020/06/25 01:40:57 - SYSTEM MONITOR STATS: Time:0.06 Reboots:0 DownDevices:0 Notifications:0 Purges:0
2020/06/25 01:42:02 - POLLER: Poller[1] WARNING: Cron is out of sync with the Poller Interval!  The Poller Interval is '60' seconds, with a maximum of a '60' second Cron, but 121 seconds have passed since the last poll!

Which means the poll is too slow (it's normally 6-8 seconds). The problem seems to be caused by a host, which is normally quite loaded with video decoding, so sometimes it fails to respond on time. This host is not that important. How could I solve this, so my other hosts are still polled every minute? Should I decrease "SNMP Timeout"? It's currently the default of 500 milliseconds and the description states "The maximum number of milliseconds Cacti will wait for an SNMP response (does not work with php-snmp support).", so I wonder whether this would have any effect.

Anyway, why is Cacti spending so much time with this host, which impacts the polling of other hosts?

P.S. I'm running latest Cacti 1.2.12 and as clear from the log I'm not using Spine.

[camerabob]

Perhaps try increasing the number of processes?

2020/06/25 01:39:45 - SYSTEM STATS: Time:44.4740 Method:cmd.php Processes:6 Threads:0 Hosts:39 HostsPerProcess:7 DataSources:880 RRDsProcessed:569

I have a virtual host with 4 cores. My setting is below:
2020/06/25 10:02:29 - SYSTEM STATS: Time:26.7852 Method:cmd.php Processes:24 Threads:0 Hosts:68 HostsPerProcess:3 DataSources:210 RRDsProcessed:208

Perhaps you may get better results.

[MrRat]

Use spine.

2020/06/26 09:23:12 - SYSTEM STATS: Time:10.0800 Method:spine Processes:2 Threads:40 Hosts:1799 HostsPerProcess:900 DataSources:50537

[gdsotirov]

Perhaps try increasing the number of processes?

In fact, I kept it at 6, because my old server was with just 2 cores, so empirically I've determined that polling in 6 processes provides best results in terms of performance. However, just two weeks ago I've migrated to a new machine with 4 cores and 8 threads, but I forgot to change this setting in Cacti. Anyway, now I'm polling in 16 threads. The polling time dropped with 1.5 to 2.0 seconds from regular, but I'll keep monitoring for system warnings in the following days and update the post if this was the solution.

[gdsotirov]

Use spine.

2020/06/26 09:23:12 - SYSTEM STATS: Time:10.0800 Method:spine Processes:2 Threads:40 Hosts:1799 HostsPerProcess:900 DataSources:50537

Impressive! I've tried to compile spine from source few years ago, but as far as I remember there ware some compilation issues and I had no time to deal with them. I have to try compiling it again when I find the time