I have cacti-1.2.19 on FreeBSD 13.0-RELEASE amd64, which collects mainly interface statistics by means of "In/Out Bits (64-bit Counters)" from ~70 devices, mainly OpenBSD firewalls and Juniper switches.
Devices with most graphs are two OpenBSD firewalls where ~20 GRE tunnels per firewall are being graphed, and new GRE tunnel is being added or removed now and then.
Yesterday I added new GRE tunnel on each of two firewalls, and created Interface Statistics - In/Out Bits (64-bit Counters) graphs for them in Cacti. This resulted in traffic not being graphed anymore for anything on those two firewalls. Other graphs for other devices are being graphed fine.
I tried to reindex device and to repopulate poller cache on device page but the problem persists.
Any Ideas what can I do to get graphs for the two devices working again?
Relevant lines from log:
Code: Select all
2022/01/27 10:14:49 - RECACHE STATS: Poller:1 RecacheTime:0.5360 DevicesRecached:2
2022/01/27 10:14:49 - PCOMMAND Device[OpenBSD Firewall NAT2] NOTE: Recache Event Detected for Device
2022/01/27 10:14:48 - PCOMMAND Device[OpenBSD Firewall NAT1] NOTE: Recache Event Detected for Device
2022/01/27 10:14:48 - SYSTEM STATS: Time:18.5147 Method:spine Processes:1 Threads:1 Hosts:70 HostsPerProcess:70 DataSources:1599 RRDsProcessed:798
2022/01/27 10:14:48 - SPINE: Poller[Main Poller] PID[6184] Device[OpenBSD Firewall NAT2] HT[1] DQ[SNMP - Interface Statistics] RECACHE ASSERT FAILED: '164327469400<13974423'
2022/01/27 10:14:48 - SPINE: Poller[Main Poller] PID[6184] Device[OpenBSD Firewall NAT1] HT[1] DQ[SNMP - Interface Statistics] RECACHE ASSERT FAILED: '164327469400<258530094'
2022/01/27 10:14:23 - RECACHE STATS: Poller:1 RecacheTime:0.5447 DevicesRecached:2
2022/01/27 10:14:23 - PCOMMAND Device[OpenBSD Firewall NAT2] NOTE: Recache Event Detected for Device
2022/01/27 10:14:22 - PCOMMAND Device[OpenBSD Firewall NAT1] NOTE: Recache Event Detected for Device
2022/01/27 10:14:22 - SYSTEM STATS: Time:22.5243 Method:spine Processes:1 Threads:1 Hosts:70 HostsPerProcess:70 DataSources:1599 RRDsProcessed:798
2022/01/27 10:14:21 - SPINE: Poller[Main Poller] PID[6148] Device[OpenBSD Firewall NAT2] HT[1] DQ[SNMP - Interface Statistics] RECACHE ASSERT FAILED: '164327466300<13971764'
2022/01/27 10:14:21 - SPINE: Poller[Main Poller] PID[6148] Device[OpenBSD Firewall NAT1] HT[1] DQ[SNMP - Interface Statistics] RECACHE ASSERT FAILED: '164327466200<258527434'