Permissions should be fine, been there, also changed user www-data shell to bash to simulate creation of rrds properly.
Host in question is name kuroo anf its local server, the increased rrds are from the other host which is sometimes turned on or off, therefore data sources number couldb e larger. This second host suffers same situation.
1. cacti.log looks perfect, poller is running fine:
Code: Select all
2022-06-30 02:04:03 - SYSTEM INTROPAGE STATS: Time:0.35, Checks:15, Details:15, Trends:10
2022-06-30 02:04:02 - SYSTEM SYSLOG STATS: Time:0.05 Deletes:0 Incoming:2 Removes:2 XFers:0 Alerts:3 Alarms:0 Reports:0
2022-06-30 02:04:02 - SYSTEM THOLD STATS: Time:0.0117 Tholds:0 TotalDevices:2 DownDevices:0 NewDownDevices:0
2022-06-30 02:04:02 - SYSTEM STATS: Time:1.2588 Method:spine Processes:8 Threads:100 Hosts:2 HostsPerProcess:1 DataSources:668 RRDsProcessed:0
2022-06-30 02:03:02 - SYSTEM INTROPAGE STATS: Time:0.23, Checks:0, Details:0, Trends:10
2022-06-30 02:03:02 - SYSTEM SYSLOG STATS: Time:0.05 Deletes:0 Incoming:0 Removes:0 XFers:0 Alerts:3 Alarms:0 Reports:0
2022-06-30 02:03:02 - SYSTEM THOLD STATS: Time:0.0056 Tholds:0 TotalDevices:2 DownDevices:0 NewDownDevices:0
2022-06-30 02:03:02 - SYSTEM STATS: Time:1.2614 Method:spine Processes:8 Threads:100 Hosts:2 HostsPerProcess:1 DataSources:668 RRDsProcessed:0
2022-06-30 02:02:02 - SYSTEM INTROPAGE STATS: Time:0.25, Checks:0, Details:0, Trends:10
2022-06-30 02:02:02 - SYSTEM SYSLOG STATS: Time:0.06 Deletes:0 Incoming:0 Removes:0 XFers:0 Alerts:3 Alarms:0 Reports:0
2022-06-30 02:02:02 - SYSTEM THOLD STATS: Time:0.0060 Tholds:0 TotalDevices:2 DownDevices:0 NewDownDevices:0
2022-06-30 02:02:02 - SYSTEM STATS: Time:1.2649 Method:spine Processes:8 Threads:100 Hosts:2 HostsPerProcess:1 DataSources:668 RRDsProcessed:0
2022-06-30 02:01:02 - SYSTEM INTROPAGE STATS: Time:0.22, Checks:0, Details:0, Trends:10
2022-06-30 02:01:02 - SYSTEM SYSLOG STATS: Time:0.05 Deletes:0 Incoming:0 Removes:0 XFers:0 Alerts:3 Alarms:0 Reports:0
2022-06-30 02:01:02 - SYSTEM THOLD STATS: Time:0.0084 Tholds:0 TotalDevices:2 DownDevices:0 NewDownDevices:0
2022-06-30 02:01:02 - SYSTEM STATS: Time:1.2579 Method:spine Processes:8 Threads:100 Hosts:2 HostsPerProcess:1 DataSources:668 RRDsProcessed:0
2022-06-30 02:00:03 - SYSTEM INTROPAGE STATS: Time:0.28, Checks:2, Details:2, Trends:10
2022-06-30 02:00:02 - SYSTEM SYSLOG STATS: Time:0.05 Deletes:0 Incoming:6 Removes:6 XFers:0 Alerts:3 Alarms:0 Reports:0
2022-06-30 02:00:02 - SYSTEM THOLD STATS: Time:0.0077 Tholds:0 TotalDevices:2 DownDevices:0 NewDownDevices:0
2022-06-30 02:00:02 - SYSTEM STATS: Time:1.2850 Method:spine Processes:8 Threads:100 Hosts:2 HostsPerProcess:1 DataSources:668 RRDsProcessed:0
2022-06-30 02:00:01 - POLLER: Poller[Main Poller] PID[846248] WARNING: Data Template 'AAA Live Internet Devices' is impacted by lack of complete information
2022-06-30 02:00:01 - POLLER: Poller[Main Poller] PID[846248] WARNING: There are 18 Data Sources not returning all data leaving rows in the poller output table. Details to follow.
2022-06-30 02:00:01 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2022-06-30 02:00:01 - POLLER: Poller[Main Poller] PID[846248] WARNING: Poller Output Table not Empty. Issues: 24, ...................some ips from disabled device here................, Additional Issues Remain. Only showing first 20
2022-06-30 01:59:05 - SYSTEM BOOST STATS: Time:2.22 RRDUpdates:4920
2022-06-30 01:59:03 - SYSTEM BOOST STATS: Time:0.17 ProcessNumber:5 RRDUpdates:903
2022-06-30 01:59:03 - SYSTEM BOOST STATS: Time:0.18 ProcessNumber:3 RRDUpdates:983
2022-06-30 01:59:03 - SYSTEM BOOST STATS: Time:0.20 ProcessNumber:2 RRDUpdates:959
2022-06-30 01:59:03 - SYSTEM BOOST STATS: Time:0.19 ProcessNumber:1 RRDUpdates:1113
2022-06-30 01:59:03 - SYSTEM BOOST STATS: Time:0.17 ProcessNumber:4 RRDUpdates:962
2022-06-30 01:59:03 - SYSTEM INTROPAGE STATS: Time:0.43, Checks:2, Details:2, Trends:10
2022-06-30 01:59:03 - SYSTEM SYSLOG STATS: Time:0.07 Deletes:0 Incoming:2 Removes:2 XFers:0 Alerts:3 Alarms:0 Reports:0
2022-06-30 01:59:03 - SYSTEM THOLD STATS: Time:0.0096 Tholds:0 TotalDevices:2 DownDevices:0 NewDownDevices:0
2022-06-30 01:59:02 - SYSTEM STATS: Time:1.2608 Method:spine Processes:8 Threads:100 Hosts:2 HostsPerProcess:1 DataSources:668 RRDsProcessed:0
2022-06-30 01:58:03 - SYSTEM INTROPAGE STATS: Time:0.72, Checks:16, Details:16, Trends:10
2022-06-30 01:58:03 - SYSTEM SYSLOG STATS: Time:0.04 Deletes:0 Incoming:0 Removes:0 XFers:0 Alerts:3 Alarms:0 Reports:0
2022-06-30 01:58:03 - SYSTEM THOLD STATS: Time:0.0070 Tholds:0 TotalDevices:2 DownDevices:0 NewDownDevices:0
2022-06-30 01:58:02 - SYSTEM STATS: Time:1.2724 Method:spine Processes:8 Threads:100 Hosts:2 HostsPerProcess:1 DataSources:668 RRDsProcessed:0
2022-06-30 01:57:18 - AUTH LOGIN: User 'js' authenticated
2022-06-30 01:57:03 - SYSTEM INTROPAGE STATS: Time:0.33, Checks:2, Details:2, Trends:10
2022-06-30 01:57:03 - SYSTEM SYSLOG STATS: Time:0.05 Deletes:0 Incoming:0 Removes:0 XFers:0 Alerts:3 Alarms:0 Reports:0
2022-06-30 01:57:03 - SYSTEM THOLD STATS: Time:0.0085 Tholds:0 TotalDevices:2 DownDevices:0 NewDownDevices:0
2022-06-30 01:57:02 - SYSTEM STATS: Time:1.2607 Method:spine Processes:8 Threads:100 Hosts:2 HostsPerProcess:1 DataSources:668 RRDsProcessed:0
2022-06-30 01:56:03 - SYSTEM INTROPAGE STATS: Time:0.22, Checks:0, Details:0, Trends:10
2022-06-30 01:56:03 - SYSTEM SYSLOG STATS: Time:0.05 Deletes:0 Incoming:0 Removes:0 XFers:0 Alerts:3 Alarms:0 Reports:0
2022-06-30 01:56:02 - SYSTEM THOLD STATS: Time:0.0060 Tholds:0 TotalDevices:1 DownDevices:0 NewDownDevices:0
2022-06-30 01:56:02 - SYSTEM STATS: Time:1.1544 Method:spine Processes:8 Threads:100 Hosts:1 HostsPerProcess:1 DataSources:668 RRDsProcessed:0
2022-06-30 01:55:03 - SYSTEM INTROPAGE STATS: Time:0.2, Checks:0, Details:0, Trends:10
2022-06-30 01:55:03 - SYSTEM SYSLOG STATS: Time:0.05 Deletes:0 Incoming:6 Removes:6 XFers:0 Alerts:3 Alarms:0 Reports:0
2022-06-30 01:55:02 - SYSTEM THOLD STATS: Time:0.0071 Tholds:0 TotalDevices:1 DownDevices:0 NewDownDevices:0
2022-06-30 01:55:02 - SYSTEM STATS: Time:1.1680 Method:spine Processes:8 Threads:100 Hosts:1 HostsPerProcess:1 DataSources:668 RRDsProcessed:0
2. Yes, boost is on. I could try stopping it, acually I tried, but not very systematically.
3. hmmm didn't know that.
So let me simulate one more time - removing all logs, removing device graphs, running automate rules on device gives me following debug log:
Code: Select all
2022-06-30 02:11:03 - SYSTEM INTROPAGE STATS: Time:0.21, Checks:0, Details:0, Trends:10
2022-06-30 02:11:03 - SYSTEM SYSLOG STATS: Time:0.05 Deletes:0 Incoming:0 Removes:0 XFers:0 Alerts:3 Alarms:0 Reports:0
2022-06-30 02:11:03 - SYSTEM THOLD STATS: Time:0.0062 Tholds:0 TotalDevices:1 DownDevices:0 NewDownDevices:0
2022-06-30 02:11:03 - SYSTEM STATS: Time:2.1266 Method:spine Processes:8 Threads:100 Hosts:1 HostsPerProcess:1 DataSources:670 RRDsProcessed:0
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075681421056] DEBUG: Device[kuroo] HT[7] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075681421056] Device[kuroo] HT[7] Total Time: 0.03 Seconds
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075689813760] DEBUG: Device[kuroo] HT[6] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075689813760] Device[kuroo] HT[6] Total Time: 0.031 Seconds
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075681421056] Device[kuroo] HT[7] DS[kuroo - Load Average - 5 Minute] Graphs[kuroo - Load Average] TT[0.73] SNMP: v2: 192.168.1.23, dsname: load_5min, oid: .1.3.6.1.4.1.2021.10.1.3.2, value: 1.07
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075681421056] Device[kuroo] HT[7] DS[kuroo - CPU - Idle] Graphs[kuroo - CPU Utilization] TT[0.71] SNMP: v2: 192.168.1.23, dsname: ssCpuIdle, oid: .1.3.6.1.4.1.2021.11.11.0, value: 90
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075681421056] Device[kuroo] HT[7] DS[kuroo - CPU - System] Graphs[kuroo - CPU Utilization] TT[0.67] SNMP: v2: 192.168.1.23, dsname: ssCpuSystem, oid: .1.3.6.1.4.1.2021.11.10.0, value: 2
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075681421056] Device[kuroo] HT[7] DS[kuroo - CPU - User] Graphs[kuroo - CPU Utilization] TT[0.60] SNMP: v2: 192.168.1.23, dsname: ssCpuUser, oid: .1.3.6.1.4.1.2021.11.9.0, value: 6
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075689813760] Device[kuroo] HT[6] DS[kuroo - Load Average - 5 Minute] Graphs[kuroo - Load Average] TT[0.45] SNMP: v2: 192.168.1.23, dsname: load_5min, oid: .1.3.6.1.4.1.2021.10.1.3.2, value: 1.07
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075689813760] Device[kuroo] HT[6] DS[kuroo - CPU - Idle] Graphs[kuroo - CPU Utilization] TT[0.41] SNMP: v2: 192.168.1.23, dsname: ssCpuIdle, oid: .1.3.6.1.4.1.2021.11.11.0, value: 90
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075689813760] Device[kuroo] HT[6] DS[kuroo - CPU - System] Graphs[kuroo - CPU Utilization] TT[0.38] SNMP: v2: 192.168.1.23, dsname: ssCpuSystem, oid: .1.3.6.1.4.1.2021.11.10.0, value: 2
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075689813760] Device[kuroo] HT[6] DS[kuroo - CPU - User] Graphs[kuroo - CPU Utilization] TT[0.36] SNMP: v2: 192.168.1.23, dsname: ssCpuUser, oid: .1.3.6.1.4.1.2021.11.9.0, value: 6
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075190974208] DEBUG: Device[kuroo] HT[9] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075190974208] Device[kuroo] HT[9] Total Time: 0.024 Seconds
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075689813760] Device[kuroo] HT[6] NOTE: There are '4' Polling Items for this Device
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075681421056] Device[kuroo] HT[7] NOTE: There are '4' Polling Items for this Device
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075673028352] DEBUG: Device[kuroo] HT[8] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075673028352] Device[kuroo] HT[8] Total Time: 0.027 Seconds
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075698206464] DEBUG: Device[kuroo] HT[5] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075190974208] Device[kuroo] HT[9] DQ[AVS - SNMP - Interface Statistics] RECACHE OID: .1.3.6.1.2.1.2.1.0, (assert: 8 = output: 8)
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075723384576] DEBUG: Device[kuroo] HT[1] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075698206464] Device[kuroo] HT[5] Total Time: 0.028 Seconds
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075723384576] Device[kuroo] HT[1] Total Time: 0.034 Seconds
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075673028352] Device[kuroo] HT[8] DS[kuroo - Load Average - 15 Minute] Graphs[kuroo - Load Average] TT[0.89] SNMP: v2: 192.168.1.23, dsname: load_15min, oid: .1.3.6.1.4.1.2021.10.1.3.3, value: 1.13
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075673028352] Device[kuroo] HT[8] DS[kuroo - Load Average - 5 Minute] Graphs[kuroo - Load Average] TT[0.83] SNMP: v2: 192.168.1.23, dsname: load_5min, oid: .1.3.6.1.4.1.2021.10.1.3.2, value: 1.07
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075673028352] Device[kuroo] HT[8] DS[kuroo - CPU - Idle] Graphs[kuroo - CPU Utilization] TT[0.78] SNMP: v2: 192.168.1.23, dsname: ssCpuIdle, oid: .1.3.6.1.4.1.2021.11.11.0, value: 90
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075673028352] Device[kuroo] HT[8] DS[kuroo - CPU - System] Graphs[kuroo - CPU Utilization] TT[0.74] SNMP: v2: 192.168.1.23, dsname: ssCpuSystem, oid: .1.3.6.1.4.1.2021.11.10.0, value: 2
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075190974208] Device[kuroo] HT[9] DQ[Net-SNMP - Get Device I/O] RECACHE OID COUNT: .1.3.6.1.4.1.2021.13.15.1.1.1, output: 28
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075681421056] Device[kuroo] HT[7] DQ[AVS - SNMP - Interface Statistics] RECACHE OID: .1.3.6.1.2.1.2.1.0, (assert: 8 = output: 8)
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075048363776] DEBUG: Device[kuroo] HT[10] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075048363776] Device[kuroo] HT[10] Total Time: 0.022 Seconds
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075689813760] Device[kuroo] HT[6] DQ[AVS - SNMP - Interface Statistics] RECACHE OID: .1.3.6.1.2.1.2.1.0, (assert: 8 = output: 8)
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075698206464] Device[kuroo] HT[5] DS[kuroo - Load Average - 5 Minute] Graphs[kuroo - Load Average] TT[0.74] SNMP: v2: 192.168.1.23, dsname: load_5min, oid: .1.3.6.1.4.1.2021.10.1.3.2, value: 1.07
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075698206464] Device[kuroo] HT[5] DS[kuroo - CPU - Idle] Graphs[kuroo - CPU Utilization] TT[0.72] SNMP: v2: 192.168.1.23, dsname: ssCpuIdle, oid: .1.3.6.1.4.1.2021.11.11.0, value: 90
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075698206464] Device[kuroo] HT[5] DS[kuroo - CPU - System] Graphs[kuroo - CPU Utilization] TT[0.70] SNMP: v2: 192.168.1.23, dsname: ssCpuSystem, oid: .1.3.6.1.4.1.2021.11.10.0, value: 2
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075698206464] Device[kuroo] HT[5] DS[kuroo - CPU - User] Graphs[kuroo - CPU Utilization] TT[0.67] SNMP: v2: 192.168.1.23, dsname: ssCpuUser, oid: .1.3.6.1.4.1.2021.11.9.0, value: 6
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075681421056] Device[kuroo] HT[7] DQ[Net-SNMP - Get Device I/O] RECACHE OID COUNT: .1.3.6.1.4.1.2021.13.15.1.1.1, output: 28
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075673028352] Device[kuroo] HT[8] NOTE: There are '4' Polling Items for this Device
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075689813760] Device[kuroo] HT[6] DQ[Net-SNMP - Get Device I/O] RECACHE OID COUNT: .1.3.6.1.4.1.2021.13.15.1.1.1, output: 28
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075723384576] Device[kuroo] HT[1] DS[kuroo - CPU - Idle] Graphs[kuroo - CPU Utilization] TT[4.74] SNMP: v2: 192.168.1.23, dsname: ssCpuIdle, oid: .1.3.6.1.4.1.2021.11.11.0, value: 90
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075723384576] Device[kuroo] HT[1] DS[kuroo - Load Average - 5 Minute] Graphs[kuroo - Load Average] TT[4.71] SNMP: v2: 192.168.1.23, dsname: load_5min, oid: .1.3.6.1.4.1.2021.10.1.3.2, value: 1.07
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075723384576] Device[kuroo] HT[1] DS[kuroo - Load Average - 15 Minute] Graphs[kuroo - Load Average] TT[4.67] SNMP: v2: 192.168.1.23, dsname: load_15min, oid: .1.3.6.1.4.1.2021.10.1.3.3, value: 1.13
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075698206464] Device[kuroo] HT[5] NOTE: There are '4' Polling Items for this Device
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075723384576] Device[kuroo] HT[1] DS[kuroo - Load Average - 1 Minute] Graphs[kuroo - Load Average] TT[4.61] SNMP: v2: 192.168.1.23, dsname: load_1min, oid: .1.3.6.1.4.1.2021.10.1.3.1, value: 1.04
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075673028352] Device[kuroo] HT[8] DQ[AVS - SNMP - Interface Statistics] RECACHE OID: .1.3.6.1.2.1.2.1.0, (assert: 8 = output: 8)
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075698206464] Device[kuroo] HT[5] DQ[AVS - SNMP - Interface Statistics] RECACHE OID: .1.3.6.1.2.1.2.1.0, (assert: 8 = output: 8)
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075673028352] Device[kuroo] HT[8] DQ[Net-SNMP - Get Device I/O] RECACHE OID COUNT: .1.3.6.1.4.1.2021.13.15.1.1.1, output: 28
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075698206464] Device[kuroo] HT[5] DQ[Net-SNMP - Get Device I/O] RECACHE OID COUNT: .1.3.6.1.4.1.2021.13.15.1.1.1, output: 28
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075048363776] Device[kuroo] HT[10] DQ[AVS - SNMP - Interface Statistics] RECACHE OID: .1.3.6.1.2.1.2.1.0, (assert: 8 = output: 8)
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075048363776] Device[kuroo] HT[10] DQ[Net-SNMP - Get Device I/O] RECACHE OID COUNT: .1.3.6.1.4.1.2021.13.15.1.1.1, output: 28
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075706599168] DEBUG: Device[kuroo] HT[4] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075706599168] Device[kuroo] HT[4] Total Time: 0.028 Seconds
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075706599168] Device[kuroo] HT[4] DS[kuroo - Load Average - 5 Minute] Graphs[kuroo - Load Average] TT[0.62] SNMP: v2: 192.168.1.23, dsname: load_5min, oid: .1.3.6.1.4.1.2021.10.1.3.2, value: 1.07
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075706599168] Device[kuroo] HT[4] DS[kuroo - CPU - Idle] Graphs[kuroo - CPU Utilization] TT[0.59] SNMP: v2: 192.168.1.23, dsname: ssCpuIdle, oid: .1.3.6.1.4.1.2021.11.11.0, value: 90
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075706599168] Device[kuroo] HT[4] DS[kuroo - CPU - System] Graphs[kuroo - CPU Utilization] TT[0.53] SNMP: v2: 192.168.1.23, dsname: ssCpuSystem, oid: .1.3.6.1.4.1.2021.11.10.0, value: 2
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075706599168] Device[kuroo] HT[4] DS[kuroo - CPU - User] Graphs[kuroo - CPU Utilization] TT[0.48] SNMP: v2: 192.168.1.23, dsname: ssCpuUser, oid: .1.3.6.1.4.1.2021.11.9.0, value: 6
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075714991872] DEBUG: Device[kuroo] HT[3] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075714991872] Device[kuroo] HT[3] Total Time: 0.028 Seconds
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075706599168] Device[kuroo] HT[4] NOTE: There are '4' Polling Items for this Device
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075731777280] DEBUG: Device[kuroo] HT[2] DEBUG: HOST COMPLETE: About to Exit Device Polling Thread Function
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075706599168] Device[kuroo] HT[4] DQ[AVS - SNMP - Interface Statistics] RECACHE OID: .1.3.6.1.2.1.2.1.0, (assert: 8 = output: 8)
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075731777280] Device[kuroo] HT[2] Total Time: 0.028 Seconds
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075723384576] Device[kuroo] HT[1] NOTE: There are '4' Polling Items for this Device
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075706599168] Device[kuroo] HT[4] DQ[Net-SNMP - Get Device I/O] RECACHE OID COUNT: .1.3.6.1.4.1.2021.13.15.1.1.1, output: 28
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075731777280] Device[kuroo] HT[2] DS[kuroo - Device I/O -nvme0n1p1] Graphs[kuroo - Device I/O - Reads/Writes nvme0n1p1] TT[1.37] SNMP: v2: 192.168.1.23, dsname: Device_Writes, oid: .1.3.6.1.4.1.2021.13.15.1.1.6.10, value: 2
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075714991872] Device[kuroo] HT[3] DS[kuroo - Device I/O -nvme0n1p3] Graphs[kuroo - Device I/O - Reads/Writes nvme0n1p3] TT[1.35] SNMP: v2: 192.168.1.23, dsname: Device_Writes, oid: .1.3.6.1.4.1.2021.13.15.1.1.6.12, value: 100843
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075714991872] Device[kuroo] HT[3] DS[kuroo - Device I/O -nvme0n1p3] Graphs[kuroo - Device I/O - Reads/Writes nvme0n1p3] TT[1.32] SNMP: v2: 192.168.1.23, dsname: Device_Reads, oid: .1.3.6.1.4.1.2021.13.15.1.1.5.12, value: 88008
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075731777280] Device[kuroo] HT[2] DS[kuroo - Device I/O -nvme0n1p1] Graphs[kuroo - Device I/O - Reads/Writes nvme0n1p1] TT[1.32] SNMP: v2: 192.168.1.23, dsname: Device_Reads, oid: .1.3.6.1.4.1.2021.13.15.1.1.5.10, value: 1355
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075714991872] Device[kuroo] HT[3] DS[kuroo - Device I/O -nvme0n1p2] Graphs[kuroo - Device I/O - Reads/Writes nvme0n1p2] TT[1.28] SNMP: v2: 192.168.1.23, dsname: Device_Writes, oid: .1.3.6.1.4.1.2021.13.15.1.1.6.11, value: 954040
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075731777280] Device[kuroo] HT[2] DS[kuroo - CPU - User] Graphs[kuroo - CPU Utilization] TT[1.28] SNMP: v2: 192.168.1.23, dsname: ssCpuUser, oid: .1.3.6.1.4.1.2021.11.9.0, value: 6
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075714991872] Device[kuroo] HT[3] DS[kuroo - Device I/O -nvme0n1p2] Graphs[kuroo - Device I/O - Reads/Writes nvme0n1p2] TT[1.22] SNMP: v2: 192.168.1.23, dsname: Device_Reads, oid: .1.3.6.1.4.1.2021.13.15.1.1.5.11, value: 978634
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075731777280] Device[kuroo] HT[2] DS[kuroo - CPU - System] Graphs[kuroo - CPU Utilization] TT[1.18] SNMP: v2: 192.168.1.23, dsname: ssCpuSystem, oid: .1.3.6.1.4.1.2021.11.10.0, value: 2
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075723384576] Device[kuroo] HT[1] DQ[AVS - SNMP - Interface Statistics] RECACHE OID: .1.3.6.1.2.1.2.1.0, (assert: 8 = output: 8)
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075714991872] Device[kuroo] HT[3] NOTE: There are '4' Polling Items for this Device
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075731777280] Device[kuroo] HT[2] NOTE: There are '4' Polling Items for this Device
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075723384576] Device[kuroo] HT[1] DQ[Net-SNMP - Get Device I/O] RECACHE OID COUNT: .1.3.6.1.4.1.2021.13.15.1.1.1, output: 28
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075731777280] Device[kuroo] HT[2] DQ[AVS - SNMP - Interface Statistics] RECACHE OID: .1.3.6.1.2.1.2.1.0, (assert: 8 = output: 8)
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075714991872] Device[kuroo] HT[3] DQ[AVS - SNMP - Interface Statistics] RECACHE OID: .1.3.6.1.2.1.2.1.0, (assert: 8 = output: 8)
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075731777280] Device[kuroo] HT[2] DQ[Net-SNMP - Get Device I/O] RECACHE OID COUNT: .1.3.6.1.4.1.2021.13.15.1.1.1, output: 28
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075714991872] Device[kuroo] HT[3] DQ[Net-SNMP - Get Device I/O] RECACHE OID COUNT: .1.3.6.1.4.1.2021.13.15.1.1.1, output: 28
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075673028352] DEBUG: walk starts at OID .1.3.6.1.4.1.2021.13.15.1.1.1
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075673028352] DEBUG: Device[kuroo] HT[8] RECACHE: Processing 2 items in the auto reindex cache for '192.168.1.23'
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075698206464] DEBUG: walk starts at OID .1.3.6.1.4.1.2021.13.15.1.1.1
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075698206464] DEBUG: Device[kuroo] HT[5] RECACHE: Processing 2 items in the auto reindex cache for '192.168.1.23'
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075048363776] DEBUG: walk starts at OID .1.3.6.1.4.1.2021.13.15.1.1.1
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075048363776] DEBUG: Device[kuroo] HT[10] RECACHE: Processing 2 items in the auto reindex cache for '192.168.1.23'
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075190974208] DEBUG: walk starts at OID .1.3.6.1.4.1.2021.13.15.1.1.1
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075190974208] DEBUG: Device[kuroo] HT[9] RECACHE: Processing 2 items in the auto reindex cache for '192.168.1.23'
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075048363776] Device[kuroo] DEBUG: Entering SNMP Ping
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075048363776] DEBUG: Device[kuroo] HT[10] In Poller, About to Start Polling
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075190974208] Device[kuroo] DEBUG: Entering SNMP Ping
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075673028352] Device[kuroo] DEBUG: Entering SNMP Ping
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075698206464] Device[kuroo] DEBUG: Entering SNMP Ping
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075681421056] DEBUG: walk starts at OID .1.3.6.1.4.1.2021.13.15.1.1.1
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075681421056] DEBUG: Device[kuroo] HT[7] RECACHE: Processing 2 items in the auto reindex cache for '192.168.1.23'
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075190974208] DEBUG: Device[kuroo] HT[9] In Poller, About to Start Polling
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075689813760] DEBUG: walk starts at OID .1.3.6.1.4.1.2021.13.15.1.1.1
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075689813760] DEBUG: Device[kuroo] HT[6] RECACHE: Processing 2 items in the auto reindex cache for '192.168.1.23'
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075681421056] Device[kuroo] DEBUG: Entering SNMP Ping
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075723384576] DEBUG: walk starts at OID .1.3.6.1.4.1.2021.13.15.1.1.1
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075723384576] DEBUG: Device[kuroo] HT[1] RECACHE: Processing 2 items in the auto reindex cache for '192.168.1.23'
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075673028352] DEBUG: Device[kuroo] HT[8] In Poller, About to Start Polling
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075698206464] DEBUG: Device[kuroo] HT[5] In Poller, About to Start Polling
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075689813760] Device[kuroo] DEBUG: Entering SNMP Ping
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075681421056] DEBUG: Device[kuroo] HT[7] In Poller, About to Start Polling
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075706599168] DEBUG: walk starts at OID .1.3.6.1.4.1.2021.13.15.1.1.1
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075706599168] DEBUG: Device[kuroo] HT[4] RECACHE: Processing 2 items in the auto reindex cache for '192.168.1.23'
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075706599168] Device[kuroo] DEBUG: Entering SNMP Ping
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075689813760] DEBUG: Device[kuroo] HT[6] In Poller, About to Start Polling
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075714991872] DEBUG: walk starts at OID .1.3.6.1.4.1.2021.13.15.1.1.1
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075714991872] DEBUG: Device[kuroo] HT[3] RECACHE: Processing 2 items in the auto reindex cache for '192.168.1.23'
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075714991872] Device[kuroo] DEBUG: Entering SNMP Ping
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075706599168] DEBUG: Device[kuroo] HT[4] In Poller, About to Start Polling
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075731777280] DEBUG: walk starts at OID .1.3.6.1.4.1.2021.13.15.1.1.1
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075731777280] DEBUG: Device[kuroo] HT[2] RECACHE: Processing 2 items in the auto reindex cache for '192.168.1.23'
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075723384576] Device[kuroo] Updating Full System Information Table
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075714991872] DEBUG: Device[kuroo] HT[3] In Poller, About to Start Polling
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075731777280] Device[kuroo] DEBUG: Entering SNMP Ping
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075723384576] Device[kuroo] DEBUG: Entering SNMP Ping
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075731777280] DEBUG: Device[kuroo] HT[2] In Poller, About to Start Polling
2022-06-30 02:11:01 - SPINE: Poller[Main Poller] PID[849329] PT[140075723384576] DEBUG: Device[kuroo] HT[1] In Poller, About to Start Polling
2022-06-30 02:10:47 - AUTOM8 NOTE: Graph Added - Device[kuroo], Graph[kuroo - Traffic - br0 - 192.168.1.23 ], DS[kuroo - Traffic - 192.168.1.23/br0] Graphs[kuroo - Traffic - br0 - 192.168.1.23 ], Rule[AVS - Create interface statistics NOT FOR HOSTS]
2022-06-30 02:10:47 - AUTOM8 NOTE: [PID: 835271] create_graph_node() Graph[kuroo - Traffic - br0 - 192.168.1.23 ] Tree Item - Added - id: (157341)
2022-06-30 02:10:47 - CMDPHP ERROR: Suggested value column error. Column name for Data Template ID 197 is not a compatible field name for tables data_template_data and data_template_rrd. Please correct this suggested value mapping
2022-06-30 02:10:47 - AUTOM8 NOTE: Graph Added - Device[kuroo], Graph[kuroo - Device I/O - Bytes Read/Written nvme0n1], DS[kuroo - Device I/O -nvme0n1] Graphs[kuroo - Device I/O - Bytes Read/Written nvme0n1], Rule[AVS - Create I/O statistics - bytes read / written]
2022-06-30 02:10:47 - AUTOM8 NOTE: Graph Added - Device[kuroo], Graph[kuroo - Device I/O - Bytes Read/Written nvme0n1p5], DS[kuroo - Device I/O -nvme0n1p5] Graphs[kuroo - Device I/O - Bytes Read/Written nvme0n1p5], Rule[AVS - Create I/O statistics - bytes read / written]
2022-06-30 02:10:46 - AUTOM8 NOTE: Graph Added - Device[kuroo], Graph[kuroo - Device I/O - Bytes Read/Written nvme0n1p4], DS[kuroo - Device I/O -nvme0n1p4] Graphs[kuroo - Device I/O - Bytes Read/Written nvme0n1p4], Rule[AVS - Create I/O statistics - bytes read / written]
2022-06-30 02:10:46 - AUTOM8 NOTE: Graph Added - Device[kuroo], Graph[kuroo - Device I/O - Bytes Read/Written nvme0n1p3], DS[kuroo - Device I/O -nvme0n1p3] Graphs[kuroo - Device I/O - Bytes Read/Written nvme0n1p3], Rule[AVS - Create I/O statistics - bytes read / written]
2022-06-30 02:10:46 - AUTOM8 NOTE: Graph Added - Device[kuroo], Graph[kuroo - Device I/O - Bytes Read/Written nvme0n1p2], DS[kuroo - Device I/O -nvme0n1p2] Graphs[kuroo - Device I/O - Bytes Read/Written nvme0n1p2], Rule[AVS - Create I/O statistics - bytes read / written]
2022-06-30 02:10:46 - AUTOM8 NOTE: Graph Added - Device[kuroo], Graph[kuroo - Device I/O - Bytes Read/Written nvme0n1p1], DS[kuroo - Device I/O -nvme0n1p1] Graphs[kuroo - Device I/O - Bytes Read/Written nvme0n1p1], Rule[AVS - Create I/O statistics - bytes read / written]
2022-06-30 02:10:46 - AUTOM8 NOTE: Graph Added - Device[kuroo], Graph[kuroo - Device I/O - Reads/Writes nvme0n1], DS[kuroo - Device I/O -nvme0n1] Graphs[kuroo - Device I/O - Reads/Writes nvme0n1], Rule[AVS - Create I/O statistics]
2022-06-30 02:10:46 - AUTOM8 NOTE: Graph Added - Device[kuroo], Graph[kuroo - Device I/O - Reads/Writes nvme0n1p5], DS[kuroo - Device I/O -nvme0n1p5] Graphs[kuroo - Device I/O - Reads/Writes nvme0n1p5], Rule[AVS - Create I/O statistics]
2022-06-30 02:10:45 - AUTOM8 NOTE: Graph Added - Device[kuroo], Graph[kuroo - Device I/O - Reads/Writes nvme0n1p4], DS[kuroo - Device I/O -nvme0n1p4] Graphs[kuroo - Device I/O - Reads/Writes nvme0n1p4], Rule[AVS - Create I/O statistics]
2022-06-30 02:10:45 - AUTOM8 NOTE: Graph Added - Device[kuroo], Graph[kuroo - Device I/O - Reads/Writes nvme0n1p3], DS[kuroo - Device I/O -nvme0n1p3] Graphs[kuroo - Device I/O - Reads/Writes nvme0n1p3], Rule[AVS - Create I/O statistics]
2022-06-30 02:10:45 - AUTOM8 NOTE: Graph Added - Device[kuroo], Graph[kuroo - Device I/O - Reads/Writes nvme0n1p2], DS[kuroo - Device I/O -nvme0n1p2] Graphs[kuroo - Device I/O - Reads/Writes nvme0n1p2], Rule[AVS - Create I/O statistics]
2022-06-30 02:10:45 - AUTOM8 NOTE: Graph Added - Device[kuroo], Graph[kuroo - Device I/O - Reads/Writes nvme0n1p1], DS[kuroo - Device I/O -nvme0n1p1] Graphs[kuroo - Device I/O - Reads/Writes nvme0n1p1], Rule[AVS - Create I/O statistics]
2022-06-30 02:10:45 - AUTOM8 NOTE: Graph Added - Device[kuroo], Graph[kuroo - Traffic], DS[kuroo - Traffic - |query_ifName| - |query_ifIP| ] Graphs[kuroo - Traffic]
2022-06-30 02:10:45 - AUTOM8 NOTE: [PID: 835271] create_graph_node() Graph[kuroo - Traffic] Tree Item - Added - id: (157340)
2022-06-30 02:10:45 - AUTOM8 NOTE: Data Check Succeeded for - Device[kuroo], GT[AVS - Interface - Traffic (bits/sec, 95th) HW]
2022-06-30 02:10:45 - AUTOM8 NOTE: Graph Added - Device[kuroo], Graph[kuroo - CPU Utilization], DS[kuroo - CPU - Idle, kuroo - CPU - System, kuroo - CPU - User] Graphs[kuroo - CPU Utilization]
2022-06-30 02:10:44 - AUTOM8 NOTE: Data Check Succeeded for - Device[kuroo], GT[Net-SNMP - CPU Utilization]
2022-06-30 02:10:44 - AUTOM8 NOTE: Graph Added - Device[kuroo], Graph[kuroo - Load Average], DS[kuroo - Load Average - 1 Minute, kuroo - Load Average - 15 Minute, kuroo - Load Average - 5 Minute] Graphs[kuroo - Load Average]
2022-06-30 02:10:44 - AUTOM8 NOTE: Data Check Succeeded for - Device[kuroo], GT[Net-SNMP - Load Average]
2022-06-30 02:10:21 - WEBUI NOTE: Cacti Log file /var/www/html/log/cacti.log, Removed by user js
2022-06-30 02:10:21 - WEBUI NOTE: Cacti Log Cleared from Web Management Interface.
This replicates the issue. What is interesting, is that realtime polls data correctly - there is no rrd for the graph on mousover, but its pure realtime presentation.