Graphs not getting populated but real time graphs works

[plasky12]

Hey guys,

I have no idea why this may be, but there seems to problem to only with one graph . The graphs are not getting populated for no reason.

However, I do have the real-time graphs plugin installed and when I click on that, the real-time graph updates just fine and is giving proper results.

I'm sorry but I don't really know where to look first. Anyone have any ideas? Thanks!
I ran the dataquery in debug mode and i got this. Not sure if this is an error .

Code: Select all

Total: 0.000000, Delta: 0.000000, Found data query XML file at '/var/www/html/cacti-1.2.18/resource/snmp_queries/cisco_saa.xml'
Total: 0.000000, Delta: 0.000000, Running Data Query [13].
Total: 0.000000, Delta: 0.000000, Found Type = '3' [SNMP Query].
Total: 0.000000, Delta: 0.000000, XML file parsed ok.
Total: 0.000000, Delta: 0.000000, Auto Bulk Walk Size Selected.
Total: 0.010000, Delta: 0.010000, Tested Bulk Walk Size 1 with a response of 0.0076.
Total: 0.020000, Delta: 0.010000, Tested Bulk Walk Size 5 with a response of 0.0091.
Total: 0.020000, Delta: 0.000000, Bulk Walk Size selected was 1.
Total: 0.020000, Delta: 0.000000, <oid_num_indexes> missing in XML file, 'Index Count Changed' emulated by counting oid_index entries
Total: 0.020000, Delta: 0.000000, Executing SNMP walk for list of indexes @ '.1.3.6.1.4.1.9.9.42.1.2.1.1.2' Index Count: 0
Total: 0.020000, Delta: 0.000000, No SNMP data returned

To confirm, this happens only with one single device for now. Others all works fine and the graphs are getting populated as usual.
Could anyone guide me here? Attached screenshots for the same

[kdsmith]

Probably not your solution, but I've been having this issue lately with a few of my devices. Everything seemed okay, but the graphs were not populating. I even blew out the device and recreated them, but nothing.

Anyway, I clicked the Console->System Utilities->Technical Support and on that I noticed that my Cacti and Spine were different versions (they weren't even close). Apparently, my Cacti was getting updated with package releases, but my Spine was not. So, I went to https://files.cacti.net/spine/ and downloaded the Spine that matched my Cacti (1.2.20) and upgraded it. Now all of those devices are getting populated graphs.

[plasky12]

Hi,

I checked this, but both my spine and cacti version are the same.
Could it be something else?
Thanks,

[TheWitness]

Likely a permissions problem in the RRA directory or the RRDfiles. Should be owned by the Web Server account. A few dozen lines from your Cacti.log will help.

[plasky12]

hi,

all the files under the rra directory are all owned by apache user. No change in that
Here are a few lines from cacti.log. It's just strange the graphs stopped plotting or there gaps in the graphs suddenly only for a few devices.
What could be the reason?

Logs
-----
2022-05-30 07:28:10 - SYSTEM MONITOR STATS: Time:0.1115 Reboots:0 DownDevices:0 Notifications:0 Purges:1
2022-05-30 07:28:09 - SYSTEM STATS: Time:7.8434 Method:spine Processes:8 Threads:50 Hosts:5346 HostsPerProcess:669 DataSources:21866 RRDsProcessed:0
2022-05-30 07:28:04 - SPINE: Poller[1] PID[5183] Device[1984] Hostname[94.105.24.36] NOTICE: HOST EVENT: Device Returned from DOWN State
2022-05-30 07:28:04 - SPINE: Poller[1] PID[5296] Device[2703] Hostname[94.105.34.44] NOTICE: HOST EVENT: Device Returned from DOWN State
2022-05-30 07:28:04 - SPINE: Poller[1] PID[5562] Device[4809] Hostname[94.105.1.150] ERROR: HOST EVENT: Device is DOWN Message: Device did not respond to SNMP
2022-05-30 07:28:03 - SPINE: Poller[1] PID[5712] ERROR: Spine Timed Out While Processing Available Thread Lock
2022-05-30 07:27:13 - SYSTEM MONITOR STATS: Time:0.1116 Reboots:0 DownDevices:0 Notifications:0 Purges:1
2022-05-30 07:27:13 - SYSTEM STATS: Time:11.5950 Method:spine Processes:8 Threads:50 Hosts:5346 HostsPerProcess:669 DataSources:21944 RRDsProcessed:0
2022-05-30 07:27:03 - SPINE: Poller[1] PID[32721] ERROR: Spine Timed Out While Processing Available Thread Lock
2022-05-30 07:26:26 - SYSTEM BOOST STATS: Time:16.38 RRDUpdates:50906
2022-05-30 07:26:24 - SYSTEM BOOST STATS: Time:14.26 ProcessNumber:6 RRDUpdates:2998
2022-05-30 07:26:24 - BOOST CHILD DEBUG: Processing Complete for Boost Process 6. It took 3 passed to complete.
2022-05-30 07:26:24 - SYSTEM BOOST STATS: Time:14.22 ProcessNumber:11 RRDUpdates:3088
2022-05-30 07:26:24 - BOOST CHILD DEBUG: Processing Complete for Boost Process 11. It took 3 passed to complete.
2022-05-30 07:26:24 - SYSTEM BOOST STATS: Time:14.19 ProcessNumber:8 RRDUpdates:3888
2022-05-30 07:26:24 - BOOST CHILD DEBUG: Processing Complete for Boost Process 8. It took 3 passed to complete.
2022-05-30 07:26:24 - SYSTEM BOOST STATS: Time:14.27 ProcessNumber:5 RRDUpdates:3894
2022-05-30 07:26:24 - BOOST CHILD DEBUG: Processing Complete for Boost Process 5. It took 3 passed to complete.
2022-05-30 07:26:24 - SYSTEM BOOST STATS: Time:14.13 ProcessNumber:9 RRDUpdates:3882
2022-05-30 07:26:24 - SYSTEM BOOST STATS: Time:14.02 ProcessNumber:13 RRDUpdates:2140
2022-05-30 07:26:24 - BOOST CHILD DEBUG: Processing Complete for Boost Process 9. It took 3 passed to complete.
2022-05-30 07:26:24 - SYSTEM BOOST STATS: Time:14.12 ProcessNumber:3 RRDUpdates:3223
2022-05-30 07:26:24 - BOOST CHILD DEBUG: Processing Complete for Boost Process 13. It took 3 passed to complete.
2022-05-30 07:26:24 - SYSTEM BOOST STATS: Time:14.07 ProcessNumber:7 RRDUpdates:3750
2022-05-30 07:26:24 - BOOST CHILD DEBUG: Processing Complete for Boost Process 3. It took 3 passed to complete.
2022-05-30 07:26:24 - SYSTEM BOOST STATS: Time:13.91 ProcessNumber:15 RRDUpdates:1754
2022-05-30 07:26:24 - BOOST CHILD DEBUG: Processing Complete for Boost Process 7. It took 3 passed to complete.
2022-05-30 07:26:24 - SYSTEM BOOST STATS: Time:13.93 ProcessNumber:14 RRDUpdates:2542
2022-05-30 07:26:24 - BOOST CHILD DEBUG: Processing Complete for Boost Process 15. It took 2 passed to complete.
2022-05-30 07:26:24 - BOOST CHILD DEBUG: Processing Complete for Boost Process 14. It took 3 passed to complete.
2022-05-30 07:26:24 - SYSTEM BOOST STATS: Time:14.09 ProcessNumber:4 RRDUpdates:3888
2022-05-30 07:26:24 - SYSTEM BOOST STATS: Time:14.03 ProcessNumber:2 RRDUpdates:3754
2022-05-30 07:26:24 - SYSTEM BOOST STATS: Time:13.80 ProcessNumber:16 RRDUpdates:3858
2022-05-30 07:26:24 - BOOST CHILD DEBUG: Processing Complete for Boost Process 4. It took 3 passed to complete.
2022-05-30 07:26:24 - BOOST CHILD DEBUG: Processing Complete for Boost Process 2. It took 3 passed to complete.
2022-05-30 07:26:24 - BOOST CHILD DEBUG: Processing Complete for Boost Process 16. It took 3 passed to complete.
2022-05-30 07:26:24 - SYSTEM BOOST STATS: Time:13.96 ProcessNumber:1 RRDUpdates:3529
2022-05-30 07:26:24 - BOOST CHILD DEBUG: Processing Complete for Boost Process 1. It took 3 passed to complete.
2022-05-30 07:26:24 - SYSTEM BOOST STATS: Time:13.79 ProcessNumber:10 RRDUpdates:3702
2022-05-30 07:26:24 - BOOST CHILD DEBUG: Processing Complete for Boost Process 10. It took 3 passed to complete.
2022-05-30 07:26:24 - SYSTEM BOOST STATS: Time:13.80 ProcessNumber:12 RRDUpdates:1016
2022-05-30 07:26:24 - BOOST CHILD DEBUG: Processing Complete for Boost Process 12. It took 2 passed to complete.
2022-05-30 07:26:18 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 11
2022-05-30 07:26:17 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 8
2022-05-30 07:26:17 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 6
2022-05-30 07:26:17 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 4
2022-05-30 07:26:17 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 5
2022-05-30 07:26:17 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 16
2022-05-30 07:26:17 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 13
2022-05-30 07:26:17 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 14
2022-05-30 07:26:17 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 3
2022-05-30 07:26:17 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 7
2022-05-30 07:26:17 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 2
2022-05-30 07:26:17 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 9
2022-05-30 07:26:17 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 10
2022-05-30 07:26:17 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 1
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 16
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processes:16, TotalRows:3858
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 16
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing 1 of 1 for Boost Process 15
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processes:15, TotalRows:1754
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 15
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 14
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processes:14, TotalRows:2542
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 14
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 10
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processes:10, TotalRows:3702
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 13
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processes:13, TotalRows:2140
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 10
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 13
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 7
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processes:7, TotalRows:3750
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 7
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 9
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processes:9, TotalRows:3882
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 9
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 1
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 11
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 8
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processes:11, TotalRows:3088
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processes:8, TotalRows:3888
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 11
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processes:1, TotalRows:3529
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 8
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 1
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 3
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 6
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processes:3, TotalRows:3223
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processes:6, TotalRows:2998
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 3
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 6
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing 1 of 1 for Boost Process 12
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processes:12, TotalRows:1016
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 2
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processes:2, TotalRows:3754
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 12
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 2
2022-05-30 07:26:10 - POLLER: Poller[1] PID[21384] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=16 --debug]
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 5
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processes:5, TotalRows:3894
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 5
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 4
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processes:4, TotalRows:3888
2022-05-30 07:26:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 4
2022-05-30 07:26:10 - POLLER: Poller[1] PID[21384] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=15 --debug]
2022-05-30 07:26:10 - POLLER: Poller[1] PID[21384] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=14 --debug]
2022-05-30 07:26:10 - POLLER: Poller[1] PID[21384] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=13 --debug]
2022-05-30 07:26:10 - POLLER: Poller[1] PID[21384] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=12 --debug]
2022-05-30 07:26:10 - POLLER: Poller[1] PID[21384] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=11 --debug]
2022-05-30 07:26:10 - POLLER: Poller[1] PID[21384] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=10 --debug]
2022-05-30 07:26:10 - POLLER: Poller[1] PID[21384] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=9 --debug]
2022-05-30 07:26:10 - POLLER: Poller[1] PID[21384] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=8 --debug]
2022-05-30 07:26:10 - POLLER: Poller[1] PID[21384] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=7 --debug]
2022-05-30 07:26:10 - POLLER: Poller[1] PID[21384] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=6 --debug]
2022-05-30 07:26:10 - POLLER: Poller[1] PID[21384] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=5 --debug]
2022-05-30 07:26:10 - POLLER: Poller[1] PID[21384] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=4 --debug]
2022-05-30 07:26:10 - POLLER: Poller[1] PID[21384] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=3 --debug]
2022-05-30 07:26:10 - SYSTEM MONITOR STATS: Time:0.1111 Reboots:0 DownDevices:2 Notifications:0 Purges:0
2022-05-30 07:26:10 - POLLER: Poller[1] PID[21384] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=2 --debug]
2022-05-30 07:26:10 - POLLER: Poller[1] PID[21384] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=1 --debug]
2022-05-30 07:26:10 - SYSTEM STATS: Time:8.1441 Method:spine Processes:8 Threads:50 Hosts:5346 HostsPerProcess:669 DataSources:22197 RRDsProcessed:0
2022-05-30 07:26:06 - SPINE: Poller[1] PID[16936] ERROR: Spine Timed Out While Processing Available Thread Lock
2022-05-30 07:26:03 - SPINE: Poller[1] PID[17339] ERROR: Spine Timed Out While Processing Available Thread Lock
2022-05-30 07:25:10 - SYSTEM MONITOR STATS: Time:0.1139 Reboots:0 DownDevices:0 Notifications:0 Purges:0
2022-05-30 07:25:10 - SYSTEM STATS: Time:8.6043 Method:spine Processes:8 Threads:50 Hosts:5346 HostsPerProcess:669 DataSources:21748 RRDsProcessed:0
2022-05-30 07:25:03 - SPINE: Poller[1] PID[10688] Device[1984] Hostname[94.105.24.36] ERROR: HOST EVENT: Device is DOWN Message: Device did not respond to SNMP
2022-05-30 07:25:03 - SPINE: Poller[1] PID[10577] Device[1406] Hostname[94.105.17.148] NOTICE: HOST EVENT: Device Returned from DOWN State
2022-05-30 07:25:03 - SPINE: Poller[1] PID[11241] ERROR: Spine Timed Out While Processing Available Thread Lock
2022-05-30 07:25:03 - SPINE: Poller[1] PID[11402] Device[6783] Hostname[94.105.31.22] NOTICE: HOST EVENT: Device Returned from DOWN State
2022-05-30 07:24:08 - SYSTEM MONITOR STATS: Time:0.1176 Reboots:0 DownDevices:0 Notifications:0 Purges:1
2022-05-30 07:24:08 - SYSTEM STATS: Time:7.4169 Method:spine Processes:8 Threads:50 Hosts:5346 HostsPerProcess:669 DataSources:21226 RRDsProcessed:0
2022-05-30 07:24:06 - SPINE: Poller[1] PID[4864] Device[2294] Hostname[94.105.19.71] NOTICE: HOST EVENT: Device Returned from DOWN State
2022-05-30 07:24:03 - SPINE: Poller[1] PID[4994] Device[2603] Hostname[94.105.33.89] ERROR: HOST EVENT: Device is DOWN Message: Device did not respond to SNMP
2022-05-30 07:24:03 - SPINE: Poller[1] PID[4760] Device[752] Hostname[94.105.19.224] ERROR: HOST EVENT: Device is DOWN Message: Device did not respond to SNMP
2022-05-30 07:24:03 - SPINE: Poller[1] PID[5392] ERROR: Spine Timed Out While Processing Available Thread Lock
2022-05-30 07:24:01 - SPINE: Poller[1] PID[4994] Device[3108] Hostname[94.105.32.149] NOTICE: HOST EVENT: Device Returned from DOWN State
2022-05-30 07:24:01 - SPINE: Poller[1] PID[4864] Device[1921] Hostname[94.105.35.43] NOTICE: HOST EVENT: Device Returned from DOWN State
2022-05-30 07:23:28 - SYSTEM BOOST STATS: Time:18.72 RRDUpdates:48547
2022-05-30 07:23:26 - SYSTEM BOOST STATS: Time:15.95 ProcessNumber:15 RRDUpdates:2296
2022-05-30 07:23:26 - BOOST CHILD DEBUG: Processing Complete for Boost Process 15. It took 3 passed to complete.
2022-05-30 07:23:26 - SYSTEM BOOST STATS: Time:15.95 ProcessNumber:6 RRDUpdates:3180
2022-05-30 07:23:26 - BOOST CHILD DEBUG: Processing Complete for Boost Process 6. It took 3 passed to complete.
2022-05-30 07:23:26 - SYSTEM BOOST STATS: Time:15.88 ProcessNumber:11 RRDUpdates:2992
2022-05-30 07:23:26 - SYSTEM BOOST STATS: Time:16.04 ProcessNumber:7 RRDUpdates:3180
2022-05-30 07:23:26 - SYSTEM BOOST STATS: Time:15.94 ProcessNumber:3 RRDUpdates:3260
2022-05-30 07:23:26 - BOOST CHILD DEBUG: Processing Complete for Boost Process 7. It took 3 passed to complete.
2022-05-30 07:23:26 - BOOST CHILD DEBUG: Processing Complete for Boost Process 11. It took 3 passed to complete.
2022-05-30 07:23:26 - BOOST CHILD DEBUG: Processing Complete for Boost Process 3. It took 3 passed to complete.
2022-05-30 07:23:26 - SYSTEM BOOST STATS: Time:15.98 ProcessNumber:4 RRDUpdates:3978
2022-05-30 07:23:26 - BOOST CHILD DEBUG: Processing Complete for Boost Process 4. It took 3 passed to complete.
2022-05-30 07:23:26 - SYSTEM BOOST STATS: Time:15.90 ProcessNumber:8 RRDUpdates:3278
2022-05-30 07:23:26 - BOOST CHILD DEBUG: Processing Complete for Boost Process 8. It took 3 passed to complete.
2022-05-30 07:23:26 - SYSTEM BOOST STATS: Time:15.98 ProcessNumber:1 RRDUpdates:3168
2022-05-30 07:23:26 - SYSTEM BOOST STATS: Time:15.78 ProcessNumber:13 RRDUpdates:2306
2022-05-30 07:23:26 - SYSTEM BOOST STATS: Time:15.75 ProcessNumber:12 RRDUpdates:714
2022-05-30 07:23:26 - BOOST CHILD DEBUG: Processing Complete for Boost Process 1. It took 3 passed to complete.
2022-05-30 07:23:26 - BOOST CHILD DEBUG: Processing Complete for Boost Process 13. It took 3 passed to complete.
2022-05-30 07:23:26 - BOOST CHILD DEBUG: Processing Complete for Boost Process 12. It took 2 passed to complete.
2022-05-30 07:23:26 - SYSTEM BOOST STATS: Time:15.79 ProcessNumber:9 RRDUpdates:3558
2022-05-30 07:23:26 - SYSTEM BOOST STATS: Time:15.76 ProcessNumber:10 RRDUpdates:3292
2022-05-30 07:23:26 - SYSTEM BOOST STATS: Time:15.65 ProcessNumber:14 RRDUpdates:1904
2022-05-30 07:23:26 - BOOST CHILD DEBUG: Processing Complete for Boost Process 9. It took 3 passed to complete.
2022-05-30 07:23:26 - BOOST CHILD DEBUG: Processing Complete for Boost Process 10. It took 3 passed to complete.
2022-05-30 07:23:26 - BOOST CHILD DEBUG: Processing Complete for Boost Process 14. It took 2 passed to complete.
2022-05-30 07:23:26 - SYSTEM BOOST STATS: Time:15.70 ProcessNumber:5 RRDUpdates:3984
2022-05-30 07:23:26 - BOOST CHILD DEBUG: Processing Complete for Boost Process 5. It took 3 passed to complete.
2022-05-30 07:23:26 - SYSTEM BOOST STATS: Time:15.42 ProcessNumber:16 RRDUpdates:3948
2022-05-30 07:23:26 - BOOST CHILD DEBUG: Processing Complete for Boost Process 16. It took 3 passed to complete.
2022-05-30 07:23:26 - SYSTEM BOOST STATS: Time:15.39 ProcessNumber:2 RRDUpdates:3509
2022-05-30 07:23:25 - BOOST CHILD DEBUG: Processing Complete for Boost Process 2. It took 3 passed to complete.
2022-05-30 07:23:18 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 15
2022-05-30 07:23:18 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 10
2022-05-30 07:23:18 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 8
2022-05-30 07:23:18 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 6
2022-05-30 07:23:18 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 16
2022-05-30 07:23:18 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 3
2022-05-30 07:23:18 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 4
2022-05-30 07:23:18 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 5
2022-05-30 07:23:18 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 13
2022-05-30 07:23:18 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 11
2022-05-30 07:23:18 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 9
2022-05-30 07:23:18 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 7
2022-05-30 07:23:18 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 1
2022-05-30 07:23:18 - BOOST CHILD DEBUG: Processing 2 of 2 for Boost Process 2
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 16
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processes:16, TotalRows:3948
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 16
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing 1 of 1 for Boost Process 14
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processes:14, TotalRows:1904
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 14
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 15
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processes:15, TotalRows:2296
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 11
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processes:11, TotalRows:2992
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 15
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing 1 of 1 for Boost Process 12
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processes:12, TotalRows:714
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 11
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 12
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 13
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processes:13, TotalRows:2306
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 13
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 10
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processes:10, TotalRows:3292
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 6
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 10
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processes:6, TotalRows:3180
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 6
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 3
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processes:3, TotalRows:3260
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 3
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 9
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processes:9, TotalRows:3558
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 9
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 5
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processes:5, TotalRows:3984
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 5
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 8
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processes:8, TotalRows:3278
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 8
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 4
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processes:4, TotalRows:3978
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 4
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 2
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processes:2, TotalRows:3509
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 2
2022-05-30 07:23:10 - POLLER: Poller[1] PID[26121] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=16 --debug]
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 7
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processes:7, TotalRows:3180
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 7
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing 1 of 2 for Boost Process 1
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processes:1, TotalRows:3168
2022-05-30 07:23:10 - BOOST CHILD DEBUG: Processing RRRtool Output for Boost Process 1
2022-05-30 07:23:10 - POLLER: Poller[1] PID[26121] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=15 --debug]
2022-05-30 07:23:10 - POLLER: Poller[1] PID[26121] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=14 --debug]
2022-05-30 07:23:10 - POLLER: Poller[1] PID[26121] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=13 --debug]
2022-05-30 07:23:10 - POLLER: Poller[1] PID[26121] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=12 --debug]
2022-05-30 07:23:10 - POLLER: Poller[1] PID[26121] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=11 --debug]
2022-05-30 07:23:10 - POLLER: Poller[1] PID[26121] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=10 --debug]
2022-05-30 07:23:10 - POLLER: Poller[1] PID[26121] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=9 --debug]
2022-05-30 07:23:10 - POLLER: Poller[1] PID[26121] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=8 --debug]
2022-05-30 07:23:10 - POLLER: Poller[1] PID[26121] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=7 --debug]
2022-05-30 07:23:10 - POLLER: Poller[1] PID[26121] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=6 --debug]
2022-05-30 07:23:10 - POLLER: Poller[1] PID[26121] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=5 --debug]
2022-05-30 07:23:10 - POLLER: Poller[1] PID[26121] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=4 --debug]
2022-05-30 07:23:10 - POLLER: Poller[1] PID[26121] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=3 --debug]
2022-05-30 07:23:10 - POLLER: Poller[1] PID[26121] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=2 --debug]
2022-05-30 07:23:10 - POLLER: Poller[1] PID[26121] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: /var/www/html/cacti-1.2.18/poller_boost.php --child=1 --debug]
2022-05-30 07:23:10 - SYSTEM MONITOR STATS: Time:0.1092 Reboots:0 DownDevices:0 Notifications:0 Purges:0
2022-05-30 07:23:09 - SYSTEM STATS: Time:8.4948 Method:spine Processes:8 Threads:50 Hosts:5346 HostsPerProcess:669 DataSources:21548 RRDsProcessed:0
2022-05-30 07:23:03 - SPINE: Poller[1] PID[21528] Device[4809] Hostname[94.105.1.150] NOTICE: HOST EVENT: Device Returned from DOWN State
2022-05-30 07:23:03 - SPINE: Poller[1] PID[21663] ERROR: Spine Timed Out While Processing Available Thread Lock
2022-05-30 07:22:09 - SYSTEM MONITOR STATS: Time:0.1016 Reboots:0 DownDevices:0 Notifications:0 Purges:0
2022-05-30 07:22:09 - SYSTEM STATS: Time:8.4614 Method:spine Processes:8 Threads:50 Hosts:5346 HostsPerProcess:669 DataSources:21737 RRDsProcessed:0
2022-05-30 07:22:05 - SPINE: Poller[1] PID[17140] ERROR: Spine Timed Out While Processing Available Thread Lock
2022-05-30 07:22:05 - SPINE: Poller[1] PID[17012] ERROR: Spine Timed Out While Processing Available Thread Lock
2022-05-30 07:22:05 - SPINE: Poller[1] PID[16906] ERROR: Spine Timed Out While Processing Available Thread Lock
2022-05-30 07:22:04 - SPINE: Poller[1] PID[17012] Device[1782] Hostname[94.105.26.205] NOTICE: HOST EVENT: Device Returned from DOWN State
2022-05-30 07:22:04 - SPINE: Poller[1] PID[17269] Device[4222] Hostname[94.105.15.176] NOTICE: HOST EVENT: Device Returned from DOWN State
2022-05-30 07:22:03 - SPINE: Poller[1] PID[17412] Device[5428] Hostname[94.105.10.251] ERROR: HOST EVENT: Device is DOWN Message: Device did not respond to SNMP
2022-05-30 07:22:03 - SPINE: Poller[1] PID[17140] Device[2549] Hostname[94.105.33.45] ERROR: HOST EVENT: Device is DOWN Message: Device did not respond to SNMP
2022-05-30 07:22:03 - SPINE: Poller[1] PID[16906] Device[1406] Hostname[94.105.17.148] ERROR: HOST EVENT: Device is DOWN Message: Device did not respond to SNMP
2022-05-30 07:22:03 - SPINE: Poller[1] PID[17543] ERROR: Spine Timed Out While Processing Available Thread Lock
2022-05-30 07:22:02 - SPINE: Poller[1] PID[17412] Device[4902] Hostname[94.105.11.91] NOTICE: HOST EVENT: Device Returned from DOWN State
2022-05-30 07:22:01 - SPINE: Poller[1] PID[17140] Device[3074] Hostname[94.105.8.66] NOTICE: HOST EVENT: Device Returned from DOWN State
2022-05-30 07:21:12 - SYSTEM MONITOR STATS: Time:0.1102 Reboots:0 DownDevices:0 Notifications:0 Purges:2
2022-05-30 07:21:12 - SYSTEM STATS: Time:10.8717 Method:spine Processes:8 Threads:50 Hosts:5346 HostsPerProcess:669 DataSources:21825 RRDsProcessed:0
2022-05-30 07:21:03 - SPINE: Poller[1] PID[11637] Device[1921] Hostname[94.105.35.43] ERROR: HOST EVENT: Device is DOWN Message: Device did not respond to SNMP
2022-05-30 07:21:03 - SPINE: Poller[1] PID[12147] ERROR: Spine Timed Out While Processing Available Thread Lock
2022-05-30 07:21:02 - SPINE: Poller[1] PID[11533] Device[855] Hostname[94.105.27.33] NOTICE: HOST EVENT: Device Returned from DOWN State
2022-05-30 07:20:24 - SYSTEM BOOST STATS: Time:16.24 RRDUpdates:70063
2022-05-30 07:20:23 - SYSTEM BOOST STATS: Time:15.11 ProcessNumber:1 RRDUpdates:4709
2022-05-30 07:20:23 - BOOST CHILD DEBUG: Processing Complete for Boost Process 1. It took 4 passed to complete.
2022-05-30 07:20:23 - SYSTEM BOOST STATS: Time:15.09 ProcessNumber:12 RRDUpdates:1835
2022-05-30 07:20:23 - SYSTEM BOOST STATS: Time:15.05 ProcessNumber:2 RRDUpdates:5079
2022-05-30 07:20:23 - BOOST CHILD DEBUG: Processing Complete for Boost Process 12. It took 2 passed to complete.
2022-05-30 07:20:23 - BOOST CHILD DEBUG: Processing Complete for Boost Process 2. It took 4 passed to complete.
2022-05-30 07:20:23 - SYSTEM BOOST STATS: Time:15.03 ProcessNumber:3 RRDUpdates:4540
2022-05-30 07:20:23 - SYSTEM BOOST STATS: Time:15.04 ProcessNumber:4 RRDUpdates:5664
2022-05-30 07:20:23 - BOOST CHILD DEBUG: Processing Complete for Boost Process 3. It took 4 passed to complete.
2022-05-30 07:20:23 - BOOST CHILD DEBUG: Processing Complete for Boost Process 4. It took 4 passed to complete.
2022-05-30 07:20:23 - SYSTEM BOOST STATS: Time:15.00 ProcessNumber:10 RRDUpdates:5118
2022-05-30 07:20:23 - SYSTEM BOOST STATS: Time:14.98 ProcessNumber:9 RRDUpdates:5652
2022-05-30 07:20:23 - BOOST CHILD DEBUG: Processing Complete for Boost Process 10. It took 4 passed to complete.
2022-05-30 07:20:23 - BOOST CHILD DEBUG: Processing Complete for Boost Process 9. It took 4 passed to complete.
2022-05-30 07:20:23 - SYSTEM BOOST STATS: Time:14.95 ProcessNumber:6 RRDUpdates:5652
2022-05-30 07:20:23 - SYSTEM BOOST STATS: Time:15.02 ProcessNumber:7 RRDUpdates:5664
2022-05-30 07:20:23 - BOOST CHILD DEBUG: Processing Complete for Boost Process 6. It took 4 passed to complete.
2022-05-30 07:20:23 - SYSTEM BOOST STATS: Time:14.90 ProcessNumber:15 RRDUpdates:2129
2022-05-30 07:20:23 - BOOST CHILD DEBUG: Processing Complete for Boost Process 7. It took 4 passed to complete.
2022-05-30 07:20:23 - BOOST CHILD DEBUG: Processing Complete for Boost Process 15. It took 3 passed to complete.

[plasky12]

Hi,

Anyone with any suggestions or help?
I still get empty graphs for some devices.

[TheWitness]

Turn on device debug for the one device, your Cacti log will get some additional entries then that should be informative.

[plasky12]

hi,

I enabled the device debug for one device but i dont find any information related to that device in the cacti log.
Really strange. graphs were plotting perfectly till a couple of weeks and now they are empty. the device is up as well.
Any other suggestion?
Thanks.

[plasky12]

When i do an rrtoold fetch for the affected device , i get this output,
rrdtool fetch xyz.rrd AVERAGE | tail

1654509600: -nan -nan
1654509900: -nan -nan
1654510200: -nan -nan
1654510500: -nan -nan
1654510800: -nan -nan
1654511100: -nan -nan
1654511400: -nan -nan
1654511700: -nan -nan
1654512000: -nan -nan
1654512300: -nan -nan

Anyway to tune this?

[plasky12]

And this is output when i do rrdtool for the affected device.

rrdtool info xyz.rrd | grep ds
ds[traffic_in].index = 0
ds[traffic_in].type = "COUNTER"
ds[traffic_in].minimal_heartbeat = 600
ds[traffic_in].min = 0.0000000000e+00
ds[traffic_in].max = 5.1200000000e+08
ds[traffic_in].last_ds = "3409684536"
ds[traffic_in].value = NaN
ds[traffic_in].unknown_sec = 7
ds[traffic_out].index = 1
ds[traffic_out].type = "COUNTER"
ds[traffic_out].minimal_heartbeat = 600
ds[traffic_out].min = 0.0000000000e+00
ds[traffic_out].max = 5.1200000000e+08
ds[traffic_out].last_ds = "3472986418"
ds[traffic_out].value = NaN
ds[traffic_out].unknown_sec = 7

[TheWitness]

The Max Value on your Data Sources is too log. Edit the Data Source and Turn on Data Source Debug to see the syntax to correct the discrepancy. If it says that the Data Source is okay, then the SNMP agent is returning the wrong ifHighSpeed, which happens.

[plasky12]

Hi,

I enabled the data source debug and this was the output for the affected device.

Data Source Debug

/usr/bin/rrdtool create \
/var/www/html/cacti-1.2.18/rra/xyz_24254.rrd \
--start 0 --step 60 \
DS:traffic_in:COUNTER:600:0:512000000 \
DS:traffic_out:COUNTER:600:0:512000000 \
RRA:AVERAGE:0.5:1:600 \
RRA:AVERAGE:0.51 \
RRA:AVERAGE:0.5:5:1440 \
RRA:AVERAGE:0.5:30:1344 \
RRA:AVERAGE:0.5:120:4320 \
RRA:MIN:0.5:1:600 \
RRA:MIN:0.51 \
RRA:MIN:0.5:5:1440 \
RRA:MIN:0.5:30:1344 \
RRA:MIN:0.5:120:4320 \
RRA:MAX:0.5:1:600 \
RRA:MAX:0.51 \
RRA:MAX:0.5:5:1440 \
RRA:MAX:0.5:30:1344 \
RRA:MAX:0.5:120:4320 \

What do i need to update or change?
I am kinda new to cacti , would really appreciate any good help

[TheWitness]

Senior moment. Data Source Info Mode. My bad. It'll look something like what you see in the attached image.

[plasky12]

Hi, Here you go.

[/img]

[TheWitness]

I'm leaning so some other issue. That max number looks low, but you are also 1 minute polling. So, I guess you'll have to keep digging. Make your permissions are good to the RRA folder from the websites perspective.