graph suddenly disappears and then appears again

[jsofari]

some of the devices that I added to cacti suddenly the graph doesn't work and after a few hours the graph runs again but creates an empty graph in the middle
i use
cacti 1.2.15
Centos 7

[TheWitness]

Upgrade.

[jsofari]

upgrade not solved the problem, graph still intermitent

[macan]

Is it problem with all devices or only few?

If only few maybe poller don't finish work on time.
Try log, find text "SYSTEM STATS". What is the poller time?

[jsofari]

This happens intermittently on several devices
poller time is 5 minute
and this log

2023-02-04 04:15:00 - SYSTEM STATS: Time:299.3050 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:7993
2023-02-04 04:10:01 - SYSTEM STATS: Time:299.5648 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:8292
2023-02-04 04:05:01 - SYSTEM STATS: Time:299.6003 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:8306
2023-02-04 04:00:00 - SYSTEM STATS: Time:298.5107 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:7102
2023-02-04 03:55:00 - SYSTEM STATS: Time:298.5996 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:8281
2023-02-04 03:50:00 - SYSTEM STATS: Time:298.6385 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:8586
2023-02-04 03:45:00 - SYSTEM STATS: Time:298.5814 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:8322
2023-02-04 03:40:00 - SYSTEM STATS: Time:298.6239 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:8286
2023-02-04 03:35:00 - SYSTEM STATS: Time:299.3131 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:8283
2023-02-04 03:30:00 - SYSTEM STATS: Time:298.2908 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:8294
2023-02-04 03:25:00 - SYSTEM STATS: Time:298.3634 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:9237
2023-02-04 03:20:00 - SYSTEM STATS: Time:298.2260 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:7648
2023-02-04 03:15:01 - SYSTEM STATS: Time:299.2798 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:8289
2023-02-04 03:10:00 - SYSTEM STATS: Time:299.3717 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:9197
2023-02-04 03:05:01 - SYSTEM STATS: Time:299.3682 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:9094
2023-02-04 03:00:00 - SYSTEM STATS: Time:299.4161 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:6346
2023-02-04 02:55:00 - SYSTEM STATS: Time:299.3329 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:4908
2023-02-04 02:50:00 - SYSTEM STATS: Time:299.3570 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:5013
2023-02-04 02:45:00 - SYSTEM STATS: Time:299.3614 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:5053
2023-02-04 02:40:00 - SYSTEM STATS: Time:299.3943 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:5746
2023-02-04 02:35:00 - SYSTEM STATS: Time:298.4263 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:6032
2023-02-04 02:30:00 - SYSTEM STATS: Time:298.4283 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:5898
2023-02-04 02:25:00 - SYSTEM STATS: Time:298.4112 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:5702
2023-02-04 02:20:00 - SYSTEM STATS: Time:298.4106 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:5931
2023-02-04 02:15:00 - SYSTEM STATS: Time:298.5236 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:7186
2023-02-04 02:10:00 - SYSTEM STATS: Time:298.6686 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:8340
2023-02-04 02:05:01 - SYSTEM STATS: Time:299.6603 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:9276
2023-02-04 02:00:01 - SYSTEM STATS: Time:299.7288 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:9948
2023-02-04 01:55:00 - SYSTEM STATS: Time:298.7388 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:9917
2023-02-04 01:50:00 - SYSTEM STATS: Time:298.7538 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:10048
2023-02-04 01:45:00 - SYSTEM STATS: Time:298.7919 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:10374
2023-02-04 01:40:00 - SYSTEM STATS: Time:299.5409 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:11327
2023-02-04 01:35:00 - SYSTEM STATS: Time:299.5246 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:11522
2023-02-04 01:30:00 - SYSTEM STATS: Time:298.7572 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:10596
2023-02-04 01:25:00 - SYSTEM STATS: Time:298.3624 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:9529
2023-02-04 01:20:00 - SYSTEM STATS: Time:298.7704 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:10643
2023-02-04 01:15:00 - SYSTEM STATS: Time:298.4433 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:10307
2023-02-04 01:10:00 - SYSTEM STATS: Time:298.4109 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:10218
2023-02-04 01:05:00 - SYSTEM STATS: Time:298.7331 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:10257
2023-02-04 01:00:00 - SYSTEM STATS: Time:298.7116 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:10094
2023-02-04 00:55:00 - SYSTEM STATS: Time:298.8372 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:10964
2023-02-04 00:50:00 - SYSTEM STATS: Time:298.7534 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:10400
2023-02-04 00:45:00 - SYSTEM STATS: Time:299.3746 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:9757
2023-02-04 00:40:00 - SYSTEM STATS: Time:298.7507 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:10017
2023-02-04 00:35:00 - SYSTEM STATS: Time:299.4515 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:10289
2023-02-04 00:30:00 - SYSTEM STATS: Time:299.4691 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:10375
2023-02-04 00:25:00 - SYSTEM STATS: Time:299.5346 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:11086
2023-02-04 00:20:00 - SYSTEM STATS: Time:298.8876 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:12197
2023-02-04 00:15:00 - SYSTEM STATS: Time:298.7602 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:10501
2023-02-04 00:10:00 - SYSTEM STATS: Time:298.4953 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:11234
2023-02-04 00:05:00 - SYSTEM STATS: Time:298.6749 Method:cmd.php Processes:1 Threads:1 Hosts:332 HostsPerProcess:332 DataSources:28036 RRDsProcessed:9467

[TheWitness]

Either upgrade to spine and go parallel that way, or use more processes for cmd.php.

[jsofari]

i will try add more proses for cmd.php first, i will inform later

[jsofari]

after I re-analyzed the graph that suddenly disappeared it only happened on devices that had less than 99% availability

[macan]

Your poller time is 300 seconds. Poller fails to complete the work on time. You can try decrease snmp/icmp timeouts too.

Try to sort your devices by poll time and try to improve the worst devices

[TheWitness]

As previously mentioned, and either not comprehended or ignored by the poster, this type of behavior can be corrected by parallelizing the collection process. Spine is the way to go, but using cmd.php processes is a nice workaround for sub thousand device sites with some limitations.