Improve Poller Performance and Stability

[hakiki]

Hi, I have a system running Cacti 1.2.19 with Spine poller 1.2.19. I found that poller performance is unstable like shown below:

Code: Select all

2022-02-14 07:36:28 - SYSTEM STATS: Time:85.3971 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:9206
2022-02-14 07:35:11 - SYSTEM STATS: Time:1809.1720 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:237766
2022-02-14 07:35:11 - SYSTEM STATS: Time:1508.7892 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:195728
2022-02-14 07:35:11 - SYSTEM STATS: Time:1208.6683 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:148606
2022-02-14 07:35:11 - SYSTEM STATS: Time:908.5960 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:107219
2022-02-14 07:35:11 - SYSTEM STATS: Time:608.0923 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:72404
2022-02-14 07:35:10 - SYSTEM STATS: Time:308.6889 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:37859
2022-02-14 06:55:22 - SYSTEM STATS: Time:1819.4318 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:244006
2022-02-14 06:55:21 - SYSTEM STATS: Time:1519.1600 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:201514
2022-02-14 06:55:21 - SYSTEM STATS: Time:1218.9484 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:158230
2022-02-14 06:55:21 - SYSTEM STATS: Time:918.7908 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:121441
2022-02-14 06:55:21 - SYSTEM STATS: Time:619.4013 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:85164
2022-02-14 06:55:19 - SYSTEM STATS: Time:317.1410 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:24044
2022-02-14 06:16:28 - SYSTEM STATS: Time:85.2553 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:50033
2022-02-14 06:11:28 - SYSTEM STATS: Time:85.0617 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:50001
2022-02-14 06:06:27 - SYSTEM STATS: Time:84.8586 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:5325
2022-02-14 06:05:14 - SYSTEM STATS: Time:1512.2130 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:264385
2022-02-14 06:05:14 - SYSTEM STATS: Time:1212.2879 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:170129
2022-02-14 06:05:14 - SYSTEM STATS: Time:912.0449 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:122869
2022-02-14 06:05:14 - SYSTEM STATS: Time:611.6144 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:82691
2022-02-14 06:05:14 - SYSTEM STATS: Time:311.5205 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:39011
2022-02-14 05:36:27 - SYSTEM STATS: Time:85.1007 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:15789
2022-02-14 05:35:09 - SYSTEM STATS: Time:607.1693 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:92384
2022-02-14 05:35:09 - SYSTEM STATS: Time:306.7381 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:36178
2022-02-14 05:25:08 - SYSTEM STATS: Time:306.1754 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:80687
2022-02-14 05:16:28 - SYSTEM STATS: Time:85.9134 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:8055
2022-02-14 05:15:13 - SYSTEM STATS: Time:311.4729 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145387 RRDsProcessed:95368

I tried to change processes value but the result didn't show much difference. Any suggestion to improve this?

[cigamit]

Check your disk I/O, I bet its maxed out on the server. You might try using "Enable On-demand RRA Updating" under the Performance settings? There are a few things you will need to do first, like make sure the webserver has write access to the RRAs.

[hakiki]

Hi cigamit,

Checked the I/O and I think it's quite low :

sar -b.png (36.97 KiB) Viewed 483 times

sar -d.png (114.6 KiB) Viewed 483 times

For the RRAs, the webserver already has access to them as I set chmod 775.
Now I try to enable On-demand RRA Updating and will monitor the result further.

Does database conversion and upgrade possibly affect this issue? Since I imported database dumped from an older system (cacti 0.8.8h ; mySQL 5.1.73), then perform conversion and upgrade (cacti 1.2.19 ; MariaDB 10.3.28).

[hakiki]

Update :
After enabling On-demand RRD Updating for roughly 1.5 hours, the poller seems not improving.
The poller is set to run every 5 minutes, and it looks like some 'processes' run overlapping over 5 minutes.

Code: Select all

2022-02-14 13:21:28 - SYSTEM STATS: Time:85.5205 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:0
2022-02-14 13:20:12 - SYSTEM STATS: Time:1210.3222 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:0
2022-02-14 13:20:12 - SYSTEM STATS: Time:910.2333 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:0
2022-02-14 13:20:12 - SYSTEM STATS: Time:610.1960 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:0
2022-02-14 13:20:12 - SYSTEM STATS: Time:310.0132 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:0
2022-02-14 12:56:27 - SYSTEM STATS: Time:85.2602 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:0
2022-02-14 12:55:13 - SYSTEM STATS: Time:1510.7286 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:263662
2022-02-14 12:55:13 - SYSTEM STATS: Time:1210.5603 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:167165
2022-02-14 12:55:12 - SYSTEM STATS: Time:910.5385 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:0
2022-02-14 12:55:12 - SYSTEM STATS: Time:610.2601 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:0
2022-02-14 12:55:12 - SYSTEM STATS: Time:310.0213 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:0
2022-02-14 12:26:27 - SYSTEM STATS: Time:84.7496 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:9033
2022-02-14 12:25:12 - SYSTEM STATS: Time:2110.1365 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:279934
2022-02-14 12:25:12 - SYSTEM STATS: Time:2410.0428 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:314118
2022-02-14 12:25:12 - SYSTEM STATS: Time:1809.9373 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:241721
2022-02-14 12:25:12 - SYSTEM STATS: Time:1509.6752 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:194273
2022-02-14 12:25:12 - SYSTEM STATS: Time:1209.4676 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:148804
2022-02-14 12:25:12 - SYSTEM STATS: Time:909.3986 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:116754
2022-02-14 12:25:12 - SYSTEM STATS: Time:610.0813 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:80393
2022-02-14 12:25:12 - SYSTEM STATS: Time:309.2323 Method:spine Processes:40 Threads:30 Hosts:2571 HostsPerProcess:65 DataSources:145389 RRDsProcessed:33919 

Also I cannot revert the "On-demand RRD Updating" status back to disabled even though I don't have remote poller.
The toggle stays at "enable" each time I disabled it and click save.

2022-02-14_15-30_001.png (38.59 KiB) Viewed 475 times