Cacti poller runs very slow

[hakiki]

I installed cacti version 1.2.27 with spine on an RHEL environment, then migrated data from other system with older cacti version (0.8.8h). Then I found out that the poller on new system runs very slow that it exceeds 298 seconds every polling cycle.

Here's steps I did:
1. Install and configure cacti + spine ver 1.2.27 on new system (RHEL 8.10)
2. restore mysql dump + migrate RRD files from older system (cacti + spine ver 0.8.8h, centos 6) to new system

Here's logs from the new system

Code: Select all

2024-10-17 18:35:02 - SYSTEM MAINT STATS: Time:0.01
2024-10-17 18:35:00 - SYSTEM STATS: Time:299.2033 Method:spine Processes:40 Threads:30 Hosts:1643 HostsPerProcess:42 DataSources:120586 RRDsProcessed:0
2024-10-17 18:35:00 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-10-17 18:35:00 - POLLER: Poller[1] PID[3319179] Maximum runtime of 298 seconds exceeded. Exiting.
2024-10-17 18:30:01 - SYSTEM MAINT STATS: Time:0.00
2024-10-17 18:30:00 - SYSTEM STATS: Time:298.2398 Method:spine Processes:40 Threads:30 Hosts:1643 HostsPerProcess:42 DataSources:120586 RRDsProcessed:0
2024-10-17 18:30:00 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-10-17 18:30:00 - POLLER: Poller[1] PID[3318372] Maximum runtime of 298 seconds exceeded. Exiting.

I use crontab to run the poller

Code: Select all

[root@mrtg-backbone ~]# cat /etc/cron.d/cacti
*/5 * * * * apache php /var/www/html/cacti/poller.php &>/dev/null

Strangely, when I run spine manually, it tooks only 33 seconds to run the poller:

Code: Select all

[root@mrtg-backbone bin]# ./spine -R -V 3 -S
SPINE: Using spine config file [/etc/spine.conf]
Version 1.2.27 starting
Total[0.0236] No Device 0 Poller Items found.
Total[0.1256] Device[4001] INFO: SNMP Device Alive, Time:0.0042 ms
Total[0.1269] Device[376] INFO: SNMP Device Alive, Time:0.0125 ms
Total[0.1290] Device[4001] HT[1] DQ[14] Legacy Uptime Result: 1163402330, Is Numeric: 1
Total[0.1315] Device[4001] HT[1] DQ[14] Extended Uptime Result: 1163402400, Is Numeric: 1
Total[0.1315] Device[4001] HT[1] DQ[14] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 1151623100 < output: 1163402400)
Total[0.1317] Device[244] INFO: SNMP Device Alive, Time:0.0129 ms
Total[0.1372] Device[376] HT[1] DQ[14] Legacy Uptime Result: 3202806046, Is Numeric: 1
Total[0.1374] Device[396] INFO: SNMP Device Alive, Time:0.0131 ms
-----------
Total[32.9088] Device[2137] HT[1] DQ[14] Legacy Uptime Result: U, Is Numeric: 0
Total[32.9088] Device[2137] HT[1] DQ[14] Extended Uptime Result: , Is Numeric: 0
Total[32.9089] Device[2137] HT[1] DQ[14] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert:  < output: )
Total[32.9098] Device[2137] HT[1] Total Time: 8 Seconds
Total[33.3404] Time: 33.3403 s, Threads: 30, Devices: 1643

[macan]

You did install 1.2.x and restore db from old version? So you have now php code 1.2.x and database 0.88? It is wrong.

1) Copy you old files include rra to new server
2) Import mysqldump from old version
= so you have now old cacti on new server
3) replace files from 1.2.x code
4) use new include/config.php, change here mysql user, password, ...
5) follow wizard

[hakiki]

Oops sorry I think I missed some explanations on my steps:

Here what I did previously:
1. Install cacti 1.2.27 + spine on new system
2. Copy rrd files and restore mysql dump to the new system
3. Continue cacti installation wizard (the setup was directed to the upgrade wizard, which includes template upgrade and database conversion)
4. Finish the installation (upgrade) wizard.

Besides of different cacti version, there are some major version jumps between old vs new eviroment:
- OS: centos 6 vs RHEL 8.10
- DB: mysql 5.1 vs mariadb 10.3
- PHP: 5.3 vs 7.2

Also yesterday I realized that spine.conf didn't have correct permission set. I fixed that so now RRD files are updated but still with very long polling time:

Code: Select all

2024-10-18 15:40:39 - SYSTEM MAINT STATS: Time:0.01
2024-10-18 15:40:38 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-10-18 15:40:38 - SYSTEM STATS: Time:5136.1373 Method:spine Processes:40 Threads:30 Hosts:1643 HostsPerProcess:42 DataSources:120586 RRDsProcessed:98663
2024-10-18 15:40:38 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-10-18 15:40:38 - POLLER: Poller[1] PID[3978476] Maximum runtime of 298 seconds exceeded. Exiting.
2024-10-18 15:40:01 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-10-18 15:40:01 - POLLER: Poller[1] PID[4028343] WARNING: Poller Output Table not Empty. Issues: 16, DS[50232, 50232, 50233, 50233, 50234, 50234, 50235, 50235, 50236, 50236, 50237, 50237, 50238, 50238, 93204, 93204]
2024-10-18 15:37:57 - SYSTEM MAINT STATS: Time:0.01
2024-10-18 15:37:55 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-10-18 15:37:55 - SYSTEM STATS: Time:3173.0291 Method:spine Processes:40 Threads:30 Hosts:1643 HostsPerProcess:42 DataSources:120586 RRDsProcessed:60335
2024-10-18 15:37:55 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-10-18 15:37:55 - POLLER: Poller[1] PID[3996036] Maximum runtime of 298 seconds exceeded. Exiting.
2024-10-18 15:35:02 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-10-18 15:35:02 - POLLER: Poller[1] PID[4025437] WARNING: Poller Output Table not Empty. Issues: 36645, DS[164976, 164976, 164977, 164977, 164978, 164978, 164979, 164979, 165024, 165024, 165025, 165025, 165026, 165026, 165027, 165027, 165028, 165028, 165029, 165029], Additional Issues Remain. Only showing first 20
2024-10-18 15:33:42 - SYSTEM MAINT STATS: Time:0.01
2024-10-18 15:33:41 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-10-18 15:33:41 - SYSTEM STATS: Time:2018.8886 Method:spine Processes:40 Threads:30 Hosts:1643 HostsPerProcess:42 DataSources:120586 RRDsProcessed:38645
2024-10-18 15:33:41 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-10-18 15:33:41 - POLLER: Poller[1] PID[4005106] Maximum runtime of 298 seconds exceeded. Exiting.
2024-10-18 15:24:06 - SYSTEM MAINT STATS: Time:0.00
2024-10-18 15:24:05 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-10-18 15:24:05 - SYSTEM STATS: Time:2042.6308 Method:spine Processes:40 Threads:30 Hosts:1643 HostsPerProcess:42 DataSources:120586 RRDsProcessed:38645
2024-10-18 15:24:05 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-10-18 15:24:05 - POLLER: Poller[1] PID[3998836] Maximum runtime of 298 seconds exceeded. Exiting.
2024-10-18 15:23:33 - CMDPHP Validation Error, Variable:id, Value:100290, 100290 Backtrace: (/data_sources.php[87]:ds_edit(), /data_sources.php[774]:get_filter_request_var(), /lib/html_utility.php[493]:die_html_input_error(), /lib/html_validate.php[64]:cacti_debug_backtrace())
2024-10-18 15:23:30 - CMDPHP Validation Error, Variable:id, Value:80808, 80808 Backtrace: (/data_sources.php[87]:ds_edit(), /data_sources.php[774]:get_filter_request_var(), /lib/html_utility.php[493]:die_html_input_error(), /lib/html_validate.php[64]:cacti_debug_backtrace())
2024-10-18 15:23:13 - CMDPHP Validation Error, Variable:id, Value:131488, 131488 Backtrace: (/data_sources.php[87]:ds_edit(), /data_sources.php[774]:get_filter_request_var(), /lib/html_utility.php[493]:die_html_input_error(), /lib/html_validate.php[64]:cacti_debug_backtrace())
2024-10-18 15:20:58 - SYSTEM MAINT STATS: Time:0.01
2024-10-18 15:20:57 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-10-18 15:20:57 - SYSTEM STATS: Time:6354.2747 Method:spine Processes:40 Threads:30 Hosts:1643 HostsPerProcess:42 DataSources:120586 RRDsProcessed:118673

[macan]

Few ideas:
chmod +s /path/to/spine - for icmp ping
is selinux or apparmor used?
DNS is working well?
Enable boost
try rebuild poller cache
try change process/threads of spine

[slademan]

I had case of cacti slowness where I defined that mysql on new server had permissions to listen database on old server. Everything was very slow, even browsing through GUI. I removed those permissions and it stopped contacting old servers database. It worked normally after that.
So just as an idea to your case.