Maximum runtime of 298 seconds exceeded Problem!

[desscartes]

Hi Everybody,

I have a problem about poller time.
It is near to 300.There is error on log files.

Iwas added weathermap plugin and I create 4 maps.After that Ihad this problem. Is it reason for poller problem?

How can I solve this problem?

Code: Select all

SPINE: Poller[0] DEBUG: Net-SNMP Close Completed
01/13/2015 03:05:41 AM - SPINE: Poller[0] Time: 22.3190 s, Threads: 100, Hosts: 3
01/13/2015 03:00:00 PM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
01/13/2015 03:00:00 PM - SYSTEM STATS: Time:298.1891 Method:spine Processes:500 Threads:500 Hosts:839 HostsPerProcess:2 DataSources:8904 RRDsProcessed:5531
01/13/2015 03:00:00 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q /var/cacti/plugins/plugins/dsstats/poller_dsstats.php]
01/13/2015 03:00:00 PM - SYSTEM THOLD STATS: Time:0.1237 Tholds:0 DownHosts:0
01/13/2015 03:00:00 PM - POLLER: Poller[0] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q /var/cacti/plugins/nectar/poller_nectar.php]
01/13/2015 03:00:00 PM - WEATHERMAP: Poller[0] Weathermap 0.97a starting - DEBUG mode is on

[cigamit]

Processes:500 Threads:500

Holy crap. Change that to something more sensible. Change Processes to the number of processors on your system and threads to a number between 10 and 25.

[desscartes]

Thanks for reply,

I have to change Maximum Concurrent Poller Processes, Am I right?

[desscartes]

Solved!
thanks a lot for your help cigamit.

Code: Select all

SYSTEM STATS: Time:298.2168 Method:spine Processes:500 Threads:500 Hosts:839 HostsPerProcess:2 DataSources:8908 RRDsProcessed:1000
01/14/2015 09:30:28 AM - SYSTEM STATS: Time:26.9255 Method:spine Processes:20 Threads:500 Hosts:839 HostsPerProcess:42 DataSources:8908 RRDsProcessed:5699
01/14/2015 09:35:27 AM - SYSTEM STATS: Time:26.1630 Method:spine Processes:20 Threads:4 Hosts:839 HostsPerProcess:42 DataSources:8912 RRDsProcessed:5701

[alexvelitsky]

Hi guys!

I have:

MariaDB [cacti]> select * from version;
+--------+
| cacti |
+--------+
| 1.2.25 |
+--------+
1 row in set (0.001 sec)

MariaDB [cacti]>

on debian. And I see same logs:

2023-12-18 05:40:08 - POLLER: Poller[1] PID[3293072] Maximum runtime of 298 seconds exceeded. Exiting.
2023-12-18 05:45:08 - POLLER: Poller[1] PID[3304537] Maximum runtime of 298 seconds exceeded. Exiting.
2023-12-18 05:50:07 - POLLER: Poller[1] PID[3314533] Maximum runtime of 298 seconds exceeded. Exiting.
2023-12-18 05:55:09 - POLLER: Poller[1] PID[3324585] Maximum runtime of 298 seconds exceeded. Exiting.
2023-12-18 06:00:07 - POLLER: Poller[1] PID[3335473] Maximum runtime of 298 seconds exceeded. Exiting.
2023-12-18 06:05:07 - POLLER: Poller[1] PID[3353622] Maximum runtime of 298 seconds exceeded. Exiting.
2023-12-18 06:10:07 - POLLER: Poller[1] PID[3372194] Maximum runtime of 298 seconds exceeded. Exiting.
2023-12-18 06:15:06 - POLLER: Poller[1] PID[3387593] Maximum runtime of 298 seconds exceeded. Exiting.
2023-12-18 06:30:07 - POLLER: Poller[1] PID[883] Maximum runtime of 298 seconds exceeded. Exiting.
2023-12-18 06:40:06 - POLLER: Poller[1] PID[3579] Maximum runtime of 298 seconds exceeded. Exiting.
2023-12-18 06:45:06 - POLLER: Poller[1] PID[6950] Maximum runtime of 298 seconds exceeded. Exiting.
2023-12-18 06:45:06 - POLLER: Poller[1] PID[10573] Maximum runtime of 298 seconds exceeded. Exiting.
2023-12-18 06:55:07 - POLLER: Poller[1] PID[13989] Maximum runtime of 298 seconds exceeded. Exiting.
2023-12-18 07:00:06 - POLLER: Poller[1] PID[17534] Maximum runtime of 298 seconds exceeded. Exiting.
2023-12-18 07:05:05 - POLLER: Poller[1] PID[23591] Maximum runtime of 298 seconds exceeded. Exiting.
2023-12-18 07:05:05 - POLLER: Poller[1] PID[20595] Maximum runtime of 298 seconds exceeded. Exiting.

But didn't find "maximum concurrent poller proccess" in my cacti 1.2.25 settings. Pls help me to solve this issue.
Thanks!

P.S. I don't use spine.

[Rno]

Can you post the SYSTEM STAT line from the log

[alexvelitsky]

Can you post the SYSTEM STAT line from the log

What do you mean with SYSTEM STAT line from the log? Maybe I understood you correctly?

2023-12-18 08:10:06 - SYSTEM STATS: Time:600.9354 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 08:10:06 - SYSTEM STATS: Time:300.7104 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 08:20:06 - SYSTEM STATS: Time:600.6947 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 08:25:06 - SYSTEM STATS: Time:600.7908 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 08:25:06 - SYSTEM STATS: Time:300.6844 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 08:35:06 - SYSTEM STATS: Time:300.4846 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 08:35:06 - SYSTEM STATS: Time:600.7222 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 08:50:06 - SYSTEM STATS: Time:600.6478 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 08:50:08 - SYSTEM STATS: Time:901.3146 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:4
2023-12-18 09:00:06 - SYSTEM STATS: Time:600.7826 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 09:00:07 - SYSTEM STATS: Time:301.5527 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 09:00:08 - SYSTEM STATS: Time:901.2994 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:13
2023-12-18 09:10:06 - SYSTEM STATS: Time:600.6909 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 09:15:15 - SYSTEM STATS: Time:309.0471 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 09:15:15 - SYSTEM STATS: Time:609.2822 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 09:20:15 - SYSTEM STATS: Time:603.6155 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 09:25:06 - SYSTEM STATS: Time:604.5571 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 09:30:07 - SYSTEM STATS: Time:605.5926 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 09:35:06 - SYSTEM STATS: Time:604.6677 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 09:40:07 - SYSTEM STATS: Time:605.7764 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 09:45:08 - SYSTEM STATS: Time:605.8638 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 09:50:06 - SYSTEM STATS: Time:603.8442 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 09:55:07 - SYSTEM STATS: Time:605.7222 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 10:00:06 - SYSTEM STATS: Time:604.5947 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 10:05:07 - SYSTEM STATS: Time:605.7280 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 10:10:07 - SYSTEM STATS: Time:605.8879 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 10:15:05 - SYSTEM STATS: Time:603.8526 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 10:20:02 - SYSTEM STATS: Time:600.6870 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 10:25:07 - SYSTEM STATS: Time:606.0020 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 10:30:06 - SYSTEM STATS: Time:604.3919 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 10:35:05 - SYSTEM STATS: Time:604.0022 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 10:40:11 - SYSTEM STATS: Time:609.1444 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0
2023-12-18 10:45:12 - SYSTEM STATS: Time:609.7124 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:85 RRDsProcessed:0

[Rno]

Yes that what I was looking for.

So 4 hosts, 85 DataSource is very low!

How your 4 host are responding to a ping, or a snmpwalk
something like that:
snmpwalk -v2c -c public hostname 1.3.6.1.2.1.1.3.0

adapt the -c public to your community

and do the check on all 4 hosts, it will give you the uptime.

[alexvelitsky]

Yes that what I was looking for.

So 4 hosts, 85 DataSource is very low!

How your 4 host are responding to a ping, or a snmpwalk
something like that:
snmpwalk -v2c -c public hostname 1.3.6.1.2.1.1.3.0

adapt the -c public to your community

and do the check on all 4 hosts, it will give you the uptime.

There's no connectivity problem:

:~$ ping 185.225.203.164
PING 185.225.203.164 (185.225.203.164) 56(84) bytes of data.
64 bytes from 185.225.203.164: icmp_seq=1 ttl=56 time=15.8 ms
64 bytes from 185.225.203.164: icmp_seq=2 ttl=56 time=14.9 ms
64 bytes from 185.225.203.164: icmp_seq=3 ttl=56 time=16.1 ms
64 bytes from 185.225.203.164: icmp_seq=4 ttl=56 time=14.9 ms
64 bytes from 185.225.203.164: icmp_seq=5 ttl=56 time=16.7 ms

--- 185.225.203.164 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 14.865/15.683/16.749/0.710 ms

also there's no snmpwalk reply delay problem.

P.S. All my 4 hosts have the same public IP 185.225.203.164.
P.P.S. htop output from my server:

CPU[*** 2.6%] Tasks: 66, 82 thr, 59 kthr; 1 running
Mem[|||||||||||||||||||##**@@@@@@@@@@@@@ 600M/2.85G] Load average: 0.19 0.30 0.40
Swp[ 0K/2.93G] Uptime: 00:13:21

[Main] [I/O]
PID USER PRI NI VIRT RES SHR S CPU%▽MEM% TIME+ Command
1449 vel 20 0 18060 6536 4684 S 0.7 0.2 0:00.11 sshd: vel@pts/0
1461 vel 20 0 8608 4840 3368 R 0.7 0.2 0:00.70 htop
1 root 20 0 100M 12700 8972 S 0.0 0.4 0:01.86 /sbin/init
221 root 20 0 82340 38828 37600 S 0.0 1.3 0:00.76 /lib/systemd/systemd-journald
251 root 20 0 26660 6864 4660 S 0.0 0.2 0:00.31 /lib/systemd/systemd-udevd
457 avahi 20 0 8664 4684 3864 S 0.0 0.2 0:01.13 avahi-daemon: running [debian.local]
458 vel 20 0 224M 134M 14080 S 0.0 4.6 0:03.06 /home/vel/bot/venv/bin/python referendum.py
459 root 20 0 6628 2732 2488 S 0.0 0.1 0:00.01 /usr/sbin/cron -f
460 messagebus 20 0 9812 5216 4196 S 0.0 0.2 0:00.81 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
463 root 20 0 302M 8912 6708 S 0.0 0.3 0:00.29 /usr/lib/policykit-1/polkitd --no-debug
464 root 20 0 222M 10596 6452 S 0.0 0.4 0:00.33 /usr/sbin/rsyslogd -n -iNONE
465 root 20 0 50116 8388 7228 S 0.0 0.3 0:00.18 /lib/systemd/systemd-logind
467 avahi 20 0 8120 356 0 S 0.0 0.0 0:00.00 avahi-daemon: chroot helper
474 root 20 0 302M 8912 6708 S 0.0 0.3 0:00.00 /usr/lib/policykit-1/polkitd --no-debug
475 root 20 0 302M 8912 6708 S 0.0 0.3 0:00.00 /usr/lib/policykit-1/polkitd --no-debug
480 root 20 0 302M 8912 6708 S 0.0 0.3 0:00.07 /usr/lib/policykit-1/polkitd --no-debug
482 root 20 0 324M 18656 16164 S 0.0 0.6 0:00.20 /usr/sbin/NetworkManager --no-daemon
483 root 20 0 16568 5772 4904 S 0.0 0.2 0:00.01 /sbin/wpa_supplicant -u -s -O DIR=/run/wpa_supplicant GROUP=netdev
484 root 20 0 222M 10596 6452 S 0.0 0.4 0:00.04 /usr/sbin/rsyslogd -n -iNONE
485 root 20 0 222M 10596 6452 S 0.0 0.4 0:00.00 /usr/sbin/rsyslogd -n -iNONE
486 root 20 0 222M 10596 6452 S 0.0 0.4 0:00.12 /usr/sbin/rsyslogd -n -iNONE
487 root 20 0 310M 11932 10164 S 0.0 0.4 0:00.17 /usr/sbin/ModemManager
494 root 20 0 310M 11932 10164 S 0.0 0.4 0:00.00 /usr/sbin/ModemManager
495 root 20 0 310M 11932 10164 S 0.0 0.4 0:00.00 /usr/sbin/ModemManager
497 root 20 0 310M 11932 10164 S 0.0 0.4 0:00.01 /usr/sbin/ModemManager
498 root 20 0 324M 18656 16164 S 0.0 0.6 0:00.01 /usr/sbin/NetworkManager --no-daemon
499 root 20 0 324M 18656 16164 S 0.0 0.6 0:00.00 /usr/sbin/NetworkManager --no-daemon
500 root 20 0 324M 18656 16164 S 0.0 0.6 0:00.03 /usr/sbin/NetworkManager --no-daemon
503 root 20 0 28504 10316 7852 S 0.0 0.3 0:00.29 /usr/sbin/cupsd -l
509 root 20 0 10216 4356 3588 S 0.0 0.1 0:00.00 /usr/sbin/vsftpd /etc/vsftpd.conf
512 root 20 0 373M 7348 6556 S 0.0 0.2 0:00.03 /usr/sbin/lightdm
537 root 20 0 373M 7348 6556 S 0.0 0.2 0:00.00 /usr/sbin/lightdm
538 root 20 0 373M 7348 6556 S 0.0 0.2 0:00.00 /usr/sbin/lightdm
541 root 20 0 373M 7348 6556 S 0.0 0.2 0:00.00 /usr/sbin/lightdm
559 root 20 0 528M 93992 64588 S 0.0 3.1 0:01.04 /usr/lib/xorg/Xorg :0 -seat seat0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
560 root 20 0 5900 1060 972 S 0.0 0.0 0:00.00 /sbin/agetty -o -p -- \u --noclear - linux
561 root 20 0 15480 9288 7996 S 0.0 0.3 0:00.03 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
613 mysql 20 0 1233M 121M 18720 S 0.0 4.2 0:03.82 /usr/sbin/mysqld
643 root 20 0 247M 16420 12824 S 0.0 0.5 0:00.66 /usr/sbin/cups-browsed
645 Debian-snm 20 0 24228 13280 9984 S 0.0 0.4 0:00.46 /usr/sbin/snmpd -LOw -u Debian-snmp -g Debian-snmp -I -smux mteTrigger mteTriggerConf -f
660 ntpsec RT 0 84920 19280 8876 S 0.0 0.6 0:00.24 /usr/sbin/ntpd -p /run/ntpd.pid -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec
675 root 20 0 4676 280 0 S 0.0 0.0 0:00.00 /usr/sbin/in.tftpd --listen --user tftp --address 0.0.0.0:69 --secure -4 -c /srv/tftp
680 root 20 0 7568 360 0 S 0.0 0.0 0:00.00 /usr/sbin/tac_plus -C /etc/tacacs+/tac_plus.conf
697 root 20 0 225M 33900 24276 S 0.0 1.1 0:00.16 /usr/sbin/apache2 -k start
710 mysql 20 0 1233M 121M 18720 S 0.0 4.2 0:00.00 /usr/sbin/mysqld
716 mysql 20 0 1233M 121M 18720 S 0.0 4.2 0:00.00 /usr/sbin/mysqld
721 root 20 0 247M 16420 12824 S 0.0 0.5 0:00.00 /usr/sbin/cups-browsed
722 root 20 0 247M 16420 12824 S 0.0 0.5 0:00.00 /usr/sbin/cups-browsed
723 root 20 0 247M 16420 12824 S 0.0 0.5 0:00.01 /usr/sbin/cups-browsed

[Rno]

Can you explain to me how you handle this:
All my 4 hosts have the same public IP 185.225.203.164

Did you use some PAT to make the access to different server ?
On cacti did you change the port on SNMP ?

And that mean about the answer of the ping, I'm not sure it come from one of the host or the device making the translation, but not all four would answer.

[alexvelitsky]

Can you explain to me how you handle this:
All my 4 hosts have the same public IP 185.225.203.164

Did you use some PAT to make the access to different server ?
On cacti did you change the port on SNMP ?

And that mean about the answer of the ping, I'm not sure it come from one of the host or the device making the translation, but not all four would answer.

I mean that one of devices - is a border router (185.225.203.164) with DNAT, and all other 3 devices are located after this DNAT in the LAN, which is directly connected to border router.

So, as the result: All my 4 hosts have the same public IP 185.225.203.164

> On cacti did you change the port on SNMP?

Do you think this can help?

[alexvelitsky]

Yeah, it seems to be this is a connectivity problem:

2023-12-17 16:05:05 - SYSTEM STATS: Time:3.6742 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:79 RRDsProcessed:52
2023-12-17 16:10:05 - SYSTEM STATS: Time:3.6561 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:79 RRDsProcessed:52
2023-12-17 16:15:11 - SYSTEM STATS: Time:7.8263 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:79 RRDsProcessed:52
2023-12-17 16:20:05 - SYSTEM STATS: Time:3.6156 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:79 RRDsProcessed:52
2023-12-17 16:35:09 - SYSTEM STATS: Time:300.6300 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:79 RRDsProcessed:0
2023-12-17 16:45:06 - SYSTEM STATS: Time:300.4598 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:79 RRDsProcessed:0
2023-12-17 16:45:12 - SYSTEM STATS: Time:604.4611 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:79 RRDsProcessed:0
2023-12-17 16:50:08 - SYSTEM STATS: Time:300.6279 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:79 RRDsProcessed:0

and so on approximately 300/600 ms. I changed SNMP port for cacti server - didn't help.
Is it possible to change runtime limit for Method:cmd.php manually in cacti version 1.2.25? Thanks!

[Rno]

Somehow you need to be able to let cacti reach you 4 routers with the same IP with something who is unique per router.
Usually it's the port that let you make the difference. That's why I was thinking about a change of the snmp port.
Otherwise you can't access the 4 routers with same IP, same port, or I'm missing something on the device that made the connection in front of those routers.

About the change of the runtime of cmd.php it's based on the poller interval and the cron job. Actually you have 5minutes cron job, so the poller has to end before this 5 minutes.
But did you have any graph that are working ?
Are they really different from one another ?

I have some doubt that cacti isn't actually reaching the same host 3 times, not 3 different hosts. Look at the In state value, it's the same, and the uptime within a few minutes, and everything else is the same value, way to close to be on different system.

How did you reach the 3 hosts for management ? did you use SSH and you can reach each one of it ?

[TheWitness]

500 threads is ridiculous. I'm polling well over 10k devices with 5 processes and 50 threads and well over 1M data sources in less than 70 seconds.

[alexvelitsky]

Somehow you need to be able to let cacti reach you 4 routers with the same IP with something who is unique per router.
Usually it's the port that let you make the difference. That's why I was thinking about a change of the snmp port.
Otherwise you can't access the 4 routers with same IP, same port, or I'm missing something on the device that made the connection in front of those routers.

About the change of the runtime of cmd.php it's based on the poller interval and the cron job. Actually you have 5minutes cron job, so the poller has to end before this 5 minutes.
But did you have any graph that are working ?
Are they really different from one another ?

I have some doubt that cacti isn't actually reaching the same host 3 times, not 3 different hosts. Look at the In state value, it's the same, and the uptime within a few minutes, and everything else is the same value, way to close to be on different system.

How did you reach the 3 hosts for management ? did you use SSH and you can reach each one of it ?

I use one same IP but 4 different SNMP ports for monitoring. By the way, problem was solved by itself. It seems to be that was some strange problem with long reply time:

2023-12-19 02:15:06 - SYSTEM STATS: Time:604.6793 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:87 RRDsProcessed:0
2023-12-19 02:20:07 - SYSTEM STATS: Time:605.8844 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:87 RRDsProcessed:0
2023-12-19 02:25:06 - SYSTEM STATS: Time:604.5697 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:87 RRDsProcessed:0
2023-12-19 02:30:05 - SYSTEM STATS: Time:604.1221 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:87 RRDsProcessed:0
2023-12-19 02:35:05 - SYSTEM STATS: Time:603.8186 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:87 RRDsProcessed:0
2023-12-19 02:40:05 - SYSTEM STATS: Time:3.4722 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:87 RRDsProcessed:0
2023-12-19 02:40:05 - SYSTEM STATS: Time:603.9674 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:87 RRDsProcessed:0
2023-12-19 02:45:05 - SYSTEM STATS: Time:604.1199 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:87 RRDsProcessed:0
2023-12-19 02:45:06 - SYSTEM STATS: Time:4.4760 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:87 RRDsProcessed:0
2023-12-19 02:50:06 - SYSTEM STATS: Time:4.5237 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:87 RRDsProcessed:0
2023-12-19 02:55:06 - SYSTEM STATS: Time:4.4852 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:87 RRDsProcessed:0
2023-12-19 03:00:06 - SYSTEM STATS: Time:4.4931 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:87 RRDsProcessed:0
2023-12-19 03:05:05 - SYSTEM STATS: Time:4.4863 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:87 RRDsProcessed:0
2023-12-19 03:10:06 - SYSTEM STATS: Time:3.5026 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:87 RRDsProcessed:0
2023-12-19 03:15:06 - SYSTEM STATS: Time:4.5933 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:87 RRDsProcessed:0
2023-12-19 03:20:06 - SYSTEM STATS: Time:4.4933 Method:cmd.php Processes:1 Threads:1 Hosts:4 HostsPerProcess:4 DataSources:87 RRDsProcessed:0

And there's no issue with all four graphs.