Cacti/Spine 1.2.18 poller problem

[eddi]

Hi, I have just finished installing and configuring on CentOS 7, Cacti/Spine 1.2.18 main and 1 remote collector then configured few hosts and graphs on the remote collector. The graphs of the last host I added are not being graphed.
Log entry on the remote collector which is configured to run every minute:
SYSTEM STATS: Time:58.8471 Method:spine Processes:8 Threads:32 Hosts:5 HostsPerProcess:1 DataSources:33 RRDsProcessed:0

php -q /var/www/html/cacti/poller.php --debug
2021-09-07 15:27:53 - POLLER: Poller[2] PID[18260] NOTE: Poller Int: '60', Cron Int: '60', Time Since Last: '50.85', Max Runtime '58', Poller Runs: '1'
2021-09-07 15:27:53 - POLLER: Poller[2] PID[18260] DEBUG: About to Spawn a Remote Process [CMD: /usr/local/spine/bin/spine, ARGS: -C '/usr/local/spine/etc/spine.conf' --mode=online --poller=2 --first=4 --last=4 --mibs]
2021-09-07 15:27:53 - POLLER: Poller[2] PID[18260] DEBUG: About to Spawn a Remote Process [CMD: /usr/local/spine/bin/spine, ARGS: -C '/usr/local/spine/etc/spine.conf' --mode=online --poller=2 --first=5 --last=5 --mibs]
2021-09-07 15:27:53 - POLLER: Poller[2] PID[18260] DEBUG: About to Spawn a Remote Process [CMD: /usr/local/spine/bin/spine, ARGS: -C '/usr/local/spine/etc/spine.conf' --mode=online --poller=2 --first=6 --last=6 --mibs]
2021-09-07 15:27:53 - POLLER: Poller[2] PID[18260] DEBUG: About to Spawn a Remote Process [CMD: /usr/local/spine/bin/spine, ARGS: -C '/usr/local/spine/etc/spine.conf' --mode=online --poller=2 --first=7 --last=7 --mibs]
2021-09-07 15:27:53 - POLLER: Poller[2] PID[18260] DEBUG: About to Spawn a Remote Process [CMD: /usr/local/spine/bin/spine, ARGS: -C '/usr/local/spine/etc/spine.conf' --mode=online --poller=2 --first=8 --last=8 --mibs]
Waiting on 5 of 5 pollers.
Waiting on 5 of 5 pollers.
Waiting on 1 of 5 pollers.
Waiting on 1 of 5 pollers.
Waiting on 1 of 5 pollers.
Waiting on 1 of 5 pollers.
.
.
.
Waiting on 1 of 5 pollers.
Waiting on 1 of 5 pollers.
2021-09-07 15:28:51 - POLLER: Poller[2] PID[18260] Maximum runtime of 58 seconds exceeded. Exiting.
2021-09-07 15:28:51 - SYSTEM STATS: Time:57.8189 Method:spine Processes:8 Threads:32 Hosts:5 HostsPerProcess:1 DataSources:33 RRDsProcessed:0
2021-09-07 15:28:51 - POLLER: Poller[2] PID[18260] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q '/var/www/html/cacti/poller_automation.php' -M]
2021-09-07 15:28:51 - POLLER: Poller[2] PID[18260] DEBUG: About to Spawn a Remote Process [CMD: /usr/bin/php, ARGS: -q '/var/www/html/cacti/poller_maintenance.php']

/usr/local/spine/bin/spine -p 2 -R -S -V 4 -C /usr/local/spine/etc/spine.conf
SPINE: Using spine config file [/usr/local/spine/etc/spine.conf]
Version 1.2.18 starting
Sending entries to remote database in 'online' mode
Total[0.3881] Spine will support multithread device polling.
Total[0.3889] DEBUG: Initial Value of Available Threads is 32 (0 outstanding)
Total[0.3894] DEBUG: Available Threads is 31 (1 outstanding)
Total[0.3899] DEBUG: Available Threads is 30 (2 outstanding)
Total[0.3903] DEBUG: Available Threads is 29 (3 outstanding)
Total[0.3906] Device[4] No Device Availability Method Selected
Total[0.3907] DEBUG: Available Threads is 28 (4 outstanding)
Total[0.3910] Device[7] INFO: SNMP Device 'z-ilo-3' has timeout 500000 (500), retries 3
Total[0.3912] Device[4] HT[1] Device has no information for recache.
Total[0.3912] Device[8] INFO: SNMP Device 'z-ilo-4' has timeout 500000 (500), retries 3
Total[0.3912] DEBUG: Available Threads is 27 (5 outstanding)
Total[0.3912] WARNING: Spine Sleeping While Waiting for 5 Threads to End
Total[0.3917] Device[5] INFO: SNMP Device 'z-ilo-1' has timeout 500000 (500), retries 3
Total[0.3923] Device[6] INFO: SNMP Device 'z-ilo-2' has timeout 500000 (500), retries 3
Total[0.3930] Device[7] No Device Availability Method Selected
Total[0.3933] Device[4] HT[1] NOTE: There are '9' Polling Items for this Device
Total[0.3937] Device[8] No Device Availability Method Selected
Total[0.3944] Device[5] No Device Availability Method Selected
Total[0.3950] Device[6] No Device Availability Method Selected
Total[0.3951] Device[7] HT[1] DQ[10] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 4187982649 < output: 4187983273)
Total[0.3964] Device[8] HT[1] DQ[10] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 4187743164 < output: 4187983573)
Total[0.3964] Device[5] HT[1] DQ[10] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 4187982926 < output: 4187983573)
Total[0.3968] Device[6] HT[1] DQ[10] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 4187982634 < output: 4187983273)
Total[0.3971] Device[7] HT[1] NOTE: There are '6' Polling Items for this Device
Total[0.3971] Device[7] INFO: SNMP Device 'z-ilo-3' has timeout 500000 (500), retries 3
Total[0.3976] Device[5] HT[1] NOTE: There are '6' Polling Items for this Device
Total[0.3976] Device[5] INFO: SNMP Device 'z-ilo-1' has timeout 500000 (500), retries 3
Total[0.3984] Device[8] HT[1] NOTE: There are '6' Polling Items for this Device
Total[0.3984] Device[8] INFO: SNMP Device 'z-ilo-4' has timeout 500000 (500), retries 3
Total[0.3988] Device[6] HT[1] NOTE: There are '6' Polling Items for this Device
Total[0.3988] Device[6] INFO: SNMP Device 'z-ilo-2' has timeout 500000 (500), retries 3
Total[0.3992] Device[7] HT[1] DS[34] TT[1.41] SNMP: v2: z-ilo-3, dsname: temp_celcius, oid: .1.3.6.1.4.1.232.6.2.6.8.1.4.0.1, value: 20
Total[0.3993] Device[7] HT[1] DS[34] TT[1.44] SNMP: v2: z-ilo-3, dsname: temp_condition, oid: .1.3.6.1.4.1.232.6.2.6.8.1.6.0.1, value: 2
Total[0.3993] Device[7] HT[1] DS[34] TT[1.45] SNMP: v2: z-ilo-3, dsname: temp_threshold, oid: .1.3.6.1.4.1.232.6.2.6.8.1.5.0.1, value: 42
Total[0.3993] Device[7] HT[1] DS[35] TT[1.47] SNMP: v2: z-ilo-3, dsname: temp_celcius, oid: .1.3.6.1.4.1.232.6.2.6.8.1.4.0.15, value: 23
Total[0.3993] Device[7] HT[1] DS[35] TT[1.48] SNMP: v2: z-ilo-3, dsname: temp_condition, oid: .1.3.6.1.4.1.232.6.2.6.8.1.6.0.15, value: 2
Total[0.3993] Device[7] HT[1] DS[35] TT[1.49] SNMP: v2: z-ilo-3, dsname: temp_threshold, oid: .1.3.6.1.4.1.232.6.2.6.8.1.5.0.15, value: 70
Total[0.3999] Device[7] HT[1] Updating Poller Items for Next Poll
Total[0.4001] Device[5] HT[1] DS[30] TT[1.83] SNMP: v2: z-ilo-1, dsname: temp_celcius, oid: .1.3.6.1.4.1.232.6.2.6.8.1.4.0.1, value: 20
Total[0.4002] Device[5] HT[1] DS[30] TT[1.85] SNMP: v2: z-ilo-1, dsname: temp_condition, oid: .1.3.6.1.4.1.232.6.2.6.8.1.6.0.1, value: 2
Total[0.4002] Device[5] HT[1] DS[30] TT[1.86] SNMP: v2: z-ilo-1, dsname: temp_threshold, oid: .1.3.6.1.4.1.232.6.2.6.8.1.5.0.1, value: 42
Total[0.4002] Device[5] HT[1] DS[31] TT[1.87] SNMP: v2: z-ilo-1, dsname: temp_celcius, oid: .1.3.6.1.4.1.232.6.2.6.8.1.4.0.15, value: 24
Total[0.4002] Device[5] HT[1] DS[31] TT[1.88] SNMP: v2: z-ilo-1, dsname: temp_condition, oid: .1.3.6.1.4.1.232.6.2.6.8.1.6.0.15, value: 2
Total[0.4002] Device[5] HT[1] DS[31] TT[1.88] SNMP: v2: z-ilo-1, dsname: temp_threshold, oid: .1.3.6.1.4.1.232.6.2.6.8.1.5.0.15, value: 70
Total[0.4005] Device[5] HT[1] Updating Poller Items for Next Poll
Total[0.4005] Device[7] HT[1] Total Time: 0.011 Seconds
Total[0.4008] Device[6] HT[1] DS[32] TT[1.27] SNMP: v2: z-ilo-2, dsname: temp_celcius, oid: .1.3.6.1.4.1.232.6.2.6.8.1.4.0.1, value: 19
Total[0.4009] Device[6] HT[1] DS[32] TT[1.29] SNMP: v2: z-ilo-2, dsname: temp_condition, oid: .1.3.6.1.4.1.232.6.2.6.8.1.6.0.1, value: 2
Total[0.4009] Device[6] HT[1] DS[32] TT[1.31] SNMP: v2: z-ilo-2, dsname: temp_threshold, oid: .1.3.6.1.4.1.232.6.2.6.8.1.5.0.1, value: 42
Total[0.4009] Device[6] HT[1] DS[33] TT[1.32] SNMP: v2: z-ilo-2, dsname: temp_celcius, oid: .1.3.6.1.4.1.232.6.2.6.8.1.4.0.15, value: 24
Total[0.4009] Device[6] HT[1] DS[33] TT[1.33] SNMP: v2: z-ilo-2, dsname: temp_condition, oid: .1.3.6.1.4.1.232.6.2.6.8.1.6.0.15, value: 2
Total[0.4009] Device[6] HT[1] DS[33] TT[1.35] SNMP: v2: z-ilo-2, dsname: temp_threshold, oid: .1.3.6.1.4.1.232.6.2.6.8.1.5.0.15, value: 70
Total[0.4010] Device[5] HT[1] Total Time: 0.01 Seconds
Total[0.4013] Device[6] HT[1] Updating Poller Items for Next Poll
Total[0.4018] Device[6] HT[1] Total Time: 0.011 Seconds
Total[0.4236] Device[4] HT[1] DS[23] TT[30.30] SCRIPT: perl /var/www/html/cacti/scripts/unix_processes.pl, output: 220
Total[0.4351] Device[8] HT[1] DS[36] TT[36.02] SNMP: v2: z-ilo-4, dsname: temp_celcius, oid: .1.3.6.1.4.1.232.6.2.6.8.1.4.0.1, value: 20
Total[0.4351] Device[8] HT[1] DS[36] TT[36.06] SNMP: v2: z-ilo-4, dsname: temp_condition, oid: .1.3.6.1.4.1.232.6.2.6.8.1.6.0.1, value: 2
Total[0.4352] Device[8] HT[1] DS[36] TT[36.08] SNMP: v2: z-ilo-4, dsname: temp_threshold, oid: .1.3.6.1.4.1.232.6.2.6.8.1.5.0.1, value: 42
Total[0.4352] Device[8] HT[1] DS[37] TT[36.09] SNMP: v2: z-ilo-4, dsname: temp_celcius, oid: .1.3.6.1.4.1.232.6.2.6.8.1.4.0.15, value: 24
Total[0.4352] Device[8] HT[1] DS[37] TT[36.10] SNMP: v2: z-ilo-4, dsname: temp_condition, oid: .1.3.6.1.4.1.232.6.2.6.8.1.6.0.15, value: 2
Total[0.4352] Device[8] HT[1] DS[37] TT[36.12] SNMP: v2: z-ilo-4, dsname: temp_threshold, oid: .1.3.6.1.4.1.232.6.2.6.8.1.5.0.15, value: 70
Total[0.4356] Device[8] HT[1] Updating Poller Items for Next Poll
Total[0.4362] Device[8] HT[1] Total Time: 0.046 Seconds
Total[0.4392] Device[4] HT[1] DS[24] TT[15.56] SCRIPT: perl /var/www/html/cacti/scripts/loadavg_multi.pl, output: 1min:0.00 5min:0.01 10min:0.05
Total[0.4525] Device[4] HT[1] DS[25] TT[13.29] SCRIPT: perl /var/www/html/cacti/scripts/unix_users.pl '', output: 2
Total[0.4645] Device[4] HT[1] DS[26] TT[11.96] SCRIPT: perl /var/www/html/cacti/scripts/linux_memory.pl 'MemFree:', output: 14574416
Total[0.4764] Device[4] HT[1] DS[27] TT[11.87] SCRIPT: perl /var/www/html/cacti/scripts/linux_memory.pl 'SwapFree:', output: 15624188
Total[0.4844] Device[4] HT[1] DS[28] TT[7.96] SCRIPT: perl /var/www/html/cacti/scripts/query_unix_partitions.pl 'get' 'available' '/dev/sda1', output: 307328
Total[0.4921] Device[4] HT[1] DS[28] TT[7.67] SCRIPT: perl /var/www/html/cacti/scripts/query_unix_partitions.pl 'get' 'used' '/dev/sda1', output: 176676
Total[0.4996] Device[4] HT[1] DS[29] TT[7.46] SCRIPT: perl /var/www/html/cacti/scripts/query_unix_partitions.pl 'get' 'available' '/dev/sda3', output: 86170864
Total[0.5060] Device[4] HT[1] DS[29] TT[6.42] SCRIPT: perl /var/www/html/cacti/scripts/query_unix_partitions.pl 'get' 'used' '/dev/sda3', output: 2530764
Total[0.5065] Device[4] HT[1] Updating Poller Items for Next Poll
Total[0.5071] Device[4] HT[1] Total Time: 0.12 Seconds
Total[0.8915] The Final Value of Threads is 0
Total[0.8948] Time: 0.8914 s, Threads: 32, Devices: 5

Any ideas?
Thanks

[eddi]

an interesting update; I decreased the poller's Processes and Threads by half and it got solved:
2021-09-08 08:23:04 - SYSTEM STATS: Time:2.3524 Method:spine Processes:4 Threads:16 Hosts:5 HostsPerProcess:2 DataSources:33 RRDsProcessed:0
2021-09-08 08:22:04 - SYSTEM STATS: Time:58.8765 Method:spine Processes:8 Threads:32 Hosts:5 HostsPerProcess:1 DataSources:33 RRDsProcessed:0