[SOLVED] polling timed out while acquiring Available Thread Lock

[cksrealm]

After months of Cacti working perfectly Im suddenly getting the following error in the logs.

polling timed out while acquiring Available Thread Lock

Can anyone provide any insight.
Running on RedHatEnterprise 8.0

[TheWitness]

That sounds like a timeout. Post your cacti stats: from the cacti log.

[cksrealm]

Hi, see below from the cacti log

Issue appears to have stopped now for some unknown reason.


2023-08-08 11:59:23 - CMDPHP SNVer STATS: hosts processed/max: 0/100. Duration: 0.01
2023-08-08 11:59:23 - SYSTEM STATS: WEATHERMAP Time:7.08 Maps:13 Warnings:155 Notes:None
2023-08-08 11:59:16 - SYSTEM MONITOR STATS: Time:0.02 Reboots:0 DownDevices:0 Notifications:0 Purges:0
2023-08-08 11:59:16 - SYSTEM THOLD STATS: Time:7.5980 Tholds:9112 TotalDevices:401 DownDevices:0 NewDownDevices:0
2023-08-08 11:59:09 - SYSTEM MAINT STATS: Time:0.02
2023-08-08 11:59:07 - SYSTEM STATS: Time:186.0664 Method:spine Processes:1 Threads:1 Hosts:400 HostsPerProcess:400 DataSources:34702 RRDsProcessed:0
2023-08-08 11:55:46 - CMDPHP SNVer STATS: hosts processed/max: 0/100. Duration: 0.01
2023-08-08 11:55:46 - SYSTEM STATS: WEATHERMAP Time:7.49 Maps:13 Warnings:155 Notes:None
2023-08-08 11:55:38 - SYSTEM MONITOR STATS: Time:0.02 Reboots:0 DownDevices:0 Notifications:0 Purges:0
2023-08-08 11:55:38 - SYSTEM THOLD STATS: Time:7.6419 Tholds:9081 TotalDevices:401 DownDevices:0 NewDownDevices:0
2023-08-08 11:55:31 - SYSTEM MAINT STATS: Time:0.02
2023-08-08 11:55:29 - SYSTEM STATS: Time:207.1550 Method:spine Processes:1 Threads:1 Hosts:400 HostsPerProcess:400 DataSources:34702 RRDsProcessed:0
2023-08-08 11:51:29 - CMDPHP SNVer STATS: hosts processed/max: 0/100. Duration: 0.01
2023-08-08 11:51:29 - SYSTEM STATS: WEATHERMAP Time:7.06 Maps:13 Warnings:155 Notes:None
2023-08-08 11:51:22 - SYSTEM MONITOR STATS: Time:0.01 Reboots:0 DownDevices:0 Notifications:0 Purges:0
2023-08-08 11:51:22 - SYSTEM THOLD STATS: Time:7.6355 Tholds:9112 TotalDevices:401 DownDevices:0 NewDownDevices:0
2023-08-08 11:51:15 - SYSTEM MAINT STATS: Time:0.03
2023-08-08 11:51:13 - SYSTEM STATS: Time:191.9437 Method:spine Processes:1 Threads:1 Hosts:400 HostsPerProcess:400 DataSources:34702 RRDsProcessed:0
2023-08-08 11:47:38 - CMDPHP SNVer STATS: hosts processed/max: 0/100. Duration: 0.01
2023-08-08 11:47:38 - SYSTEM STATS: WEATHERMAP Time:7.74 Maps:13 Warnings:155 Notes:None
2023-08-08 11:47:30 - SYSTEM MONITOR STATS: Time:0.01 Reboots:0 DownDevices:0 Notifications:0 Purges:0
2023-08-08 11:47:30 - SYSTEM THOLD STATS: Time:7.6752 Tholds:9074 TotalDevices:401 DownDevices:0 NewDownDevices:0
2023-08-08 11:47:23 - SYSTEM MAINT STATS: Time:0.05
2023-08-08 11:47:21 - SYSTEM STATS: Time:198.9717 Method:spine Processes:1 Threads:1 Hosts:400 HostsPerProcess:400 DataSources:34702 RRDsProcessed:0
2023-08-08 11:43:47 - CMDPHP SNVer STATS: hosts processed/max: 0/100. Duration: 0.01
2023-08-08 11:43:47 - SYSTEM STATS: WEATHERMAP Time:7.48 Maps:13 Warnings:155 Notes:None
2023-08-08 11:43:40 - SYSTEM MONITOR STATS: Time:0.01 Reboots:0 DownDevices:0 Notifications:0 Purges:0
2023-08-08 11:43:40 - SYSTEM THOLD STATS: Time:7.9184 Tholds:9058 TotalDevices:401 DownDevices:0 NewDownDevices:0
2023-08-08 11:43:32 - SYSTEM MAINT STATS: Time:0.07
2023-08-08 11:43:31 - SYSTEM STATS: Time:209.3814 Method:spine Processes:1 Threads:1 Hosts:400 HostsPerProcess:400 DataSources:34702 RRDsProcessed:0

[TheWitness]

Change your spine threads to 10 under:

Console > Data Collection > Data Collectors > Edit

Then post the results.

[cksrealm]

Have done so, have a remote poller setup as well which I have disabled as having some issues with it so have disabled that for now.

[TheWitness]

Yea, it's safe to make them both 10.

[cksrealm]

Having major issues with the remote poller at the moment, its only set to poll one device at the moment and its exceeding the 5 minutes limit for some reason, I think there is something wrong with the spine setup on it that Im trying to work out.

[TheWitness]

Likely you don't have spine configured properly there. Login to the remote data collector and do the following:

Code: Select all

cd /your/spine/path/bin
./spine -R -V 3 -S -p=X

Replace X with the ID of the remote poller.

[cksrealm]

That appeared to work with no issues.

# ./spine -R -V 3 -S -p=2
SPINE: Using spine config file [../etc/spine.conf]
Version 1.2.23 starting
Sending entries to remote database in 'online' mode
Total[0.2256] Spine will support multithread device polling.
Total[0.2261] DEBUG: Initial Value of Available Threads is 2 (0 outstanding)
Total[0.2265] DEBUG: Available Threads is 1 (1 outstanding)
Total[0.2276] Device[416] INFO: SNMP Device '10.x.x.x:161' has a timeout of 500000 (500), with 3 retries
Total[0.2536] Device[416] Checking for System Information Update
Total[0.2536] Device[416] Updating Full System Information Table
Total[0.4505] Device[416] HT[1] DQ[4] Legacy Uptime Result: 231879582, Is Numeric: 1
Total[0.4750] Device[416] HT[1] DQ[4] Extended Uptime Result: 230970200, Is Numeric: 1
Total[0.4751] Device[416] HT[1] DQ[4] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 103348300 < output: 230970200)
Total[0.4751] Device[416] HT[1] DQ[5] RECACHE OID: .1.3.6.1.2.1.1.3.0, (assert: 103348300 < output: 230970200)
Total[0.4760] Device[416] HT[1] NOTE: There are '10' Polling Items for this Device
Total[0.4760] Device[416] INFO: SNMP Device '10.x.x.x:161' has a timeout of 500000 (500), with 3 retries
Total[0.4797] Device[416] HT[1] DS[12061] TT[3.58] SS[0] SERVER: /var/www/html/cacti/scripts/ss_hstats.php ss_hstats '416' uptime, output: 103369800
Total[0.5055] Device[416] HT[1] DS[12059] TT[25.81] SNMP: v2: 10.x.x.x, dsname: 5min_cpu, oid: .1.3.6.1.4.1.9.2.1.58.0, value: 1
Total[0.5055] Device[416] HT[1] DS[12062] TT[25.86] SNMP: v2: 10.x.x.x dsname: bgpPeerState, oid: .1.3.6.1.2.1.15.3.1.2.10.0.22.10, value: 6
Total[0.5056] Device[416] HT[1] DS[12063] TT[25.87] SNMP: v2: 10.x.x.x, dsname: bgpPeerState, oid: .1.3.6.1.2.1.15.3.1.2.10.253.253.250, value: 6
Total[0.5056] Device[416] HT[1] DS[12064] TT[25.88] SNMP: v2: 10.x.x.x, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.2, value: 2889709193324
Total[0.5056] Device[416] HT[1] DS[12064] TT[25.90] SNMP: v2: 10.x.x.x, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.2, value: 43072009013
Total[0.5056] Device[416] HT[1] DS[12065] TT[25.91] SNMP: v2: 10.x.x.x, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.3, value: 190198167059
Total[0.5056] Device[416] HT[1] DS[12065] TT[25.93] SNMP: v2: 10.x.x.x, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.3, value: 4744210034552
Total[0.5056] Device[416] HT[1] DS[12066] TT[25.94] SNMP: v2: 10.x.x.x, dsname: traffic_in, oid: .1.3.6.1.2.1.31.1.1.1.6.4, value: 0
Total[0.5056] Device[416] HT[1] DS[12066] TT[25.95] SNMP: v2: 10.x.x.x, dsname: traffic_out, oid: .1.3.6.1.2.1.31.1.1.1.10.4, value: 0
Total[0.5058] Device[416] HT[1] Total Time: 0.28 Seconds
Total[0.7775] Time: 0.7773 s, Threads: 2, Devices: 1

[TheWitness]

See what it's like with 10 threads. You have a Cacti log on that server too.

[TheWitness]

Also, the paths for spine on the remote poller (from the GUI), can be different. So, from the Remote Pollers user interface go to Console > Configuration > Settings > Paths and make sure they are all accurate.

[cksrealm]

Yeah all the paths are correct, I have enabled the remote poller again with 10 threads, will let you know in 10 minutes once a cycle has run.

[TheWitness]

Yea, getting late here.

[cksrealm]

Just to check something, when running a remote poller you only need to add the RDB host settings to the remote poller and not the main poller right?

IM referring to

RDB_Host x.x.x.x
RDB_Database cacti
RDB_User cacti
RDB_Pass xxxxxxxxxxxx
RDB_Port 3306

[TheWitness]

Yea, you do. They should point back to the Main Cacti Server, and of course the connectivity has to be there.