2 Devices: "Polling timed out while waiting for 4 Threads to End" - Little Help on Poller Statistics?

[Osiris]

I suspect you need to do this

Code: Select all

ALTER TABLE poller_output_boost ADD COLUMN last_updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, ADD INDEX last updated(last_updated);

[macan]

I have met the same problem today. Log shows a lot of messages ...polling sleeping while waiting for X Threads to End.
I tried everything - update mysql, cacti and spine (1.2.25), delete, disable problematic hosts but still the same result.
I tried decrease spine process/thread from 1/10 to 1/7 and it is working now.

I found one bug. Go to console -> Management -> Data sources. Have you here any damaged data sources? I had there few DS with name "Damaged data source" and these DS cause problem with poller. It wasn't main problem but i can cause another things

[Jeeves]

I suspect you need to do this

Code: Select all

ALTER TABLE poller_output_boost ADD COLUMN last_updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, ADD INDEX last updated(last_updated);

It's been a busy week, so I haven't had time to stick my head in and test some of these out, but I appreciate your feedback!

I have met the same problem today. Log shows a lot of messages ...polling sleeping while waiting for X Threads to End.
I tried everything - update mysql, cacti and spine (1.2.25), delete, disable problematic hosts but still the same result.
I tried decrease spine process/thread from 1/10 to 1/7 and it is working now.

I found one bug. Go to console -> Management -> Data sources. Have you here any damaged data sources? I had there few DS with name "Damaged data source" and these DS cause problem with poller. It wasn't main problem but i can cause another things

I flicked through the data sources, none of them say damaged? Where would I see that text? I searched for 'damaged' and got no results on the DS page.

[TheWitness]

Talking to macan about this one. I did a spine update this morning, to at least acknowledge device 0, which is a special class of device that I had removed support for years ago, and re-introduced at the request of Ian Berry, the Cacti founder. So, though it's still supported, it will not be counted if it's not in use now.

With regard to the threads not ending, I have asked macan to try and find the Cacti version where that regression was introduced. Might take a few days for him to find it.

[Jeeves]

Talking to macan about this one. I did a spine update this morning, to at least acknowledge device 0, which is a special class of device that I had removed support for years ago, and re-introduced at the request of Ian Berry, the Cacti founder. So, though it's still supported, it will not be counted if it's not in use now.

With regard to the threads not ending, I have asked macan to try and find the Cacti version where that regression was introduced. Might take a few days for him to find it.

I'm not sure I follow how device 0 relates to my particular scenario, but if you think it does, I'm happy to go down that rabbithole.

[Osiris]

Macan is having similar issues and is trying to find the breaking change by testing older versions.

[macan]

Have you got this error in your mysql error log?
2023-11-07 9:12:00 469892 [Warning] Aborted connection 469892 to db: 'cacti' user: 'cactiuser' host: 'localhost' (Got an error reading communication packets)
2023-11-07 9:14:00 470362 [Warning] Aborted connection 470362 to db: 'cacti' user: 'cactiuser' host: 'localhost' (Got an error reading communication packets)
2023-11-07 9:14:00 470361 [Warning] Aborted connection 470361 to db: 'cacti' user: 'cactiuser' host: 'localhost' (Got an error reading communication packets)
2023-11-07 9:15:01 470593 [Warning] Aborted connection 470593 to db: 'cacti' user: 'cactiuser' host: 'localhost' (Got an error reading communication packets)
2023-11-07 9:15:01 470594 [Warning] Aborted connection 470594 to db: 'cacti' user: 'cactiuser' host: 'localhost' (Got an error reading communication packets)

[macan]

It seems to be resolved. In my case there were more things. Please try:
- change availability method from "ping and snmp" and "ping or snmp" to snmp uptime or similar.
- check snmp and ping timeout. I have there 15 seconds snmp timeout for one device. It is nonsence, I don't know, why was there
- you can use actual spine from 1.2.x branch

[Jeeves]

Have you got this error in your mysql error log?
2023-11-07 9:12:00 469892 [Warning] Aborted connection 469892 to db: 'cacti' user: 'cactiuser' host: 'localhost' (Got an error reading communication packets)
2023-11-07 9:14:00 470362 [Warning] Aborted connection 470362 to db: 'cacti' user: 'cactiuser' host: 'localhost' (Got an error reading communication packets)
2023-11-07 9:14:00 470361 [Warning] Aborted connection 470361 to db: 'cacti' user: 'cactiuser' host: 'localhost' (Got an error reading communication packets)
2023-11-07 9:15:01 470593 [Warning] Aborted connection 470593 to db: 'cacti' user: 'cactiuser' host: 'localhost' (Got an error reading communication packets)
2023-11-07 9:15:01 470594 [Warning] Aborted connection 470594 to db: 'cacti' user: 'cactiuser' host: 'localhost' (Got an error reading communication packets)

I searched for the word 'aborted' and did not have any hits.

It seems to be resolved. In my case there were more things. Please try:
- change availability method from "ping and snmp" and "ping or snmp" to snmp uptime or similar.
- check snmp and ping timeout. I have there 15 seconds snmp timeout for one device. It is nonsence, I don't know, why was there
- you can use actual spine from 1.2.x branch

So, I have not logged into Cacti since my last response in this thread.
Today, we're right back to normal. No change to cacti or the underlying host, no changes to infrastructure/ISPs.
I can't explain it and it feels like a bug that my PING only device was responding just fine, but getting flagged in the logs as Poller timeouts. And yet here we are;

cacti-2023-perfomanceBackToNorm.png (9.87 KiB) Viewed 1517 times

I will look though your provided list but I don't feel like they largely apply. E.g.

change availability method from "ping and snmp" and "ping or snmp" to snmp uptime or similar.

My device doesn't use SNMP, it is ICMP ping only.
With "SNMP Version" set to "Not in Use" the options for SNMP do not exist under the "Availability" section.
Only "ICMP Ping" "UDP Ping" and "TCP Ping" exist.

check snmp and ping timeout. I have there 15 seconds snmp timeout for one device. It is nonsence, I don't know, why was there

On this device Ping Timeout is set identically to other devices; 400.
I should note however the I assume this to be ms, but there is no unit of measurement listed.

you can use actual spine from 1.2.x branch

My version of spine is the current release version; 1.2.25

[TheWitness]

This is fixed in the 1.2.x branch and will be a part of the 1.2.26 release.

[Jeeves]

This is fixed in the 1.2.x branch and will be a part of the 1.2.26 release.

I'm not even sure WHAT broke, but excellent! Thanks!