Facing polling issues post upgrade to cacti 1.2.23

Post support questions that directly relate to Linux/Unix operating systems.

Moderators: Developers, Moderators

Post Reply
mahavirj
Posts: 13
Joined: Sun Sep 20, 2020 9:46 am

Facing polling issues post upgrade to cacti 1.2.23

Post by mahavirj »

Post upgrading cacti from 1.2.21 to 1.2.23, cacti collector process is taking more than 290 seconds, which ideally used to happen in 60-70 seconds.
collector.png
collector.png (17.86 KiB) Viewed 884 times
Cacti Graphs are having gaps in-between
sample_device_poll.png
sample_device_poll.png (43.04 KiB) Viewed 884 times
Below actions taken till now :
1. Verified all the parameters in technical support
2. repair_database --> without any errors
3. exectured poller_output_empty script
4. rebuild-resource-cache
5. rebuild-poller-cache
techinal_support.png
techinal_support.png (452.61 KiB) Viewed 884 times
Top
User avatar
macan
Cacti Guru User
Posts: 1106
Joined: Tue Mar 18, 2008 2:30 am
Location: Czech

Re: Facing polling issues post upgrade to cacti 1.2.23

Post by macan »

any errors or warning in log?
You can try new spine version (download and compile) from https://github.com/Cacti/spine/tree/1.2.x

There was any issue in spine 1.2.23:
https://github.com/Cacti/spine/blob/1.2.x/CHANGELOG
Let the Cacti grow!
Top
mahavirj
Posts: 13
Joined: Sun Sep 20, 2020 9:46 am

Re: Facing polling issues post upgrade to cacti 1.2.23

Post by mahavirj »

No Error/warnings in logs related to poller

already running spine 1.2.23
Top
mahavirj
Posts: 13
Joined: Sun Sep 20, 2020 9:46 am

Re: Facing polling issues post upgrade to cacti 1.2.23

Post by mahavirj »

can someone help me out with the problem.
Due to this issue DB got crashed.
Top
User avatar
macan
Cacti Guru User
Posts: 1106
Joined: Tue Mar 18, 2008 2:30 am
Location: Czech

Re: Facing polling issues post upgrade to cacti 1.2.23

Post by macan »

Try to sort your device by poll time. Is there any device with very long poll time?
How many down devices you have? if you had too many down devices and too long timeouts, it could cause the poller to run long
Let the Cacti grow!
Top
mahavirj
Posts: 13
Joined: Sun Sep 20, 2020 9:46 am

Re: Facing polling issues post upgrade to cacti 1.2.23

Post by mahavirj »

The max polling time one or two of device taking is 20seconds, from remaining 20% devices takes less than 5 seconds and 80% devices are polled in millisecond interval.

Down devices count is 25 approx.

The issue has started just post upgrading cacti to 1.2.23.
Top
User avatar
macan
Cacti Guru User
Posts: 1106
Joined: Tue Mar 18, 2008 2:30 am
Location: Czech

Re: Facing polling issues post upgrade to cacti 1.2.23

Post by macan »

please show cacti.log - at least one poller cycle. There should be any error/warning
Let the Cacti grow!
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Re: Facing polling issues post upgrade to cacti 1.2.23

Post by TheWitness »

Update spine to the 1.2.x branch
There was a regression in the 1.2.23 release.
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
mahavirj
Posts: 13
Joined: Sun Sep 20, 2020 9:46 am

Re: Facing polling issues post upgrade to cacti 1.2.23

Post by mahavirj »

Shall I proceed with installation of cacti-spine release 1.2.23 as below
release_1.2.23.png
release_1.2.23.png (12.85 KiB) Viewed 827 times
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Re: Facing polling issues post upgrade to cacti 1.2.23

Post by TheWitness »

Use the 1.2.x branch on GitHub, not the release tarball.
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
mahavirj
Posts: 13
Joined: Sun Sep 20, 2020 9:46 am

Re: Facing polling issues post upgrade to cacti 1.2.23

Post by mahavirj »

Thanks, can you please share me the link or do you want me to download the recent branch named : Preparing for 1.2.24 release
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Re: Facing polling issues post upgrade to cacti 1.2.23

Post by TheWitness »

Code: Select all

git clone -b 1.2.x https://github.com/cacti/cacti.git
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
mahavirj
Posts: 13
Joined: Sun Sep 20, 2020 9:46 am

Re: Facing polling issues post upgrade to cacti 1.2.23

Post by mahavirj »

Thanks now the primary issue is solved post updating to branch 1.2.x, Poller time reduced in between 45-80 seconds.

However now I am facing new issue.

Post updating to recommended branch, weathermap plugin(0.98) stopped working

Hence to bring back the plugin, I installed a plugin from : https://github.com/Cacti/plugin_weathermap.git

Post enabling new weathermap works however cacti polling time again spikes in between 260-290 seconds. To confirm the same I disabled the plugin and polling time reduces to said before.

Can you please check for same.

Polling stats below:
2023-02-21 14:51:04 - SYSTEM STATS: Time:62.2528 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 14:45:55 - SYSTEM STATS: Time:53.6700 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 14:44:18 - SYSTEM STATS: Time:256.7625 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 14:39:11 - SYSTEM STATS: Time:249.5203 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 14:34:18 - SYSTEM STATS: Time:256.0257 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 14:29:35 - SYSTEM STATS: Time:273.6222 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 14:24:26 - SYSTEM STATS: Time:264.1048 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 14:20:07 - SYSTEM STATS: Time:305.6012 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 14:14:25 - SYSTEM STATS: Time:263.0573 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 14:09:11 - SYSTEM STATS: Time:248.7488 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 14:04:45 - SYSTEM STATS: Time:283.7497 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 14:00:25 - SYSTEM STATS: Time:323.7485 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 13:54:07 - SYSTEM STATS: Time:245.5016 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 13:49:14 - SYSTEM STATS: Time:252.2138 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 13:44:22 - SYSTEM STATS: Time:260.8185 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 13:39:19 - SYSTEM STATS: Time:257.6375 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 13:34:11 - SYSTEM STATS: Time:249.5179 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 13:29:28 - SYSTEM STATS: Time:266.2009 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 13:25:10 - SYSTEM STATS: Time:308.2520 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 13:19:15 - SYSTEM STATS: Time:253.2777 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 13:14:27 - SYSTEM STATS: Time:264.5856 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 13:09:11 - SYSTEM STATS: Time:249.7552 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 13:00:52 - SYSTEM STATS: Time:50.5807 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 12:55:56 - SYSTEM STATS: Time:53.9250 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 12:50:51 - SYSTEM STATS: Time:48.6005 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 12:45:44 - SYSTEM STATS: Time:42.0275 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 12:40:56 - SYSTEM STATS: Time:54.8136 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 12:35:53 - SYSTEM STATS: Time:52.1809 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 12:30:56 - SYSTEM STATS: Time:53.9103 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 12:25:59 - SYSTEM STATS: Time:57.9687 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 12:20:51 - SYSTEM STATS: Time:48.6606 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 12:15:48 - SYSTEM STATS: Time:46.2762 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 12:10:57 - SYSTEM STATS: Time:55.4332 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 12:05:56 - SYSTEM STATS: Time:54.5659 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197122 RRDsProcessed:0
2023-02-21 12:01:15 - SYSTEM STATS: Time:73.1352 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197112 RRDsProcessed:0
2023-02-21 11:55:53 - SYSTEM STATS: Time:51.5742 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197112 RRDsProcessed:0
2023-02-21 11:50:53 - SYSTEM STATS: Time:51.0164 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197112 RRDsProcessed:0
2023-02-21 11:45:51 - SYSTEM STATS: Time:49.8547 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197112 RRDsProcessed:0
2023-02-21 11:40:43 - SYSTEM STATS: Time:41.4537 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197112 RRDsProcessed:0
2023-02-21 11:36:05 - SYSTEM STATS: Time:63.5662 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197112 RRDsProcessed:0
2023-02-21 11:30:51 - SYSTEM STATS: Time:49.7444 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197112 RRDsProcessed:0
2023-02-21 11:25:48 - SYSTEM STATS: Time:45.7860 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197112 RRDsProcessed:0
2023-02-21 11:20:48 - SYSTEM STATS: Time:45.8497 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197112 RRDsProcessed:0
2023-02-21 11:15:43 - SYSTEM STATS: Time:41.7671 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197112 RRDsProcessed:0
2023-02-21 11:10:42 - SYSTEM STATS: Time:40.9601 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197112 RRDsProcessed:0
2023-02-21 11:06:22 - SYSTEM STATS: Time:79.8741 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197112 RRDsProcessed:0
2023-02-21 11:00:49 - SYSTEM STATS: Time:48.0022 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197112 RRDsProcessed:0
2023-02-21 10:55:45 - SYSTEM STATS: Time:43.7018 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197112 RRDsProcessed:0
2023-02-21 10:50:58 - SYSTEM STATS: Time:55.9579 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197112 RRDsProcessed:0
2023-02-21 10:45:49 - SYSTEM STATS: Time:47.5997 Method:spine Processes:22 Threads:16 Hosts:6549 HostsPerProcess:298 DataSources:197112 RRDsProcessed:0
Top
User avatar
TheWitness
Developer
Posts: 17007
Joined: Tue May 14, 2002 5:08 pm
Location: MI, USA
Contact:
Contact TheWitness

Re: Facing polling issues post upgrade to cacti 1.2.23

Post by TheWitness »

Dude I have north of 30k hosts and I don't use anything near as aggressive as that. Your server better have 100+ threads...
True understanding begins only when we realize how little we truly understand...

Life is an adventure, let yours begin with Cacti!

Author of dozens of Cacti plugins and customization's. Advocate of LAMP, MariaDB, IBM Spectrum LSF and the world of batch. Creator of IBM Spectrum RTM, author of quite a bit of unpublished work and most of Cacti's bugs.
_________________
Official Cacti Documentation
GitHub Repository with Supported Plugins
Percona Device Packages (no support)
Interesting Device Packages

For those wondering, I'm still here, but lost in the shadows. Yearning for less bugs. Who want's a Cacti 1.3/2.0? Streams anyone?
Top
Post Reply

Return to “Help: Linux/Unix Specific”

Who is online

Users browsing this forum: No registered users and 1 guest