Explanation of the sleep time

[dovid]

Hi,

I set up a new cacti box (our original box was done by a tech that is no longer with the company). On the old box we are monitoring 55 devices with no issues. On the new box we are watching 34 devices. For some reason on the new box it can't complete the poller with just 18 devices. I ran strace on the PID that runs poller.php and it showed a lot of sleeping. I added some of my own debug code under

Code: Select all

if ($poller_runs_completed == 1) {
                                $sleep_time = $poller_interval - $cur_loop_time - $overhead_time;
                        } else {
                                $sleep_time = $poller_interval - $cur_loop_time;
                        }

I added:

Code: Select all

echo "\nDOVID_12: poller_interval = $poller_interval\n";
echo "\nDOVID_12: overhead_time = $overhead_time\n";
echo "\nDOVID_12: cur_loop_time = $cur_loop_time\n";

The output in the logs was:

Code: Select all

DOVID_12: poller_interval = 60
DOVID_12: overhead_time = 0.076318025588989
DOVID_12: cur_loop_time = 9.7454109191895
DOVID_11: SLEEP TIME IS 50.178271055222

I understand this is affected by Settings -> Poller Interval. It was set to one minute (like the old system is). Even if I lower it to 10 seconds it seems we still have 10 seconds of sleep between graphs. Can anyone explain to me why cacti sleeps here? Is there some other setting I am missing?


TIA.

[cigamit]

Cacti only "sleeps" when you are doing your cron interval different than your poller interval.

Say your cron is set at 5 minutes, but you want you poller at 1 minute. Your cron launches the poller, we poll, but we know we can't close down the process as we have to run a few more (and your cron won't be launching those) so we determine the amount of time it took to poll, and we subtract that from the polling interval, and that give us how long to sleep until the next time we have to poll.

For multiplicities sake, I always recommend everyone to set the poller internal and cron interval to the same. Much easier to just launch a fresh process each time. The other options are generally there for people who have less control over their environment.

[dovid]

Thanks. I changed the cron to run every minute instead of every 5 minutes. I do see on the old system in the logs:
Waiting on X of 10 pollers.
and on the new system
Waiting on 1 of 2 pollers.

How do I increase the pollers so we can query all of our systems? It seems like cacti is not finishing in time

[cigamit]

Post your STATs line from the cacti log. Are you using spine or cmd.php? Is BOOST enabled?

[dovid]

Post your STATs line from the cacti log. Are you using spine or cmd.php? Is BOOST enabled?

As I was working on the issue I switched from cmd.php to spine. I am not sure what boost is but it seems like it is enabled. Below are the settings that I have for boost.



Below are the logs for the last hour

[root@noc2 cacti]# cat cacti.log| grep STAT | grep '2020/06/01 05:'
2020/06/01 05:00:01 - SYSTEM STATS: Time:0.4195 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2028 RRDsProcessed:0
2020/06/01 05:02:00 - SYSTEM STATS: Time:298.4114 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2055 RRDsProcessed:0
2020/06/01 05:02:00 - SYSTEM STATS: Time:0.3967 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2055 RRDsProcessed:0
2020/06/01 05:03:00 - SYSTEM STATS: Time:0.3894 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2055 RRDsProcessed:0
2020/06/01 05:04:00 - SYSTEM STATS: Time:0.3775 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2055 RRDsProcessed:0
2020/06/01 05:05:00 - SYSTEM STATS: Time:0.3746 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2055 RRDsProcessed:0
2020/06/01 05:06:00 - SYSTEM STATS: Time:299.1618 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2069 RRDsProcessed:0
2020/06/01 05:06:00 - SYSTEM STATS: Time:0.3791 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2069 RRDsProcessed:0
2020/06/01 05:07:00 - SYSTEM STATS: Time:0.3799 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2069 RRDsProcessed:0
2020/06/01 05:08:00 - SYSTEM STATS: Time:0.3778 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2069 RRDsProcessed:0
2020/06/01 05:09:00 - SYSTEM STATS: Time:0.4466 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2069 RRDsProcessed:0
2020/06/01 05:09:06 - SYSTEM STATS: Time:244.5442 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1998 RRDsProcessed:0
2020/06/01 05:10:01 - SYSTEM STATS: Time:54.5648 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1998 RRDsProcessed:0
2020/06/01 05:10:07 - SYSTEM STATS: Time:0.4512 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1998 RRDsProcessed:0
2020/06/01 05:11:07 - SYSTEM STATS: Time:0.5206 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1998 RRDsProcessed:0
2020/06/01 05:12:07 - SYSTEM STATS: Time:0.5368 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1998 RRDsProcessed:0
2020/06/01 05:14:00 - SYSTEM STATS: Time:298.9014 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1989 RRDsProcessed:0
2020/06/01 05:14:01 - SYSTEM STATS: Time:0.3675 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1989 RRDsProcessed:0
2020/06/01 05:15:01 - SYSTEM STATS: Time:0.3969 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1989 RRDsProcessed:0
2020/06/01 05:16:01 - SYSTEM STATS: Time:0.3699 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1989 RRDsProcessed:0
2020/06/01 05:17:00 - SYSTEM STATS: Time:238.8309 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2043 RRDsProcessed:0
2020/06/01 05:17:01 - SYSTEM STATS: Time:0.3103 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1989 RRDsProcessed:0
2020/06/01 05:17:06 - SYSTEM STATS: Time:4.3320 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1979 RRDsProcessed:0
2020/06/01 05:17:06 - SYSTEM STATS: Time:6.3790 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2043 RRDsProcessed:0
2020/06/01 05:18:00 - SYSTEM STATS: Time:0.3980 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2043 RRDsProcessed:0
2020/06/01 05:18:06 - SYSTEM STATS: Time:4.3823 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1979 RRDsProcessed:0
2020/06/01 05:19:00 - SYSTEM STATS: Time:0.3836 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2043 RRDsProcessed:0
2020/06/01 05:20:00 - SYSTEM STATS: Time:58.7878 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1979 RRDsProcessed:0
2020/06/01 05:20:00 - SYSTEM STATS: Time:0.3801 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2043 RRDsProcessed:0
2020/06/01 05:20:06 - SYSTEM STATS: Time:4.4575 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1979 RRDsProcessed:0
2020/06/01 05:21:07 - SYSTEM STATS: Time:5.2711 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2005 RRDsProcessed:0
2020/06/01 05:21:07 - SYSTEM STATS: Time:5.3917 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1979 RRDsProcessed:0
2020/06/01 05:23:01 - SYSTEM STATS: Time:59.7632 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2005 RRDsProcessed:0
2020/06/01 05:24:02 - SYSTEM STATS: Time:60.7240 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2005 RRDsProcessed:0
2020/06/01 05:26:00 - SYSTEM STATS: Time:118.0261 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2005 RRDsProcessed:0
2020/06/01 05:26:00 - SYSTEM STATS: Time:0.3772 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2005 RRDsProcessed:0
2020/06/01 05:27:01 - SYSTEM STATS: Time:119.6910 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1977 RRDsProcessed:0
2020/06/01 05:28:01 - SYSTEM STATS: Time:60.6259 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1977 RRDsProcessed:0
2020/06/01 05:30:00 - SYSTEM STATS: Time:119.2576 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1977 RRDsProcessed:0
2020/06/01 05:30:01 - SYSTEM STATS: Time:0.4701 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1977 RRDsProcessed:0
2020/06/01 05:30:52 - SYSTEM BOOST STATS: Time:50.5400 RRDUpdates:109164
2020/06/01 05:31:01 - SYSTEM STATS: Time:0.3886 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1977 RRDsProcessed:0
2020/06/01 05:32:01 - SYSTEM STATS: Time:179.9650 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2023 RRDsProcessed:0
2020/06/01 05:34:00 - SYSTEM STATS: Time:119.1299 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2023 RRDsProcessed:0
2020/06/01 05:34:00 - SYSTEM STATS: Time:0.3782 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2023 RRDsProcessed:0
2020/06/01 05:35:00 - SYSTEM STATS: Time:0.3333 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2023 RRDsProcessed:0
2020/06/01 05:36:00 - SYSTEM STATS: Time:0.3244 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2023 RRDsProcessed:0
2020/06/01 05:37:01 - SYSTEM STATS: Time:239.7953 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2031 RRDsProcessed:0
2020/06/01 05:37:06 - SYSTEM STATS: Time:5.4366 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2031 RRDsProcessed:0
2020/06/01 05:37:06 - SYSTEM STATS: Time:5.2799 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2118 RRDsProcessed:0
2020/06/01 05:38:01 - SYSTEM STATS: Time:0.5086 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2031 RRDsProcessed:0
2020/06/01 05:38:07 - SYSTEM STATS: Time:5.4254 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2118 RRDsProcessed:0
2020/06/01 05:39:01 - SYSTEM STATS: Time:0.3873 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2031 RRDsProcessed:0
2020/06/01 05:39:07 - SYSTEM STATS: Time:5.7125 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2118 RRDsProcessed:0
2020/06/01 05:40:01 - SYSTEM STATS: Time:0.3836 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2031 RRDsProcessed:0
2020/06/01 05:40:07 - SYSTEM STATS: Time:5.4762 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2118 RRDsProcessed:0
2020/06/01 05:41:07 - SYSTEM STATS: Time:5.4380 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1957 RRDsProcessed:0
2020/06/01 05:41:07 - SYSTEM STATS: Time:5.3794 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2118 RRDsProcessed:0
2020/06/01 05:43:02 - SYSTEM STATS: Time:60.8621 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1957 RRDsProcessed:0
2020/06/01 05:44:03 - SYSTEM STATS: Time:60.9221 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1957 RRDsProcessed:0
2020/06/01 05:46:00 - SYSTEM STATS: Time:116.9859 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1957 RRDsProcessed:0
2020/06/01 05:46:00 - SYSTEM STATS: Time:0.3719 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1957 RRDsProcessed:0
2020/06/01 05:47:01 - SYSTEM STATS: Time:120.0846 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1952 RRDsProcessed:0
2020/06/01 05:48:01 - SYSTEM STATS: Time:60.5685 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1952 RRDsProcessed:0
2020/06/01 05:50:00 - SYSTEM STATS: Time:118.2206 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1952 RRDsProcessed:0
2020/06/01 05:50:00 - SYSTEM STATS: Time:0.3762 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1952 RRDsProcessed:0
2020/06/01 05:51:00 - SYSTEM STATS: Time:0.3871 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1952 RRDsProcessed:0
2020/06/01 05:52:00 - SYSTEM STATS: Time:178.4972 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2046 RRDsProcessed:0
2020/06/01 05:53:01 - SYSTEM STATS: Time:60.6226 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2046 RRDsProcessed:0
2020/06/01 05:53:06 - SYSTEM STATS: Time:5.3792 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2046 RRDsProcessed:0
2020/06/01 05:53:06 - SYSTEM STATS: Time:5.3037 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2014 RRDsProcessed:0
2020/06/01 05:54:01 - SYSTEM STATS: Time:0.4460 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2046 RRDsProcessed:0
2020/06/01 05:54:06 - SYSTEM STATS: Time:5.4432 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2014 RRDsProcessed:0
2020/06/01 05:55:01 - SYSTEM STATS: Time:0.5290 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2046 RRDsProcessed:0
2020/06/01 05:55:07 - SYSTEM STATS: Time:6.0517 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2014 RRDsProcessed:0
2020/06/01 05:57:01 - SYSTEM STATS: Time:59.8010 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2014 RRDsProcessed:0
2020/06/01 05:57:07 - SYSTEM STATS: Time:5.7219 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1982 RRDsProcessed:0
2020/06/01 05:57:07 - SYSTEM STATS: Time:6.3817 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:2014 RRDsProcessed:0
2020/06/01 05:59:01 - SYSTEM STATS: Time:59.6241 Method:spine Processes:2 Threads:1 Hosts:31 HostsPerProcess:16 DataSources:1982 RRDsProcessed:0
[root@noc2 cacti]#

[cigamit]

Goto Console > Data Collectors > Main Poller and change the number of processes to be equal the number of CPUs you have in your Cacti server, and change the number of threads to around 12 (you can tweak these up or down later).

Most of your pollings are fine, but one of them is hanging up ever so often.

[dovid]

Thank you. That seems to have fixed it.