I Broke It

[jeffhochberg]

OK - so obviously I made a huge mistake and broke my Cacti server.

I've been running Cacti for several months. I've been getting errors about the poller and cron not running in sync:

WARNING: Cron is out of sync with the Poller Interval for poller id 1! The Poller Interval is 60 seconds, with a maximum of a 60 seconds, but 299 seconds have passed since the last poll!

A few weeks ago, I read an article on how to properly configure Cacti for 1 minute polling, so I followed the recommendations. I cloned the default template and made the recommended modifications, the rebuilt the poller cache.

Things seemed to be OK for a few days, then I started receiving a different message about the poller:

Maximum runtime of 298 seconds exceeded for poller id 1. Exiting.

I went back to the document I read about adjusting the templates for 1 minute polling, as well as recommendations on how to properly configure the poller and cron entries (despite no longer using cron). I was using 1 and 1 respectively. I changed them back to 30 seconds and 5 minutes. I rebuilt the poller cache again.

After that, things really broke. I started getting messages like this:

WARNING: Poller Output Table not empty for poller id 1. Issues: 65, DS[92, 93, 94, 95, 95, 99, 100, 101, 102, 103, 104, 105, 106, 107, 107, 108, 108, 109, 109, 114], Additional Issues Remain. Only showing first 20.

I started receiving email notifications every few seconds, so I ended up shutting down cactid, Apache, and MySQL. So at this this point, I'm effectively shut down. I'm considering rebuilding Cacti from scratch, but if there's any way to recover this instance, that would be my preference.

Am I completely SOL at this point?

[cigamit]

Honestly, it looks like you changed your Poller/Cron settings to 1 minute, without changing the actual Cron Job that runs the poller from 5 minutes to 1 minute.

[jeffhochberg]

Thanks so much for your response!

Well, am I still relying on cron since I switched to using cactid.service? I thought that switching over to using cactid.service would remove the need to have the cron job. Is that not the case? I actually removed the cron job at one point and was relying solely on cactid.service. It wasn't until I modified the Poller Interval and Cron/Daemon interval that I started seeing this issue. I just added the cron job back - still not entirely sure if it's necessary though.

I just started seeing this message in the logs as well:

2021-02-08 17:09:01 - POLLER: Poller[Main Poller] WARNING: There are 1 processes detected as overrunning a polling cycle, please investigate


/etc/cron.d/cacti:

*/1 * * * * www-data php /var/www/html/cacti/poller.php > /dev/null 2>&1

systemctl status cactid.service

● cactid.service - Cacti Daemon Main Poller Service
Loaded: loaded (/lib/systemd/system/cactid.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-02-08 16:54:39 EST; 7min ago
Process: 534840 ExecStart=/var/www/html/cacti/cactid.php (code=exited, status=0/SUCCESS)
Main PID: 534858 (php)
Tasks: 2 (limit: 2281)
Memory: 39.5M
CGroup: /system.slice/cactid.service
├─534858 php /var/www/html/cacti/cactid.php
└─536511 /usr/bin/php -q /var/www/html/cacti/poller.php --force

Feb 08 16:54:38 radio01 systemd[1]: Starting Cacti Daemon Main Poller Service...
Feb 08 16:54:39 radio01 cactid.php[534840]: Starting Cacti Daemon ... [OK]
Feb 08 16:54:39 radio01 systemd[1]: Started Cacti Daemon Main Poller Service.

[jeffhochberg]

@cigamit - thanks for your feedback. Oddly enough, I did have the Poller Interval and Cron/Daemon Interval set to 1 minute & 1 minute respectively before but found a comment someone made that said to "NEVER" set the Cron interval lower than 5 minutes. They also recommended using 30 seconds for the Poller Interval. So it was those recommendations that prompted me to change the values to 30 seconds and 5 minutes respectively.

I went back to the poller settings and modified the Poller Sync Interval and Cron/Daemon Interval...

From:

Poller Interval: 30 seconds
Cron/Daemon Interval: 5 minutes

To:

Poller Interval: 1 minute
Cron/Daemon Interval: 1 minute

After making the changes, I rebuilt the poller cache and resource cache. Since then everything has been running very smoothly. In fact, even more smoothly than before.

Thanks so much!!!

[jmaris]

Hi,
do you happen to know whether the cactid service still needs the cron job to run or not...
It's not clear to me either.
thanks
Jan

[jeffhochberg]

No - as long as you’re using cactid.service then it is no longer necessary to set up a cron job. In fact, that would probably cause issues so make sure it’s only one or the other.

I’ve since switched to using cactid.service and it’s been working perfectly.