thold host up sends thousands of emails

[videodude]

Hello,

There are a couple of posts about this on here, but none seem to resolve the issue I am experiencing or be the same as I am experiencing.
I am running:
Cacti 0.8.8b installed from Debian.
Thold 0.5 from github
Settings 0.71 from github

The issue pertains to the recovery of a host:
Its the same if the host is set to use a 'Global' or a 'List Below' host up/down email notification, when the host goes down, I get a single message notifying me the host has gone down - this is fantastic.

However as soon as the host comes back up, I receive 50,000+ emails in the space of about 10 minutes until I turn off the notification of the host - which stopped the torrent of email advising the host was back up.
When I turn it back on after about an hour or so, its OK - but I can't disable it then reenable it - it will continue to send the messages.

There are no settings that I can see to change this, nothing in the Database that is out of the ordinary, or that I can see..
I took down the corporate mail server with this - and I have had to set all host up/down notifications to disabled until I can resolve this..

Just so the email contains as much information as I can, in settings, the Re-Alerting Repeat Alert after specified number of poller cycles - in Thold Settings is set to 9999.
And there are no host configurable options for re-alerting anywhere else...

More than happy to post database configs, and anything else anyone can suggest to try and figure this out..
Its a wonderful piece of software and resource, but this is just very frustrating..
I am just now relying on service failure to determine if a host is down, as opposed to ping or snmp failure.

Thanks in advance.

Chris

[phalek]

H,

That makes only sense if the function responsible for sending out the "up" emails is not completing, but stops right after sending out the emails ( afterwards it truncates the table it just processed ).

Can you provide a link to the GIT repository you've used ?

Did you try the one available from the documentation section ( http://docs.cacti.net/plugin:thold ) ?

[videodude]

Sorry it was the one from the docs.cacti.net...
My mistake, I went through the install notes again (its been up for a while) and found where it was downloaded from.

Hmmm.. ok - that kinda makes sense, is there a log that I can review to see what is causing the function to fail??

[phalek]

If it's a php error, you may need to enable error logging in php.ini:

Code: Select all

error_log =syslog

then you will see PHP errors in your syslog ( CentOS: /var/log/messages )

[videodude]

Ok will do that..

(Just gotta setup a /dev/null email address that the mail server can pipe all the mail to)

I will take a look and report back shortly.

Thanks again for the advise

[videodude]

Ok - so what I am seeing in /var/log/cacti/cacti.log

04/29/2016 09:21:21 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1142', SQL:"TRUNCATE TABLE poller_time'
04/29/2016 09:21:21 PM - POLLER: Poller[0] WARNING: Poller Output Table not Empty. Issues Found: 50, Data Sources: users(DS[8]), load_1min(DS[15]), prCount(DS[27]), prMax(DS[27]), prMin(DS[27]), users(DS[28]), prCount(DS[45]), prMax(DS[45]), prMin(DS[45]), users(DS[46]), load_5min(DS[53]), prCount(DS[63]), prMax(DS[63]), prMin(DS[63]), users(DS[64]), cpu(DS[76]), prCount(DS[81]), prMax(DS[81]), prMin(DS[81]), discards_in(DS[97]), Additional Issues Remain. Only showing first 20
04/29/2016 09:21:21 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1142', SQL:"TRUNCATE TABLE poller_output'
04/29/2016 09:21:21 PM - PHPSVR: Poller[0] PHP Script Server has Started - Parent is spine
04/29/2016 09:21:21 PM - SYSTEM STATS: Time:0.1304 Method:spine Processes:1 Threads:25 Hosts:27 HostsPerProcess:27 DataSources:55 RRDsProcessed:32

...

04/29/2016 09:21:21 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1142', SQL:"TRUNCATE TABLE plugin_thold_host_failed'

The three 'ERRORS' in the log consist of the following:

04/29/2016 09:23:21 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1142', SQL:"TRUNCATE TABLE poller_time'
04/29/2016 09:23:22 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1142', SQL:"TRUNCATE TABLE poller_output'
04/29/2016 09:23:23 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1142', SQL:"TRUNCATE TABLE plugin_thold_host_failed'


Once the host goes away, I get the poller_time error, but I seem to be getting a lot of poller_output and the plugin_thold_host_failed constantly.

[phalek]

Google tells me this:

Code: Select all

MySQL Error: #1142 - XXXXX command denied to user

So please check mysql/database permissions

[videodude]

I did that, it has everything but drop - which I granted it but the problem still persists.

[videodude]

Sorry I can also confirm that if I change the username I get a failed to connect to database when I attempt to do anything within the Cacti UI.
So the user and password are all correct..

[phalek]

try connecting to the database yourself with that username and password (mysql -u ... -p cacti ) and then try running the truncate command( TRUNCATE TABLE plugin_thold_host_failed ).

What does it do then ?

[videodude]

Hmm..

It appears that I need to grant it DROP permissions for it to work...
This is a bit odd, would prefer that user not have drop permissions to the database, but I guess if it works, I will just need to keep it there

I think I was seeing residual emails still flying through the mailserver.

[phalek]

yep:

TRUNCATE TABLE empties a table completely. It requires the DROP privilege as of MySQL 5.1.16. (Before 5.1.16, it requires the DELETE privilege).

[videodude]

Yeah... bit annoying, but oh well


Thank you for your assistance... I appreciate it..