Host Down E-mails Repeated

[GrL]

Hello All!

I configured the ability of my Cacti to send e-mails a pair of days ago and I received multiple e-mails today that the same host is down. The messages were sent with various pauses between messages. I see no threashold templates that do this. I think the monitor plugin causes that thold sends those messages. Can you help me make Cacti send such messages only once for a host if it stays down?

Thank you if you answer.

The information about my Cacti is below.

Cacti Version - 0.8.7c
Plugin Architecture - 2.2
Poller Type - Cactid v
Server Info - Linux 2.6.32-4-pve
Web Server - Apache/2.0.63 (CentOS)
PHP - 5.1.6
PHP Extensions - libxml, xml, wddx, tokenizer, sysvshm, sysvsem, sysvmsg, standard, SimpleXML, sockets, SPL, shmop, session, Reflection, pspell, posix, mime_magic, iconv, hash, gmp, gettext, ftp, exif, date, curl, ctype, calendar, bz2, zlib, pcre, openssl, apache2handler, gd, ldap, mysql, mysqli, PDO, pdo_mysql, pdo_sqlite, snmp, eAccelerator
MySQL - 5.0.82sp1
RRDTool - 1.2.23
SNMP - 5.1.2
Plugins

• Global Plugin Settings (settings - v0.5)
  Large Site Performane Booster for Cacti (boost - v2.2)
  Device Monitoring (monitor - v0.8.2)
  Network Discovery (discovery - v0.8.5)
  Network Tools (tools - v0.3)
  Syslog Monitoring (syslog - v0.5.2)
  Device Tracking (mactrack - v2.6)
  Login Page Mod (loginmod - v1.0)
  Update Checker (update - v0.4)
  FlowView (flowview - v0.6)
  Host Info (hostinfo - v0.2)
  Error Images (errorimage - v0.1)
  PHP Network Weathermap (weathermap - v0.95b)
  docs
  Cacti-ReportIt (reportit - v0.6.1)
  Router Configs (routerconfigs - v0.1)
  WMI Query (wmi - v0.1)
  Realtime for Cacti (realtime - v0.35)

[blc]

the same thing is happening in my cacti enviorment too.
havent found a way to solve it yet.
anyone got any idea on why does is send several e-mail about the same host ?

[phalek]

hm ... how many data sources does this host have, and how many mails do you get from it ?

[blc]

i got 1 data source for this host and i get 5 emails from it before it stop sending.
the time between each mail is 1 minute.

[phalek]

Do you have 1 minute polling enabled ?

[blc]

Thanks phalek for the lead,
I've re-configured the data template "Associated RRA's" to include the 1 minute template and changed the "Step" field to 60.
This seems to solve the problem. I'm not familiar with why its working or why it didn't worked from the first place and I'll be happy to be pointed to a document to explain me that.
Now it stoped sending multiple e-mails.

[phalek]

Well, the only thing I can tell you is that Thold is running a function called "thold_update_host_status" after each polling cycle (includes/poller.php).

Within that function it sends out an email alert for all hosts that match the following SQL statement:

Code: Select all

SELECT * FROM host WHERE disabled="" AND status=' . HOST_DOWN . ' AND status_event_count=' . $ping_failure_count

Now the "ping_failure_count" is set on the settings page and should be the "Failure Count".

Based on what you say, the settings on the RRAs does have an influence on the mentioned SQL statement.

I assume that due to the RRA the poller simply skips that host until the 5 minutes have passed again, therefore not updating the failure count field.

[TheHack]

I have faced the same issue since a while and couldn't find a solution. Even checking the code did not show any obvious bugs. Finally I checked the database entry.
There was the recommendation to clear the thold_failed_hosts content in the settings table. I did it a couple of times and it seems to improve. However after a couple of days the down alerts continue.

Then I realized that the field thold_failed_hosts is only 512 characters long. The field is just too short for keeping all the host id's (including comma) which are down. Extending this field to 1024 or even more characters can solve the issue. See the SQL commmand below.

Code: Select all

ALTER TABLE `settings` CHANGE `value` `value` VARCHAR( 1024 ) NOT NULL DEFAULT ''

[dragossto]

good to know, i don't monitor so many host,and for me not happen this issue

[TheWitness]

That whole use of the settings table is wrong. I'll have to discuss this with Cigamit and determine the proper resolution in the code. Making the settings table too large, or even 'text' presents huge scalability problems for Cacti, so I don't want to have to set it too large.

TheWitness

[uhtred]

we did have the same problem this weekend, cause of many down devices for a test.
so i worked on a solution and here is my result:

1. new table in db --> plugin_thold_host_failed
2. change polling.php in thold/includes/polling.php

The only point, i dont know where the column thold_failed_hosts in table settings come from.
The diff's are based on files of thold v0.4.6

greetings

[TheWitness]

Crap. If forgot all about this for 0.4.6. Let me prepare 0.4.7 today.

TheWitness

[uhtred]

sry, there was a mistake in my polling.diff.
so i did upload the correct one.

[TheWitness]

Ok, doing the merge now.

[TheWitness]

Test SVN 1809 when it becomes available on the public facing SVN. It'll take a few minutes. I did things a bit differently as you will note.

TheWitness