Threshold Notification not work after upgrade to Cacti v1.2.27 with Thold (1.8.1)

[jason23]

Hello Sir,

I am able to send to Test Mail to the mail box. But the alert notification mail cannot be sent even the Threshold is being triggered.

May I know if anyone can provide some solution for me?

Cacti Version: v1.2.27
PHP 5.6.40 (cli)
MySQL 5.7.43

Thank you.

Jason

2024-05-25 19:56:13 - SYSTEM THOLD STATS: Time:0.85 Tholds:251 TotalDevices:458 DownDevices:4 NewDownDevices:0
2024-05-25 19:56:13 - SYSTEM THOLD STATS: Time:0.85 Tholds:251 TotalDevices:458 DownDevices:4 NewDownDevices:0
2024-05-25 19:56:12 - POLLER: Poller[Main Poller] PID[12008] WARNING: Detected process that is exited and did not unregister first! (thold, master, 1, 12008)
2024-05-25 19:56:12 - SYSTEM MAINT STATS: Time:0.02
2024-05-25 19:56:12 - SYSTEM MAINT STATS: Time:0.01
2024-05-25 19:56:12 - CMDPHP SQL Backtrace: (D:\Inetpub\wwwroot\cacti\poller_reports.php[148]:db_fetch_assoc_prepared(), D:\Inetpub\wwwroot\cacti\lib\database.php[727]:db_execute_prepared())
2024-05-25 19:56:12 - CMDPHP ERROR: A DB Row Failed!, Error: Unknown column 'enabled' in 'where clause'
2024-05-25 19:56:12 - CMDPHP SQL Backtrace: (D:\Inetpub\wwwroot\cacti\poller_reports.php[148]:db_fetch_assoc_prepared(), D:\Inetpub\wwwroot\cacti\lib\database.php[727]:db_execute_prepared())
2024-05-25 19:56:12 - CMDPHP ERROR: A DB Row Failed!, Error: Unknown column 'enabled' in 'where clause'
2024-05-25 19:56:10 - SNMPAGENT WARNING: No notification receivers configured for event: cactiNotifyDeviceFailedPoll (CACTI-MIB), severity: medium
2024-05-25 19:56:10 - SNMPAGENT WARNING: No notification receivers configured for event: cactiNotifyDeviceFailedPoll (CACTI-MIB), severity: medium
2024-05-25 19:56:10 - SYSTEM STATS: Time:67.9653 Method:cmd.php Processes:10 Threads:1 Hosts:458 HostsPerProcess:46 DataSources:14164 RRDsProcessed:0
2024-05-25 19:56:10 - SYSTEM STATS: Time:67.8780 Method:cmd.php Processes:10 Threads:1 Hosts:458 HostsPerProcess:46 DataSources:14164 RRDsProcessed:0
2024-05-25 19:55:05 - CMDPHP WARNING: cmd.php poller has run over its polling interval and therefore is ending
2024-05-25 19:55:05 - CMDPHP WARNING: cmd.php poller has run over its polling interval and therefore is ending
2024-05-25 19:55:04 - CMDPHP WARNING: cmd.php poller has run over its polling interval and therefore is ending
2024-05-25 19:55:03 - CMDPHP WARNING: cmd.php poller has run over its polling interval and therefore is ending
2024-05-25 19:55:02 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-05-25 19:55:02 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.

[TheWitness]

Run the cli script:

Code: Select all

php -q upgrade_database.php --forcever=0.8.8

That should add that column.

[TheWitness]

You might want to increase the concurrent processes or go to spine, or linux + spine.

[jason23]

Run the cli script:

Code: Select all

php -q upgrade_database.php --forcever=0.8.8

That should add that column.

Thanks Witness, I am now in Cacti v1.2.27, you mean I need to run this CLI too?

[jason23]

You might want to increase the concurrent processes or go to spine, or linux + spine.

As the server is being with 60 Virtual CPUs with 48GB Ram, so I have set the "Poller Settings" as below, or do you have any suggestion for me?

Poller Type: cmd.php
Poller Sync Interval: Every 2 hours
Poller Interval: Every 5 mins
Cron/Daemon Interval: Every 5 mins

Number of PHP Script Servers: 10
Script and Script Server Timeout Value: 60

Data Collector Processes: 120
Threads per Process:60

Thanks so much

[TheWitness]

Well, with cmd.php you need to edit the data collector and increase the processes. Most of the rest of that is relevant to spine.

[jason23]

Well, with cmd.php you need to edit the data collector and increase the processes. Most of the rest of that is relevant to spine.

Thanks Witness, I can receive the email alert after back the Email Address for the Admin Account.

But I discover that most of the Threshold "Current Bandwidth" is not correct.
The Link should just only have 1G Bandwidth, but the Current it shows as 7T. So may be that is the reason to trigger with the alert.

And many of the Graphs are only update intermittently, I don't know how to fix with the issue.

Thank you.

[jason23]

I have tried to revert the Instance VM Snapshot to the previous status, and now the Graph and Threshold Data can be refreshed. And I have added the email address for the "admin" account, and created a "Notification List", but the alert notification email still cannot send to the below.

"Warning Notification List"
"Alert Notification List"
"Notify accounts"

And I discover that the "Current" show in Threshold is not match with the Graph showing.

Now still being with OLD PHP Version (5.6.40), and using with cmd.php to POLL.

[jason23]

I have upgraded to PHP 8.3.7, and made some changes as below, and the Graph and Threshold Data become accurate now, and the notification mail also can be sent.

php.ini >>
cgi.force_redirect = 0 (From 1 to 0)
memory_limit = 800M (From 128M to 800M)
max_execution_time = 300 (From 60 to 300)

my.ini >>
sql_mode=NO_ENGINE_SUBSTITUTION,NO_AUTO_CREATE_USER

Thanks so much.

[jason23]

Sorry that I have encounter the problem again, the Threshold cannot be trigger as Alert (RED Color), but just showing as Notice (Yellow), and I am not able to receive the alert email too.
And even if the Threshold is being triggered as ALERT (Red Color), I also need to wait for a long long time to receive the alert mail by lucky (Not sure it will send it out).

Would anyone can help me to solve this issue?

Date Tue, 04 Jun 2024 13:32:08 +0800
Cacti Version 1.2.27
Cacti OS win32
NET-SNMP Version NET-SNMP version: 5.5
RRDtool Version Configured 1.4.0+
RRDtool Version Found 1.4.5
Devices 460
Graphs 5,940
Data Sources SNMP Get: 2,027
SNMP Query: 4,726
Script Query: 4
Script Server: 2
Total: 6,759
****************************************************************************************
Interval 300
Type cmd.php
Items Action[0]: 14,155
Action[2]: 2
Total: 14,157
Concurrent Processes Name: Main Poller, Procs: 10
Max Threads Name: Main Poller, Threads: 60
PHP Servers 10
Minimum Connections: Main Server: Current: 3000, Min Required: 800
Assumes 100 spare connections for Web page users and other various connections.
The minimum required can vary greatly if there is heavy user Graph viewing activity.
Each browser tab can use upto 10 connections depending on the browser.
Script Timeout 60
Max OID 10
Last Run Statistics Time:298.4994 Method:cmd.php Processes:10 Threads:1 Hosts:454 HostsPerProcess:46 DataSources:14156 RRDsProcessed:0
****************************************************************************************
PHP Version 8.3.7
PHP OS WINNT
PHP uname Windows NT HKUTMPWV724 10.0 build 14393 (Windows Server 2016) AMD64
PHP SNMP Installed. Note: If you are planning on using SNMPv3, you must remove php-snmp and use the Net-SNMP toolset.
max_execution_time 300
memory_limit 800M

Cacti Error Log

2024-06-04 13:30:06 - POLLER: Poller[Main Poller] PID[9104] WARNING: There are 1 processes detected as overrunning a polling cycle, please investigate
2024-06-04 13:30:06 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-06-04 13:30:06 - POLLER: Poller[Main Poller] PID[9104] WARNING: Scheduled Task is out of sync with the Poller Interval! The Poller Interval is '300' seconds, with a maximum of a '300' second Scheduled Task, but 305.2 seconds have passed since the last poll!
2024-06-04 13:30:03 - CMDPHP WARNING: cmd.php poller has run over its polling interval and therefore is ending
2024-06-04 13:30:02 - CMDPHP WARNING: cmd.php poller has run over its polling interval and therefore is ending
2024-06-04 13:30:00 - SYSTEM STATS: Time:298.4994 Method:cmd.php Processes:10 Threads:1 Hosts:454 HostsPerProcess:46 DataSources:14156 RRDsProcessed:0
2024-06-04 13:30:00 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-06-04 13:30:00 - POLLER: Poller[Main Poller] PID[9504] Maximum runtime of 298 seconds exceeded. Exiting.
2024-06-04 13:25:04 - SYSTEM THOLD STATS: Time:3.36 Tholds:1111 TotalDevices:454 DownDevices:0 NewDownDevices:0
2024-06-04 13:25:04 - CMDPHP PHP ERROR Backtrace: (D:\Inetpub\wwwroot\cacti\plugins\thold\poller_thold.php[99]:perform_thold_processes(), D:\Inetpub\wwwroot\cacti\plugins\thold\poller_thold.php[154]:thold_check_all_thresholds(), D:\Inetpub\wwwroot\cacti\plugins\thold\includes\polling.php[332]:thold_check_threshold(), D:\Inetpub\wwwroot\cacti\plugins\thold\thold_functions.php[2219]:get_thold_emails(), D:\Inetpub\wwwroot\cacti\plugins\thold\thold_functions.php[7596]:CactiErrorHandler())
2024-06-04 13:25:04 - ERROR PHP DEPRECATED: strlen(): Passing null to parameter #1 ($string) of type string is deprecated in file: D:\Inetpub\wwwroot\cacti\plugins\thold\thold_functions.php on line: 7596
2024-06-04 13:25:04 - CMDPHP PHP ERROR Backtrace: (D:\Inetpub\wwwroot\cacti\plugins\thold\poller_thold.php[99]:perform_thold_processes(), D:\Inetpub\wwwroot\cacti\plugins\thold\poller_thold.php[154]:thold_check_all_thresholds(), D:\Inetpub\wwwroot\cacti\plugins\thold\includes\polling.php[332]:thold_check_threshold(), D:\Inetpub\wwwroot\cacti\plugins\thold\thold_functions.php[2219]:get_thold_emails(), D:\Inetpub\wwwroot\cacti\plugins\thold\thold_functions.php[7596]:CactiErrorHandler())
2024-06-04 13:25:04 - ERROR PHP DEPRECATED: strlen(): Passing null to parameter #1 ($string) of type string is deprecated in file: D:\Inetpub\wwwroot\cacti\plugins\thold\thold_functions.php on line: 7596
2024-06-04 13:25:04 - CMDPHP PHP ERROR Backtrace: (D:\Inetpub\wwwroot\cacti\plugins\thold\poller_thold.php[99]:perform_thold_processes(), D:\Inetpub\wwwroot\cacti\plugins\thold\poller_thold.php[154]:thold_check_all_thresholds(), D:\Inetpub\wwwroot\cacti\plugins\thold\includes\polling.php[332]:thold_check_threshold(), D:\Inetpub\wwwroot\cacti\plugins\thold\thold_functions.php[2219]:get_thold_emails(), D:\Inetpub\wwwroot\cacti\plugins\thold\thold_functions.php[7596]:CactiErrorHandler())
2024-06-04 13:25:04 - ERROR PHP DEPRECATED: strlen(): Passing null to parameter #1 ($string) of type string is deprecated in file: D:\Inetpub\wwwroot\cacti\plugins\thold\thold_functions.php on line: 7596
2024-06-04 13:25:04 - CMDPHP PHP ERROR Backtrace: (D:\Inetpub\wwwroot\cacti\plugins\thold\poller_thold.php[99]:perform_thold_processes(), D:\Inetpub\wwwroot\cacti\plugins\thold\poller_thold.php[154]:thold_check_all_thresholds(), D:\Inetpub\wwwroot\cacti\plugins\thold\includes\polling.php[332]:thold_check_threshold(), D:\Inetpub\wwwroot\cacti\plugins\thold\thold_functions.php[2219]:get_thold_emails(), D:\Inetpub\wwwroot\cacti\plugins\thold\thold_functions.php[7596]:CactiErrorHandler())
2024-06-04 13:25:04 - ERROR PHP DEPRECATED: strlen(): Passing null to parameter #1 ($string) of type string is deprecated in file: D:\Inetpub\wwwroot\cacti\plugins\thold\thold_functions.php on line: 7596
2024-06-04 13:25:04 - CMDPHP PHP ERROR Backtrace: (D:\Inetpub\wwwroot\cacti\plugins\thold\poller_thold.php[99]:perform_thold_processes(), D:\Inetpub\wwwroot\cacti\plugins\thold\poller_thold.php[154]:thold_check_all_thresholds(), D:\Inetpub\wwwroot\cacti\plugins\thold\includes\polling.php[332]:thold_check_threshold(), D:\Inetpub\wwwroot\cacti\plugins\thold\thold_functions.php[2219]:get_thold_emails(), D:\Inetpub\wwwroot\cacti\plugins\thold\thold_functions.php[7596]:CactiErrorHandler())
2024-06-04 13:25:04 - ERROR PHP DEPRECATED: strlen(): Passing null to parameter #1 ($string) of type string is deprecated in file: D:\Inetpub\wwwroot\cacti\plugins\thold\thold_functions.php on line: 7596
2024-06-04 13:25:04 - CMDPHP PHP ERROR Backtrace: (D:\Inetpub\wwwroot\cacti\plugins\thold\poller_thold.php[99]:perform_thold_processes(), D:\Inetpub\wwwroot\cacti\plugins\thold\poller_thold.php[154]:thold_check_all_thresholds(), D:\Inetpub\wwwroot\cacti\plugins\thold\includes\polling.php[332]:thold_check_threshold(), D:\Inetpub\wwwroot\cacti\plugins\thold\thold_functions.php[2219]:get_thold_emails(), D:\Inetpub\wwwroot\cacti\plugins\thold\thold_functions.php[7596]:CactiErrorHandler())
2024-06-04 13:25:04 - ERROR PHP DEPRECATED: strlen(): Passing null to parameter #1 ($string) of type string is deprecated in file: D:\Inetpub\wwwroot\cacti\plugins\thold\thold_functions.php on line: 7596
2024-06-04 13:25:04 - CMDPHP WARNING: cmd.php poller has run over its polling interval and therefore is ending

2024-06-04 14:30:00 - POLLER: Poller[Main Poller] PID[9600] Maximum runtime of 298 seconds exceeded. Exiting.
2024-06-04 14:29:17 - CMDPHP PHP ERROR WARNING Backtrace: (D:\Inetpub\wwwroot\cacti\plugins\thold\thold.php[144]:thold_edit(), D:\Inetpub\wwwroot\cacti\plugins\thold\thold.php[2266]:draw_edit_form(), D:\Inetpub\wwwroot\cacti\lib\html_form.php[57]:CactiErrorHandler())
2024-06-04 14:29:17 - ERROR PHP WARNING: Undefined array key "default" in file: D:\Inetpub\wwwroot\cacti\lib\html_form.php on line: 57
2024-06-04 14:29:17 - CMDPHP form_edit Backtrace: (D:\Inetpub\wwwroot\cacti\plugins\thold\thold.php[144]:thold_edit(), D:\Inetpub\wwwroot\cacti\plugins\thold\thold.php[2266]:draw_edit_form(), D:\Inetpub\wwwroot\cacti\lib\html_form.php[56]:cacti_debug_backtrace())

Thank you.

[SynDeddy]

Bro, i have same problem here. Email alert doesn't send it, just always pending all the time.
Send test email was normal, reporting works. Anyone can help me?

1.jpg (351.59 KiB) Viewed 3597 times

[jason23]

I have tried to increase the below number, and it is able to send the alert mail.

cmd.php

Collection Settings
Processes = From 10 to 40
Data Collector Processes = From 200 to 120