Failed to update in Graph and the Data in Threshold

[jason23]

Hello Sir,

Recently I have upgrade from v0.8.8h to v1.2.27, after being upgraded, I can see the Graph Data for a short period of time, but now it was failed to update with the latest data information.

I have tried to upgrade from cmc.php to spine, but also failed to solve the issue.

May I know if anyone can provide some solution for me?

Cacti Version: v1.2.27
PHP 5.6.40 (cli)
MySQL 5.7.43

Cacti Version 1.2.27
Cacti OS win32
NET-SNMP Version NET-SNMP version: 5.5
RRDtool Version Configured 1.4.0+
RRDtool Version Found 1.4.5
Devices 463
Graphs 5,947
Data Sources SNMP Get: 2,028
SNMP Query: 4,729
Script Query: 4
Script Server: 2
Total: 6,763
Interval 300
Type SPINE 1.2.27 Copyright 2004-2023 by The Cacti Group
Items Action[0]: 14,162
Action[2]: 2
Total: 14,164
Concurrent Processes Name: Main Poller, Procs: 10
Max Threads Name: Main Poller, Threads: 60
PHP Servers 15
Minimum Connections: Main Server: Current: 100000, Min Required: 850
Assumes 100 spare connections for Web page users and other various connections.
The minimum required can vary greatly if there is heavy user Graph viewing activity.
Each browser tab can use upto 10 connections depending on the browser.
Script Timeout 60
Max OID 10
Last Run Statistics Time:298.9038 Method:spine Processes:10 Threads:60 Hosts:458 HostsPerProcess:46 DataSources:14164 RRDsProcessed:0

Virtual Server Config:
60 CPUs
48GB Ram

Poller Settings:
Number of PHP Script Servers = 15
Data Collector Processes = 60
Threads per Process = 40

Cacti Log

2024-05-28 16:20:02 - SYSTEM STATS: Time:298.7260 Method:spine Processes:10 Threads:60 Hosts:458 HostsPerProcess:46 DataSources:14164 RRDsProcessed:0
2024-05-28 16:20:02 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-05-28 16:20:02 - POLLER: Poller[Main Poller] PID[9360] Maximum runtime of 298 seconds exceeded. Exiting.
2024-05-28 16:15:19 - SYSTEM STATS: Time:298.9038 Method:spine Processes:10 Threads:60 Hosts:458 HostsPerProcess:46 DataSources:14164 RRDsProcessed:0
2024-05-28 16:15:19 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-05-28 16:15:19 - POLLER: Poller[Main Poller] PID[2064] Maximum runtime of 298 seconds exceeded. Exiting.
2024-05-28 16:15:01 - SYSTEM STATS: Time:298.8368 Method:spine Processes:10 Threads:60 Hosts:458 HostsPerProcess:46 DataSources:14164 RRDsProcessed:0
2024-05-28 16:15:01 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-05-28 16:15:01 - POLLER: Poller[Main Poller] PID[5820] Maximum runtime of 298 seconds exceeded. Exiting.

Thank you.

Jason

[macan]

find in forums "gaps in graph" and "poller overrun".

You should update php a net-snmp too

[jason23]

find in forums "gaps in graph" and "poller overrun".

You should update php a net-snmp too

Hi Macan,

I have changed with the below amount, but it still being with the same result.

max_heap_table_size = 800M ==> 2G

2024-05-28 18:05:03 - SYSTEM STATS: Time:298.8379 Method:spine Processes:10 Threads:60 Hosts:458 HostsPerProcess:46 DataSources:14164 RRDsProcessed:0
2024-05-28 18:05:03 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-05-28 18:05:03 - POLLER: Poller[Main Poller] PID[9124] Maximum runtime of 298 seconds exceeded. Exiting.
2024-05-28 18:00:01 - SYSTEM STATS: Time:298.6973 Method:spine Processes:10 Threads:60 Hosts:458 HostsPerProcess:46 DataSources:14164 RRDsProcessed:0
2024-05-28 18:00:01 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2024-05-28 18:00:01 - POLLER: Poller[Main Poller] PID[11640] Maximum runtime of 298 seconds exceeded. Exiting.

Am I need to upgrade to the latest version of PHP first??
VS16 x64 Thread Safe (2024-May-08 09:15:56) ???

Thank you

[macan]

PHP 7.4 or 8.x

[jason23]

PHP 7.4 or 8.x

Thanks Macan, let me have a try to install the PHP 8.3 (8.3.7).

I was installed with the current 5.x PHP Version because of I have failed to access with the Installation Page of Cacti (From 0.8.8h to 1.2.27 with PHP 8.3 before. Always showing > PHP Module Issue / Failed to access Database / Check with config.php.

[macan]

did you install php mysqli module?

Sorry, I made a mistake, mysql_pdo is correct module.

[jason23]

did you install php mysqli module?

I didn't enable the "mysqli" module, and the below are the current module setting

;extension=php_bz2.dll
;extension=php_curl.dll
extension=php_com_dotnet.dll
;extension=php_fileinfo.dll
extension=php_gd2.dll
extension=php_gettext.dll
extension=php_gmp.dll
extension=php_intl.dll
;extension=php_imap.dll
;extension=php_interbase.dll
extension=php_ldap.dll
extension=php_mbstring.dll
extension=php_exif.dll ; Must be after mbstring as it depends on it
;extension=php_mysql.dll
;extension=php_mysqli.dll
;extension=php_oci8_12c.dll ; Use with Oracle Database 12c Instant Client
extension=php_openssl.dll
;extension=php_pdo_firebird.dll
extension=php_pdo_mysql.dll
;extension=php_pdo_oci.dll
;extension=php_pdo_odbc.dll
;extension=php_pdo_pgsql.dll
;extension=php_pdo_sqlite.dll
;extension=php_pgsql.dll
;extension=php_shmop.dll

; The MIBS data available in the PHP distribution must be installed.
; See http://www.php.net/manual/en/snmp.installation.php
extension=php_snmp.dll

;extension=php_soap.dll
extension=php_sockets.dll
;extension=php_sqlite3.dll
;extension=php_sybase_ct.dll
;extension=php_tidy.dll
;extension=php_xmlrpc.dll
;extension=php_xsl.dll

Thank you.

[jason23]

Or would it be the problem of the old RRDtool??

RRDtool Version Found 1.4.5

[macan]

in your case it's not about server performance. Similar cacti installation works fine (more than 400 devices), 6 cpu, 12GB RAM:
Concurrent Processes Name: Main Poller, Procs: 2
Max Threads Name: Main Poller, Threads: 10
PHP Servers 3
Minimum Connections: Main Server: Current: 151, Min Required: 126
Script Timeout 30
Max OID 35

You can try:
- enable boost (configuration -> settings -> performance)
- increase log level and check logs
- change max OID, script timeou, device snmp and ping timeout
- check dns resolving

[jason23]

in your case it's not about server performance. Similar cacti installation works fine (more than 400 devices), 6 cpu, 12GB RAM:
Concurrent Processes Name: Main Poller, Procs: 2
Max Threads Name: Main Poller, Threads: 10
PHP Servers 3
Minimum Connections: Main Server: Current: 151, Min Required: 126
Script Timeout 30
Max OID 35

You can try:
- enable boost (configuration -> settings -> performance)
- increase log level and check logs
- change max OID, script timeou, device snmp and ping timeout
- check dns resolving

Thanks Macan, I have tried to revert the Instance VM Snapshot to the previous status, I will try to update with PHP, Spine, and other setting one by one. Avoid hitting the issue again.

[TheWitness]

6 CPU's and 600 threads. I'm not surprised it's not working. I know many customers with over 30k devices that poll with 4-5 processes and 20 threads, but on servers with 100 threads+