Polling error after upgrade from version 1.2.22 to 1.2.23

[TheWitness]

Just lower the two negative values until the system comes up with a value above zero for both numbers. That's it. Might take you two restarts.

[acvfriend]

Seem it crash again when polling, I just stopped the polling and copy the log...

===
2023-01-04 10:15:01 - POLLER: Poller[1] PID[13362] Maximum runtime of 298 second
s exceeded. Exiting.
2023-01-04 10:15:01 - SYSTEM WARNING: Primary Admin account notifications disabl
ed! Unable to send administrative Email.
2023-01-04 10:15:01 - SYSTEM STATS: Time:299.5431 Method:cmd.php Processes:1 Thr
eads:1 Hosts:326 HostsPerProcess:326 DataSources:329 RRDsProcessed:0
2023-01-04 10:15:02 - SYSTEM MAINT STATS: Time:0.01
2023-01-04 10:15:28 - SYSTEM STATS: Time:27.3597 Method:cmd.php Processes:1 Thre
ads:1 Hosts:326 HostsPerProcess:326 DataSources:329 RRDsProcessed:329
2023-01-04 10:15:29 - CMDPHP ERROR: A DB Exec Failed!, Error: Table 'cacti.user_
auth_row_cache' doesn't exist
2023-01-04 10:15:29 - CMDPHP SQL Backtrace: (/poller_maintenance.php[117]:remov
e_aged_row_cache(), /poller_maintenance.php[186]:db_execute_prepared())
2023-01-04 10:15:29 - CMDPHP ERROR: A DB Exec Failed!, Error: Table 'cacti.user_
auth_row_cache' doesn't exist
2023-01-04 10:15:29 - CMDPHP SQL Backtrace: (/poller_maintenance.php[117]:remov
e_aged_row_cache(), /poller_maintenance.php[186]:db_execute_prepared())
2023-01-04 10:15:29 - SYSTEM MAINT STATS: Time:0.01
2023-01-04 10:20:29 - SYSTEM STATS: Time:27.3540 Method:cmd.php Processes:1 Thre
ads:1 Hosts:326 HostsPerProcess:326 DataSources:329 RRDsProcessed:329
2023-01-04 10:20:29 - CMDPHP ERROR: A DB Exec Failed!, Error: Table 'cacti.user_
auth_row_cache' doesn't exist
2023-01-04 10:20:29 - CMDPHP SQL Backtrace: (/poller_maintenance.php[117]:remov
e_aged_row_cache(), /poller_maintenance.php[186]:db_execute_prepared())
2023-01-04 10:20:29 - CMDPHP ERROR: A DB Exec Failed!, Error: Table 'cacti.user_
auth_row_cache' doesn't exist
2023-01-04 10:20:29 - CMDPHP SQL Backtrace: (/poller_maintenance.php[117]:remov
e_aged_row_cache(), /poller_maintenance.php[186]:db_execute_prepared())
2023-01-04 10:20:29 - SYSTEM MAINT STATS: Time:0.01
2023-01-04 10:22:30 - INSTALL: always: Spawning background process: /bin/php '/d
ata/www/html/cacti/install/background.php' 1672798950.6107
2023-01-04 10:22:30 - INSTALL: Checking arguments
2023-01-04 10:22:30 - INSTALL: always: Setting PHP Option max_execution_time = 0
2023-01-04 10:22:30 - INSTALL: always: Setting PHP Option memory_limit = -1
2023-01-04 10:22:31 - INSTALL: always: Starting UPGRADE Process for v1.2.23
2023-01-04 10:22:31 - INSTALL: always: No tables where found or selected for con
version
2023-01-04 10:22:31 - INSTALL: always: Switched from to /tmp/cduWu7xei
2023-01-04 10:22:31 - INSTALL: always: NOTE: Using temporary file for db cache:
/tmp/cduWu7xei
2023-01-04 10:22:31 - INSTALL: always: Upgrading from v1.2.22 (DB 1.2.23 (DB: 1.
2.22)) to v1.2.23
2023-01-04 10:22:31 - INSTALL: always: Found 11 templates to install
2023-01-04 10:22:31 - INSTALL: always: About to import Package #1 'Apache_Webser
ver.xml.gz'.
2023-01-04 10:22:32 - INSTALL: always: Import of Package #1 'Apache_Webserver.xm

[TheWitness]

Goto the CLI directory and run this:

Code: Select all

php -q upgrade_database.php --forcever=1.2.22

[acvfriend]

php -q upgrade_database.php --forcever=1.2.22
NOTE: Repairing Tables for Local Database
Upgrading from v1.2.22
Upgrading from v1.2.22 (DB 1.2.23) to v1.2.23
++[xxxxxx]#
[xxxxxx]#

[TheWitness]

The two missing tables should be there now.

[acvfriend]

Haha, another error come now after adjusted the sql setting value

===
2023-01-04 10:50:02 - SYSTEM WARNING: System Polling is Disabled! Therefore, data collection from the poller will be suspended till re-enabled.
2023-01-04 10:45:52 - CMDPHP SQL Backtrace: (/utilities.php[25]:include(), /include/auth.php[27]:require_once(), /include/global.php[532]:include_once(), /include/global_settings.php[62]:db_fetch_assoc(), /lib/database.php[677]:db_fetch_assoc_prepared(), /lib/database.php[697]:db_execute_prepared())
2023-01-04 10:45:52 - CMDPHP ERROR: A DB Row Failed!, Error: Out of sort memory, consider increasing server sort buffer size
2023-01-04 10:45:52 - CMDPHP SQL Backtrace: (/utilities.php[25]:include(), /include/auth.php[27]:require_once(), /include/global.php[531]:include_once(), /include/global_arrays.php[2923]:api_plugin_hook(), /lib/plugins.php[83]:api_plugin_run_plugin_hook(), /lib/plugins.php[225]:plugin_config_arrays(), /lib/plugins.php[1218]:api_plugin_load_realms(), /lib/plugins.php[1183]:db_fetch_assoc(), /lib/database.php[677]:db_fetch_assoc_prepared(), /lib/database.php[697]:db_execute_prepared())
2023-01-04 10:45:52 - CMDPHP ERROR: A DB Row Failed!, Error: Out of sort memory, consider increasing server sort buffer size
2023-01-04 10:45:52 - CMDPHP SQL Backtrace: (/utilities.php[25]:include(), /include/auth.php[27]:require_once(), /include/global.php[531]:include_once(), /include/global_arrays.php[1088]:db_fetch_assoc(), /lib/database.php[677]:db_fetch_assoc_prepared(), /lib/database.php[697]:db_execute_prepared())
2023-01-04 10:45:52 - CMDPHP ERROR: A DB Row Failed!, Error: Out of sort memory, consider increasing server sort buffer size
2023-01-04 10:45:50 - CMDPHP SQL Backtrace: (/utilities.php[25]:include(), /include/auth.php[27]:require_once(), /include/global.php[532]:include_once(), /include/global_settings.php[62]:db_fetch_assoc(), /lib/database.php[677]:db_fetch_assoc_prepared(), /lib/database.php[697]:db_execute_prepared())
2023-01-04 10:45:50 - CMDPHP ERROR: A DB Row Failed!, Error: Out of sort memory, consider increasing server sort buffer size
2023-01-04 10:45:50 - CMDPHP SQL Backtrace: (/utilities.php[25]:include(), /include/auth.php[27]:require_once(), /include/global.php[531]:include_once(), /include/global_arrays.php[2923]:api_plugin_hook(), /lib/plugins.php[83]:api_plugin_run_plugin_hook(), /lib/plugins.php[225]:plugin_config_arrays(), /lib/plugins.php[1218]:api_plugin_load_realms(), /lib/plugins.php[1183]:db_fetch_assoc(), /lib/database.php[677]:db_fetch_assoc_prepared(), /lib/database.php[697]:db_execute_prepared())
2023-01-04 10:45:50 - CMDPHP ERROR: A DB Row Failed!, Error: Out of sort memory, consider increasing server sort buffer size
2023-01-04 10:45:50 - CMDPHP SQL Backtrace: (/utilities.php[25]:include(), /include/auth.php[27]:require_once(), /include/global.php[531]:include_once(), /include/global_arrays.php[1088]:db_fetch_assoc(), /lib/database.php[677]:db_fetch_assoc_prepared(), /lib/database.php[697]:db_execute_prepared())
2023-01-04 10:45:50 - CMDPHP ERROR: A DB Row Failed!, Error: Out of sort memory, consider increasing server sort buffer size
2023-01-04 10:45:23 - CMDPHP SQL Backtrace: (/utilities.php[25]:include(), /include/auth.php[27]:require_once(), /include/global.php[532]:include_once(), /include/global_settings.php[62]:db_fetch_assoc(), /lib/database.php[677]:db_fetch_assoc_prepared(), /lib/database.php[697]:db_execute_prepared())
2023-01-04 10:45:23 - CMDPHP ERROR: A DB Row Failed!, Error: Out of sort memory, consider increasing server sort buffer size
2023-01-04 10:45:23 - CMDPHP SQL Backtrace: (/utilities.php[25]:include(), /include/auth.php[27]:require_once(), /include/global.php[531]:include_once(), /include/global_arrays.php[2923]:api_plugin_hook(), /lib/plugins.php[83]:api_plugin_run_plugin_hook(), /lib/plugins.php[225]:plugin_config_arrays(), /lib/plugins.php[1218]:api_plugin_load_realms(), /lib/plugins.php[1183]:db_fetch_assoc(), /lib/database.php[677]:db_fetch_assoc_prepared(), /lib/database.php[697]:db_execute_prepared())
2023-01-04 10:45:23 - CMDPHP ERROR: A DB Row Failed!, Error: Out of sort memory, consider increasing server sort buffer size
2023-01-04 10:45:23 - CMDPHP SQL Backtrace: (/utilities.php[25]:include(), /include/auth.php[27]:require_once(), /include/global.php[531]:include_once(), /include/global_arrays.php[1088]:db_fetch_assoc(), /lib/database.php[677]:db_fetch_assoc_prepared(), /lib/database.php[697]:db_execute_prepared())
2023-01-04 10:45:23 - CMDPHP ERROR: A DB Row Failed!, Error: Out of sort memory, consider increasing server sort buffer size
2023-01-04 10:45:21 - CMDPHP SQL Backtrace: (/utilities.php[25]:include(), /include/auth.php[27]:require_once(), /include/global.php[532]:include_once(), /include/global_settings.php[62]:db_fetch_assoc(), /lib/database.php[677]:db_fetch_assoc_prepared(), /lib/database.php[697]:db_execute_prepared())
2023-01-04 10:45:21 - CMDPHP ERROR: A DB Row Failed!, Error: Out of sort memory, consider increasing server sort buffer size
2023-01-04 10:45:21 - CMDPHP SQL Backtrace: (/utilities.php[25]:include(), /include/auth.php[27]:require_once(), /include/global.php[531]:include_once(), /include/global_arrays.php[2923]:api_plugin_hook(), /lib/plugins.php[83]:api_plugin_run_plugin_hook(), /lib/plugins.php[225]:plugin_config_arrays(), /lib/plugins.php[1218]:api_plugin_load_realms(), /lib/plugins.php[1183]:db_fetch_assoc(), /lib/database.php[677]:db_fetch_assoc_prepared(), /lib/database.php[697]:db_execute_prepared())
2023-01-04 10:45:21 - CMDPHP ERROR: A DB Row Failed!, Error: Out of sort memory, consider increasing server sort buffer size
2023-01-04 10:45:21 - CMDPHP SQL Backtrace: (/utilities.php[25]:include(), /include/auth.php[27]:require_once(), /include/global.php[531]:include_once(), /include/global_arrays.php[1088]:db_fetch_assoc(), /lib/database.php[677]:db_fetch_assoc_prepared(), /lib/database.php[697]:db_execute_prepared())
2023-01-04 10:45:21 - CMDPHP ERROR: A DB Row Failed!, Error: Out of sort memory, consider increasing server sort buffer size
===

[acvfriend]

Better now but still cannot polling within 300 seconds.

===
2023-01-04 11:05:00 - SYSTEM MAINT STATS: Time:0.01
2023-01-04 11:05:00 - SYSTEM STATS: Time:298.5888 Method:cmd.php Processes:1 Threads:1 Hosts:326 HostsPerProcess:326 DataSources:329 RRDsProcessed:0
2023-01-04 11:05:00 - SYSTEM WARNING: Primary Admin account notifications disabled! Unable to send administrative Email.
2023-01-04 11:05:00 - POLLER: Poller[1] PID[8022] Maximum runtime of 298 seconds exceeded. Exiting.
2023-01-04 11:00:00 - SYSTEM MAINT STATS: Time:0.01
2023-01-04 11:00:00 - SYSTEM STATS: Time:298.5945 Method:cmd.php Processes:1 Threads:1 Hosts:326 HostsPerProcess:326 DataSources:329 RRDsProcessed:0
===

[Osiris]

Use spine.

[acvfriend]

Install spine and test as below

===
2023-01-04 13:05:02 - SYSTEM MAINT STATS: Time:0.01
2023-01-04 13:05:01 - SYSTEM STATS: Time:299.5981 Method:spine Processes:1 Threads:1 Hosts:326 HostsPerProcess:326 DataSources:329 RRDsProcessed:0
===

[acvfriend]

At the end I changed the crontab command from run by user <cacti> to <root>, also add "--force" after polling.php, seem work now.

*/5 * * * * root /usr/bin/php /var/www/html/cacti/poller.php --force > /dev/null 2>&1

===
2023-01-04 15:10:29 - SYSTEM MAINT STATS: Time:0.01
2023-01-04 15:10:29 - SYSTEM STATS: Time:27.3584 Method:spine Processes:1 Threads:1 Hosts:326 HostsPerProcess:326 DataSources:329 RRDsProcessed:328
2023-01-04 15:06:26 - SYSTEM MAINT STATS: Time:0.01
2023-01-04 15:06:25 - SYSTEM STATS: Time:30.3684 Method:spine Processes:1 Threads:1 Hosts:326 HostsPerProcess:326 DataSources:329 RRDsProcessed:328
===

[acvfriend]

Seem something wrong on rrdcleaner.php, but not effect to polling

===

2023-01-04 15:36:58 - CMDPHP SQL Backtrace: (/rrdcleaner.php[72]:list_rrd(), /rrdcleaner.php[368]:db_fetch_assoc(), /lib/database.php[677]:db_fetch_assoc_prepared(), /lib/database.php[697]:db_execute_prepared())
2023-01-04 15:36:58 - CMDPHP ERROR: A DB Row Failed!, Error: Unknown column 'test_date' in 'order clause'

===

[TheWitness]

Go to the documentation and learn how to "test" spine. Documentation can be found at https://docs.cacti.net. That bit was logged in GitHub earlier this morning. Your issue is mainly training at this point. You'll be find after you stumble a few more times. Just cutting your teeth.

[cigamit]

Method:spine Processes:1 Threads:1

You are still running everything on a single process and thread, that's going to take forever on any poller for that many devices.

Goto Data Collection > Data Pollers > Main Poller and increase the number of processes higher (I normally match the number of CPUs in my server). Then set threads to around 10.