spine 1.2.9 regularly crashing

[TheWitness]

That seems real odd. The is a repair_databse.php, run it is non force mosde and post the output, don't repair any template issues, just post the output.

[hektor]

root@graph-1 07:43:40 ~> php -q /usr/local/cacti/cli/repair_database.php
Repairing All Cacti Database Tables
Repairing Table -> 'aggregate_graph_templates' Successful
Repairing Table -> 'aggregate_graph_templates_graph' Successful
Repairing Table -> 'aggregate_graph_templates_item' Successful
Repairing Table -> 'aggregate_graphs' Successful
Repairing Table -> 'aggregate_graphs_graph_item' Successful
Repairing Table -> 'aggregate_graphs_items' Successful
Repairing Table -> 'automation_devices' Successful
Repairing Table -> 'automation_graph_rule_items' Successful
Repairing Table -> 'automation_graph_rules' Successful
Repairing Table -> 'automation_ips' Successful
Repairing Table -> 'automation_match_rule_items' Successful
Repairing Table -> 'automation_networks' Successful
Repairing Table -> 'automation_processes' Successful
Repairing Table -> 'automation_snmp' Successful
Repairing Table -> 'automation_snmp_items' Successful
Repairing Table -> 'automation_templates' Successful
Repairing Table -> 'automation_tree_rule_items' Successful
Repairing Table -> 'automation_tree_rules' Successful
Repairing Table -> 'cdef' Successful
Repairing Table -> 'cdef_items' Successful
Repairing Table -> 'color_template_items' Successful
Repairing Table -> 'color_templates' Successful
Repairing Table -> 'colors' Successful
Repairing Table -> 'data_debug' Successful
Repairing Table -> 'data_input' Successful
Repairing Table -> 'data_input_data' Successful
Repairing Table -> 'data_input_fields' Successful
Repairing Table -> 'data_local' Successful
Repairing Table -> 'data_source_profiles' Successful
Repairing Table -> 'data_source_profiles_cf' Successful
Repairing Table -> 'data_source_profiles_rra' Successful
Repairing Table -> 'data_source_purge_action' Successful
Repairing Table -> 'data_source_purge_temp' Successful
Repairing Table -> 'data_source_stats_daily' Successful
Repairing Table -> 'data_source_stats_hourly' Successful
Repairing Table -> 'data_source_stats_hourly_cache' Successful
Repairing Table -> 'data_source_stats_hourly_last' Successful
Repairing Table -> 'data_source_stats_monthly' Successful
Repairing Table -> 'data_source_stats_weekly' Successful
Repairing Table -> 'data_source_stats_yearly' Successful
Repairing Table -> 'data_template' Successful
Repairing Table -> 'data_template_data' Successful
Repairing Table -> 'data_template_rrd' Successful
Repairing Table -> 'external_links' Successful
Repairing Table -> 'graph_local' Successful
Repairing Table -> 'graph_template_input' Successful
Repairing Table -> 'graph_template_input_defs' Successful
Repairing Table -> 'graph_templates' Successful
Repairing Table -> 'graph_templates_gprint' Successful
Repairing Table -> 'graph_templates_graph' Successful
Repairing Table -> 'graph_templates_item' Successful
Repairing Table -> 'graph_tree' Successful
Repairing Table -> 'graph_tree_items' Successful
Repairing Table -> 'host' Successful
Repairing Table -> 'host_graph' Successful
Repairing Table -> 'host_snmp_cache' Successful
Repairing Table -> 'host_snmp_query' Successful
Repairing Table -> 'host_template' Successful
Repairing Table -> 'host_template_graph' Successful
Repairing Table -> 'host_template_snmp_query' Successful
Repairing Table -> 'plugin_config' Successful
Repairing Table -> 'plugin_db_changes' Successful
Repairing Table -> 'plugin_hooks' Successful
Repairing Table -> 'plugin_intropage_panel' Successful
Repairing Table -> 'plugin_intropage_trends' Successful
Repairing Table -> 'plugin_intropage_user_setting' Successful
Repairing Table -> 'plugin_realms' Successful
Repairing Table -> 'plugin_rrdclean' Successful
Repairing Table -> 'plugin_rrdclean_action' Successful
Repairing Table -> 'plugin_spikekill_templates' Successful
Repairing Table -> 'poller' Successful
Repairing Table -> 'poller_command' Successful
Repairing Table -> 'poller_data_template_field_mappings' Successful
Repairing Table -> 'poller_item' Successful
Repairing Table -> 'poller_output' Successful
Repairing Table -> 'poller_output_boost' Successful
Repairing Table -> 'poller_output_boost_arch_1587961417' Successful
Repairing Table -> 'poller_output_boost_processes' Successful
Repairing Table -> 'poller_output_realtime' Successful
Repairing Table -> 'poller_reindex' Successful
Repairing Table -> 'poller_resource_cache' Successful
Repairing Table -> 'poller_time' Successful
Repairing Table -> 'processes' Successful
Repairing Table -> 'reports' Successful
Repairing Table -> 'reports_items' Successful
Repairing Table -> 'sessions' Successful
Repairing Table -> 'settings' Successful
Repairing Table -> 'settings_tree' Successful
Repairing Table -> 'settings_user' Successful
Repairing Table -> 'settings_user_group' Successful
Repairing Table -> 'sites' Successful
Repairing Table -> 'snmp_query' Successful
Repairing Table -> 'snmp_query_graph' Successful
Repairing Table -> 'snmp_query_graph_rrd' Successful
Repairing Table -> 'snmp_query_graph_rrd_sv' Successful
Repairing Table -> 'snmp_query_graph_sv' Successful
Repairing Table -> 'snmpagent_cache' Successful
Repairing Table -> 'snmpagent_cache_notifications' Successful
Repairing Table -> 'snmpagent_cache_textual_conventions' Successful
Repairing Table -> 'snmpagent_managers' Successful
Repairing Table -> 'snmpagent_managers_notifications' Successful
Repairing Table -> 'snmpagent_mibs' Successful
Repairing Table -> 'snmpagent_notifications_log' Successful
Repairing Table -> 'user_auth' Successful
Repairing Table -> 'user_auth_cache' Successful
Repairing Table -> 'user_auth_group' Successful
Repairing Table -> 'user_auth_group_members' Successful
Repairing Table -> 'user_auth_group_perms' Successful
Repairing Table -> 'user_auth_group_realm' Successful
Repairing Table -> 'user_auth_perms' Successful
Repairing Table -> 'user_auth_realm' Successful
Repairing Table -> 'user_domains' Successful
Repairing Table -> 'user_domains_ldap' Successful
Repairing Table -> 'user_log' Successful
Repairing Table -> 'vdef' Successful
Repairing Table -> 'vdef_items' Successful
Repairing Table -> 'version' Successful

NOTE: Running some Data Query repair scripts
NOTE: Checking for Invalid Cacti Templates
NOTE: 121 Invalid Data Input Data Rows based upon template mappings found in Data Templates
NOTE: 540 Invalid Data Input Data rows based upon field mappings found in Data Templates

WARNING: Cacti Template Problems found in your Database. Using the '--force' option will remove
the invalid records. However, these changes can be catastrophic to existing data sources. Therefore, you
should contact your support organization prior to proceeding with that repair.

[hektor]

Rebuild poller cash does not help.

Or maybe it did?

The messages have vanished two hours ago.

Poller rebuild for this host was way before.
Hours (some of them) later?

At the moment I am still in the process of recovery
(only have 2k or my 3-4k systems back in cacti)

Adding new systems is pretty slow (hours),
putting something new in my "trees" lasts (hours?)

And just I did the upgrade for centos 7.,7.

My feeling is that mnariadb might be a real bottleneck.
Just to give you a class of may scaling:

2100 Hosts, 280k Data Sources & Diagrams.
Just want to get clean before adding another 1k (mostly minor devices)

I am running three pollers now (one with important systems (~40)
one just with one (at the moment) and the rest on the main poller
with the DB on it (~around 30s spine runtime, 32 CPUs).

[hektor]

Alright.

Issued a rebuild on poller cache on all those "nasty" hosts
with their "empty results"-

Will wait. Patiently. I'll be back in 24h - or earlier, if it helped.

[bmfmancini]

Hey Hey

I have this exact issue I reported that on git

*** Error in `/usr/local/spine/bin/spine': double free or corruption (out): 0x00007f45d8007540 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81679)[0x7f4615af3679]
/usr/lib64/mysql/libmysqlclient.so.18(+0x256d5)[0x7f4616c006d5]
/usr/lib64/mysql/libmysqlclient.so.18(+0x2a3d6)[0x7f4616c053d6]
/usr/lib64/mysql/libmysqlclient.so.18(+0x27b84)[0x7f4616c02b84]
/usr/lib64/mysql/libmysqlclient.so.18(mysql_close+0x1c)[0x7f4616c02bbc]
/usr/local/spine/bin/spine[0x40cf3d]
/usr/local/spine/bin/spine[0x40f2ac]
/lib64/libpthread.so.0(+0x7e65)[0x7f461634de65]
/lib64/libc.so.6(clone+0x6d)[0x7f4615b7088d]

https://github.com/Cacti/spine/issues/15

I dont have the same db issues but periodically I get the same issues along with 2006 SQL server has gone away right at the same time the above errors show up

[hektor]

"The empy result" messages are gone. Rebuild poller cache helped, although it lasted a bit.

And indeed: spine seems to crash on connectivity issues with the SQL server.

Anyway: my problems are solved at the moment.

Thx for the patience and the good hints!

[netniV]

Please try updating to 1.2.12 and see if that resolves the issue.