spine 1.2.9 regularly crashing

[TheWitness]

Hector, 1.2.11 is officially released.

[hektor]

Thx. Installed. Still crashing.

Will try to find out how I get a core file.

[hektor]

Finally I have it:

the line in question is in sql.c:333

Here is the outpout from gdb:
------------------------

Core was generated by `/usr/local/spine/bin/spine -C /usr/local/spine/etc/spine.conf --mode online --p'.
Program terminated with signal 6, Aborted.

(gdb) where
#0 0x00007fb9f4de7337 in raise () from /lib64/libc.so.6
#1 0x00007fb9f4de8a28 in abort () from /lib64/libc.so.6
#2 0x00007fb9f4e29e87 in __libc_message () from /lib64/libc.so.6
#3 0x00007fb9f4e32679 in _int_free () from /lib64/libc.so.6
#4 0x00007fb9f61d93e7 in vio_delete () from /usr/lib64/mysql/libmysqlclient.so.18
#5 0x00007fb9f61b16d5 in end_server () from /usr/lib64/mysql/libmysqlclient.so.18
#6 0x00007fb9f61b63d6 in cli_advanced_command () from /usr/lib64/mysql/libmysqlclient.so.18
#7 0x00007fb9f61b3b84 in mysql_close_slow_part () from /usr/lib64/mysql/libmysqlclient.so.18
#8 0x00007fb9f61b3bbc in mysql_close () from /usr/lib64/mysql/libmysqlclient.so.18
#9 0x00000000004051aa in db_disconnect (mysql=<optimized out>) at sql.c:333
#10 0x000000000040cfbd in poll_host (host_id=host_id@entry=1647, host_thread=host_thread@entry=1, last_host_thread=last_host_thread@entry=1,
host_data_ids=host_data_ids@entry=0, host_time=host_time@entry=0x7fb9f346fcb0 "2020-04-07 15:02:02", host_errors=host_errors@entry=0x7fb9f346fca8,
host_time_double=host_time_double@entry=1586264522.599179) at poller.c:1660
#11 0x000000000040f35c in child (arg=<optimized out>) at poller.c:77
#12 0x00007fb9f53f8e65 in start_thread () from /lib64/libpthread.so.0
#13 0x00007fb9f4eaf88d in clone () from /lib64/libc.so.6

What can I do now?

[TheWitness]

Can you check if your MariaDB server is restarting?

[hektor]

No.

[root@somehost ~]# ls -latrc /tmp/
[...]
-rw-------. 1 root root 143691776 7. Apr 15:02 core-spine-sig6-user497-group487-pid4899-time1586264534
-rw-------. 1 root root 140759040 7. Apr 15:36 core-spine-sig6-user497-group487-pid31152-time1586266593
-rw-------. 1 root root 100958208 7. Apr 15:36 core-spine-sig6-user497-group487-pid31178-time1586266593
-rw-------. 1 root root 87777280 7. Apr 15:38 core-spine-sig6-user497-group487-pid32373-time1586266736
-rw-------. 1 root root 111095808 7. Apr 15:44 core-spine-sig11-user497-group487-pid5041-time1586267080
-rw-------. 1 root root 93704192 7. Apr 15:48 core-spine-sig6-user497-group487-pid8113-time1586267303
-rw-------. 1 root root 89858048 7. Apr 15:56 core-spine-sig6-user497-group487-pid13423-time1586267790
-rw-------. 1 root root 154169344 7. Apr 16:03 core-spine-sig6-user497-group487-pid19380-time1586268213
-rw-------. 1 root root 124358656 7. Apr 16:16 core-spine-sig11-user497-group487-pid29186-time1586268981
-rw-------. 1 root root 150925312 7. Apr 16:17 core-spine-sig6-user497-group487-pid29588-time1586269057
-rw-------. 1 root root 140849152 7. Apr 16:19 core-spine-sig6-user497-group487-pid30819-time1586269157
-rw-------. 1 root root 154304512 7. Apr 17:00 core-spine-sig6-user497-group487-pid28846-time1586271629
-rw-------. 1 root root 78028800 7. Apr 17:15 core-spine-sig6-user497-group487-pid7240-time1586272543
-rw-------. 1 root root 152162304 7. Apr 17:16 core-spine-sig6-user497-group487-pid8092-time1586272585
-rw-------. 1 root root 159412224 7. Apr 17:24 core-spine-sig11-user497-group487-pid13176-time1586273045
-rw-------. 1 root root 152559616 7. Apr 17:28 core-spine-sig6-user497-group487-pid16994-time1586273321
-rw-------. 1 root root 149110784 7. Apr 17:39 core-spine-sig6-user497-group487-pid24301-time1586273987
-rw-------. 1 root root 61472768 7. Apr 17:43 core-spine-sig6-user497-group487-pid25747-time1586274207
-rw-------. 1 root root 158478336 7. Apr 18:07 core-spine-sig6-user497-group487-pid13335-time1586275672
-rw-------. 1 root root 162414592 7. Apr 19:00 core-spine-sig11-user497-group487-pid19959-time1586278813
-rw-------. 1 root root 118104064 7. Apr 19:04 core-spine-sig6-user497-group487-pid22267-time1586279059
-rw-------. 1 root root 121135104 7. Apr 19:06 core-spine-sig6-user497-group487-pid24637-time1586279198
drwxrwxrwt. 10 root root 4096 7. Apr 19:12 .
-rw-------. 1 root root 159477760 7. Apr 19:12 core-spine-sig6-user497-group487-pid28457-time1586279565
[root@somehost ~]# ps -ef | fgrep mysql
mysql 1453 1 0 14:58 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
mysql 2248 1453 13 14:58 ? 00:35:27 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
root 6906 2405 0 19:25 pts/0 00:00:00 grep -F --color=auto mysql

I will have a look at all the core dumps later,
but for today I am through as my son and I "survived" a small explosive attack

[TheWitness]

Is that all spine 1.2.11? I so, can you do a bt on a few of them and post the results. But only if you are on 1.2.11

[hektor]

It was spine 1.2.11

Buuut: this is from a remote poller and meanwhile my central cacti DB is corrupted such that mysqld crashes on my main poller.

So before fixing all this (I have really bad days at the moment), we can close this. - I'll be back.

[hektor]

After a weak of depression I conclude this:

maybe my SSDs were corrupted, Dell did not help very much.
With new SSDs and lots of backup tries of the mariadb (and a crashing mysqld!)
the system is back in work and seems to behave well.

[TheWitness]

You never said if you updated spine to 1.2.11?

[hektor]

Sorry. Forgot that. Yes, the system is now running 1.2.11.
(I love systems with uptodate software).

[hektor]

can't open a new help topic, just a test here which seems to work.

[hektor]

I have a zombie DS.

Errors in cacti.log:

2020/04/24 01:06:11 - SPINE: Poller[3] PID[7677] Device[2054] DS[681465] ERROR: Empty result [A.B.8.26]: ''
2020/04/24 01:06:11 - SPINE: Poller[3] PID[7677] Device[2054] DS[681465] ERROR: Empty result [A.B.8.26]: ''

DS not to find (see image)

[TheWitness]

Sorry. Forgot that. Yes, the system is now running 1.2.11.
(I love systems with uptodate software).

Maybe one of these days.

[TheWitness]

I have a zombie DS.

Errors in cacti.log:

2020/04/24 01:06:11 - SPINE: Poller[3] PID[7677] Device[2054] DS[681465] ERROR: Empty result [A.B.8.26]: ''
2020/04/24 01:06:11 - SPINE: Poller[3] PID[7677] Device[2054] DS[681465] ERROR: Empty result [A.B.8.26]: ''

DS not to find (see image)

Rebuild the poller cache. Looks like you might have had some database issues along the way, otherwise, you should have found it.

[hektor]

I wish it would be as simple as that.

Rebuild poller cash does not help.

The DS ID does not get listed, but I can do a

http://my.cacti.de/cacti/data_sources.p ... &id=681465

and I get something listed, I can't change or delete.

I have ~1k of these datasources I want to get rid of
and there was for sure some hardware induced DB glitch.
Whenever it was ...

How can I proceed?
I have the ID of the DSs, any MySQL (mariadb) magic?
Or some patch I should test?

I do not only love up2date software, I love no errors in my cacti-log, also.