FATAL: Spine Encountered a Segmentation Fault

[jschoot]

Hello,

I have upgraded my Cacti from 1.1.38 to 1.2.3 and also updated Spine from 1.1.38 to 1.2.3.

Now i get the following error from Spine randomly:
SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [11, Resource temporarily unavailable] (Spine thread)

I already lowered the concurrent processes to 4 and threads to 10. but still they happen, but less often.

This is the Technical report from Cacti:
Technical Support [Summary]
General Information
Date Wed, 29 May 2019 15:19:07 +0200
Cacti Version 1.2.3
Cacti OS unix
RSA Fingerprint b3:4e:1f:19:68:41:e1:a9:58:43:15:c7:74:27:5d:67
NET-SNMP Version NET-SNMP version: 5.7.3
RRDtool Version Configured 1.7.2+
RRDtool Version Found 1.7.2
Devices 132
Graphs 1128
Data Sources Script/Command: 17
SNMP Get: 127
SNMP Query: 993
Total: 1137
Poller Information
Interval 300
Type SPINE 1.2.3 Copyright 2004-2019 by The Cacti Group
Items Action[0]: 2195
Action[1]: 17
Total: 2212
Concurrent Processes 4
Max Threads 10
PHP Servers 5
Script Timeout 30
Max OID 20
Last Run Statistics Time:8.0926 Method:spine Processes:5 Threads:20 Hosts:132 HostsPerProcess:27 DataSources:2212 RRDsProcessed:1126
System Memory
MemTotal 973.12 M
MemFree 204.61 M
MemAvailable 463.39 M
Buffers 95.26 M
Cached 297.02 M
Active 353.98 M
Inactive 289.06 M
SwapTotal 625.00 M
SwapFree 496.62 M
PHP Information
PHP Version 5.6.40-0+deb8u3
PHP OS Linux
PHP uname Linux NetworkMonitor 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u2 (2019-05-13) x86_64
PHP SNMP Installed
max_execution_time 60
memory_limit 800M
MariaDB Tuning (/etc/my.cnf) - [ Documentation ] Note: Many changes below require a database restart
Variable Current Value Recommended Value Comments
version 10.3.15-MariaDB-1:10.3.15+maria~stretch-log >= 5.6 MySQL 5.6+ and MariaDB 10.0+ are great releases, and are very good versions to choose. Make sure you run the very latest release though which fixes a long standing low level networking issue that was causing spine many issues with reliability.
collation_server utf8mb4_unicode_ci = utf8mb4_unicode_ci When using Cacti with languages other than English, it is important to use the utf8mb4_unicode_ci collation type as some characters take more than a single byte.
character_set_client utf8mb4 = utf8mb4 When using Cacti with languages other than English, it is important to use the utf8mb4 character set as some characters take more than a single byte.
max_connections 150 >= 100 Depending on the number of logins and use of spine data collector, MariaDB will need many connections. The calculation for spine is: total_connections = total_processes * (total_threads + script_servers + 1), then you must leave headroom for user connections, which will change depending on the number of concurrent login accounts.
max_allowed_packet 16777216 >= 16777216 With Remote polling capabilities, large amounts of data will be synced from the main server to the remote pollers. Therefore, keep this value at or above 16M.
max_heap_table_size 64M >= 16M If using the Cacti Performance Booster and choosing a memory storage engine, you have to be careful to flush your Performance Booster buffer before the system runs out of memory table space. This is done two ways, first reducing the size of your output column to just the right size. This column is in the tables poller_output, and poller_output_boost. The second thing you can do is allocate more memory to memory tables. We have arbitrarily chosen a recommended value of 10% of system memory, but if you are using SSD disk drives, or have a smaller system, you may ignore this recommendation or choose a different storage engine. You may see the expected consumption of the Performance Booster tables under Console -> System Utilities -> View Boost Status.
tmp_table_size 128M >= 16M When executing subqueries, having a larger temporary table size, keep those temporary tables in memory.
join_buffer_size 256M >= 31M When performing joins, if they are below this size, they will be kept in memory and never written to a temporary file.
innodb_file_per_table ON = ON When using InnoDB storage it is important to keep your table spaces separate. This makes managing the tables simpler for long time users of MariaDB. If you are running with this currently off, you can migrate to the per file storage by enabling the feature, and then running an alter statement on all InnoDB tables.
innodb_file_format Barracuda = Barracuda When using innodb_file_per_table, it is important to set the innodb_file_format to Barracuda. This setting will allow longer indexes important for certain Cacti tables.
innodb_large_prefix 1 = 1 If your tables have very large indexes, you must operate with the Barracuda innodb_file_format and the innodb_large_prefix equal to 1. Failure to do this may result in plugins that can not properly create tables.
innodb_buffer_pool_size 256M >= 243M InnoDB will hold as much tables and indexes in system memory as is possible. Therefore, you should make the innodb_buffer_pool large enough to hold as much of the tables and index in memory. Checking the size of the /var/lib/mysql/cacti directory will help in determining this value. We are recommending 25% of your systems total memory, but your requirements will vary depending on your systems size.
innodb_doublewrite OFF = OFF With modern SSD type storage, this operation actually degrades the disk more rapidly and adds a 50% overhead on all write operations.
innodb_lock_wait_timeout 50 >= 50 Rogue queries should not for the database to go offline to others. Kill these queries before they kill your system.
innodb_flush_method O_DIRECT eq O_DIRECT Maximum I/O performance happens when you use the O_DIRECT method to flush pages.
innodb_flush_log_at_timeout 5 >= 3 As of MariaDB 10.3.15-1:10.3.15+maria~stretch-log, the you can control how often MariaDB flushes transactions to disk. The default is 1 second, but in high I/O systems setting to a value greater than 1 can allow disk I/O to be more sequential
innodb_read_io_threads 32 >= 32 With modern SSD type storage, having multiple read io threads is advantageous for applications with high io characteristics.
innodb_write_io_threads 16 >= 16 With modern SSD type storage, having multiple write io threads is advantageous for applications with high io characteristics.
innodb_buffer_pool_instances 1 >= 3 MariaDB will divide the innodb_buffer_pool into memory regions to improve performance. The max value is 64. When your innodb_buffer_pool is less than 1GB, you should use the pool size divided by 128MB. Continue to use this equation upto the max of 64.
innodb_io_capacity 400 >= 5000 If you have SSD disks, use this suggestion. If you have physical hard drives, use 200 * the number of active drives in the array. If using NVMe or PCIe Flash, much larger numbers as high as 100000 can be used.
innodb_io_capacity_max 2000 >= 10000 If you have SSD disks, use this suggestion. If you have physical hard drives, use 2000 * the number of active drives in the array. If using NVMe or PCIe Flash, much larger numbers as high as 200000 can be used.



Does anyone have an idea what else can cause this?

[jschoot]

In a day time I got multiple different variants of segmentation faults.

2019/05/30 23:05:10 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [11, Resource temporarily unavailable] (Spine thread)
2019/05/30 23:05:10 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [110, Connection timed out] (Spine thread)
2019/05/30 23:00:09 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [4, Interrupted system call] (Spine thread)
2019/05/30 15:10:03 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [11, Resource temporarily unavailable] (Spine thread)
2019/05/29 18:05:03 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [95, Operation not supported] (Spine thread)
2019/05/29 18:05:03 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [95, Operation not supported] (Spine thread)
2019/05/29 16:40:02 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [95, Operation not supported] (Spine thread)
2019/05/29 15:10:08 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [11, Resource temporarily unavailable] (Spine thread)


Any idea?

[netniV]

Try running spine in debug mode to see if there are more details.

[jschoot]

I have the spine debug log attached to this post.

I do not see anything that I can explain to causing an FATAL error..

[netniV]

Have you made sure you've run through all the bootstrap/build steps to clean and make spine?

[jschoot]

I downloaded 1.2.4 and did a complete clean build. It works better but I still get errors..

2019/08/01 10:30:03 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [11, Resource temporarily unavailable] (Spine thread)
2019/08/01 10:15:03 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [95, Operation not supported] (Spine thread)
2019/08/01 10:15:03 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [95, Operation not supported] (Spine thread)
2019/08/01 10:10:02 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [95, Operation not supported] (Spine thread)
2019/08/01 10:10:02 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [95, Operation not supported] (Spine thread)
2019/08/01 10:10:02 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [95, Operation not supported] (Spine thread)
2019/08/01 10:10:02 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [95, Operation not supported] (Spine thread)
2019/08/01 10:05:04 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [95, Operation not supported] (Spine thread)
2019/08/01 10:00:08 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [11, Resource temporarily unavailable] (Spine parent)
2019/08/01 10:00:03 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [95, Operation not supported] (Spine thread)
2019/08/01 10:00:03 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [95, Operation not supported] (Spine thread)
2019/08/01 10:00:03 - SPINE: Poller[1] FATAL: Spine Encountered a Segmentation Fault [95, Operation not supported] (Spine thread)

[netniV]

Do you have all the required libraries installed? Have you checked for any updates on them? If it's an older version of MySQL have you tried being that forward slightly to see if anything corrects it ? (MySQL client libraries have been known to cause issues from time to time).

[netniV]

Generally speaking a segmentation fault is normally caused when a pointer is held to memory that isn't part of the process or has been released and thus should not still be held.

Try doing something like:

Code: Select all

bash$ gdb spine
gdb> run arguments
(spine will stop at the segmentation fault)
gdb> where
gdb> list

[jschoot]

I think it is solved.

Downloaded the latest version.
Updated all tools
Cleaned all existing builds
And followed the build instructions properly.

Thanks for your help.