Spine: Spine Encountered a Segmentation Fault (Spine parent)

[seerow]

Hello all,

sine i moved my cacti server to another hardware platform i've very strange problems with the spine poller.
Everytime then i started the polling process (manual or via cron) i get an "SPINE: Poller[0] FATAL: Spine Encountered a Segmentation Fault (Spine parent)" error.
If i try to run spine in verbosity mode, the output is the following:

SPINE: Using spine config file [../etc/spine.conf]
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The path_php_server variable is /var/www/html/cacti/script_server.php
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The path_cactilog variable is /var/www/html/cacti/log/cacti.log
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The log_destination variable is 1 (FILE)
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The path_php variable is /usr/local/php/bin/php
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The availability_method variable is 1
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The ping_recovery_count variable is 3
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The ping_failure_count variable is 2
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The ping_method variable is 1
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The ping_retries variable is 3
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The ping_timeout variable is 400
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The log_perror variable is 1
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The log_pwarn variable is 0
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The log_pstats variable is 0
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The threads variable is 1
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The polling interval is 300 seconds
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The number of concurrent processes is 4
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The script timeout is 25
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The number of php script servers to run is 1
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: StartHost='-1', EndHost='-1', TotalPHPScripts='0'
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The PHP Script Server is Not Required
02/24/2009 09:13:39 AM - SPINE: Poller[0] DEBUG: The Maximum SNMP OID Get Size is 10
02/24/2009 09:13:39 AM - SPINE: Poller[0] FATAL: Spine Encountered a Segmentation Fault (Spine parent)

Here my cacti environment:

Cacti Version - 0.8.7d
Plugin Architecture - 2.4
Poller Type - CMD.php
Server Info - Linux 2.6.18-92.1.22.el5PAE
Web Server - Apache/2.2.3 (Red Hat)
PHP - 5.2.6
PHP Extensions - xmlwriter, libxml, dom, xmlreader, xml, tokenizer, session, pcre, SimpleXML, SPL, PDO, sockets, snmp, SQLite, standard, Reflection, posix, pdo_sqlite, mysqli, mysql, mbstring, json, iconv, hash, gettext, gd, filter, dba, date, ctype, zlib, openssl, apache2handler
MySQL - 5.0.45
RRDTool - 1.3.5
SNMP - 5.3.1
Plugins

• Configuration manager (configmanager - v0.76)
  Device Monitoring (monitor - v0.8.2)
  Simple Links page (links - v0.3)
  Poller Performance Report (pollperf - v0.1)
  Host Info (hostinfo - v0.2)

Here are my actual troubleshooting steps:

1. upgraded cacti to 0.8.7d
2. upgraded php to 5.2.6
3. upgraded spine to 0.8.7c
4. mysqlcheck -uxxx -pxxx -c -B cacti -> result cacti.poller_item was marked as crashed, repaired via repair table

[dragossto]

i have the same error
and i found this error in system log

*** glibc detected *** /usr/bin/spine: free(): invalid next size (fast): 0x08131b18 ***
======= Backtrace: =========
/lib/libc.so.6[0xb7c37564]
/lib/libc.so.6(cfree+0x90)[0xb7c3b010]
/usr/bin/spine[0x804fa78]
/usr/bin/spine[0x805167e]
/lib/libpthread.so.0[0xb7d43369]
/lib/libc.so.6(clone+0x5e)[0xb7ca0cfe]
======= Memory map: ========
08048000-08059000 r-xp 00000000 03:01 366938 /usr/bin/spine
08059000-0805a000 rw-p 00010000 03:01 366938 /usr/bin/spine
0805a000-0830d000 rw-p 0805a000 00:00 0 [heap]
b1f00000-b1f35000 rw-p b1f00000 00:00 0
b1f35000-b2000000 ---p b1f35000 00:00 0
b2100000-b2101000 ---p b2100000 00:00 0
b2101000-b2900000 rwxp b2101000 00:00 0
b2900000-b2901000 ---p b2900000 00:00 0
b2901000-b3100000 rwxp b2901000 00:00 0
b3100000-b3101000 ---p b3100000 00:00 0
b3101000-b3900000 rwxp b3101000 00:00 0
b3900000-b3901000 ---p b3900000 00:00 0
b3901000-b4100000 rwxp b3901000 00:00 0
b4900000-b4901000 ---p b4900000 00:00 0
b4901000-b5100000 rwxp b4901000 00:00 0
b5900000-b5901000 ---p b5900000 00:00 0
b5901000-b6100000 rwxp b5901000 00:00 0
b6100000-b6101000 ---p b6100000 00:00 0
b6101000-b6900000 rwxp b6101000 00:00 0
b6900000-b694b000 rw-p b6900000 00:00 0
b694b000-b6a00000 ---p b694b000 00:00 0
b72e8000-b731e000 r--p 00000000 03:01 90974 /usr/lib/locale/en_US/LC_CTYPE
b731e000-b7327000 r-xp 00000000 03:01 91478 /lib/libnss_files-2.7.so
b7327000-b7329000 rw-p 00008000 03:01 91478 /lib/libnss_files-2.7.so
b7b3c000-b7b3e000 rw-p b7b3c000 00:00 0
b7b3e000-b7b40000 r-xp 00000000 03:01 91619 /lib/libdl-2.7.so
b7b40000-b7b42000 rw-p 00001000 03:01 91619 /lib/libdl-2.7.so
b7b42000-b7b7e000 r-xp 00000000 03:01 110330 /usr/lib/libssl.so.0.9.8
b7b7e000-b7b82000 rw-p 0003b000 03:01 110330 /usr/lib/libssl.so.0.9.8
b7b82000-b7b96000 r-xp 00000000 03:01 91450 /lib/libnsl-2.7.so
b7b96000-b7b98000 rw-p 00013000 03:01 91450 /lib/libnsl-2.7.so
b7b98000-b7b9a000 rw-p b7b98000 00:00 0
b7b9a000-b7ba3000 r-xp 00000000 03:01 91369 /lib/libcrypt-2.7.so
b7ba3000-b7ba5000 rw-p 00008000 03:01 91369 /lib/libcrypt-2.7.so
b7ba5000-b7bcc000 rw-p b7ba5000 00:00 0
b7bcc000-b7d11000 r-xp 00000000 03:01 66939 /lib/libc-2.7.so
b7d11000-b7d12000 r--p 00145000 03:01 66939 /lib/libc-2.7.so
b7d12000-b7d14000 rw-p 00146000 03:01 66939 /lib/libc-2.7.so
b7d14000-b7d18000 rw-p b7d14000 00:00 0
b7d18000-b7d3c000 r-xp 00000000 03:01 91648 /lib/libm-2.7.so
b7d3c000-b7d3e000 rw-p 00023000 03:01 91648 /lib/libm-2.7.so
b7d3e000-b7d51000 r-xp 00000000 03:01 91480 /lib/libpthread-2.7.so
b7d51000-b7d53000 rw-p 00012000 03:01 91480 /lib/libpthread-2.7.so
b7d53000-b7d55000 rw-p b7d53000 00:00 0
b7d55000-b7d68000 r-xp 00000000 03:01 188173 /usr/lib/libz.so.1.2.3
b7d68000-b7d69000 rw-p 00012000 03:01 188173 /usr/lib/libz.so.1.2.3
b7d69000-b7e8d000 r-xp 00000000 03:01 110316 /usr/lib/libcrypto.so.0.9.8
b7e8d000-b7ea2000 rw-p 00123000 03:01 110316 /usr/lib/libcrypto.so.0.9.8
b7ea2000-b7ea5000 rw-p b7ea2000 00:00 0
b7ea5000-b7fbb000 r-xp 00000000 03:01 47716 /usr/lib/mysql/libmysqlclient_r.so.15.0.0
b7fbb000-b7fea000 rw-p 00116000 03:01 47716 /usr/lib/mysql/libmysqlclient_r.so.15.0.0
b7fea000-b7feb000 rw-p b7fea000 00:00 0
b7feb000-b806a000 r-xp 00000000 03:01 1000738 /usr/lib/libnetsnmp.so.15.1.2
b806a000-b806c000 rw-p 0007f000 03:01 1000738 /usr/lib/libnetsnmp.so.15.1.2
b806c000-b808c000 rw-p b806c000 00:00 0
b8093000-b809d000 r-xp 00000000 03:01 30010 /usr/lib/libgcc_s.so.1
b809d000-b809e000 rw-p 00009000 03:01 30010 /usr/lib/libgcc_s.so.1
b809e000-b809f000 rw-p b809e000 00:00 0
b809f000-b80bb000 r-xp 00000000 03:01 91984 /lib/ld-2.7.so
b80bb000-b80bd000 rw-p 0001b000 03:01 91984 /lib/ld-2.7.so
bfda8000-bfdbc000 rwxp bffeb000 00:00 0 [stack]
bfdbc000-bfdbd000 rw-p bffff000 00:00 0
ffffe000-fffff000 r-xp 00000000 00:00 0 [vdso]

an also in cacti log i have this warning

02/27/2009 12:43:17 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
02/27/2009 12:43:17 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
02/27/2009 12:43:17 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
02/27/2009 12:42:45 PM - CMDPHP: Poller[0] ERROR: A DB Exec Failed!, Error:'1054', SQL:"UPDATE thold_data SET tcheck = 0'
02/27/2009 12:42:45 PM - CMDPHP: Poller[0] ERROR: SQL Assoc Failed!, Error:'1054', SQL:"SELECT * FROM thold_data WHERE thold_enabled = 'on' AND tcheck = 1"

check the data base and no error found
so i miss something , but i dont see what :(

[seerow]

Hi,

i can't see any error message regarding spine or cacti in my syslog. All seems to be runing fine. If start spine manualy via ./spine i also get an error in my cacti.log, but this is the same error:

cd /usr/local/spine/bin/
./spine
SPINE: Using spine config file [../etc/spine.conf]
02/27/2009 02:13:07 PM - SPINE: Poller[0] FATAL: Spine Encountered a Segmentation Fault (Spine parent)

[smckee]

I also have this issue. The spine seems to crash but it leaves orphaned processes running. Details on the crash:

*** glibc detected *** spine: free(): invalid next size (fast): 0x00000000108bb250 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3448071684]
/lib64/libc.so.6(cfree+0x8c)[0x3448074ccc]
spine[0x4081b8]
spine[0x409913]
/lib64/libpthread.so.0[0x3448c06307]
/lib64/libc.so.6(clone+0x6d)[0x34480d1ded]
======= Memory map: ========
00400000-00412000 r-xp 00000000 fd:00 23642147 /usr/bin/spine
00611000-00612000 rw-p 00011000 fd:00 23642147 /usr/bin/spine
00612000-00615000 rw-p 00612000 00:00 0
00811000-00814000 rw-p 00011000 fd:00 23642147 /usr/bin/spine
106cd000-10c12000 rw-p 106cd000 00:00 0 [heap]
407e2000-407e3000 ---p 407e2000 00:00 0
407e3000-411e3000 rw-p 407e3000 00:00 0
41435000-41436000 ---p 41435000 00:00 0
41436000-41e36000 rw-p 41436000 00:00 0
41e36000-41e37000 ---p 41e36000 00:00 0
41e37000-42837000 rw-p 41e37000 00:00 0
42837000-42838000 ---p 42837000 00:00 0
42838000-43238000 rw-p 42838000 00:00 0
43238000-43239000 ---p 43238000 00:00 0
43239000-43c39000 rw-p 43239000 00:00 0
43c39000-43c3a000 ---p 43c39000 00:00 0
43c3a000-4463a000 rw-p 43c3a000 00:00 0
4463a000-4463b000 ---p 4463a000 00:00 0
4463b000-4503b000 rw-p 4463b000 00:00 0
4503b000-4503c000 ---p 4503b000 00:00 0
4503c000-45a3c000 rw-p 4503c000 00:00 0
45a3c000-45a3d000 ---p 45a3c000 00:00 0
45a3d000-4643d000 rw-p 45a3d000 00:00 0
4643d000-4643e000 ---p 4643d000 00:00 0
4643e000-46e3e000 rw-p 4643e000 00:00 0
46e3e000-46e3f000 ---p 46e3e000 00:00 0
46e3f000-4783f000 rw-p 46e3f000 00:00 0
3230c00000-3230d2c000 r-xp 00000000 fd:00 24184492 /usr/lib64/mysql/libmysqlclient_r.so.15.0.0
3230d2c000-3230f2b000 ---p 0012c000 fd:00 24184492 /usr/lib64/mysql/libmysqlclient_r.so.15.0.0
3230f2b000-3230f73000 rw-p 0012b000 fd:00 24184492 /usr/lib64/mysql/libmysqlclient_r.so.15.0.0
3230f73000-3230f74000 rw-p 3230f73000 00:00 0
3231000000-323108f000 r-xp 00000000 fd:00 23641900 /usr/lib64/libnetsnmp.so.10.0.1
323108f000-323128f000 ---p 0008f000 fd:00 23641900 /usr/lib64/libnetsnmp.so.10.0.1
323128f000-3231293000 rw-p 0008f000 fd:00 23641900 /usr/lib64/libnetsnmp.so.10.0.1
3231293000-32312c7000 rw-p 3231293000 00:00 0
3447c00000-3447c1a000 r-xp 00000000 fd:00 22216898 /lib64/ld-2.5.so
3447e1a000-3447e1b000 r--p 0001a000 fd:00 22216898 /lib64/ld-2.5.so
3447e1b000-3447e1c000 rw-p 0001b000 fd:00 22216898 /lib64/ld-2.5.so
3448000000-344814a000 r-xp 00000000 fd:00 22216900 /lib64/libc-2.5.so
344814a000-3448349000 ---p 0014a000 fd:00 22216900 /lib64/libc-2.5.so
3448349000-344834d000 r--p 00149000 fd:00 22216900 /lib64/libc-2.5.so
344834d000-344834e000 rw-p 0014d000 fd:00 22216900 /lib64/libc-2.5.so
344834e000-3448353000 rw-p 344834e000 00:00 0
3448400000-3448402000 r-xp 00000000 fd:00 22216904 /lib64/libdl-2.5.so
3448402000-3448602000 ---p 00002000 fd:00 22216904 /lib64/libdl-2.5.so
3448602000-3448603000 r--p 00002000 fd:00 22216904 /lib64/libdl-2.5.so
3448603000-3448604000 rw-p 00003000 fd:00 22216904 /lib64/libdl-2.5.so
3448800000-3448882000 r-xp 00000000 fd:00 22217001 /lib64/libm-2.5.so
3448882000-3448a81000 ---p 00082000 fd:00 22217001 /lib64/libm-2.5.so
3448a81000-3448a82000 r--p 00081000 fd:00 22217001 /lib64/libm-2.5.so
3448a82000-3448a83000 rw-p 00082000 fd:00 22217001 /lib64/libm-2.5.so
3448c00000-3448c15000 r-xp 00000000 fd:00 22217000 /lib64/libpthread-2.5.so
3448c15000-3448e14000 ---p 00015000 fd:00 22217000 /lib64/libpthread-2.5.so
3448e14000-3448e15000 r--p 00014000 fd:00 22217000 Aborted

My configuration:

Cacti Version - 0.8.7d
Plugin Architecture - 2.4
Poller Type - Cactid v
Server Info - Linux 2.6.18-128.1.1.el5
Web Server - Apache/2.2.3 (Scientific Linux)
PHP - 5.1.6
PHP Extensions - libxml, xml, wddx, tokenizer, sysvshm, sysvsem, sysvmsg, standard, SimpleXML, sockets, SPL, shmop, session, Reflection, pspell, posix, mime_magic, iconv, hash, gmp, gettext, ftp, exif, date, curl, ctype, calendar, bz2, zlib, pcre, openssl, apache2handler, dbase, mysql, mysqli, PDO, pdo_mysql, pdo_sqlite, snmp
MySQL - 5.0.45-log
RRDTool - 1.2.30
SNMP - 5.3.1
Plugins

• Global Plugin Settings (settings - v0.5)
  Update Checker (update - v0.4)
  Network Tools (tools - v0.3)
  Host Info (hostinfo - v0.2)
  Network Discovery (discovery - v0.8.4)
  FlowView (flowview - v0.5.2)
  PHP Network Weathermap (weathermap - v0.95b)
  Thresholds (thold - v0.3.9)
  Device Monitoring (monitor - v0.8.2)
  Device Tracking (mactrack - v1.1)
  manage
  Create Aggregate Graphs (aggregate - v0.63)
  Poller Performance Report (pollperf - v0.32)
  SuperLinks (superlinks - v0.72)

-bash-3.2$ spine --version
SPINE 0.8.7c Copyright 2002-2008 by The Cacti Group

Maybe I will try to build off the spine source repository head. If this helps I will post a follow-up.

[TheWitness]

Run the following in the build directory for the binary in question:

Code: Select all

gdb ./spine
b *0x4081b8
run

Post the results.

TheWitness

[dragossto]

cacti-spine-0.8.7c# gdb ./spine
GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-slackware-linux"...
(gdb) b *0x4081b8
Breakpoint 1 at 0x4081b8
(gdb) run
Starting program: /usr/src/cacti-spine-0.8.7c/spine
Warning:
Cannot insert breakpoint 1.
Error accessing memory address 0x4081b8: Input/output error.

(gdb)

and same to me
spine leave orphan process.
also i post here=http://forums.cacti.net/viewtopic.php?t=23926&start=75 the full cacti log debug.

best regards

[TheWitness]

Are you sure we are dealing with the same binary here?

[dragossto]

very sure
compile from
http://www.cacti.net/downloads/spine/ca ... .7c.tar.gz

when using bootstrap
i have this warning
NOTE: Running auto-tools to verify buildability
/usr/share/aclocal/nspr.m4:8: warning: underquoted definition of AM_PATH_NSPR
/usr/share/aclocal/nspr.m4:8: run info '(automake)Extending aclocal'
/usr/share/aclocal/nspr.m4:8: or see http://sources.redhat.com/automake/auto ... ng-aclocal

i try once again
end have the same error
gdb ./spine
GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-slackware-linux"...
(gdb) b *0x4081b8
Breakpoint 1 at 0x4081b8
(gdb) run
Starting program: /usr/src/cacti-spine-0.8.7c/spine
Warning:
Cannot insert breakpoint 1.
Error accessing memory address 0x4081b8: Input/output error.

[TheWitness]

What about this one?

0x409913

TheWitness

[dragossto]

i dont know, from a lib or other module
use by spine

if tell me where to look, i whil

[TheWitness]

If you want to do a gotomeeting. PM me and I'll set one up. But only if you can do it today.

TheWitness

[dragossto]

still try to solve the problem
but no luck
only resolv the orphan process by killing them

a little script

Code: Select all

#!/bin/sh
pathscript=/etc/cron.min
pathcacti=/var/www/htdocs/cacti
cmdphp=`which php`
logdate=`date +"%Y-%m-%d_%H_%M"`
#make diretory error.log
declare -i spipro=`ps -A x|grep spine|grep -v grep|wc -l`
declare -i spinkill=0
declare -i zz=0


if [ $spipro -ge "1" ]
    then
    proc=`ps -A x|grep spine|grep -v grep|awk '{print $1}'`
    for nrproc in $proc
        do
        kill -s QUIT $nrproc
        kill -9 $nrproc
    done
    zz=`cat $pathscript/zzz`
    if [ $zz -ge "11" ]
        then
        $cmdphp $pathcacti/cli/rebuild_poller_cache.php > /dev/null 2>&1
        echo "0">$pathscript/zzz
        else
        zz=zz+1
        echo $zz>$pathscript/zzz
    fi
    echo "1"> $pathscript/spoolchk
    spinkill=1
    su -c `$cmdphp $pathcacti/poller.php >> $pathscript/error.log/$logdate.log2` &
fi

declare -i chkspoll=`cat $pathscript/spoolchk`
if [ $chkspoll == 0 ]
    then
    su -c `$cmdphp $pathcacti/poller.php >> $pathscript/error.log/$logdate.log1` &
fi

if [ $spinkill == 0 ]
    then
    echo "0"> $pathscript/spoolchk
fi

[dragossto]

no good luck
i still try to solve the problem,but
so far i dont have segmentantion fault
using the above script,

some one tell me why
give me this error

SPINE: Poller[0] ERROR: SQL Failed! Error:'1062', Message:'Duplicate entry '8--2009-03-09 17:01:02' for key 1', SQL Fragment:'INSERT INTO poller_output (local_data_id, rrd_name, time, output) VALUES (8,'','2009-03-09 17:01:02','min:3.8390 avg:3.8485 max:3.8580 dev:0.0095 loss:0.0000'),(554,'','2009-03-09 17:01:02','min:0.7270 avg:0.7270 max:0.7270 dev:0.0000 loss:0.0000'),(570,'','2009-03-09 17:01:02','min:53.5250 avg:53.5250 max:53.5250 dev:0.0000 loss:0.0000')'

allmost every time
when poller make the polling

[meralias]

I'm experiencing the same segfault as well, using the spine v0.8.7c. System is a RedHat EL5 64bit server. Kernel: Linux server 2.6.18-92.1.22.el5 #1 SMP Fri Dec 5 09:28:22 EST 2008 x86_64 x86_64 x86_64 GNU/Linux


GDB output:

*** glibc detected *** /usr/local/cacti/sources/cacti-spine-0.8.7c/spine: munmap_chunk(): invalid pointer: 0x0000000006563420 ***
======= Backtrace: =========
/lib64/libc.so.6(cfree+0x1b6)[0x305dc74d86]
/usr/lib64/libnetsnmp.so.10(netsnmp_transport_free+0x43)[0x3a31a5e453]
/usr/lib64/libnetsnmp.so.10(snmp_sess_close+0xb9)[0x3a31a33ab9]
/usr/local/cacti/sources/cacti-spine-0.8.7c/spine[0x40877c]
/usr/local/cacti/sources/cacti-spine-0.8.7c/spine[0x4099a3]
/lib64/libpthread.so.0[0x305e8062f7]
/lib64/libc.so.6(clone+0x6d)[0x305dcd1b6d]
======= Memory map: ========
00400000-00412000 r-xp 00000000 fd:05 73892098 /usr/local/cacti/sources/cacti-spine-0.8.7c/spine
00611000-00612000 rw-p 00011000 fd:05 73892098 /usr/local/cacti/sources/cacti-spine-0.8.7c/spine
00612000-00615000 rw-p 00612000 00:00 0
06461000-06a65000 rw-p 06461000 00:00 0
40d3f000-40d40000 ---p 40d3f000 00:00 0
40d40000-41740000 rw-p 40d40000 00:00 0
41852000-41853000 ---p 41852000 00:00 0
41853000-42253000 rw-p 41853000 00:00 0
42253000-42254000 ---p 42253000 00:00 0
42254000-42c54000 rw-p 42254000 00:00 0
42c54000-42c55000 ---p 42c54000 00:00 0
42c55000-43655000 rw-p 42c55000 00:00 0
43655000-43656000 ---p 43655000 00:00 0
43656000-44056000 rw-p 43656000 00:00 0
44056000-44057000 ---p 44056000 00:00 0
44057000-44a57000 rw-p 44057000 00:00 0
305d800000-305d81a000 r-xp 00000000 fd:00 444710 /lib64/ld-2.5.so
305da1a000-305da1b000 r--p 0001a000 fd:00 444710 /lib64/ld-2.5.so
305da1b000-305da1c000 rw-p 0001b000 fd:00 444710 /lib64/ld-2.5.so
305dc00000-305dd4a000 r-xp 00000000 fd:00 444711 /lib64/libc-2.5.so
305dd4a000-305df49000 ---p 0014a000 fd:00 444711 /lib64/libc-2.5.so
305df49000-305df4d000 r--p 00149000 fd:00 444711 /lib64/libc-2.5.so
305df4d000-305df4e000 rw-p 0014d000 fd:00 444711 /lib64/libc-2.5.so
305df4e000-305df53000 rw-p 305df4e000 00:00 0
305e000000-305e002000 r-xp 00000000 fd:00 444712 /lib64/libdl-2.5.so
305e002000-305e202000 ---p 00002000 fd:00 444712 /lib64/libdl-2.5.so
305e202000-305e203000 r--p 00002000 fd:00 444712 /lib64/libdl-2.5.so
305e203000-305e204000 rw-p 00003000 fd:00 444712 /lib64/libdl-2.5.so
305e800000-305e815000 r-xp 00000000 fd:00 444716 /lib64/libpthread-2.5.so
305e815000-305ea14000 ---p 00015000 fd:00 444716 /lib64/libpthread-2.5.so
305ea14000-305ea15000 r--p 00014000 fd:00 444716 /lib64/libpthread-2.5.so
305ea15000-305ea16000 rw-p 00015000 fd:00 444716 /lib64/libpthread-2.5.so
305ea16000-305ea1a000 rw-p 305ea16000 00:00 0
305ec00000-305ec14000 r-xp 00000000 fd:01 32926 /usr/lib64/libz.so.1.2.3
305ec14000-305ee13000 ---p 00014000 fd:01 32926 /usr/lib64/libz.so.1.2.3
305ee13000-305ee14000 rw-p 00013000 fd:01 32926 /usr/lib64/libz.so.1.2.3
3060000000-3060015000 r-xp 00000000 fd:00 444731 /lib64/libnsl-2.5.so
3060015000-3060214000 ---p 00015000 fd:00 444731 /lib64/libnsl-2.5.so
3060214000-3060215000 r--p 00014000 fd:00 444731 /lib64/libnsl-2.5.so
3060215000-3060216000 rw-p 00015000 fd:00 444731 /lib64/libnsl-2.5.so
3060216000-3060218000 rw-p 3060216000 00:00 0
3060400000-306040d000 r-xp 00000000 fd:00 444721 /lib64/libgcc_s-4.1.2-20080102.so.1
306040d000-306060d000 ---p 0000d000 fd:00 444721 /lib64/libgcc_s-4.1.2-20080102.so.1
306060d000-306060e000 rw-p 0000d000 fd:00 444721 /lib64/libgcc_s-4.1.2-20080102.so.1
3060c00000-3060d25000 r-xp 00000000 fd:00 444725 /lib64/libcrypto.so.0.9.8b
3060d25000-3060f25000 ---p 00125000 fd:00 444725 /lib64/libcrypto.so.0.9.8b
3060f25000-3060f44000 rw-p 00125000 fd:00 444725 /lib64/libcrypto.so.0.9.8b
3060f44000-3060f48000 rw-p 3060f44000 00:00 0
3061400000-3061402000 r-xp 00000000 fd:00 444730 /lib64/libcom_err.so.2.1
3061402000-3061601000 ---p 00002000 fd:00 444730 /lib64/libcom_err.so.2.1
3061601000-3061602000 rw-p 00001000 fd:00 444730 /lib64/libcom_err.so.2.1
3061800000-3061802000 r-xp 00000000 fd:00 444729 /lib64/libkeyutils-1.2.so
3061802000-3061a01000 ---p 00002000 fd:00 444729 /lib64/libkeyutils-1.2.so
3061a01000-3061a02000 rw-p 00001000 fd:00 444729 /lib64/libkeyutils-1.2.so
3061c00000-3061c11000 r-xp 00000000 fd:00 444728 /lib64/libresolv-2.5.so
3061c11000-3061e11000 ---p 00011000 fd:00 444728 /lib64/libresolv-2.5.so
3061e11000-3061e12000 r--p 00011000 fd:00 444728 /lib64/libresolv-2.5.so
3061e12000-3061e13000 rw-p 00012000 fd:00 444728 /lib64/libresolv-2.5.so
3061e13000-3061e15000 rw-p 3061e13000 00:00 0
33f5800000-33f5882000 r-xp 00000000 fd:00 444421 /lib64/libm-2.5.so
33f5882000-33f5a81000 ---p 00082000 fd:00 444421 /lib64/libm-2.5.so
33f5a81000-33f5a82000 r--p 00081000 fd:00 444421 /lib64/libm-2.5.so
33f5a82000-33f5a83000 rw-p 00082000 fd:00 444421 /lib64/libm-2.5.so
33f5c00000-33f5c09000 r-xp 00000000 fd:00 444418 /lib64/libcrypt-2.5.so
33f5c09000-33f5e08000 ---p 00009000 fd:00 444418 /lib64/libcrypt-2.5.so
33f5e08000-33f5e09000 r--p 00008000 fd:00 444418 /lib64/libcrypt-2.5.so
33f5e09000-33f5e0a000 rw-p 00009000 fd:00 444418 /lib64/libcrypt-2.5.so
33f5e0a000-33f5e38000 rw-p 33f5e0a000 00:00 0
3a31a00000-3a31a90000 r-xp 00000000 fd:01 33226 /usr/lib64/libnetsnmp.so.10.0.3
3a31a90000-3a31c8f000 ---p 00090000 fd:01 33226 /usr/lib64/libnetsnmp.so.10.0.3
3a31c8f000-3a31c93000 rw-p 0008f000 fd:01 33226 /usr/lib64/libnetsnmp.so.10.0.3
3a31c93000-3a31cc7000 rw-p 3a31c93000 00:00 0
3c60800000-3c6083b000 r-xp 00000000 fd:00 444456 /lib64/libsepol.so.1
3c6083b000-3c60a3b000 ---p 0003b000 fd:00 444456 /lib64/libsepol.so.1
3c60a3b000-3c60a3c000 rw-p 0003b000 fd:00 444456 /lib64/libsepol.so.1
3c60a3c000-3c60a46000 rw-p 3c60a3c000 00:00 0
3c60c00000-3c60c08000 r-xp 00000000 fd:01 33935 /usr/lib64/libkrb5support.so.0.1
3c60c08000-3c60e07000 ---p 00008000 fd:01 33935 /usr/lib64/libkrb5support.so.0.1
3c60e07000-3c60e08000 rw-p 00007000 fd:01 33935 /usr/lib64/libkrb5support.so.0.1
3c61000000-3c6102c000 r-xp 00000000 fd:01 34200 /usr/lib64/libgssapi_krb5.so.2.2
3c6102c000-3c6122c000 ---p 0002c000 fd:01 34200 /usr/lib64/libgssapi_krb5.so.2.2
3c6122c000-3c6122e000 rw-p 0002c000 fd:01 34200 /usr/lib64/libgssapi_krb5.so.2.2
3c61c00000-3c61c24000 r-xp 00000000 fd:01 33944 /usr/lib64/libk5crypto.so.3.1
3c61c24000-3c61e23000 ---p 00024000 fd:01 33944 /usr/lib64/libk5crypto.so.3.1
3c61e23000-3c61e25000 rw-p 00023000 fd:01 33944 /usr/lib64/libk5crypto.so.3.1
3c62000000-3c62015000 r-xp 00000000 fd:00 444460 /lib64/libselinux.so.1
3c62015000-3c62215000 ---p 00015000 fd:00 444460 /lib64/libselinux.so.1
3c62215000-3c62217000 rw-p 00015000 fd:00 444460 /lib64/libselinux.so.1
3c62217000-3c62218000 rw-p 3c62217000 00:00 0
3c62400000-3c62443000 r-xp 00000000 fd:00 444466 /lib64/libssl.so.0.9.8b
3c62443000-3c62643000 ---p 00043000 fd:00 444466 /lib64/libssl.so.0.9.8b
3c62643000-3c62649000 rw-p 00043000 fd:00 444466 /lib64/libssl.so.0.9.8b
3c62800000-3c6288f000 r-xp 00000000 fd:01 34199 /usr/lib64/libkrb5.so.3.3
3c6288f000-3c62a8e000 ---p 0008f000 fd:01 34199 /usr/lib64/libkrb5.so.3.3
3c62a8e000-3c62a92000 rw-p 0008e000 fd:01 34199 /usr/lib64/libkrb5.so.3.3
3e29600000-3e2972c000 r-xp 00000000 fd:01 98154 /usr/lib64/mysql/libmysqlclient_r.so.15.0.0
3e2972c000-3e2992b000 ---p 0012c000 fd:01 98154 /usr/lib64/mysql/libmysqlclient_r.so.15.0.0
3e2992b000-3e29973000 rw-p 0012b000 fd:01 98154 /usr/lib64/mysql/libmysqlclient_r.so.15.0.0
3e29973000-3e29974000 rw-p 3e29973000 00:00 0
2aaaac000000-2aaaac14e000 rw-p 2aaaac000000 00:00 0
2aaaac14e000-2aaab0000000 ---p 2aaaac14e000 00:00 0
2aaab0000000-2aaab01e0000 rw-p 2aaab0000000 00:00 0
2aaab01e0000-2aaab4000000 ---p 2aaab01e0000 00:00 0
2aaab4000000-2aaab4163000 rw-p 2aaab4000000 00:00 0
2ae478486000-2ae478489000 rw-p 2ae478486000 00:00 0
2ae47849f000-2ae4784a8000 rw-p 2ae47849f000 00:00 0
2ae4784c0000-2ae4784ca000 r-xp 00000000 fd:00 444444 /lib64/libnss_files-2.5.so
2ae4784ca000-2ae4786c9000 ---p 0000a000 fd:00 444444 /lib64/libnss_files-2.5.so
2ae4786c9000-2ae4786ca000 r--p 00009000 fd:00 444444 /lib64/libnss_files-2.5.so
2ae4786ca000-2ae4786cb000 rw-p 0000a000 fd:00 444444 /lib64/libnss_files-2.5.so
2ae4786cb000-2ae47bc9a000 r--p 00000000 fd:01 290472 /usr/lib/locale/locale-archive
7fff3260e000-7fff32624000 rw-p 7fff3260e000 00:00 0 [stack]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]

Program received signal SIGABRT, Aborted.
[Switching to Thread 1130711360 (LWP 31672)]
0x000000305dc30155 in raise () from /lib64/libc.so.6

[meralias]

Issue appears to be intermittent on my server - sometimes it runs through cleanly. Another time it failed, I got the following GDB trace:

*** glibc detected *** /usr/local/cacti/sources/cacti-spine-0.8.7c/spine: free(): invalid next size (fast): 0x000000000c2bd640 ***
======= Backtrace: =========
/lib64/libc.so.6[0x305dc71634]
/lib64/libc.so.6(cfree+0x8c)[0x305dc74c5c]
/usr/local/cacti/sources/cacti-spine-0.8.7c/spine[0x40b94d]
/usr/local/cacti/sources/cacti-spine-0.8.7c/spine[0x407a4b]
/lib64/libpthread.so.0[0x305e8062f7]
/lib64/libc.so.6(clone+0x6d)[0x305dcd1b6d]
======= Memory map: ========
00400000-00415000 r-xp 00000000 fd:05 73892099 /usr/local/cacti/sources/cacti-spine-0.8.7c/spine
00614000-00616000 rw-p 00014000 fd:05 73892099 /usr/local/cacti/sources/cacti-spine-0.8.7c/spine
00616000-00618000 rw-p 00616000 00:00 0
0c1fc000-0c800000 rw-p 0c1fc000 00:00 0
4001e000-4001f000 ---p 4001e000 00:00 0
4001f000-40a1f000 rw-p 4001f000 00:00 0
40a1f000-40a20000 ---p 40a1f000 00:00 0
40a20000-41420000 rw-p 40a20000 00:00 0
41420000-41421000 ---p 41420000 00:00 0
41421000-41e21000 rw-p 41421000 00:00 0
41f9f000-41fa0000 ---p 41f9f000 00:00 0
41fa0000-429a0000 rw-p 41fa0000 00:00 0
429a0000-429a1000 ---p 429a0000 00:00 0
429a1000-433a1000 rw-p 429a1000 00:00 0
433a1000-433a2000 ---p 433a1000 00:00 0
433a2000-43da2000 rw-p 433a2000 00:00 0
305d800000-305d81a000 r-xp 00000000 fd:00 444710 /lib64/ld-2.5.so
305da1a000-305da1b000 r--p 0001a000 fd:00 444710 /lib64/ld-2.5.so
305da1b000-305da1c000 rw-p 0001b000 fd:00 444710 /lib64/ld-2.5.so
305dc00000-305dd4a000 r-xp 00000000 fd:00 444711 /lib64/libc-2.5.so
305dd4a000-305df49000 ---p 0014a000 fd:00 444711 /lib64/libc-2.5.so
305df49000-305df4d000 r--p 00149000 fd:00 444711 /lib64/libc-2.5.so
305df4d000-305df4e000 rw-p 0014d000 fd:00 444711 /lib64/libc-2.5.so
305df4e000-305df53000 rw-p 305df4e000 00:00 0
305e000000-305e002000 r-xp 00000000 fd:00 444712 /lib64/libdl-2.5.so
305e002000-305e202000 ---p 00002000 fd:00 444712 /lib64/libdl-2.5.so
305e202000-305e203000 r--p 00002000 fd:00 444712 /lib64/libdl-2.5.so
305e203000-305e204000 rw-p 00003000 fd:00 444712 /lib64/libdl-2.5.so
305e800000-305e815000 r-xp 00000000 fd:00 444716 /lib64/libpthread-2.5.so
305e815000-305ea14000 ---p 00015000 fd:00 444716 /lib64/libpthread-2.5.so
305ea14000-305ea15000 r--p 00014000 fd:00 444716 /lib64/libpthread-2.5.so
305ea15000-305ea16000 rw-p 00015000 fd:00 444716 /lib64/libpthread-2.5.so
305ea16000-305ea1a000 rw-p 305ea16000 00:00 0
305ec00000-305ec14000 r-xp 00000000 fd:01 32926 /usr/lib64/libz.so.1.2.3
305ec14000-305ee13000 ---p 00014000 fd:01 32926 /usr/lib64/libz.so.1.2.3
305ee13000-305ee14000 rw-p 00013000 fd:01 32926 /usr/lib64/libz.so.1.2.3
3060000000-3060015000 r-xp 00000000 fd:00 444731 /lib64/libnsl-2.5.so
3060015000-3060214000 ---p 00015000 fd:00 444731 /lib64/libnsl-2.5.so
3060214000-3060215000 r--p 00014000 fd:00 444731 /lib64/libnsl-2.5.so
3060215000-3060216000 rw-p 00015000 fd:00 444731 /lib64/libnsl-2.5.so
3060216000-3060218000 rw-p 3060216000 00:00 0
3060400000-306040d000 r-xp 00000000 fd:00 444721 /lib64/libgcc_s-4.1.2-20080102.so.1
306040d000-306060d000 ---p 0000d000 fd:00 444721 /lib64/libgcc_s-4.1.2-20080102.so.1
306060d000-306060e000 rw-p 0000d000 fd:00 444721 /lib64/libgcc_s-4.1.2-20080102.so.1
3060c00000-3060d25000 r-xp 00000000 fd:00 444725 /lib64/libcrypto.so.0.9.8b
3060d25000-3060f25000 ---p 00125000 fd:00 444725 /lib64/libcrypto.so.0.9.8b
3060f25000-3060f44000 rw-p 00125000 fd:00 444725 /lib64/libcrypto.so.0.9.8b
3060f44000-3060f48000 rw-p 3060f44000 00:00 0
3061400000-3061402000 r-xp 00000000 fd:00 444730 /lib64/libcom_err.so.2.1
3061402000-3061601000 ---p 00002000 fd:00 444730 /lib64/libcom_err.so.2.1
3061601000-3061602000 rw-p 00001000 fd:00 444730 /lib64/libcom_err.so.2.1
3061800000-3061802000 r-xp 00000000 fd:00 444729 /lib64/libkeyutils-1.2.so
3061802000-3061a01000 ---p 00002000 fd:00 444729 /lib64/libkeyutils-1.2.so
3061a01000-3061a02000 rw-p 00001000 fd:00 444729 /lib64/libkeyutils-1.2.so
3061c00000-3061c11000 r-xp 00000000 fd:00 444728 /lib64/libresolv-2.5.so
3061c11000-3061e11000 ---p 00011000 fd:00 444728 /lib64/libresolv-2.5.so
3061e11000-3061e12000 r--p 00011000 fd:00 444728 /lib64/libresolv-2.5.so
3061e12000-3061e13000 rw-p 00012000 fd:00 444728 /lib64/libresolv-2.5.so
3061e13000-3061e15000 rw-p 3061e13000 00:00 0
33f5800000-33f5882000 r-xp 00000000 fd:00 444421 /lib64/libm-2.5.so
33f5882000-33f5a81000 ---p 00082000 fd:00 444421 /lib64/libm-2.5.so
33f5a81000-33f5a82000 r--p 00081000 fd:00 444421 /lib64/libm-2.5.so
33f5a82000-33f5a83000 rw-p 00082000 fd:00 444421 /lib64/libm-2.5.so
33f5c00000-33f5c09000 r-xp 00000000 fd:00 444418 /lib64/libcrypt-2.5.so
33f5c09000-33f5e08000 ---p 00009000 fd:00 444418 /lib64/libcrypt-2.5.so
33f5e08000-33f5e09000 r--p 00008000 fd:00 444418 /lib64/libcrypt-2.5.so
33f5e09000-33f5e0a000 rw-p 00009000 fd:00 444418 /lib64/libcrypt-2.5.so
33f5e0a000-33f5e38000 rw-p 33f5e0a000 00:00 0
3a31a00000-3a31a90000 r-xp 00000000 fd:01 33226 /usr/lib64/libnetsnmp.so.10.0.3
3a31a90000-3a31c8f000 ---p 00090000 fd:01 33226 /usr/lib64/libnetsnmp.so.10.0.3
3a31c8f000-3a31c93000 rw-p 0008f000 fd:01 33226 /usr/lib64/libnetsnmp.so.10.0.3
3a31c93000-3a31cc7000 rw-p 3a31c93000 00:00 0
3c60800000-3c6083b000 r-xp 00000000 fd:00 444456 /lib64/libsepol.so.1
3c6083b000-3c60a3b000 ---p 0003b000 fd:00 444456 /lib64/libsepol.so.1
3c60a3b000-3c60a3c000 rw-p 0003b000 fd:00 444456 /lib64/libsepol.so.1
3c60a3c000-3c60a46000 rw-p 3c60a3c000 00:00 0
3c60c00000-3c60c08000 r-xp 00000000 fd:01 33935 /usr/lib64/libkrb5support.so.0.1
3c60c08000-3c60e07000 ---p 00008000 fd:01 33935 /usr/lib64/libkrb5support.so.0.1
3c60e07000-3c60e08000 rw-p 00007000 fd:01 33935 /usr/lib64/libkrb5support.so.0.1
3c61000000-3c6102c000 r-xp 00000000 fd:01 34200 /usr/lib64/libgssapi_krb5.so.2.2
3c6102c000-3c6122c000 ---p 0002c000 fd:01 34200 /usr/lib64/libgssapi_krb5.so.2.2
3c6122c000-3c6122e000 rw-p 0002c000 fd:01 34200 /usr/lib64/libgssapi_krb5.so.2.2
3c61c00000-3c61c24000 r-xp 00000000 fd:01 33944 /usr/lib64/libk5crypto.so.3.1
3c61c24000-3c61e23000 ---p 00024000 fd:01 33944 /usr/lib64/libk5crypto.so.3.1
3c61e23000-3c61e25000 rw-p 00023000 fd:01 33944 /usr/lib64/libk5crypto.so.3.1
3c62000000-3c62015000 r-xp 00000000 fd:00 444460 /lib64/libselinux.so.1
3c62015000-3c62215000 ---p 00015000 fd:00 444460 /lib64/libselinux.so.1
3c62215000-3c62217000 rw-p 00015000 fd:00 444460 /lib64/libselinux.so.1
3c62217000-3c62218000 rw-p 3c62217000 00:00 0
3c62400000-3c62443000 r-xp 00000000 fd:00 444466 /lib64/libssl.so.0.9.8b
3c62443000-3c62643000 ---p 00043000 fd:00 444466 /lib64/libssl.so.0.9.8b
3c62643000-3c62649000 rw-p 00043000 fd:00 444466 /lib64/libssl.so.0.9.8b
3c62800000-3c6288f000 r-xp 00000000 fd:01 34199 /usr/lib64/libkrb5.so.3.3
3c6288f000-3c62a8e000 ---p 0008f000 fd:01 34199 /usr/lib64/libkrb5.so.3.3
3c62a8e000-3c62a92000 rw-p 0008e000 fd:01 34199 /usr/lib64/libkrb5.so.3.3
3e29600000-3e2972c000 r-xp 00000000 fd:01 98154 /usr/lib64/mysql/libmysqlclient_r.so.15.0.0
3e2972c000-3e2992b000 ---p 0012c000 fd:01 98154 /usr/lib64/mysql/libmysqlclient_r.so.15.0.0
3e2992b000-3e29973000 rw-p 0012b000 fd:01 98154 /usr/lib64/mysql/libmysqlclient_r.so.15.0.0
3e29973000-3e29974000 rw-p 3e29973000 00:00 0
2aaaaaec9000-2aaaab02c000 rw-p 2aaaaaec9000 00:00 0
2aaaac000000-2aaaac229000 rw-p 2aaaac000000 00:00 0
2aaaac229000-2aaab0000000 ---p 2aaaac229000 00:00 0
2aaab0000000-2aaab0135000 rw-p 2aaab0000000 00:00 0
2aaab0135000-2aaab4000000 ---p 2aaab0135000 00:00 0
2b41fe5d8000-2b41fe5db000 rw-p 2b41fe5d8000 00:00 0
2b41fe5f1000-2b41fe5fa000 rw-p 2b41fe5f1000 00:00 0
2b41fe612000-2b41fe61c000 r-xp 00000000 fd:00 444444 /lib64/libnss_files-2.5.so
2b41fe61c000-2b41fe81b000 ---p 0000a000 fd:00 444444 /lib64/libnss_files-2.5.so
2b41fe81b000-2b41fe81c000 r--p 00009000 fd:00 444444 /lib64/libnss_files-2.5.so
2b41fe81c000-2b41fe81d000 rw-p 0000a000 fd:00 444444 /lib64/libnss_files-2.5.so
2b41fe81d000-2b4201dec000 r--p 00000000 fd:01 290472 /usr/lib/locale/locale-archive
7fffac4bc000-7fffac4d2000 rw-p 7fffac4bc000 00:00 0 [stack]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]

Program received signal SIGABRT, Aborted.
[Switching to Thread 1138366784 (LWP 29755)]
0x000000305dc30155 in raise () from /lib64/libc.so.6