Spine 0.8.7d-pre2 Available for Testing

[TheWitness]

A quick utility.

With regard to data loss. Can you poll once with the good version at medium and then the new version and send me the output.

./spine -R --stdout -V 3

TheWitness

The new version will stop those segfaults.

[skinty]

No segfaults so far. I ran a technical support and I'm not seeing any issues with the tables. I'm sending the output in PM shortly.

[mxtommy]

As requested in my other thread in the linux forum, I tried the pre2 version, but am still getting the same segfaults right after recvmsg.

Code: Select all

[pid 11697] write(6, "R\1\0\0\3UPDATE host SET status=\'3\',"..., 342) = 342
[pid 11697] read(6, "0\0\0\1\0\1\0\2\0\1\0(Rows matched: 1  Cha"..., 16384) = 52
[pid 11697] poll([{fd=6, events=POLLIN|POLLPRI}], 1, 0) = 0
[pid 11697] write(6, "[\0\0\0\3SELECT data_query_id, actio"..., 95) = 95
[pid 11697] read(6, "\1\0\0\1\5U\0\0\2\3def\tcacti_new\16poller_r"..., 16384) = 443
[pid 11697] sendmsg(7, {msg_name(16)={sa_family=AF_INET, sin_port=htons(162), sin_addr=inet_addr("OURIP")}, msg_iov(1)=[{"0,\2\1\0\4\tOURCOMMUNITY\240\34\2\4sviP\2\1\0\2\1\0000\16"..., 46}], msg_controllen=24, {cmsg_len=24, cmsg_level=SOL_IP, cmsg_type=, ...}, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 46
[pid 11697] gettimeofday({1238069175, 564821}, NULL) = 0
[pid 11697] gettimeofday({1238069175, 564905}, NULL) = 0
[pid 11697] select(8, [7], NULL, NULL, {0, 499916}) = 1 (in [7], left {0, 499916})
[pid 11697] recvmsg(7, {msg_name(16)={sa_family=AF_INET, sin_port=htons(162), sin_addr=inet_addr("OURIP")}, msg_iov(1)=[{"0\36\2\1\0\4\tOURCOMMUNITY\242\16\2\4sviP\2\1\0\2\1\0000\0"..., 65536}], msg_controllen=0, msg_flags=0}, 0) = 32
[pid 11697] --- SIGSEGV (Segmentation fault) @ 0 (0) ---
[pid 11697] rt_sigaction(SIGSEGV, {SIG_DFL}, {0x8054a80, [SEGV], SA_RESTART}, 8) = 0
[pid 11697] write(1, "FATAL: Spine Encountered a Segme"..., 61FATAL: Spine Encountered a Segmentation Fault (Spine thread)
) = 61

Code: Select all

DEBUG: In Poller, About to Start Polling of Host
DEBUG: The Value of Active Threads is 1
Host[157] DEBUG: ICMP Host Alive, Try Count:1, Time:0.2210 ms
Host[157] PING: Result ICMP: Host is Alive
Host[157] RECACHE: Processing 1 items in the auto reindex cache for 'OURIP'
FATAL: Spine Encountered a Segmentation Fault (Spine thread)

If there's any other debugging tools I can run to find out what spine is doing at that moment please let me know.

Thanks!

[skinty]

Code: Select all

03/26/2009 10:51:16 AM - SYSTEM STATS: Time:74.1688 Method:spine Processes:8 Threads:20 Hosts:364 HostsPerProcess:46 DataSources:19010 RRDsProcessed:9352
03/26/2009 10:50:49 AM - SPINE: Poller[0] Host[81] ERROR: The POPEN timed out
03/26/2009 10:50:48 AM - SPINE: Poller[0] Host[264] ERROR: The POPEN timed out
03/26/2009 10:50:44 AM - SPINE: Poller[0] Host[274] ERROR: Empty result [xxxxxxx]: '/usr/bin/php -q /usr/share/cacti/scripts/cisco_cpu_usage.php xxxxxxxx, , 161, 500 get oneMin 1'
03/26/2009 10:50:40 AM - SPINE: Poller[0] Host[274] ERROR: Empty result [xxxxxxx]: '/usr/bin/php -q /usr/share/cacti/scripts/cisco_cpu_usage.php xxxxxxxxx, , 161, 500 get fiveMin 1'
03/26/2009 10:50:38 AM - SPINE: Poller[0] Host[346] ERROR: The POPEN timed out
03/26/2009 10:50:36 AM - SPINE: Poller[0] Host[398] ERROR: The POPEN timed out
03/26/2009 10:50:31 AM - SPINE: Poller[0] Host[399] ERROR: The POPEN timed out
03/26/2009 10:50:30 AM - SPINE: Poller[0] Host[336] ERROR: The POPEN timed out
03/26/2009 10:50:29 AM - SPINE: Poller[0] Host[474] ERROR: The POPEN timed out
03/26/2009 10:50:28 AM - SPINE: Poller[0] Host[479] ERROR: The POPEN timed out
03/26/2009 10:50:27 AM - SPINE: Poller[0] Host[473] ERROR: The POPEN timed out
03/26/2009 10:50:27 AM - SPINE: Poller[0] WARNING: SS[0] The PHP Script Server did not respond in time and will therefore be restarted
03/26/2009 10:50:27 AM - SPINE: Poller[0] Host[499] ERROR: The POPEN timed out
03/26/2009 10:50:24 AM - SPINE: Poller[0] Host[472] ERROR: The POPEN timed out
03/26/2009 10:50:20 AM - SPINE: Poller[0] Host[313] ERROR: The POPEN timed out
03/26/2009 10:50:20 AM - SPINE: Poller[0] Host[305] ERROR: The POPEN timed out
03/26/2009 10:50:17 AM - SPINE: Poller[0] Host[26] DS[211] WARNING: SNMP timeout detected [500 ms], ignoring host '192.168.2.16'
03/26/2009 10:50:17 AM - SPINE: Poller[0] Host[26] DS[210] WARNING: SNMP timeout detected [500 ms], ignoring host '192.168.2.16'
03/26/2009 10:50:14 AM - SPINE: Poller[0] WARNING: Host[322] DataQuery[19] Reindex Check FAILED: No SNMP Session. If not an SNMP host, don't use Uptime Goes Backwards!
03/26/2009 10:50:14 AM - SPINE: Poller[0] WARNING: Host[322] DataQuery[1] Reindex Check FAILED: No SNMP Session. If not an SNMP host, don't use Uptime Goes Backwards! 

TCP ping still showing the uptime error issue. However, it does seem to be fairly fast. I'm working on converting a couple of hosts to snmpv3, so I should have something to look later for that as well.

-ryan

[TheWitness]

Stinky,

Here is your problem with the new Spine, not enough MySQL connections:

Code: Select all

MYSQL: Connection Failed: Error:'2002', Message:'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (11)'
MYSQL: Connection Failed: Error:'2002', Message:'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (11)'
MYSQL: Connection Failed: Error:'2002', Message:'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (11)'
MYSQL: Connection Failed: Error:'2002', Message:'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (11)'
MYSQL: Connection Failed: Error:'2002', Message:'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (11)'
MYSQL: Connection Failed: Error:'2002', Message:'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (11)'
MYSQL: Connection Failed: Error:'2002', Message:'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (11)'
MYSQL: Connection Failed: Error:'2002', Message:'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (11)'
MYSQL: Connection Failed: Error:'2002', Message:'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (11)'
MYSQL: Connection Failed: Error:'2002', Message:'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (11)'
MYSQL: Connection Failed: Error:'2002', Message:'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (11)'
MYSQL: Connection Failed: Error:'2002', Message:'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (11)'

TheWitness

[burn26]

checking how to run the C++ preprocessor... /lib/cpp
configure: error: C++ preprocessor "/lib/cpp" fails sanity check
See `config.log' for more details.

cpp (GCC) 4.1.2 20071124 (Red Hat 4.1.2-42)
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[sizulku]

got this error

Code: Select all

04/03/2009 09:40:01 AM - SPINE: Poller[0] FATAL: Spine Encountered a Segmentation Fault (Spine parent)
04/03/2009 09:40:01 AM - SPINE: Poller[0] FATAL: Spine Encountered a Segmentation Fault (Spine parent)

[TheWitness]

Please test SVN. I have been unable to muster time to issue pre3 as I have not snmpv3 testing done as of yet.

TheWitness

[sizulku]

stil getting error with SVN, any help would be great

Code: Select all

04/06/2009 10:07:01 PM - SPINE: Poller[0] FATAL: Spine Encountered a Segmentation Fault (Spine parent)  
04/06/2009 10:07:01 PM - SPINE: Poller[0] FATAL: Spine Encountered a Segmentation Fault (Spine parent)  
04/06/2009 10:07:00 PM - SYSTEM STATS: Time:58.5090 Method:spine Processes:2 Threads:20 Hosts:99 HostsPerProcess:50 DataSources:282 RRDsProcessed:0  
04/06/2009 10:07:00 PM - POLLER: Poller[0] Maximum runtime of 58 seconds exceeded. Exiting.  
04/06/2009 10:06:01 PM - SPINE: Poller[0] FATAL: Spine Encountered a Segmentation Fault (Spine parent)  
04/06/2009 10:06:01 PM - SPINE: Poller[0] FATAL: Spine Encountered a Segmentation Fault (Spine parent)  
04/06/2009 10:06:01 PM - SYSTEM STATS: Time:59.5154 Method:spine Processes:2 Threads:20 Hosts:99 HostsPerProcess:50 DataSources:282 RRDsProcessed:0  

[dononeil]

I had build issues with the latest SVN as of this morning:

checking for HP-UX gethostbyname_r... no
configure: creating ./config.status
config.status: creating Makefile
config.status: creating config/config.h
config.status: executing depfiles commands
rm -f spine spine
rm -rf .libs _libs
rm -f *.o
rm -f *.lo
source='sql.c' object='sql.o' libtool=no \
DEPDIR=.deps depmode=none /bin/sh ./config/depcomp \
gcc -DHAVE_CONFIG_H -I. -I. -I./config -I/usr/include/net-snmp -I/usr/include/net-snmp/.. -I/usr/include/mysql -g -O2 -c sql.c
: command not foundine 2:
: command not foundine 5:
: command not foundine 10:
: command not foundine 15:
: command not foundine 20:
: command not foundine 25:
: command not foundine 27:
'/config/depcomp: line 64: syntax error near unexpected token `in
'/config/depcomp: line 64: `case "$depmode" in
make: *** [sql.o] Error 2


Trying to build on SLES10.2 64 Bit. 87c built ok for me.

[TheWitness]

Rerun autoconf prior to configure.

TheWitness

[dononeil]

Rerun autoconf prior to configure.

TheWitness

Didn't help.... I re-rean autoconf and then a make and I still get the same errors.

DEPDIR=.deps depmode=none /bin/sh ./config/depcomp \
gcc -DHAVE_CONFIG_H -I. -I. -I./config -I/usr/include/net-snmp -I/usr/include/net-snmp/.. -I/usr/include/mysql -g -O2 -c sql.c
: command not foundine 2:
: command not foundine 5:
: command not foundine 10:
: command not foundine 15:
: command not foundine 20:
: command not foundine 25:
: command not foundine 27:
'/config/depcomp: line 64: syntax error near unexpected token `in
'/config/depcomp: line 64: `case "$depmode" in
make: *** [sql.o] Error 2

Looks like it may not be finding the correct mysql libs or something... any ideas? I'd really like to test this today.

[thebofh]

I'm an old cacti user. I've just recently tried to get cacti going again. I'm running into the "WARNING: SNMP timeout detected [500ms], ignoring host" problem with spine Version 0.8.7d-pre2. I've tried a few different versions of spine and the problem persists. I'm using SNMPv3 authPriv on my network. Here is a log:

Code: Select all

$ ./spine -V 5 -C /etc/cacti/spine.conf 
SPINE: Using spine config file [/etc/cacti/spine.conf]
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The path_php_server variable is /usr/share/cacti/site/script_server.php
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The path_cactilog variable is /var/log/cacti/cacti.log
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The log_destination variable is 1 (FILE)
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The path_php variable is /usr/bin/php
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The availability_method variable is 3
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The ping_recovery_count variable is 3
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The ping_failure_count variable is 2
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The ping_method variable is 3
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The ping_retries variable is 2
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The ping_timeout variable is 400
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The log_perror variable is 1
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The log_pwarn variable is 1
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The log_pstats variable is 1
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The threads variable is 10
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The polling interval is 300 seconds
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The number of concurrent processes is 1
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The script timeout is 25
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The number of php script servers to run is 1
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: StartHost='-1', EndHost='-1', TotalPHPScripts='0'
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The PHP Script Server is Not Required
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The Maximum SNMP OID Get Size is 10
04/09/2009 10:26:59 AM - SPINE: Poller[0] Version 0.8.7d-pre2 starting
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: MySQL is Thread Safe!
04/09/2009 10:26:59 AM - SPINE: Poller[0] SPINE: Initializing Net-SNMP API
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: SNMP Header Version is 5.4.1
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: SNMP Library Version is 5.4.1
04/09/2009 10:26:59 AM - SPINE: Poller[0] SPINE: Initializing PHP Script Server(s)
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: Initial Value of Active Threads is 0
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polling of Host
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 1
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polling of Host
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 2
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polling of Host
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 3
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] DEBUG: TCP Host Alive, Try Count:1, Time:15.2090 ms
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] PING: Result TCP: Host is Alive
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[0] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 2
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] Host has no information for recache.
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] NOTE: There are '7' Polling Items for this Host
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] DEBUG: The POPEN returned the following File Descriptor 5
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] DS[3] SCRIPT: perl /usr/share/cacti/site/scripts/linux_memory.pl MemFree:, output: 1453712
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] DEBUG: The POPEN returned the following File Descriptor 5
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] DS[4] SCRIPT: perl /usr/share/cacti/site/scripts/linux_memory.pl SwapFree:, output: 409616
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] DEBUG: The POPEN returned the following File Descriptor 5
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[2] DEBUG: TCP Host Alive, Try Count:1, Time:29.0811 ms
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[2] PING: Result TCP: Host is Alive
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[2] Host has no information for recache.
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] DS[5] SCRIPT: perl /usr/share/cacti/site/scripts/loadavg_multi.pl, output: 1min:0.15 5min:0.29 10min:0.23
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] DEBUG: The POPEN returned the following File Descriptor 5
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[2] NOTE: There are '15' Polling Items for this Host
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] DS[6] SCRIPT: perl /usr/share/cacti/site/scripts/unix_users.pl , output: 1
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] DEBUG: The POPEN returned the following File Descriptor 5
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] DS[7] SCRIPT: perl /usr/share/cacti/site/scripts/unix_processes.pl, output: 66
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] DEBUG: The POPEN returned the following File Descriptor 5
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] DS[8] SCRIPT: perl /usr/share/cacti/site/scripts/query_unix_partitions.pl  get available /dev/sda1, output: 6017824
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] DEBUG: The POPEN returned the following File Descriptor 5
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] DS[8] SCRIPT: perl /usr/share/cacti/site/scripts/query_unix_partitions.pl  get used /dev/sda1, output: 1434360
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[1] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
04/09/2009 10:26:59 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 1
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[2] DS[9] WARNING: SNMP timeout detected [500 ms], ignoring host 'foo.example.com'
04/09/2009 10:26:59 AM - SPINE: Poller[0] Host[2] DS[9] SNMP: v3: foo.example.com, dsname: users, oid: .1.3.6.1.2.1.25.1.5.0, value: U
04/09/2009 10:27:00 AM - SPINE: Poller[0] Host[2] DS[10] WARNING: SNMP timeout detected [500 ms], ignoring host 'foo.example.com'
04/09/2009 10:27:00 AM - SPINE: Poller[0] Host[2] DS[10] SNMP: v3: foo.example.com, dsname: proc, oid: .1.3.6.1.2.1.25.1.6.0, value: U
04/09/2009 10:27:00 AM - SPINE: Poller[0] Host[2] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
04/09/2009 10:27:00 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 0
04/09/2009 10:27:00 AM - SPINE: Poller[0] DEBUG: Thread Cleanup Complete
04/09/2009 10:27:00 AM - SPINE: Poller[0] DEBUG: PHP Script Server Pipes Closed
04/09/2009 10:27:00 AM - SPINE: Poller[0] DEBUG: Allocated Variable Memory Freed
04/09/2009 10:27:00 AM - SPINE: Poller[0] DEBUG: MYSQL Free & Close Completed
04/09/2009 10:27:00 AM - SPINE: Poller[0] Time: 0.7995 s, Threads: 10, Hosts: 3

If I run snmpget manually on those OIDs I get the following:

Code: Select all

$ snmpwalk -v 3 -a SHA -x AES -n '' -l authPriv -u myuser -A mysecret -X mysecret foo.example.com .1.3.6.1.2.1.25.1.5.0
HOST-RESOURCES-MIB::hrSystemNumUsers.0 = Gauge32: 0
$ snmpwalk -v 3 -a SHA -x AES -n '' -l authPriv -u myuser -A mysecret -X mysecret foo.example.com .1.3.6.1.2.1.25.1.6.0
HOST-RESOURCES-MIB::hrSystemProcesses.0 = Gauge32: 49
$

I've tried compiling spine with and without the '--enable-nifty-popen' option. Same results. Here's my uname (Ubuntu 8.10):

Code: Select all

$ uname -a
Linux cacti 2.6.27-11-server #1 SMP Wed Apr 1 21:34:13 UTC 2009 x86_64 GNU/Linux

Any ideas?

[TheWitness]

SNMPv3 Issue resolved. This was caused by changes in the net-snmp API starting with net-snmp 5.4. I believe it to be a bug in net-snmp. However, the changes in pre3 will correct the behavior non the less.

TheWitness

[TheWitness]

Pre3 is now released again... I need some confirmation. This will be the final release candidate.

http://forums.cacti.net/viewtopic.php?t=32060

TheWitness