cactid/spine - buffer Overflow on 64 bits sys

Kro · Post by **Kro** » Wed Apr 02, 2008 12:22 pm

Hello,

I'm installing cacti on a new host.
Poller is working find with cmd.php but not with spine.

I compilled without error and there is no error in cacti log...

The only thing is cacti stop to update graph.
RRDsProcessed:0 with spine.

Code: Select all

04/02/2008 07:15:00 PM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
04/02/2008 07:15:00 PM - SYSTEM STATS: Time:298.8481 Method:spine Processes:1 Threads:1 Hosts:2 HostsPerProcess:2 DataSources:15 [b]RRDsProcessed:0[/b]

Any idea why I got this issue ?

Kro · Post by **Kro** » Thu Apr 03, 2008 5:36 am

There is a way to test cactid poller alone ?

How I can debug what is happens ?

Kro · Post by **Kro** » Thu Apr 03, 2008 7:13 am

I found something strange.

I started spine manually and here after the result returned

Code: Select all

ben@cacti2:/home/share/cactid$ sudo ./spine 1 2
SPINE: Using spine config file [spine.conf]
SPINE: Version 0.8.7b starting
*** glibc detected *** ./spine: free(): invalid next size (fast): 0x0000000000703ff0 ***
======= Backtrace: =========
/lib/libc.so.6[0x2af320316b0a]
/lib/libc.so.6(cfree+0x8c)[0x2af32031a6fc]
./spine[0x40bd1f]
./spine[0x40c0a5]
./spine[0x4091b5]
./spine[0x409d43]
/lib/libpthread.so.0[0x2af31fbf5317]
/lib/libc.so.6(clone+0x6d)[0x2af320378d5d]
======= Memory map: ========
00400000-00412000 r-xp 00000000 08:01 21218365                           /home/share/cactid/spine
00611000-00613000 rw-p 00011000 08:01 21218365                           /home/share/cactid/spine
00613000-0073e000 rw-p 00613000 00:00 0                                  [heap]
40000000-40001000 ---p 40000000 00:00 0
40001000-40801000 rw-p 40001000 00:00 0
40801000-40802000 ---p 40801000 00:00 0
40802000-41002000 rw-p 40802000 00:00 0
41002000-41003000 ---p 41002000 00:00 0
41003000-41803000 rw-p 41003000 00:00 0
2aaaaaab0000-2aaaaaabd000 r-xp 00000000 08:01 12140558                   /lib/libgcc_s.so.1
2aaaaaabd000-2aaaaacbd000 ---p 0000d000 08:01 12140558                   /lib/libgcc_s.so.1
2aaaaacbd000-2aaaaacbe000 rw-p 0000d000 08:01 12140558                   /lib/libgcc_s.so.1
2aaaac000000-2aaaac021000 rw-p 2aaaac000000 00:00 0
2aaaac021000-2aaab0000000 ---p 2aaaac021000 00:00 0
2af31ed59000-2af31ed76000 r-xp 00000000 08:01 12142542                   /lib/ld-2.6.1.so
2af31ed76000-2af31ed7b000 rw-p 2af31ed76000 00:00 0
2af31ed7b000-2af31edba000 r--p 00000000 08:01 12156930                   /usr/lib/locale/en_US.utf8/LC_CTYPE
2af31edba000-2af31edc1000 r--s 00000000 08:01 12108754                   /usr/lib/gconv/gconv-modules.cache
2af31ef75000-2af31ef77000 rw-p 0001c000 08:01 12142542                   /lib/ld-2.6.1.so
2af31ef77000-2af31f008000 r-xp 00000000 08:01 12111608                   /usr/lib/libnetsnmp.so.10.0.1
2af31f008000-2af31f208000 ---p 00091000 08:01 12111608                   /usr/lib/libnetsnmp.so.10.0.1
2af31f208000-2af31f20c000 rw-p 00091000 08:01 12111608                   /usr/lib/libnetsnmp.so.10.0.1
2af31f20c000-2af31f240000 rw-p 2af31f20c000 00:00 0
2af31f240000-2af31f409000 r-xp 00000000 08:01 12108945                   /usr/lib/libmysqlclient_r.so.15.0.0
2af31f409000-2af31f609000 ---p 001c9000 08:01 12108945                   /usr/lib/libmysqlclient_r.so.15.0.0
2af31f609000-2af31f654000 rw-p 001c9000 08:01 12108945                   /usr/lib/libmysqlclient_r.so.15.0.0
2af31f654000-2af31f656000 rw-p 2af31f654000 00:00 0
2af31f656000-2af31f7b1000 r-xp 00000000 08:01 12109820                   /usr/lib/libcrypto.so.0.9.8
2af31f7b1000-2af31f9b1000 ---p 0015b000 08:01 12109820                   /usr/lib/libcrypto.so.0.9.8
2af31f9b1000-2af31f9d4000 rw-p 0015b000 08:01 12109820                   /usr/lib/libcrypto.so.0.9.8
2af31f9d4000-2af31f9d7000 rw-p 2af31f9d4000 00:00 0
2af31f9d7000-2af31f9ed000 r-xp 00000000 08:01 12109871                   /usr/lib/libz.so.1.2.3.3
2af31f9ed000-2af31fbed000 ---p 00016000 08:01 12109871                   /usr/lib/libz.so.1.2.3.3
2af31fbed000-2af31fbee000 rw-p 00016000 08:01 12109871                   /usr/lib/libz.so.1.2.3.3
2af31fbee000-2af31fbef000 rw-p 2af31fbee000 00:00 0
2af31fbef000-2af31fc05000 r-xp 00000000 08:01 12142823                   /lib/libpthread-2.6.1.so
2af31fc05000-2af31fe04000 ---p 00016000 08:01 12142823                   /lib/libpthread-2.6.1.so
2af31fe04000-2af31fe06000 rw-p 00015000 08:01 12142823                   /lib/libpthread-2.6.1.so
2af31fe06000-2af31fe0a000 rw-p 2af31fe06000 00:00 0
2af31fe0a000-2af31fe8a000 r-xp 00000000 08:01 12142813                   /lib/libm-2.6.1.so
2af31fe8a000-2af320089000 ---p 00080000 08:01 12142813                   /lib/libm-2.6.1.so
2af320089000-2af32008b000 rw-p 0007f000 08:01 12142813                   /lib/libm-2.6.1.so
2af32008b000-2af3200a1000 r-xp 00000000 08:01 12142815                   /lib/libnsl-2.6.1.so
2af3200a1000-2af3202a0000 ---p 00016000 08:01 12142815                   /lib/libnsl-2.6.1.so
2af3202a0000-2af3202a2000 rw-p 00015000 08:01 12142815                   /lib/libnsl-2.6.1.so
2af3202a2000-2af3202a5000 rw-p 2af3202a2000 00:00 0
2af3202a5000-2af3203f7000 r-xp 00000000 08:01 12142545                   /lib/libc-2.6.1.so
2af3203f7000-2af3205f6000 ---p 00152000 08:01 12142545                   /lib/libc-2.6.1.so
2af3205f6000-2af3205f9000 r--p 00151000 08:01 12142545                   /lib/libc-2.6.1.so
2af3205f9000-2af3205fb000 rw-p 00154000 08:01 12142545                   /lib/libc-2.6.1.so
2af3205fb000-2af320600000 rw-p 2af3205fb000 00:00 0
2af320600000-2af320605000 r-xp 00000000 08:01 12142547                   /lib/libcrypt-2.6.1.so
2af320605000-2af320804000 ---p 00005000 08:01 12142547                   /lib/libcrypt-2.6.1.so
2af320804000-2af320806000 rw-p 00004000 08:01 12142547                   /lib/libcrypt-2.6.1.so
2af320806000-2af320834000 rw-p 2af320806000 00:00 0
2af320834000-2af320836000 r-xp 00000000 08:01 12142812                   /lib/libdl-2.6.1.so
2af320836000-2af320a36000 ---p 00002000 08:01 12142812                   /lib/libdl-2.6.1.so
2af320a36000-2af320a38000 rw-p 00002000 08:01 12142812                   /lib/libdl-2.6.1.so
2af320a38000-2af320a3a000 rw-p 2af320a38000 00:00 0
2af320a3a000-2af320a44000 r-xp 00000000 08:01 12142818                   /lib/libnss_files-2.6.1.so
2af320a44000-2af320c43000 ---p 0000a000 08:01 12142818                   /lib/libnss_files-2.6.1.so
2af320c43000-2af320c45000 rw-p 00009000 08:01 12142818                   /lib/libnss_files-2.6.1.so
7fff8bd3c000-7fff8bd51000 rw-p 7fff8bd3c000 00:00 0                      [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vdso]
Aborted

I'm lost :/

Kro · Post by **Kro** » Thu Apr 03, 2008 7:30 am

Ok, it looks like the same issue than here : http://forums.cacti.net/about24216-0-asc-15.html

I'm also running it on a Linux 64 bits.

Kro · Post by **Kro** » Thu Apr 03, 2008 8:47 am

So, I updated Downed Device Detection to determine if a host is available for polling with SNMP only.

Now, when I run ./spine 1 2, it works without buffer overflow error

ben@cacti2:/home/share/cactid$ sudo ./spine 1 2
SPINE: Using spine config file [spine.conf]
SPINE: Version 0.8.7b starting
SPINE: Time: 0.1790 s, Threads: 10, Hosts: 3

returned

04/03/2008 03:46:36 PM - SYSTEM STATS: Time:95.9559 Method:spine Processes:1 Threads:10 Hosts:3 HostsPerProcess:3 DataSources:20 RRDsProcessed:13

but when spine is launch through poller.php, I have again a time out.

04/03/2008 03:45:00 PM - POLLER: Poller[0] Maximum runtime of 298 seconds exceeded. Exiting.
04/03/2008 03:45:00 PM - SYSTEM STATS: Time:298.7935 Method:spine Processes:1 Threads:10 Hosts:3 HostsPerProcess:3 DataSources:20 RRDsProcessed:0

Post by **TheWitness** » Thu Apr 03, 2008 12:24 pm

Please use the SVN version of Spine for now. When I get a few moments to focus on it, it will have 0.8.7c released.

Regards,

TheWitness

Kro · Post by **Kro** » Thu Apr 03, 2008 1:27 pm

I'm a little bit lost with SVN.

Should I download and compile this version: http://svn.cacti.net/cgi-bin/viewvc.cgi ... hes/0.8.7/ ?

Thanks.

Post by **TheWitness** » Thu Apr 03, 2008 2:37 pm

Use the "main", although, I think they are identical.

Regards,

TheWitness

Kro · Post by **Kro** » Wed Apr 09, 2008 9:16 am

I don't understand how to download all files :/
Any link to download all one shot ?

Thx

Kro · Post by **Kro** » Thu Apr 10, 2008 3:33 am

anybody to tell me how to download svn version of cactid/spine ?

Thx

Post by **gandalf** » Fri Apr 11, 2008 7:16 am

Please install svn and checkout head. Currently, we do not have nightly builds
Reinhard

Kro · Post by **Kro** » Fri Apr 11, 2008 7:49 am

In fact I already downloaded svn version last time but there was no spine files.
I just did it again and now I have spine svn version too...

I will try to compile it right now.

Btw, thank you

Kro · Post by **Kro** » Fri Apr 11, 2008 8:31 am

Ok, I compiled svn version using following command:

- autoreconf
- libtoolize --force
- configure
- make

I finally get spine v0.8.7c and... it is working on 64bit sys!

Thanks!

Kro · Post by **Kro** » Fri Apr 11, 2008 10:50 am

arrghhh no wait. Graph are not populated anymore :/

From cacti logs I can see that RRds are processed... What is happens ?

04/11/2008 05:45:47 PM - SPINE: Poller[0] Host[404] DS[4271] WARNING: Result from SNMP not valid. Partial Result: ...
04/11/2008 05:45:49 PM - SPINE: Poller[0] Host[404] DS[4271] WARNING: Result from SNMP not valid. Partial Result: ...
04/11/2008 05:45:49 PM - SPINE: Poller[0] Host[404] DS[4272] WARNING: Result from SNMP not valid. Partial Result: ...
04/11/2008 05:45:49 PM - SPINE: Poller[0] Host[404] DS[4272] WARNING: Result from SNMP not valid. Partial Result: ...
04/11/2008 05:45:52 PM - SYSTEM STATS: Time:50.9251 Method:spine Processes:1 Threads:10 Hosts:298 HostsPerProcess:298 DataSources:4537 RRDsProcessed:3101

Kro · Post by **Kro** » Mon Apr 14, 2008 9:36 am

I disabled all hosts to keep only one and started to debug.

First I ran ./spine --verbosity=5 432 432

All scripts returned valid values except for network interfaces.

04/14/2008 04:14:34 PM - SPINE: Poller[0] Host[432] DS[4593] WARNING: Result from SNMP not valid. Partial Result: ...
04/14/2008 04:14:34 PM - SPINE: Poller[0] Host[432] DS[4593] SNMP: v2: 10.254.1.17, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.2, value: U
04/14/2008 04:14:34 PM - SPINE: Poller[0] Host[432] DS[4593] WARNING: Result from SNMP not valid. Partial Result: ...
04/14/2008 04:14:34 PM - SPINE: Poller[0] Host[432] DS[4593] SNMP: v2: 10.254.1.17, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.2, value: U

I ran manually snmpget and get valid awnser:

snmpget -v 2c -c <Community> <HostIp> .1.3.6.1.2.1.2.2.1.10.2
returned: IF-MIB::ifInOctets.2 = Counter32: 4215928416

snmpget -v 2c -c <Community> <HostIp> .1.3.6.1.2.1.2.2.1.16.2
returned: IF-MIB::ifOutOctets.2 = Counter32: 2536489958

I tried to follow gandalf's instructions about NaNs graph without succes.

I repeated this for various hosts (routers/servers/ DNS servers/proxies...) and found that only SNMP polling do not return any values with spine (it works manually & with php poller).

I'm really stuck here :'(

Cacti

cactid/spine - buffer Overflow on 64 bits sys

cactid/spine - buffer Overflow on 64 bits sys

Who is online