Device down if use ICMP in Availability/Reachability Options

[brightm]

I'm trying to upgrade from the Windows version of spine - 0.8.7e to 0.8.7g and am also having the same problem where all my hosts/devices are reported as down using ICMP ping. If i manually change a device to use udp ping the device moves to recovering. Any ideas?

Thanks

[TheWitness]

For one of the hosts, change to icmp. Then run the following

Code: Select all

./spine -V 5 -f <hostid> -l <hostid> -R

Post the output

[brightm]

Hi Witness,

Thanks for the reply, sorry i haven't been able to get back to you sooner. Here is the output from the command:

• C:\Temp\Downloads\Cacti\Spine\spine-0.8.7g>.\spine -V 5 -f 10.10.0.2 -l 10.10.0.
  2 -R
  NOTE: The Shell Command Exists in the current directory
  SPINE: Using spine config file [spine.conf]
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The path_php_server variable is
  c:/wamp/www/cacti/script_server.php
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The path_cactilog variable is C
  :/wamp/www/cacti/log/cacti.log
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The log_destination variable is
  1 (FILE)
  cygwin warning:
  MS-DOS style path detected: C:/wamp/www/cacti/log/cacti.log
  Preferred POSIX equivalent is: /cygdrive/c/wamp/www/cacti/log/cacti.log
  CYGWIN environment variable option "nodosfilewarning" turns off this warning.
  Consult the user's guide for more details about POSIX paths:
  http://cygwin.com/cygwin-ug-net/using.h ... -pathnames
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The path_php variable is c:/wam
  p/bin/php/php5.2.11/php.exe
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The availability_method variabl
  e is 3
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The ping_recovery_count variabl
  e is 3
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The ping_failure_count variable
  is 2
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The ping_method variable is 1
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The ping_retries variable is 2
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The ping_timeout variable is 40
  0
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The snmp_retries variable is 3
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The log_perror variable is 1
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The log_pwarn variable is 1
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The boost_redirect variable is
  0
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The log_pstats variable is 1
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The threads variable is 15
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The polling interval is 300 sec
  onds
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The number of concurrent proces
  ses is 2
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The script timeout is 25
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The number of php script server
  s to run is 5
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: StartHost='10', EndHost='10', T
  otalPHPScripts='0'
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The PHP Script Server is Not Re
  quired
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: The Maximum SNMP OID Get Size i
  s 40
  10/06/2010 08:14:14 PM - SPINE: Poller[0] Version 0.8.7g starting
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: MySQL is Thread Safe!
  10/06/2010 08:14:14 PM - SPINE: Poller[0] SPINE: Initializing Net-SNMP API
  10/06/2010 08:14:14 PM - SPINE: Poller[0] DEBUG: Issues with SNMP Header Version
  information, assuming old version of Net-SNMP.
  10/06/2010 08:14:15 PM - SPINE: Poller[0] SPINE: Initializing PHP Script Server(
  s)
  10/06/2010 08:14:15 PM - SPINE: Poller[0] NOTE: Spine did not detect multithread
  ed device polling.
  10/06/2010 08:14:15 PM - SPINE: Poller[0] NOTE: Spine is behaving in a PRE 0.8.7
  g manner
  10/06/2010 08:14:15 PM - SPINE: Poller[0] DEBUG: Initial Value of Active Threads
  is 0
  10/06/2010 08:14:15 PM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
  10/06/2010 08:14:15 PM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polli
  ng of Host
  10/06/2010 08:14:15 PM - SPINE: Poller[0] DEBUG: The Value of Active Threads is
  1
  10/06/2010 08:14:15 PM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polli
  ng of Host
  10/06/2010 08:14:15 PM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
  10/06/2010 08:14:15 PM - SPINE: Poller[0] DEBUG: The Value of Active Threads is
  2
  10/06/2010 08:14:15 PM - SPINE: Poller[0] Host[0] TH[1] DEBUG: HOST COMPLETE: Ab
  out to Exit Host Polling Thread Function
  10/06/2010 08:14:15 PM - SPINE: Poller[0] DEBUG: The Value of Active Threads is
  1
  10/06/2010 08:14:15 PM - SPINE: Poller[0] Host[10] PING: Result ICMP: Ping timed
  out
  10/06/2010 08:14:15 PM - SPINE: Poller[0] Host[10] TH[1] NOTE: There are '212' P
  olling Items for this Host
  10/06/2010 08:14:15 PM - SPINE: Poller[0] Host[10] TH[1] DEBUG: HOST COMPLETE: A
  bout to Exit Host Polling Thread Function
  10/06/2010 08:14:15 PM - SPINE: Poller[0] DEBUG: The Value of Active Threads is
  0
  10/06/2010 08:14:15 PM - SPINE: Poller[0] DEBUG: Thread Cleanup Complete
  10/06/2010 08:14:15 PM - SPINE: Poller[0] DEBUG: PHP Script Server Pipes Closed
  10/06/2010 08:14:15 PM - SPINE: Poller[0] DEBUG: Allocated Variable Memory Freed
  
  10/06/2010 08:14:15 PM - SPINE: Poller[0] DEBUG: MYSQL Free & Close Completed
  10/06/2010 08:14:15 PM - SPINE: Poller[0] Time: 0.7030 s, Threads: 15, Hosts: 2
  
  C:\Temp\Downloads\Cacti\Spine\spine-0.8.7g>

I ran the command form the download directory because the older version is in operation. Hope that's ok (path variables, etc).

[TheWitness]

10.10.x.x is NOT a 'host_id', it's an 'ip address'. You need to use 'host_id's. Make that change and try again.

TheWitness

[wedge]

Here is mine. I'm having the same issue as well. I have a few hosts that I only monitor via ICMP ping. All of these hosts show down despite being able to ping them. This started after my upgrade from 0.8.7e to 0.8.7.g.

I'm using W2K3 Server.

Code: Select all

C:\Spine>.\spine -V 5 -f 6 -l 6 -R
NOTE: The Shell Command Exists in the /bin directory
SPINE: Using spine config file [spine.conf]
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The path_php_server variable is
 c:/Apache2/htdocs/cacti/script_server.php
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The path_cactilog variable is C
:/Apache2/htdocs/cacti/log/cacti.log
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The log_destination variable is
 1 (FILE)
cygwin warning:
  MS-DOS style path detected: C:/Apache2/htdocs/cacti/log/cacti.log
  Preferred POSIX equivalent is: /cygdrive/c/Apache2/htdocs/cacti/log/cacti.log
  CYGWIN environment variable option "nodosfilewarning" turns off this warning.
  Consult the user's guide for more details about POSIX paths:
    http://cygwin.com/cygwin-ug-net/using.html#using-pathnames
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The path_php variable is C:/php/php.exe
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The availability_method variable is 4
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The ping_recovery_count variable is 3
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The ping_failure_count variable is 2
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The ping_method variable is 1
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The ping_retries variable is 1
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The ping_timeout variable is 400
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The snmp_retries variable is 3
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The log_perror variable is 0
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The log_pwarn variable is 0
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The boost_redirect variable is0
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The log_pstats variable is 0
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The threads variable is 50
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The polling interval is 60 seconds
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The number of concurrent processes is 4
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The script timeout is 30
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The number of php script servers to run is 1
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: StartHost='6', EndHost='6', TotalPHPScripts='0'
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The PHP Script Server is Not Required
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: The Maximum SNMP OID Get Size is 60
10/10/2010 11:51:47 AM - SPINE: Poller[0] Version 0.8.7g starting
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: MySQL is Thread Safe!
10/10/2010 11:51:47 AM - SPINE: Poller[0] SPINE: Initializing Net-SNMP API
10/10/2010 11:51:47 AM - SPINE: Poller[0] DEBUG: Issues with SNMP Header Version information, assuming old version of Net-SNMP.
10/10/2010 11:51:48 AM - SPINE: Poller[0] SPINE: Initializing PHP Script Server(s)
10/10/2010 11:51:48 AM - SPINE: Poller[0] NOTE: Spine did not detect multithreaded device polling.
10/10/2010 11:51:48 AM - SPINE: Poller[0] NOTE: Spine is behaving in a 0.8.7g manner
10/10/2010 11:51:48 AM - SPINE: Poller[0] DEBUG: Initial Value of Active Threads is 0
10/10/2010 11:51:48 AM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
10/10/2010 11:51:48 AM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polling of Host
10/10/2010 11:51:48 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is1
10/10/2010 11:51:48 AM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
10/10/2010 11:51:48 AM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polling of Host
10/10/2010 11:51:48 AM - SPINE: Poller[0] Host[0] TH[1] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
10/10/2010 11:51:48 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is2
10/10/2010 11:51:48 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is1
10/10/2010 11:51:48 AM - SPINE: Poller[0] Host[6] PING Result: ICMP: Ping timedout
10/10/2010 11:51:48 AM - SPINE: Poller[0] Host[6] TH[1] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
10/10/2010 11:51:48 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is0
10/10/2010 11:51:48 AM - SPINE: Poller[0] DEBUG: Thread Cleanup Complete
10/10/2010 11:51:48 AM - SPINE: Poller[0] DEBUG: PHP Script Server Pipes Closed
10/10/2010 11:51:48 AM - SPINE: Poller[0] DEBUG: Allocated Variable Memory Freed
10/10/2010 11:51:48 AM - SPINE: Poller[0] DEBUG: MYSQL Free & Close Completed
10/10/2010 11:51:48 AM - SPINE: Poller[0] Time: 1.4050 s, Threads: 50, Hosts: 

[TheWitness]

Pm me a tcpdump of that trace. Filter on the destination host. Also, increase the timeout to a few seconds.

TheWitness

[wedge]

I'll get you a trace as soon as I figure out how to do it on windows. I have also noticed this:

On the device screen it reports a successful ICMP ping like this.



But on the main devices screen it is still reported as down.



I'm not sure if this helps but I thought I'd mention it.

[TheWitness]

Wireshark. Download it.

[wedge]

I found windump.

I'll get you the traces shortly.

:edit:

I don't seem to have the ability to PM people, or upload cap files.

[TheWitness]

Send via e-mail.

[wedge]

Sent.

[brightm]

Hi Witness,

Once again sorry for the long delay in my reply!

Here is the output you ask for with the correct hostid value this time and a trace attached:

• C:\Temp\Downloads\Cacti\Spine\spine-0.8.7g>.\spine -V 5 -f 2 -l 2 -R
  NOTE: The Shell Command Exists in the current directory
  SPINE: Using spine config file [spine.conf]
  10/19/2010 11:12:15 AM - SPINE: Poller[0] DEBUG: The path_php_server variable is
  c:/wamp/www/cacti/script_server.php
  10/19/2010 11:12:15 AM - SPINE: Poller[0] DEBUG: The path_cactilog variable is C
  :/wamp/www/cacti/log/cacti.log
  10/19/2010 11:12:15 AM - SPINE: Poller[0] DEBUG: The log_destination variable is
  1 (FILE)
  cygwin warning:
  MS-DOS style path detected: C:/wamp/www/cacti/log/cacti.log
  Preferred POSIX equivalent is: /cygdrive/c/wamp/www/cacti/log/cacti.log
  CYGWIN environment variable option "nodosfilewarning" turns off this warning.
  Consult the user's guide for more details about POSIX paths:
  http://cygwin.com/cygwin-ug-net/using.h ... -pathnames
  10/19/2010 11:12:15 AM - SPINE: Poller[0] DEBUG: The path_php variable is c:/wam
  p/bin/php/php5.2.11/php.exe
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: The availability_method variabl
  e is 3
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: The ping_recovery_count variabl
  e is 3
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: The ping_failure_count variable
  is 2
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: The ping_method variable is 1
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: The ping_retries variable is 2
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: The ping_timeout variable is 40
  0
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: The snmp_retries variable is 3
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: The log_perror variable is 1
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: The log_pwarn variable is 1
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: The boost_redirect variable is
  0
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: The log_pstats variable is 1
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: The threads variable is 15
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: The polling interval is 300 sec
  onds
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: The number of concurrent proces
  ses is 2
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: The script timeout is 25
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: The number of php script server
  s to run is 5
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: StartHost='2', EndHost='2', Tot
  alPHPScripts='0'
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: The PHP Script Server is Not Re
  quired
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: The Maximum SNMP OID Get Size i
  s 40
  10/19/2010 11:12:16 AM - SPINE: Poller[0] Version 0.8.7g starting
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: MySQL is Thread Safe!
  10/19/2010 11:12:16 AM - SPINE: Poller[0] SPINE: Initializing Net-SNMP API
  10/19/2010 11:12:16 AM - SPINE: Poller[0] DEBUG: Issues with SNMP Header Version
  information, assuming old version of Net-SNMP.
  10/19/2010 11:12:17 AM - SPINE: Poller[0] SPINE: Initializing PHP Script Server(
  s)
  10/19/2010 11:12:17 AM - SPINE: Poller[0] NOTE: Spine did not detect multithread
  ed device polling.
  10/19/2010 11:12:17 AM - SPINE: Poller[0] NOTE: Spine is behaving in a PRE 0.8.7
  g manner
  10/19/2010 11:12:17 AM - SPINE: Poller[0] DEBUG: Initial Value of Active Threads
  is 0
  10/19/2010 11:12:17 AM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
  10/19/2010 11:12:17 AM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polli
  ng of Host
  10/19/2010 11:12:17 AM - SPINE: Poller[0] Host[0] TH[1] DEBUG: HOST COMPLETE: Ab
  out to Exit Host Polling Thread Function
  10/19/2010 11:12:17 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is
  1
  10/19/2010 11:12:17 AM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
  10/19/2010 11:12:17 AM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polli
  ng of Host
  10/19/2010 11:12:17 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is
  2
  10/19/2010 11:12:17 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is
  1
  10/19/2010 11:12:17 AM - SPINE: Poller[0] Host[2] PING: Result ICMP: Ping timed
  out
  10/19/2010 11:12:17 AM - SPINE: Poller[0] Host[2] TH[1] NOTE: There are '1190' P
  olling Items for this Host
  10/19/2010 11:12:18 AM - SPINE: Poller[0] Host[2] TH[1] DEBUG: HOST COMPLETE: Ab
  out to Exit Host Polling Thread Function
  10/19/2010 11:12:18 AM - SPINE: Poller[0] DEBUG: The Value of Active Threads is
  0
  10/19/2010 11:12:18 AM - SPINE: Poller[0] DEBUG: Thread Cleanup Complete
  10/19/2010 11:12:18 AM - SPINE: Poller[0] DEBUG: PHP Script Server Pipes Closed
  10/19/2010 11:12:18 AM - SPINE: Poller[0] DEBUG: Allocated Variable Memory Freed
  
  10/19/2010 11:12:18 AM - SPINE: Poller[0] DEBUG: MYSQL Free & Close Completed
  10/19/2010 11:12:18 AM - SPINE: Poller[0] Time: 3.7340 s, Threads: 15, Hosts: 2

Thanks in advance for your help.

[blkcamarozr28]

I also have this problem.
Running CentOS v5.5
Cacti: Version 0.8.7g (All patches applied as of 10/21/2010)
SPINE: Version 0.8.7g (All patches applied as of 10/21/2010)

When Downed Device Detection is set to ICMP it errors out in the logs. However it configuration menu shows 0.3ms for ICMP Pings.
SPINE: Poller[0] Host[11] Hostname[172.31.x.x] ERROR: HOST EVENT: Host is DOWN Message: UDP: Ping timed out

If I change the Downed Decvice Detection to SNMP it works fine.

[blkcamarozr28]

Here's my permissions on spine.

[root@mrtg3 bin]# ll -la |grep spine
-rwxr-xr-x 1 root root 216449 Oct 21 16:59 spine
[root@mrtg3 bin]# pwd
/usr/bin

[TFC]

If polling with spine, you need to set the setuid bit to root and change ownership to root as well. Only after this will ICMP ping work in Spine.

TheWitness

How can i do that? what is the command?