Since updating to 0.8.8a i get these "SPINE: Poller[0] Host[1467] TH[1] DS[325680] WARNING: SNMP timeout detected [2000 ms], ignoring host" Error Messages in my log.
I already tried to reduce the max_oids Value for the affected Hosts to 1 (in steps of /2 (starting with 32) and increasing the snmp_timeout Value up to 2000ms (in steps of 500ms starting with 500) with no result. I also tried to rebuild the poller cache and reindexing the hosts (with the cli script).
The Problem is with about 160 of 1560 devices, All of them have snmp_version 1 (i have some other snmp_version 1 devices which work fine though).
If i use snmpwalk on the devices i get the expected results (with the same timeout value set by using the -t option). I am also confused how spine can finish in 0.16 seconds but state that there was a snmptimeout of 2000ms.
The consequence of these errors is that i have missing graphs for 2 of the 160 hosts and Nan Values (empty graphs) for the others.
spine output with verbosity=5 on one of the hosts:
Code: Select all
SPINE: Using spine config file [/etc/cacti/spine.conf]
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The path_php_server variable is /usr/share/cacti/site/script_server.php
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The path_cactilog variable is /var/log/cacti/cacti.log
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The log_destination variable is 1 (FILE)
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The path_php variable is /usr/bin/php
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The availability_method variable is 2
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The ping_recovery_count variable is 3
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The ping_failure_count variable is 2
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The ping_method variable is 3
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The ping_retries variable is 1
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The ping_timeout variable is 60
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The snmp_retries variable is 3
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The log_perror variable is 1
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The log_pwarn variable is 1
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The boost_redirect variable is 1
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The log_pstats variable is 0
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The threads variable is 20
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The polling interval is 300 seconds
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The number of concurrent processes is 1
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The script timeout is 10
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The number of php script servers to run is 1
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: StartHost='1369', EndHost='1369', TotalPHPScripts='0'
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The PHP Script Server is Not Required
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The Maximum SNMP OID Get Size is 32
12/10/2012 04:57:28 PM - SPINE: Poller[0] Version 0.8.8a starting
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: MySQL is Thread Safe!
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: Spine is running asroot.
12/10/2012 04:57:28 PM - SPINE: Poller[0] SPINE: Initializing Net-SNMP API
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: Issues with SNMP Header Version information, assuming old version of Net-SNMP.
12/10/2012 04:57:28 PM - SPINE: Poller[0] SPINE: Initializing PHP Script Server(s)
12/10/2012 04:57:28 PM - SPINE: Poller[0] NOTE: Spine will support multithread device polling.
12/10/2012 04:57:28 PM - SPINE: Poller[0] NOTE: Spine is behaving in a 0.8.7g+ manner
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: Initial Value of Active Threads is 0
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 1
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polling of Host
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: Valid Thread to be Created
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: In Poller, About to Start Polling of Host
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 2
12/10/2012 04:57:28 PM - SPINE: Poller[0] Host[1369] DEBUG: Entering SNMP Ping
12/10/2012 04:57:28 PM - SPINE: Poller[0] Host[0] TH[1] Total Time: 0.0016 Seconds
12/10/2012 04:57:28 PM - SPINE: Poller[0] Host[0] TH[1] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 1
12/10/2012 04:57:28 PM - SPINE: Poller[0] Host[1369] SNMP Result: Host responded to SNMP
12/10/2012 04:57:28 PM - SPINE: Poller[0] Host[1369] TH[1] RECACHE: Processing 1 items in the auto reindex cache for '172.21.231.1'
12/10/2012 04:57:28 PM - SPINE: Poller[0] Host[1369] TH[1] Recache DataQuery[1] OID: .1.3.6.1.2.1.1.3.0, output: 2137204600
12/10/2012 04:57:28 PM - SPINE: Poller[0] Host[1369] TH[1] NOTE: There are '71' Polling Items for this Host
12/10/2012 04:57:28 PM - SPINE: Poller[0] Host[1369] TH[1] DS[310557] WARNING: SNMP timeout detected [2000 ms], ignoring host '172.21.231.1'
12/10/2012 04:57:28 PM - SPINE: Poller[0] Host[1369] TH[1] DS[310557] SNMP: v1: 172.21.231.1, dsname: 5min_cpu, oid: .1.3.6.1.4.1.9.2.1.58.0, value: U
12/10/2012 04:57:28 PM - SPINE: Poller[0] Host[1369] TH[1] DS[347591] WARNING: SNMP timeout detected [2000 ms], ignoring host '172.21.231.1'
12/10/2012 04:57:28 PM - SPINE: Poller[0] Host[1369] TH[1] DS[347591] SNMP: v1: 172.21.231.1, dsname: traffic_in, oid: .1.3.6.1.2.1.2.2.1.10.1, value: U
12/10/2012 04:57:28 PM - SPINE: Poller[0] Host[1369] TH[1] Total Time: 0.092 Seconds
12/10/2012 04:57:28 PM - SPINE: Poller[0] Host[1369] TH[1] DEBUG: HOST COMPLETE: About to Exit Host Polling Thread Function
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: The Value of Active Threads is 0
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: Thread Cleanup Complete
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: PHP Script Server Pipes Closed
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: Allocated Variable Memory Freed
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: MYSQL Free & Close Completed
12/10/2012 04:57:28 PM - SPINE: Poller[0] DEBUG: Net-SNMP Close Completed
12/10/2012 04:57:28 PM - SPINE: Poller[0] Time: 0.1618 s, Threads: 20, Hosts: 2
Cacti Info:
Code: Select all
General Information
Date Mon, 10 Dec 2012 16:48:41 +0100
Cacti Version 0.8.8a
Cacti OS unix
SNMP Version NET-SNMP version: 5.4.3
RRDTool Version RRDTool 1.4.x
Hosts 1559
Graphs 347891
Data Sources Script/Command: 5
SNMP: 2288
SNMP Query: 346759
Total: 349052
Poller Information
Interval 300
Type SPINE 0.8.8a Copyright 2002-2012 by The Cacti Group
Items Action[0]: 835976
Action[1]: 5
Total: 835981
Concurrent Processes 1
Max Threads 20
PHP Servers 1
Script Timeout 10
Max OID 32
Last Run Statistics Time:212.2792 Method:spine Processes:1 Threads:20 Hosts:1560 HostsPerProcess:1560 DataSources:835981 RRDsProcessed:0
PHP Information
PHP Version 5.4.4-10
PHP OS Linux
PHP uname Linux srv01 3.2.0-3-amd64 #1 SMP Mon Jul 23 02:45:17 UTC 2012 x86_64
PHP SNMP Installed
max_execution_time 30
memory_limit 1024M
Can anyone give me a hint, where the error might be?
Thanks!
Dtr