SNMP timeout errors with SPINE

[thardie]

I've done quite a bit of digging into this. I'm 99% sure this is a SPINE/netsnmp bug. I see this show up on my network for at least 1 host every interval. I've done sniffs at the time, and every single get request has a response in the sniff. I'd ordinarility post the sniffs, but they have our production SNMP community string in them. I've done several tests, and every time poller complains about a timeout, the sniff shows every get has a response.

FYI, this is the error I'm talking about (I'm running 216 hosts, 1 process, 10 threads per process, 10 requests max):


05/28/2010 09:45:15 PM - SPINE: Poller[0] Host[134] DS[2874] WARNING: SNMP timeout detected [500 ms], ignoring host '208.93.137.20'
05/28/2010 09:45:15 PM - SPINE: Poller[0] Host[134] DS[2873] WARNING: SNMP timeout detected [500 ms], ignoring host '208.93.137.20'
05/28/2010 09:45:15 PM - SPINE: Poller[0] Host[134] DS[2872] WARNING: SNMP timeout detected [500 ms], ignoring host '208.93.137.20'
05/28/2010 09:45:15 PM - SPINE: Poller[0] Host[134] DS[2871] WARNING: SNMP timeout detected [500 ms], ignoring host '208.93.137.20'
05/28/2010 09:45:15 PM - SPINE: Poller[0] Host[134] DS[2870] WARNING: SNMP timeout detected [500 ms], ignoring host '208.93.137.20'
05/28/2010 09:45:15 PM - SPINE: Poller[0] Host[134] DS[2869] WARNING: SNMP timeout detected [500 ms], ignoring host '208.93.137.20'
05/28/2010 09:45:15 PM - SPINE: Poller[0] Host[134] DS[2868] WARNING: SNMP timeout detected [500 ms], ignoring host '208.93.137.20'
05/28/2010 09:45:15 PM - SPINE: Poller[0] Host[134] DS[2094] WARNING: SNMP timeout detected [500 ms], ignoring host '208.93.137.20'
05/28/2010 09:45:15 PM - SPINE: Poller[0] Host[134] DS[2094] WARNING: SNMP timeout detected [500 ms], ignoring host '208.93.137.20'
05/28/2010 09:45:15 PM - SPINE: Poller[0] Host[134] DS[2093] WARNING: SNMP timeout detected [500 ms], ignoring host '208.93.137.20'

I'm a C/C++ developer, and will be happy to assist in debugging this with a developer.

[skol]

Same thing, although I've got 10,000 devices. Seems to not affect the graphing of cacti, just this error might be spurious?

[TheWitness]

There are multiple reasons for this:

1) Device snmp community is wrong
2) SNMP timeout too low
3) Device Max OID's are too high causing longer time to gather data
4) SNMP Ping does not work correctly for the device.

Remediation steps:

1) Increase SNMP Timeout. For busy devices, I have had to go as high as 4 seconds.
2) Reduce Max OID's (remember if it's a WAN device aka high latency, this will slow polling).

TheWitness

[skol]

Remediation steps:

1) Increase SNMP Timeout. For busy devices, I have had to go as high as 4 seconds.
2) Reduce Max OID's (remember if it's a WAN device aka high latency, this will slow polling).

These settings have been in use before the errors started happening (both global setting and host setting) :

- SNMP timeout = 2000
- Max OID's = 1

Spine still shows :

Code: Select all

09/27/2010 09:36:56 AM - SPINE: Poller[0] Host[3370] DS[71096] WARNING: SNMP timeout detected [500 ms], ignoring host 'hostname'

Graphs continue to graph, however.

[TheWitness]

That DS is likely wonkers.

TheWitness

[skol]

As of today the errors stopped, without any human interaction. /shrug