Problem with SNMP timeouts on Ubuntu 8.04

[pm3kx]

We are having problems with consistent timeouts to the same set of hosts using Cacti spine. We are using Cacti. 0.8.7e with Spine .0.8.7e. Patches for Spine have been compiled in. We are running the following modules, aggregate, autom8, boost, mobile, settings, thold, and update. What happens is we will see snmp timeouts to several of our Cisco Access Points. Its generally the same set of hosts a random seeming selection of a few access points in a few particular buildings. I have checked all the obvious network things (link speed/duplex, etc). I have also tried varying the snmp timeout and max oids per request. None of these suggestions work. The oddest thing about this is that command line snmp stops working as well. Yet you can manually walk the same host on an identical Ubuntu machine without any trouble at the exact same time. It can also be accessed from Solaris machine. I've tried to replicate the behavior without cacti by running snmpwalk 30 times at 30 seconds intervals to one of the hosts having issues. I could not replicate the behavior. I'm using 1 process and 15 threads. Using more than one process seems to cause more issues. Currently we have 917 hosts and 38828 data sources.

Top statistics:
top - 17:02:58 up 4 days, 6:03, 1 user, load average: 0.32, 0.22, 0.11
Tasks: 123 total, 1 running, 122 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.6%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 16627708k total, 2472024k used, 14155684k free, 225424k buffers
Swap: 2936824k total, 0k used, 2936824k free, 1938540k cached

Also, we store the database on a separate server. Our cacti host does polling only. I also was able to replicate the problem on the database host by turning up cacti on it for a short period of time. This was before I installed boost. I am including the debug level log from one polling cycle to help.

Any suggestions would be appreciated.

[Alphadog]

How long does it take for your Poller to run ?
Try inrceasing the snmp timeout value ?

[pm3kx]

The poller on average takes about 40 seconds to run. I've tried increasing the SNMP timeout and decreasing the maximum oids per request to 1. Neither had any effect. I know its something cacti related because I can snmpwalk the same host from an identical machine at the same time the host is timing out from Cacti.

[TheWitness]

What version of SNMP? If v3, you may need to make a few modifications to spine. This may also have something to do with your backend. No certain.

TheWitness

[pm3kx]

We are using SNMP version 2. As for our backend, I'm not sure what you mean. We are using ubuntu 8.04 on a server with dual Intel Xeon 2.66 GHZ processors which are quad core processors with 16 gig of memoy. Are there any known issues with Ubuntu package versions of cacti?

The only possible issue I can think of is that the cacti server is behind a firewall, but that seems unlikely because I can snmpwalk from another machine behind the same firewall at the same time.

Are there any known issues with snmp on Cisco 1130 APs that aren't well publicized?

[TheWitness]

More than likely it's a traffic prioritization issue. Could be QOS or something like this. SNMP is one of the lowest priority traffics. Spine works just fine though. With very busy devices, I have had to set the TO to as much as 4-6 seconds. Could be the device too. You'll have to login and monitor things when the timeouts occur to see.

TheWitness