Connectivity problem with Cisco 4948

matt85 · Post by **matt85** » Wed Dec 02, 2009 12:10 pm

Hi Cacti experts,

I'm new to Cacti and the forum. Please excuse if the issue has been discussed before, but I wasn't able to find any answers so far.

I'd like to use Cacti to monitor a number of Cisco 4948 Switch interfaces. The Cacti installation has been up and running for a while and is being used successfully to monitor several other network components.
Now I've added the Cisco Switches and keep encountering the same problem: Some of them are identified as "down" or at least intermittently have no SNMP connection. After adding a device, the SNMP info is displayed correctly and the TCP ping is successful. When I list the associated data queries, I also get a correct result, which allows me to create the graphs for the interfaces I need. So far, so good.
However, when listing the devices, some of them are consistently reported as "down" and their graphs are not created. When I click on a device to check the details, the SNMP info and ping results are fine. Some of the switches I added work just fine - they are "up" and graphs are drawn. I can see no pattern as to which devices work and which don't. They have identical configurations (as far as SNMP is concerned) and the same IOS version. When deleting a group of devices and adding them again, other switches than before might be "up" or "down". When I change downed device detection to "Ping or SNMP", the device is detected as "up" and a graph created. However, it only contains "NaN" values.
I worked my way through gandalf's debugging tutorial (http://docs.cacti.net/manual:087:4_help.2_debugging). This is what I found:

- Hosts are detected as "down" (or get SNMP timeouts if I change down detection setting)
- Scripts / Permissions are OK, they work for some devices
- RRD files are either not created or contain "NaN"s, depending on down detection setting
- Minimum & Maximum settings are OK, results should be within bounds
- No problems found in the rrdtool graph statement
- When using snmpwalk from my Cacti-Server, I can reliably retrieve all values from all devices
- Tried setting Maximum SNMP OID's Per SNMP Get Request to 1 -> no difference

My system:

Cacti Version: 0.8.7e
Cacti OS: unix
SNMP Version: NET-SNMP version: 5.3.0.1
RRDTool Version: RRDTool 1.2.x
Poller: spine

I would really appreciate any pointers as to where I should be looking for the problem. Thank you very much!

Cheers,

Matt

Post by **gandalf** » Wed Dec 02, 2009 5:15 pm

Please try changing SNMP timeout and/or Max OID for those devices. Using MAX OID = 1 may solve problems
R.

matt85 · Post by **matt85** » Thu Dec 03, 2009 8:25 am

Thank you very much for the quick reply!
I tried setting Max OID to 1 and changed the timeout to 5 seconds. Neither seems to make a difference, though

A timeout problem seems to make sense, though. Still, I can't quite put my finger on it.

Any other ideas?

Post by **gandalf** » Thu Dec 03, 2009 2:42 pm

A tcpdump/wireshark trace may shed some light on it
R.

matt85 · Post by **matt85** » Mon Dec 14, 2009 9:59 am

Sorry this took a while - some other projects came up and I had to get root access for the Cacti-Server.

Anyways, here's what tcpdump is telling me.

This is a poll cycle for a switch that is working:

Code: Select all

user@cacti-srv:~> sudo /usr/sbin/tcpdump -evv host 10.18.1.9
tcpdump: listening on eth3, link-type EN10MB (Ethernet), capture size 96 bytes
15:35:03.520928 [*cacti-srv-mac*] (oui Unknown) > [*switch_1-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl  64, id 24820, offset 0, flags [DF], proto: TCP (6), length: 60) cacti-srv.foo.bar.52891 > 10.18.1.9.ssh: S, cksum 0x27cb (correct), 1775188063:1775188063(0) win 5840 <mss 1460,sackOK,timestamp 1386764459 0,nop,wscale 2>
15:35:03.533330 [*switch_1-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 249, id 34021, offset 0, flags [none], proto: TCP (6), length: 44) 10.18.1.9.ssh > cacti-srv.foo.bar.52891: S, cksum 0xe947 (correct), 3446503109:3446503109(0) ack 1775188064 win 4128 <mss 536>
15:35:03.533369 [*cacti-srv-mac*] (oui Unknown) > [*switch_1-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl  64, id 24821, offset 0, flags [DF], proto: TCP (6), length: 40) cacti-srv.foo.bar.52891 > 10.18.1.9.ssh: ., cksum 0xf6b8 (correct), 1:1(0) ack 1 win 5840
15:35:03.533421 [*cacti-srv-mac*] (oui Unknown) > [*switch_1-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl  64, id 24822, offset 0, flags [DF], proto: TCP (6), length: 40) cacti-srv.foo.bar.52891 > 10.18.1.9.ssh: F, cksum 0xf6b7 (correct), 1:1(0) ack 1 win 5840
15:35:03.533485 [*cacti-srv-mac*] (oui Unknown) > [*switch_1-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 106: (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: UDP (17), length: 92) cacti-srv.foo.bar.46636 > 10.18.1.9.snmp:  { SNMPv3 { F=r } { USM B=0 T=0 U= } { ScopedPDU E=  C= { GetRequest(14) [|snmp] } } }
15:35:03.545484 [*switch_1-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 249, id 34022, offset 0, flags [none], proto: TCP (6), length: 40) 10.18.1.9.ssh > cacti-srv.foo.bar.52891: ., cksum 0xfd67 (correct), 1:1(0) ack 2 win 4128
15:35:03.547429 [*switch_1-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 150: (tos 0x0, ttl 249, id 5019, offset 0, flags [none], proto: UDP (17), length: 136) 10.18.1.9.snmp > cacti-srv.foo.bar.46636:  { SNMPv3 { F= } { USM B=5 T=30248391 U= [|snmp]} { ScopedPDU [|snmp]} }
15:35:03.547551 [*cacti-srv-mac*] (oui Unknown) > [*switch_1-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 176: (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: UDP (17), length: 162) cacti-srv.foo.bar.46636 > 10.18.1.9.snmp:  { SNMPv3 { F=apr } { USM B=2 T=31798124 U=Ã¡Â·! [|snmp]} { ScopedPDU [|snmp]} }
15:35:03.560989 [*switch_1-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 168: (tos 0x0, ttl 249, id 5020, offset 0, flags [none], proto: UDP (17), length: 154) 10.18.1.9.snmp > cacti-srv.foo.bar.46636:  { SNMPv3 { F=a } { USM B=5 T=30248391 U=CaÃ¡Â·! [|snmp]} { ScopedPDU [|snmp]} }
15:35:03.561074 [*cacti-srv-mac*] (oui Unknown) > [*switch_1-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 176: (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: UDP (17), length: 162) cacti-srv.foo.bar.46636 > 10.18.1.9.snmp:  { SNMPv3 { F=apr } { USM B=5 T=30248391 U=Ã¡Â·! [|snmp]} { ScopedPDU [|snmp]} }
15:35:03.577887 [*switch_1-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 442: (tos 0x0, ttl 249, id 5021, offset 0, flags [none], proto: UDP (17), length: 428) 10.18.1.9.snmp > cacti-srv.foo.bar.46636:  { SNMPv3 { F=ap } { USM B=5 T=30248391 U=Ã¡Â·! [|snmp]} { ScopedPDU [|snmp]} }
15:35:03.592863 [*cacti-srv-mac*] (oui Unknown) > [*switch_1-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 106: (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: UDP (17), length: 92) cacti-srv.foo.bar.46637 > 10.18.1.9.snmp:  { SNMPv3 { F=r } { USM B=0 T=0 U= } { ScopedPDU E=  C= { GetRequest(14) [|snmp] } } }
15:35:03.605979 [*switch_1-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 150: (tos 0x0, ttl 249, id 5022, offset 0, flags [none], proto: UDP (17), length: 136) 10.18.1.9.snmp > cacti-srv.foo.bar.46637:  { SNMPv3 { F= } { USM B=5 T=30248391 U= [|snmp]} { ScopedPDU [|snmp]} }
15:35:03.606093 [*cacti-srv-mac*] (oui Unknown) > [*switch_1-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 184: (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: UDP (17), length: 170) cacti-srv.foo.bar.46637 > 10.18.1.9.snmp:  { SNMPv3 { F=apr } { USM B=5 T=30248391 U=Ã¡Â·! [|snmp]} { ScopedPDU [|snmp]} }
15:35:03.643620 [*switch_1-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 60: (tos 0xc0, ttl 249, id 34023, offset 0, flags [none], proto: TCP (6), length: 40) 10.18.1.9.ssh > cacti-srv.foo.bar.52891: FP, cksum 0xfd5e (correct), 1:1(0) ack 2 win 4128
15:35:03.643643 [*cacti-srv-mac*] (oui Unknown) > [*switch_1-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 54: (tos 0xc0, ttl  64, id 0, offset 0, flags [DF], proto: TCP (6), length: 40) cacti-srv.foo.bar.52891 > 10.18.1.9.ssh: ., cksum 0xf6b6 (correct), 2:2(0) ack 2 win 5840
15:35:03.703207 [*switch_1-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 183: (tos 0x0, ttl 249, id 5023, offset 0, flags [none], proto: UDP (17), length: 169) 10.18.1.9.snmp > cacti-srv.foo.bar.46637:  { SNMPv3 { F=ap } { USM B=5 T=30248391 U=CÃ¡Â·! [|snmp]} { ScopedPDU [|snmp]} }
15:35:03.703420 [*cacti-srv-mac*] (oui Unknown) > [*switch_1-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 184: (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: UDP (17), length: 170) cacti-srv.foo.bar.46637 > 10.18.1.9.snmp:  { SNMPv3 { F=apr } { USM B=5 T=30248391 U=Ã¡Â·! [|snmp]} { ScopedPDU [|snmp]} }
15:35:03.720509 [*switch_1-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 183: (tos 0x0, ttl 249, id 5024, offset 0, flags [none], proto: UDP (17), length: 169) 10.18.1.9.snmp > cacti-srv.foo.bar.46637:  { SNMPv3 { F=ap } { USM B=5 T=30248391 U=CÃ¡Â·! [|snmp]} { ScopedPDU [|snmp]} }
15:35:03.720658 [*cacti-srv-mac*] (oui Unknown) > [*switch_1-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 184: (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: UDP (17), length: 170) cacti-srv.foo.bar.46637 > 10.18.1.9.snmp:  { SNMPv3 { F=apr } { USM B=5 T=30248391 U=Ã¡Â·! [|snmp]} { ScopedPDU [|snmp]} }
15:35:03.739121 [*switch_1-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 183: (tos 0x0, ttl 249, id 5025, offset 0, flags [none], proto: UDP (17), length: 169) 10.18.1.9.snmp > cacti-srv.foo.bar.46637:  { SNMPv3 { F=ap } { USM B=5 T=30248391 U=CÃ¡Â·! [|snmp]} { ScopedPDU [|snmp]} }
15:35:03.739122 [*cacti-srv-mac*] (oui Unknown) > [*switch_1-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 184: (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: UDP (17), length: 170) cacti-srv.foo.bar.46637 > 10.18.1.9.snmp:  { SNMPv3 { F=apr } { USM B=5 T=30248391 U=Ã¡Â·! [|snmp]} { ScopedPDU [|snmp]} }
15:35:03.756250 [*switch_1-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 183: (tos 0x0, ttl 249, id 5026, offset 0, flags [none], proto: UDP (17), length: 169) 10.18.1.9.snmp > cacti-srv.foo.bar.46637:  { SNMPv3 { F=ap } { USM B=5 T=30248391 U=CÃ¡Â·! [|snmp]} { ScopedPDU [|snmp]} }

And a cycle for one of those that isn't working:

Code: Select all

user@cacti-srv:~> sudo /usr/sbin/tcpdump -evv host 10.18.1.10
tcpdump: listening on eth3, link-type EN10MB (Ethernet), capture size 96 bytes
15:40:45.699513 [*cacti-srv-mac*] (oui Unknown) > [*switch_2-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl  64, id 5507, offset 0, flags [DF], proto: TCP (6), length: 60) cacti-srv.foo.bar.35135 > 10.18.1.10.ssh: S, cksum 0x8f0a (correct), 2128130893:2128130893(0) win 5840 <mss 1460,sackOK,timestamp 1386849998 0,nop,wscale 2>
15:40:45.712028 [*switch_2-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 249, id 20855, offset 0, flags [none], proto: TCP (6), length: 44) 10.18.1.10.ssh > cacti-srv.foo.bar.35135: S, cksum 0xa53b (correct), 411772184:411772184(0) ack 2128130894 win 4128 <mss 536>
15:40:45.712043 [*cacti-srv-mac*] (oui Unknown) > [*switch_2-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl  64, id 5508, offset 0, flags [DF], proto: TCP (6), length: 40) cacti-srv.foo.bar.35135 > 10.18.1.10.ssh: ., cksum 0xb2ac (correct), 1:1(0) ack 1 win 5840
15:40:45.712187 [*cacti-srv-mac*] (oui Unknown) > [*switch_2-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl  64, id 5509, offset 0, flags [DF], proto: TCP (6), length: 40) cacti-srv.foo.bar.35135 > 10.18.1.10.ssh: F, cksum 0xb2ab (correct), 1:1(0) ack 1 win 5840
15:40:45.712211 [*cacti-srv-mac*] (oui Unknown) > [*switch_2-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 106: (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: UDP (17), length: 92) cacti-srv.foo.bar.46673 > 10.18.1.10.snmp:  { SNMPv3 { F=r } { USM B=0 T=0 U= } { ScopedPDU E=  C= { GetRequest(14) [|snmp] } } }
15:40:45.725180 [*switch_2-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 249, id 20856, offset 0, flags [none], proto: TCP (6), length: 40) 10.18.1.10.ssh > cacti-srv.foo.bar.35135: ., cksum 0xb95b (correct), 1:1(0) ack 2 win 4128
15:40:45.727054 [*switch_2-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 150: (tos 0x0, ttl 249, id 46667, offset 0, flags [none], proto: UDP (17), length: 136) 10.18.1.10.snmp > cacti-srv.foo.bar.46673:  { SNMPv3 { F= } { USM B=4 T=30248733 U= [|snmp]} { ScopedPDU [|snmp]} }
15:40:45.727266 [*cacti-srv-mac*] (oui Unknown) > [*switch_2-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 175: (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: UDP (17), length: 161) cacti-srv.foo.bar.46673 > 10.18.1.10.snmp:  { SNMPv3 { F=apr } { USM B=10 T=4063649 U=CÃ¢Â·! [|snmp]} { ScopedPDU [|snmp]} }
15:40:45.739974 [*switch_2-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 168: (tos 0x0, ttl 249, id 46668, offset 0, flags [none], proto: UDP (17), length: 154) 10.18.1.10.snmp > cacti-srv.foo.bar.46673:  { SNMPv3 { F=a } { USM B=4 T=30248733 U=CaÃ¢Â·! [|snmp]} { ScopedPDU [|snmp]} }
15:40:45.823435 [*switch_2-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 60: (tos 0xc0, ttl 249, id 20857, offset 0, flags [none], proto: TCP (6), length: 40) 10.18.1.10.ssh > cacti-srv.foo.bar.35135: FP, cksum 0xb952 (correct), 1:1(0) ack 2 win 4128
15:40:45.823448 [*cacti-srv-mac*] (oui Unknown) > [*switch_2-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 54: (tos 0xc0, ttl  64, id 0, offset 0, flags [DF], proto: TCP (6), length: 40) cacti-srv.foo.bar.35135 > 10.18.1.10.ssh: ., cksum 0xb2aa (correct), 2:2(0) ack 2 win 5840
15:40:50.729644 [*cacti-srv-mac*] (oui Unknown) > [*switch_2-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 175: (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: UDP (17), length: 161) cacti-srv.foo.bar.46673 > 10.18.1.10.snmp:  { SNMPv3 { F=apr } { USM B=10 T=4063654 U=CÃ¢Â·! [|snmp]} { ScopedPDU [|snmp]} }
15:40:50.741503 [*switch_2-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 168: (tos 0x0, ttl 249, id 46669, offset 0, flags [none], proto: UDP (17), length: 154) 10.18.1.10.snmp > cacti-srv.foo.bar.46673:  { SNMPv3 { F=a } { USM B=4 T=30248738 U=CaÃ¢Â·! [|snmp]} { ScopedPDU [|snmp]} }
15:40:55.737543 [*cacti-srv-mac*] (oui Unknown) > [*switch_2-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 175: (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: UDP (17), length: 161) cacti-srv.foo.bar.46673 > 10.18.1.10.snmp:  { SNMPv3 { F=apr } { USM B=10 T=4063659 U=CÃ¢Â·! [|snmp]} { ScopedPDU [|snmp]} }
15:40:55.749983 [*switch_2-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 168: (tos 0x0, ttl 249, id 46670, offset 0, flags [none], proto: UDP (17), length: 154) 10.18.1.10.snmp > cacti-srv.foo.bar.46673:  { SNMPv3 { F=a } { USM B=4 T=30248743 U=CaÃ¢Â·! [|snmp]} { ScopedPDU [|snmp]} }
15:41:00.741563 [*cacti-srv-mac*] (oui Unknown) > [*switch_2-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 175: (tos 0x0, ttl  64, id 0, offset 0, flags [DF], proto: UDP (17), length: 161) cacti-srv.foo.bar.46673 > 10.18.1.10.snmp:  { SNMPv3 { F=apr } { USM B=10 T=4063664 U=CÃ¢Â·! [|snmp]} { ScopedPDU [|snmp]} }
15:41:00.753566 [*switch_2-mac*] (oui Unknown) > [*cacti-srv-mac*] (oui Unknown), ethertype IPv4 (0x0800), length 168: (tos 0x0, ttl 249, id 46671, offset 0, flags [none], proto: UDP (17), length: 154) 10.18.1.10.snmp > cacti-srv.foo.bar.46673:  { SNMPv3 { F=a } { USM B=4 T=30248748 U=CaÃ¢Â·! [|snmp]} { ScopedPDU [|snmp]} }

Switch_2 does seem to be answering all requests, but maybe not correctly. Any ideas what to look for now?

Thank you very much for your help.

Post by **gandalf** » Mon Dec 14, 2009 2:03 pm

That's SNMP V3, is it?
There have been rumours around that. So please, which kind of parameters are you using? AuthPriv? AuthNoPriv? ...
R.

matt85 · Post by **matt85** » Tue Dec 15, 2009 4:09 am

Yeah, we're using SNMP v3 with AuthPriv...

matt85 · Post by **matt85** » Tue Dec 15, 2009 5:27 am

Now you mentioned it, I decided to try setting up SNMP v2. Graphing seems to work without a problem with SNMP v2. Now I feel kind of stupid for not testing this before.

So the problem is definitely with SNMP v3. Do you have any further info on the "rumors" around v3 problems?
I'm trying to use v3 with AuthPriv, MD5 and DES.

Post by **TheWitness** » Tue Dec 15, 2009 11:54 am

What spine poller are you using. Can you please test the latest SVN and see if that helps with the random connectivity problems.

TheWitness

artmik · Post by **artmik** » Wed Dec 16, 2009 7:06 am

Hi experts,

I have the same problem, as matt85 described. But in contrast to him, I HAVE to use SNMPv3. Could you please inform me about the rootcause of this situation and how I can fix it?

tcpdump output:

IP x.x.x.x.port > y.y.y.y.snmp F=apr U=snmp [|snmp][|snmp]
IP y.y.y.y.snmp > x.x.x.x.port F= U=sn [|snmp][|snmp]

I use the following command:

snmpget -v 3 -l authPriv -u someuser -c somecommunity -a SHA -A somepass1 -x DES -X somepass2 -e someEngineId y.y.y.y:161 system.sysUpTime.0

Regards, artmik

matt85 · Post by **matt85** » Wed Dec 16, 2009 10:54 am

I am using spine 0.8.7e.
I only just discovered the patches at http://cacti.net/spine_download_patches.php
Since I don't have a backup of the Cacti server (and can't easily pull one), I'd like to try the patches before the latest SVN. However, I've run into some trouble updating autoconf. I'll have to resolve that first and then try the patches. I'll keep you posted.

Connectivity problem with Cisco 4948

Connectivity problem with Cisco 4948

Who is online