Cacti snmp Version 3 issues

[lostinthedessert]

This is a brand-new Cacti install trying to make use of SNMP v3. At present there are 2 targets: localhost and a remote Linux
machine. I am unabel to use snmp v3 to retrieve data from either localhost or the remote system. The environment looks like:

CentOS release 6.4 (Final)

Server: Apache/2.2.15 (CentOS)

Cacti Version 0.8.8b
Cacti Variables
Operating System: unix
PHP SNMP Support: yes

using poller, no spine

mysql Ver 14.14 Distrib 5.1.69, for redhat-linux-gnu (x86_64) using readline 5.1

PHP 5.3.3 (cli) (built: Jul 12 2013 20:35:47)

rrdtool-1.3.8-6.el6.x86_64

NET-SNMP version: 5.5


==== on Cacti web console – add target machine as 'Local Linux MAchine';
SNMP Options Version 3, Auth Protocol MD5, Privacy Protocol AES;
local SNMP username, password and passphrase; defaults for remaining SNMPoptions.



==== under section 'Associated Data Queries' fo ritem "SNMP - Interface Statistics' click on 'Verbose Query';

in the 'Data Query Debug Information' these lines appear:
Data Query Debug Information

+ Running data query [1].
+ Found type = '3' [SNMP Query].
+ Found data query XML file at '/usr/share/cacti/resource/snmp_queries/interface.xml'
+ XML file parsed ok.
+ Executing SNMP get for num of indexes @ '.1.3.6.1.2.1.2.1.0' Index Count:
+ Executing SNMP walk for list of indexes @ '.1.3.6.1.2.1.2.2.1.1' Index Count: 0
+ No SNMP data returned



==== tcpdump on the Cacti server does show snmp traffic out and in when poller runs:

[root@cap_monitor log]# tcpdump -nnX -c 20 host 10.4.11.50
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:30:02.021972 IP 10.3.27.118.40843 > 10.4.11.50.161: F=r U= E= C= GetRequest(14)
0x0000: 4500 005c 0000 4000 4011 ffe2 0a03 1b76 E..\..@.@......v
0x0010: 0a04 0b32 9f8b 00a1 0048 3b08 303e 0201 ...2.....H;.0>..
0x0020: 0330 1102 0401 a9ee 4a02 0300 ffe3 0401 .0......J.......
0x0030: 0402 0103 0410 300e 0400 0201 0002 0100 ......0.........
0x0040: 0400 0400 0400 3014 0400 0400 a00e 0204 ......0.........
0x0050: 42c6 bbb5 0201 0002 0100 3000 B.........0.
14:30:02.023245 IP 10.4.11.50.161 > 10.3.27.118.40843: F= U= E= 0x800x000x1F0x880x800xFE0xD00x3C0x0E0xB90x7A0x370x520x000x000x000x00 C= Report(31) .1.3.6.1.6.3.15.1.1.4.0=1
0x0000: 4500 0090 0000 4000 4011 ffae 0a04 0b32 E.....@.@......2
0x0010: 0a03 1b76 00a1 9f8b 007c df29 3072 0201 ...v.....|.)0r..
0x0020: 0330 1102 0401 a9ee 4a02 0300 ffe3 0401 .0......J.......
0x0030: 0002 0103 0422 3020 0411 8000 1f88 80fe ....."0.........
0x0040: d03c 0eb9 7a37 5200 0000 0002 0112 0202 .<..z7R.........
0x0050: 00ca 0400 0400 0400 3036 0411 8000 1f88 ........06......
0x0060: 80fe d03c 0eb9 7a37 5200 0000 0004 00a8 ...<..z7R.......
0x0070: 1f02 0442 c6bb b502 0100 0201 0030 1130 ...B.........0.0
0x0080: 0f06 0a2b 0601 0603 0f01 0104 0041 0101 ...+.........A..
14:30:02.023449 IP 10.3.27.118.40843 > 10.4.11.50.161: F=apr U=manage-doa-stations [!scoped PDU]fb_c7_cd_03_bd_cf_28_26_1f_a2_2b_60_2f_cb_b9_a5_18_ed_65_be_a4_5c_f5_d1_9a_5b_7d_8c_7d_b9_5c_72_97_2b_a2_7e_ea_3e_e4_9c_3f_20_ea_4c_6a_2f_fe_5f_b9_92_3d_f1_ea
0x0000: 4500 00b8 0000 4000 4011 ff86 0a03 1b76 E.....@.@......v
0x0010: 0a04 0b32 9f8b 00a1 00a4 3b64 3081 9902 ...2......;d0...
0x0020: 0103 3011 0204 01a9 ee49 0203 00ff e304 ..0......I......
0x0030: 0107 0201 0304 4a30 4804 1180 001f 8880 ......J0H.......
0x0040: fed0 3c0e b97a 3752 0000 0000 0201 1202 ..<..z7R........
0x0050: 0200 ca04 146d 616e 6167 652d 646f 612d .....manage-doa-
0x0060: 7374 6174 696f 6e73 2004 0c49 820e b143 stations...I...C
0x0070: 7c0d 7fb9 56eb 4504 08ac 1095 314f 020c |...V.E.....1O..
0x0080: 3704 35fb c7cd 03bd cf28 261f a22b 602f 7.5......(&..+`/
0x0090: cbb9 a518 ed65 bea4 5cf5 d19a 5b7d 8c7d .....e..\...[}.}
0x00a0: b95c 7297 2ba2 7eea 3ee4 9c3f 20ea 4c6a .\r.+.~.>..?..Lj
0x00b0: 2ffe 5fb9 923d f1ea /._..=..
14:30:02.024228 IP 10.4.11.50.161 > 10.3.27.118.40843: F= U=manage-doa-stations E= 0x800x000x1F0x880x800xFE0xD00x3C0x0E0xB90x7A0x370x520x000x000x000x00 C= Report(28) .1.3.6.1.6.3.15.1.1.3.0=1
0x0000: 4500 00a2 0000 4000 4011 ff9c 0a04 0b32 E.....@.@......2
0x0010: 0a03 1b76 00a1 9f8b 008e 9077 3081 8302 ...v.......w0...
0x0020: 0103 3011 0204 01a9 ee49 0203 00ff e304 ..0......I......
0x0030: 0100 0201 0304 3630 3404 1180 001f 8880 ......604.......
0x0040: fed0 3c0e b97a 3752 0000 0000 0201 1202 ..<..z7R........
0x0050: 0200 ca04 146d 616e 6167 652d 646f 612d .....manage-doa-
0x0060: 7374 6174 696f 6e73 2004 0004 0030 3304 stations.....03.
0x0070: 1180 001f 8880 fed0 3c0e b97a 3752 0000 ........<..z7R..
0x0080: 0000 0400 a81c 0201 0002 0100 0201 0030 ...............0
0x0090: 1130 0f06 0a2b 0601 0603 0f01 0103 0041 .0...+.........A
0x00a0: 0101 ..
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
[root@cap_monitor log]#


==== as does the snmpd task on the target: (OPTIONS="-LS0-6d -Lf /var/log/snmpd -p /var/run/snmpd.pid -a -V -d")

Received 64 bytes from UDP: [10.3.27.118]:40843->[10.4.11.50]
0000: 30 3E 02 01 03 30 11 02 04 01 A9 EE 4A 02 03 00 0>...0......J...
0016: FF E3 04 01 04 02 01 03 04 10 30 0E 04 00 02 01 ..........0.....
0032: 00 02 01 00 04 00 04 00 04 00 30 14 04 00 04 00 ..........0.....
0048: A0 0E 02 04 42 C6 BB B5 02 01 00 02 01 00 30 00 ....B.........0.

Received SNMP packet(s) from UDP: [10.3.27.118]:40843->[10.4.11.50]

Sending 116 bytes to UDP: [10.3.27.118]:40843->[10.4.11.50]
0000: 30 72 02 01 03 30 11 02 04 01 A9 EE 4A 02 03 00 0r...0......J...
0016: FF E3 04 01 00 02 01 03 04 22 30 20 04 11 80 00 ........."0 ....
0032: 1F 88 80 FE D0 3C 0E B9 7A 37 52 00 00 00 00 02 .....<..z7R.....
0048: 01 12 02 02 00 CA 04 00 04 00 04 00 30 36 04 11 ............06..
0064: 80 00 1F 88 80 FE D0 3C 0E B9 7A 37 52 00 00 00 .......<..z7R...
0080: 00 04 00 A8 1F 02 04 42 C6 BB B5 02 01 00 02 01 .......B........
0096: 00 30 11 30 0F 06 0A 2B 06 01 06 03 0F 01 01 04 .0.0...+........
0112: 00 41 01 01 .A..


Received 156 bytes from UDP: [10.3.27.118]:40843->[10.4.11.50]
0000: 30 81 99 02 01 03 30 11 02 04 01 A9 EE 49 02 03 0.....0......I..
0016: 00 FF E3 04 01 07 02 01 03 04 4A 30 48 04 11 80 ..........J0H...
0032: 00 1F 88 80 FE D0 3C 0E B9 7A 37 52 00 00 00 00 ......<..z7R....
0048: 02 01 12 02 02 00 CA 04 14 6D 61 6E 61 67 65 2D .........manage-
0064: 64 6F 61 2D 73 74 61 74 69 6F 6E 73 20 04 0C 49 doa-stations ..I
0080: 82 0E B1 43 7C 0D 7F B9 56 EB 45 04 08 AC 10 95 ...C|...V.E.....
0096: 31 4F 02 0C 37 04 35 FB C7 CD 03 BD CF 28 26 1F 1O..7.5......(&.
0112: A2 2B 60 2F CB B9 A5 18 ED 65 BE A4 5C F5 D1 9A .+`/.....e..\...
0128: 5B 7D 8C 7D B9 5C 72 97 2B A2 7E EA 3E E4 9C 3F [}.}.\r.+.~.>..?
0144: 20 EA 4C 6A 2F FE 5F B9 92 3D F1 EA .Lj/._..=..

Received SNMP packet(s) from UDP: [10.3.27.118]:40843->[10.4.11.50]

Sending 134 bytes to UDP: [10.3.27.118]:40843->[10.4.11.50]
0000: 30 81 83 02 01 03 30 11 02 04 01 A9 EE 49 02 03 0.....0......I..
0016: 00 FF E3 04 01 00 02 01 03 04 36 30 34 04 11 80 ..........604...
0032: 00 1F 88 80 FE D0 3C 0E B9 7A 37 52 00 00 00 00 ......<..z7R....
0048: 02 01 12 02 02 00 CA 04 14 6D 61 6E 61 67 65 2D .........manage-
0064: 64 6F 61 2D 73 74 61 74 69 6F 6E 73 20 04 00 04 doa-stations ...
0080: 00 30 33 04 11 80 00 1F 88 80 FE D0 3C 0E B9 7A .03.........<..z
0096: 37 52 00 00 00 00 04 00 A8 1C 02 01 00 02 01 00 7R..............
0112: 02 01 00 30 11 30 0F 06 0A 2B 06 01 06 03 0F 01 ...0.0...+......
0128: 01 03 00 41 01 01 ...A..

============ running snmpget on the first mib named in the 'Data Query Debug Information' section -

[root@cap_monitor cacti]# snmpget -u manage-doa-stations -l priv -x AES -A 'NUtestSNMPv3install' -X 'NUtestSNMPv3install' 10.4.11.50 '.1.3.6.1.2.1.2.1.0'
IF-MIB::ifNumber.0 = INTEGER: 4
[root@cap_monitor cacti]#

=== and checking tcpdump seems to indicate only the first snmpget executes, and snmpwalk does not:

[root@cap_monitor log]# tcpdump -nnX -c 20 host 10.4.11.50
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:21:50.237852 IP 10.3.27.118.46703 > 10.4.11.50.161: F=r U= E= C= GetRequest(14)
0x0000: 4500 005c 0000 4000 4011 ffe2 0a03 1b76 E..\..@.@......v
0x0010: 0a04 0b32 b66f 00a1 0048 3b08 303e 0201 ...2.o...H;.0>..
0x0020: 0330 1102 0472 f411 7302 0300 ffe3 0401 .0...r..s.......
0x0030: 0402 0103 0410 300e 0400 0201 0002 0100 ......0.........
0x0040: 0400 0400 0400 3014 0400 0400 a00e 0204 ......0.........
0x0050: 46b7 def5 0201 0002 0100 3000 F.........0.
16:21:50.238936 IP 10.4.11.50.161 > 10.3.27.118.46703: F= U= E= 0x800x000x1F0x880x800xFE0xD00x3C0x0E0xB90x7A0x370x520x000x000x000x00 C= Report(31) .1.3.6.1.6.3.15.1.1.4.0=3
0x0000: 4500 0090 0000 4000 4011 ffae 0a04 0b32 E.....@.@......2
0x0010: 0a03 1b76 00a1 b66f 007c 22b1 3072 0201 ...v...o.|".0r..
0x0020: 0330 1102 0472 f411 7302 0300 ffe3 0401 .0...r..s.......
0x0030: 0002 0103 0422 3020 0411 8000 1f88 80fe ....."0.........
0x0040: d03c 0eb9 7a37 5200 0000 0002 0118 0202 .<..z7R.........
0x0050: 019b 0400 0400 0400 3036 0411 8000 1f88 ........06......
0x0060: 80fe d03c 0eb9 7a37 5200 0000 0004 00a8 ...<..z7R.......
0x0070: 1f02 0446 b7de f502 0100 0201 0030 1130 ...F.........0.0
0x0080: 0f06 0a2b 0601 0603 0f01 0104 0041 0103 ...+.........A..
16:21:50.239119 IP 10.3.27.118.46703 > 10.4.11.50.161: F=apr U=manage-doa-stations [!scoped PDU]42_fe_10_58_9f_c2_4d_66_1a_30_f9_dc_33_62_83_e7_ea_1d_60_22_43_03_14_75_b3_08_05_8a_4b_81_bf_d1_66_85_8e_19_13_75_00_69_cb_99_ad_ea_da_32_3e_a8_5f_a3_a4_23_2e
0x0000: 4500 00b7 0000 4000 4011 ff87 0a03 1b76 E.....@.@......v
0x0010: 0a04 0b32 b66f 00a1 00a3 3b63 3081 9802 ...2.o....;c0...
0x0020: 0103 3011 0204 72f4 1172 0203 00ff e304 ..0...r..r......
0x0030: 0107 0201 0304 4930 4704 1180 001f 8880 ......I0G.......
0x0040: fed0 3c0e b97a 3752 0000 0000 0201 1802 ..<..z7R........
0x0050: 0201 9b04 136d 616e 6167 652d 646f 612d .....manage-doa-
0x0060: 7374 6174 696f 6e73 040c d327 03ac 5d5e stations...'..]^
0x0070: 75d8 3ebd caa4 0408 1f55 1fa1 2595 d615 u.>......U..%...
0x0080: 0435 42fe 1058 9fc2 4d66 1a30 f9dc 3362 .5B..X..Mf.0..3b
0x0090: 83e7 ea1d 6022 4303 1475 b308 058a 4b81 ....`"C..u....K.
0x00a0: bfd1 6685 8e19 1375 0069 cb99 adea da32 ..f....u.i.....2
0x00b0: 3ea8 5fa3 a423 2e >._..#.
16:21:50.239959 IP 10.4.11.50.161 > 10.3.27.118.46703: F=ap U=manage-doa-stations [!scoped PDU]5d_52_de_e6_60_d2_8a_e3_85_69_df_08_77_f4_37_6f_6c_a2_a4_e2_41_e1_ed_bf_87_1c_54_77_76_eb_78_98_15_c5_43_25_c8_05_29_db_4f_4c_e7_4d_8e_97_98_0a_dd_ff_09_74_29_3b
0x0000: 4500 00b8 0000 4000 4011 ff86 0a04 0b32 E.....@.@......2
0x0010: 0a03 1b76 00a1 b66f 00a4 eb9f 3081 9902 ...v...o....0...
0x0020: 0103 3011 0204 72f4 1172 0203 00ff e304 ..0...r..r......
0x0030: 0103 0201 0304 4930 4704 1180 001f 8880 ......I0G.......
0x0040: fed0 3c0e b97a 3752 0000 0000 0201 1802 ..<..z7R........
0x0050: 0201 9b04 136d 616e 6167 652d 646f 612d .....manage-doa-
0x0060: 7374 6174 696f 6e73 040c abaa c7f1 3de1 stations......=.
0x0070: 713c 916e d843 0408 4376 4509 65d6 718d q<.n.C..CvE.e.q.
0x0080: 0436 5d52 dee6 60d2 8ae3 8569 df08 77f4 .6]R..`....i..w.
0x0090: 376f 6ca2 a4e2 41e1 edbf 871c 5477 76eb 7ol...A.....Twv.
0x00a0: 7898 15c5 4325 c805 29db 4f4c e74d 8e97 x...C%..).OL.M..
0x00b0: 980a ddff 0974 293b .....t);
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel

==== cacti.log certain supports this idea:

10/15/2013 04:25:01 PM - CMDPHP: Poller[0] WARNING: SNMP Get Timeout for Host:'10.4.11.50', and OID:'.1.3.6.1.2.1.1.3.0'
10/15/2013 04:25:01 PM - SYSTEM STATS: Time:0.2453 Method:cmd.php Processes:1 Threads:N/A Hosts:3 HostsPerProcess:3 DataSources:23 RRDsProcessed:6


==== on the Cacti server command line the snmpget and snmpwalk commands return data:

[root@cap_monitor cacti]# snmpget -u manage-doa-stations -l priv -x AES -A 'NUtestSNMPv3install' -X 'NUtestSNMPv3install' 10.4.11.50 '.1.3.6.1.2.1.2.1.0'
IF-MIB::ifNumber.0 = INTEGER: 4
[root@cap_monitor cacti]# snmpwalk -u manage-doa-stations -l priv -x AES -A 'NUtestSNMPv3install' -X 'NUtestSNMPv3install' 10.4.11.50 '.1.3.6.1.2.1.2.2.1.1'
IF-MIB::ifIndex.1 = INTEGER: 1
IF-MIB::ifIndex.2 = INTEGER: 2
IF-MIB::ifIndex.3 = INTEGER: 3
IF-MIB::ifIndex.4 = INTEGER: 4


==== but somehow Cacti seems unable to receive --

Does anyone have an idea where I might look next?
Is there some trace I might use to pin down what I have wrong in the configuration?

Thanks for any insight -