SNMPv3 polling returning NaN value

[cactaceae]

The Cacti cmd poller is getting NaN value for a specific poll that can be successfully queried using snmpwalk. We are using version 0.8.7g.

Here are some entries from the Cacti log at DEBUG:

12/15/2011 11:11:35 AM - CMDPHP: Poller[0] Host[205] DS[1300] SNMP: v3: 10.xx.xx.xx, dsname: snmp_oid, oid: .1.3.6.1.4.1.8962.2.1.2.1.1.10, output: U
12/15/2011 11:16:40 AM - CMDPHP: Poller[0] Host[205] DS[1300] SNMP: v3: 10.xx.xx.xx, dsname: snmp_oid, oid: .1.3.6.1.4.1.8962.2.1.2.1.1.10, output: U
12/15/2011 11:21:35 AM - CMDPHP: Poller[0] Host[205] DS[1300] SNMP: v3: 10.xx.xx.xx, dsname: snmp_oid, oid: .1.3.6.1.4.1.8962.2.1.2.1.1.10, output: U
12/15/2011 11:26:35 AM - CMDPHP: Poller[0] Host[205] DS[1300] SNMP: v3: 10.xx.xx.xx, dsname: snmp_oid, oid: .1.3.6.1.4.1.8962.2.1.2.1.1.10, output: U
12/15/2011 11:31:36 AM - CMDPHP: Poller[0] Host[205] DS[1300] SNMP: v3: 10.xx.xx.xx, dsname: snmp_oid, oid: .1.3.6.1.4.1.8962.2.1.2.1.1.10, output: U
12/15/2011 11:36:34 AM - CMDPHP: Poller[0] Host[205] DS[1300] SNMP: v3: 10.xx.xx.xx, dsname: snmp_oid, oid: .1.3.6.1.4.1.8962.2.1.2.1.1.10, output: U

Here is results from snmpwalk:

snmpwalk -v3 -u xxxxxx -l authPriv -a MD5 -A xxxxxx -x DES -X xxxxxx -Os 10.xx.xx.xx .1.3.6.1.4.1.8962.2.1.2.1.1.10
iso.3.6.1.4.1.8962.2.1.2.1.1.10.0 = INTEGER: 679016

rrdtool dump shows polling is getting NaN:

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE rrd SYSTEM "http://oss.oetiker.ch/rrdtool/rrdtool.dtd">
<!-- Round Robin Database Dump -->
<rrd>
<version>0003</version>
<step>300</step> <!-- Seconds -->
<lastupdate>1323978392</lastupdate> <!-- 2011-12-15 11:46:32 PST -->

<ds>
<name> snmp_oid </name>
<type> GAUGE </type>
<minimal_heartbeat>600</minimal_heartbeat>
<min>0.0000000000e+00</min>
<max>1.0000000000e+06</max>

<!-- PDP Status -->
<last_ds>U</last_ds>
<value>NaN</value>
<unknown_sec> 92 </unknown_sec>
</ds>

<!-- Round Robin Archives -->
<rra>
<cf>AVERAGE</cf>
<pdp_per_row>1</pdp_per_row> <!-- 300 seconds -->

<params>
<xff>5.0000000000e-01</xff>
</params>
<cdp_prep>
<ds>
<primary_value>NaN</primary_value>
<secondary_value>0.0000000000e+00</secondary_value>
<value>NaN</value>
<unknown_datapoints>0</unknown_datapoints>
</ds>
</cdp_prep>
<database>
<!-- 2011-12-13 18:10:00 PST / 1323828600 --> <row><v>NaN</v></row>
<!-- 2011-12-13 18:15:00 PST / 1323828900 --> <row><v>NaN</v></row>
<!-- 2011-12-13 18:20:00 PST / 1323829200 --> <row><v>NaN</v></row>
<!-- 2011-12-13 18:25:00 PST / 1323829500 --> <row><v>NaN</v></row>
<!-- 2011-12-13 18:30:00 PST / 1323829800 --> <row><v>NaN</v></row>

What additional information can I provide that will assist with troubleshooting this issue?

[gandalf]

Is this happening on a single host or are multiple hosts showing this issue?
R.

[cactaceae]

This is happening on a single host. This single host is the only SNMPv3 device currently being monitored. All the other 50+ SNMPv2 hosts are being monitored and graphed correctly.

[gandalf]

Which exact setup has this host releated to SNMP parameters (please modify credentials before posting)
R.

[cactaceae]

Version 3
SNMP Username (v3) xxxxxx
SNMP Password (v3) xxxxxxxx
SNMP Auth Protocol (v3) MD5 (default)
SNMP Privacy Passphrase (v3) xxxxxxxx
SNMP Privacy Protocol (v3) DES (default)
SNMP Context (v3)
SNMP Port 161
SNMP Timeout 500
Maximum OID's Per Get Request 10

[cactaceae]

any follow up on this please?
thank you

[gandalf]

You're using a non-empty context?
R.

[cactaceae]

the context is empty

[gandalf]

We use a similar setting successfully on several switches. So, IMHO, this is not a general issue. Are you able to provide a wireshark trace of such an SNMP communication between cacti and target? You may pm or email that directly to me to avoid security issues.
BTB: the snmpwalk you've posted above has been run from the cacti system as well?
R.

[cactaceae]

catpure when snmpwalk the value:

16:32:43.523348 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 92)
xx.xx.xx.xx.43383 > xx.xx.xx.xx.161: [bad udp cksum d0d5!] { SNMPv3 { F=r } { USM B=0 T=0 U= } { ScopedPDU E= C= { GetRequest(14) R=1456283550 } } }
16:32:43.556076 IP (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto UDP (17), length 156)
xx.xx.xx.xx.161 > xx.xx.xx.xx.43383: [udp sum ok] { SNMPv3 { F= } { USM B=17 T=7831740 U= } { ScopedPDU E= 0x800x000x1F0x880x040x300x300x3A0x300x360x3A0x390x320x3A0x320x410x3A0x310x360x3A0x320x30 C= { Report(32) R=1456283550 .1.3.6.1.6.3.15.1.1.4.0=10682 } } }
16:32:43.556473 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 189)
xx.xx.xx.xx.43383 > xx.xx.xx.xx.161: [bad udp cksum cbe2!] { SNMPv3 { F=apr } { USM B=17 T=7831740 U=xxxxxx } { ScopedPDU [!scoped PDU]f9_12_e7_26_90_0c_e9_ef_e3_7a_de_d0_c8_a9_46_9e_07_cb_73_00_73_48_6c_51_5d_09_03_20_44_df_f8_30_68_61_f4_4d_1f_5f_84_19_39_3f_97_ad_67_f7_ae_fc_0e_b9_60_80_ed_43_c8_be_0f_64_bc_50_a9_66_b6_27} }
16:32:43.592305 IP (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto UDP (17), length 197)
xx.xx.xx.xx.161 > xx.xx.xx.xx.43383: [udp sum ok] { SNMPv3 { F=ap } { USM B=17 T=7831740 U=xxxxxx } { ScopedPDU [!scoped PDU]53_3c_f3_1e_e9_40_16_e4_cf_41_9e_36_0e_fd_29_94_13_63_e2_89_34_6e_fd_af_eb_f4_7e_fa_fb_72_56_c4_52_ca_6a_e2_53_72_3e_dd_0d_be_70_cd_3b_80_36_dd_67_44_69_f0_23_e8_a0_92_e2_06_7d_a0_82_e8_68_96_49_2e_69_08_ef_bc_36_a0} }
16:32:43.592551 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 189)
xx.xx.xx.xx.43383 > xx.xx.xx.xx.161: [bad udp cksum 74b!] { SNMPv3 { F=apr } { USM B=17 T=7831740 U=xxxxxx } { ScopedPDU [!scoped PDU]b3_af_bd_25_e6_7c_b4_a5_b2_9b_c6_fe_b4_cb_c6_60_4c_ef_d0_d1_33_e9_9d_57_e8_37_9e_fb_61_5a_66_b7_33_08_db_c1_7e_7c_53_e1_8d_ac_c4_2d_0a_e8_a5_8f_df_12_6e_8a_02_4b_9e_91_4a_7f_60_43_3f_7f_06_88} }
16:32:43.627533 IP (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto UDP (17), length 197)
xx.xx.xx.xx.161 > xx.xx.xx.xx.43383: [udp sum ok] { SNMPv3 { F=ap } { USM B=17 T=7831740 U=xxxxxx } { ScopedPDU [!scoped PDU]6f_f2_eb_5d_37_d0_db_08_a4_d5_1f_7e_08_34_6c_ff_75_70_87_f3_66_a2_c9_4a_dc_56_3c_40_cf_19_65_7f_59_0e_f3_22_09_55_f0_4a_be_12_2b_aa_00_b8_02_b7_e3_8d_e5_b1_52_f3_3a_6a_47_46_01_38_8f_9b_a3_b8_c9_a8_ed_de_73_d3_ef_ec} }



capture from poller for same value in cacti:

16:35:02.104332 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 92)
xx.xx.xx.xx.35483 > xx.xx.xx.xx.161: [bad udp cksum 1817!] { SNMPv3 { F=r } { USM B=0 T=0 U= } { ScopedPDU E= C= { GetRequest(14) R=53577365 } } }
16:35:02.151103 IP (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto UDP (17), length 156)
xx.xx.xx.xx.161 > xx.xx.xx.xx.35483: [udp sum ok] { SNMPv3 { F= } { USM B=17 T=7831878 U= } { ScopedPDU E= 0x800x000x1F0x880x040x300x300x3A0x300x360x3A0x390x320x3A0x320x410x3A0x310x360x3A0x320x30 C= { Report(32) R=53577365 .1.3.6.1.6.3.15.1.1.4.0=10683 } } }
16:35:02.151370 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 189)
xx.xx.xx.xx.35483 > xx.xx.xx.xx.161: [bad udp cksum e02b!] { SNMPv3 { F=apr } { USM B=17 T=7831878 U=xxxxxx } { ScopedPDU [!scoped PDU]ac_53_58_5d_0a_cb_1f_8f_8c_6d_d2_e9_58_23_e8_5e_09_e4_a2_37_6d_c3_4c_49_4f_cb_54_1b_44_b5_13_0c_60_27_16_7b_37_be_b0_e4_79_14_df_82_47_f8_58_28_5c_4e_02_15_bb_fa_03_5c_76_c2_11_d5_20_17_52_64} }
16:35:02.207071 IP (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto UDP (17), length 189)
xx.xx.xx.xx.161 > xx.xx.xx.xx.35483: [udp sum ok] { SNMPv3 { F=ap } { USM B=17 T=7831878 U=xxxxxx } { ScopedPDU [!scoped PDU]9d_ba_a9_d5_1c_4b_fb_f0_1d_28_f4_95_19_ac_9f_ac_72_ff_74_b3_b5_27_60_88_5d_fc_1d_d7_c9_32_85_be_17_22_e9_c9_e1_64_7e_a5_bd_58_97_5a_9a_7b_3b_ba_36_68_b6_68_71_1b_e7_b7_cd_05_03_07_4d_04_59_22} }
16:35:02.267682 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 92)
xx.xx.xx.xx.41223 > xx.xx.xx.xx.161: [bad udp cksum a9fe!] { SNMPv3 { F=r } { USM B=0 T=0 U= } { ScopedPDU E= C= { GetRequest(14) R=53577367 } } }
16:35:02.317755 IP (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto UDP (17), length 156)
xx.xx.xx.xx.161 > xx.xx.xx.xx.41223: [udp sum ok] { SNMPv3 { F= } { USM B=17 T=7831878 U= } { ScopedPDU E= 0x800x000x1F0x880x040x300x300x3A0x300x360x3A0x390x320x3A0x320x410x3A0x310x360x3A0x320x30 C= { Report(32) R=53577367 .1.3.6.1.6.3.15.1.1.4.0=10684 } } }
16:35:02.317948 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 189)
xx.xx.xx.xx.41223 > xx.xx.xx.xx.161: [bad udp cksum 347d!] { SNMPv3 { F=apr } { USM B=17 T=7831878 U=xxxxxx } { ScopedPDU [!scoped PDU]20_b4_58_e9_7a_c2_25_e7_99_fa_60_da_82_48_25_45_2c_6e_75_3e_d6_27_5f_7a_7f_d4_56_e8_56_0d_a7_43_66_48_53_a9_3a_bc_ca_e8_2f_3b_fe_15_ed_5d_53_a5_e7_ad_85_8a_03_2b_69_e4_77_c2_84_37_56_6d_e5_19} }
16:35:02.368475 IP (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto UDP (17), length 189)
xx.xx.xx.xx.161 > xx.xx.xx.xx.41223: [udp sum ok] { SNMPv3 { F=ap } { USM B=17 T=7831878 U=xxxxxx } { ScopedPDU [!scoped PDU]91_6c_79_14_ea_ef_79_f0_08_76_84_67_09_e6_15_ab_0d_c1_6b_b8_c2_d6_3a_6e_64_5d_1e_d9_63_65_d8_15_e4_76_c5_67_42_57_c8_07_7a_f4_3b_61_b0_05_fb_3f_b3_e4_27_1d_a1_cb_fa_1e_7c_b3_f5_c8_0f_1e_1a_a3} }

[gandalf]

Is php-snmp installed (see e.g. "about" page)? If yes, which version?
R.

[gandalf]

We use -O fntev as snmp output options. Please try those along with the -t <timeout> parameter.
In case php-snmp is in place, you may want to uninstall and retry (this will switch to net-snmp libraries for SNMP requests)
R.

[cactaceae]

okay, we scraped the entire Cacti setup and started a completely new install using 0.8.7i with plugin architecture build. We configured all the settings and are not using php-snmp. There are no plugins being used yet as we wanted to go with a clean install. The poller is still pulling incorrect values.


snmpwalk -v3 -u xxxxxx -l authPriv -a MD5 -A xxxxxx -x DES -X xxxxxx -Oftenv 10.131.255.4 .1.3.6.1.4.1.8962.2.1.2.1.7.1
INTEGER: 0

from RRA directory:

filename = "xxxxxx_snmp_oid_14.rrd"
rrd_version = "0003"
step = 300
last_update = 1332270601
header_size = 2048
ds[snmp_oid].index = 0
ds[snmp_oid].type = "GAUGE"
ds[snmp_oid].minimal_heartbeat = 600
ds[snmp_oid].min = 0.0000000000e+00
ds[snmp_oid].max = 1.0000000000e+00
ds[snmp_oid].last_ds = "U"
ds[snmp_oid].value = NaN
ds[snmp_oid].unknown_sec = 1
rra[0].cf = "AVERAGE"
rra[0].rows = 600
rra[0].cur_row = 415
rra[0].pdp_per_row = 1
rra[0].xff = 5.0000000000e-01
rra[0].cdp_prep[0].value = NaN
rra[0].cdp_prep[0].unknown_datapoints = 0
rra[1].cf = "AVERAGE"
rra[1].rows = 700
rra[1].cur_row = 486
rra[1].pdp_per_row = 6
rra[1].xff = 5.0000000000e-01
rra[1].cdp_prep[0].value = 0.0000000000e+00
rra[1].cdp_prep[0].unknown_datapoints = 2
rra[2].cf = "AVERAGE"
rra[2].rows = 775
rra[2].cur_row = 219
rra[2].pdp_per_row = 24
rra[2].xff = 5.0000000000e-01
rra[2].cdp_prep[0].value = NaN
rra[2].cdp_prep[0].unknown_datapoints = 14
rra[3].cf = "AVERAGE"
rra[3].rows = 797
rra[3].cur_row = 173
rra[3].pdp_per_row = 288
rra[3].xff = 5.0000000000e-01
rra[3].cdp_prep[0].value = NaN
rra[3].cdp_prep[0].unknown_datapoints = 230
rra[4].cf = "MAX"
rra[4].rows = 600
rra[4].cur_row = 270
rra[4].pdp_per_row = 1
rra[4].xff = 5.0000000000e-01
rra[4].cdp_prep[0].value = NaN
rra[4].cdp_prep[0].unknown_datapoints = 0
rra[5].cf = "MAX"
rra[5].rows = 700
rra[5].cur_row = 33
rra[5].pdp_per_row = 6
rra[5].xff = 5.0000000000e-01
rra[5].cdp_prep[0].value = -inf
rra[5].cdp_prep[0].unknown_datapoints = 2
rra[6].cf = "MAX"
rra[6].rows = 775
rra[6].cur_row = 52
rra[6].pdp_per_row = 24
rra[6].xff = 5.0000000000e-01
rra[6].cdp_prep[0].value = NaN
rra[6].cdp_prep[0].unknown_datapoints = 14
rra[7].cf = "MAX"
rra[7].rows = 797
rra[7].cur_row = 49
rra[7].pdp_per_row = 288
rra[7].xff = 5.0000000000e-01
rra[7].cdp_prep[0].value = NaN
rra[7].cdp_prep[0].unknown_datapoints = 230


from cacti log:

03/20/2012 12:10:02 PM - CMDPHP: Poller[0] Host[2] DS[14] SNMP: v3: xx.xx.xx.xx
, dsname: snmp_oid, oid: .1.3.6.1.4.1.8962.2.1.2.1.7.1, output: U

[cactaceae]

okay, feeling dumb now as the solution was staring me in the face all the time. I tried to verify the oid using snmpget and that result blew up so I printed the full oid return value using onlly -Of and the answer was right there as the correct full oid string is .1.3.6.1.4.1.8962.2.1.2.1.7.1.0 and not .1.3.6.1.4.1.8962.2.1.2.1.7.1

problem solved. thank you again for all your assistance.

[gandalf]

Oops, missed that. Good you've found it.
R.