Getting NaN on inbound only on ONE graph

[djcronos]

Hi all,

I've already went through the debugging NaN's documentation, and I did not find any errors at all. Everything looks fine.

I am trying to graph an HP ProCurve switch - all other ports on the switch are showing both inbound and outbound - this is the only one.

I don't know if this will help, but here is what Graph Management shows for debug:

Code: Select all

 RRDTool Command:
 
 /usr/bin/rrdtool graph - \
 --imgformat=PNG \
 --start=-86400 \
 --end=-60 \
 --title="h-sw1.garlic.com - Traffic - 24" \
 --rigid \
 --base=1000 \
 --height=120 \
 --width=500 \
 --alt-autoscale-max \
 --lower-limit=0 \
 --vertical-label="bits per second" \
 --slope-mode \
 DEF:a="/opt/cacti/rra/hsw1_garlic_com_traffic_in_3481.rrd":traffic_in:AVERAGE \
 DEF:b="/opt/cacti/rra/hsw1_garlic_com_traffic_in_3481.rrd":traffic_out:AVERAGE \
 CDEF:cdefa=a,8,* \
 CDEF:cdefe=b,8,* \
 AREA:cdefa#00CF00:"Inbound"  \
 GPRINT:cdefa:LAST:" Current\:%8.2lf %s"  \
 GPRINT:cdefa:AVERAGE:"Average\:%8.2lf %s"  \
 GPRINT:cdefa:MAX:"Maximum\:%8.2lf %s\n"  \
 LINE1:cdefe#002A97:"Outbound"  \
 GPRINT:cdefe:LAST:"Current\:%8.2lf %s"  \
 GPRINT:cdefe:AVERAGE:"Average\:%8.2lf %s"  \
 GPRINT:cdefe:MAX:"Maximum\:%8.2lf %s" 
 
 RRDTool Says:
 
 OK

Here's what's really weird - we have an old RRDTool on another machine, and it's graphing the results just fine. It can see both inbound and outbound.

We use Cacti 0.8.6j with the patches as well. We're graphing about 2000 graphs, and this is the ONLY one that is causing us grief.

As far as poller, we are using cmd.php with a max of 1 concurrent poller processes, 4 max threads per process, 4 php script servers, and a script and server timeout of 25 seconds.

Also, (i think it isn't related) I notice that there are TONS of the following, each polling period:

Code: Select all

 05/07/2007 12:11:13 PM - CMDPHP: Poller[0] Host[24] DS[3417] WARNING: Result from SNMP not valid.  Partial Result:
 05/07/2007 12:11:24 PM - CMDPHP: Poller[0] Host[48] DS[554] WARNING: Result from SNMP not valid.  Partial Result:
 05/07/2007 12:11:24 PM - CMDPHP: Poller[0] Host[49] DS[562] WARNING: Result from SNMP not valid.  Partial Result:
 05/07/2007 12:11:29 PM - CMDPHP: Poller[0] Host[54] DS[613] WARNING: Result from SNMP not valid.  Partial Result:
 05/07/2007 12:11:38 PM - CMDPHP: Poller[0] Host[60] DS[632] WARNING: Result from SNMP not valid.  Partial Result:
 05/07/2007 12:11:44 PM - CMDPHP: Poller[0] Host[68] DS[804] WARNING: Result from SNMP not valid.  Partial Result:
 05/07/2007 12:11:48 PM - CMDPHP: Poller[0] Host[73] DS[931] WARNING: Result from SNMP not valid.  Partial Result:
 05/07/2007 12:11:51 PM - CMDPHP: Poller[0] Host[82] DS[964] WARNING: Result from SNMP not valid.  Partial Result: No Such Instance cur
 05/07/2007 12:11:52 PM - CMDPHP: Poller[0] Host[84] DS[981] WARNING: Result from SNMP not valid.  Partial Result: No Such Instance cur
 05/07/2007 12:11:53 PM - CMDPHP: Poller[0] Host[85] DS[989] WARNING: Result from SNMP not valid.  Partial Result: No Such Instance cur
 05/07/2007 12:11:54 PM - CMDPHP: Poller[0] Host[86] DS[997] WARNING: Result from SNMP not valid.  Partial Result: No Such Instance cur
 05/07/2007 12:11:54 PM - CMDPHP: Poller[0] Host[94] DS[3328] WARNING: Result from SNMP not valid.  Partial Result:
 05/07/2007 12:11:55 PM - CMDPHP: Poller[0] Host[96] DS[3358] WARNING: Result from SNMP not valid.  Partial Result:
 05/07/2007 12:15:05 PM - CMDPHP: Poller[0] Host[4] DS[68] WARNING: Result from SNMP not valid.  Partial Result:
 05/07/2007 12:15:11 PM - CMDPHP: Poller[0] Host[8] DS[2640] WARNING: Result from SNMP not valid.  Partial Result:
 05/07/2007 12:15:13 PM - CMDPHP: Poller[0] Host[12] DS[187] WARNING: Result from SNMP not valid.  Partial Result:
 05/07/2007 12:15:35 PM - CMDPHP: Poller[0] Host[18] DS[2728] WARNING: Result from SNMP not valid.  Partial Result:
 05/07/2007 12:15:47 PM - CMDPHP: Poller[0] Host[20] DS[2826] WARNING: Result from SNMP not valid.  Partial Result:

There are tons more, I removed all the duplicates.

Any insight on my issues are greatly appreciated, as always!

[djcronos]

So I tried to add more graphs and now it seems that every graph that I add from now on only shows outbound and not inbound.

Any ideas how I may troubleshoot this?

Thanks all in advance.

[djcronos]

Anyone?

Heck, I'll pay someone if they can fix it. Send me a PM - you gotta be good with debugging NaN's though - and if would help if you had any experience with debugging NaN's that are only showing up for inbound and not outbound.

I'm pulling my hair out over here!

[BSOD2600]

On Host[24], click on DS[3417] to see what snmp query its related to. Then look in your snmp cache for Host[24] and see what OID cacti is polling for DS[3417]. Manually snmpget/snmpwalk that OID yourself and see if its valid or not.

[djcronos]

Code: Select all

05/09/2007 01:15:03 PM - CMDPHP: Poller[0] Host[4] DS[68] WARNING: Result from SNMP not valid.  Partial Result:
05/09/2007 01:15:03 PM - CMDPHP: Poller[0] Host[4] DS[68] SNMP: v1: www.xxx.com, dsname: traffic_out, oid: .1.3.6.1.2.1.2.2.1.16.7, output: U
05/09/2007 01:15:03 PM - CMDPHP: Poller[0] DEBUG: SQL Exec: "insert into poller_output (local_data_id, rrd_name, time, output) values (68, 'traffic_out', '2007-05-09 13:15:03', 'U')"

I do an snmpget on OID .1.3.6.1.2.1.2.2.1.16.7 and get the following:

Code: Select all

Error in packet
Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: IF-MIB::ifOutOctets.7

If I wanted it to poll the right results, what would I do?

Also, is this really related to the NaN's? Or is just coincidentally another issue?

Thanks.

[BSOD2600]

1) Have you tried manually reindexing the host (click the green circle) ?
2) What Re-Index Method are you using for the SNMP - Interface Statistics data query?
3) snmpwalk .1.3.6.1.2.1.2.2.1 and post the results.

[djcronos]

1) I have before, and I just went through each host that was having an issue and clicked on it again. What exactly would that accomplish? I thought that just refreshes the list?

2) Uptime goes backward.

3)

Code: Select all

Error in packet
Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: IF-MIB::ifEntry

[djcronos]

If I were to import a template and it overwrote one normally used, would that only affect new graphs I create?

I'm trying to think of what has been changed ever, and the only thing I can think of might be a template I imported but never worked properly...

[djcronos]

Found the issue - when I imported the Trango Templates, it overwrote my default template to graph SNMP results - importing the Cisco 7200 templates fixed the issue again.

Thanks for being so kind and helpful - perhaps this could be another addition to debugging NaN's.

[gandalf]

I will consider this
Reinhard