Spine not polling correct OIDs from Netscaler

[codyjohn26]

We are seeing a very peculiar issue when graphing our Citrix Netscaler VIPs. We graph in/out bytes via a COUNTER SNMP OID for each VIP. This has always worked great, until a recent upgrade to the Citrix boxes.

Now it appears that Spine is somehow mangling the OID that I specify for the data source. At first the obvious conclusion was that the Citrix upgrade broke the SNMP stat collecting. However, I was able to quickly refute this by graphing the OID independently of Cacti using Perl and GD::Graph. I have ruled out rrdtool, by confirming in the logs that it is simply using the incorrect values passed to it by Spine. A tcpdump confirms that the problem lies somewhere between spine and snmpwalk, because the SNMP GET requests kicked off during the polling cycle are grabbing incorrect OIDs.

For example, let's look at an OID for vsvrTotalResponseBytes for one particular VIP: .1.3.6.1.4.1.5951.4.1.3.1.1.33.9.97.117.99.49.95.104.116.116.112.

Here is the output of an snmpwalk performed on the OID at roughly the same time that poller performed the same task (notice the discrepancy in return values):

== Spine ==
11:40:01 AM - SPINE: Poller[0] Host[44] DS[2030] SNMP: v1: 10.1.4.233, dsname: snmp_oid, oid: .1.3.6.1.4.1.5951.4.1.3.1.1.33.9.97.117.99.49.95.104.116.116.112, value: 2034440842

== SNMP ==
cacti02:/opt/apache2/htdocs/cacti/log # date; snmpwalk -v 2c -c xxxxx -m /usr/share/snmp/mibs/NS-9.1.MIB-smiv2.txt 10.1.4.233 .1.3.6.1.4.1.5951.4.1.3.1.1.33.9.97.117.99.49.95.104.116.116.112
Tue Feb 9 11:40:04 MST 2010
NS-ROOT-MIB::vsvrTotalResponseBytes."auc1_http" = Counter64: 20372200549

That leaves us with:
Spine: 2034440842
SNMP: 20373300549

The even more peculiar thing is that it appears Spine is using the same erroneous OID for multiple different data sources, so many graphs for different VIPs on one host end up looking the same.

Has anyone see anything similar to this or have any suggestion on what to look for? I'm at a bit of a loss at the moment. Thanks.

--
Cody

[codyjohn26]

BTW, this problem first affected cacti/spine 0.8.7d. I have upgraded both to 0.8.7e without affect. We have another cacti box running 0.8.7d which is able to graph these OIDs successfully.

[codyjohn26]

Another thing worth mentioning, if I switch to the PHP poller, the correct data is returned. However, using this method, the Netscaler CPU spikes between ~60-100%

[gandalf]

Please contact TheWitness
R.

[codyjohn26]

Per advice from TheWitness, I set "Maximum OIDs Per GET Request" to 1 and that solved the problem. Apparently the Netscaler was not returning results in the same order in which snmp had asked for them.

Thanks!

--
Cody