Invalid SNMP Result

[rizlo100]

Hey All,

Recently did a new install of cacti 0.8.7g and spine 0.8.7g, running on top of CentOS 5.6 x64.

Up until yesterday things were going very well. But somewhere around 5:30ish PM, I started getting invalid SNMP results on some OID's. In the logfile I am seeing the following.

07/08/2011 01:21:57 PM - SPINE: Poller[0] Host[11] TH[1] DS[527] SNMP: v2: 01-w01.prod, dsname: sscClientOutda, oid: .1.3.6.1.4.1.12345.8.2.4.0, value: U

Prior to yesterday at 5:30pm this was graphing without any issue. I can do a walk, get, bulkget against the OID and get proper results without any issue. Just cacti can not (using spine or cmd.php)

[tor-cacti.prod:~]# snmpget -v2c -c public 01-w01:4000 .1.3.6.1.4.1.12345.8.2.4.0
SNMPv2-SMI::enterprises.12345.8.2.4.0 = Counter32: 0

[tor-cacti.prod:~/cacti-stuff/cacti-spine-0.8.7g]# snmpbulkwalk -c public -v2c -Cr50 01-w01.prod:4000 .1.3.6.1.4.1.12345.8.2.4.0
SNMPv2-SMI::enterprises.12345.8.2.4.0 = Counter32: 0


I've gone through and decreased Max OID's per get request with no change, tried cmd.php and same problem. This is affecting multiple hosts.


Here is an example of a successful run.

07/06/2011 03:55:03 PM - SPINE: Poller[0] Host[11] TH[1] DS[527] SNMP: v2: 01-w01.prod, dsname: sscClientOutda, oid: .1.3.6.1.4.1.12345.8.2.4.0, value: 0

The only thing I can see happening around there is the Discover plugin made a run Starting around 5:40, and the poller through the following warnings at the end.

07/07/2011 05:47:01 PM - DISCOVER: Poller[0] 761 IPs Scanned, 130 IPs Responded to Ping, 79 Responded to SNMP, 0 Device Added, 0 Graphs Added to Cacti
07/07/2011 05:50:02 PM - SPINE: Poller[0] Host[2] ERROR: Empty result [209.X.X.X]: '/var/www/cacti/scripts/ss_fping.php ss_fping 209.X.X.X'

And since I've been seeing the value: U since. Unfortunatly I didn't have much logging enable yesterday evening, but I have no idea why all of a sudden it would break...

[rizlo100]

Here is a SS of the data template. No changes made to it since import.

[rizlo100]

I've been able to track down this problem.

It seems that in the cacti DB under the tables "poller_item" the field for "snmp_port" was incorrect. I have no idea how this was changed as in my data templates it's clearly set for port 4000.

Once I was manually updated the DB tables with phpmyadmin my graphs started working again and no more "Value: U" returns. This leads me to 2 questions that I hope someone will be able to address.

1) What overwrote these values. I can only assume that prior to yesterdays Discover run they were correct in the DB as the graphs were working. I'm running under the assumption that something in the Discover run overwrote the table fields.

2) How come when I added a NEW host that the values (snmp port) from the data template did not carry over into the db? Again, in the data template I've set the port to 4000, and yet, after adding a new host around noon today, I had to correct the DB fields manually as well.

[gandalf]

I'm not sure why "(snmp) port" sows up 4 times on your data template. I assume that this might me the root cause
R.

[rizlo100]

I wonder if this could be a result of importing templates from a older version of cacti? The origional data templates came from a export on a 0.8.6i system. I also went through a number of the default templates that came with Cacti and a some have the port listed 4 times, some only once and a few with it twice.

Since when adding new hosts the DB does not pick up the proper fields for the templates, I feel my only resolution is to blow away the DB and start again.

[gandalf]

I'd try to "repair" the template. But in case loosing the db is of no importance, you may go that way

R.

[rizlo100]

I ended up leaving the templates as-is. After some more digging, I found many of the additional "SNMP Port" fields were really just there for no reason, they don't get written to the DB and as best as I can tell, have no effect. The only field that seems to take effect is the "Port", this is listed second on my data templates.

Once I update all the port numbers in that field, the template has been working without issue. I may go back later on and see about removing the useless fields, but other things are more important right now such as getting my historical data showing.

Thanks for your help!