Many graphs stopped suddenly several days ago

[jeremyp]

Running Cacti 0.8.6g on Solaris 9 with RRDTOOL 1.2.11 and Net-SNMP 5.1(I think)

All of a sudden, on Friday, many of my custom graphs ceased to work. I keep getting the error "Result from SNMP not valid. Partial Result: ... ", but running an snmpget on those OIDs show that the correct data is returned.

I switched to debug mode, and here is some of the log output for the problem I have:

11/01/2005 11:56:58 AM - CACTID: Poller[0] Host[2] DS[291] SNMP: v1: PRVAAD01, dsname: webct_node10_sess, oid: .1.3.6.1.4.1.140.625.430.1.50.32.99.48.55.50.99.51.98.98.54.48.54.97.54.56.49.52.97.99.98.100.48.101.50.53.99.50.54.51.53.53.50.53, value: U
11/01/2005 11:56:58 AM - CACTID: Poller[0] DEBUG: SQLCMD: insert into poller_output (local_data_id,rrd_name,time,output) values (291,'webct_node10_sess','2005-11-01 11:56:57','U')
11/01/2005 11:56:58 AM - CACTID: Poller[0] Host[2] DS[377] WARNING: Result from SNMP not valid. Partial Result: ...
11/01/2005 11:56:58 AM - CACTID: Poller[0] Host[2] DS[377] SNMP: v1: PRVAAD01, dsname: webct_node11_sess, oid: .1.3.6.1.4.1.140.625.430.1.50.32.49.51.52.52.52.55.97.49.53.98.55.52.54.55.50.52.100.97.51.53.100.54.98.50.53.48.97.52.48.57.97.102, value: U
11/01/2005 11:56:58 AM - CACTID: Poller[0] DEBUG: SQLCMD: insert into poller_output (local_data_id,rrd_name,time,output) values (377,'webct_node11_sess','2005-11-01 11:56:57','U')
11/01/2005 11:56:58 AM - CACTID: Poller[0] Host[2] DS[292] WARNING: Result from SNMP not valid. Partial Result: ...
11/01/2005 11:56:58 AM - CACTID: Poller[0] Host[2] DS[292] SNMP: v1: PRVAAD01, dsname: webct_node17_sess, oid: .1.3.6.1.4.1.140.625.430.1.50.32.49.53.102.100.51.101.101.102.49.99.102.100.99.57.53.48.50.99.99.98.55.54.56.57.50.99.97.99.98.57.48.100, value:
11/01/2005 11:56:58 AM - CACTID: Poller[0] DEBUG: SQLCMD: insert into poller_output (local_data_id,rrd_name,time,output) values (292,'webct_node17_sess','2005-11-01 11:56:57','U')
11/01/2005 11:56:58 AM - CACTID: Poller[0] Host[2] DS[293] WARNING: Result from SNMP not valid. Partial Result: ...
11/01/2005 11:56:58 AM - CACTID: Poller[0] Host[2] DS[293] SNMP: v1: PRVAAD01, dsname: webct_node18_sess, oid: .1.3.6.1.4.1.140.625.430.1.50.32.99.55.56.54.48.98.57.49.52.54.52.50.99.48.48.99.97.55.101.54.55.52.53.51.49.50.48.98.48.97.57.99, value: U
11/01/2005 11:56:58 AM - CACTID: Poller[0] DEBUG: SQLCMD: insert into poller_output (local_data_id,rrd_name,time,output) values (293,'webct_node18_sess','2005-11-01 11:56:57','U')

Anyone have any ideas why this would just stop working all fo a sudden and how I can fix it?

[jeremyp]

Forgot to add I'm using cactid as the poller, but receive the same problem with the php poller.

[BSOD2600]

Would you mind posting an snmpwalk of one of those long OIDs which cacti should be polling?

You using cactid 0.8.6f-1 ?

[jeremyp]

I'm using version 0.8.6e of CactiD.

Performing an SNMPWalk on one of the long OIDs and I get:

(This is for the BEA Weblogic OID jvmJavaRuntimeHeapFreeCurrent)



bash-2.05$ snmpwalk -v 1 -c mycommunity@mynode ADMINSERVER:1461 .1.3.6.1.4.1.140.625.340.1.25.32.98.49.102.50.101.100.53.49.51.102.100.53.56.100.51.102.53.100.102.50.55.53.56.56.52.50.50.51.55.98.54.102
SNMPv2-SMI::enterprises.140.625.340.1.25.32.98.49.102.50.101.100.53.49.51.102.100.53.56.100.51.102.53.100.102.50.55.53.56.56.52.50.50.51.55.98.54.102 = INTEGER: 735524424


And all the OIDs are correct still, so I don't know why the graphs stopped suddenly like they did. It's really confusing me, and I'm not sure how to get them working again - I don't want to have to delete and recreate if possible, but I'm unsure of what else to do. And I've got to get these things working asap too!

[jeremyp]

Any ideas? If I can't figure this out soon I'll have to use a different solution, which I don't want to do

[BSOD2600]

I'm not sure offhand. Another use a while ago had the same problem with long OIDs, and I thought the devs made a patch for it.

At any rate, a dev will have to comment on why it isnt working.

[jeremyp]

Ok, I'll wait on the devs a little bit then. I don't think it's the OID length though, or else they wouldn't have even begun to graph anything, right? Poking around the database, I can see the entire OIDs I specified listed, so I don't think that's it. Thanks for your help though!

[jeremyp]

Should I run the poller reindex script found at http://forums.cacti.net/viewtopic.php?t=8583&start=60 ? I'd really really love to get these working! I've followed most of the steps at the "Graph stopped working" thread, and I'm at my wits end!

[jeremyp]

Another weird thing here - when I was following along with the other thread, one of the suggestions was to truncate a table. I did so, and then all of a sudden one graph item had a value, but wouldn't graph - so I'm unsure of what that means....

[dgamble]

Can you just confirm that the OIDs being polled do, in fact, return values when queried with snmpget?

I just ask because the example snmpwalk you posted doesn't match any of the OIDs in the log file you posted.

Can you post a log entry for a failed OID, and then a query of you trying to snmpget that object?

[jeremyp]

Oh, certainly. Here's a response from one of the above log entries:

Log Entry:
11/01/2005 11:56:58 AM - CACTID: Poller[0] Host[2] DS[291] SNMP: v1: PRVAAD01, dsname: webct_node10_sess, oid: .1.3.6.1.4.1.140.625.430.1.50.32.99.48.55.50.99.51.98.98.54.48.54.97.54.56.49.52.97.99.98.100.48.101.50.53.99.50.54.51.53.53.50.53, value: U
11/01/2005 11:56:58 AM - CACTID: Poller[0] DEBUG: SQLCMD: insert into poller_output (local_data_id,rrd_name,time,output) values (291,'webct_node10_sess','2005-11-01 11:56:57','U')
11/01/2005 11:56:58 AM - CACTID: Poller[0] Host[2] DS[377] WARNING: Result from SNMP not valid. Partial Result: ...





Snmpget:
bash-2.05$ ./snmpget -v 1 -c COMMUNITY PRVAAD01 .1.3.6.1.4.1.140.625.430.1.50.32.99.48.55.50.99.51.98.98.54.48.54.97.54.56.49.52.97.99.98.100.48.101.50.53.99.50.54.51.53.53.50.53
SNMPv2-SMI::enterprises.140.625.430.1.50.32.99.48.55.50.99.51.98.98.54.48.54.97.54.56.49.52.97.99.98.100.48.101.50.53.99.50.54.51.53.53.50.53 = INTEGER: 215
bash-2.05$

[dgamble]

That OID is 131 characters in length, which is just under the previous limit. Just to be sure, you may want to check the bug database to make sure your database columns have the right size, though, which could cause an issue with other objects.

I encountered a similar issue with cactid last week, although the problem was with 0.8.6f, and working with 0.8.6e- and it was with string objects, while you have integers here.

There's a decent chance the problem is a bug, so, you may want to open one up in the bug database so that one of the devels can look at it. As I noted, parsing seems to have changed a bit in 0.8.6f, so, it could be that this version works and/or provides more information on your problem.

Good luck!

[rony]

The database field for OID was increased to 255 chars in 0.8.6g.

[jeremyp]

The database field for OID was increased to 255 chars in 0.8.6g.

Then do you have ides, Rony?

[dgamble]

The database field for OID was increased to 255 chars in 0.8.6g.

Then do you have ides, Rony?

As I noted, the issue is probably not the oid length.

Try the suggestions in my message here.