BIND9 script doesn't work after upgrade (SOLVED-Workaround?)

lcano · Post by **lcano** » Thu Dec 17, 2009 7:47 am

Hi all,

not sure if it can be a cacti version problem, or a script problem, or probably I am the problem

. The issue is that i have and old version of cacti (0.8.6.j) running a .xml data query to collect Bind9 statistics. I installed a new cacti updated to the last version (0.8.7.e), and exported the data query used in the 0.8.6j version with its dependences (host template, graph template, etc...). Then, I added the same host in the new version that I was montoring perfectly with the last one. Every Graph template and data query in the new one works perfectly, but the data query for Bind9 statistics doesn't.

I tried to troubleshoot it issuing "verbose query" in the bind9 statistics data query, getting this result:

Code: Select all

+ Running data query [10].
+ Found type = '3' [snmp query].
+ Found data query XML file at '/aplicaciones/cacti-0.8.7e/resource/snmp_queries/bind9-stats-snmp.xml'
+ XML file parsed ok.
+ Executing SNMP walk for list of indexes @ '.1.3.6.1.4.1.2021.55.1'
+ No SNMP data returned
+ Found data query XML file at '/aplicaciones/cacti-0.8.7e/resource/snmp_queries/bind9-stats-snmp.xml'
+ Found data query XML file at '/aplicaciones/cacti-0.8.7e/resource/snmp_queries/bind9-stats-snmp.xml'
+ Found data query XML file at '/aplicaciones/cacti-0.8.7e/resource/snmp_queries/bind9-stats-snmp.xml'

Then, I issue the specific snmpwalk query manually from the command line of the server, to the OID which reports "No SNMP data returned". And my susprise is that I get the reply normaly. In fact, I tried the same query in the old cacti version and I get the same answer:

Code: Select all

lognet4:/aplicaciones/cacti/resource/snmp_queries # snmpbulkwalk -c <community> -v2c <host> .1.3.6.1.4.1.2021.55.1
UCD-SNMP-MIB::ucdavis.55.1.1 = INTEGER: 1

At this point, I get blocked.

Same server to query, from different cacti servers. Same data query configured (exported from the old one and imported in the new one), and it doesn't work in the new one. Do I have to be careful with any specific issue when exporting the data template? Any idea of how can I continue the troubleshooting?

Thanks in advance.

Post by **gandalf** » Fri Dec 18, 2009 8:28 am

It is possible, that following issues show up:
- Cacti changed interface of "cacti_snmp_walk" and stuff. Those were frequently used within scripts. But this seems to be a pure SNMP things, so it should not be an issue
- your query seems to make use of the SNMP exec option. This has changed to SNMP execute (or was it vice versa?) and needs specific attention when using net-snmp V5.4.x

R.

lcano · Post by **lcano** » Fri Dec 18, 2009 9:04 am

I agree with you. It seems more something related to the way that new cacti version uses snmp. Below, the data query that fails. Note that snmp version is under 5.4.x.

lognet54:/aplicaciones/cacti-0.8.7e/resource/snmp_queries # snmpwalk -V
NET-SNMP version: 5.3.rc2
lognet54:/aplicaciones/cacti-0.8.7e/resource/snmp_queries # snmpbulkwalk -V
NET-SNMP version: 5.3.rc2

Code: Select all

lognet54:/aplicaciones/cacti-0.8.7e/resource/snmp_queries # more bind9-stats-snmp.xml 
<query>
        <name>Get BIND 9 Statistics via SNMP</name>
        <description>Queries a host for a list of zones</description>
        <oid_index>.1.3.6.1.4.1.2021.55.1</oid_index>
        
        <fields>
                <bindIndex>
                        <name>Index</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>input</direction>
                        <oid>.1.3.6.1.4.1.2021.55.1</oid>
                </bindIndex>
                <bindName>
                        <name>Name</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>input</direction>
                        <oid>.1.3.6.1.4.1.2021.55.2</oid>
                </bindName>
                <bindSuccess>
                        <name>Successful Query Count</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.2021.55.3</oid>
                </bindSuccess>
                <bindFailed>
                        <name>Failed Query Count</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.2021.55.4</oid>
                </bindFailed>
                <bindNxdomain>
                        <name>NXDOMAIN Query Count</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.2021.55.5</oid>
                </bindNxdomain>
                <bindNxrrset>
                        <name>NXRRSET Query Count</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.2021.55.6</oid>
                </bindNxrrset>
                <bindReferred>
                        <name>Referred Query Count</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.2021.55.7</oid>
                </bindReferred>
                <bindRecursive>
                        <name>Recursive Query Count</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.2021.55.8</oid>
                </bindRecursive>
        </fields>
</query>

Post by **TheWitness** » Fri Dec 18, 2009 9:44 pm

So, just out of curiosity, did you apply the patches from the main web site? This might solve your problem. Try sending a wireshark output when running the command via PM to me.

TheWitness

lcano · Post by **lcano** » Sun Dec 20, 2009 12:54 pm

Hi Thewitness,

This is the information of the system. I installed pathes for that version, as recommended in the webpage.

I've just sent you a MP with the required information.

Code: Select all

General Information
Date 	Sun, 20 Dec 2009 18:50:45 +0100
Cacti Version 	0.8.7e
Cacti OS 	unix
SNMP Version 	NET-SNMP version: 5.3.rc2
RRDTool Version 	RRDTool 1.2.x
Hosts 	71
Graphs 	5002
Data Sources 	Script/Command: 10
SNMP: 30
SNMP Query: 4923
Script Query: 6
Script - Script Server (PHP): 35
Total: 5004

Poller Information
Interval 	300
Type 	spine
Items 	Action[0]: 9876
Action[1]: 22
Action[2]: 35
Total: 9933
Concurrent Processes 	1
Max Threads 	4
PHP Servers 	1
Script Timeout 	25
Max OID 	10
Last Run Statistics 	Time:14.4325 Method:spine Processes:1 Threads:4 Hosts:72 HostsPerProcess:72 DataSources:9933 RRDsProcessed:4925

Thanks

Luis

Post by **TheWitness** » Sun Dec 20, 2009 2:01 pm

What does your apache error log look like?

TheWitness

lcano · Post by **lcano** » Mon Dec 21, 2009 4:21 am

Hi,

from my point of view, no strange messages in the apache error log file. I sent it to you via MP, with the screenshot of the poller settings.

I tried with a second diferente server, running also same BIND9 version, and the behaviour is the same. I also checked the SNMP configuration in the server and it seems no problem on it. As you could see, if I do SNMP queries from the operationg system, it replies perfectly.

Post by **TheWitness** » Mon Dec 21, 2009 9:27 am

Well looking at your error_log, you system is a bit hosed. Looks like your upgrade was not a clean one. Best to handle this like a support call. You should PM me your availability.

TheWitness

lcano · Post by **lcano** » Mon Dec 28, 2009 4:05 am

Hi, Thewitness,

don't know if you received my PM

Post by **TheWitness** » Tue Dec 29, 2009 11:13 am

Well, still on vacation, but I did take a second look at your wireshark capture, and it's a bit hosed. For example, you perform a bulk request of:

SNMPv2-SMI::enterprises.2021.55.1 (1.3.6.1.4.1.2021.55.1): Value (Null)

However, the response is way wrong. Please change the MAXOID's for this host to 1, save, and then retry the verbose query again and send along another wireshark capture.

What I can confirm is that the bulk request is returning invalid data, I will provide a screen shot from Wireshark that will demonstrate this momentarily. So, it's the Agent on the far side having issues with "bulk" type requests whose response results in a boundary check error on the Cacti side that Cacti was not written to handle.

It's a bit of a bug, but if you agent were responding correctly, we would not be having this conversation.

TheWitness

Post by **TheWitness** » Tue Dec 29, 2009 11:20 am

Please see the graphic attached.

TheWitness

Post by **TheWitness** » Tue Dec 29, 2009 11:23 am

Oh, sorry, that's my dog at ~3 months roughly 4 years ago.

TheWitness

lcano · Post by **lcano** » Tue Dec 29, 2009 12:22 pm

Hi Thewitness,

I didn't know you were on vacations, so sorry. On the other hand, really cute dog. Nice picture in the middle of the forum

Going on with the test, I changed the maximun OID for this host as you recomended, I did again the verbose query, getting a capture with tcpdump. Unfortunately, same result

(attached),. As you confirmed, some more information in the stream which shouldn't be there

Also attached the screenshot where I configured maxoid=1 for the host, and of course, SAVE button.

Trying to get you more information, I also launch spine for the specific host with verbosity=10, and it seems that it continues getting the value=10 for maxoid. I don't understand

(file3 attached)...

Then, I check directly the database, but I suppose that this conf is the general one, which will be override for the specific for any host:

Code: Select all

mysql> select value from settings where name = 'max_get_size';
+-------+
| value |
+-------+
| 10    | 
+-------+
1 row in set (0.00 sec)

Even more, trying to check if that can solve the issue, I changed temporarely the global setting:

Code: Select all

mysql> select value from settings where name = 'max_get_size';
+-------+
| value |
+-------+
| 1     | 
+-------+
1 row in set (0.00 sec)

but I had the same capture with TCPDUMP, where I can see some strange OID's in the answer.

Finaly, I tried to reproduce exactly the same snmp query that cacti should do (from my understanding), and I don't get those strange OID's in the answer:

Code: Select all

lognet54:/aplicaciones/cacti-0.8.7e/include # snmpbulkget -c /publica/-mad -v2c 10.118.244.174 .1.3.6.1.4.1.2021.55.1
UCD-SNMP-MIB::ucdavis.55.2.1 = INTEGER: 1
UCD-SNMP-MIB::ucdavis.55.3.1 = STRING: "Estadisticas Peticiones DNS"
UCD-SNMP-MIB::ucdavis.55.4.1 = INTEGER: 18298078
UCD-SNMP-MIB::ucdavis.55.5.1 = INTEGER: 362996
UCD-SNMP-MIB::ucdavis.55.6.1 = INTEGER: 4128555
UCD-SNMP-MIB::ucdavis.55.7.1 = INTEGER: 466733
UCD-SNMP-MIB::ucdavis.55.8.1 = INTEGER: 5049297
UCD-SNMP-MIB::ucdavis.55.9.1 = INTEGER: 21447945
UCD-SNMP-MIB::versionIndex.0 = INTEGER: 1
UCD-SNMP-MIB::versionTag.0 = STRING: 5.3.0.1

Any other test that could help to find out if it is a bug, please let me know.

Thanks.

Luis

Post by **TheWitness** » Tue Dec 29, 2009 1:52 pm

It's still performing a bulk get request. Change snmp to v1 and try again. Send me the new capture once that's done.

TheWitness

lcano · Post by **lcano** » Tue Dec 29, 2009 4:04 pm

Here it is:

BIND9 script doesn't work after upgrade (SOLVED-Workaround?)

BIND9 script doesn't work after upgrade (SOLVED-Workaround?)

Who is online