Data Query help

[sbindl01]

I'm new to Data Queries and have been trying to figure out how to apply them for my situation. I have several thousand Alcatel 7750s that I'm polling, and need QoS statistics for each port/interface. The mib is a little different than the standard ifMib.IfTable in that it's entries are OID.IfIndex.
The Mib i'm getting values from is OID.ifIndex.qNumber (1.3.6.1.4.1.6527.3.1.2.2.12.1.1.4.1.ifIndex.1-16 for qos q's 1-6 on that particular interface).
Is this possible via a Data Query? I've been trying to figure out by reading the doc but no luck (confused).
Any help would be appreciated!

[sbindl01]

Also...I've tried to get the values via a perl script as entry to a Data Input, as well as a php script for a script server data input. Both aren't scaling very well.

[gandalf]

Also...I've tried to get the values via a perl script as entry to a Data Input, as well as a php script for a script server data input. Both aren't scaling very well.

Sure
R.

[gandalf]

I'm new to Data Queries and have been trying to figure out how to apply them for my situation. I have several thousand Alcatel 7750s that I'm polling, and need QoS statistics for each port/interface. The mib is a little different than the standard ifMib.IfTable in that it's entries are OID.IfIndex.
The Mib i'm getting values from is OID.ifIndex.qNumber (1.3.6.1.4.1.6527.3.1.2.2.12.1.1.4.1.ifIndex.1-16 for qos q's 1-6 on that particular interface).
Is this possible via a Data Query? I've been trying to figure out by reading the doc but no luck (confused).
Any help would be appreciated!

Please see tha DQ walkthrough at 1st link of my sig and start with the very simplest XML. Then come back to us with the results and let's turn to the next step.
R.

[sbindl01]

mib_table.png (163.63 KiB) Viewed 1299 times

Ok....I'm working through the doc and starting to get it, but my confusion still seems to be my trying to apply these examples to situation i'm trying to solve. The examples are based on mib tables that are structured as OID.Index for each value....my table is OID.Index.OID. There is no number of indexes.

Here is example of what I'm trying to get...
I can get a list of interface indexes form the standard interfaces table and say it i get 37781504 as the ifIndex for a port. I then need to go get the qos counters for that index in an Alcatel specific mib. The table is setup such that it has an entry in the table for each specific q based on it's q number, in this case each port has 16 qos queues with some counters for each queue (i'm looking for ingress.inprofile.forwarded and ingress.outofprofile.forwarded and adding them together to get total forwarded in each q)

inprofile = .1.3.6.1.4.1.6527.3.1.2.2.12.1.1.4
outofprofile = .1.3.6.1.4.1.6527.3.1.2.2.12.1.1.5

so for each q it is
OIDInProfile.ifIndex.Qnumber
OIDOutProfile.ifIndex.Qnumber

.1.3.6.1.4.1.6527.3.1.2.2.12.1.1.4.37781504.1
.1.3.6.1.4.1.6527.3.1.2.2.12.1.1.5.37781504.1

.1.3.6.1.4.1.6527.3.1.2.2.12.1.1.4.37781504.2
.1.3.6.1.4.1.6527.3.1.2.2.12.1.1.5.37781504.2

and so on up to 16....
Looking via a mib browser, it seems the index is acually oid.1.ifIndex.qNumber
I'll attach a screen shot of the mib table, might make better sense to you, along with
a current graph that is using a script at the moment.

My php scripts i've created for a script/scriptserver data input requires an ifIndex,
then in a loop i get the values
$ifIndx=37781504;
for ($i=1; $i<16;$i++) {
$q[$i] = snmp2_get .1.3.6.1.4.1.6527.3.1.2.2.12.1.1.4.$ifIndex.$i ;
$q[$i] = $q[$i] + snmp2_get .1.3.6.1.4.1.6527.3.1.2.2.12.1.1.5.$ifIndex.$i ;
}
then returns the array to cacti for my data source
return "q1: $q[1] q2: $q[2] ..."
into a single data source that has q1, q2, q2, q4... for the rrd file data source items.

This works fine...just doesn't scale to tens of thousands of interfaces

I looked at this doc http://docs.cacti.net/howto:data_query_templates
Was wondering if the snmp does not exist example is relevant here?

graph_image.png (90.34 KiB) Viewed 1299 times

mib_table.png (163.63 KiB) Viewed 1299 times

[sbindl01]

sorry for the double upload of the image...

i made one mistake in my message

.1.3.6.1.4.1.6527.3.1.2.2.12.1.1.4.37781504.1
.1.3.6.1.4.1.6527.3.1.2.2.12.1.1.5.37781504.1

.1.3.6.1.4.1.6527.3.1.2.2.12.1.1.4.37781504.2
.1.3.6.1.4.1.6527.3.1.2.2.12.1.1.5.37781504.2

should be

.1.3.6.1.4.1.6527.3.1.2.2.12.1.1.4.1.37781504.1
.1.3.6.1.4.1.6527.3.1.2.2.12.1.1.5.1.37781504.1

.1.3.6.1.4.1.6527.3.1.2.2.12.1.1.4.1.37781504.2
.1.3.6.1.4.1.6527.3.1.2.2.12.1.1.5.1.37781504.2

which shows 1.ifIndex.Qnum as the index....

[maguac]

Am also having an issue getting my snmp query to work properly. I have gone thorough the data walk through but I dont seem to be doing something right. Please assist with setting the query xml correctly.
Am trying to graph the resource usage (Resource -Max, Peak, Current, Denied) on my firewall using the following:
Resource usage by OID names
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitMax."3x".macAddresses = Gauge32: 766
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitMax."3x".concurrentConns = Gauge32: 11700
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitMax."3x".natTranslations = Gauge32: 3066
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitMax."3x".hosts = Gauge32: 3066
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitMax."3x".ipsecSessions = Gauge32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitMax."3x".asdmSessions = Gauge32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitMax."3x".sshSessions = Gauge32: 1
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitMax."3x".telnetSessions = Gauge32: 1
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitCurrentUsage."3x".macAddresses = Gauge32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitCurrentUsage."3x".concurrentConns = Gauge32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitCurrentUsage."3x".natTranslations = Gauge32: 9
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitCurrentUsage."3x".hosts = Gauge32: 5
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitCurrentUsage."3x".ipsecSessions = Gauge32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitCurrentUsage."3x".asdmSessions = Gauge32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitCurrentUsage."3x".sshSessions = Gauge32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitCurrentUsage."3x".telnetSessions = Gauge32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitPeakUsage."3x".macAddresses = Gauge32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitPeakUsage."3x".concurrentConns = Gauge32: 8
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitPeakUsage."3x".natTranslations = Gauge32: 9
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitPeakUsage."3x".hosts = Gauge32: 5
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitPeakUsage."3x".ipsecSessions = Gauge32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitPeakUsage."3x".asdmSessions = Gauge32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitPeakUsage."3x".sshSessions = Gauge32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitPeakUsage."3x".telnetSessions = Gauge32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitReqsDeniedCount."3x".macAddresses = Counter32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitReqsDeniedCount."3x".concurrentConns = Counter32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitReqsDeniedCount."3x".natTranslations = Counter32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitReqsDeniedCount."3x".hosts = Counter32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitReqsDeniedCount."3x".ipsecSessions = Counter32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitReqsDeniedCount."3x".asdmSessions = Counter32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitReqsDeniedCount."3x".sshSessions = Counter32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitReqsDeniedCount."3x".telnetSessions = Counter32: 0

Resource usage by OIDS


.1.3.6.1.4.1.9.9.480.1.1.2.1.5.2.51.120.2 = Gauge32: 766
.1.3.6.1.4.1.9.9.480.1.1.2.1.5.2.51.120.3 = Gauge32: 11700
.1.3.6.1.4.1.9.9.480.1.1.2.1.5.2.51.120.8 = Gauge32: 3066
.1.3.6.1.4.1.9.9.480.1.1.2.1.5.2.51.120.15 = Gauge32: 3066
.1.3.6.1.4.1.9.9.480.1.1.2.1.5.2.51.120.16 = Gauge32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.5.2.51.120.17 = Gauge32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.5.2.51.120.18 = Gauge32: 1
.1.3.6.1.4.1.9.9.480.1.1.2.1.5.2.51.120.19 = Gauge32: 1
.1.3.6.1.4.1.9.9.480.1.1.2.1.8.2.51.120.2 = Gauge32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.8.2.51.120.3 = Gauge32: 2
.1.3.6.1.4.1.9.9.480.1.1.2.1.8.2.51.120.8 = Gauge32: 9
.1.3.6.1.4.1.9.9.480.1.1.2.1.8.2.51.120.15 = Gauge32: 5
.1.3.6.1.4.1.9.9.480.1.1.2.1.8.2.51.120.16 = Gauge32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.8.2.51.120.17 = Gauge32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.8.2.51.120.18 = Gauge32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.8.2.51.120.19 = Gauge32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.9.2.51.120.2 = Gauge32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.9.2.51.120.3 = Gauge32: 8
.1.3.6.1.4.1.9.9.480.1.1.2.1.9.2.51.120.8 = Gauge32: 9
.1.3.6.1.4.1.9.9.480.1.1.2.1.9.2.51.120.15 = Gauge32: 5
.1.3.6.1.4.1.9.9.480.1.1.2.1.9.2.51.120.16 = Gauge32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.9.2.51.120.17 = Gauge32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.9.2.51.120.18 = Gauge32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.9.2.51.120.19 = Gauge32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.10.2.51.120.2 = Counter32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.10.2.51.120.3 = Counter32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.10.2.51.120.8 = Counter32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.10.2.51.120.15 = Counter32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.10.2.51.120.16 = Counter32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.10.2.51.120.17 = Counter32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.10.2.51.120.18 = Counter32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.10.2.51.120.19 = Counter32: 0

if the resource class to which the firewall belongs to is changed the middle part of the OID changes eg:


CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitMax."10x".macAddresses = Gauge32: 2555
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitMax."10x".concurrentConns = Gauge32: 39000
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitMax."10x".natTranslations = Gauge32: 10220
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitMax."10x".hosts = Gauge32: 10220
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitMax."10x".ipsecSessions = Gauge32: 0
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitMax."10x".asdmSessions = Gauge32: 2
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitMax."10x".sshSessions = Gauge32: 3
CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitMax."10x".telnetSessions = Gauge32: 3

.1.3.6.1.4.1.9.9.480.1.1.2.1.5.3.49.48.120.2 = Gauge32: 2555
.1.3.6.1.4.1.9.9.480.1.1.2.1.5.3.49.48.120.3 = Gauge32: 39000
.1.3.6.1.4.1.9.9.480.1.1.2.1.5.3.49.48.120.8 = Gauge32: 10220
.1.3.6.1.4.1.9.9.480.1.1.2.1.5.3.49.48.120.15 = Gauge32: 10220
.1.3.6.1.4.1.9.9.480.1.1.2.1.5.3.49.48.120.16 = Gauge32: 0
.1.3.6.1.4.1.9.9.480.1.1.2.1.5.3.49.48.120.17 = Gauge32: 2
.1.3.6.1.4.1.9.9.480.1.1.2.1.5.3.49.48.120.18 = Gauge32: 3
.1.3.6.1.4.1.9.9.480.1.1.2.1.5.3.49.48.120.19 = Gauge32: 3

QUERY:

<interface>
<name>ECME Resource Usage</name>
<oid_index>.1.3.6.1.4.1.9.9.480.1.1.2.1.3</oid_index>
<index_title_format>|chosen_order_field|</index_title_format>
<fields>
<crlResourceLimitIndex>
<name>Index</name>
<method>walk</method>
<source>OID/REGEXP:.*\"\.(.*)</source>
<direction>input</direction>
<oid>.1.3.6.1.4.1.9.9.480.1.1.2.1.3</oid>
</crlResourceLimitIndex>

<Max>
<name>Maximum Connections Rate</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.9.9.480.1.1.2.1.5</oid>
</Max>
<Current>
<name>Current Connections Rate</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.9.9.480.1.1.2.1.8</oid>
</Current>
<Peak>
<name>Peak Connections Rate</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.9.9.480.1.1.2.1.9</oid>
</Peak>
<Denied>
<name>Denied Connections Rate</name>
<method>walk</method>
<source>value</source>
<direction>output</direction>
<oid>.1.3.6.1.4.1.9.9.480.1.1.2.1.10</oid>
</Denied>

</fields>
</interface>

Poller Cache:
Firewall telnetSessions SNMP Version: 2, Community: public, OID: .1.3.6.1.4.1.9.9.480.1.1.2.1.8.1
RRD: /local/data2/cacti/rra/firewall_peak_65010.rrd
Firewall telnetSessions SNMP Version: 2, Community: public, OID: .1.3.6.1.4.1.9.9.480.1.1.2.1.10.1
RRD: /local/data2/cacti/rra/firewall_peak_65010.rrd
Firewall telnetSessions SNMP Version: 2, Community: public, OID: .1.3.6.1.4.1.9.9.480.1.1.2.1.5.1
RRD: /local/data2/cacti/rra/firewall_peak_65010.rrd
Firewall telnetSessions SNMP Version: 2, Community: public, OID: .1.3.6.1.4.1.9.9.480.1.1.2.1.9.1
RRD: /local/data2/cacti/rra/firewall_peak_65010.rrd

Data Query Debug Information
+ Running data query [19].
+ Found type = '3' [SNMP Query].
+ Found data query XML file at '/local/data2/cacti/resource/snmp_queries/ecme_crlResourceLimit.xml'
+ XML file parsed ok.
+ <oid_num_indexes> missing in XML file, 'Index Count Changed' emulated by counting oid_index entries
+ Executing SNMP walk for list of indexes @ '.1.3.6.1.4.1.9.9.480.1.1.2.1.3' Index Count: 8
+ Index found at OID: 'CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitValueType."3x".macAddresses' value: 'absolute(1)'
+ Index found at OID: 'CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitValueType."3x".concurrentConns' value: 'absolute(1)'
+ Index found at OID: 'CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitValueType."3x".natTranslations' value: 'absolute(1)'
+ Index found at OID: 'CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitValueType."3x".hosts' value: 'absolute(1)'
+ Index found at OID: 'CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitValueType."3x".ipsecSessions' value: 'absolute(1)'
+ Index found at OID: 'CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitValueType."3x".asdmSessions' value: 'absolute(1)'
+ Index found at OID: 'CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitValueType."3x".sshSessions' value: 'absolute(1)'
+ Index found at OID: 'CISCO-L4L7MODULE-RESOURCE-LIMIT-MIB::crlResourceLimitValueType."3x".telnetSessions' value: 'absolute(1)'
+ Located input field 'crlResourceLimitIndex' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.4.1.9.9.480.1.1.2.1.3'
+ Found item [crlResourceLimitIndex='macAddresses'] index: 1 [from regexp oid parse]
+ Found item [crlResourceLimitIndex='concurrentConns'] index: 1 [from regexp oid parse]
+ Found item [crlResourceLimitIndex='natTranslations'] index: 1 [from regexp oid parse]
+ Found item [crlResourceLimitIndex='hosts'] index: 1 [from regexp oid parse]
+ Found item [crlResourceLimitIndex='ipsecSessions'] index: 1 [from regexp oid parse]
+ Found item [crlResourceLimitIndex='asdmSessions'] index: 1 [from regexp oid parse]
+ Found item [crlResourceLimitIndex='sshSessions'] index: 1 [from regexp oid parse]
+ Found item [crlResourceLimitIndex='telnetSessions'] index: 1 [from regexp oid parse]

[sbindl01]

I think I got it. Scouring the forum I did find others that were trying to solve the same thing I was (particularly Alcatel 7x50 SR router QoS queue counters) and applying the concepts in the data query documentation was able to create an XML, data source, graph template etc.

I did have one other question for the developers. I am grabbing roughly 32 counters (16 in profile queues and 16 out of profile queues) for a particular ifIndex.
I see in the poller cache there is a poller cache item for each individual output field (so 32 per router interface). What effect will this have on performance when i add say 5000 routers and several links per router? So for say 2 links per router that will be 320,000 poller cache items (but still just 5000 rrd files). That's just for the ingress counters, It will add another 320k when i build the egress counter xml and apply.

I guess I'll find out when I get this all built

-SB

[sbindl01]

Well...i answered my scaling question.....it broke (poller_output table not empty errors in the logs for every polling cycle)

However...i installed and configured boost plugin!
09/04/2012 10:00:40 AM - SYSTEM STATS: Time:38.4741 Method:spine Processes:16 Threads:25 Hosts:3222 HostsPerProcess:202 DataSources:309011 RRDsProcessed:0


booyah!

[gandalf]

Wow, 300k data sources is a "significant" amount ...
R.