[HowTo] use the index at an arbitrary place in the OID field

[cbscpe]

Hi,

First thanks for this tutorial. And now to my question. I have here the case that I have a table I want to read via SNMP which has several columns and also contains history information I'm not interested in but the last intervall. However the ifIindex is not at the end of the OID I'm interested in. The table is the "hdsl2Shdsl15MinIntervalTable" .1.3.6.1.2.1.10.48.1.6 and I'm interested in the values of ES .1.3.6.1.2.1.10.48.1.6.1.2, SES .1.3.6.1.2.1.10.48.1.6.1.3, etc. But the OID I'm interested in are at

.1.3.6.1.2.1.10.48.1.6.1.[2-6].ifIndex.1.2.1.1

when looking at the definitions of a field I did not find any hint how ifIndex (or any other value matched at query->oid_index_parse) can be placed elsewhere than the end of the OID of the field (query->fields->ifIndex->oid). I do not want to do a walk for

.1.3.6.1.2.1.10.48.1.6.1.[2-6].ifIndex

as in most cases this is huge amount of data as the following indexes (1.2.1.1) have a lot of possible values (built by the InvIndex which depending on the installation can have the values 1 to 10, the EndPointSide, which has the values 1 and 2, EndpointWirePair which can be 1,2,3,4 and 15MinIntervalNumber which ranges typically from 1 to 32 and for some devices 1 to 96) so I'm interested only in about 1% of the available table. (The table is defined in RFC 4319)

Regards

Peter

[gandalf]

Some remarks, no solution:

Please find current documentation at first link of my sig (scroll down to the SNMP Data Query chapter).

It is possible to use a "multi-dimensional" index, e.g. ifIndex.1.2.1.1 by using <oid_index_parse> with OID/REGEXP. This table is scanned only when re-indexing, so perhaps it's affordable to scan the whole table.

As a last resort, you may create a Script Data Query to avoid overhead. Find documentation of this at the link from my sig.

There have been feature requests for adding constant parts of an OID behind the appended index. There's a user contrib in the Feature Request forum that may be incorporated into next cacti version

Reinhard

[cbscpe]

Thanks for your reply. I hope the required change makes it to the next release My intention is to poll several thousand SHDSL links, I doubt a script is a good solution. A solution using SNMP directly (and as far as I understand the only chance to let SPINE do it) would certainly help keep the CPU load. I'll have a try with the script.


Peter

[gandalf]

My intention is to poll several thousand SHDSL links, I doubt a script is a good solution.

Agreed. So I'd try it the pure SNMP way using multi-dimensional indices. Is a matter of poking around a bit to get an idea of the overhead. I'm quite sure it is more acceptable than scripting.
Last question. When snmpwalking that table, does the device suffer from a inacceptable CPU load? That would be quite bad, then.
Reinhard

[cbscpe]

Yes you are right. I'll try both, the script and your proposal and I will have a look at the load of the cacti server and of the SHDSL device.

The devices themselves are not suffering from snmpwalking these tables. There is no impact on traffic through these devices.

[gandalf]

I'd prefer the pure SNMP way, then. Using snmpbulkwalk (= SNMP V2c at least) will reduce traffic impact to a great extend.
Reinhard

[cbscpe]

First a question to snmpbulkwalk, does cacti (poller or spine) use snmpbulkwalk to index the device? And yes I will definitively go the pure SNMP way for the Pardayne DSLAMs to start.

But now back to my SHDSL Table. I tried to set up everthing as explained in this tutorial, but now I'm stuck. When I associate the Graph Template to the Data Query I expected that I can select the fields in the XML with the drop-down lists right to the Data Source: <internal data source name>. Only the drop-downs are empty. Now I suppose I made an error in my XML. But I expect some error messages. Also I'm wondering why in the documentation the in Chapter 12.2. it tells me to use the <query> tag and when looking at the interface.xml sample it says <interface>. What did I miss?

My XML looks like.

Code: Select all

<query>
	<name>SHDSL Line Statistics</name>
	<description>Query the SHDSL Counters ES, SES, CRC, LOSWS, UAS</description>
	<oid_index>.1.3.6.1.2.1.10.48.1.6.1.2</oid_index>
	<oid_index_parse>OID/REGEXP:.*\.(10010[0-9][0-9].\1\.2\.1\.1)$</oid_index_parse>

	<fields>
		<hdsl2Shdsl15MinIntervalES>
			<name>Error Seconds</name>
			<method>get</method>
			<source>value</source>
			<direction>output</direction>
			<oid>.1.3.6.1.2.1.10.48.1.6.1.2</oid>
		</hdsl2Shdsl15MinIntervalES>
		<hdsl2Shdsl15MinIntervalSES>
			<name>Severly Error Seconds</name>
			<method>get</method>
			<source>value</source>
			<direction>output</direction>
			<oid>.1.3.6.1.2.1.10.48.1.6.1.3</oid>
		</hdsl2Shdsl15MinIntervalSES>
		<hdsl2Shdsl15MinIntervalCRC>
			<name>Checksum Errors</name>
			<method>get</method>
			<source>value</source>
			<direction>output</direction>
			<oid>.1.3.6.1.2.1.10.48.1.6.1.4</oid>
		</hdsl2Shdsl15MinIntervalCRC>
		<hdsl2Shdsl15MinIntervalLOSWS>
			<name>Loss of Sync Word Seconds</name>
			<method>get</method>
			<source>value</source>
			<direction>output</direction>
			<oid>.1.3.6.1.2.1.10.48.1.6.1.5</oid>
		</hdsl2Shdsl15MinIntervalLOSWS>
		<hdsl2Shdsl15MinIntervalUAS>
			<name>Unavailable Seconds</name>
			<method>get</method>
			<source>value</source>
			<direction>output</direction>
			<oid>.1.3.6.1.2.1.10.48.1.6.1.6</oid>
		</hdsl2Shdsl15MinIntervalUAS>
	</fields>
</query>

Also some claryfing question. Now as I'm using not only the ifIndex but also the other indices of the table to get my oid_index_parse, I cannot use this index to look up values in another table than the table I'm working on as other tables do not have the same indexing, e.g. I can't use ifName, ifDescr and/or ifAlias any more.

[cbscpe]

Found the error why the XML did not parse correctly. However cacti tells me now Success [0 Items, 0 Rows]. When I'm doing a snmpbulkwalk by hand he returns the table as expected and doing a grep with oid_index_parse returns exactly the indexes required. What further debugging can I use?

Regards

Peter

[cbscpe]

Now my XML file looks like (the REGEXP was wrong, ].\1 instead of ]\.1 .)

Code: Select all

<interface>
        <name>SHDSL Line Statistics</name>
        <description>Query the SHDSL Counters ES, SES, CRC, LOSWS, UAS</description>
        <oid_index>.1.3.6.1.2.1.10.48.1.6.1.2</oid_index>
        <oid_index_parse>OID/REGEXP:.*\.(10010[0-9][0-9]\.1\.2\.1\.1)$</oid_index_parse>
        <index_order>tableIndex</index_order>
        <index_order_type>numeric</index_order_type>
        <fields>
                <tableIndex>
                        <name>Table Indexes</name>
                        <method>walk</method>
                        <source>OID/REGEXP:.*\.(10010[0-9][0-9]\.1\.2\.1\.1)$</source>
                        <direction>input</direction>
                        <oid>.1.3.6.1.2.1.10.48.1.6.1.2</oid>
                </tableIndex>
                <hdsl2Shdsl15MinIntervalES>
                        <name>Error Seconds</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.2.1.10.48.1.6.1.2</oid>
                </hdsl2Shdsl15MinIntervalES>
                <hdsl2Shdsl15MinIntervalSES>
                        <name>Severly Error Seconds</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.2.1.10.48.1.6.1.3</oid>
                </hdsl2Shdsl15MinIntervalSES>
                <hdsl2Shdsl15MinIntervalCRC>
                        <name>Checksum Errors</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.2.1.10.48.1.6.1.4</oid>
                </hdsl2Shdsl15MinIntervalCRC>
                <hdsl2Shdsl15MinIntervalLOSWS>
                        <name>Loss of Sync Word Seconds</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.2.1.10.48.1.6.1.5</oid>
                </hdsl2Shdsl15MinIntervalLOSWS>
                <hdsl2Shdsl15MinIntervalUAS>
                        <name>Unavailable Seconds</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.2.1.10.48.1.6.1.6</oid>
                </hdsl2Shdsl15MinIntervalUAS>
        </fields>
</interface>

I had to add an input field so cacti did really something. Reading the documentation for me it seemed input fields are really optional. However strange things happen. Now I get a "Success [3104 Items, 3104 Rows]". Due to the REGEXP, i expected that only rows with 1.2.1.1 at the end of the OID would make it to the index. Like when I do a

• [root@lpczur010 snmp_queries]# snmpbulkwalk -c public -v 2c ldszur002 .1.3.6.1.2.1.10.48.1.6.1.2 | grep 10010[0-9][0-9]\.1\.2\.1\.1\
  SNMPv2-SMI::transmission.48.1.6.1.2.1001001.1.2.1.1 = Gauge32: 0
  SNMPv2-SMI::transmission.48.1.6.1.2.1001002.1.2.1.1 = Gauge32: 0
  SNMPv2-SMI::transmission.48.1.6.1.2.1001003.1.2.1.1 = Gauge32: 0
  SNMPv2-SMI::transmission.48.1.6.1.2.1001005.1.2.1.1 = Gauge32: 0
  SNMPv2-SMI::transmission.48.1.6.1.2.1001007.1.2.1.1 = Gauge32: 0
  SNMPv2-SMI::transmission.48.1.6.1.2.1001013.1.2.1.1 = Gauge32: 0
  SNMPv2-SMI::transmission.48.1.6.1.2.1001015.1.2.1.1 = Gauge32: 0
  SNMPv2-SMI::transmission.48.1.6.1.2.1001016.1.2.1.1 = Gauge32: 0
  SNMPv2-SMI::transmission.48.1.6.1.2.1001017.1.2.1.1 = Gauge32: 0
  SNMPv2-SMI::transmission.48.1.6.1.2.1001019.1.2.1.1 = Gauge32: 0
  SNMPv2-SMI::transmission.48.1.6.1.2.1001020.1.2.1.1 = Gauge32: 0
  [root@lpczur010 snmp_queries]#

I have read Chapter 12.2 several times. And I did my best to catch the meaning of the english text. But now I'm really confused.

Instead the debugging shows me a lot of indexes matched I did not expect.

• + Running data query [11].
  + Found type = '3' [snmp query].
  + Found data query XML file at '/var/www/html/cacti/resource/snmp_queries/shdsl-statistcs-tableIndexes.xml'
  + XML file parsed ok.
  + Executing SNMP walk for list of indexes @ '.1.3.6.1.2.1.10.48.1.6.1.2'
  + Located input field 'tableIndex' [walk]
  + Executing SNMP walk for data @ '.1.3.6.1.2.1.10.48.1.6.1.2'
  + Found item [tableIndex='1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.0 [from regexp oid parse]
  + Found item [tableIndex='1001001.1.2.1.1'] index: 1001001.1.2.1.1 [from regexp oid parse]
  + Found item [tableIndex='1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.2'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.2 [from regexp oid parse]
  + Found item [tableIndex='1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.3'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.3 [from regexp oid parse]
  + Found item [tableIndex='1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.4'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.4 [from regexp oid parse]
  + Found item [tableIndex='1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.5'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.5 [from regexp oid parse]
  + Found item [tableIndex='1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.6'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.6 [from regexp oid parse]
  + Found item [tableIndex='1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.7'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.7 [from regexp oid parse]
  + Found item [tableIndex='1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.8'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.8 [from regexp oid parse]
  + Found item [tableIndex='1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.9'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.9 [from regexp oid parse]
  + Found item [tableIndex='1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.10'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.10 [from regexp oid parse]
  + Found item [tableIndex='1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.11'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.11 [from regexp oid parse]

I have no clue of what is going wrong now. Any advice? Thanks in advance

Peter

[gandalf]

Code: Select all

<query>
	<name>SHDSL Line Statistics</name>
	<description>Query the SHDSL Counters ES, SES, CRC, LOSWS, UAS</description>
	<oid_index>.1.3.6.1.2.1.10.48.1.6.1.2</oid_index>
	<oid_index_parse>OID/REGEXP:.*\.(10010[0-9][0-9].\1\.2\.1\.1)$</oid_index_parse>

I'm quite sure, that the second \ is mis-placed. Should read (10010[0-9][0-9]\.1\.2\.1\.1) instead.
In general, in case an error occurs, please return to the starting point and lets go forward step-by-step. Cacti does not much of an error checking for SNMP Queries. And you won't see any output apart from "Verbose Query" to debug your XML.
Sometimes, importing the XML to some XML application (e.g. a modern browser) will show XML syntax errors that would otherwise not be found that easy
Reinhard

[gandalf]

Code: Select all

                <tableIndex>
                        <name>Table Indexes</name>
                        <method>walk</method>
                        <source>OID/REGEXP:.*\.(10010[0-9][0-9]\.1\.2\.1\.1)$</source>
                        <direction>input</direction>
                        <oid>.1.3.6.1.2.1.10.48.1.6.1.2</oid>
                </tableIndex>

Please replace the <source>OID/REGEXP by <source>value. The index will alreday be determined by the <oid_index_parse>
Reinhard

[gandalf]

Split off due to the increasing size of this discussion
Reinhard

[cbscpe]

Sure the second backslash was at the wrong place. Alghoutg grep returns the same matches in both case, but this only by coincidence of the OID structure and because . (any character) matches also the real . and \1 matches 1, don't know whether ereg_replace does the same, but the corrected regexp does not give the expected result.

Now I changed <source> to value and still get all the rows Success [3104 Items, 3104 Rows].

• + Running data query [11].
  + Found type = '3' [snmp query].
  + Found data query XML file at '/var/www/html/cacti/resource/snmp_queries/shdsl-statistcs.xml'
  + XML file parsed ok.
  + Executing SNMP walk for list of indexes @ '.1.3.6.1.2.1.10.48.1.6.1.2'
  + Located input field 'tableIndex' [walk]
  + Executing SNMP walk for data @ '.1.3.6.1.2.1.10.48.1.6.1.2'
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.0 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.1 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.2 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.3 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.4 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.5 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.6 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.7 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.8 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.9 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.10 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.11 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.12 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.13 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.14 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.15 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.16 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.17 [from value]

(list truncated)

I expected to have only the ones from my snmpbulkwalk with the grep from the shell. Otherwise the selection of the indexes to graph will be too long and it would display almost 100times more entry to create a graph from.

I changed also the oid in the field <tableIndex> to .1.3.6.1.2.1.2.2.1.1, then I get at least only the interfaces and not all rows of the SHDSL table, but I doubt this will work.

Still, why do I require a input field? They have no use, especially as the indexes matched be oid_parse_index are not usefull anyhow, as they cannot be used to extract e.g. ifName or ifDescr to be put on the graphs automatically.

I expect that oid_index_parse will only select the indexes that match against the regular expression. Is that right?

Also you said I should go back to the starting point. Typically when changing XML what I do is delete the data query from the device and then I delete associated graph from the data query. Or should I even go back one step further?

Regards

Peter

[cbscpe]

One more question. You are using ereg_replace in lib/data_query.php which does the following according the PHP doc.

This function scans string for matches to pattern, then replaces the matched text with replacement.

The modified string is returned. (Which may mean that the original string is returned if there are no matches to be replaced.)

The sentence in paranthesis makes my worry, whether my expression really filters anything as I expected a OID/REGEXP should do.

Regards

Peter

[gandalf]

Also you said I should go back to the starting point. Typically when changing XML what I do is delete the data query from the device and then I delete associated graph from the data query. Or should I even go back one step further?

We should first get the index(_parse) stuff right. I do not worry about other fields as long as the index is correct (which it is not ATM).
In terms of the docs, this part is covered by http://docs.cacti.net/node/226. Currently the index reported by

• + Running data query [11].
  + Found type = '3' [snmp query].
  + Found data query XML file at '/var/www/html/cacti/resource/snmp_queries/shdsl-statistcs.xml'
  + XML file parsed ok.
  + Executing SNMP walk for list of indexes @ '.1.3.6.1.2.1.10.48.1.6.1.2'
  + Located input field 'tableIndex' [walk]
  + Executing SNMP walk for data @ '.1.3.6.1.2.1.10.48.1.6.1.2'
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.0 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.1 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.2 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.3 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.4 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.5 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.6 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.7 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.8 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.9 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.10 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.11 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.12 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.13 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.14 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.15 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.16 [from value]
  + Found item [tableIndex='0'] index: 1.3.6.1.2.1.10.48.1.6.1.2.1001001.1.2.1.17 [from value]

is still wrong. It should report 1001001.1.2.1.0 and the like. So let's stick to this bit. Perhaps I got my instructions wrong, so please post XML along with Verbose Query output as long as we are struggeling with this case.
To be honest, multi-dimensional indices have always been kind of a hazzle and I always have to remember each and every time.
I'll try to google for the last time I tried to help ...
Reinhard