Data Query almost works

jjhans · Post by **jjhans** » Thu Jan 03, 2008 11:06 am

I'm trying to build a data query to read the counters from Ricoh Aficio printers (number of pages printed and such).

I created the graph template, the data template, the XML file, and the data query. On the data query, I set up the associated graph templates. Up to here, everything seems to work. The XML file seems to be read without problems, and the pop-up menu of input fields is populated correctly. When I run the query on a host, that works too. But when I create the graphs, I run into problems.

The first sign of a problem is the title of the created graph and data source. The data query Suggested Value is set to "host_description| - |query_counterName|".

What I actually get as the title is "Ricoh 4500 - |query_counterName|". So the host variable is getting filled in, but not the query variable.

In addition, when I open up the created data query, it shows the message, "Data query data sources must be created through New Graphs." Which, of course, this one was.

By the way, I'm running cacti 0.8.6j; I suppose I should try to upgrade before asking questions here, but I'm hoping this is relatively straightforward and not a bug in this version. (Or, if this is a known bug, finding that out would be helpful too!)

Any thoughts? The XML file is below.

PS: in the built-in snmp query xml files, some have <query> as the top-level element, while others have <interface>. I can't find anything that says why. Are these equivalent? I tried both in my file, without success.

Code: Select all

<query>
        <name>Caprio - Ricoh Counters</name>
        <description>Queries a Ricoh printer</description>
        <oid_index>.1.3.6.1.4.1.367.3.2.1.2.19.5.1.2</oid_index>
        
        <index_order>counterName:counterIndex</index_order>
        <index_order_type>numeric</index_order_type>
        <index_title_format>|chosen_order_field|</index_title_format>

        <fields>
                <counterIndex>
                        <name>Index</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>input</direction>
                        <oid>.1.3.6.1.4.1.367.3.2.1.2.19.5.1.2</oid>

                </counterIndex>
                <counterName>
                        <name>Counter Name</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>input</direction>
                        <oid>.1.3.6.1.4.1.367.3.2.1.2.19.5.1.5</oid>
                </counterName>
                <counterValue>
                        <name>Counter Value</name>
                        <method>walk</method>
                        <source>value</source>
                        <direction>output</direction>
                        <oid>.1.3.6.1.4.1.367.3.2.1.2.19.5.1.9</oid>
                </counterValue>


        </fields>
</query>

Post by **gandalf** » Thu Jan 03, 2008 1:23 pm

Please post the result of a walk against the counterName OID. Unfortunetaly, there has been a related bug that meanwhile has been fixed.
Reinhard

jjhans · Post by **jjhans** » Thu Jan 03, 2008 1:37 pm

The counterName OID looks like this:

Code: Select all

1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.1	Counter: Machine Total
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.2	Counter:Copy:Total
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.3	Counter:Copy:Black & White
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.4	Counter:Copy:Single/Two-color
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.5	Counter:Copy:Full Color
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.6	Counter:FAX:Total
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.7	Counter:FAX:Black & White
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.8	Counter:Print:Total
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.9	Counter:Print:Black & White
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.10	Counter:Print:Single/Two-col.
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.11	Counter:Print:Full Color
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.12	Counter: Machine Total
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.13	Total Prints: Full Color
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.14	Total Prints: Monocolor
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.15	Development: Color
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.16	Development: Black & White
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.17	Copier: Color
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.18	Copier: Black & White
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.19	Printer: Color
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.20	Printer: Black & White
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.21	Total Prints: Color
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.22	Total Prints: Black & White
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.23	Printer: Full Color
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.24	Printer: Monocolor
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.25	Total Prints: Full Color (GPC)
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.26	Copier: Black & White
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.27	Copier: Single Color
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.28	Copier: Two-color
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.29	Copier: Full Color
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.30	Fax: Black & White
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.31	Fax: Single Color
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.32	Printer: Black & White
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.33	Printer: 1 or 2 Clr. Toner(s)
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.34	Printer: Full Color
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.35	From Storage: Black & White
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.36	From Storage: Single Color
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.37	From Storage: Two-color
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.38	From Storage: Full Color
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.39	A3,DLT
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.40	No. of Printed Sides in Duplex
1.3.6.1.4.1.367.3.2.1.2.19.5.1.5.41	Number of Staples Used

For what it's worth, the counterIndex OID is a small integer associated with each counter, and naturally counterValue is the (monotonically increasing) value I'm trying to capture.

Incidentally, I notice now that the first and twelfth entries have the same name, and also the same index... and for that matter, the same value, as they both report the same data, the total number of pages printed. (Because they both report the same value, it won't be a problem if cacti gets the two confused with each other.) Could this cause the problem? Will I need to use the OID as an index as well as (or instead of) the others?

Post by **gandalf** » Thu Jan 03, 2008 3:39 pm

Thank you for posting this. There has been a problem with index values holding spaces, just as in your case. This error has been resolved but no patch has been published yet (it "should" be in 087 SVN, if I remember correctly). And I fear, that even a colon will be stripped from the values. Is it possible to change those counterNames, at least for a single one, just as a try?
The issue with the index occuring twice is bad. Please post a walk for the index as well. But you may just derive the index from the name field as well to avoid any problem on this side.
Reinhard

jjhans · Post by **jjhans** » Thu Jan 03, 2008 6:33 pm

Solved! After a few more false starts, I added the following line (which is apparently undocumented but can be found in the win32_procs.xml file) into my xml file:

Code: Select all

<index_type>nonunique</index_type>

Now, it works! So far, at least... I'm not sure what exactly that line does, so I don't know what the ramifications will be.

So the colons and spaces don't seem to be causing any problem at all... perhaps because they're only in one of two columns that are used to index?

There's still one small problem, though... the |query_counterName| is working, but it's truncating it to the first 15 characters. So something like "Total Prints: Black & White" is turning into "Total Prints: B".

Anyway, thanks for the help!

BSOD2600 · Post by **BSOD2600** » Thu Jan 03, 2008 7:19 pm

Heh, thats my script you found that in. TheWitness told me about that "nonunique" variable a long time ago. Yes, it should be documented, gandalf

.

Post by **gandalf** » Fri Jan 04, 2008 8:58 am

BSOD2600 wrote:Heh, thats my script you found that in. TheWitness told me about that "nonunique" variable a long time ago. Yes, it should be documented, gandalf .

Fine, and what the hell, apart from the obvious, does this option do? Allow for non-unique indices, that's all?
Reinhard

jjhans · Post by **jjhans** » Fri Jan 04, 2008 9:53 am

It appears that if a duplicate index is encountered WITHOUT the "nonunique" setting, all attempts to create a data source from the query will fail silently. If a duplicate index is encountered when "nonunique" is enabled, it will succeed, and if you try to graph one of the items with a duplicate index, I imagine the software might pick any of the same-indexed items. (In my case, where the same data appears twice and has the same index in both cases, this behavior is perfect.)

It seems to me that the default behavior of the software should be to allow non-unique indexes, but to warn the user if the query finds one, making the <nonunique> tag obsolete.

On the other hand, users can always use scripts instead of direct SNMP queries whenever the results from the query aren't completely clean and straightforward, and cacti seems to be moving in that direction, increasing the performance and ease of adding scripts. Or at least, that seems to have been the dev team's opinion as of 2005: http://bugs.cacti.net/view.php?id=363

Post by **gandalf** » Fri Jan 04, 2008 3:51 pm

jjhans wrote:... I imagine the software might pick any of the same-indexed items. (In my case, where the same data appears twice and has the same index in both cases, this behavior is perfect.)

Yes, that's the question. Which of the nin-unique indices is taken? Is it safe? As I do not have a device of this kind, I cam't verify on my own.
I'm willing to document this feature. But I have to know this first
Reinhard

BSOD2600 · Post by **BSOD2600** » Fri Jan 04, 2008 3:55 pm

gandalf wrote:Fine, and what the hell, apart from the obvious, does this option do? Allow for non-unique indices, that's all?
Reinhard

Looking through my chat logs with him, here are some excerpts:

5/24/2005
Me: so you know a fix for my reindexing problem on the win32 monitor running process script ?

TheWitness: yes, a new tag. Check CVS. However, you need to force the re-index to work off of process name and not PID. The nonunique <index_type> will allow that

Me: so if this is cvs code... then it wouldn't work for people w/o the upgrade

TheWitness: if you pick it without the <index_type> it will default to PID since there will always be more than one "svchost" process.3

TheWitness: nonunique tells Cacti (with my patch) that it can ignore non unique process names in determining the correct OID to use for calculating uniqueness

TheWitness: we have to get smarter though

Me: I assume older cacti versions will just ignore that field ?

TheWitness: right

TheWitness: I think anyway

jjhans · Post by **jjhans** » Fri Jan 04, 2008 5:57 pm

gandalf wrote:Yes, that's the question. Which of the nin-unique indices is taken? Is it safe? As I do not have a device of this kind, I cam't verify on my own.
I'm willing to document this feature. But I have to know this first
Reinhard

It seems safe to say in the documentation that which one it chooses is undefined, even if it happens to be consistent; that's kind of the nature of a non-unique index, right? Consider: In MySQL, if you "SELECT * FROM some_table WHERE some_column = some_value LIMIT 1", and there are multiple rows that match, then I'm pretty sure the database reserves the right to return whichever one it damn well pleases; the rows are considered unordered. Of course, OIDs are not unordered, so it might be more reasonable here to say it always picks the lowest/highest/whatever one.

I'll try to glance at the code on monday and see if I can figure out exactly what's going on here. In the meantime, here's my experience. My data query, when run, looked approximately like this:

Code: Select all

10	Counter: Machine Total	
200	Counter:Copy:Total	
201	Counter:Copy:Black & White	
202	Counter:Copy:Single/Two-color	
203	Counter:Copy:Full Color	
300	Counter:FAX:Total	
301	Counter:FAX:Black & White	
10	Counter: Machine Total	
600	Total Prints: Full Color	
601	Total Prints: Monocolor	
602	Development: Color
...

As you can see, two entries have the number 10 and the name "Counter: Machine Total." I hit the check box next to the first one, and told it to create the graph. When the page reloaded, the second one was grayed out, indicating that a graph had been created, but not the first. Both queries return the same value, so for all I know and care it could choose a different one each time it re-indexes, or even each time it polls.

It seems safe to say that if I needed it to pick one or the other, I should be using the OID as the index, or adding my own logic with a script, instead of simply using nonunique indexes. Which is why I think it's safe to say that the precise behavior is undefined -- if I care which one it chooses, I shouldn't be using it.

kanada · Post by **kanada** » Fri Apr 25, 2008 7:33 am

Hello,

jjhans can you upload this templates?

Regards,
Alex.

crashergs · Post by **crashergs** » Fri Jul 11, 2008 1:59 pm

Yes,

it would be very nice if you can provide your work to the community?

marekmust · Post by **marekmust** » Tue Oct 13, 2009 3:14 am

hello, is there any chance you can upload youre templates

Mika2008 · Post by **Mika2008** » Thu Dec 03, 2009 4:39 am

Up hwere i can download this template please?

Data Query almost works

Data Query almost works

Who is online