Understanding Data Query Question

[denis.king]

Cacti version 0.8.7a

Background:
I have a device that does not use SNMP, but HTTP req/reply instead. The device has configured users that are mostly static, but do get removed and added from time to time. For each user, there are ~10 different stats that I am interested in graphing within cacti.

Script:
I have written a perl script that does the HTTP req/reply, and prints the correct output for cacti putposes. My question is actually "what is the correct output"

After reviewing the host_cpu.php/.xml example and reading http://docs.cacti.net/node/225
, my understanding was that there are 3 main input argument for the script and each of them should return data in a different format.
"Index" -> My script will print all the configured users on the device, each on a new line.
Query |desiredStat| -> My script will retun all the users (effectively indexes) with ":" as the delimeter and only the desired stat. So if there are 5 configured users, my script will output 5 lines, with each line having user1:desiredStatInfo. (this is where my confusion is...see bleow)
get |index| |desiredStat| -> My script will return a single value given the user name and the desired stat name.

I thought I was on the right track until I loaded the XML file for my target device. Within cacti I loaded the data query for my device and got the following from the verbose query, which confused me:

+ Running data query [15].
+ Found type = '4 '[script query].
+ Found data query XML file at 'C:/Inetpub/wwwroot/cacti/resource/script_queries/subscriber_stats.xml'
+ XML file parsed ok.
+ Executing script for list of indexes 'C:\Inetpub\wwwroot\cacti\scripts\Solace\ConfiguredSubscribers.pl 192.168.130.50 index'
+ Executing script query 'C:\Inetpub\wwwroot\cacti\scripts\Solace\ConfiguredSubscribers.pl 192.168.130.50 query index'
+ Found data query XML file at 'C:/Inetpub/wwwroot/cacti/resource/script_queries/subscriber_stats.xml'
+ Found data query XML file at 'C:/Inetpub/wwwroot/cacti/resource/script_queries/subscriber_stats.xml'
+ Found data query XML file at 'C:/Inetpub/wwwroot/cacti/resource/script_queries/subscriber_stats.xml'


Questions:
1-From reading this can I assume that cacti is running my script using the input arguments of "query |index|", where index is one of my configured users, that it received from the first script call that just used "index"?
2-If 1 is true, then my understanding from the examples that I have read is incorrect. Moreover, if each of my indexes has many stat values associated with it, then should the output be index:stat1:stat2:...statn?
3-Is the number of "Found data query XML file" in the above verbose output a representation of the number of outputs from the script?

Hope this makes sense.

Thanks,
DK

[denis.king]

To further define where my confusion is coming from.

I see now that cacti is indeed running the script using the input args "query index".

If I look at the php script that is being used in the example query_host_cpu.php, the following code is used to handle these arguements:

for ($i=0;($i<sizeof($arr_index));$i++) {
if ($arg == "usage") {
print $arr_index[$i] . "!" . $arr[$i] . "\n";
}elseif ($arg == "index") {
print $arr_index[$i] . "!" . $arr_index[$i] . "\n"; }
}


The bolded code above which is executed when the input args is query index prints the same thing twice......unless I am missing something. If I can understand this line of code, I can understand what my script should output when the input args is "query index".

I added my XML file if anyone is interested.

Thanks

[BSOD2600]

1) wrong forum moving.

2) This is a complicated subject, but do you realize there are two methods of accomplishing your goal? One uses a Data Query and the other is a Data Input Method. Data queries are more dynamic and can show data inside the device's screen (like the cpu/disk space templates). Data Input Method is basically a script that gets executed and then returns the data to cacti. The latter is obviously more easy since it requires less programming, but isn't quite as user friendly for GUI representation of the data.

Which do you want yours to be?

[denis.king]

I have setup several data input methods scripts thus far that poll my device and output all the data that I need. I sm no expert but I understand this approach and it is working fine for what I need.

data input methods are good if you have a known stat or several stats that you can poll and graph.

The problem that I am trying to solve is different (at least I think so). I can't use a data input method to poll a stat if I am not even sure if the stat even exists first. Therefore, I want to first poll the device for a list of connected users. With this list, then I know what stats I need as they are associated with each connected user. Therefore, I thought I could find out what users are connected as and indexe much like I see other query's finding out what disk partititons exist or how many CPUs exits, etc, etc.
From this index, I can then have the associated stats for each index (effecitvely a user) polled.

I can certainly understand that data query's are more complex, but I am not convinced that the data input method helps my cause here.
Perhaps I am being over ambitious using cacti, but I remain optimistic.

does this make sense?

[BSOD2600]

I can't use a data input method to poll a stat if I am not even sure if the stat even exists first. Therefore, I want to first poll the device for a list of connected users. With this list, then I know what stats I need as they are associated with each connected user. Therefore, I thought I could find out what users are connected as and indexe much like I see other query's finding out what disk partititons exist or how many CPUs exits, etc, etc

Gotcha.

Alright, so the first thing I noticed about your xml file is that it doesn't quite follow the existing cpu/disk templates. For example the ones which live in cacti\resource\script_queries. These are the scripts cacti use to use before switching over to the PHP script server (cacti\resource\script_server).

I'd suggest you play around with the query_host_partitions.php or query_host_cpu.php scripts and how the output data, to learn how to make yours work properly. For example:

Code: Select all

C:\inetpub\wwwroot\cacti\scripts>php -q C:\inetpub\wwwroot\cacti\scripts\query_host_partitions.php localhost public 2 index
1
2
3
4
5
6
7
8
9

C:\inetpub\wwwroot\cacti\scripts>php -q C:\inetpub\wwwroot\cacti\scripts\query_host_partitions.php localhost public 2 get total 1
0
C:\inetpub\wwwroot\cacti\scripts>php -q C:\inetpub\wwwroot\cacti\scripts\query_host_partitions.php localhost public 2 get total 2
73986359296
C:\inetpub\wwwroot\cacti\scripts>php -q C:\inetpub\wwwroot\cacti\scripts\query_host_partitions.php localhost public 2 get description 2
C: Label:WinXP  Serial Number ec4119f
C:\inetpub\wwwroot\cacti\scripts>

Have you read the articles regarding Data queries? http://docs.cacti.net/node/499

[denis.king]

Thanks for the info.

I have read the link that you provided. Specifically:

"One requirement for any data query in Cacti, is that it has some unique value that defines each row in the list. This concept follows that of a 'primary key' in SQL, and makes sure that each row in the list can be uniquely referenced. Examples of these index values are 'ifIndex' for SNMP network interfaces or the device name for partitions."

The user names that exist on my device are unique and I plan to use the names as the actual unique index in to the DB. Each name will have many other stats associated with it so that should work. First I need to find out how many names there are, and their associated stats.

My XML file is slightly different because I thought what I was trying to do is different (maybe not). The main difference that I see is that most of my stats that I have listed in the XML file are "outputs" and not "inputs" as the other examples. I have to say that I am not totally clear on what is correct here. From the docs, my understanding is that any value that I need to graph needs to be an output in the XML file. That is what I have done.......I am just not sure what the input does?

The screen dump that you have provided is exactly what I would expect from the script (and my script for that matter). I was confused with the "query index" args that Cacti uses with it loads the XML file and runs the script. I couldn't understand what the expected output would be for that. I will play some more with these examples, and see what I come up with.

Thanks,
DK

[denis.king]

Have a look at my script outputs. It matches much like the examples that you have provided.


C:\Inetpub\wwwroot\cacti\scripts\Solace>perl SubscriberGenStats.pl 192.168.130.50 index
denis
john
C:\Inetpub\wwwroot\cacti\scripts\Solace>perl SubscriberGenStats.pl 192.168.130.50 query directdocsin
denis:0
john:6201764
C:\Inetpub\wwwroot\cacti\scripts\Solace>perl SubscriberGenStats.pl 192.168.130.50 query numscriptions
denis:0
john:1
C:\Inetpub\wwwroot\cacti\scripts\Solace>perl SubscriberGenStats.pl 192.168.130.50 get numsubscriptions john
1
C:\Inetpub\wwwroot\cacti\scripts\Solace>perl SubscriberGenStats.pl 192.168.130.50 query index
denis:denis
john:john

I am not sure if the last example is correct but that is what I see the host_cpu example doing.

Anything look strange to you?

[BSOD2600]

Well I'd have to agree the output of your script would appear to be ok. The only thing I'm not sure about is the textual output of the index, instead of numbers. There isn't any way your device represents the users by a number instead of a name?

As for inputs/outputs, think of Input values are to be displayed on the screen and outputs, obviously stored in the rrd file. You can change / duplicate the input/output values in the xml script for debugging, to make sure the script is properly returning the data you're after too.

[denis.king]

Good info on the input/output.

I agree, I am not sure how the alpha index will get handled. Depends how the indexes are actually handled within RRD. Initially, I assumed that the <index_order_type> element within the XML file was used to tell cacti that the index will actually be alphabetic. I have sense learned that its for ordering purposes only...I think.

I guess the only way to know if this will work is to move to the next level into cacti to see how it will respond to this index type.

worst case scenerio, I will have to have to script generate some numericial representation of the user name(s). This will be a pain to correlate within cacti for monitoring, but possible.


I will let you know what my next stumbling block becomes

DK

[denis.king]

OK - some progress made.

The script, XML file, data template and graph template all seem to be configured correctly.

When I go to the "associated graph/data template", the drop down boxes for the associated data templates are not populated with anything. My understanding is that this is taken from the XML file with elements that have a direction defined to be "output". If that is correct, could my alpha index be causing the problem here?

BTW - I attached my latest XML file if needed.

[gandalf]

IMHO, script path should include /full/path/to/perl |path_cacti|...
And there's a typo at index_title_format.
Did you follow my walkthrough at http://docs.cacti.net/node/284?
Reinhard

[denis.king]

TypeO corrected, full path added and yes, I have reviewed your example. I have had it printed and on my desk for 1 week now.....

Problem not solved at this point:

The verbose query of my XML file looks like this:
Running data query [15].
+ Found type = '4 '[script query].
+ Found data query XML file at 'C:/Inetpub/wwwroot/cacti/resource/script_queries/subscriber_stats.xml'
+ XML file parsed ok.
+ Executing script for list of indexes 'c:\Inetpub\wwwroot\cacti\scripts\Solace\SubscriberGenStats.pl 192.168.130.50 index'
+ Executing script query 'c:\Inetpub\wwwroot\cacti\scripts\Solace\SubscriberGenStats.pl 192.168.130.50 query index'
+ Found item [SubIndexID='denis'] index: denis
+ Found item [SubIndexID='john'] index: john
+ Executing script query 'c:\Inetpub\wwwroot\cacti\scripts\Solace\SubscriberGenStats.pl 192.168.130.50 query directdocsin'
+ Found item [SubDirDocsIn='0'] index: denis
+ Found item [SubDirDocsIn='6521931'] index: john
+ Executing script query 'c:\Inetpub\wwwroot\cacti\scripts\Solace\SubscriberGenStats.pl 192.168.130.50 query persistdocsin'
+ Found item [SubPerDocsIn='0'] index: denis
+ Found item [SubPerDocsIn='0'] index: john
+ Executing script query 'c:\Inetpub\wwwroot\cacti\scripts\Solace\SubscriberGenStats.pl 192.168.130.50 query avgmsgrate'
+ Found item [SubAvgMsgRate='0'] index: denis
+ Found item [SubAvgMsgRate='0'] index: john
+ Executing script query 'c:\Inetpub\wwwroot\cacti\scripts\Solace\SubscriberGenStats.pl 192.168.130.50 query numsubscriptions'
+ Found item [SubNumTopics='0'] index: denis
+ Found item [SubNumTopics='1'] index: john
+ Executing script query 'c:\Inetpub\wwwroot\cacti\scripts\Solace\SubscriberGenStats.pl 192.168.130.50 query datachanneldiscards'
+ Found item [SubDataChDis='0'] index: denis
+ Found item [SubDataChDis='41160'] index: john
+ Executing script query 'c:\Inetpub\wwwroot\cacti\scripts\Solace\SubscriberGenStats.pl 192.168.130.50 query transmitcongestiondiscards'
+ Found item [SubCongestionDis='0'] index: denis
+ Found item [SubCongestionDis='45425'] index: john
+ Executing script query 'c:\Inetpub\wwwroot\cacti\scripts\Solace\SubscriberGenStats.pl 192.168.130.50 query cugdiscards'
+ Found item [SubCUGDis='0'] index: denis
+ Found item [SubCUGDis='7629649'] index: john
+ Found data query XML file at 'C:/Inetpub/wwwroot/cacti/resource/script_queries/subscriber_stats.xml'
+ Found data query XML file at 'C:/Inetpub/wwwroot/cacti/resource/script_queries/subscriber_stats.xml'
+ Found data query XML file at 'C:/Inetpub/wwwroot/cacti/resource/script_queries/subscriber_stats.xml'


All looks good as far as up to 295....


For pages node/296-308, I setup two data source items called DirectDocsIn and NumSubscriptions, which is two of my script outputs that I would like to graph

For pages node/318, I setup two Graph template items called DirectDocsIn and NumSubscriptions. I don't really understand the CDEF stuff, but for the purposes of a simple graph, don't think's it's important for this example.

When I to the node/333 section of your example, the associated data templates drop down box(s) do not have my output values from my XML file. Not sure what I have missed, but I will try all of this to see if my alpha index has anything to do with it......I suspect not...yet anyway.

Any idea?

[BSOD2600]

Sweet, looks like you've got the hard part done now. The fact you're getting the correct data shown for your script, basically means it's working.

You said you created a Data Template, with the fields DirectDocsIn and NumSubscriptions (with the intention to expand those once you get everything working, I'd assume)? You specified its Data Input Method is Get SNMP Data (Indexed) right?

In your Data Query, you associated your Data Template (and their associated fields) with the output fields from your XML file?

[denis.king]

"You said you created a Data Template, with the fields DirectDocsIn and NumSubscriptions (with the intention to expand those once you get everything working, I'd assume)? You specified its Data Input Method is Get SNMP Data (Indexed) right?"

<DK> Yes, I am using DirectDocsIn and NumSubscriptions as samples now, but would like to graph all my defined outputs once I get my head around all of this.
I am not using Get SNMP Data (Indexed) for two reasons: 1- I am not using SNMP at all as discsussed above and 2-The example http://docs.cacti.net/node/284? uses Get Script Data (Indexed) (Page node/308 specifically). Given I am not using SNMP, but a perl script that uses HTTP to talk to a device, I figure this is the right selection. Why would I want to use Get SNMP Data (Indexed)?

In your Data Query, you associated your Data Template (and their associated fields) with the output fields from your XML file?

<DK> I am not exactly sure what portion of the setup you are talking about here. The data template that I have setup has two data source items configured.....DirectDocsIn and NumSubscriptions, which are two of the items I have set as outputs in the XML file. My understanding from the example is that these can be called anything.....I went with my XML naming convention nonetheless....I could have called them X and Y.
If you are talking about Data Query-> Graph Template -> my associated data template has two entires for data sources which are exactly what I setup in my data template (directdocsin and numsubscriptions), but still no list in the drop down box with all my XML outputs..

Any further thoughs?

[BSOD2600]

I am not using Get SNMP Data (Indexed) for two reasons:

Sorry, brain fart. You're right. I was using my Win32 running process template (SNMP) as an example to look off of.

but still no list in the drop down box with all my XML outputs..

I'm fairly sure those drop downs are coupled together with the Data Source Item(s) listed in the associated Data Template and the output fields defined in the XML file. If your Data Template is already using the "Get Script Data (Indexed)" data input method, then I'm not entirely sure whats the deal. Mind attaching your exported templates and XML file for me to see?