OIDs are the same for every data source?

[jorhett]

I am building a new data source template and defining 5 different OIDs which should be stored in 5 different data sources in the RRD file.

Oddly, I have found that after carefully defining all 5 with different OIDs, only the last one was saved. When I click between each data source, the OID remains the same for all 5 of them. Image is attached.

[rsanto]

work as design LOL

create a data source for each OID or a script pointed to a data source referenced to that script.

This is the reason why you have no help for your post. Take some time to read the manual and search before posting, most of my questions where answered on this forum. That is way now I contribute as far as my knowledge gets.

Regards

[jorhett]

work as design LOL

create a data source for each OID or a script pointed to a data source referenced to that script.

This is the reason why you have no help for your post. Take some time to read the manual and search before posting, most of my questions where answered on this forum. That is way now I contribute as far as my knowledge gets.

I did read the manual, and I have read dozens of threads on this, and none of them addressed this issue. In fact, I found little more than people pointing out that the docs don't match the product and can't be followed, which is true.

It can't possibly work this way by design. How could five data sources have the same OID? That makes sense in absolutely zero situations. It created five DS in the RRD file, but it only collects data from a single OID? I need 5 different data sources stored in the RRD, not 1 data source stored 5 times. Nobody ever needs the latter.

I've been using RRD since it was created for MRTG, and I know monitoring and RRD inside and out. And your docs make no sense, no how, no way. So don't say "read the docs" when the docs are inaccurate, out of date, incomplete, etc. The docs have absolutely nothing that tells me why this is done this way

Read the forums and you'll find nobody that says "oh, I figured it out from the docs". If you stop being nasty about it, and help people learn to do it "correctly" then I might help update the docs so that others have something useful to use.

Now, if you could bother yourself to be helpful-- I have 26 data sources from this product. I do not want or need 26 RRD files. Many of the built in cacti data sources put multiple ds in a single RRD, so it's not impossible. Let's take a simple example: I want to store tempC and tempF in the same file, two different DS in the same RRD. How do I accomplish this?

[jorhett]

Take some time to read the manual and search before posting, most of my questions where answered on this forum. That is way now I contribute as far as my knowledge gets.

Just to point out how useless your rant is, let's show you how far your suggestions go:

Search the forum:
The following words in your search query were ignored because they are too common words: oid source rrd data
Um... that's all the words that I have?!

Read the manual:
http://www.cacti.net/downloads/docs/htm ... p_oid.html

This page describes how to map an OID to a graph item. No page in the documentation at all mentions the "New" button to the right or what it does. Clicking that button and creating a new DS creates a new ds in the RRD file. That makes sense... so for each ds in the file we'd map a different OID/source of data, right?

You say no. You say it's not intended to, and that I should read the manual, but the manual simply doesn't mention this. The functionality works as one would expect... half way. There simply isn't any wordage that a manual or search can find that would have told me what this is supposed to do.

I'm sorry, but you are wrong. Being rude to me about reading the docs was just being personally nasty. Congratulations, you got to be nasty to me. Did it make your day better?

[gandalf]

I have to ask for a wrapup. Some bits are missing to understand.
Let me start with some "assumptions". Please clarify at this level:

- you do have 5 different OIDs, that you want to fetch data from?
- It is important to know, whether the data is fetched via SNMP (aka: you have to OIDs) or whether it is fetched via script.
- Next, it is important to know, whether those OIDs are part of an SNMP table, aka multiple indexes are existing in general.
- you do want to store them in a single rrd file (is this a hard requirement, a wish or just the way you understood so far?)

hth
R.

[jorhett]

I have to ask for a wrapup. Some bits are missing to understand.
Let me start with some "assumptions". Please clarify at this level:

- you do have 5 different OIDs, that you want to fetch data from?

Yes.

- It is important to know, whether the data is fetched via SNMP (aka: you have to OIDs) or whether it is fetched via script.

I would prefer SNMP. I think I've learned how to do script, but the SNMP parts of this are high wizardry. Half in the system, half in XML files...?

- Next, it is important to know, whether those OIDs are part of an SNMP table, aka multiple indexes are existing in general.

I have a bit of both. For these five queries shown on screen they are internal sensors and there will only be one of these. Separately I'll need to query for external sensors which may or may not be installed.

- you do want to store them in a single rrd file (is this a hard requirement, a wish or just the way you understood so far?)

Well, I'm moving over about 400 existing RRD files where the data is already one file per sensor, and I'd prefer to keep this format. It's not impossible to break 400 RRD files into ~5k files but...

[jorhett]

So I have got snmp queries for indexed data working just fine. It's silly to have to spend half your time in Cacti punching the same buttons time after time, and then half the time in XML format. I really wish the template creation was documented

Anyway, indexed data for external sensors plugged in works fine. However the query against the internal sensors is not working quite right. Here's the snmpwalk:

Code: Select all

snmpwalk -c public -v 1 -On goose 1.3.6.1.4.1.17373 
.1.3.6.1.4.1.17373.3.1.1.0 = STRING: "MiniGoose II"
.1.3.6.1.4.1.17373.3.1.2.0 = STRING: "3.8.0"
.1.3.6.1.4.1.17373.3.1.3.0 = STRING: "MiniGoose II"
.1.3.6.1.4.1.17373.3.1.4.0 = STRING: "00:19:85:E0:6F:F4"
.1.3.6.1.4.1.17373.3.1.5.0 = STRING: "http://192.168.13.133"
.1.3.6.1.4.1.17373.3.1.6.0 = INTEGER: 0
.1.3.6.1.4.1.17373.3.1.7.0 = STRING: "ITW"
.1.3.6.1.4.1.17373.3.1.8.1.2.0 = INTEGER: 1
*snip a bunch of zeroed out stuff not in my xml file*
.1.3.6.1.4.1.17373.3.2.1.2.1 = STRING: "28864F8002000056"
.1.3.6.1.4.1.17373.3.2.1.3.1 = STRING: "MiniGoose II"
.1.3.6.1.4.1.17373.3.2.1.4.1 = Gauge32: 1
.1.3.6.1.4.1.17373.3.2.1.5.1 = INTEGER: 21
.1.3.6.1.4.1.17373.3.2.1.6.1 = INTEGER: 70
.1.3.6.1.4.1.17373.3.2.1.7.1 = INTEGER: 0
.1.3.6.1.4.1.17373.3.2.1.8.1 = INTEGER: 0
.1.3.6.1.4.1.17373.3.2.1.9.1 = INTEGER: 0
.1.3.6.1.4.1.17373.3.2.1.10.1 = INTEGER: 0
.1.3.6.1.4.1.17373.3.2.1.11.1 = INTEGER: 0
.1.3.6.1.4.1.17373.3.2.1.12.1 = INTEGER: 0
.1.3.6.1.4.1.17373.3.2.1.13.1 = INTEGER: 0
.1.3.6.1.4.1.17373.3.2.1.14.1 = INTEGER: 0
.1.3.6.1.4.1.17373.3.2.1.15.1 = INTEGER: 0

My query xml file looks like this:

Code: Select all

<interface>
	<name>Get ITWatchdogs *Goose Internal Sensors</name>
	<description>Retrieve a list of Internal *Goose sensors</description>
	<oid_num_indexes>.1.3.6.1.4.1.17373.3.1.8.1.2.0</oid_num_indexes>
	<oid_index>.1.3.6.1.4.1.17373.3.2.1.2</oid_index>
	<oid_index_parse>OID/REGEXP:.*\.([0-9]{1,3})$</oid_index_parse>
	<index_title_format>climateName:climateSerial</index_title_format>
	<index_order_type>numeric</index_order_type>
	<fields>
		<climateSerial>
			<name>Serial Number</name>
			<method>walk</method>
			<source>value</source>
			<direction>input</direction>
			<oid>.1.3.6.1.4.1.17373.3.2.1.2</oid>
		</climateSerial>
        <climateName>
            <name>Name</name>
            <method>walk</method>

...snip snip everything below 3.2.1.2 is identical except the names change and after status they are all "output" values.

When I apply this to a host and do the verbose debug query, it finds all of the input values just fine:

Code: Select all

+ Running data query [15].
+ Found type = '3' [SNMP Query].
+ Found data query XML file at '/usr/share/cacti/resource/snmp_queries/ITW-Goose-internal.xml'
+ XML file parsed ok.
+ Executing SNMP get for num of indexes @ '.1.3.6.1.4.1.17373.3.1.8.1.2.0' Index Count: 1
+ Executing SNMP walk for list of indexes @ '.1.3.6.1.4.1.17373.3.2.1.2' Index Count: 1
+ Index found at OID: '1.3.6.1.4.1.17373.3.2.1.2.1' value: '28864F8002000056'
+ index_parse at OID: '1.3.6.1.4.1.17373.3.2.1.2.1' results: '1'
+ Located input field 'climateSerial' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.4.1.17373.3.2.1.2'
+ Found item [climateSerial='28864F8002000056'] index: 1 [from value]
+ Located input field 'climateName' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.4.1.17373.3.2.1.3'
+ Found item [climateName='MiniGoose II'] index: 1 [from value]
+ Located input field 'climateAvail' [walk]
+ Executing SNMP walk for data @ '.1.3.6.1.4.1.17373.3.2.1.4'
+ Found item [climateAvail='1'] index: 1 [from value]

So in short, it appears to work. However I'm not getting any values back. Is there some way to fake out the indexed data bit such that I can get these values? The unit even kindly has 3.1.8.1.2 which says "there is 1 internal climate sensor" which is always true... what am I getting wrong here?

[jorhett]

FYI, back to the original question. For anyone who reads this thread I think right now the answer is:

The UI design is confusing. It allows you to define multiple DS in the RRD file but this only works for retrieving data from script-supplied data sources. When doing an SNMP query, only one OID is stored in the database so it simply doesn't work even though it looks right.

Which leads to a simple Feature Request: how about we fix this? The UI is already well designed for this. All we have to do is fix the DB work such that each DS in the RRD could be associated with an OID. For many situations this would obsolete any need for external snmp query files, and make it much simpler for people who only want a handful of OIDs pulled from a device.

[gandalf]

So I have got snmp queries for indexed data working just fine. It's silly to have to spend half your time in Cacti punching the same buttons time after time, and then half the time in XML format. I really wish the template creation was documented

I honestly suppose it IS documented at 1st link of my sig with some complete walkthroughs.
R.

[gandalf]

FYI, back to the original question. For anyone who reads this thread I think right now the answer is:

The UI design is confusing. It allows you to define multiple DS in the RRD file but this only works for retrieving data from script-supplied data sources. When doing an SNMP query, only one OID is stored in the database so it simply doesn't work even though it looks right.

Which leads to a simple Feature Request: how about we fix this? The UI is already well designed for this. All we have to do is fix the DB work such that each DS in the RRD could be associated with an OID. For many situations this would obsolete any need for external snmp query files, and make it much simpler for people who only want a handful of OIDs pulled from a device.

For sake of flexibility, we have chosen exactly the current design. Parts of what you see as inflexibility has to be dedicated to rrdtool; sorry to say that.
But please know, that current rrdtool "officially" does not allow to add or delete a data source item from any existing rrd file. So in case you want to modify your Data Query XML (and, btw: all OIDs of an SNMP Data Query are in that XML, nothing is in the UI), you would loose all your history of existing rrd files. Utmost flexibility is delivered when you use a single rrd file for each <field>. You then are able to combine those data in every way you can imagine.
And usually, this is neither a disk issue nor a disk space issue.

I agree, that the difference between a "multi data source item" Data Template for scripts and the "single data source item" Data Template for SNMP Data Queries is confusing and not easy to understand. That's why the tuto's are there. And currently, I see no flexibel approach to change that.

R.

[jorhett]

For sake of flexibility, we have chosen exactly the current design. Parts of what you see as inflexibility has to be dedicated to rrdtool; sorry to say that.

Based on what? rrdtool can do multiple ds in a file. I have tens of thousands of those

But please know, that current rrdtool "officially" does not allow to add or delete a data source item from any existing rrd file. So in case you want to modify your Data Query XML (and, btw: all OIDs of an SNMP Data Query are in that XML, nothing is in the UI), you would loose all your history of existing rrd files. Utmost flexibility is delivered when you use a single rrd file for each <field>. You then are able to combine those data in every way you can imagine.

I totally agree that one file per data source may work for most people, and is easiest to understand. But I fail to understand why it is explicitly prevented, when the data structure in SQL would allow for this with only a minor tweak and the UI is already designed for this. There are people like myself who know RRD and know what they want to store and how. Since I have like thousands of these files already existing I want to import...

And see also the comment below about I/O capacity.

Further, if I decided to change this format in the future I can write a script to iterate over the existing RRDs, export them, adjust the export to add a DS, and reimport them. Time from start of writing that script to being done testing it would be less than an hour. (okay, I've done this a few dozen times already). Point of fact, if we can do something intelligent about this for the next version of Cacti I could probably generalize this well enough to do it within Cacti.

You have to put your OIDs in the XML, then you have to map them in the UI. It's half here, half there, and both are incomplete without the other. This makes your templates patchwork. More than half of the Templates I find on the Internet (including your own template repository) are useless since they don't include the xml files. This one I am working on here was exactly that, it's listed in the official repository but doesn't work because the xml files are missing. For the smaller Mini and MicroGoose units they only support internal sensors (not indexed) so a template for them could be complete without an XML file if you had multi-OID support.

And usually, this is neither a disk issue nor a disk space issue.

It's an I/O capacity problem. To understand my answer you need to grok the effect of implementing your choice: It's a "hundreds of thousands of files in a directory" issue. We have 28k rrd files today. If we have to split on data sources, we will have more than 200k files. Even when using the split-on-hostid option, we'd have more than 4k files in a single host's directory.

Now what does this mean? Right now to make a graph for that target we read a single RRD file. In your scenario we'd have to read dozens of RRD files. Furthermore, on writes we'd have to do aggregates for dozens of separate files. We used to have lots of separate files. We cut I/O for reads to roughly 30% by combining the data for single targets into single files. RRDtool is tremendously more efficient when all the data to aggregate for RRAs are encompassed in the same set of linear reads.

I agree, that the difference between a "multi data source item" Data Template for scripts and the "single data source item" Data Template for SNMP Data Queries is confusing and not easy to understand.

I think the words you are looking for is "not mentioned at all". It would be easy to understand if the idea was mentioned even once. There's no confusion except by its absence.

That's why the tuto's are there. And currently, I see no flexibel approach to change that.

What? Um, I am working from your tutorials. It says "do this" and "do this". It never once mentions these limitations, it never once says why you can or cannot do this or that. The documentation does not *explain* anything, it just runs you through doing one thing once, without explaining a single one of the boundary considerations.

I am not trying to complain, but I am growing tired of people saying "read the docs" with a pointer to the main page. If you think that this is documented, point me to the page. Because I've read the 0.87 docs (which show up in google searches), the 0.88 docs (which do not) and also the docs you point to in your link (which overlap with only minor differences) and nothing that you or rsanto say is documented is there.

Once I figure all of this stuff out, I'm going to send you patches for the documentation and also patches for how to implement multi datasources for OIDs for the next version of Cacti. Because I think it's important, and it will make a lot of templates self-standing.

I have a personal goal of making it easy to implement either (1) everything in the UI and (2) everything in XML or YAML or whatever, without having to do both. I prefer #2 but many people prefer #1. I think the current system of half-here half-there is baffling at best. Most importantly, if we make it easy to do everything in #2 I think we can (A) talk a lot of vendors into providing templates and (B) generate complete templates automatically from MIB files much better than the current hack-it approach allows.

[jorhett]

So I have got snmp queries for indexed data working just fine. It's silly to have to spend half your time in Cacti punching the same buttons time after time, and then half the time in XML format. I really wish the template creation was documented

I honestly suppose it IS documented at 1st link of my sig with some complete walkthroughs.
R.

Really? Seriously man, search your forums. Everyone comes here trying to figure it out. They post questions from the documentation, with specific inquiries. They get treated poorly and they move on to other things instead. Do you ever wonder why there's very little momentum behind Cacti? It would appear that the entire momentum behind Cacti is... You. Wouldn't you like that to change?

I've been working with MRTG since RRDtool was invented (wasn't usable for me before then). I've personally created hundreds of tools for RRD file management, for Nagios RRD usage, for Cricket, for Zenoss and dozens of hand-built internal company systems. I know RRD inside and out, I know network monitoring inside and out. And I can't follow your documentation.

I know what it's like to be so close to a project you can't read the documentation any more. I have some of those myself. But read your forums, read how people come here and can't follow it. How they ask specific questions based on specific statements in the documentation, and receive nasty responses like "read the docs" and "search the forums". Searches for any of the relevant words in your forums return no results due to overusage. It would be different if someone said "search using these words" or "look at these threads", but they don't. People get discouraged and they leave.

I made a decision to convert a lot of the custom hand-built RRD systems to Cacti so that it was easier to document and upgrade over time. Some things I am going to need to introduce whole new concepts to Cacti, like custom polling intervals in Spine. I expect that to be more difficult. What has shocked me has been how hard it has been to do really, really simple things. I have really struggled because what you consider to be documentation is completely inadequate to build a template for even a simple four-oid network unit.

At this point I've resigned myself to the fact that my deliverables for getting these units with no support in Cacti working has grown to:

* learning the entire database structure - done
* building a better map of the database structure - almost done
* learning how templates really work by creating/diffing - in progress
* begging people on the forum to explain things that are completely never mentioned in the docs - not even close
* rewriting the documentation with what I learn - in progress
* writing patches to make it easier to define all this in XML format which I can maintain - in progress

That's a lot more work that I originally signed up for. That said, I'm committed to doing the work because out of the open source projects out there, this is the easiest thing to work with when you have thousands of RRD files. (most new things don't use RRD any more)

So what I'm trying to say is, like Dude, help me out. If you can help me understand what the documentation doesn't say, I will work on the documentation for you. I will make patches for features I want. And I will make it easier to support new units, so that we both get more people using this and helping out. I can't promise you any more momentum than just me, but in my experience if you guys will stop just telling people to "read the docs" when they are asking questions FROM THE DOCS you might get some more.

[jorhett]

So I have got snmp queries for indexed data working just fine. It's silly to have to spend half your time in Cacti punching the same buttons time after time, and then half the time in XML format. I really wish the template creation was documented

I honestly suppose it IS documented at 1st link of my sig with some complete walkthroughs.
R.

Are you aware that this link goes to the 0.87 version of the documentation and the screens don't match?

[rsanto]

nice post

[gandalf]

For sake of flexibility, we have chosen exactly the current design. Parts of what you see as inflexibility has to be dedicated to rrdtool; sorry to say that.

Based on what? rrdtool can do multiple ds in a file. I have tens of thousands of those

I KNOW, that rrdtool can cope with multiple ds in a single file. The issue is, that once that file exists, you can't change the ds' any more. But on a Data Query, we have to expect that indexes are changing and thus the ds' have to change. And this is, where rrdtool can't keep up.
So we use a single rrd file for each ds to have full flexibility. You may want to follow similar discussions on the rrdtool forum. And from my point of view you will end up with exact the setup we have chosen.

But please know, that current rrdtool "officially" does not allow to add or delete a data source item from any existing rrd file. So in case you want to modify your Data Query XML (and, btw: all OIDs of an SNMP Data Query are in that XML, nothing is in the UI), you would loose all your history of existing rrd files. Utmost flexibility is delivered when you use a single rrd file for each <field>. You then are able to combine those data in every way you can imagine.

I totally agree that one file per data source may work for most people, and is easiest to understand. But I fail to understand why it is explicitly prevented, when the data structure in SQL would allow for this with only a minor tweak and the UI is already designed for this. There are people like myself who know RRD and know what they want to store and how. Since I have like thousands of these files already existing I want to import...

This is valid but has been concerned as a corner case up to now. From my knowledge, discussions in teh rrdtool forum show, that there's no negative performance when using the current file setup.
But honestly, I did not test this to a very deep extend.

Further, if I decided to change this format in the future I can write a script to iterate over the existing RRDs, export them, adjust the export to add a DS, and reimport them. Time from start of writing that script to being done testing it would be less than an hour. (okay, I've done this a few dozen times already). Point of fact, if we can do something intelligent about this for the next version of Cacti I could probably generalize this well enough to do it within Cacti.

I know and I did this several times in the past. But it seems too complicated for the average Cacti user. But you may know, that now we have some contributed rrdtool stuff which will address this. And for Cacti 0.9.0, I wrote some similar utilities to tackle similar cases. But my approach was NOT based on your use case.

Our plans are to change the XML file handling and put the stuff compeletely into the DB, getting rid of files as much as we can.

And usually, this is neither a disk issue nor a disk space issue.

It's an I/O capacity problem. To understand my answer you need to grok the effect of implementing your choice: It's a "hundreds of thousands of files in a directory" issue. We have 28k rrd files today. If we have to split on data sources, we will have more than 200k files. Even when using the split-on-hostid option, we'd have more than 4k files in a single host's directory.

Now what does this mean? Right now to make a graph for that target we read a single RRD file. In your scenario we'd have to read dozens of RRD files. Furthermore, on writes we'd have to do aggregates for dozens of separate files. We used to have lots of separate files. We cut I/O for reads to roughly 30% by combining the data for single targets into single files. RRDtool is tremendously more efficient when all the data to aggregate for RRAs are encompassed in the same set of linear reads.

We have around 100,000 files on disk in a single directory. While you may consider this as not optimal, we perform about 2,000,000 writes using BOOST in less than 5 minutes. Well, it's a SAN ...

I agree, that the difference between a "multi data source item" Data Template for scripts and the "single data source item" Data Template for SNMP Data Queries is confusing and not easy to understand.

I think the words you are looking for is "not mentioned at all". It would be easy to understand if the idea was mentioned even once. There's no confusion except by its absence.

In case you point me to the palce where you expect more comments/help/documentation and some text you'd like to see there, I will add this in about no time. I always am happy if users help to improve documentation.

And same applies, but with a more relaxed time frame, to code contributions.
Don't get me wrong: I appreciate your will to improve Cacti very much
R.