Graph element order bug

[nisrak]

Hello,
I have a rather complex graph template that I deployed on many devices. The graph uses many CDEFs which gets rather difficult to understand, though it works for all previously deployed graphs. I am now deploying it to a new device and it randomly is changing the order of the graph elements, thereby displaying completely wrong things on the graph (due to the CDEFs).

Here is an example of the RRD command for a previously deployed graph:

RRDTool Command:

/var/home/uhsitar/packages/rrdtools/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-60 \
--title="its99 Rice trng Dashboard" \
--rigid \
--base=1000 \
--height=250 \
--width=600 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label="" \
--slope-mode \
--font TITLE:12: \
--font AXIS:8: \
--font LEGEND:10: \
--font UNIT:8: \
DEF:a="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_556.rrd":Heap_Committed:AVERAGE \
DEF:b="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_556.rrd":Nonheap_Committed:AVERAGE \
DEF:c="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_556.rrd":Nonheap_Used:AVERAGE \
DEF:d="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_556.rrd":Heap_Used:AVERAGE \
DEF:e="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_556.rrd":Heap_Max:AVERAGE \
DEF:f="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_556.rrd":Nonheap_Max:AVERAGE \
DEF:g="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_556.rrd":time_ratio:AVERAGE \
DEF:h="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_556.rrd":latency:AVERAGE \
DEF:i="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_556.rrd":derived_cpu:AVERAGE \
DEF:j="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_556.rrd":session:AVERAGE \
DEF:ba="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_556.rrd":request_count:AVERAGE \
DEF:bb="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_556.rrd":error_count:AVERAGE \
DEF:bc="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_556.rrd":response_max:AVERAGE \
DEF:bd="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_556.rrd":read_max_db:AVERAGE \
DEF:be="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_556.rrd":read_avg_db:AVERAGE \
DEF:bf="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_556.rrd":restarted:MAX \
DEF:bg="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_556.rrd":not_responding:MAX \
CDEF:cdefbb=i,e,* \
CDEF:cdefbc=g,100,* \
CDEF:cdefbf=i,g,-,e,* \
CDEF:cdefbg=i,g,-,100,* \
CDEF:cdefbj=j,LOG,e,*,10,/ \
CDEF:cdefcd=ba,LOG,e,*,10,/ \
CDEF:cdefdc=bc,LOG,e,*,10,/ \
CDEF:cdefdg=h,LOG,e,*,10,/ \
CDEF:cdefea=bd,LOG,e,*,10,/ \
CDEF:cdefee=be,LOG,e,*,10,/ \
CDEF:cdefei=bf,500000000,* \
CDEF:cdefej=bg,500000000,* \


And here is the RRD command for my newly deployed graph:

RRDTool Command:

/var/home/uhsitar/packages/rrdtools/bin/rrdtool graph - \
--imgformat=PNG \
--start=-86400 \
--end=-60 \
--title="kfs72 Kuali dev1 Dashboard" \
--rigid \
--base=1000 \
--height=250 \
--width=600 \
--alt-autoscale-max \
--lower-limit=0 \
--vertical-label="" \
--slope-mode \
--font TITLE:12: \
--font AXIS:8: \
--font LEGEND:10: \
--font UNIT:8: \
DEF:a="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_578.rrd":Heap_Committed:AVERAGE \
DEF:b="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_578.rrd":Nonheap_Committed:AVERAGE \
DEF:c="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_578.rrd":Nonheap_Used:AVERAGE \
DEF:d="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_578.rrd":Heap_Used:AVERAGE \
DEF:e="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_578.rrd":Heap_Max:AVERAGE \
DEF:f="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_578.rrd":Nonheap_Max:AVERAGE \
DEF:g="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_578.rrd":time_ratio:AVERAGE \
DEF:h="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_578.rrd":derived_cpu:AVERAGE \
DEF:i="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_578.rrd":session:AVERAGE \
DEF:j="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_578.rrd":request_count:AVERAGE \
DEF:ba="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_578.rrd":error_count:AVERAGE \
DEF:bb="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_578.rrd":response_max:AVERAGE \
DEF:bc="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_578.rrd":latency:AVERAGE \
DEF:bd="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_578.rrd":read_max_db:AVERAGE \
DEF:be="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_578.rrd":read_avg_db:AVERAGE \
DEF:bf="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_578.rrd":restarted:MAX \
DEF:bg="/var/home/uhsitar/htdocs/its/sitar-i/rra/other_apps_default_error_578.rrd":not_responding:MAX \
CDEF:cdefbb=i,e,* \
CDEF:cdefbc=g,100,* \
CDEF:cdefbf=i,g,-,e,* \
CDEF:cdefbg=i,g,-,100,* \
CDEF:cdefbj=j,LOG,e,*,10,/ \
CDEF:cdefcd=ba,LOG,e,*,10,/ \
CDEF:cdefdc=bc,LOG,e,*,10,/ \
CDEF:cdefdg=h,LOG,e,*,10,/ \
CDEF:cdefea=bd,LOG,e,*,10,/ \
CDEF:cdefee=be,LOG,e,*,10,/ \
CDEF:cdefei=bf,500000000,* \
CDEF:cdefej=bg,500000000,* \


Note that the "latency" stat is (h) for previous graphs, but (bc) for new deploys. I have no idea why it would change the order or how to fix this? Since they use the same template, I cannot change the CDEFs without ruining all previously deployed graphs. Any help/advice would be great.

Thanks.
- Nisrak

[gandalf]

Were both of them deployed using the same version of cacti? If so, which one was used?
R.

[nisrak]

Yes, both were deployed with Cacti 0.8.7e. I have also tried re-deploying several times (and re-linking to the old data source so as not to lose data), but keep getting the different element order.

Thanks.
- Nisrak

[gandalf]

That sounds like some sql does not use "order by" as it should use. To make it sure, please export your graph template (including all references) and post them.
R.

[nisrak]

Ok, is there a way to force the query to use a certain 'order by'? I don't mind if its different as long as it is consistent so I can make CDEFs that will always work.

Attached is the exported graph template.

Thanks.
- Nisrak

[gandalf]

Ok, is there a way to force the query to use a certain 'order by'? I don't mind if its different as long as it is consistent so I can make CDEFs that will always work.

Attached is the exported graph template.

Thanks.
- Nisrak

That's not an option on your side. It requires a code change. But I will have to analyze first. It was only an assumption
R.

[gandalf]

I had some errors while importing that template. Then, perhaps as a result of that, I found some issues with the graph template.
I tried to fix them, but guessing from time to time. Please try to import on a test system and verify then.
If this is not possible, please post at least a screenshot of both the graph templates items list (when editing that graph template) and a screenshot of the "Graph Item Inputs" as well.
R.

[nisrak]

I think you may have posted the wrong xml file. The one you posted contains a different graph template called "1 Cisco - CPU Usage" and edits a few other things.
I did import it into my test environment and did see any changes (except the creation of the 1 Cisco template.

Attached is a screenshot of the Dashboard graph template. It shows the graph elements of the template and the data items associated with them. I pasted them together as you would see them in the actual Cacti interface.

Thanks again for all the help!

- Nisrak

[gandalf]

Then, let's do it the hard way.
I suspect, that two issues have to be solved.
First, I do not understand, why there are (no Task) items that shall be graphed. That may make sense, when a CDEF is attached that refers to specific, position dependant data sources.
Then, in the INPUTS, there are duplicates. I suppose, that many of them are "empty" = when selecting them, you won't find any checkbox selected. You should delete them (what I did when I worked on your template).
All of this sounds a bit crazy; yes, the fact that you received a template that I at least did not intend to refer to other specific stuff makes me think that the template itself is broken. There are so many weird issues around this ...
R:

[nisrak]

Sorry I havent replied. Finally had the chance to try this. On my test environment I removed all of the extra data input items on the graph template and fixed all of the CDEFs to fit the new item order. It works fine on a new test graph that I made.
Am now experiencing 2 strange errors though:

First, after creating a graph, I associate the Data Source items with each graph element, 1 item will not associate correctly. The attached image shows this. The "Heap Non-Committed" always goes back to "None" when I click Save. The graph is displaying the Heap Non-Commited stat correctly though!

The other issue is trying to get previously created graphs working with the new changes to the template. Several of the Data Source items do not show up!! In this case, the Heap Non-Committed and Session stats do not appear in the drop-down list. The data template has 42 items, but there are only 40 in the drop-down list! I also manually checked the rrd file with "rrdtool info" and all the 42 stats are there! Why would they not show up on the drop-down list?

I think everything will be working fine if I can get these last few things figured out. Thanks so much for the help!

Thanks.
- Nisrak

[nisrak]

Figured it all out! It seems that part of the initial problem was that my data template had duplicates, causing the duplicates in my graph template to be super unpredictable. By deleting duplicate graph template items (and in some cases deleting all and re-selecting) I got the graph template to work. One thing to note is for a few items I had to delete entries in the data template and re-add it because it was not showing up on the drop-down items for some data sources.
Thanks a bunch for your help!

Thanks.
- Nisrak

[TheWitness]

That's a massive Data Template and Graph Template. Is there any reason you did not go with less Data Sources in the Data Template. I would think you would want this to be spread over 5+ graphs. When they get this big, it's pretty tricky to debug them. Nice thing is, when they are done, they are done.

TheWitness

[nisrak]

Yeah, I have told the users that this graph is getting way too big, but they want to see all the stats at the same time. As it currently is, it is really hard to even understand most of the stats (lots of them have crazy CDEFs).

I have thought about ways of displaying multiple zoomable graphs side by side on some kind of basic PHP page, but don't have any time to develop an add-on. Now that it works, I will just let them use this huge graph and maybe one day give them a better visualization.

As for the huge data template, we do not use SNMP so repeatedly querying can be costly. I am currently developing a new data gathering method that will be easily scalable and uses SSH to gather large amounts of data for later parsing. When it and several other customizations are complete and documented, we will be making our project publicly available.

Thanks.
- Nisrak

[TheWitness]

As for the huge data template, we do not use SNMP so repeatedly querying can be costly.

Yea, it's better to grab everything, and drop it in a temp file so that other scripts can leverage it. The more elegant method is to write either a plugin that gathers the data asynchronously, or a cron based poller. Then cacti can go after the local data.

I am currently developing a new data gathering method that will be easily scalable and uses SSH to gather large amounts of data for later parsing. When it and several other customizations are complete and documented, we will be making our project publicly available.

That's the way to go.

TheWitness