Zooming too far = high CPU use rrdtool process

[pezz]

G'day guys,

I was wondering if there are any known issues with the "zoom" functionality of cacti, as I find if you zoom in to a graph too far it creates a high CPU usage rrdtool process on my system, which can be done multiple times if you want. I end up having to kill the process as the page never returns until I do so...

I have used probably the last 3 versions of cacti (I'm on 0.8.6e now) and have seen it in all versions - both using rrdtool 1.2.x and 1.0.x

Is there a work-around or fix for this, as it seems anybody can cause high CPU load on my box if they choose to - simply by over-using the zoom function.

Is there any way I can "limit" how far someone can zoom in? As it seems to happen when you zoom in under a 5 minute interval on a graph.

Cheers, and TIA - love cacti!

[rony]

Another user was reporting this problem. I have never been able to reproduce.

Are you running windows or unix. And if so, where did you get RRDtool from. Did you compile it, or get it pre-compiled?

It's not a cacti bug, I'm pretty sure it's a RRDTool issue.

[pezz]

It is essentially compiled by portage as I run a Gentoo system.

If there is some refernce post on these forums or some tip on how I can use a pre-compiled version of rrdtool that doesn't exhibit the behaviour that you could point me at then that'd be great!

Cheers.

[rony]

No reference, I'm just trying to get an idea what is causing this..

[SithLord]

Hi pezz,

I am the other user who is facing the same problem as you. I too use Cacti on a Gentoo server and I have the same problem. For further info, look at these links :

• Cacti forum thread
• Cacti bug tracker issue

If you have any idea, would be great !

[TheWitness]

There are circumstances that cause the older versions of RRDTool to enter race conditions.

One of them was associated with Area/Stack graphs. This problem occured in 1.0.x and 1.2.x until a certain point release. It's definately fixed in 1.2.10 that much I know. It happened when you have malformed graph item ordering.

Question, for the graphs in question, do they ever appear accurate? If so, please post a few things 1) Graph Syntax 2) RRDfiles in XML format.

I will report the bug to Tobi.

TheWitness

[TheWitness]

Also, does this only happen with either 95th percentile or Bandwidth type graphs?

TheWitness

[pezz]

This problem happens on all of my graphs whether they be area / stack or line graphs. I also upgraded to rrdtool 1.2.10 to see if it fixed the issue, and it didn't.

I have since gone back to 1.2.6 as I like to keep in sync with stable packages.

I'll pass on the info you have requested when I get the chance.

Cheers.

[TheWitness]

Are they all graphs types or just bandwidth/95th percentile type? Does it only happen when you zoom in? Is there a point where the CPU starts racing. Please provide additional details and we can release a patch to zoom.js. (i.e. If we know what the minimum "safe" timespan is (end_time - begin_time) we can prevent the zoom feature from going that low.)

TheWitness

[rony]

It's minimum is 1 second right now.

I have been able to zoom down to that in many many rrdtool files.

How many DS's are in the graphs that do this? How old are those files? Have they been around for a long time?

[emat]

Hi all same problem here, I'm also having this with all my graphs.

I'm also using gentoo linux 2005
I'm using rrdtool-1.2.6r1 which is concidered stable.

Here's some syntax

Code: Select all

/usr/bin/rrdtool graph - \
--imgformat=PNG \
--start=1120464662 \
--end=1120476453 \
--title="Hum outside" \
--base=1000 \
--height=220 \
--width=700 \
--upper-limit=100 \
--lower-limit=0 \
COMMENT:"From 2005/07/04 10\:11\:02 To 2005/07/04 13\:27\:33\c" \
COMMENT:"  \n" \
--vertical-label="Procent" \
--slope-mode \
DEF:a="/var/www/localhost/htdocs/cacti/rra/test_snmp_oid_11.rrd":snmp_oid:AVERAGE \
LINE2:a#6EA100:"Luchtvochtigheid"  \
GPRINT:a:LAST:"Current\:%8.2lf %s"  \
GPRINT:a:AVERAGE:"Average\:%8.2lf %s"  \
GPRINT:a:MAX:"Maximum\:%8.2lf %s"  \
GPRINT:a:MIN:"Minimum%8.2lf %s\n"  \
HRULE:85#FF3932:"Maximum Waarde"  \
HRULE:50#4444FF:"Minimum Waarde"  \
HRULE:65#B1441E:"Streefwaarde\n" 

The last thing I changed before I noticed this behavious is when I've changed the standard RRA's into this:

Code: Select all

Name   	                             Steps   	 Rows   	 Timespan
Daily (5 Minute Average) 	        1 	           600 	             86400
Weekly (5 Minute Average)             1 	         2200 	          604800 
Yearly (5 Minute Average)              1 	          105120 	33053184
Monthly (5 Minute Average)           1          	9000 	        2678400

The point of this is to have a resolution of 5 minutes for every day, week, month and year

There is only 1 datasource for the graphs
The rrd's are about 10 days old
If I zoom down to 20 minutes, rrdtool hangs with 99% cpu

If you want to know more, i can try to give you answers,.....

Greetings

[SithLord]

Here, it happens with ALL kinds of graphs and ONLY happens when zooming (hence my feature request to disable zooming via prefs on Mantis bugtracker because for us, this is a major/critical issue and can be used for DoS'in a server hosting Cacti).

Here is what rrdtool was doing when the CPU was toping at 100% usage :

Code: Select all

/usr/bin/rrdtool graph -
--imgformat=PNG
--start=1120451367
--end=1120452092
--title="Orthanc - Memory"
--base=1000
--height=150
--width=500
--alt-autoscale-max
--lower-limit=0
COMMENT:From 2005-07-04 06\:29\:27 To 2005-07-04 06\:41\:32\c
COMMENT:  \n
--vertical-label=Size (bytes)
--slope-mode
--font TITLE:9:/usr/local/opt/TrueTypeFonts/tahomabd.ttf
--font AXIS:7:/usr/local/opt/TrueTypeFonts/tahoma.ttf
--font LEGEND:7:/usr/local/opt/TrueTypeFonts/lucon.ttf
--font UNIT:8:/usr/local/opt/TrueTypeFonts/tahoma.ttf
DEF:a=/usr/local/opt/cacti/rra/orthanc_mlk_mem_total_88.rrd:mlk_mem_total:MAX
DEF:b=/usr/local/opt/cacti/rra/orthanc_mlk_mem_free_87.rrd:mlk_mem_free:AVERAGE
DEF:c=/usr/local/opt/cacti/rra/orthanc_mlk_mem_buffers_85.rrd:mlk_mem_buffers:AVERAGE
DEF:d=/usr/local/opt/cacti/rra/orthanc_mlk_mem_cache_86.rrd:mlk_mem_cache:AVERAGE
CDEF:cdefa=a,1024,*
CDEF:cdefc=b,1024,*
CDEF:cdefh=c,1024,*
CDEF:cdefbc=d,1024,*
LINE1:"defa#000000:Total
GPRINT:cdefa:MAX:-%17.2lf %sB\n
AREA:cdefc#8F005C:Free
GPRINT:cdefc:LAST:   - Current\:%8.2lf %sB
GPRINT:cdefc:MIN:- Min\:%8.2lf %sB
GPRINT:cdefc:AVERAGE:- Avg\:%8.2lf %sB
GPRINT:cdefc:MAX:- Max\:%8.2lf %sB\n
AREA:cdefh#FF5700:Buffers:STACK
GPRINT:cdefh:LAST:- Current\:%8.2lf %sB
GPRINT:cdefh:MIN:- Min\:%8.2lf %sB
GPRINT:cdefh:AVERAGE:- Avg\:%8.2lf %sB
GPRINT:cdefh:MAX:- Max\:%8.2lf %sB\n
AREA:cdefbc#FFC73B:Cache:STACK
GPRINT:cdefbc:LAST:  - Current\:%8.2lf %sB
GPRINT:cdefbc:MIN:- Min\:%8.2lf %sB
GPRINT:cdefbc:AVERAGE:- Avg\:%8.2lf %sB
GPRINT:cdefbc:MAX:- Max\:%8.2lf %sB\n

and here is the graph I was zooming. Selecting any kind of zone for zooming results in a CPU usage at 100% on the server with a staled rrdtool command (which must be killed with kill -9/-KILL pid)

[TheWitness]

Guys,

Dump the rrd files associated with the graph syntax in the attached and attach the XML files. I will research. It will take a few days though. On vacation till the 9th.

TheWitness

[emat]

K,

Here's the rrd file it's quite huge (1,8 Mb)
The Graph Template in xml
The Data Template in xml

It concerns the processes graph.
It's one of the default graphs which is available with a new install.
I Have modified nothing about these graphs.
The only thing I modified are the RRA values because i wanted more detail (hence the rrd file size 1,8 MB) (see earlier post) on this topic

I hope I gave enough info to help, if not give me detailed description of what to do,....

PS :

If I look at the previous posts it all concerns gentoo.........
I'm not sure (don't kill me if i'm wrong ) but this problem started when I changed the RRA values to get more detailed graphs from the past)

[TheWitness]

emat,

Did you simply make changes from the UI or did you use the "RRDtool tune" function, or both? Another method would be to change the settings in Cacti and then delete the RRD file..

Again, I will look at these on the 9th or thereafter.

Thanks,

TheWitness