rrdtool 1.4.3 + php 5.3.1 = Slow Graphing (rrdtool@100% cpu)

[ckawchuk]

Hi All,

Sorry to bother the Cacti Gurus, but been scratching our collective heads off here for the past few weeks.

Have been running cacti @0.8.7e for a while - under OSX 10.6.2 Server.

Recently did ye olde Upgrade to OSX 10.6.3 Server, which puts in a newer version of php (amongst other things it changes).

Graphs now render rather slowly.... as rrdtool spikes to 100% cpu load when displaying graphs - and stays there for a good 8-10 seconds. (we used to render graphs in sub 1s times, for multiple graphs).

The server is an Apple Xserve 2009 Model, i.e.:
8 x QuadCore Intel i7's at 2.5 ghz
12 Gb DDR3-1066 Mhz RAM
1 TB RAID 1
i.e. lots-o-horsepower on this puppy, lots of free RAM too. (enough that all our rrd databases end up being filesystem-cached in memory - and it's still aufully slow).

I'm looking towards a werid php + rrdtool interaction; as it's rrdtool which goes to 100% cpu load (although it does get launched from cacti's php script when viewing).

Any pointers?

This issue is virtuall identical to this previous post:

http://forums.cacti.net/viewtopic.php?t=34597

although I dont necessarily have the option of downgrading to php 5.2 (I was running this under php 5.3.0 prior to the OSX upgrade).

Thanks in advance to anyone who's run into this before. I'm sure it's "something dumb" I forgot to tweak in some .ini file somewhere =)

- Chris.

[TheWitness]

Definately a bug. If you take the graph syntax and post to a shall and save of the graph, does the problem go away? If so, it's likely an issue with PHP and it's handling of pipe output and possibly the passthrough command in PHP. If you also take the latest lib/rrd.php from SVN it, may correct that issue if it's happening. However, before you go do that, if you are running the Plugin Architecture, you will have to hand compare. They are very different.

TheWitness

[ckawchuk]

Definately a bug. If you take the graph syntax and post to a shall and save of the graph, does the problem go away?

Yes - If I take the graph syntax and invoke rrdtool from the shell, it renders the images very quickly. Hence, Yes, I'm thinking towards an apache+php+rrd interaction.

I'll do as you suggest and grab the latest lib/rrd.php and have a look

- Chris.

[senger]

What is the current situation with respect to this problem. I had been running 0.8.7.e. After the cacti problems related to the PHP update included in the OS X 10.6.4 upgrade I have now updated Cacti to 0.8.7.g and I am seeing the slow graphing times mentioned in this post. Is there a newer version of /lib/rrd.php that addresses this problem?

- Steve

[bosmacs]

I'm having the same problem with 10.6.4 and cacti 0.8.7g. Anyone have any status or viable workarounds?

[gandalf]

I'm having the same problem with 10.6.4 and cacti 0.8.7g. Anyone have any status or viable workarounds?

It is very important to do as adviced by Larry (TheWitness): run the rrdtool command from cli to compare.
There are performance issues with rrdtool 1.4.x, which we have to separate from and possible php/cacti issues.
R.

[bosmacs]

I should have mentioned, I tried this before posting; I took the shell command from the graph's debug page, and it returns almost instantly.

FWIW:

PHP 5.3.2 (cli)
RRDtool 1.4.3
Apache/2.2.14 (Unix)

[bosmacs]

I'm not sure if this helps, but I wrote a simple php script to exec/popen/proc_open/shell_exec/passthru rrdtool and they all return nearly instantly when run from a terminal. All of them likewise exhibit the same slow behavior (with rrdtool pegged at 100%) when accessed via the webserver/apache.

[gwk]

Well this is not a solution (it does fix the problem but not in a nice way) I did find some potentially more useful information. Testing a hunch I attempted to run the rrdtool command as the _www user on a 10.6.5 machine
flummoxed by the fact that _www has no login shell /usr/bin/false I switched the _www users shell to bash
"sudo dscl ." >"change /Users/_www /usr/bin/false /bin/bash" so that I might su to the user from a root shell and run the rrdtool command, the first time it ran just as slowly as it does when executed from the web however subsequent runs were very quick I suspected ok its just caching or something but then I restarted apache sudo launchctl stop org.apache.httpd sudo launchctl start org.apache.httpd and every graph loads very quickly with no appreciable spike in cpu utilization.

I suspect there might be some kind of sandboxing of the apache process going on as it running as a user with no login shell. I will keep looking since this is a little strange if the no login shell was that big of a problem I wouldn't exepct the calls to rrdtool to work at all.

[gandalf]

RRDTool 1.3++ is known to cache the fonts. If there's a huge font list, this may slow down processing significantly.
R.

[gwk]

That sounds plausible when I attached instruments to one of the running rrdtools processes while refreshing a large list of graphs (before the shell change for _www) almost 60% of the calls were in libfreetype

http://www.gwk.ca/images/instruments_rrdtool.png

[bosmacs]

@gwk, I tried your workaround to no avail (10.6., rrdtool 1.4.3). rrdtool processes are slow when run under _www, even after setting a login shell, and no matter how many times I run them. However, I do see similar behavior when run under my user account; slow the first time, very fast every subsequent time.

I also see a lot of time spent in FT_LoadGlyph when sampling the active rrdtool processes. Where/how does rrdtool cache the fonts? Incidently, this is a stock system -- no custom fonts installed.

[bosmacs]

Just another observation; I think it's definitely a font cache issue. fc-list returns 287 fonts immediately when run as a normal user. From a _www shell, it returns the same list after roughly 10 seconds.

[bosmacs]

Well, creating a _www-writable .fontconfig directory in /Library/WebServer made fc-list (and rrdtool) return quickly under the _www account (the second time). Cache files do appear in that directory, but web server performance is still slow.

[bosmacs]

OK, fc-cache -fv fixed the problem (there are now many cache files in .fontconfig directory I created, instead of two previously). Cacti graphs appear without any noticable delay. With that done, I've disabled the _www shell by setting it back to /usr/bin/false and everything appears to be humming along.