Memorize: image caching plugin

[johnrembo]

hi all,

I've recently got stuck on a small problem. I use aggregate plugin which generates huge images from variety of data-sources. Those images are continously being watched by service-desk team, moreover - several applications are fetching them as well.

I've wrote a small experimental plugin which caches images in shared-memory after they were watched. It also adds a small informational legend on the bottom of the image.

without memorize plug-in generating one image on quad-core machine with 4gigs of ram takes from 2.5 to 6 seconds (depending on poller is running or not). With memorize plugin installed, cached images are returned within 1 microsecond and less (from 4000 to 7000 times faster).
Browser renders images on the fly and pooler gets extra cpu resources, while RAM consumption is relatively small.

moreover, png images are compressed up to the max, so average image size drops ~10%:
without memorize: 91.73 KB (93927 bytes)
with memorize: 78.96 KB (80853 bytes)

here are some examples:

when poller isn't running:

when poller is running:


after simple refresh memorize starts doing it's job:

poller isn't running:

poller is running:


memorize statistics when poller isn't running:

Code: Select all

displayed graphs count:	4
rendered from rrd:	2
rendered from cache:	2
avg rrd render time:	2.600489
avg cache render time:	0.000336
benefit(times): 	7739.550595
used cache memory size:	361947

memorize statistics when poller is running:

Code: Select all

displayed graphs count:	4
rendered from rrd:	2
rendered from cache:	2
avg rrd render time:	6.138474
avg cache render time:	0.001444
benefit(times): 	4251.020776
used cache memory size:	361982

one big disadvantage of memorize is that it doesn't run on windows falimy servers because php has no build-in support to shared memory on windows os.

after downloading it, please see "README". Most likely you will have to tune some kernel parameters to cache more images.

Also, make sure you have the following extensions enabled in php.ini (or statically compiled-in):
php-gd2 with FreeType lib - to see statistical information on image
shmop - to cache images in memory
sysvsem
sysvshm

this plugin was tested with cacti 0.8.7b + PIA 2.1
on freebsd-7 and debian-4 systems (+reported working on gentoo)
running on php5.2 + mysql 5.1+apache 2.2
whithout any aditional kernel tunig

[streaker69]

I'm getting an error:

Warning: Call-time pass-by-reference has been deprecated - argument passed by value; If you would like to pass it by reference, modify the declaration of [runtime function name](). If you would like to enable call-time pass-by-reference, you can set allow_call_time_pass_reference to true in your INI file. However, future versions may not support this any longer. in /var/www/html/plugins/memorize/setup.php on line 117

I think this is gonna be a great plugin though. It would be amazing if it could be implemented into core Cacti code and enabled based upon OS that's running Cacti.

Edit: nevermind, fixed it. Had to change the setting in php.ini for 'allow_call_time_pass_reference"

I think that's normally off by default when PHP is installed.

[Howie]

one big disadvantage of memorize is that it doesn't run on windows falimy servers because php has no build-in support to shared memory on windows os.

It could always fall back to a file-backed cache - statically serving the image is still going to be quicker than a trip to rrdtool-land. This is what Boost does.

[streaker69]

one big disadvantage of memorize is that it doesn't run on windows falimy servers because php has no build-in support to shared memory on windows os.

It could always fall back to a file-backed cache - statically serving the image is still going to be quicker than a trip to rrdtool-land. This is what Boost does.

Does Weathermap currently do anything like this? (hint hint).

[Howie]

one big disadvantage of memorize is that it doesn't run on windows falimy servers because php has no build-in support to shared memory on windows os.

It could always fall back to a file-backed cache - statically serving the image is still going to be quicker than a trip to rrdtool-land. This is what Boost does.

Does Weathermap currently do anything like this? (hint hint).

Yes

It "caches" the map to the output/ folder
It can use poller_output to save running rrdtool
If you have another mechanism for caching graphs transparently (Boost, this plugin), then it will use that.

What are you hinting at? It's too subtle for me...

[streaker69]

Yes

It "caches" the map to the output/ folder
It can use poller_output to save running rrdtool
If you have another mechanism for caching graphs transparently (Boost, this plugin), then it will use that.

What are you hinting at? It's too subtle for me...

Ahh, I didn't realize it was caching, must just be my slow Cacti box that makes it seem like it's not.

[johnrembo]

I'm getting an error:

Warning: Call-time pass-by-reference has been deprecated - argument passed by value; If you would like to pass it by reference, modify the declaration of [runtime function name](). If you would like to enable call-time pass-by-reference, you can set allow_call_time_pass_reference to true in your INI file. However, future versions may not support this any longer. in /var/www/html/plugins/memorize/setup.php on line 117

I think this is gonna be a great plugin though. It would be amazing if it could be implemented into core Cacti code and enabled based upon OS that's running Cacti.

Edit: nevermind, fixed it. Had to change the setting in php.ini for 'allow_call_time_pass_reference"

I think that's normally off by default when PHP is installed.

damn! i've forgotten to revert php settings to their defaults while testing :\
it whould be enough to remove ampersand on line 117

anyway, thanx for the tip - i've reuploaded the file (no need for you to redownload).

[johnrembo]

one big disadvantage of memorize is that it doesn't run on windows falimy servers because php has no build-in support to shared memory on windows os.

It could always fall back to a file-backed cache - statically serving the image is still going to be quicker than a trip to rrdtool-land. This is what Boost does.

hmm...now i'm thinking maybe I was wrong?

Note: Versions of Windows previous to Windows 2000 do not support shared memory. Under Windows, Shmop will only work when PHP is running as a web server module, such as Apache or IIS (CLI and CGI will not work).

is there anyone running windows installation? it would be nice to see what this plugin vomits about it. maybe there still is a hope...

[Howie]

My suggestion would be to do this like the perl CPAN Cache::* modules - so you'd have a Cache object which provides Store, Delete, Fetch/Exists and Flush functions, then a couple of (simple) backends for file and SHM. That way the 'front' part of the plugin doesn't change, and the back is nice and neat.

Also, you might want to switch from Arial to Vera or DeJa Vu, since I don't think it's legal to redistribute Arial (especially for something that doesn't actually work in Windows ) - Vera and DeJa Vu are free to distribute with open source software.

I just tried it on my test server, and it broke all my graphs, but that's probably due to sysctl settings. On FreeBSD, some of those things need a reboot, which I don't want to do right now.

[johnrembo]

My suggestion would be to do this like the perl CPAN Cache::* modules - so you'd have a Cache object which provides Store, Delete, Fetch/Exists and Flush functions, then a couple of (simple) backends for file and SHM. That way the 'front' part of the plugin doesn't change, and the back is nice and neat.

Also, you might want to switch from Arial to Vera or DeJa Vu, since I don't think it's legal to redistribute Arial (especially for something that doesn't actually work in Windows ) - Vera and DeJa Vu are free to distribute with open source software.

I just tried it on my test server, and it broke all my graphs, but that's probably due to sysctl settings. On FreeBSD, some of those things need a reboot, which I don't want to do right now.

thanx for the tips Howie. I'll note that.

this thing is far from perfect and, trully speaking, it have never had to reach a day light, but, assuming it did - i'll have to develop it now

ps: this plugin was originaly developed on debian-linux machine and now seems to work fine on 5 different servers. I simply had no opportunity to try this on freebsd box because got stuck i recompiling php with gd support;). I'll try it asap.

[Howie]

this thing is far from perfect and, trully speaking, it have never had to reach a day light, but, assuming it did - i'll have to develop it now

Hehe. See what you have done?

I simply had no opportunity to try this on freebsd box because got stuck i recompiling php with gd support;). I'll try it asap.

I'm happy to test on FreeBSD for you - just not on my mail server I have a spare FreeBSD system to play with.

[frdrk]

Hi, this seems like great plugin . I'm trying it out now on my gentoo box with 0.8.7b but I'm getting broken images. I'm gonna try recompile php with external gd-support (i think i used the internal before), and see if things work better.

edit:

Hm, no.. still no images.

Code: Select all

gd
GD Support 	enabled
GD Version 	2.0 or higher
FreeType Support 	enabled
FreeType Linkage 	with freetype
FreeType Version 	2.3.7
T1Lib Support 	enabled
GIF Read Support 	enabled
GIF Create Support 	enabled
JPG Support 	enabled
PNG Support 	enabled
WBMP Support 	enabled

That should work, right?

When I'm viewing a graph FF tells me that "Blablabla can't be viewed because it contains errors."

Cache statistics shows nothing;

Code: Select all

displayed graphs count:	0
rendered from rrd:	0
rendered from cache:	0
avg rrd render time:	0
avg cache render time:	0
benefit(times): 	0
used cache memory size:	0

[johnrembo]

Hi, this seems like great plugin . I'm trying it out now on my gentoo box with 0.8.7b but I'm getting broken images. I'm gonna try recompile php with external gd-support (i think i used the internal before), and see if things work better.

edit:

Hm, no.. still no images.

Code: Select all

gd
GD Support 	enabled
GD Version 	2.0 or higher
FreeType Support 	enabled
FreeType Linkage 	with freetype
FreeType Version 	2.3.7
T1Lib Support 	enabled
GIF Read Support 	enabled
GIF Create Support 	enabled
JPG Support 	enabled
PNG Support 	enabled
WBMP Support 	enabled

That should work, right?

When I'm viewing a graph FF tells me that "Blablabla can't be viewed because it contains errors."

Cache statistics shows nothing;

Code: Select all

displayed graphs count:	0
rendered from rrd:	0
rendered from cache:	0
avg rrd render time:	0
avg cache render time:	0
benefit(times): 	0
used cache memory size:	0

i've upgraded my post. forgot to mension the most important extension - shmop

[frdrk]

i've upgraded my post. forgot to mension the most important extension - shmop

Haha . Remerging php now with the +sharedmem use-flag

edit

ok, there we go. Niice

Code: Select all

displayed graphs count:	18
rendered from rrd:	9
rendered from cache:	9
avg rrd render time:	1.300332
avg cache render time:	0.000818
benefit(times): 	1589.647922
used cache memory size:	518870

To get this running on gentoo one has to emerge php with the sharedmem and sysvipc useflags.

[bodo.bach]

Hi,

thanx for your nice plugin.

I am using cacti 0.87b und PA 2.0 on cactiEZ 0.4

1. in Settings/Memorize if pressing SHOW or FLUSH buttom gives in the Browser-Statusline "ERROR on page." and nothing is happend. Seemed to be a Java compatibility problem, i am using IE7 & IE6 with JRE 1.607 and JRE 1.5

2. It speeds up the graphs, but sometimes, calling graphs for a device the first time, it shows no graphs (placeholder is shown), then i press the refresh buttom in the browser .... now the graphs are shown......

You have a idea to fix it ?