cacti-0.8.6i memory and performance issues (memory leak?).

[phizer]

I am experiencing a strange issue with poller.php runing out of memory on a box with 4 GB installed. I have been able to reproduce the same symptoms and behaviour on two Dell servers of identical hardware with different OS architectures (one x86 and one x86_64).

Dual Intel Xeon 3.4GHZ - EM64T
4 GB RAM
10K SCSI drives - hardware RAID 5

Server1:
Fedora Core 5
Linux cacti 2.6.18-1.2257.fc5smp #1 SMP i686 i686 i386 GNU/Linux
PHP 5.1.6
mysql Ver 14.12 Distrib 5.0.27, for redhat-linux-gnu (i686)
cacti-0.8.6i-5.fc5
cacti-cactid-0.8.6i-1

Server2:
Red Hat Enterprise Linux ES release 4 (Nahant Update 4)
Linux cacti 2.6.9-42.0.3.ELsmp #1 SMP x86_64 x86_64 x86_64 GNU/Linux
PHP 4.3.9
mysql Ver 14.7 Distrib 4.1.20, for redhat-linux-gnu (x86_64) using readline 4.3
cacti-0.8.6i
cacti-cactid-0.8.6i-1

Cacti on both boxes has these patches:
poller_output_remainder.patch
dec06-vulnerability-poller-0.8.6i.patch
dec06-vulnerability-scripts-0.8.6i.patch
import_template_argument_space_removal.patch

I have @ 14000 RRDs. Mysql is using a slightly modified my-HUGE.cnf (I have also used the default settings) config and I changed the storage engine over to InnoDB. Memory_limit is set to 128M in the active php.ini config. The LIMIT variable in (cacti-dir)/lib/poller.php is set to 50000.

The problem is that when I run (cacti-dir)/poller.php, after gathering data and while processing RRDs (long output of "OK" messages) the output then freezes. Before the freeze the system is under a reasonable and balanced load. Once it begins hanging Mysqld shoots up in CPU usage @ 80-100% while maintaining the prior mem usage of @ 584MB. Poller.php remains constant in CPU usage @ 25% but begins to grow in memory usage up to the physical limit of what is installed in the box, then the following error is output and poller.php exits out:

"Allowed memory size of 134217728 bytes exhausted (tried to allocate 10 bytes)"

The exhausted memory size in the error message is whatever size that memory_limit is set to in the loaded php.ini config, which I have set up to a value up to 4 GB, it only prolongs how long it will take to generate the error message and exit.

Again, this is reproducable on both servers, not sure what I am missing. Any input or assitance with this is very appreciated.

-Phizer

[Exo7]

Do you have custom template / data query using external scripts running ? I remember reading somewhere a version of a mysql template causing issues with memory and the mysql process.
You might get more info on where it is really hanging if you enable debug logging for 1 polling cycle, but due to the number of rrd you have, it might cause some trouble with disk space...

[phizer]

I am not using any external scripts. It hangs duing this SQL select:

01/23/2007 09:15:42 AM - CMDPHP: Poller[0] DEBUG: SQL Assoc: "select poller_output.output, poller_output.time, poller_output.local_data_id, poller_item.rrd_path, poller_item.rrd_name, poller_item.rrd_num from (poller_output,poller_item) where (poller_output.local_data_id=poller_item.local_data_id and poller_output.rrd_name=poller_item.rrd_name)

running this select statement from the command line I get the same hang from Mysqld, and then the Mysql client grows in memory usage in the same fashion that poller.php grows when this is run via cacti. So the problem lies within this select statement and the data in the poller_output and poller_item tables. Changing the my.cnf settings back to the origional default settings has no effect, so it doesnt seem to be a Mysql performance issue, it just doesnt like this query.

Any other thoughts?

[phizer]

It looks like the poller_output table was not purging itself, there were a couple million rows with data going back a few days in that table and this select statement could not execute. I dropped and re-created the poller_output table and all seems to work now.

Has anyone experienced this issue before?

[TheWitness]

It is manifested when a) you are running an older version of Cacti, but more importantly b) you don't have enough memory allocated to php (in php.ini file) to handle all the rows that are returned.

In order versions, once you go down that slippery slope, your poller output table just grows and grows because PHP has to exit.

TheWitness

[phizer]

Thanks Witness, I'll upgrade to 0.8.6j as well. Looking back when this manifested, memory_limit in the loaded php.ini file was set to 32M, it was changed to 128M as part of diagnostics after the table had already filled with stale output data but before the problem was identified, hence why dropping and re-creating the table was the last needed step.

Thanks for the definitive explanation of cause and affect.

[melchandra]

I believe I had this problem again running 8.6j - cacti stopped updating the RRD files. When I check the poller_output table it had 800,000 records in it. I emptied the table with phpmyadmin and everything went back to normal.

I've already increased my php.ini memory limit to 64Mb so that's not the problem. I'm only polling 4000 or so oids.

[TheWitness]

I have seen this number change between PHP versions. So, increase it and continue to watch. Depending on what version, I alway's truncate this table at the very end. So, there is a lot of depends:

Version of Cacti
Plugin Architecture
PHP Version

Are the biggest "If's".

TheWitness