Cacti Not Freeing up Semaphores???

[MikeSullivan]

I installed cacti about two weeks ago. Recently I was removing some virtual hosts from my httpd.conf file and then restarted apache. I was surprised when I received the following error:

Ouch! ap_mm_create(1048576, "/var/run/httpd.mm.27323") failed
Error: MM: mm:core: failed to acquire semaphore (No space left on device): OS: Invalid argument

Anyway, after running the error through Google, I discovered the "ipcs" command and how to determine what was using up the semaphores:

tux:~$ ipcs

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status

------ Semaphore Arrays --------
key semid owner perms nsems status
0x00000000 1028 getreg 600 1
0x00000000 1029 getreg 600 1
0x00000000 1030 getreg 600 1
0x00000000 1031 getreg 600 1

//-- snipped the other 500 lines --//
(there was over 500 lines for the getreg user)

The user "getreg" is the user that the cacti cron job runs under. I then used a little script to free up the semaphores:

#!/bin/sh
ipcs | egrep getreg | awk '{print $2}' | xargs -n1 ipcrm sem;
ipcs | egrep getreg | awk '{print $1}' | xargs -n1 ipcrm sem

After that, apache starts up again just fine. Very quickly though, all of the semaphores are again used up.

Does anyone have any ideas on how to remedy this?? I have searched the net high and low for possible configurations problems, but there just isn't much info available.

Thanks!

[raX]

I've never run into this problem myself, even with cacti installations that have been running for over a year. I am suspecting that the cause could be with a script you are calling to gather data (I could be wrong though).

If I were going to debug this problem, that is the first place I would check.

-Ian

[MikeSullivan]

The only scripts I am currently using are the ones that came with cacti. I am only monitorring three routers, two switches (36 total ports), and the box on which cacti is installed. They were all setup using the SNMP interface scripts under the "Data Gathering" category of the console. Is it possible I am monitoring too much?

I didn't mean to imply necessarily that cacti is the culprit, but I am baffled as to what could be wrong and was hoping someone else had run into this particular problem. I have managed a workaround with a cron job to free up the semaphores in the meantime, but would really like to find the root cause of the problem.

Any more ideas??

[phriday613]

this is an old thread, but ill post.. who knows, it may help someone else!


i have a problem with semaphores as well.. my webserver/fileserver i have x server on. In using KDE, i noticed i have 468 processes running, at once!!! it was killing everything!! apache had 50mb along with php, mysql and perl, i had no memory left! 128mb is in, but it should be fine!

first, i found a temporary fix with this little command:

echo 250 32000 32 250 > /proc/sys/kernel/sem

also, i switched over to gnome.. I know i know, i shouldnt be running x on a webserver or fileserver, but the box was a learning tool before it became a server of any kind.. since, i just left it on

hope this helps anyone..
-michael

[raX]

I must admit I have seen this kind of thing happen before as well. It really only happens when there is some sort of network outtage causing timeouts all over the place. Since cmd.php is not multi-threaded it just waits for each process to finish. Then after five minutes another cmd.php starts, many times before the previous one has finished. Hence your sitution...

The obvious short term fix for this problem is to have cmd.php use a lock file and not spawn if another copy is not currently running. The long term solution is to make cmd.(c|php|pl) multi-threaded.

-Ian

[deadzen]

that's a freebsd issue

[gilson]

that's a freebsd issue

I have the same problem on RH73 and RH80.

If you need to restart Apache and there is no semaphores available, you can try this script in Red Hat (restart_apache.sh):

Code: Select all

#!/bin/sh
service httpd stop
for a in `ipcs -s|grep apache|awk '{print $2}'`; do ipcrm sem $a; done
for a in `ipcs -m|grep apache|awk '{print $2}'`; do ipcrm shm $a; done
service httpd start

-Gilson

[gregm]

This problem is actaully a known bug between apache and MM.

Some users have it, some don't.. i had it, and downgraded, and it seems to be working..

Rather than explain it here, I'll link you to this thread in another forum:

http://forum.plesk.com/showthread.php?s=&threadid=6226