A lot of zombie PHP processes

[Dyr]

I have a server, installed from CactiEZ and a bit of update (CentOS 4.3->4.6, fresh Apache, fresh NAGIOS, etc).
At this server working Cacti (with DB at another server) and NAGIOS.
All worked fine until this friday.
Suddenly at weekend, load average up to 250 (from normal 1-2), and "ps -ax" show me a lot of processes like that:

Code: Select all

23354 ?        Z      0:00 [php] <defunct> 
23355 ?        Z      0:00 [php] <defunct> 
23356 ?        Z      0:00 [php] <defunct> 
23357 ?        Z      0:00 [php] <defunct> 
23358 ?        Z      0:00 [php] <defunct> 
23359 ?        Z      0:00 [php] <defunct> 
23360 ?        Z      0:00 [php] <defunct> 
23361 ?        R      3:44 /usr/local/spine/spine 76 97 
23362 ?        Z      0:00 [php] <defunct> 
23363 ?        Z      0:00 [php] <defunct> 
23364 ?        R      3:52 /usr/local/spine/spine 98 117 
23365 ?        Z      0:00 [php] <defunct> 
23366 ?        Z      0:00 [php] <defunct> 
23367 ?        R      0:02 /usr/bin/php -q /var/www/html/plugins/syslog/syslog_process.php 
23368 ?        Z      0:00 [php] <defunct> 
23369 ?        Z      0:00 [php] <defunct> 
23370 ?        Z      0:00 [php] <defunct> 
23371 ?        Z      0:00 [php] <defunct> 
23372 ?        Z      0:00 [php] <defunct> 
23373 ?        Z      0:00 [php] <defunct> 
23374 ?        Z      0:00 [php] <defunct> 
23375 ?        Z      0:00 [php] <defunct> 
23376 ?        Z      0:00 [php] <defunct> 
23377 ?        Z      0:00 [php] <defunct> 
23378 ?        Z      0:00 [php] <defunct> 
23379 ?        Z      0:00 [php] <defunct> 
23380 ?        Z      0:00 [php] <defunct> 
23381 ?        Z      0:00 [php] <defunct> 
23382 ?        Z      0:00 [php] <defunct> 
23383 ?        Z      0:00 [php] <defunct> 
23384 ?        Z      0:00 [php] <defunct>

After I killed all php processes, they again come in two hours. =(
There are now graphs in this period, off cause.
I don't understand what happens and what should I do to fix.. =(
I don't change anything at this week.

Cacti Version - 0.8.7b
Plugin Architecture - 2.0
Poller Type - Cactid v
Server Info - Linux 2.6.9-67.0.22.ELsmp
Web Server - Apache/2.0.63 (CentOS)
PHP - 5.1.6
PHP Extensions - libxml, xml, wddx, tokenizer, sysvshm, sysvsem, sysvmsg, standard, SimpleXML, sockets, SPL, shmop, session, Reflection, pspell, posix, mime_magic, iconv, hash, gmp, gettext, ftp, exif, date, curl, ctype, calendar, bz2, zlib, pcre, openssl, apache2handler, gd, ldap, mysql, mysqli, PDO, pdo_mysql, pdo_sqlite, snmp, eAccelerator
MySQL - 5.0.51a-3ubuntu5.1
RRDTool - 1.2.23
SNMP - 5.1.2
Plugins

• Global Plugin Settings (settings - v0.5)
  Thresholds (thold - v0.3.9)
  Large Site Performane Booster for Cacti (boost - v1.5)
  Network Tools (tools - v0.3)
  Syslog Monitoring (syslog - v0.5.2)
  Device Tracking (mactrack - v1.1)
  RRD File Cleaner (RRD Cleaner - v0.36)
  Update Checker (update - v0.4)
  FlowView (flowview - v0.5.1)
  Host Info (hostinfo - v0.2)
  Error Images (errorimage - v0.1)
  Create Aggregate Graphs (aggregate - v0.63)
  Documents (docs - v0.1)
  report it! (report it! - v0.5.0)
  Realtime for Cacti (realtime - v0.34)
  PHP Network Managing (PHP Network Managing - v0.6)

Detailed system information attached below (cause formatting left out then insert as simple text)

[computer_guru]

I am having a similar issue on my CactiEZ 0.4 server running CentOS 4.7.


I believe this is caused by a bad update to httpd (Apache) or Php5. Bad meaning it causes issues, not corrupt.


I have 100+ zombie PHP processes right now. All created in the last few minutes. This will continue until the server crashes.


Apache Version: 2.0.63-2.e14s1.centos.2
PHP Version: 5.1.6-3.e14s1.10

I'm still trying to fix this issue.

[computer_guru]

this command killed all zombies:

ps -ef | grep spine | awk '{print "kill -9 "$2}' | sh


now trying to find permanent fix!

apparently spine is borken!

[computer_guru]

I think I found a solution! Reinstall latest version of Spine. Read article below for special instructions:

http://www.paulscomputerservice.net/art ... .php?ID=64

[philcordier]

I rebuilt spine per those instructions, and also upgraded to latest 0.8.7c, and still have the same problem. I had to put in a cron job to reboot every hour, otherwise the system just crawls to a halt....

Anyone have any better ideas??

I think I found a solution! Reinstall latest version of Spine. Read article below for special instructions:

http://www.paulscomputerservice.net/art ... .php?ID=64

[computer_guru]

Are you using the same version of Apache, PHP, and CentOS as me? Maybe your problem has similar symptoms, but different cause.

[istein001]

Seeing same issue... multiple dead zombie's for spine, all seem to have the "script server" still running with no input. When i kill off the old spine processes i get errors messages from them.

www# ps -aux | grep cacti
cacti 60891 10.9 0.9 55020 35788 ?? S 2:10PM 0:04.33 /usr/local/bin/php /usr/local/share/cacti/poller.php
cacti 60900 0.7 0.0 4236 1708 ?? D 2:10PM 0:00.46 /usr/local/bin/.libs/rrdtool -
cacti 40093 0.0 0.3 16524 10212 ?? I 7May09 0:00.44 /usr/local/bin/spine 0 356
cacti 40094 0.0 0.4 30444 15348 ?? I 7May09 0:00.23 /usr/local/bin/php -q /usr/local/share/cacti/script_server.php spine 0
cacti 40271 0.0 0.0 0 0 ?? Z 7May09 0:00.24 <defunct>
cacti 50028 0.0 0.4 23692 15216 ?? I Tue11AM 0:01.18 /usr/local/bin/spine 0 367
cacti 50029 0.0 0.4 30444 15348 ?? I Tue11AM 0:00.24 /usr/local/bin/php -q /usr/local/share/cacti/script_server.php spine 0
cacti 51527 0.0 0.0 0 0 ?? Z Tue11AM 0:00.24 <defunct>
cacti 52186 0.0 0.3 19596 13260 ?? I Thu12PM 0:00.69 /usr/local/bin/spine 0 55
cacti 52187 0.0 0.4 30444 15348 ?? I Thu12PM 0:00.23 /usr/local/bin/php -q /usr/local/share/cacti/script_server.php spine 0
cacti 52778 0.0 0.0 0 0 ?? Z Thu12PM 0:00.24 <defunct>
cacti 52807 0.0 0.0 0 0 ?? Z Thu12PM 0:00.24 <defunct>
cacti 52964 0.0 0.0 0 0 ?? Z Tue11AM 0:00.24 <defunct>
cacti 52966 0.0 0.0 0 0 ?? Z Tue11AM 0:00.24 <defunct>
cacti 60889 0.0 0.0 3456 1164 ?? Is 2:10PM 0:00.00 /bin/sh -c /usr/local/bin/php /usr/local/share/cacti/poller.php > /dev/nul
cacti 60899 0.0 0.0 3456 1172 ?? I 2:10PM 0:00.00 sh -c /usr/local/bin/rrdtool -
cacti 70088 0.0 0.3 20620 14168 ?? I Thu12PM 0:00.72 /usr/local/bin/spine 0 55
cacti 70089 0.0 0.4 30444 15348 ?? I Thu12PM 0:00.23 /usr/local/bin/php -q /usr/local/share/cacti/script_server.php spine 0
cacti 70469 0.0 0.0 0 0 ?? Z Thu12PM 0:00.24 <defunct>
cacti 70496 0.0 0.0 0 0 ?? Z Thu12PM 0:00.01 <defunct>
cacti 70762 0.0 0.0 0 0 ?? Z Thu12PM 0:00.24 <defunct>
root 65933 0.0 0.0 420 260 p0 D+ 2:11PM 0:00.00 grep cacti

www#killall -9 spine

cacti.log ->
05/18/2009 02:17:21 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
05/18/2009 02:17:21 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
05/18/2009 02:17:21 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating
05/18/2009 02:17:21 PM - PHPSVR: Poller[0] ERROR: Input Expected, Script Server Terminating

www# uname -a
FreeBSD www.somewhere.com 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Sun Feb 24 19:59:52 UTC 2008 root@logan.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386

Cacti Version 0.8.7d
SPINE 0.8.7b

[gandalf]

Please upgrade to spine087d. There have been fixes in that area of the code
Reinhard

[istein001]

I'm seeing the same issue with 0.8.7d and spine v0.8.7b. I have it setup with 1 concurrent polller process, 15 threads and only 1 php script server.

If i kill off the zombie spine processes cacti appears to speed up noticably. After checking the next day there are usually 1 to 4 new ones with defunct childrend.

Anyone else found a fix for this?