Mysql zombie attack killing cacti on Debian

theph1th · Post by **theph1th** » Thu Jul 14, 2005 3:37 am

First post to this group, but have been lurking to find out this problem in particular.
Cacti works great ..but not for long. Mysql falls over after a few hours with zombie processes, which start ramping in number (8 every 5 mins) after 2 hours or so. When they get to a few hundred the web page gives a mysql_connect() failure errno 11 (or similar message every time it refreshes).

Running Debian Sarge on a Linkstation (200MHz PPC) as a small soho server -hence no load issues.
Setup is: kernel 2.4.17 + cacti0.8.6f-2+cactid0.8.6.e +Mysql 4.0.24 installed via apt-get.

Although the Mysql defunct processes may not be associated with cacti - they do appear at nice five min intervals, and nothing else uses it since removed and reinstalled mysql, cacti. Have picked up some basic using of cacti but next to nothing about mysql so please be gentle.

Killing (killall -9 mysqld) the zombies and stop and start mysql and they just come back (like a classic horror movie) but a reboot leaves it clean for a couple of hours, then they start marching again! (help!)

There's no warnings/mention I can find in any logs, and would prefer not to have a cron job to kill and restart them.

Post by **TheWitness** » Thu Jul 14, 2005 11:36 am

Do you have hung Cactid processes as well? If not, I had the same problem with open connections closed automatically by setting a 5 minute timeout on any mysql connection. It is cleared up in MySQL 4.1.11 that I run at home though.

TheWitness

theph1th · Post by **theph1th** » Thu Jul 14, 2005 12:45 pm

Thanks for the reply.
I've no hung processes for cactid as far as I can see and its really by association since if no mysql was needed I wouldn't use it.

Lots of activemysqld processes running - is this correct?
I'll try and give more info..
There are 8 mysqld zombies made every 5 mins (same as the running sub processes.

ps axf output (mysql part):

12383 pts/0 S 0:00 /bin/sh /usr/bin/mysqld_safe
12402 pts/0 S 0:00 \_ /bin/sh /usr/bin/mysqld_safe
12403 pts/0 S 0:01 \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --p
id-file=/var/lib/mysql/kuro-box.pid --skip-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
12407 pts/0 S 0:00 | \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql
--pid-file=/var/lib/mysql/kuro-box.pid --skip-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
12408 pts/0 S 0:00 | \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=m
ysql --pid-file=/var/lib/mysql/kuro-box.pid --skip-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
12409 pts/0 S 0:00 | \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=m
ysql --pid-file=/var/lib/mysql/kuro-box.pid --skip-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
12410 pts/0 S 0:00 | \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=m
ysql --pid-file=/var/lib/mysql/kuro-box.pid --skip-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
12411 pts/0 S 0:00 | \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=m
ysql --pid-file=/var/lib/mysql/kuro-box.pid --skip-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
12412 pts/0 S 0:00 | \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=m
ysql --pid-file=/var/lib/mysql/kuro-box.pid --skip-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
12413 pts/0 S 0:00 | \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=m
ysql --pid-file=/var/lib/mysql/kuro-box.pid --skip-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
12414 pts/0 S 0:00 | \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=m
ysql --pid-file=/var/lib/mysql/kuro-box.pid --skip-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
12415 pts/0 S 0:00 | \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=m
ysql --pid-file=/var/lib/mysql/kuro-box.pid --skip-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
12416 pts/0 Z 0:00 | \_ [mysqld] <defunct>
12425 pts/0 Z 0:00 | \_ [mysqld] <defunct>
12436 pts/0 Z 0:00 | \_ [mysqld] <defunct>
12441 pts/0 Z 0:00 | \_ [mysqld] <defunct>
12470 pts/0 Z 0:00 | \_ [mysqld] <defunct>
12474 pts/0 Z 0:00 | \_ [mysqld] <defunct>
... ... ...

And cacti log has a typical entry:
07/14/2005 06:15:12 PM - SYSTEM STATS: Time: 6.1912 s, Method: cactid, Processes: 1, Threads: 5, Hosts: 5, Hosts/Process: 5, Data Sources 15, RRDs Processed 10
NB. I've only got 4 hosts - the server and 3 routers polled by snmp.

How might I set a 5min timeout if it would do the job?
Most things are held at Debian default install with the min to get working.

Answers always much appreciated

Graph attached to show what things do

Post by **TheWitness** » Thu Jul 14, 2005 3:13 pm

Edit your my.ini or my.conf (Windows vs. Linux) file and set the following:

wait_timeout=360

TheWitness

theph1th · Post by **theph1th** » Thu Jul 14, 2005 5:41 pm

TheWitness many thanks
Bit of a delay to watch what it was doing.
I've been trying to find the problem for a long time.
I put this in the mysqld section. and did /etc/init.d/mysql stop then start.
I'm usng the debian my-small.cnf file unchanged except for this (worth posting?)

Unfortunately the zombies are still coming.

The cacti 'debug' output in case it gives any clues:
07/14/2005 11:35:10 PM - CACTID: Poller[0] Host[2] SNMP Result: Host responded to SNMP
07/14/2005 11:35:10 PM - CACTID: Poller[0] DEBUG: In Poller, About to Start Polling of Host
07/14/2005 11:35:10 PM - CACTID: Poller[0] DEBUG: The Value of Active Threads is 5
07/14/2005 11:35:10 PM - CACTID: Poller[0] DEBUG: Valid Thread to be Created
07/14/2005 11:35:10 PM - CACTID: Poller[0] DEBUG: Falling back to UDP Ping due to inability to set asroot permissions
07/14/2005 11:35:10 PM - CACTID: Poller[0] MYSQL: Connected to MySQL database 'cacti' on '127.0.0.1'...
07/14/2005 11:35:10 PM - CACTID: Poller[0] DEBUG: Falling back to UDP Ping due to inability to set asroot permissions
07/14/2005 11:35:10 PM - CACTID: Poller[0] DEBUG: The Value of Active Threads is 4
07/14/2005 11:35:10 PM - CACTID: Poller[0] DEBUG: Valid Thread to be Created
07/14/2005 11:35:10 PM - CACTID: Poller[0] MYSQL: Connecting to MySQL database 'cacti' on '127.0.0.1'...
07/14/2005 11:35:10 PM - CACTID: Poller[0] DEBUG: In Poller, About to Start Polling of Host
07/14/2005 11:35:10 PM - CACTID: Poller[0] Host[1] DEBUG: The POPEN returned the following File Descriptor 14
07/14/2005 11:35:10 PM - CACTID: Poller[0] MYSQL: Connected to MySQL database 'cacti' on '127.0.0.1'...
07/14/2005 11:35:10 PM - CACTID: Poller[0] DEBUG: The Value of Active Threads is 3
07/14/2005 11:35:10 PM - CACTID: Poller[0] DEBUG: Valid Thread to be Created
07/14/2005 11:35:10 PM - CACTID: Poller[0] MYSQL: Connecting to MySQL database 'cacti' on '127.0.0.1'...
07/14/2005 11:35:10 PM - CACTID: Poller[0] DEBUG: In Poller, About to Start Polling of Host
07/14/2005 11:35:10 PM - CACTID: Poller[0] DEBUG: SQLCMD: update host set status='3',status_event_count='0', status_fail_date='0000-00-00 00:00:00',status_rec_date='0000-00-00 00:00:00',status_last_error='',min_time='0.000000',max_time='0.000000',cur_time='0.000000',avg_time='0.000000',total_polls='289',failed_polls='0',availability='100.0000' where id='1'
07/14/2005 11:35:10 PM - CACTID: Poller[0] Host[1] SNMP Result: Host does not require SNMP
07/14/2005 11:35:10 PM - CACTID: Poller[0] DEBUG: Falling back to UDP Ping due to inability to set asroot permissions

Post by **TheWitness** » Thu Jul 14, 2005 6:22 pm

You are running an old version of Cactid. Please upgrade.

TheWitness

Post by **TheWitness** » Thu Jul 14, 2005 6:23 pm

Also, please post your cactid.conf. You can sanitize the user/password sections.

TheWitness

theph1th · Post by **theph1th** » Fri Jul 15, 2005 8:35 am

Currently on cactid 0.8.6e but will attempt to compile the next one in this evening.
Nothing in the cactid.conf file apart from user/passwords really.
I changed the DB_Host from localhost to 127.0.0.1 in case that would help
but no joy. Database is the default cacti

Will let you know if next cactid helps.

thanks for your help

Post by **TheWitness** » Fri Jul 15, 2005 10:04 am

Since you are the only user experiencing this, I would suggest it has something to do with your build. Have you tried using cmd.php instead?

EDIT: How many concurrent processes are you running?

Thanks,

TheWitness

theph1th · Post by **theph1th** » Fri Jul 15, 2005 12:54 pm

It does seem very unusual (I've only seen it on one other posting) but they appeared to get round by doing something completely different.
Latest cactid stable is 0.8.6e too? I did also use cmd.php before with exactly the same effect.

The box is different in that its a hacked Linkstation (a hard disk with a very small PPCprocessor and network port meant for file sharing). Could it be something kernel related with a modified 2.4.17? I'll ask the guys who did the hack.

I didn't build it as such - just apt-get 'd them. This maybe the next thing to look at and will strip out and then build. Concurrent cactid processes were meant to be max 5.

thanks
downlala

Post by **TheWitness** » Fri Jul 15, 2005 4:32 pm

Power PC's are somewhat rare these days...

TheWitness

theph1th · Post by **theph1th** » Sun Jul 17, 2005 4:52 pm

Just a follow up..
Haven't got anything conclusive but will crow-bar in a cron job restart to keep it moving. Some PPCs are getting used now on embedded systems (low power & flexible etc).

I'm getting a feeling my issue probably isn't to do with Cacti, but it merely showed up a rarely occuring problem.

Many thanks for your input alll the same.

Thanks too for a great package
-hope to use it in long term remote hardware logging eventually.
P

Mysql zombie attack killing cacti on Debian

Mysql zombie attack killing cacti on Debian

Who is online