memory load is VERY high

[MPI]

first off, i'm not 100% sure if this is even cacti related but cacti was the only change that i made to the system before this issue started happenening.

basically, i installed cacti, everything was fine, then i couldn't use it for bandwidth monitoring so to uninstall, i basically removed the cacti folder and deleted the cron job. followed by removing the mysql username i had made for it....

i don't think is relevant but i'm on RHEL4

a day or two later, the server started crashing....i've been monitoring it since and there is a very high memory usage.

the server has 8gigs of ram.

right now apache is handling 1,428 requests and the server is using 75% of the ram!!!

i've hit a load of up to 3,500 requests and i've NEVER used 75% of ram...max was like 40% or 50%

first i thought it was apache but i havn't made any changes to apache...and as i said, the only change i had made was cacti...

can you guys think of anything that might have caused this to happen?

I'd appreciate any help.

[TheWitness]

My guess is that you should uninstall MySQL. It's pretty obvious it is something you did. Is there server internet facing and also, did you get affected by the recent exploit?

TheWitness

[MPI]

My guess is that you should uninstall MySQL.

how does this have ANYTHING to do with mysql? can you please explain?

Is there server internet facing?

what do you mean?

did you get affected by the recent exploit?

this one:
http://www.milw0rm.com/exploits/3029
...?

i just searched for "exploit" and that came up, did not know about it until now....what exactly does it do?

EDIT

holy crap, you mean someone got root shell access to my machine?! how can i find out what they've done?!!

[TheWitness]

1) Where is all the memory going? You should research that first
2) Internet facing, mean's someone from the internet can access it direcly (http port is exposed)

TheWitness

[MPI]

1) Where is all the memory going? You should research that first

this is all i can think of...

here is a snapshot of top sorted from high memory usage to low...just took it...70% memory usage, 1,580 connections.

http://www.MegaShare.com/images/15315
...now imagine 12,000 of those apache processes each taking 0.2% ram.

is there any other way to check to see where the memory is going?

2) Internet facing, mean's someone from the internet can access it direcly (http port is exposed)

then yes, it is but i had cacti in a folder like http://ip.address/cacti0.8.1 ....this was for OTHER reasons...how could this have happened?!

please help me fix this.

p.s. from what i've read i don't think what i'm about to say will matter but i have iptables enabled....and every port is blocked except for 80 and 443(ssl)

[TheWitness]

It is apparent that you have someone trying to "hack" your box. You have several connections to httpd. Don't know why or what they are doing.

Review your apache error and access logs to see what is going on. Sorry, but I can be of further assistance.

To clear the httpd connections, you would do the following (OS dependent)

Either:

/etc/init.d/httpd restart

or

/sbin/init.d/httpd restart

or

/etc/rc.d/init.d/httpd restart

TheWitness

[MPI]

It is apparent that you have someone trying to "hack" your box. You have several connections to httpd. Don't know why or what they are doing.

Review your apache error and access logs to see what is going on. Sorry, but I can be of further assistance.

To clear the httpd connections, you would do the following (OS dependent)

Either:

/etc/init.d/httpd restart

or

/sbin/init.d/httpd restart

or

/etc/rc.d/init.d/httpd restart

TheWitness

i've been up for the past 30 hours because i have to keep restarting apache every hour because it eats up all the memory.....i've already checked all the access logs and error logs with no luck.

at this point, is it obvious that someone got into the machine?

[MPI]

how can i check to see if my cmd.php file is vonurable?

because I think that i might have downloaded cacti AFTER the patch was released....

i still have the same copy that i installed on my server...found the tar.gz file in my temp folder.....it is "cacti-0.8.6i.tar.gz" ....is this the vonurable version?

also, i'm still curius...why did you tell me to re-install mysql?

[TheWitness]

All versions other than 0.8.6j are vulnerable without a patch. It it simply there as a GZ file or is it installed. I am quite convinced you don't know what you are doing... Sorry...

TheWitness

[MPI]

I am quite convinced you don't know what you are doing... Sorry...

i admit, i don't know what i'm doing but you know, i'm doing the best i can to figure this out.

its just there as a gz file....this is in a unaccessible location.......for uninstalling, is there a certain procedure thats supposed to be followed or would doing what i described in my 1st post do the job?

p.s when i stop apache, memory usage goes down to almost nothing...but when i fire it up again....connections shoot up to 1,000-1,500 andmemory makes its way up to 80%+

[MPI]

question about the exploit.

once the cacti folder is removed....the "hacker" loses control...right?

[MPI]

i addedd this and it seems to be helping....

RLimitMEM 1188431872
RLimitCPU 240

50% memory...1,500 connections but thats still too high

[MPI]

i addedd this and it seems to be helping....

RLimitMEM 1188431872
RLimitCPU 240

50% memory...1,500 connections but thats still too high

nevermind ...

[MPI]

My guess is that you should uninstall MySQL.

i'm still VERY curious on why you said this...because i just tried this...

httpd stop
service mysql stop
httpd start (note that i didn't turn on mysql)

i got to 1,500 connections and ram usage was around 10% so you were right, it has something to do with mysql.

but i'm curious as to why you said uninstall it....WHERE in mysql do you think the problem resides? and why would uninstalling fix it?

[TheWitness]

Other than the fact that my mind is in the machine, I don't have a clue. It's somewhat an act of clairvoyance or just plain luck...

TheWiitness