Kernel Panic when running cmd.php

Post general support questions here that do not specifically fall into the Linux or Windows categories.

Moderators: Developers, Moderators

Post Reply
hick
Posts: 3
Joined: Tue Jul 30, 2002 3:57 pm

Kernel Panic when running cmd.php

Post by hick »

Recently I started getting kernel panics after crontab runs:

/var/www/html/cacti/cmd.php > /dev/null 2>&1

How do I go about debuging this to find out what's wrong?

It happens when I run it manually from commandline also.

As this is recent and only happens with Cacti and it takes the entire server down after a day or so, to the point that a hard reboot is required as the system wont even allow a 'shutdown -r' command.

I've #'s the line out of crontab until I can figure out what's wrong.

Output is as follows:

/var/log/kernel

Code: Select all


Jul 30 15:00:08 homer kernel: Unable to handle kernel paging request at virtual address fbf7c070
Jul 30 15:00:08 homer kernel: Unable to handle kernel paging request at virtual address fbf7c070
Jul 30 15:00:08 homer kernel:  printing eip:
Jul 30 15:00:08 homer kernel:  printing eip:
Jul 30 15:00:08 homer kernel: c0154439
Jul 30 15:00:08 homer kernel: c0154439
Jul 30 15:00:08 homer kernel: *pde = 00000000
Jul 30 15:00:08 homer kernel: *pde = 00000000
Jul 30 15:00:08 homer kernel: Oops: 0000
Jul 30 15:00:08 homer kernel: Oops: 0000
Jul 30 15:00:08 homer kernel: Kernel 2.4.9-34
Jul 30 15:00:08 homer kernel: Kernel 2.4.9-34
Jul 30 15:00:08 homer kernel: CPU:    0
Jul 30 15:00:08 homer kernel: CPU:    0
Jul 30 15:00:08 homer kernel: EIP:    0010:[proc_pid_stat+281/592]    Not tainted
Jul 30 15:00:08 homer kernel: EIP:    0010:[proc_pid_stat+281/592]    Not tainted
Jul 30 15:00:08 homer kernel: EIP:    0010:[<c0154439>]    Not tainted
Jul 30 15:00:08 homer kernel: EIP:    0010:[<c0154439>]    Not tainted
Jul 30 15:00:08 homer kernel: EFLAGS: 00010206
Jul 30 15:00:08 homer kernel: EFLAGS: 00010206
Jul 30 15:00:08 homer kernel: EIP is at proc_pid_stat [kernel] 0x119
Jul 30 15:00:08 homer kernel: EIP is at proc_pid_stat [kernel] 0x119
Jul 30 15:00:08 homer kernel: eax: fbf7c000   ebx: 00000000   ecx: 00000008   edx: 00003309
Jul 30 15:00:08 homer kernel: eax: fbf7c000   ebx: 00000000   ecx: 00000008   edx: 00003309
Jul 30 15:00:08 homer kernel: esi: eeb60000   edi: c0119c42   ebp: 00000000   esp: db8e7f20
Jul 30 15:00:08 homer kernel: esi: eeb60000   edi: c0119c42   ebp: 00000000   esp: db8e7f20
Jul 30 15:00:08 homer kernel: ds: 0018   es: 0018   ss: 0018
Jul 30 15:00:08 homer kernel: ds: 0018   es: 0018   ss: 0018
Jul 30 15:00:08 homer kernel: Process w (pid: 4308, stackpage=db8e7000)
Jul 30 15:00:08 homer kernel: Process w (pid: 4308, stackpage=db8e7000)
Jul 30 15:00:08 homer kernel: Stack: 5a000212 00000000 ffffffff 00000000 00000000 00000000 00000000 00000000
Jul 30 15:00:08 homer kernel: Stack: 5a000212 00000000 ffffffff 00000000 00000000 00000000 00000000 00000000
Jul 30 15:00:08 homer kernel:        00000000 00000000 ef3aec80 000001ff eeb60000 d5816000 c015253c eeb60000
Jul 30 15:00:08 homer kernel:        00000000 00000000 ef3aec80 000001ff eeb60000 d5816000 c015253c eeb60000
Jul 30 15:00:08 homer kernel:        d5816000 00000000 f53d0000 00000000 e0667440 ffffffea 000001ff c0136016
Jul 30 15:00:08 homer kernel:        d5816000 00000000 f53d0000 00000000 e0667440 ffffffea 000001ff c0136016
Jul 30 15:00:08 homer kernel: Call Trace: [proc_info_read+76/256] proc_info_read [kernel] 0x4c
Jul 30 15:00:08 homer kernel: Call Trace: [proc_info_read+76/256] proc_info_read [kernel] 0x4c
Jul 30 15:00:08 homer kernel: Call Trace: [<c015253c>] proc_info_read [kernel] 0x4c
Jul 30 15:00:08 homer kernel: Call Trace: [<c015253c>] proc_info_read [kernel] 0x4c
Jul 30 15:00:08 homer kernel: [sys_read+150/256] sys_read [kernel] 0x96
Jul 30 15:00:08 homer kernel: [sys_read+150/256] sys_read [kernel] 0x96
Jul 30 15:00:08 homer kernel: [<c0136016>] sys_read [kernel] 0x96
Jul 30 15:00:08 homer kernel: [<c0136016>] sys_read [kernel] 0x96
Jul 30 15:00:08 homer kernel: [sys_open+125/176] sys_open [kernel] 0x7d
Jul 30 15:00:08 homer kernel: [sys_open+125/176] sys_open [kernel] 0x7d
Jul 30 15:00:08 homer kernel: [<c0135b3d>] sys_open [kernel] 0x7d
Jul 30 15:00:08 homer kernel: [<c0135b3d>] sys_open [kernel] 0x7d
Jul 30 15:00:08 homer kernel: [system_call+51/56] system_call [kernel] 0x33
Jul 30 15:00:08 homer kernel: [system_call+51/56] system_call [kernel] 0x33
Jul 30 15:00:08 homer kernel: [<c0106f6b>] system_call [kernel] 0x33
Jul 30 15:00:08 homer kernel: [<c0106f6b>] system_call [kernel] 0x33
Jul 30 15:00:08 homer kernel:
Jul 30 15:00:08 homer kernel:
Jul 30 15:00:08 homer kernel:
Jul 30 15:00:08 homer kernel: Code: 8b 50 70 eb 04 89 f6 31 d2 8b 46 34 50 8b 46 60 50 8b 86 20
Jul 30 15:00:08 homer kernel:
Jul 30 15:00:08 homer kernel: Code: 8b 50 70 eb 04 89 f6 31 d2 8b 46 34 50 8b 46 60 50 8b 86 20
Jul 30 15:00:10 homer kernel:  <1>Unable to handle kernel paging request at virtual address fbf7c070
Jul 30 15:00:10 homer kernel:  <1>Unable to handle kernel paging request at virtual address fbf7c070
Jul 30 15:00:10 homer kernel:  printing eip:
Jul 30 15:00:10 homer kernel:  printing eip:
Jul 30 15:00:10 homer kernel: c0154439
Jul 30 15:00:10 homer kernel: c0154439
Jul 30 15:00:10 homer kernel: *pde = 00000000
Jul 30 15:00:10 homer kernel: *pde = 00000000
Jul 30 15:00:10 homer kernel: Oops: 0000
Jul 30 15:00:10 homer kernel: Oops: 0000
Jul 30 15:00:10 homer kernel: Kernel 2.4.9-34
Jul 30 15:00:10 homer kernel: Kernel 2.4.9-34
Jul 30 15:00:10 homer kernel: CPU:    0
Jul 30 15:00:10 homer kernel: CPU:    0
Jul 30 15:00:10 homer kernel: EIP:    0010:[proc_pid_stat+281/592]    Not tainted
Jul 30 15:00:10 homer kernel: EIP:    0010:[proc_pid_stat+281/592]    Not tainted
Jul 30 15:00:10 homer kernel: EIP:    0010:[<c0154439>]    Not tainted
Jul 30 15:00:10 homer kernel: EIP:    0010:[<c0154439>]    Not tainted
Jul 30 15:00:10 homer kernel: EFLAGS: 00010206
Jul 30 15:00:10 homer kernel: EFLAGS: 00010206
Jul 30 15:00:10 homer kernel: EIP is at proc_pid_stat [kernel] 0x119
Jul 30 15:00:10 homer kernel: EIP is at proc_pid_stat [kernel] 0x119
Jul 30 15:00:10 homer kernel: eax: fbf7c000   ebx: 00000000   ecx: 00000008   edx: 00003309
Jul 30 15:00:10 homer kernel: eax: fbf7c000   ebx: 00000000   ecx: 00000008   edx: 00003309
Jul 30 15:00:10 homer kernel: esi: eeb60000   edi: c0119c42   ebp: 00000000   esp: cb535f20
Jul 30 15:00:10 homer kernel: esi: eeb60000   edi: c0119c42   ebp: 00000000   esp: cb535f20
Jul 30 15:00:10 homer kernel: ds: 0018   es: 0018   ss: 0018
Jul 30 15:00:10 homer kernel: ds: 0018   es: 0018   ss: 0018
Jul 30 15:00:10 homer kernel: Process ps (pid: 4353, stackpage=cb535000)
Jul 30 15:00:10 homer kernel: Process ps (pid: 4353, stackpage=cb535000)
Jul 30 15:00:10 homer kernel: Stack: 5a000212 00000000 ffffffff 00000000 00000000 00000000 00000000 00000000
Jul 30 15:00:10 homer kernel: Stack: 5a000212 00000000 ffffffff 00000000 00000000 00000000 00000000 00000000
Jul 30 15:00:10 homer kernel:        00000000 00000000 ef3aec80 000001ff eeb60000 d16fb000 c015253c eeb60000
Jul 30 15:00:10 homer kernel:        00000000 00000000 ef3aec80 000001ff eeb60000 d16fb000 c015253c eeb60000
Jul 30 15:00:10 homer kernel:        d16fb000 00000000 f53cf000 00000000 cdd4c140 ffffffea 000001ff c0136016
Jul 30 15:00:10 homer kernel:        d16fb000 00000000 f53cf000 00000000 cdd4c140 ffffffea 000001ff c0136016
Jul 30 15:00:10 homer kernel: Call Trace: [proc_info_read+76/256] proc_info_read [kernel] 0x4c
Jul 30 15:00:10 homer kernel: Call Trace: [proc_info_read+76/256] proc_info_read [kernel] 0x4c
Jul 30 15:00:10 homer kernel: Call Trace: [<c015253c>] proc_info_read [kernel] 0x4c
Jul 30 15:00:10 homer kernel: Call Trace: [<c015253c>] proc_info_read [kernel] 0x4c
Jul 30 15:00:10 homer kernel: [sys_read+150/256] sys_read [kernel] 0x96
Jul 30 15:00:10 homer kernel: [sys_read+150/256] sys_read [kernel] 0x96
Jul 30 15:00:10 homer kernel: [<c0136016>] sys_read [kernel] 0x96
Jul 30 15:00:10 homer kernel: [<c0136016>] sys_read [kernel] 0x96
Jul 30 15:00:10 homer kernel: [sys_open+125/176] sys_open [kernel] 0x7d
Jul 30 15:00:10 homer kernel: [sys_open+125/176] sys_open [kernel] 0x7d
Jul 30 15:00:11 homer kernel: [<c0135b3d>] sys_open [kernel] 0x7d
Jul 30 15:00:11 homer kernel: [<c0135b3d>] sys_open [kernel] 0x7d
Jul 30 15:00:11 homer ucd-snmp[706]: Connection from 127.0.0.1
Jul 30 15:00:11 homer kernel: [system_call+51/56] system_call [kernel] 0x33
Jul 30 15:00:11 homer kernel: [system_call+51/56] system_call [kernel] 0x33
Jul 30 15:00:11 homer kernel: [<c0106f6b>] system_call [kernel] 0x33
Jul 30 15:00:11 homer kernel: [<c0106f6b>] system_call [kernel] 0x33
Jul 30 15:00:11 homer ucd-snmp[706]: Connection from 127.0.0.1
Jul 30 15:00:11 homer kernel:
Jul 30 15:00:11 homer kernel:
Jul 30 15:00:11 homer kernel:
Jul 30 15:00:11 homer kernel: Code: 8b 50 70 eb 04 89 f6 31 d2 8b 46 34 50 8b 46 60 50 8b 86 20
Jul 30 15:00:11 homer kernel:
Jul 30 15:00:11 homer kernel: Code: 8b 50 70 eb 04 89 f6 31 d2 8b 46 34 50 8b 46 60 50 8b 86 20
Top
raX
Lead Developer
Posts: 2243
Joined: Sat Oct 13, 2001 7:00 pm
Location: Carlisle, PA
Contact:
Contact raX

Post by raX »

Ouch... The first thing you can do is tail cacti's log file when running cmd.php, this should tell you what processes cmd.php is executing.

The second thing I would try is to closely watch the output of 'ps fax' to see if cmd.php is spawning a sain amount of processes.

Other than that though, it is really impossible for cmd.php to soley cause your kernel to panic. Perhaps cmd.php is spawning a process which in turn causes the problem?

The fact that it takes days for the machine to die leads me to believe it is some sort of memory problem. The kernel errors looked very memory-like with such errors as:
Unable to handle kernel paging request at virtual address fbf7c070
Try running 'vmstat 1' while this is going on. What sort of I/O is going on?

I would be curious to hear what exactly is causing this. Of course, you could always try upgrading your kernel. Come to think of it, 2.4.9 was about the version where there was some major VM instability...

Tell me what you find.

-Ian
Top
hick
Posts: 3
Joined: Tue Jul 30, 2002 3:57 pm

Ok.. I will give it a try..

Post by hick »

Thanks for the reply...

I will take your advice and give it run over the next week with your suggestions and report back :-)

But I think you are correct because when I just tested the 'vmstat' command, I got a segmentation fault.

I will try replacing the DIMMS and see what happens...
Top
User avatar
youngmug
Posts: 15
Joined: Wed Jun 19, 2002 4:34 pm
Location: California, USA
Contact:
Contact youngmug

Post by youngmug »

What Linux Distro are you using? I am getting panics also, but they happen at 401a, which is when the daily cron job runs. After eliminating all the unnecessary jobs, I was able to run the box continuously for over 16 days before a fault. I am using MandrakeLinux 8.2 ProSuite Server for i586. Also, I am running Apache 2.0.36 and a PHP 4.2.2 CVS that I compiled myself.

If I find any more information on this issue, I will post on this thread. (I am looking at trying Cacti on the latest ML 9.00beta, which made several improvments with some of the processes).
Top
hick
Posts: 3
Joined: Tue Jul 30, 2002 3:57 pm

ok..

Post by hick »

Ok... It's been two weeks now since I have changed out the last DIMM in the memory chain and everything has been super stable.

Oh, to answer youngmugs question:

I am running RedHat 7.2.

So it seems my issue was with a DIMM that did not like that motherboard. I moved it to another computer and it works fine.
Top
Post Reply

Return to “Help: General”

Who is online

Users browsing this forum: No registered users and 4 guests