PHP :: Bug #56720 :: Apache Children Crash

Bug #56720	Apache Children Crash - Signal 10
Submitted:	2005-12-13 02:36 UTC	Modified:	2006-03-03 20:53 UTC
From:	alain dot russell at blackpepper dot co dot nz	Assigned:
Status:	Closed	Package:	APC (PECL)
PHP Version:	5.1.1	OS:	Mac OSX
Private report:	No	CVE-ID:	None

View Add Comment Developer Edit

[2005-12-13 02:36 UTC] alain dot russell at blackpepper dot co dot nz

Description:
------------
After about 20 hours of running I see a lot of errors in the 
apache error_log as below:
[Tue Dec 13 18:38:06 2005] [notice] child pid 7430 exit 
signal Bus error (10)

We have recently switched from eaccelerator because it was 
segfaulting and eventually caused the server to stop 
responding totally.

Apache seems to still respond to some PHP pages when I start 
to see the errors but others just crash straight away. Quite 
often when the errors start to regularly appear in the log I 
see a few of these starting to appear as well:

[Tue Dec 13 13:22:04 2005] [apc-warning] GC cache entry '/
Library/WebServer/Documents/tizwine/tiz1.peppersuite.co.nz/
cache/byid/137.php' (dev=234881026 ino=16316034) was on gc-
list for 3752 seconds
/private/tmp/tmpIoAEhe/APC-3.0.8/apc_cache.c:730: failed 
assertion `entry->ref_count == 0'
[Tue Dec 13 13:22:04 2005] [notice] child pid 23651 exit 
signal Abort trap (6)

There are 2 ways to fix it, One is clean the APC cache, the 
other is to restart Apache. Either way will work fine and 
Apache runs error free for another 15-20 hours.

When Apache is erroring there is also a lot of fragmentation 
in the apc.php dashboard page .. upwards of 45-50% at times.

I've also noticed that I dont get any files in /tmp even 
though I have this setup below ?

I'm running Apache 2.0.55, PHP 5.1.1, APC 3.0.8.

APC Config
extension="apc.so"
apc.enabled=1
apc.enable_cli=1
apc.shm_segments=1
apc.optimization=0
apc.shm_size=64
apc.ttl=7200
apc.user_ttl=7200
apc.num_files_hint=2000
apc.mmap_file_mask=/tmp/apc.XXXXXX

Please let me know if any more info can be supplied to help 
tracking this down.

Expected result:
----------------
Apache to keep responding

Actual result:
--------------
Apache crashes after a while

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports

[2005-12-13 02:58 UTC] rasmus@php.net

APC unlinks the file after mmaping it to ensure the OS will clean it up properly on exit.  It's a common shared memory allocation trick.

Which MPM are you running with Apache2?  If it is anything other than prefork, all bets are off.

If you can reproduce this, attach gdb to one of these Apache children and try to get it to crash.  Then type "bt" and get me a backtrace.  Also do a "p entry".  This smells like a memory corruption which doesn't necessarily have anything to do with APC.  Do you have any exotic extensions loaded?

[2005-12-13 03:07 UTC] alain dot russell at blackpepper dot co dot nz

Apache MPM is prefork .. 
Not really an exotic PHP build I dont think - config below

./configure \
--prefix=/usr/local/php5 \
--enable-sockets \
--enable-calendar \
--with-zlib \
--with-xml \
--with-libxml-dir=/opt/local \
--with-gd \
--with-jpeg-dir=/opt/local \
--with-png-dir=/opt/local \
--with-mysql=/usr/local/mysql \
--with-mysqli=/usr/local/mysql/bin/mysql_config \
--with-apxs2=/usr/local/apache2/bin/apxs \
--with-pspell=/opt/local \
--with-openssl=/usr \
--with-zlib=/usr \
--with-mcrypt=/opt/local \
--with-curl=/opt/local \
--with-bz2=/opt/local \
--disable-cgi \
--with-mhash=/opt/local \
--enable-mbstring=all \
--enable-mbregex \
--enable-ftp \
--enable-bcmath \
--with-imap=/users/admin/documents/imap-2004a \
--with-imap-ssl=/users/admin/documents/imap-2004a \
--disable-debug

I've got a Crash report that Mac OSX generates as follows - 
is this the same as a backtrace ?

Date/Time:      2005-12-13 19:54:47 +1300
OS Version:     10.3.9 (Build 7W98)
Report Version: 2

Command: httpd
Path:    /usr/local/apache2/bin/httpd
Version: ??? (???)
PID:     10297
Thread:  0

Exception:  EXC_BAD_ACCESS (0x0001)
Codes:      KERN_PROTECTION_FAILURE (0x0002) at 0x0000006c

Thread 0 Crashed:
0   libphp5.so  0x02d0ab0c _efree + 0x144
1   libphp5.so  0x02d2de64 _zval_dtor_func + 0xb4
2   libphp5.so  0x02d205bc _zval_dtor + 0x38
3   libphp5.so  0x02d1c980 _zval_ptr_dtor + 0x60
4   libphp5.so  0x02d3db1c zend_hash_apply_deleter + 0x80
5   libphp5.so  0x02d3ddd8 
zend_hash_graceful_reverse_destroy + 0x40
6   libphp5.so  0x02d1c044 shutdown_executor + 0xe0
7   libphp5.so  0x02d301a0 zend_deactivate + 0x10c
8   libphp5.so  0x02ccc6d8 php_request_shutdown + 0x40c
9   libphp5.so  0x02dd6318 php_apache_request_dtor + 0x20
10  libphp5.so  0x02dd6afc php_handler + 0x61c
11  httpd       0x000228c4 ap_run_handler + 0x74
12  httpd       0x000231dc ap_invoke_handler + 0xfc
13  httpd       0x00008464 ap_process_request + 0x78
14  httpd       0x00002830 ap_process_http_connection + 0x90
15  httpd       0x0001b738 ap_run_process_connection + 0x74
16  httpd       0x00009c90 child_main + 0x4c4
17  httpd       0x00009e40 make_child + 0x140
18  httpd       0x0000a134 perform_idle_server_maintenance + 
0x1e4
19  httpd       0x0000a60c ap_mpm_run + 0x468
20  httpd       0x0000be48 main + 0x894
21  httpd       0x00001e24 _start + 0x188 (crt.c:267)
22  dyld        0x8fe1a278 _dyld_start + 0x64

PPC Thread State:
  srr0: 0x02d0ab0c srr1: 0x0200f030                vrsave: 
0x00000000
    cr: 0x24000224  xer: 0x00000000   lr: 0x02d0a9e4  ctr: 
0x02d2de48
    r0: 0x005d5110   r1: 0xbfffb3d0   r2: 0x02109970   r3: 
0x02109980
    r4: 0x02108850   r5: 0xbfffb410   r6: 0x031d6680   r7: 
0x00000001
    r8: 0x031d92f0   r9: 0x0000006c  r10: 0x031dbf0c  r11: 
0x00000006
   r12: 0x02d1c920  r13: 0x04952e18  r14: 0x000407dc  r15: 
0x00043570
   r16: 0x000407dc  r17: 0x0003e108  r18: 0x000407f4  r19: 
0x00040808
   r20: 0x000407d8  r21: 0x000397d4  r22: 0x0003e2c8  r23: 
0x0004081c
   r24: 0x0003e164  r25: 0x0003e1f8  r26: 0x00000000  r27: 
0x0495ae50
   r28: 0x000410b0  r29: 0x031d6a00  r30: 0xbfffb3d0  r31: 
0x02d0a9e4

Binary Images Description:
    0x1000 -    0x3dfff httpd   /usr/local/apache2/bin/httpd
   0x61000 -    0x62fff mod_access.so   /usr/local/apache2/
modules/mod_access.so
   0x65000 -    0x66fff mod_auth.so     /usr/local/apache2/
modules/mod_auth.so
   0x69000 -    0x69fff mod_auth_anon.so        /usr/local/
apache2/modules/mod_auth_anon.so
   0x6c000 -    0x6dfff mod_auth_dbm.so         /usr/local/
apache2/modules/mod_auth_dbm.so
   0x70000 -    0x70fff mod_env.so      /usr/local/apache2/
modules/mod_env.so
   0x74000 -    0x84fff libaprutil-0.0.dylib    /usr/local/
apache2/lib/libaprutil-0.0.dylib
   0x8c000 -    0x8dfff mod_expires.so  /usr/local/apache2/
modules/mod_expires.so
   0x90000 -    0x91fff mod_headers.so  /usr/local/apache2/
modules/mod_headers.so
   0x95000 -    0x9afff mod_auth_digest.so      /usr/local/
apache2/modules/mod_auth_digest.so
   0x9f000 -    0xcffff libssl.0.9.7.dylib      /sw/lib/
libssl.0.9.7.dylib
   0xe2000 -    0xe4fff mod_deflate.so  /usr/local/apache2/
modules/mod_deflate.so
   0xe7000 -    0xe9fff mod_ext_filter.so       /usr/local/
apache2/modules/mod_ext_filter.so
   0xec000 -    0xeffff mod_log_config.so       /usr/local/
apache2/modules/mod_log_config.so
   0xf3000 -    0xf7fff mod_mime_magic.so       /usr/local/
apache2/modules/mod_mime_magic.so
   0xfa000 -    0xfbfff mod_usertrack.so        /usr/local/
apache2/modules/mod_usertrack.so
  0x205000 -   0x20cfff mod_include.so  /usr/local/apache2/
modules/mod_include.so
  0x210000 -   0x211fff mod_setenvif.so         /usr/local/
apache2/modules/mod_setenvif.so
  0x214000 -   0x214fff mod_asis.so     /usr/local/apache2/
modules/mod_asis.so
  0x219000 -   0x21bfff mod_mime.so     /usr/local/apache2/
modules/mod_mime.so
  0x21e000 -   0x21ffff mod_vhost_alias.so      /usr/local/
apache2/modules/mod_vhost_alias.so
  0x222000 -   0x222fff mod_dir.so      /usr/local/apache2/
modules/mod_dir.so
  0x227000 -   0x242fff libexpat.0.dylib        /usr/local/
apache2/lib/libexpat.0.dylib
  0x249000 -   0x249fff libpspell.15.dylib      /opt/local/
lib/libpspell.15.dylib
  0x24c000 -   0x250fff mod_status.so   /usr/local/apache2/
modules/mod_status.so
  0x253000 -   0x258fff mod_autoindex.so        /usr/local/
apache2/modules/mod_autoindex.so
  0x25c000 -   0x25dfff mod_alias.so    /usr/local/apache2/
modules/mod_alias.so
  0x260000 -   0x261fff upload_progress_meter.so        /
usr/local/php5/lib/php/extensions/no-debug-non-zts-20050922/
upload_progress_meter.so
  0x266000 -   0x266fff mod_auth_cookie.so      /usr/local/
apache2/modules/mod_auth_cookie.so
  0x26d000 -   0x285fff libapr-0.0.dylib        /usr/local/
apache2/lib/libapr-0.0.dylib
  0x291000 -   0x296fff mod_negotiation.so      /usr/local/
apache2/modules/mod_negotiation.so
  0x29a000 -   0x29cfff mod_auth_mysql.so       /usr/local/
apache2/modules/mod_auth_mysql.so
  0x2aa000 -   0x2b5fff mod_rewrite.so  /usr/local/apache2/
modules/mod_rewrite.so
  0x2bb000 -   0x2dbfff mod_ssl.so      /usr/local/apache2/
modules/mod_ssl.so
  0x2fb000 -   0x30bfff libz.1.dylib    /opt/local/lib/libz.
1.dylib
  0x30f000 -   0x319fff mod_webcatalog2.so      /usr/local/
apache2/cgi-bin/WebCatalogEngine/mod_webcatalog2.so
  0x32f000 -   0x3fdfff libcrypto.0.9.7.dylib   /sw/lib/
libcrypto.0.9.7.dylib
  0x45c000 -   0x489fff libmhash.2.dylib        /opt/local/
lib/libmhash.2.dylib
  0x48e000 -   0x4b5fff libpng.3.dylib  /opt/local/lib/
libpng.3.dylib
  0x4de000 -   0x4ebfff libbz2.1.0.dylib        /opt/local/
lib/libbz2.1.0.dylib
  0x605000 -   0x625fff libmcrypt.4.dylib       /opt/local/
lib/libmcrypt.4.dylib
  0x66e000 -   0x6a7fff libssl.0.9.8.dylib      /opt/local/
lib/libssl.0.9.8.dylib
  0x6be000 -   0x6e1fff libjpeg.62.dylib        /opt/local/
lib/libjpeg.62.dylib
  0x725000 -   0x732fff apc.so  /usr/local/php5/lib/php/
extensions/no-debug-non-zts-20050922/apc.so
 0x1008000 -  0x10ddfff libiconv.2.dylib        /opt/local/
lib/libiconv.2.dylib
 0x1278000 -  0x136afff libcrypto.0.9.8.dylib   /opt/local/
lib/libcrypto.0.9.8.dylib
 0x158e000 -  0x15b8fff libcurl.3.dylib         /opt/local/
lib/libcurl.3.dylib
 0x2985000 -  0x3030fff libphp5.so      /usr/local/apache2/
modules/libphp5.so
 0x3641000 -  0x373bfff libxml2.2.dylib         /opt/local/
lib/libxml2.2.dylib
 0x3d51000 -  0x3e14fff libaspell.15.dylib      /opt/local/
lib/libaspell.15.dylib
0x20470000 - 0x20545fff libiconv.2.dylib        /sw/lib/
libiconv.2.dylib
0x76080000 - 0x76085fff libltdl.3.dylib         /usr/lib/
libltdl.3.dylib
0x8fe00000 - 0x8fe4ffff dyld    /usr/lib/dyld
0x93a50000 - 0x93a54fff libmathCommon.A.dylib   /usr/lib/
system/libmathCommon.A.dylib
0x94125000 - 0x94138fff libpam.1.dylib  /usr/lib/libpam.
1.dylib
0x94610000 - 0x9462afff libresolv.9.dylib       /usr/lib/
libresolv.9.dylib
0x94640000 - 0x94649fff libz.1.dylib    /usr/lib/libz.
1.dylib
0x95f30000 - 0x9601efff libiconv.2.dylib        /usr/lib/
libiconv.2.dylib
0x9784a000 - 0x97999fff libSystem.B.dylib       /usr/lib/
libSystem.B.dylib

[2005-12-13 04:23 UTC] rasmus@php.net

It's sort of a backtrace, but it isn't a backtrace that matches your initial report.  This one shows no sign of apc involvement at all.  Just generic heap corruption as far as I can tell.  Any chance you could remove all extensions that aren't critical to what you are doing and see if it still happens?

[2005-12-13 04:39 UTC] alain dot russell at blackpepper dot co dot nz

Ok .. 
Rebuilding with extensions we dont need removed.
New configure as follows

./configure \
--prefix=/usr/local/php5 \
--enable-sockets \
--with-xml \
--with-libxml-dir=/opt/local \
--with-gd \
--with-jpeg-dir=/opt/local \
--with-png-dir=/opt/local \
--with-mysql=/usr/local/mysql \
--with-mysqli=/usr/local/mysql/bin/mysql_config \
--with-apxs2=/usr/local/apache2/bin/apxs \
--with-pspell=/opt/local \
--with-openssl=/usr \
--with-zlib=/usr \
--with-mcrypt=/opt/local \
--with-curl=/opt/local \
--disable-cgi \
--with-mhash=/opt/local \
--enable-ftp \
--with-imap=/users/admin/documents/imap-2004a \
--with-imap-ssl=/users/admin/documents/imap-2004a

Will see how it goes - should know in a few hours if there 
is a change.

[2005-12-13 13:50 UTC] alain dot russell at blackpepper dot co dot nz

Checked this morning and the same problem appears .. 
Screen shots of the Log file and APC Dashboard page below

http://utility.blackpepper.co.nz/apc/log.png
http://utility.blackpepper.co.nz/apc/apc.png

Let me know what else I can supply

[2005-12-13 15:20 UTC] alain dot russell at blackpepper dot co dot nz

Not sure if it helps but we've just had apache stop responding 
to some pages. None of the pages have recently been edited - 
they are unchanged so would have not had to have been updated 
at all in the cache.

APC Memory usage was still well under the limit when Apache 
started crahing as well.

[2005-12-14 03:01 UTC] alain dot russell at blackpepper dot co dot nz

Hi Rasmus,
I've rebuilt php + apc with debug and am running it now to 
see if I can get a backtrace.

I was just looking at the error_log and its started 
scrolling pretty fast with this:

http://utility.blackpepper.co.nz/apc/leaks.png

(If that helps some more)

[2005-12-15 02:20 UTC] alain dot russell at blackpepper dot co dot nz

Well, Apache started to crash again today causing a restart. I 
have looked and looked for the core file but cannot seem to 
find it anywhere on my system ? So unfortunately can't supply 
it.. any idea where I should look ?

[2005-12-16 15:36 UTC] alain dot russell at blackpepper dot co dot nz

Just an update .. 

Really have not seen the problem again while running a debug 
build .. The cache is now full .. first time I think I have 
seen this (eaccelerator used to always fill the cache)

And I now have high fragmentation again but am not seeing 
crashes .. ?

http://utility.blackpepper.co.nz/apc/apc2.png

[2005-12-22 05:39 UTC] alain dot russell at blackpepper dot co dot nz

Another update.. 
Have not had a signal(10) crash basically since the debug 
build. Lots of memory leak errors in the logs but none of the 
above behaviour ..Any ideas ?

[2006-02-24 10:45 UTC] rasmus@php.net

Try current CVS.  There have been a number of fixes recently.

[2006-02-24 15:17 UTC] alain dot russell at blackpepper dot co dot nz

thanks .. without sounding stupid .. where/how do I get the 
sources from CVS ?

[2006-02-24 15:50 UTC] rasmus@php.net

cvs -d :pserver:cvsread@cvs.php.net:/repository login
cvs -d :pserver:cvsread@cvs.php.net:/repository co pecl/apc
cd pecl/apc
phpize
./configure --enable-apc-mmap
make
make install

[2006-02-24 21:42 UTC] alain dot russell at blackpepper dot co dot nz

Thanks, that's all installed OK. I'm now running 3.0.9-dev 
and PHP 5.1.2

I'm still seeing a lot of memory leaks - I have managed to 
track one of them down to some simple code - example below:

<?

$li_slideshowid = 12;

function phpResequence() {
	global $li_slideshowid;
}

?>

This will cause an entry in the error log of apache every 
time I run the script. eg:

[Sat Feb 25 15:38:50 2006]  Script:  '/Library/WebServer/
Documents/secure.hostnz.com/15framesoffame/88leak.php'
/Users/admin/Desktop/pecl/apc/apc_compile.c(1821) :  Freeing 
0x005EC318 (140 bytes), script=/Library/WebServer/Documents/
secure.hostnz.com/15framesoffame/88leak.php

Hope that helps track it down.
It does not leak when APC.so is not enabled.

Thanks
Alain

[2006-02-25 02:19 UTC] rasmus@php.net

Those aren't real leaks.  You compiled your PHP in DEBUG mode and it is just a message from the pool mechanism saying it is cleaning up 140 bytes.  Disable debug mode and those messages will disappear.

[2006-03-03 20:53 UTC] alain dot russell at blackpepper dot co dot nz

No Problems with the build from CVS .. been running solid on a 
production server for the week now and zero problems.

	php.net \| support \| documentation \| report a bug \| advanced search \| search howto \| statistics \| random bug \| login
go to bug id or search bugs for


Copyright © 2001-2024 The PHP Group All rights reserved.	Last updated: Tue May 07 13:01:35 2024 UTC