Here are the php.ini configuration parameters:
----------------------------------------------

extension=apc.so
apc.enabled=1
apc.shm_segments=1
apc.optimization=0
apc.shm_size=384M
apc.ttl=0
apc.user_ttl=0
apc.num_files_hint=4096
apc.mmap_file_mask=/tmp/apc.XXXXXX
apc.canonicalize=0
apc.write_lock = On
apc.stat=0

Any chance you can narrow this down to a bit of PHP code that can reproduce this?

I think this particular fault happened during symfony2's UniversalClassLoader 
trying to do a load some class, other failures are 
elsewhere.

I need to note that both APC 3.1.10 and 3.1.11 have worked fine both under our 
development and QA / staging environments and 
machines (of similar profiles as provided below), although 3.1.11 was tested 
less than 3.1.10, but no APC issues on those 
machines.

Our error logs are full of require_once() failures (3.1.10) or 
UniversalClassLoader + require() failures (3.1.11), depending on 
where the thing choose to crash.

Here's the PHP log error messages we're getting:

[Thu Jun 28 15:09:51 2012] [error] [client x.y.z.a] PHP Fatal error:  require():  
in /service/releases/kenya2-
155383/local/git/vendor/symfony/src/Symfony/Component/ClassLoader/UniversalClass
Loader.php on line 229

[Thu Jun 28 15:09:51 2012] [error] [client a.b.c.d] PHP Fatal error:  require(): 
Failed opening required 
'/service/local/git/vendor/symfon\xdc\xd1*\xddc/Symfony/Component/HttpFoundatif\
x80\x91\r.' 
include_path='/service/web:/service/local/apache/lib/php:.') in 
/service/releases/kenya2-
155383/local/git/vendor/symfony/src/Symfony/Component/ClassLoader/UniversalClass
Loader.php 

If you notice, the string 
'/service/local/git/vendor/symfon\xdc\xd1*\xddc/Symf...' is somewhat corrupted.

Line 229 of UniversalClassLoader.php is the 'require' line below:
vendor/symfony/src/Symfony/Component/ClassLoader/UniversalClassLoader.php --

221 /**
222  * Loads the given class or interface.
223  *
224  * @param string $class The name of the class
225  */
226 public function loadClass($class)
227 {
228     if ($file = $this->findFile($class)) {
229         require $file;
230     }
231 }


----------------

Here's some more 'corrupted' log entries from another machine that failed during 
our rollout:

[Thu Jun 28 15:02:04 2012] [error] [client a.b.c.d] PHP Fatal error:  require(): 
Failed opening required 
'/service/local/git/vendor/symfony/src/Syp\x05;\xddy/Bundle/DoctrineBundle/Conne
ctionFa\xd2\xff%% ' 
(include_path='/service/web:/service/local/apache/lib/php:.') in 
/service/releases/kenya2-
155383/local/git/vendor/symfony/src/Symfony/Component/ClassLoader/UniversalClass
Loader.php on line 229

[Thu Jun 28 15:02:04 2012] [error] [client a.b.c.d] PHP Fatal error:  require():  
in /service/releases/kenya2-
155383/local/git/vendor/symfony/src/Symfony/Component/ClassLoader/UniversalClass
Loader.php on line 229

-----------------

Let me know if there's anything else I can get for you in terms of logs or 
whatever.  If someone wants to see the issue 
happening live, let me know - I can't make any promises, but I might be able to 
swing something.

Sure, the corrupted string problem is known and is fixed in trunk. I'm not that 
interested in the details of the crashes from 3.1.10 since those all look like 
the problem we already fixed.

Actually all the error messages and stack traces in this bug (including the 
error messages with corrupted 
strings) are from trunk (3.1.11 according to package.xml - pulled as of 
6/27/2012).

3.1.10 was actually worse.  The corruption was so bad in 3.1.10 that a single 
machine was not able to start 
about 75% of the time or more.  In 3.1.11 it seems to be better - the failure 
rate was down to about 10-15%.

Still 10-15% is still a big problem - we're not really setup to handle that as 
we're used to 0% failure in 
5.3...

Interesting, this would indicate that there is one more place where I am not 
null-terminating interned strings. It is fixed in 2 places in trunk which is 
likely why you are seeing less problems. I'll try to track down this third place.
Thanks.

@ mmucklo at corp dot oodle dot com

I was trying to reproduce this with demo symfony2 project but got no crash. Does 
the segfault happen under the heavy load only? It would be very useful if you 
could repro this fails on some publicly available symfony2 app/project, so one 
could 
debug then.

Does that crash from the backtrace (with "getlanguages") always happen on that 
code? Does it happen also if you include that class/function isolated from the 
other stuff?

Also, you've said there are many other exts present, may be you could turn off a 
one or another and see something has changed.

Unfortunately it looks like it only happens on heavy load.  I suppose I can try 
to reproduce and see if the same backtrace happens each time.

Our code base is not just symfony2, it's a bit of a hybrid, nevertheless I might 
be able to try and load test on our end and see if I can reproduce it, but it 
might not be a simple setup as our production and development environments 
differ significantly.

I ran some tests locally using apache bench and got a few cores, but they were not quite the same (happened during cleanup, but apc_string.c 
was still in the stack trace)...

#0  0x00007f7d979dd68a in gc_zobj_possible_root (zv=0x6a346e8) at /home/mmucklo/service/external/build/php-5.4.4/Zend/zend_gc.c:197
#1  0x00007f7d979b554f in destroy_zend_class (pce=<value optimized out>) at /home/mmucklo/service/external/build/php-
5.4.4/Zend/zend_opcode.c:288
#2  0x00007f7d979cb3c3 in zend_hash_clean (ht=0x1d576d0) at /home/mmucklo/service/external/build/php-5.4.4/Zend/zend_hash.c:596
#3  0x00007f7d8690c770 in apc_interned_strings_shutdown () at /tmp/pear/temp/APC/apc_string.c:239
#4  0x00007f7d86904cde in apc_module_shutdown () at /tmp/pear/temp/APC/apc_main.c:894
#5  0x00007f7d868fb6df in zm_shutdown_apc (type=<value optimized out>, module_number=44) at /tmp/pear/temp/APC/php_apc.c:373
#6  0x00007f7d979c42d5 in module_destructor (module=0x56b5210) at /home/mmucklo/service/external/build/php-5.4.4/Zend/zend_API.c:2262
#7  0x00007f7d979cad35 in zend_hash_apply_deleter (ht=0x7f7d981bb700, p=0x56b51b0) at /home/mmucklo/service/external/build/php-
5.4.4/Zend/zend_hash.c:650
#8  0x00007f7d979cafc8 in zend_hash_graceful_reverse_destroy (ht=0x7f7d981bb700) at /home/mmucklo/service/external/build/php-
5.4.4/Zend/zend_hash.c:687
#9  0x00007f7d979bebce in zend_shutdown () at /home/mmucklo/service/external/build/php-5.4.4/Zend/zend.c:823
#10 0x00007f7d9796188a in php_module_shutdown () at /home/mmucklo/service/external/build/php-5.4.4/main/main.c:2346
#11 0x00007f7d97961949 in php_module_shutdown_wrapper (sapi_globals=<value optimized out>) at /home/mmucklo/service/external/build/php-
5.4.4/main/main.c:2314
#12 0x00007f7d97a66611 in php_apache_child_shutdown (tmp=<value optimized out>) at /home/mmucklo/service/external/build/php-
5.4.4/sapi/apache2handler/sapi_apache2.c:398
#13 0x00007f7d9858845e in run_cleanups (pool=0x5b19438) at memory/unix/apr_pools.c:2352
#14 apr_pool_destroy (pool=0x5b19438) at memory/unix/apr_pools.c:814
#15 0x000000000049e5de in clean_child_exit (code=0) at prefork.c:196
#16 0x000000000049ec9b in just_die (sig=<value optimized out>) at prefork.c:328
#17 <signal handler called>
#18 0x00007f7d97a332f0 in ZEND_INIT_NS_FCALL_BY_NAME_SPEC_CONST_HANDLER (execute_data=0x9593760) at /home/mmucklo/service/external/build/php-
5.4.4/Zend/zend_vm_execute.h:1326
#19 0x00007f7d97a26930 in execute (op_array=0x5f90c78) at /home/mmucklo/service/external/build/php-5.4.4/Zend/zend_vm_execute.h:410
#20 0x00007f7d86b2baa9 in xdebug_execute (op_array=0x5f90c78) at /tmp/pear/temp/xdebug/xdebug.c:1390
#21 0x00007f7d97a397f1 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /home/mmucklo/service/external/build/php-
5.4.4/Zend/zend_vm_execute.h:669
#22 0x00007f7d97a26930 in execute (op_array=0x5f8e340) at /home/mmucklo/service/external/build/php-5.4.4/Zend/zend_vm_execute.h:410
#23 0x00007f7d86b2baa9 in xdebug_execute (op_array=0x5f8e340) at /tmp/pear/temp/xdebug/xdebug.c:1390
#24 0x00007f7d97a397f1 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /home/mmucklo/service/external/build/php-
5.4.4/Zend/zend_vm_execute.h:669
#25 0x00007f7d97a26930 in execute (op_array=0x5f90b00) at /home/mmucklo/service/external/build/php-5.4.4/Zend/zend_vm_execute.h:410
#26 0x00007f7d86b2baa9 in xdebug_execute (op_array=0x5f90b00) at /tmp/pear/temp/xdebug/xdebug.c:1390
#27 0x00007f7d97a397f1 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /home/mmucklo/service/external/build/php-
5.4.4/Zend/zend_vm_execute.h:669
#28 0x00007f7d97a26930 in execute (op_array=0x5f8b948) at /home/mmucklo/service/external/build/php-5.4.4/Zend/zend_vm_execute.h:410
#29 0x00007f7d86b2baa9 in xdebug_execute (op_array=0x5f8b948) at /tmp/pear/temp/xdebug/xdebug.c:1390
#30 0x00007f7d97a397f1 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /home/mmucklo/service/external/build/php-
5.4.4/Zend/zend_vm_execute.h:669
#31 0x00007f7d97a26930 in execute (op_array=0x5f8ab70) at /home/mmucklo/service/external/build/php-5.4.4/Zend/zend_vm_execute.h:410
#32 0x00007f7d86b2baa9 in xdebug_execute (op_array=0x5f8ab70) at /tmp/pear/temp/xdebug/xdebug.c:1390
#33 0x00007f7d97a397f1 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /home/mmucklo/service/external/build/php-
5.4.4/Zend/zend_vm_execute.h:669
#34 0x00007f7d97a26930 in execute (op_array=0x5f8a2a0) at /home/mmucklo/service/external/build/php-5.4.4/Zend/zend_vm_execute.h:410
#35 0x00007f7d86b2baa9 in xdebug_execute (op_array=0x5f8a2a0) at /tmp/pear/temp/xdebug/xdebug.c:1390
#36 0x00007f7d97a397f1 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /home/mmucklo/service/external/build/php-
5.4.4/Zend/zend_vm_execute.h:669

...
goes on and on

Hrm.. there are a few things going on in that trace. First, you are using xdebug 
and apc together. For debugging that is ok, but you will never want to run both 
in production. Also, note that the crash is on module shutdown. As in, it is on 
shutting down the web server. See the <signal handler called> section there? 
Your process received a signal telling the server to shut down. And while 
crashes on shutdown really should be fixed, I am less worried about them because 
they do not affect actual user requests when they happen after the signal 
handler like that. So, I would be much more interested in any traces you can get 
that do not include xdebug nor module_shutdown.

Using the demo application included with Symfony2, I haven’t been able to repro a crash using my setup and up to 500 concurrent requests and up to 100000 total. I’ve been able to run ab –c 500 –n 100000, as well as smaller increments. I can still use the demo application myself while the load test is running.

My setup:
(a typical good production lamp server)
12GB memory
4 quad core CPUs (8-way xeon)
CentOS 6.0 x64
PDO MySQL (separate database server)
PHP 5.4.4 (built Jun 14 2012 17:30, official build installed from CentOS repo using yum)
APC 3.1.10-1
Symfony 2.1.0-dev

Using Composer, I’ve setup the demo app on my CentOS box using the official instructions from: https://github.com/symfony/symfony-standard

The reporter is running XDebug+APC whereas I’m just running APC (shouldn’t use both). Otherwise the reporter’s setup is  CentOS 5.8 vs Centos 6.0 (which shouldn’t matter).

I did have an issue with the configuration app with APC enabled, which is a known issue (https://github.com/symfony/symfony/issues/4185). I disabled APC to run that, then enabled APC to use the Demo app (APC was always enabled when I tested the symfony2 demo app).


The reporter said that they could also repro this bug using their app, though they haven’t shared it. Maybe its doing something special.

We're using symfony 2.0, but our code base predates symfony.  There are at least 
1.8k files that are not symfony related, not including Smarty templates.

Symfony is rather something of an appendage at this point, although most of our 
code does route through the Kernel.  Parts of our site heavily rely up on it, 
some parts don't as much, but we've been primarily migrating and implementing 
new functionality in the symfony side of things.

As for stack traces, at this point, then, please ignore the second stack trace, 
as it's a DEV environment.

We're going to try to get more production stack traces for you.

Please remove xdebug and try again.

@mmucklo at corp dot oodle dot com

Ok, as there is no feedback for that long time, but the bug exists anyway ... 
Once you've said you could swing something for getting a possibility to debug on 
your environment. Could you please do it now? Getting an access to the failing 
environment with a debug possibility would bring us way to much forward.

This is a bit anecdotal, I realize, but we did try switching to PHP
5.3.14 + APC 3.1.11 (trunk) in production, and still experienced some
grave problems.  Things do seem to run okay, but the load on the
servers goes berserk.  Going back to 3.1.9 makes things okay again.
We can simulate the problems by simply stopping the servers,
swapping the apc.so files (3.1.9 w/ 3.1.11/trunk), and restarting them.

The initial problem still remains with PHP 5.4.4 and APC.  The
*first* stack trace shows and says all.

*xdebug is off* in the first stack trace (the second stack trace was
just a (failed) attempt to reproduce the problem in our dev
environment, if I could, I would delete it at this point to eliminate
future confusion)...

If someone wants, I can coordinate a time for them to see this
happen in our production environment with a screen share or
something.  Please email me directly.

Okay, I just ran a fresh 5.4.4 build in production with APC trunk (3.1.11).  Here's a fresh new core + log entries (minus referrer info):


Log Entries:
[Tue Jul 17 17:49:19 2012] [notice] child pid 31167 exit signal Segmentation fault (11), possible coredump in /tmp
[Tue Jul 17 17:49:22 2012] [error] [client 10.1.38.43] PHP Fatal error:  require(): Failed opening required 
'/service/local/git/vendor/symfony/sr\x04\xda*\xdd\x90\xd9*\xddy/Component/Security/Acl/Permission/+q\xf9Yk' (include_path='/service/web:/service/local/apache/lib/php:.') 
in /service/releases/kenya2-156328/local/git/vendor/symfony/src/Symfony/Component/ClassLoader/UniversalClassLoader.php on line 229
[Tue Jul 17 17:49:22 2012] [error] [client 10.1.38.43] PHP Fatal error:  require(): Failed opening required 
'/service/local/git/vendor/symfony/sr\x04\xda*\xdd\x90\xd9*\xddy/Component/Security/Acl/Permission/+q\xf9Yk' (include_path='/service/web:/service/local/apache/lib/php:.') 
in /service/releases/kenya2-156328/local/git/vendor/symfony/src/Symfony/Component/ClassLoader/UniversalClassLoader.php on line 229
[Tue Jul 17 17:49:24 2012] [error] [client 10.1.38.43] PHP Fatal error:  require(): Failed opening required 
'/service/local/git/vendor/symfony/sr\x04\xda*\xdd\x90\xd9*\xddy/Component/Security/Acl/Permission/+q\xf9Yk' (include_path='/service/web:/service/local/apache/lib/php:.') 
in /service/releases/kenya2-156328/local/git/vendor/symfony/src/Symfony/Component/ClassLoader/UniversalClassLoader.php on line 229
[Tue Jul 17 17:49:30 2012] [error] [client 10.1.38.43] PHP Fatal error:  require(): Failed opening required 
'/service/local/git/vendor/symfony/sr\x04\xda*\xdd\x90\xd9*\xddy/Component/Security/Acl/Permission/+q\xf9Yk' (include_path='/service/web:/service/local/apache/lib/php:.') 
in /service/releases/kenya2-156328/local/git/vendor/symfony/src/Symfony/Component/ClassLoader/UniversalClassLoader.php on line 229

Core Dump #1:

Program terminated with signal 11, Segmentation fault.
#0  0xf548ccab in apc_new_interned_string (arKey=0x9bf9ecc "getForeignTableName", nKeyLength=20) at /tmp/pear/temp/APC/apc_string.c:86
86      /tmp/pear/temp/APC/apc_string.c: No such file or directory.
        in /tmp/pear/temp/APC/apc_string.c
(gdb) bt
#0  0xf548ccab in apc_new_interned_string (arKey=0x9bf9ecc "getForeignTableName", nKeyLength=20) at /tmp/pear/temp/APC/apc_string.c:86
#1  0xf548213c in apc_string_pmemcpy (str=0x9bf9ecc "getForeignTableName", len=20, pool=0xe04f0fd8) at /tmp/pear/temp/APC/apc_compile.c:284
#2  0xf5482856 in my_copy_zval (dst=0xe0506ca8, src=0x9bfccb8, ctxt=0xffd30970) at /tmp/pear/temp/APC/apc_compile.c:340
#3  0xf548371f in apc_copy_op_array (dst=0xe05067a8, src=0x9bf96f8, ctxt=0xffd30970) at /tmp/pear/temp/APC/apc_compile.c:1104
#4  0xf5483cd9 in my_copy_function (dst=0x0, src=0x9bf96f8, ctxt=0xffd30970) at /tmp/pear/temp/APC/apc_compile.c:472
#5  0xf54825ab in my_copy_hashtable_ex (dst=0xe04f1200, src=0x9be2840, copy_fn=0xf5483c3c <my_copy_function>, holds_ptrs=0, ctxt=0xffd30970, check_fn=0xf5484c00 
<my_check_copy_function>) at /tmp/pear/temp/APC/apc_compile.c:927
#6  0xf5482b6d in my_copy_class_entry (dst=0xe04f11e8, src=0x9be2828, ctxt=0xffd30970) at /tmp/pear/temp/APC/apc_compile.c:668
#7  0xf548401c in apc_copy_new_classes (op_array=0x9be14e0, old_count=909, ctxt=0xffd30970) at /tmp/pear/temp/APC/apc_compile.c:1437
#8  0xf5485566 in apc_compile_cache_entry (key=0xffd31b20, h=0xffd31cc0, type=8, t=1342572540, op_array=0xffd309e4, cache_entry=0xdafa3c58) at 
/tmp/pear/temp/APC/apc_main.c:457
#9  0xf5485b2a in my_compile_file (h=0xffd31cc0, type=8) at /tmp/pear/temp/APC/apc_main.c:610
#10 0xf75722ef in phar_compile_file (file_handle=0xffd31cc0, type=8) at /source/release/kenya2/external/build/php-5.4.4/ext/phar/phar.c:3391
#11 0xf769a633 in compile_filename (type=8, filename=0x9be14c4) at Zend/zend_language_scanner.l:625
#12 0xf773a3f2 in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER (execute_data=0xf57c56d8) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_vm_execute.h:27378
#13 0xf76ed76e in execute (op_array=0xdceb037c) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_vm_execute.h:410
#14 0xf76bf0cb in zend_call_function (fci=0xffd31ef0, fci_cache=0xffd31ec0) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_execute_API.c:958
#15 0xf76ddb2d in zend_call_method (object_pp=0xdceb2c50, obj_ce=0xdceaff00, fn_proxy=0xdceb2c4c, function_name=0xdceb37d0 
"symfony\\component\\classloader\\apcuniversalclassloader::loadclass\002", function_name_len=69,
    retval_ptr_ptr=0xffd31f80, param_count=1, arg1=0x9bbdd00, arg2=0x0) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_interfaces.c:97
#16 0xf75d2ee0 in zif_spl_autoload_call (ht=1, return_value=0x9be148c, return_value_ptr=0xffd320b4, this_ptr=0x0, return_value_used=1) at 
/source/release/kenya2/external/build/php-5.4.4/ext/spl/php_spl.c:436
#17 0xf76bedff in zend_call_function (fci=0xffd320e0, fci_cache=0xffd320c0) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_execute_API.c:980
#18 0xf76bf960 in zend_lookup_class_ex (name=0xdd2a440c "Doctrine\\DBAL\\Platforms\\AbstractPlatform", name_length=40, key=0xe03ec0d8, use_autoload=1, ce=0xffd32168)
    at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_execute_API.c:1127
#19 0xf76c0633 in zend_fetch_class_by_name (class_name=0xdd2a440c "Doctrine\\DBAL\\Platforms\\AbstractPlatform", class_name_len=40, key=0xe03ec0d8, fetch_type=4)
    at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_execute_API.c:1607
#20 0xf76efaa5 in ZEND_FETCH_CLASS_SPEC_CONST_HANDLER (execute_data=0xf57c5660) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_vm_execute.h:1173
#21 0xf76ed76e in execute (op_array=0x9be1348) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_vm_execute.h:410
#22 0xf76bf0cb in zend_call_function (fci=0xffd322f0, fci_cache=0xffd322c0) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_execute_API.c:958
#23 0xf76ddb2d in zend_call_method (object_pp=0xdceb2c50, obj_ce=0xdceaff00, fn_proxy=0xdceb2c4c, function_name=0xdceb37d0 
"symfony\\component\\classloader\\apcuniversalclassloader::loadclass\002", function_name_len=69,
    retval_ptr_ptr=0xffd32380, param_count=1, arg1=0x9bb7860, arg2=0x0) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_interfaces.c:97
#24 0xf75d2ee0 in zif_spl_autoload_call (ht=1, return_value=0x9bdd234, return_value_ptr=0xffd324b4, this_ptr=0x0, return_value_used=1) at 
/source/release/kenya2/external/build/php-5.4.4/ext/spl/php_spl.c:436
#25 0xf76bedff in zend_call_function (fci=0xffd324e0, fci_cache=0xffd324c0) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_execute_API.c:980
#26 0xf76bf960 in zend_lookup_class_ex (name=0xdd2a3e4c "Doctrine\\DBAL\\Platforms\\MySqlPlatform", name_length=37, key=0xe03cb538, use_autoload=1, ce=0xffd32568)
    at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_execute_API.c:1127
#27 0xf76c0633 in zend_fetch_class_by_name (class_name=0xdd2a3e4c "Doctrine\\DBAL\\Platforms\\MySqlPlatform", class_name_len=37, key=0xe03cb538, fetch_type=4)
    at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_execute_API.c:1607
#28 0xf76efaa5 in ZEND_FETCH_CLASS_SPEC_CONST_HANDLER (execute_data=0xf57c5540) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_vm_execute.h:1173
#29 0xf76ed76e in execute (op_array=0x9bb73a8) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_vm_execute.h:410
#30 0xf76bf0cb in zend_call_function (fci=0xffd326e0, fci_cache=0xffd326c0) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_execute_API.c:958
#31 0xf7600ede in zif_call_user_func (ht=2, return_value=0x9b9dfc8, return_value_ptr=0x0, this_ptr=0x0, return_value_used=0) at /source/release/kenya2/external/build/php-
5.4.4/ext/standard/basic_functions.c:4726
#32 0xf76ee439 in zend_do_fcall_common_helper_SPEC (execute_data=0xf57c3070) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_vm_execute.h:642
#33 0xf76ed76e in execute (op_array=0x9b97960) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_vm_execute.h:410
#34 0xf76cc557 in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend.c:1279
#35 0xf767770e in php_execute_script (primary_file=0xffd34a50) at /source/release/kenya2/external/build/php-5.4.4/main/main.c:2473
#36 0xf77523df in php_handler (r=0x9646f78) at /source/release/kenya2/external/build/php-5.4.4/sapi/apache2handler/sapi_apache2.c:667
#37 0x0807afc2 in ap_run_handler (r=0x9646f78) at config.c:157
#38 0x0807b397 in ap_invoke_handler (r=0x9646f78) at config.c:376
#39 0x080a46d5 in ap_process_request (r=0x9646f78) at http_request.c:282
#40 0x080a2099 in ap_process_http_connection (c=0x963a9e0) at http_core.c:190
#41 0x0808092a in ap_run_process_connection (c=0x963a9e0) at connection.c:43
#42 0x080cb42e in child_main (child_num_arg=<value optimized out>) at prefork.c:667
#43 0x080cb606 in make_child (s=<value optimized out>, slot=10) at prefork.c:768
#44 0x080cc18d in ap_mpm_run (_pconf=0x92330a8, plog=0x927b1c8, s=0x923c010) at prefork.c:903
#45 0x0806ac71 in main (argc=3, argv=0xffd34fc4) at main.c:753

Core Dump #2:

Program terminated with signal 11, Segmentation fault.
#0  0xf548ccab in apc_new_interned_string (arKey=0xdca41bec "getForeignTableName", nKeyLength=20) at /tmp/pear/temp/APC/apc_string.c:86
86      /tmp/pear/temp/APC/apc_string.c: No such file or directory.
        in /tmp/pear/temp/APC/apc_string.c
(gdb) bt
#0  0xf548ccab in apc_new_interned_string (arKey=0xdca41bec "getForeignTableName", nKeyLength=20) at /tmp/pear/temp/APC/apc_string.c:86
#1  0xf548213c in apc_string_pmemcpy (str=0xdca41bec "getForeignTableName", len=20, pool=0xe0424aa0) at /tmp/pear/temp/APC/apc_compile.c:284
#2  0xf5482856 in my_copy_zval (dst=0xe043a770, src=0xdca449d8, ctxt=0xffd30960) at /tmp/pear/temp/APC/apc_compile.c:340
#3  0xf548371f in apc_copy_op_array (dst=0xe043a270, src=0xdca41418, ctxt=0xffd30960) at /tmp/pear/temp/APC/apc_compile.c:1104
#4  0xf5483cd9 in my_copy_function (dst=0x0, src=0xdca41418, ctxt=0xffd30960) at /tmp/pear/temp/APC/apc_compile.c:472
#5  0xf54825ab in my_copy_hashtable_ex (dst=0xe0424cc8, src=0xdca2aaf8, copy_fn=0xf5483c3c <my_copy_function>, holds_ptrs=0, ctxt=0xffd30960, check_fn=0xf5484c00 
<my_check_copy_function>) at /tmp/pear/temp/APC/apc_compile.c:927
#6  0xf5482b6d in my_copy_class_entry (dst=0xe0424cb0, src=0xdca2aae0, ctxt=0xffd30960) at /tmp/pear/temp/APC/apc_compile.c:668
#7  0xf548401c in apc_copy_new_classes (op_array=0xdca29838, old_count=909, ctxt=0xffd30960) at /tmp/pear/temp/APC/apc_compile.c:1437
#8  0xf5485566 in apc_compile_cache_entry (key=0xffd31b10, h=0xffd31cb0, type=8, t=1342572508, op_array=0xffd309d4, cache_entry=0xdafa3c58) at 
/tmp/pear/temp/APC/apc_main.c:457
#9  0xf5485b2a in my_compile_file (h=0xffd31cb0, type=8) at /tmp/pear/temp/APC/apc_main.c:610
#10 0xf75722ef in phar_compile_file (file_handle=0xffd31cb0, type=8) at /source/release/kenya2/external/build/php-5.4.4/ext/phar/phar.c:3391
#11 0xf769a633 in compile_filename (type=8, filename=0xdca2981c) at Zend/zend_language_scanner.l:625
#12 0xf773a3f2 in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER (execute_data=0xf57c56d8) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_vm_execute.h:27378
#13 0xf76ed76e in execute (op_array=0xdceaed10) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_vm_execute.h:410
#14 0xf76bf0cb in zend_call_function (fci=0xffd31ee0, fci_cache=0xffd31eb0) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_execute_API.c:958
#15 0xf76ddb2d in zend_call_method (object_pp=0xdceb15e4, obj_ce=0xdceae894, fn_proxy=0xdceb15e0, function_name=0xdceb2164 
"symfony\\component\\classloader\\apcuniversalclassloader::loadclass\002", function_name_len=69,
    retval_ptr_ptr=0xffd31f70, param_count=1, arg1=0xdca23b90, arg2=0x0) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_interfaces.c:97
#16 0xf75d2ee0 in zif_spl_autoload_call (ht=1, return_value=0xdca297e4, return_value_ptr=0xffd320a4, this_ptr=0x0, return_value_used=1) at 
/source/release/kenya2/external/build/php-5.4.4/ext/spl/php_spl.c:436
#17 0xf76bedff in zend_call_function (fci=0xffd320d0, fci_cache=0xffd320b0) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_execute_API.c:980
#18 0xf76bf960 in zend_lookup_class_ex (name=0xdd2a440c "Doctrine\\DBAL\\Platforms\\AbstractPlatform", name_length=40, key=0xe03ec0d8, use_autoload=1, ce=0xffd32158)
    at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_execute_API.c:1127
#19 0xf76c0633 in zend_fetch_class_by_name (class_name=0xdd2a440c "Doctrine\\DBAL\\Platforms\\AbstractPlatform", class_name_len=40, key=0xe03ec0d8, fetch_type=4)
    at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_execute_API.c:1607
#20 0xf76efaa5 in ZEND_FETCH_CLASS_SPEC_CONST_HANDLER (execute_data=0xf57c5660) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_vm_execute.h:1173
#21 0xf76ed76e in execute (op_array=0xdca296a0) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_vm_execute.h:410
#22 0xf76bf0cb in zend_call_function (fci=0xffd322e0, fci_cache=0xffd322b0) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_execute_API.c:958
#23 0xf76ddb2d in zend_call_method (object_pp=0xdceb15e4, obj_ce=0xdceae894, fn_proxy=0xdceb15e0, function_name=0xdceb2164 
"symfony\\component\\classloader\\apcuniversalclassloader::loadclass\002", function_name_len=69,
    retval_ptr_ptr=0xffd32370, param_count=1, arg1=0xdca1d70c, arg2=0x0) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_interfaces.c:97
#24 0xf75d2ee0 in zif_spl_autoload_call (ht=1, return_value=0xdca2558c, return_value_ptr=0xffd324a4, this_ptr=0x0, return_value_used=1) at 
/source/release/kenya2/external/build/php-5.4.4/ext/spl/php_spl.c:436
#25 0xf76bedff in zend_call_function (fci=0xffd324d0, fci_cache=0xffd324b0) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_execute_API.c:980
#26 0xf76bf960 in zend_lookup_class_ex (name=0xdd2a3e4c "Doctrine\\DBAL\\Platforms\\MySqlPlatform", name_length=37, key=0xe03cb538, use_autoload=1, ce=0xffd32558)
    at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_execute_API.c:1127
#27 0xf76c0633 in zend_fetch_class_by_name (class_name=0xdd2a3e4c "Doctrine\\DBAL\\Platforms\\MySqlPlatform", class_name_len=37, key=0xe03cb538, fetch_type=4)
    at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_execute_API.c:1607
#28 0xf76efaa5 in ZEND_FETCH_CLASS_SPEC_CONST_HANDLER (execute_data=0xf57c5540) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_vm_execute.h:1173
#29 0xf76ed76e in execute (op_array=0xdca1d254) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_vm_execute.h:410
#30 0xf76bf0cb in zend_call_function (fci=0xffd326d0, fci_cache=0xffd326b0) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_execute_API.c:958
#31 0xf7600ede in zif_call_user_func (ht=2, return_value=0xdca7ba1c, return_value_ptr=0x0, this_ptr=0x0, return_value_used=0) at 
/source/release/kenya2/external/build/php-5.4.4/ext/standard/basic_functions.c:4726
#32 0xf76ee439 in zend_do_fcall_common_helper_SPEC (execute_data=0xf57c3070) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_vm_execute.h:642
#33 0xf76ed76e in execute (op_array=0xdca75438) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend_vm_execute.h:410
#34 0xf76cc557 in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /source/release/kenya2/external/build/php-5.4.4/Zend/zend.c:1279
#35 0xf767770e in php_execute_script (primary_file=0xffd34a40) at /source/release/kenya2/external/build/php-5.4.4/main/main.c:2473
#36 0xf77523df in php_handler (r=0x9646f78) at /source/release/kenya2/external/build/php-5.4.4/sapi/apache2handler/sapi_apache2.c:667
#37 0x0807afc2 in ap_run_handler (r=0x9646f78) at config.c:157
#38 0x0807b397 in ap_invoke_handler (r=0x9646f78) at config.c:376
#39 0x080a46d5 in ap_process_request (r=0x9646f78) at http_request.c:282
#40 0x080a2099 in ap_process_http_connection (c=0x963a9e0) at http_core.c:190
#41 0x0808092a in ap_run_process_connection (c=0x963a9e0) at connection.c:43
#42 0x080cb42e in child_main (child_num_arg=<value optimized out>) at prefork.c:667
#43 0x080cb606 in make_child (s=<value optimized out>, slot=0) at prefork.c:768
#44 0x080cb694 in startup_children (number_to_start=5) at prefork.c:786
#45 0x080cbebc in ap_mpm_run (_pconf=0x92330a8, plog=0x927b1c8, s=0x923c010) at prefork.c:1007
#46 0x0806ac71 in main (argc=3, argv=0xffd34fc4) at main.c:753

Thanks for the new patches. They look pretty much similar the the first one 
though.

What actually came into my mind - as it's only reproduceable on production, were 
there any possibility to record your production traffic and to reply it then on 
development? Never did it myself but tcpdump cap files can be replied with 
tcpreply (or wireshark?).

I just fixed a stat=0 related bug in trunk. Any chance you could try svn trunk 
and see if this is still happening? I don't think I fixed this one, but since I 
can't reproduce it here it would be good to know if this stat=0 fix changes 
anything for you.

Just tried it with the newly released 3.1.11 from the PECL site (http://pecl.php.net/package/APC):

Not as many Segfaults, just one core file which is a change, however still require once failures all over the place:

Logs:

[Thu Jul 19 16:32:04 2012] [error] [client 76.248.89.117] PHP Fatal error:  require_once(): Failed opening required 
'common/classes/yaml/PartnerConfi\x95Av\xe5'' (include_path='/service/web:/service/local/apache/lib/php:.') in /service/releases/latvia-
156486/web/common/classes/partner/Partner.php on line 3
[Thu Jul 19 16:32:04 2012] [error] [client 207.46.204.189] PHP Fatal error:  require_once():  in /service/releases/latvia-
156486/web/common/classes/partner/Partner.php on line 3
[Thu Jul 19 16:32:04 2012] [error] [client 184.172.241.89] PHP Fatal error:  require_once():  in /service/releases/latvia-
156486/web/common/classes/partner/Partner.php on line 3
[Thu Jul 19 16:32:04 2012] [error] [client 65.52.110.34] PHP Fatal error:  require_once():  in /service/releases/latvia-
156486/web/common/classes/partner/Partner.php on line 3
[Thu Jul 19 16:32:04 2012] [error] [client 184.73.232.108] PHP Fatal error:  require_once():  in /service/releases/latvia-
156486/web/common/classes/partner/Partner.php on line 3
[Thu Jul 19 16:32:04 2012] [error] [client 69.90.134.4] PHP Fatal error:  require_once():  in /service/releases/latvia-
156486/web/common/classes/partner/Partner.php on line 3
[Thu Jul 19 16:32:04 2012] [error] [client 107.22.97.247] PHP Fatal error:  require_once():  in /service/releases/latvia-
156486/web/common/classes/partner/Partner.php on line 3
[Thu Jul 19 16:32:04 2012] [error] [client 75.47.106.237] PHP Fatal error:  require_once(): Failed opening required 
'common/classes/yaml/PartnerConfi\x95Av\xe5'' (include_path='/service/web:/service/local/apache/lib/php:.') in /service/releases/latvia-
156486/web/common/classes/partner/Partner.php on line 3

Stack:

Program terminated with signal 11, Segmentation fault.
#0  0xf551fceb in apc_new_interned_string (arKey=0xa59f0f8 "discipline", nKeyLength=11) at /tmp/pear/temp/APC/apc_string.c:86
86      /tmp/pear/temp/APC/apc_string.c: No such file or directory.
        in /tmp/pear/temp/APC/apc_string.c
(gdb) bt
#0  0xf551fceb in apc_new_interned_string (arKey=0xa59f0f8 "discipline", nKeyLength=11) at /tmp/pear/temp/APC/apc_string.c:86
#1  0xf55151c8 in apc_string_pmemcpy (str=0xa59f0f8 "discipline", len=11, pool=0xe0bf0c00) at /tmp/pear/temp/APC/apc_compile.c:284
#2  0xf55158e2 in my_copy_zval (dst=0xe0c0cf48, src=0xa5e27cc, ctxt=0xff8c57e0) at /tmp/pear/temp/APC/apc_compile.c:340
#3  0xf5515aa6 in my_copy_zval_ptr (dst=0xe0c0cf40, src=0xa5e2a1c, ctxt=0xff8c57e0) at /tmp/pear/temp/APC/apc_compile.c:219
#4  0xf5515637 in my_copy_hashtable_ex (dst=0xe0c0cc18, src=0xa5e2888, copy_fn=0xf5515a38 <my_copy_zval_ptr>, holds_ptrs=1, ctxt=0xff8c57e0, check_fn=0) at 
/tmp/pear/temp/APC/apc_compile.c:927
#5  0xf551592e in my_copy_zval (dst=0xe0c0cc08, src=0xa5e27b0, ctxt=0xff8c57e0) at /tmp/pear/temp/APC/apc_compile.c:351
#6  0xf5515aa6 in my_copy_zval_ptr (dst=0xe0c0cc00, src=0xa84dc3c, ctxt=0xff8c57e0) at /tmp/pear/temp/APC/apc_compile.c:219
#7  0xf5515637 in my_copy_hashtable_ex (dst=0xe0bf0c98, src=0xa60bfdc, copy_fn=0xf5515a38 <my_copy_zval_ptr>, holds_ptrs=1, ctxt=0xff8c57e0, check_fn=0) at 
/tmp/pear/temp/APC/apc_compile.c:927
#8  0xf551592e in my_copy_zval (dst=0xe0bf0c88, src=0xa60bfc0, ctxt=0xff8c57e0) at /tmp/pear/temp/APC/apc_compile.c:351
#9  0xf55161d0 in apc_copy_zval (dst=0x66657268, src=0xa60bfc0, ctxt=0xff8c57e0) at /tmp/pear/temp/APC/apc_compile.c:992
#10 0xf5514674 in apc_cache_store_zval (dst=0x0, src=0xa60bfc0, ctxt=0xff8c57e0) at /tmp/pear/temp/APC/apc_cache.c:1114
#11 0xf5514783 in apc_cache_make_user_entry (info=0xa60bf88 "Attribute-YAML_id", info_len=18, val=0xa60bfc0, ctxt=0xff8c57e0, ttl=1717924456) at 
/tmp/pear/temp/APC/apc_cache.c:1158
#12 0xf550fffd in _apc_store (strkey=0xa60bf88 "Attribute-YAML_id", strkey_len=18, val=0xa60bfc0, ttl=0, exclusive=0) at /tmp/pear/temp/APC/php_apc.c:610
#13 0xf5510125 in apc_store_helper (ht=<value optimized out>, return_value=0xa84ecdc, return_value_ptr=0x0, this_ptr=0x0, return_value_used=0, exclusive=0) at 
/tmp/pear/temp/APC/php_apc.c:669
#14 0xf7781439 in zend_do_fcall_common_helper_SPEC (execute_data=0xf585ac90) at /source/release/latvia/external/build/php-5.4.4/Zend/zend_vm_execute.h:642
#15 0xf778076e in execute (op_array=0xa7c09bc) at /source/release/latvia/external/build/php-5.4.4/Zend/zend_vm_execute.h:410
#16 0xf77520cb in zend_call_function (fci=0xff8c5a40, fci_cache=0xff8c5a20) at /source/release/latvia/external/build/php-5.4.4/Zend/zend_execute_API.c:958
#17 0xf7693fdd in zif_call_user_func_array (ht=2, return_value=0xad16984, return_value_ptr=0x0, this_ptr=0x0, return_value_used=1) at 
/source/release/latvia/external/build/php-5.4.4/ext/standard/basic_functions.c:4751
#18 0xf7781439 in zend_do_fcall_common_helper_SPEC (execute_data=0xf5855984) at /source/release/latvia/external/build/php-5.4.4/Zend/zend_vm_execute.h:642
#19 0xf778076e in execute (op_array=0xa2f73d4) at /source/release/latvia/external/build/php-5.4.4/Zend/zend_vm_execute.h:410
#20 0xf775f557 in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /source/release/latvia/external/build/php-5.4.4/Zend/zend.c:1279
#21 0xf770a70e in php_execute_script (primary_file=0xff8c7db0) at /source/release/latvia/external/build/php-5.4.4/main/main.c:2473
#22 0xf77e53df in php_handler (r=0x9c9dc50) at /source/release/latvia/external/build/php-5.4.4/sapi/apache2handler/sapi_apache2.c:667
#23 0x0807afc2 in ap_run_handler (r=0x9c9dc50) at config.c:157
#24 0x0807b397 in ap_invoke_handler (r=0x9c9dc50) at config.c:376
#25 0x080a46d5 in ap_process_request (r=0x9c9dc50) at http_request.c:282
#26 0x080a2099 in ap_process_http_connection (c=0x9c97dd0) at http_core.c:190
#27 0x0808092a in ap_run_process_connection (c=0x9c97dd0) at connection.c:43
#28 0x080cb42e in child_main (child_num_arg=<value optimized out>) at prefork.c:667
#29 0x080cb606 in make_child (s=<value optimized out>, slot=9) at prefork.c:768
#30 0x080cc18d in ap_mpm_run (_pconf=0x98950a8, plog=0x98dd1c8, s=0x989e010) at prefork.c:903
#31 0x0806ac71 in main (argc=3, argv=0xff8c8324) at main.c:753

What exactly is that code doing? It looks to me like it is calling 
call_user_func_array() which calls some user-defined function which in turn does 
an apc_store('Attribute-YAML_id', ??) with some nasty-looking value there. An 
object containing an array or is it a nested array? Could you perhaps find that 
bit of code and see if you could write a simplified version that causes this 
segfault? If I can reproduce it here I can likely fix it. My best guess is that 
it is an interned string issue when storing nested arrays in the user cache, but 
my quick little test of that here worked ok.

I’ve run unit tests from symfony2 and doctrine and PHP Core’s PHPT tests on 2 version of PHP

There is an increase in failure rate of PHP with APC. I ran valgrind on symfony2 following Rasmus’s email about debugging APC (using his valgrind options).

I’m not seeing a crash, but hopefully this information will be helpful. 



PHP-5.4.5-dev	Php info:
	Without APC
Symfony2	http://131.107.220.66/PFTT-Results/Share/Bug62442/php545dev_no-apc_symfony2.txt

phpunit -c vendor/symfony/symfony
Symfony2 w/ valgrind	http://131.107.220.66/PFTT-Results/Share/Bug62442/php545dev_no-apc_symfony2_valgrind.txt

USE_ZEND_ALLOC=0 valgrind --tool=memcheck --suppressions=/root/lerdorf_suppressions.txt --leak-check=yes --track-origins=yes --num-callers=30 --show-reachable=yes phpunit -c vendor/symfony/symfony
Doctrine2	php545dev_no-apc_doctrine2.txt

PHPT	http://131.107.220.66/PFTT-Results/Share/Bug62442/php545dev_no-apc_doctrine2.txt

5 fails
	APC-3.1.12-dev
Symfony2	http://131.107.220.66/PFTT-Results/Share/Bug62442/php545dev_no-apc_phpt.txt

phpunit -c vendor/symfony/symfony

Symfony2 w/ valgrind	http://131.107.220.66/PFTT-Results/Share/Bug62442/php545dev_apc_symfony2_valgrind.txt

USE_ZEND_ALLOC=0 valgrind --tool=memcheck --suppressions=/root/lerdorf_suppressions.txt --leak-check=yes --track-origins=yes --num-callers=30 --show-reachable=yes phpunit -c vendor/symfony/symfony/
Doctrine2	http://131.107.220.66/PFTT-Results/Share/Bug62442/php545dev_apc_doctrine2.txt

phpunit -c vendor/doctrine/common
PHPT	http://131.107.220.66/PFTT-Results/Share/Bug62442/php545dev_apc_phpt.txt

10 fails 
APC PHPT tests	http://131.107.220.66/PFTT-Results/Share/Bug62442/php545dev_apc_phpt_tests.txt

2 fails



PHP-5.4.5	Php info:
	With APC
Symfony2	http://131.107.220.66/PFTT-Results/Share/Bug62442/php545_no-apc_symfony2.txt

Doctrine2	http://131.107.220.66/PFTT-Results/Share/Bug62442/php545_no-apc_doctrine2.txt

PHPT	http://131.107.220.66/PFTT-Results/Share/Bug62442/php545_apc_phpt.txt

28 Fails
	APC-3.1.12-dev
Symfony2	http://131.107.220.66/PFTT-Results/Share/Bug62442/php545_apc_symfony2.txt

Doctrine2	http://131.107.220.66/PFTT-Results/Share/Bug62442/php545_apc_doctrine2.txt

PHPT	http://131.107.220.66/PFTT-Results/Share/Bug62442/php545_apc_phpt.txt

31 fails
APC PHPT tests	http://131.107.220.66/PFTT-Results/Share/Bug62442/php545_apc_phpt_tests.txt

2 fails

Sorry for the delay in replay - per Rasmus - we have a lot of APC caching of 
YAML arrays / strings / keys + values, etc.

I saw that some updates on PHP and APC were done, however I had a major crash 
and burn this time (PHP 5.4.5 plus APC 3.1.12 - trunk #326931) - it core dumped 
all over the place.  I saw some recent APC changes around shutdown so I think 
maybe there's a new bug?

Core was generated by `/service/local/apache/bin/httpd -f 
/service/conf/httpd.php.conf'.
Program terminated with signal 11, Segmentation fault.
#0  zend_mm_remove_from_free_list (heap=0x844be80, p=<value optimized out>)
    at /source/release/latvia/external/build/php-5.4.5/Zend/zend_alloc.c:786
786	/source/release/latvia/external/build/php-5.4.5/Zend/zend_alloc.c: No 
such file or directory.
	in /source/release/latvia/external/build/php-5.4.5/Zend/zend_alloc.c
(gdb) bt
#0  zend_mm_remove_from_free_list (heap=0x844be80, p=<value optimized out>)
    at /source/release/latvia/external/build/php-5.4.5/Zend/zend_alloc.c:786
#1  _zend_mm_free_int (heap=0x844be80, p=<value optimized out>)
    at /source/release/latvia/external/build/php-5.4.5/Zend/zend_alloc.c:2106
#2  0xf54a7193 in apc_php_free (p=0xdcd81e50) at 
/tmp/pear/temp/APC/apc_zend.c:43
#3  0xf54a5ff3 in apc_deactivate () at /tmp/pear/temp/APC/apc_main.c:980
#4  apc_request_shutdown () at /tmp/pear/temp/APC/apc_main.c:1037
#5  0xf549ba55 in zm_deactivate_apc (type=1, module_number=44) at 
/tmp/pear/temp/APC/php_apc.c:407
#6  0xf76f08b8 in zend_deactivate_modules () at 
/source/release/latvia/external/build/php-5.4.5/Zend/zend_API.c:2333
#7  0xf7696dbf in php_request_shutdown (dummy=0x0) at 
/source/release/latvia/external/build/php-5.4.5/main/main.c:1750
#8  0xf7772b4e in php_apache_request_dtor (r=0x87fbc68)
    at /source/release/latvia/external/build/php-
5.4.5/sapi/apache2handler/sapi_apache2.c:507
#9  php_handler (r=0x87fbc68) at /source/release/latvia/external/build/php-
5.4.5/sapi/apache2handler/sapi_apache2.c:679
#10 0x0807afc2 in ap_run_handler (r=0x87fbc68) at config.c:157
#11 0x0807b397 in ap_invoke_handler (r=0x87fbc68) at config.c:376
#12 0x080a46d5 in ap_process_request (r=0x87fbc68) at http_request.c:282
#13 0x080a2099 in ap_process_http_connection (c=0x87f5de8) at http_core.c:190
#14 0x0808092a in ap_run_process_connection (c=0x87f5de8) at connection.c:43
#15 0x080cb42e in child_main (child_num_arg=<value optimized out>) at 
prefork.c:667
#16 0x080cb606 in make_child (s=<value optimized out>, slot=2) at prefork.c:768
#17 0x080cb694 in startup_children (number_to_start=3) at prefork.c:786
#18 0x080cbebc in ap_mpm_run (_pconf=0x83f30a8, plog=0x843b1c8, s=0x83fc010) at 
prefork.c:1007
#19 0x0806ac71 in main (argc=3, argv=0xffa55cc4) at main.c:753

Thanks for staying on this, could you please check again certanly for that last 
crash again?

I updated our APC with the latest from trunk (3.1.12 v #326996 as of 8/6/2012), and ran things again.

Lots of cores again, here's a typical trace:

Program terminated with signal 11, Segmentation fault.
#0  0xf7744097 in zend_mm_remove_from_free_list (heap=0x87e9e60, p=<value optimized out>) at 
/source/release/latvia/external/build/php-5.4.5/Zend/zend_alloc.c:833
833     /source/release/latvia/external/build/php-5.4.5/Zend/zend_alloc.c: No such file or directory.
        in /source/release/latvia/external/build/php-5.4.5/Zend/zend_alloc.c
(gdb) bt
#0  0xf7744097 in zend_mm_remove_from_free_list (heap=0x87e9e60, p=<value optimized out>) at 
/source/release/latvia/external/build/php-5.4.5/Zend/zend_alloc.c:833
#1  _zend_mm_free_int (heap=0x87e9e60, p=<value optimized out>) at /source/release/latvia/external/build/php-
5.4.5/Zend/zend_alloc.c:2101
#2  0xf55211a3 in apc_php_free (p=0x8c1a1d8) at /tmp/pear/temp/APC/apc_zend.c:43
#3  0xf551b5e1 in my_free_hashtable (ht=0x8c26504, type=1) at /tmp/pear/temp/APC/apc_compile.c:920
#4  0xf551df8a in apc_free_class_entry_after_execution (src=0x8c264c4) at /tmp/pear/temp/APC/apc_compile.c:2030
#5  0xf551ff1c in apc_deactivate () at /tmp/pear/temp/APC/apc_main.c:949
#6  apc_request_shutdown () at /tmp/pear/temp/APC/apc_main.c:1041
#7  0xf5515a55 in zm_deactivate_apc (type=1, module_number=44) at /tmp/pear/temp/APC/php_apc.c:407
#8  0xf776a8b8 in zend_deactivate_modules () at /source/release/latvia/external/build/php-5.4.5/Zend/zend_API.c:2333
#9  0xf7710dbf in php_request_shutdown (dummy=0x0) at /source/release/latvia/external/build/php-5.4.5/main/main.c:1750
#10 0xf77ecb4e in php_apache_request_dtor (r=0x8b99c50) at /source/release/latvia/external/build/php-
5.4.5/sapi/apache2handler/sapi_apache2.c:507
#11 php_handler (r=0x8b99c50) at /source/release/latvia/external/build/php-5.4.5/sapi/apache2handler/sapi_apache2.c:679
#12 0x0807afc2 in ap_run_handler (r=0x8b99c50) at config.c:157
#13 0x0807b397 in ap_invoke_handler (r=0x8b99c50) at config.c:376
#14 0x080a46d5 in ap_process_request (r=0x8b99c50) at http_request.c:282
#15 0x080a2099 in ap_process_http_connection (c=0x8b93dd0) at http_core.c:190
#16 0x0808092a in ap_run_process_connection (c=0x8b93dd0) at connection.c:43
#17 0x080cb42e in child_main (child_num_arg=<value optimized out>) at prefork.c:667
#18 0x080cb606 in make_child (s=<value optimized out>, slot=15) at prefork.c:768
#19 0x080cc18d in ap_mpm_run (_pconf=0x87910a8, plog=0x87d91c8, s=0x879a010) at prefork.c:903
#20 0x0806ac71 in main (argc=3, argv=0xfffd0a54) at main.c:753

The crashes should be gone now, could you pleas pick up the current APC trunk for 
testing again?

Yes updated APC to trunk today and gave it a shot, unfortunately some unit tests 
are missing - seem to be in package.xml, but not in tests/ folder:

ERROR: file /tmp/pear/temp/tmpJhlz6f/APC-3.1.12/tests/apc54_bug62502.phpt does not 
exist

I can edit package.xml manually and try again, otherwise someone may want to 
update package.xml first?

Hi,

thanks for remarking that, I've updated the package.xml . Generally a one missing 
test isn't essential for you to test I think as the issue you're expieriencing 
more complex. So if you have to package, you could surely locally correct the 
package.xml. Otherwise just go phpize way.

Getting something very similar
- PHP 5.4.4
- APC 3.1.12
- Apache 2.2.10
- SLES 11
- also a huge site and this happens randomly, but using 3.1.10 or 3.1.11 fixes 
it completely

#0  apc_new_interned_string (arKey=0x7fac7f94a2b0 "load", nKeyLength=5) at 
/tmp/pear/temp/APC/apc_string.c:86
#1  0x00007fac73180bd7 in apc_string_pmemcpy (str=0x7fac7f94a2b0 "load", len=5, 
pool=0x7fac6b7a32c8) at /tmp/pear/temp/APC/apc_compile.c:291
#2  0x00007fac731845b3 in my_copy_zval (dst=0x7fac6b7a53e8, src=0x7fac7f94b908, 
ctxt=0x7fff65dfbbd0) at /tmp/pear/temp/APC/apc_compile.c:348
#3  0x00007fac73181cfb in apc_copy_op_array (dst=0x7fac6b7a41f8, 
src=0x7fac7f949e80, ctxt=0x7fff65dfbbd0) at 
/tmp/pear/temp/APC/apc_compile.c:1211
#4  0x00007fac73182576 in my_copy_function (dst=0x7fac6b7a41f8, 
src=0x7fac7f949e80, ctxt=0x7fff65dfbbd0) at /tmp/pear/temp/APC/apc_compile.c:490
#5  0x00007fac73184292 in my_copy_hashtable_ex (dst=0x7fac6b7a39d0, src=<value 
optimized out>, copy_fn=0x7fac731824f0 <my_copy_function>, holds_ptrs=0, 
ctxt=0x7fff65dfbbd0, 
    check_fn=0x7fac73180aa0 <my_check_copy_function>) at 
/tmp/pear/temp/APC/apc_compile.c:1034
#6  0x00007fac73182900 in my_copy_class_entry (dst=0x7fac6b7a39a8, 
src=0x7fac7f9490f0, ctxt=0x7fff65dfbbd0) at /tmp/pear/temp/APC/apc_compile.c:687
#7  0x00007fac73183ada in apc_copy_new_classes (op_array=<value optimized out>, 
old_count=<value optimized out>, ctxt=0x7fff65dfbbd0) at 
/tmp/pear/temp/APC/apc_compile.c:1544
#8  0x00007fac73185420 in apc_compile_cache_entry (key=0x7fff65dfcdf0, 
h=0x7fff65dfd070, type=<value optimized out>, t=<value optimized out>, 
op_array=0x7fff65dfce48, 
    cache_entry=0x7fff65dfce50) at /tmp/pear/temp/APC/apc_main.c:446
#9  0x00007fac73185c5b in my_compile_file (h=0x7fff65dfd070, type=8) at 
/tmp/pear/temp/APC/apc_main.c:603
#10 0x00007fac7b9c97fa in phar_compile_file (file_handle=0x7fff65dfd070, type=8) 
at /usr/local/src/php/php-5.4.4/ext/phar/phar.c:3391

The original issue was reported exactly with 3.1.11. Looks like the bug is in every version, but your app just hits that with 3.1.12. How were it with testing trunk? There was a lot of fixes ever since last release. Also a repro code in PHP were helpful.

This might be useful. We are seeing this issue on a number of boxes running

PHP 5.3.3-7+squeeze14
APC 3.1.13

We started on APC version 3.1.9 but all updates have resulted in the same issues. It happens on "require_once" calls. It is random. We also have a number of servers not effected by this at all. Here are the differences I can offer for your considerations:

Servers that HAVE the problem are hosted on a 64bit system running on Amazon AWS EC2 instances.

Servers that DO NOT have the problem are hosted on 32bit system running on regular hardware managed by us.

I am not sure there is any correlation here, but thought I pass it along. Thanks for working on this bug.

I'm going to try to get a test out today...12/20/2012

I took the latest code from trunk.  I rolled it out to our servers today.

Out of 12 machines I turned on, 1 of the machines was seg faulting and giving errors due to a seemingly corrupted / 
failed to load file that's stored in the user cache space.

There were a couple of errors from other machines in UniversalClassLoader.php about some file that didn't exist (but it 
had a confused filename, as if the filename as was a key for something in the user-cache).

I've rolled back to our prior state on those machines and all is well again.

Revisiting this bug.

If anyone has the time and the old core dump, I would appreciate the following 
run on gdb

(gdb) list

(gdb) x/12i $pc

at the coredump.

No feedback was provided. The bug is being suspended because
we assume that you are no longer experiencing the problem.
If this is not the case and you are able to provide the
information that was requested earlier, please do so and
change the status of the bug back to "Re-Opened". Thank you.