Thank you for this bug report. To properly diagnose the problem, we
need a short but complete example script to be able to reproduce
this bug ourselves. 

A proper reproducing script starts with <?php and ends with ?>,
is max. 10-20 lines long and does not require any external 
resources such as databases, etc. If the script requires a 
database to demonstrate the issue, please make sure it creates 
all necessary tables, stored procedures etc.

Please avoid embedding huge scripts into the report.

It's probably not possible to boil into a single script as we're running symfony2 
which is pulling in probably hundreds of files and various dependencies just to 
generate a single page.

It does seem to happen rather consistently under simple manual testing of our site 
on a QA machine.

Is there anything else we can do to help isolate the issue?

From what i could analyse yet, this issue locale related. It happens on prefork 
child shutdown, so also PHP module shutdown. From this point not very bad. The 
PCRE patterns are cached by locale, so it might be reproduceable with a scenario 
like this

- set locale
- do some pcre stuff
- change locale
- do some pcre stuff
.......

This is most likely a race condition in MSHUTDOWN while freeing PCRE cache under 
Apache prefork.

You could try a simple script with this scenario. I'll be doing the same in the 
meantime. Or maybe you recognize this pattern in your app? Unfortunately that's 
all I could read from the BT so far.

Thanks

Upgraded to PHP 5.4.17, fewer cores, it seems, but still seeing a couple so 
far...

The backtrace has changed, though:
----------------------------------

Core was generated by `/service/local/apache/bin/httpd -f 
/service/conf/httpd.qvc.conf'.
Program terminated with signal 11, Segmentation fault.
#0  __memcmp_sse2 () at ../sysdeps/x86_64/memcmp.S:57
57		movl	(%rdi),	%eax
(gdb) bt
#0  __memcmp_sse2 () at ../sysdeps/x86_64/memcmp.S:57
#1  0x00007f15016adb0a in zend_mm_check_ptr (heap=0xdc3a10, ptr=0x7f14dc01e2a8, 
silent=0, __zend_filename=0x7f1501c250a8 "/workspace/source/external/build/php-
5.4.17-apache/Zend/zend_opcode.c", __zend_lineno=364, __zend_orig_filename=0x0, 
__zend_orig_lineno=0)
    at /workspace/source/external/build/php-5.4.17-apache/Zend/zend_alloc.c:1515
#2  0x00007f15016ad65a in zend_mm_check_ptr (heap=0xdc3a10, ptr=0x7f14dc01e2a8, 
silent=1, __zend_filename=0x7f1501c250a8 "/workspace/source/external/build/php-
5.4.17-apache/Zend/zend_opcode.c", __zend_lineno=364, __zend_orig_filename=0x0, 
__zend_orig_lineno=0)
    at /workspace/source/external/build/php-5.4.17-apache/Zend/zend_alloc.c:1416
#3  0x00007f15016af182 in _zend_mm_free_int (heap=0xdc3a10, p=0x7f14dc01e2a8, 
__zend_filename=0x7f1501c250a8 "/workspace/source/external/build/php-5.4.17-
apache/Zend/zend_opcode.c", __zend_lineno=364, __zend_orig_filename=0x0, 
__zend_orig_lineno=0)
    at /workspace/source/external/build/php-5.4.17-apache/Zend/zend_alloc.c:2064
#4  0x00007f15016b080d in _efree (ptr=0x7f14dc01e2a8, 
__zend_filename=0x7f1501c250a8 "/workspace/source/external/build/php-5.4.17-
apache/Zend/zend_opcode.c", __zend_lineno=364, __zend_orig_filename=0x0, 
__zend_orig_lineno=0) at /workspace/source/external/build/php-5.4.17-
apache/Zend/zend_alloc.c:2436
#5  0x00007f15016d95a1 in destroy_op_array (op_array=0x1470ab8) at 
/workspace/source/external/build/php-5.4.17-apache/Zend/zend_opcode.c:364
#6  0x00007f15016d890f in destroy_zend_function (function=0x1470ab8) at 
/workspace/source/external/build/php-5.4.17-apache/Zend/zend_opcode.c:112
#7  0x00007f15016d8929 in zend_function_dtor (function=0x1470ab8) at 
/workspace/source/external/build/php-5.4.17-apache/Zend/zend_opcode.c:124
#8  0x00007f15016f9ecf in zend_hash_destroy (ht=0x1470698) at 
/workspace/source/external/build/php-5.4.17-apache/Zend/zend_hash.c:560
#9  0x00007f15016d923d in destroy_zend_class (pce=0x3586b48) at 
/workspace/source/external/build/php-5.4.17-apache/Zend/zend_opcode.c:296
#10 0x00007f15016fa2bc in zend_hash_apply_deleter (ht=0xdc4370, p=0x3586b30) at 
/workspace/source/external/build/php-5.4.17-apache/Zend/zend_hash.c:650
#11 0x00007f15016fa979 in zend_hash_reverse_apply (ht=0xdc4370, 
apply_func=0x7f15016d200e <clean_non_persistent_class>) at 
/workspace/source/external/build/php-5.4.17-apache/Zend/zend_hash.c:804
#12 0x00007f15016d287e in shutdown_executor () at 
/workspace/source/external/build/php-5.4.17-apache/Zend/zend_execute_API.c:303
#13 0x00007f15016e6e56 in zend_deactivate () at 
/workspace/source/external/build/php-5.4.17-apache/Zend/zend.c:938
#14 0x00007f1501652b03 in php_request_shutdown (dummy=0x0) at 
/workspace/source/external/build/php-5.4.17-apache/main/main.c:1800
#15 0x00007f1501792808 in php_apache_request_dtor (r=0x1383150) at 
/workspace/source/external/build/php-5.4.17-
apache/sapi/apache2handler/sapi_apache2.c:507
#16 0x00007f1501793079 in php_handler (r=0x1383150) at 
/workspace/source/external/build/php-5.4.17-
apache/sapi/apache2handler/sapi_apache2.c:679
#17 0x0000000000441d90 in ap_run_handler (r=0x1383150) at config.c:158
#18 0x00000000004453ee in ap_invoke_handler (r=0x1383150) at config.c:376
#19 0x000000000048ca60 in ap_process_request (r=0x1383150) at http_request.c:282
#20 0x0000000000489a08 in ap_process_http_connection (c=0x13752d0) at 
http_core.c:190
#21 0x0000000000449330 in ap_run_process_connection (c=0x13752d0) at 
connection.c:43
#22 0x00000000004b9bc8 in child_main (child_num_arg=<value optimized out>) at 
prefork.c:667
#23 0x00000000004b9ec4 in make_child (s=0xc51c80, slot=16) at prefork.c:768
#24 0x00000000004bab17 in perform_idle_server_maintenance (_pconf=<value 
optimized out>, plog=<value optimized out>, s=<value optimized out>) at 
prefork.c:903
#25 ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>, s=
<value optimized out>) at prefork.c:1107
#26 0x000000000042e524 in main (argc=3, argv=0x7fffd6d0d018) at main.c:753

Yep, the last BT is very far from the first one. It even doesn't mention PCRE at 
all. But nevertheless, seems the last BT is done with a debug build. Was there 
something interesting on stderr? Also the last BT might be not reproducable with a 
release build. I'm stuck reproducing your first BT, sadly.

I'm not understanding you about the "debug" build part?  I don't think it gives a 
backtrace without --enable-debug, unless you are referring to apache being 
compiled in Debug mode as well...

Yes, i meant PHP --enable-debug. That's a bit tricky, with gcc using --enable-
debug directly will define some macros which switch the code parts. Using no --
enable-debug one still can enforce debug symbols using sometihng like
CFLAGS="-ggdb -O3" CXXFLAGS="$CFLAGS" ./configure
That's why i meant it might be not reproduceable without --enable-debug as the 
code corresponding to the last BT is different then. Despite all that, i think the  
first PCRE backtrace is a real bug.

Okay, I recompiled without --enable-debug, and pounded the server and got two core files that are exactly the same backtrace-wise...


Program terminated with signal 11, Segmentation fault.
#0  _zend_mm_free_int (heap=0x1bea970, p=0x7f1393ded878) at /workspace/source/external/build/php-5.4.17-apache/Zend/zend_alloc.c:2100
2100	/workspace/source/external/build/php-5.4.17-apache/Zend/zend_alloc.c: No such file or directory.
	in /workspace/source/external/build/php-5.4.17-apache/Zend/zend_alloc.c
(gdb) bt
#0  _zend_mm_free_int (heap=0x1bea970, p=0x7f1393ded878) at /workspace/source/external/build/php-5.4.17-apache/Zend/zend_alloc.c:2100
#1  0x00007f13b8619e65 in destroy_op_array (op_array=0x28ba910) at /workspace/source/external/build/php-5.4.17-
apache/Zend/zend_opcode.c:364
#2  0x00007f13b863062b in zend_hash_destroy (ht=0x28b9748) at /workspace/source/external/build/php-5.4.17-apache/Zend/zend_hash.c:560
#3  0x00007f13b861a22e in destroy_zend_class (pce=<value optimized out>) at /workspace/source/external/build/php-5.4.17-
apache/Zend/zend_opcode.c:296
#4  0x00007f13b86302b5 in zend_hash_apply_deleter (ht=0x1beb2d0, p=0x27a7a10) at /workspace/source/external/build/php-5.4.17-
apache/Zend/zend_hash.c:650
#5  0x00007f13b86303c9 in zend_hash_reverse_apply (ht=0x1beb2d0, apply_func=0x7f13b8614c30 <clean_non_persistent_class>) at 
/workspace/source/external/build/php-5.4.17-apache/Zend/zend_hash.c:804
#6  0x00007f13b8618486 in shutdown_executor () at /workspace/source/external/build/php-5.4.17-apache/Zend/zend_execute_API.c:303
#7  0x00007f13b86239e2 in zend_deactivate () at /workspace/source/external/build/php-5.4.17-apache/Zend/zend.c:938
#8  0x00007f13b85bff3b in php_request_shutdown (dummy=<value optimized out>) at /workspace/source/external/build/php-5.4.17-
apache/main/main.c:1800
#9  0x00007f13b86d08e7 in php_apache_request_dtor (r=0x30de620) at /workspace/source/external/build/php-5.4.17-
apache/sapi/apache2handler/sapi_apache2.c:507
#10 php_handler (r=0x30de620) at /workspace/source/external/build/php-5.4.17-apache/sapi/apache2handler/sapi_apache2.c:679
#11 0x0000000000441d20 in ap_run_handler (r=0x30de620) at config.c:158
#12 0x000000000044534e in ap_invoke_handler (r=0x30de620) at config.c:376
#13 0x000000000048c180 in ap_process_request (r=0x30de620) at http_request.c:282
#14 0x0000000000489140 in ap_process_http_connection (c=0x219b640) at http_core.c:190
#15 0x00000000004492b0 in ap_run_process_connection (c=0x219b640) at connection.c:43
#16 0x00000000004b9078 in child_main (child_num_arg=<value optimized out>) at prefork.c:667
#17 0x00000000004b9374 in make_child (s=0x1a78c80, slot=14) at prefork.c:768
#18 0x00000000004b9fc7 in perform_idle_server_maintenance (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized 
out>) at prefork.c:903
#19 ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at prefork.c:1107
#20 0x000000000042e564 in main (argc=3, argv=0x7fff7846b3c8) at main.c:753

Core was generated by `/service/local/apache/bin/httpd -f /service/conf/httpd.qvc.conf'.
Program terminated with signal 11, Segmentation fault.
#0  _zend_mm_free_int (heap=0x1bea970, p=0x7f1393ded878) at /workspace/source/external/build/php-5.4.17-apache/Zend/zend_alloc.c:2100
2100	/workspace/source/external/build/php-5.4.17-apache/Zend/zend_alloc.c: No such file or directory.
	in /workspace/source/external/build/php-5.4.17-apache/Zend/zend_alloc.c
(gdb) bt
#0  _zend_mm_free_int (heap=0x1bea970, p=0x7f1393ded878) at /workspace/source/external/build/php-5.4.17-apache/Zend/zend_alloc.c:2100
#1  0x00007f13b8619e65 in destroy_op_array (op_array=0x5953800) at /workspace/source/external/build/php-5.4.17-
apache/Zend/zend_opcode.c:364
#2  0x00007f13b863062b in zend_hash_destroy (ht=0x5a51cc8) at /workspace/source/external/build/php-5.4.17-apache/Zend/zend_hash.c:560
#3  0x00007f13b861a22e in destroy_zend_class (pce=<value optimized out>) at /workspace/source/external/build/php-5.4.17-
apache/Zend/zend_opcode.c:296
#4  0x00007f13b86302b5 in zend_hash_apply_deleter (ht=0x1beb2d0, p=0x4e33d70) at /workspace/source/external/build/php-5.4.17-
apache/Zend/zend_hash.c:650
#5  0x00007f13b86303c9 in zend_hash_reverse_apply (ht=0x1beb2d0, apply_func=0x7f13b8614c30 <clean_non_persistent_class>) at 
/workspace/source/external/build/php-5.4.17-apache/Zend/zend_hash.c:804
#6  0x00007f13b8618486 in shutdown_executor () at /workspace/source/external/build/php-5.4.17-apache/Zend/zend_execute_API.c:303
#7  0x00007f13b86239e2 in zend_deactivate () at /workspace/source/external/build/php-5.4.17-apache/Zend/zend.c:938
#8  0x00007f13b85bff3b in php_request_shutdown (dummy=<value optimized out>) at /workspace/source/external/build/php-5.4.17-
apache/main/main.c:1800
#9  0x00007f13b86d08e7 in php_apache_request_dtor (r=0x21a94c0) at /workspace/source/external/build/php-5.4.17-
apache/sapi/apache2handler/sapi_apache2.c:507
#10 php_handler (r=0x21a94c0) at /workspace/source/external/build/php-5.4.17-apache/sapi/apache2handler/sapi_apache2.c:679
#11 0x0000000000441d20 in ap_run_handler (r=0x21a94c0) at config.c:158
#12 0x000000000044534e in ap_invoke_handler (r=0x21a94c0) at config.c:376
#13 0x000000000048c180 in ap_process_request (r=0x21a94c0) at http_request.c:282
#14 0x0000000000489140 in ap_process_http_connection (c=0x219b640) at http_core.c:190
#15 0x00000000004492b0 in ap_run_process_connection (c=0x219b640) at connection.c:43
#16 0x00000000004b9078 in child_main (child_num_arg=<value optimized out>) at prefork.c:667
#17 0x00000000004b9374 in make_child (s=0x1a78c80, slot=4) at prefork.c:768
#18 0x00000000004b967e in startup_children (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at 
prefork.c:786
#19 ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>, s=<value optimized out>) at prefork.c:1007
#20 0x000000000042e564 in main (argc=3, argv=0x7fff7846b3c8) at main.c:753
(gdb) quit