Same segfault, other stacktrace, don't think it help a lot :

Program terminated with signal 11, Segmentation fault.
#0  _zend_mm_alloc_int (heap=0x143a330, size=82) at /usr/src/php-5.4.13/Zend/zend_alloc.c:2016
2016                    ZEND_MM_CHECK_BLOCK_LINKAGE(best_fit);

(gdb) bt
#0  _zend_mm_alloc_int (heap=0x143a330, size=82) at /usr/src/php-5.4.13/Zend/zend_alloc.c:2016
#1  0x0000000000691791 in _zend_hash_quick_add_or_update (ht=0x1675e18, arKey=0x7fc905b7fb50 "regexChar", nKeyLength=<value optimized out>, h=8246864001117707262, pData=0x1, nDataSize=8, pDest=0x7fc9207513a8, flag=1) at /usr/src/php-5.4.13/Zend/zend_hash.c:330
#2  0x00000000006a9948 in _get_zval_cv_lookup_BP_VAR_W (ptr=0x7fc9207513a8, var=<value optimized out>) at /usr/src/php-5.4.13/Zend/zend_execute.c:281
#3  0x000000000070557a in _get_zval_ptr_ptr_cv_BP_VAR_W (execute_data=0x7fc9207512c8) at /usr/src/php-5.4.13/Zend/zend_execute.c:442
#4  ZEND_ASSIGN_SPEC_CV_VAR_HANDLER (execute_data=0x7fc9207512c8) at /usr/src/php-5.4.13/Zend/zend_vm_execute.h:33048
#5  0x00000000006e8990 in execute (op_array=0x1a22840) at /usr/src/php-5.4.13/Zend/zend_vm_execute.h:410
#6  0x0000000000676473 in zend_call_function (fci=0x7fff3c616460, fci_cache=<value optimized out>) at /usr/src/php-5.4.13/Zend/zend_execute_API.c:958
#7  0x000000000055bf1a in zim_reflection_method_invokeArgs (ht=<value optimized out>, return_value=0x166df40, return_value_ptr=<value optimized out>, this_ptr=<value optimized out>, return_value_used=<value optimized out>)
    at /usr/src/php-5.4.13/ext/reflection/php_reflection.c:3017
#8  0x00000000006fb5dc in zend_do_fcall_common_helper_SPEC (execute_data=0x7fc92074d7f8) at /usr/src/php-5.4.13/Zend/zend_vm_execute.h:642
#9  0x00000000006e8990 in execute (op_array=0x17b52b0) at /usr/src/php-5.4.13/Zend/zend_vm_execute.h:410
#10 0x0000000000681d9e in zend_execute_scripts (type=8, retval=<value optimized out>, file_count=3) at /usr/src/php-5.4.13/Zend/zend.c:1315
#11 0x000000000062746e in php_execute_script (primary_file=<value optimized out>) at /usr/src/php-5.4.13/main/main.c:2492
#12 0x0000000000730fda in main (argc=<value optimized out>, argv=<value optimized out>) at /usr/src/php-5.4.13/sapi/fpm/fpm/fpm_main.c:1924
(gdb) list
2021    
2022            remaining_size = block_size - true_size;
2023    
2024            if (remaining_size < ZEND_MM_ALIGNED_MIN_HEADER_SIZE) {
2025                    true_size = block_size;
2026                    ZEND_MM_BLOCK(best_fit, ZEND_MM_USED_BLOCK, true_size);
2027            } else {
2028                    zend_mm_free_block *new_free_block;
2029    
2030                    /* prepare new free block */
(gdb) p *best_fit->info._prev
Cannot access memory at address 0x64696c61766e49

Same as the last, seems ASCII data instead of memory pointer :

$ echo $'\x64\x69\x6c\x61\x76\x6e\x49'
dilavnI

Do you have a small script that can reproduce this easily (even if it's just 
0.018% of the time)?

No, sadly, for the moment we do not have any small script to reproduce it. It happen some times in our production servers, but never in our development one, so, for the moment, we can't try to reduce the script to a minimal test case...

Good news of the day : We have collected some core dumps, and the URL producing the segfault is always the same.

Bad news of the day : This URL does a lot of work, so it's not a "little script".

Bad news of the day 2 : If we restart php-fpm, for a few minutes it will not segfault, we have to let some users hit the server first, wait a bit, and it will start to segfault.

Bad news of the day 3 : As we have to wait for traffic to see the segfault we can't reproduce it under valgrind.

Sometimes segfault occur in php_request_shutdown (57 times since a few days) and sometimes in php_execute_script (32 times in the same timespan).

Here are two segfaults occuring during php_execute_script :

Program terminated with signal 11, Segmentation fault.
#0  _zend_mm_alloc_int (heap=0x143a330, size=72) at /usr/src/php-5.4.13/Zend/zend_alloc.c:2016
2016                    ZEND_MM_CHECK_BLOCK_LINKAGE(best_fit);
(gdb) p best_fit
$1 = (zend_mm_free_block *) 0x1c7e050
(gdb) p *best_fit
$2 = {info = {_size = 7308604897320202088, _prev = 28263411883601481}, prev_free_block = 0x1c7e710, next_free_block = 0x143a728, parent = 0x687461703f2f6e75, child = {0x31243d, 0x59}}

core.php-fpm.11335

#0  _zend_mm_realloc_int (heap=0x143a330, p=0x1665e78, size=452) at /usr/src/php-5.4.13/Zend/zend_alloc.c:2151
2151                            if (ZEND_MM_IS_FREE_BLOCK(next_block)) {
(gdb) p *next_block
Cannot access memory at address 0x656d616e75d9cdd0

do you use any non-offcial php extension, includes the exts at PECL?

@laruence :

Yes, exactly : rar-3.0.1 mongo-1.3.5 APC-3.1.13

And PHP compiled from sources (5.4.13) with :

./configure --disable-all --prefix=/usr/local/php-5.4.13 --enable-fpm --enable-ctype --enable-mbstring --enable-gd-native-ttf --enable-zip --with-mcrypt --with-openssl --with-gd --with-jpeg-dir=/usr/lib --with-freetype-dir --with-curl --with-pcre-regex --with-gettext --enable-pdo --with-pdo-mysql=mysqlnd --with-iconv --enable-fileinfo --enable-filter --enable-json --enable-session --enable-hash --enable-libxml --enable-dom --enable-libxml --enable-simplexml --enable-bcmath

could you please disable all these exts and try ?

I assume the segfault is caused by some out-bounder write

@laruence :

> could you please disable all these exts and try ?

As we are unable to reproduce the bug in a dev server, and only able to reproduce it after a buch of traffic went to it, it's not an option to disable modules in production.

But i have a hint : We do not use rar plugin for this request.
And another hint : Since I restarted php-fpm at 11h46 today (to test with valgrind) it never segfaulted (previously we had almost 1 segfault / hour), like if it's an APC cache "random bug" corrupting something at compile time.

So two solutions :
 * As i though in the past, the bug takes times to come (users / requests / time ...)
 * The bug does not happen at every restart of PHP-FPM but take place at bytecode-compiling time, murphy helping in not producing the bug when I want to try, leading me to think that a fresh php-fpm does not segfault.


> I assume the segfault is caused by some out-bounder write

Same asumption here.

No feedback was provided. The bug is being suspended because
we assume that you are no longer experiencing the problem.
If this is not the case and you are able to provide the
information that was requested earlier, please do so and
change the status of the bug back to "Re-Opened". Thank you.