php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #74403 Sporadic FPM segfaults
Submitted: 2017-04-10 11:08 UTC Modified: 2017-04-14 02:52 UTC
From: tony2001@php.net Assigned: laruence (profile)
Status: Assigned Package: PCRE related
PHP Version: 7.1.4RC1 OS: Linux
Private report: No CVE-ID: None
Have you experienced this issue?
Rate the importance of this bug to you:

 [2017-04-10 11:08 UTC] tony2001@php.net
Description:
------------
Patch ebfd93f725eb9b9cfadbbde98047efe76f658da6 introduced an issue that is both hard to debug and reproduce. FPM processes start to crash in completely random places (mostly pointing to the executor code), all FPM children at once.

Unfortunately, I don't have a reproduce case and I don't know of any reliable way to reproduce it, other than to start FPM on production server under a heavy load and leave it for a while.

I've spent a lot of time debugging it and have found that the problem goes away as soon as I revert ebfd93f725eb9b9cfadbbde98047efe76f658da6. After looking at the patch closely, I've noticed that the patch changes one tricky part of the code managing refcount of the string stored in the cache. In the new version that part is gone and reverting this part of the patch fixes the issue, see attached patch.

Test script:
---------------
.

Expected result:
----------------
No crashes.

Actual result:
--------------
Sporadic FPM segfaults.

Patches

pcre-crashes-fix.diff (last revision 2017-04-10 11:09 UTC) by tony2001@php.net)

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2017-04-10 11:09 UTC] tony2001@php.net
The following patch has been added/updated:

Patch Name: pcre-crashes-fix.diff
Revision:   1491822546
URL:        https://bugs.php.net/patch-display.php?bug=74403&patch=pcre-crashes-fix.diff&revision=1491822546
 [2017-04-10 11:17 UTC] requinix@php.net
-Summary: Sporadi +Summary: Sporadic FPM segfaults
 [2017-04-10 11:17 UTC] requinix@php.net
There have been a few php-fpm crashes reported over the last couple months. Can you tell if any of them are related to this?
 [2017-04-10 11:22 UTC] tony2001@php.net
https://bugs.php.net/bug.php?id=73740 - this one looks familiar.
 [2017-04-10 11:23 UTC] nikic@php.net
-Assigned To: +Assigned To: laruence
 [2017-04-10 11:45 UTC] laruence@php.net
-Status: Assigned +Status: Feedback
 [2017-04-10 11:45 UTC] laruence@php.net
Thank you for this bug report. To properly diagnose the problem, we
need a backtrace to see what is happening behind the scenes. To
find out how to generate a backtrace, please read
http://bugs.php.net/bugs-generating-backtrace.php for *NIX and
http://bugs.php.net/bugs-generating-backtrace-win32.php for Win32

Once you have generated a backtrace, please submit it to this bug
report and change the status back to "Open". Thank you for helping
us make PHP better.

as we discussed early in mails, I don't think that diretctly relates to that change,

anyway, could you please paste the backtrace out?

thanks
 [2017-04-11 09:51 UTC] tony2001@php.net
Reverted the attached patch in my private branch and deployed FPM on one production server.
After about 6 hours I've got several cores with the backtraces below. Number of segfaults on several hundred servers since March 22: 0.

backtrace #1:
#0  zend_hash_find_bucket (ht=ht@entry=0xdd28e1cadaf9342c, ht=ht@entry=0xdd28e1cadaf9342c, key=0x7f37b66bdf00, key@entry=0x7f57b453b718)
    at /home/tony/php-build/php-src/Zend/zend_hash.c:496
#1  zend_hash_find (ht=ht@entry=0x7f37b4539720, key=key@entry=0x7f37b66bdf00) at /home/tony/php-build/php-src/Zend/zend_hash.c:1988
#2  0x00000000006da2e6 in zend_hash_find_ind (key=0x7f37b66bdf00, ht=0x7f37b4539720) at /home/tony/php-build/php-src/Zend/zend_hash.h:278
#3  ZEND_ISSET_ISEMPTY_DIM_OBJ_SPEC_CV_CV_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:37856
#4  0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#5  0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c18720) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#6  0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800
#7  0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#8  0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c18660) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#9  0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800
#10 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#11 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c18480) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#12 0x000000000065b919 in zend_call_function (fci=fci@entry=0x7ffcffa5d8d0, fci_cache=fci_cache@entry=0x7ffcffa5d8a0)
    at /home/tony/php-build/php-src/Zend/zend_execute_API.c:858
#13 0x0000000000689db3 in zend_call_method (object=0x0, obj_ce=<optimized out>, fn_proxy=0x7f3833c55870, function_name=0x7f3833c74248 "main_autoload", 
    function_name_len=<optimized out>, retval_ptr=retval_ptr@entry=0x0, param_count=1, arg1=0x7f3833c18470, arg2=0x0)
    at /home/tony/php-build/php-src/Zend/zend_interfaces.c:104
#14 0x0000000000525c4e in zif_spl_autoload_call (execute_data=<optimized out>, return_value=<optimized out>)
    at /home/tony/php-build/php-src/ext/spl/php_spl.c:421
#15 0x00007f382e4ca680 in memtrack_execute_internal (current_execute_data=0x7f3833c18410, return_value=0x7ffcffa5dbc0)
    at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:563
#16 0x000000000065bb57 in zend_call_function (fci=fci@entry=0x7ffcffa5dc00, fci_cache=fci_cache@entry=0x7ffcffa5dbd0)
    at /home/tony/php-build/php-src/Zend/zend_execute_API.c:880
#17 0x000000000065bee2 in zend_lookup_class_ex (name=name@entry=0x7f37b66bdf00, key=key@entry=0x0, use_autoload=use_autoload@entry=1)
    at /home/tony/php-build/php-src/Zend/zend_execute_API.c:1040
#18 0x000000000065c635 in zend_fetch_class (class_name=0x7f37b66bdf00, fetch_type=512) at /home/tony/php-build/php-src/Zend/zend_execute_API.c:1364
#19 0x00000000006e3d2a in ZEND_FETCH_CLASS_SPEC_CV_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:2330
#20 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#21 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c18070) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#22 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800
#23 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#24 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c14e60) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#25 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800
#26 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#27 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c134c0) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#28 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800
(gdb) p p
$1 = (Bucket *) 0x7f57b453b718
(gdb) p *p
Cannot access memory at address 0x7f57b453b718


backtrace #2:
#0  zend_verify_abstract_class (ce=0x7f37a95d414000) at /home/tony/php-build/php-src/Zend/zend_execute_API.c:1441
#1  0x00000000006b6f24 in ZEND_VERIFY_ABSTRACT_CLASS_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:1539
#2  0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#3  0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c15af0) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#4  0x000000000071673c in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:29532
#5  0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#6  0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c15720) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#7  0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800
#8  0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#9  0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c156a0) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#10 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800
#11 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#12 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c154c0) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#13 0x000000000065b919 in zend_call_function (fci=fci@entry=0x7ffcffa5d0c0, fci_cache=fci_cache@entry=0x7ffcffa5d090)
    at /home/tony/php-build/php-src/Zend/zend_execute_API.c:858
#14 0x0000000000689db3 in zend_call_method (object=0x0, obj_ce=<optimized out>, fn_proxy=0x7f3833c55810, function_name=0x7f3833c762e8 "main_autoload", 
    function_name_len=<optimized out>, retval_ptr=retval_ptr@entry=0x0, param_count=1, arg1=0x7f3833c154b0, arg2=0x0)
    at /home/tony/php-build/php-src/Zend/zend_interfaces.c:104
#15 0x0000000000525c4e in zif_spl_autoload_call (execute_data=<optimized out>, return_value=<optimized out>)
    at /home/tony/php-build/php-src/ext/spl/php_spl.c:421
#16 0x00007f382e4ca680 in memtrack_execute_internal (current_execute_data=0x7f3833c15450, return_value=0x7ffcffa5d3b0)
    at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:563
#17 0x000000000065bb57 in zend_call_function (fci=fci@entry=0x7ffcffa5d3f0, fci_cache=fci_cache@entry=0x7ffcffa5d3c0)
    at /home/tony/php-build/php-src/Zend/zend_execute_API.c:880
#18 0x000000000065bee2 in zend_lookup_class_ex (name=name@entry=0x7f37b4487cd0, key=0x7f37b4487b98, use_autoload=use_autoload@entry=1)
    at /home/tony/php-build/php-src/Zend/zend_execute_API.c:1040
#19 0x000000000065c7d8 in zend_fetch_class_by_name (class_name=0x7f37b4487cd0, key=<optimized out>, fetch_type=fetch_type@entry=512)
    at /home/tony/php-build/php-src/Zend/zend_execute_API.c:1386
#20 0x00000000007148d7 in ZEND_INIT_STATIC_METHOD_CALL_SPEC_CONST_CONST_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:5667
#21 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#22 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c152e0) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#23 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800
#24 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#25 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c15180) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#26 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800
#27 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#28 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c14c30) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
(gdb) p *ce
Cannot access memory at address 0x7f37a95d414000
(gdb) p ce
$1 = (zend_class_entry *) 0x7f37a95d414000
 [2017-04-12 16:29 UTC] tony2001@php.net
-Status: Feedback +Status: Open
 [2017-04-13 02:57 UTC] laruence@php.net
from the backtrace, it's really not like a problem that is the "key change" caused

could you try without memtrack ?

anyway, @tony2001, I understand your concern, but we should try to find the root cause, you patch may wroks, but it also may cover some real problems.
 [2017-04-13 11:55 UTC] tony2001@php.net
> from the backtrace, it's really not like a problem that is the "key change" caused
That's why I called the segfaults "sporadic". They happen in random places, apparently because the interned string used for the hash gets broken in the process.
 [2017-04-14 02:52 UTC] laruence@php.net
could you try to understand what the different did your patch makes?

I will try to explain it again:

you patch change :
    if (!ZSTR_IS_INTERNED(key) || !(GC_FLAGS(key) & IS_STR_PERMANENT)) {
        pce = zend_hash_str_update_mem(&PCRE_G(pcre_cache),
                ZSTR_VAL(key), ZSTR_LEN(key), &new_entry, sizeof(pcre_cache_entry));
#if HAVE_SETLOCALE
        if (key != regex) {
            zend_string_release(key);
        }
#endif
    } else {
        pce = zend_hash_update_mem(&PCRE_G(pcre_cache), key, &new_entry, sizeof(pcre_cache_entry));
    }
--------------------------------------------------------------------------------------------------
to:
	if (!ZSTR_IS_INTERNED(regex) || !(GC_FLAGS(regex) & IS_STR_PERMANENT)) {
		zend_string *str = zend_string_init(ZSTR_VAL(regex), ZSTR_LEN(regex), 1);
		GC_REFCOUNT(str) = 0; /* will be incremented by zend_hash_update_mem() */
		ZSTR_H(str) = ZSTR_H(regex);
		regex = str;
 	}
 
	pce = zend_hash_update_mem(&PCRE_G(pcre_cache), regex, &new_entry, sizeof(pcre_cache_entry));

----------------------------------
but to me, they are actually doing the exactlly same thing

pce = zend_hash_str_update_mem(&PCRE_G(pcre_cache),
                ZSTR_VAL(key), ZSTR_LEN(key), &new_entry, sizeof(pcre_cache_entry));

will allocated a persistent key for str_key because &PCRG_G(pcre_cache) is a persistent hashtable

so, as I said, your patch may cover some real problems here.

please try to disable memtrack
 
PHP Copyright © 2001-2018 The PHP Group
All rights reserved.
Last updated: Sun Nov 19 01:31:42 2017 UTC