php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login

go to bug id or search bugs for

Bug #74403 Sporadic FPM segfaults
Submitted: 2017-04-10 11:08 UTC Modified: 2021-09-19 04:22 UTC
From: tony2001@php.net Assigned: cmb (profile)
Status: No Feedback Package: PCRE related
PHP Version: 7.1.4RC1 OS: Linux
Private report: No CVE-ID: None
View Developer Edit
 [2017-04-10 11:08 UTC] tony2001@php.net 
Description:
------------
Patch ebfd93f725eb9b9cfadbbde98047efe76f658da6 introduced an issue that is both hard to debug and reproduce. FPM processes start to crash in completely random places (mostly pointing to the executor code), all FPM children at once.

Unfortunately, I don't have a reproduce case and I don't know of any reliable way to reproduce it, other than to start FPM on production server under a heavy load and leave it for a while.

I've spent a lot of time debugging it and have found that the problem goes away as soon as I revert ebfd93f725eb9b9cfadbbde98047efe76f658da6. After looking at the patch closely, I've noticed that the patch changes one tricky part of the code managing refcount of the string stored in the cache. In the new version that part is gone and reverting this part of the patch fixes the issue, see attached patch.

Test script:
---------------
.

Expected result:
----------------
No crashes.

Actual result:
--------------
Sporadic FPM segfaults.

Patches

pcre-crashes-fix.diff (last revision 2017-04-10 11:09 UTC by tony2001@php.net)

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2017-04-10 11:09 UTC] tony2001@php.net 
The following patch has been added/updated:

Patch Name: pcre-crashes-fix.diff
Revision:   1491822546
URL:        https://bugs.php.net/patch-display.php?bug=74403&patch=pcre-crashes-fix.diff&revision=1491822546
 [2017-04-10 11:17 UTC] requinix@php.net
-Summary: Sporadi +Summary: Sporadic FPM segfaults
 [2017-04-10 11:17 UTC] requinix@php.net 
There have been a few php-fpm crashes reported over the last couple months. Can you tell if any of them are related to this?
 [2017-04-10 11:22 UTC] tony2001@php.net 
https://bugs.php.net/bug.php?id=73740 - this one looks familiar.
 [2017-04-10 11:23 UTC] nikic@php.net
-Assigned To: +Assigned To: laruence
 [2017-04-10 11:45 UTC] laruence@php.net
-Status: Assigned +Status: Feedback
 [2017-04-10 11:45 UTC] laruence@php.net 
Thank you for this bug report. To properly diagnose the problem, we
need a backtrace to see what is happening behind the scenes. To
find out how to generate a backtrace, please read
http://bugs.php.net/bugs-generating-backtrace.php for *NIX and
http://bugs.php.net/bugs-generating-backtrace-win32.php for Win32

Once you have generated a backtrace, please submit it to this bug
report and change the status back to "Open". Thank you for helping
us make PHP better.

as we discussed early in mails, I don't think that diretctly relates to that change,

anyway, could you please paste the backtrace out?

thanks
 [2017-04-11 09:51 UTC] tony2001@php.net 
Reverted the attached patch in my private branch and deployed FPM on one production server.
After about 6 hours I've got several cores with the backtraces below. Number of segfaults on several hundred servers since March 22: 0.

backtrace #1:
#0  zend_hash_find_bucket (ht=ht@entry=0xdd28e1cadaf9342c, ht=ht@entry=0xdd28e1cadaf9342c, key=0x7f37b66bdf00, key@entry=0x7f57b453b718)
    at /home/tony/php-build/php-src/Zend/zend_hash.c:496
#1  zend_hash_find (ht=ht@entry=0x7f37b4539720, key=key@entry=0x7f37b66bdf00) at /home/tony/php-build/php-src/Zend/zend_hash.c:1988
#2  0x00000000006da2e6 in zend_hash_find_ind (key=0x7f37b66bdf00, ht=0x7f37b4539720) at /home/tony/php-build/php-src/Zend/zend_hash.h:278
#3  ZEND_ISSET_ISEMPTY_DIM_OBJ_SPEC_CV_CV_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:37856
#4  0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#5  0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c18720) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#6  0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800
#7  0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#8  0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c18660) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#9  0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800
#10 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#11 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c18480) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#12 0x000000000065b919 in zend_call_function (fci=fci@entry=0x7ffcffa5d8d0, fci_cache=fci_cache@entry=0x7ffcffa5d8a0)
    at /home/tony/php-build/php-src/Zend/zend_execute_API.c:858
#13 0x0000000000689db3 in zend_call_method (object=0x0, obj_ce=<optimized out>, fn_proxy=0x7f3833c55870, function_name=0x7f3833c74248 "main_autoload", 
    function_name_len=<optimized out>, retval_ptr=retval_ptr@entry=0x0, param_count=1, arg1=0x7f3833c18470, arg2=0x0)
    at /home/tony/php-build/php-src/Zend/zend_interfaces.c:104
#14 0x0000000000525c4e in zif_spl_autoload_call (execute_data=<optimized out>, return_value=<optimized out>)
    at /home/tony/php-build/php-src/ext/spl/php_spl.c:421
#15 0x00007f382e4ca680 in memtrack_execute_internal (current_execute_data=0x7f3833c18410, return_value=0x7ffcffa5dbc0)
    at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:563
#16 0x000000000065bb57 in zend_call_function (fci=fci@entry=0x7ffcffa5dc00, fci_cache=fci_cache@entry=0x7ffcffa5dbd0)
    at /home/tony/php-build/php-src/Zend/zend_execute_API.c:880
#17 0x000000000065bee2 in zend_lookup_class_ex (name=name@entry=0x7f37b66bdf00, key=key@entry=0x0, use_autoload=use_autoload@entry=1)
    at /home/tony/php-build/php-src/Zend/zend_execute_API.c:1040
#18 0x000000000065c635 in zend_fetch_class (class_name=0x7f37b66bdf00, fetch_type=512) at /home/tony/php-build/php-src/Zend/zend_execute_API.c:1364
#19 0x00000000006e3d2a in ZEND_FETCH_CLASS_SPEC_CV_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:2330
#20 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#21 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c18070) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#22 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800
#23 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#24 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c14e60) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#25 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800
#26 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#27 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c134c0) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#28 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800
(gdb) p p
$1 = (Bucket *) 0x7f57b453b718
(gdb) p *p
Cannot access memory at address 0x7f57b453b718


backtrace #2:
#0  zend_verify_abstract_class (ce=0x7f37a95d414000) at /home/tony/php-build/php-src/Zend/zend_execute_API.c:1441
#1  0x00000000006b6f24 in ZEND_VERIFY_ABSTRACT_CLASS_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:1539
#2  0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#3  0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c15af0) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#4  0x000000000071673c in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:29532
#5  0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#6  0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c15720) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#7  0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800
#8  0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#9  0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c156a0) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#10 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800
#11 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#12 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c154c0) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#13 0x000000000065b919 in zend_call_function (fci=fci@entry=0x7ffcffa5d0c0, fci_cache=fci_cache@entry=0x7ffcffa5d090)
    at /home/tony/php-build/php-src/Zend/zend_execute_API.c:858
#14 0x0000000000689db3 in zend_call_method (object=0x0, obj_ce=<optimized out>, fn_proxy=0x7f3833c55810, function_name=0x7f3833c762e8 "main_autoload", 
    function_name_len=<optimized out>, retval_ptr=retval_ptr@entry=0x0, param_count=1, arg1=0x7f3833c154b0, arg2=0x0)
    at /home/tony/php-build/php-src/Zend/zend_interfaces.c:104
#15 0x0000000000525c4e in zif_spl_autoload_call (execute_data=<optimized out>, return_value=<optimized out>)
    at /home/tony/php-build/php-src/ext/spl/php_spl.c:421
#16 0x00007f382e4ca680 in memtrack_execute_internal (current_execute_data=0x7f3833c15450, return_value=0x7ffcffa5d3b0)
    at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:563
#17 0x000000000065bb57 in zend_call_function (fci=fci@entry=0x7ffcffa5d3f0, fci_cache=fci_cache@entry=0x7ffcffa5d3c0)
    at /home/tony/php-build/php-src/Zend/zend_execute_API.c:880
#18 0x000000000065bee2 in zend_lookup_class_ex (name=name@entry=0x7f37b4487cd0, key=0x7f37b4487b98, use_autoload=use_autoload@entry=1)
    at /home/tony/php-build/php-src/Zend/zend_execute_API.c:1040
#19 0x000000000065c7d8 in zend_fetch_class_by_name (class_name=0x7f37b4487cd0, key=<optimized out>, fetch_type=fetch_type@entry=512)
    at /home/tony/php-build/php-src/Zend/zend_execute_API.c:1386
#20 0x00000000007148d7 in ZEND_INIT_STATIC_METHOD_CALL_SPEC_CONST_CONST_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:5667
#21 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#22 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c152e0) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#23 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800
#24 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#25 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c15180) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
#26 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800
#27 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414
#28 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c14c30) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481
(gdb) p *ce
Cannot access memory at address 0x7f37a95d414000
(gdb) p ce
$1 = (zend_class_entry *) 0x7f37a95d414000
 [2017-04-12 16:29 UTC] tony2001@php.net
-Status: Feedback +Status: Open
 [2017-04-13 02:57 UTC] laruence@php.net 
from the backtrace, it's really not like a problem that is the "key change" caused

could you try without memtrack ?

anyway, @tony2001, I understand your concern, but we should try to find the root cause, you patch may wroks, but it also may cover some real problems.
 [2017-04-13 11:55 UTC] tony2001@php.net 
> from the backtrace, it's really not like a problem that is the "key change" caused
That's why I called the segfaults "sporadic". They happen in random places, apparently because the interned string used for the hash gets broken in the process.
 [2017-04-14 02:52 UTC] laruence@php.net 
could you try to understand what the different did your patch makes?

I will try to explain it again:

you patch change :
    if (!ZSTR_IS_INTERNED(key) || !(GC_FLAGS(key) & IS_STR_PERMANENT)) {
        pce = zend_hash_str_update_mem(&PCRE_G(pcre_cache),
                ZSTR_VAL(key), ZSTR_LEN(key), &new_entry, sizeof(pcre_cache_entry));
#if HAVE_SETLOCALE
        if (key != regex) {
            zend_string_release(key);
        }
#endif
    } else {
        pce = zend_hash_update_mem(&PCRE_G(pcre_cache), key, &new_entry, sizeof(pcre_cache_entry));
    }
--------------------------------------------------------------------------------------------------
to:
	if (!ZSTR_IS_INTERNED(regex) || !(GC_FLAGS(regex) & IS_STR_PERMANENT)) {
		zend_string *str = zend_string_init(ZSTR_VAL(regex), ZSTR_LEN(regex), 1);
		GC_REFCOUNT(str) = 0; /* will be incremented by zend_hash_update_mem() */
		ZSTR_H(str) = ZSTR_H(regex);
		regex = str;
 	}
 
	pce = zend_hash_update_mem(&PCRE_G(pcre_cache), regex, &new_entry, sizeof(pcre_cache_entry));

----------------------------------
but to me, they are actually doing the exactlly same thing

pce = zend_hash_str_update_mem(&PCRE_G(pcre_cache),
                ZSTR_VAL(key), ZSTR_LEN(key), &new_entry, sizeof(pcre_cache_entry));

will allocated a persistent key for str_key because &PCRG_G(pcre_cache) is a persistent hashtable

so, as I said, your patch may cover some real problems here.

please try to disable memtrack
 [2021-09-08 16:56 UTC] cmb@php.net
-Status: Assigned +Status: Feedback -Assigned To: laruence +Assigned To: cmb
 [2021-09-08 16:56 UTC] cmb@php.net 
Is this still an issue with any of the actively supported PHP
versions[1]?

[1] <https://www.php.net/supported-versions.php>
 [2021-09-19 04:22 UTC] php-bugs at lists dot php dot net 
No feedback was provided. The bug is being suspended because
we assume that you are no longer experiencing the problem.
If this is not the case and you are able to provide the
information that was requested earlier, please do so and
change the status of the bug back to "Re-Opened". Thank you.
 
PHP Copyright © 2001-2025 The PHP Group
All rights reserved. 		Last updated: Fri Oct 31 21:00:02 2025 UTC