php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #80154 php-fpm segfaults for an unknown reason
Submitted: 2020-09-27 08:28 UTC Modified: 2020-12-01 11:07 UTC
Votes:7
Avg. Score:4.1 ± 1.0
Reproduced:7 of 7 (100.0%)
Same Version:2 (28.6%)
Same OS:4 (57.1%)
From: sriccio at swisccenter dot com Assigned:
Status: Open Package: opcache
PHP Version: 7.4.10 OS: Centos 8
Private report: No CVE-ID: None
View Developer Edit
Welcome! If you don't have a Git account, you can't do anything here.
If you reported this bug, you can edit this bug over here.
(description)
Block user comment
Status: Assign to:
Package:
Bug Type:
Summary:
From: sriccio at swisccenter dot com
New email:
PHP Version: OS:

 

 [2020-09-27 08:28 UTC] sriccio at swisccenter dot com
Description:
------------
On our shared hosting servers, sometime (often) there php-fpm segfaults for an unknown reason.

However we suspect it might be related to opcache, as if we disable it for certain sites, it seems these aren't segfaulting anymore.

Packages we are using are Remi's php74-php-fpm-7.4.10-1.el8.x86_64$

opcache config in pools

php_admin_value[opcache.file_cache_only] = 1
php_admin_value[opcache.file_cache] = "/home/xyz/.data/opcache/"
php_admin_value[opcache.max_accelerated_files] = 10000
php_admin_value[opcache.memory_consumption] = 128


Expected result:
----------------
No segfault happening

Actual result:
--------------
[root@web21 ~]# coredumpctl info 1817197
           PID: 1817197 (php-fpm)
           UID: 1098 (xyz)
           GID: 1100 (xyz)
        Signal: 11 (SEGV)
     Timestamp: Sun 2020-09-27 09:56:25 CEST (16min ago)
  Command Line: php74-fpm: pool xyz
    Executable: /opt/remi/php74/root/usr/sbin/php-fpm
 Control Group: /system.slice/php74-php-fpm.service
          Unit: php74-php-fpm.service
         Slice: system.slice
       Boot ID: 504629e6e9164777bb74535d8219c1ee
    Machine ID: 06e6f1ebaaae4c8aa0611d132201376a
      Hostname: web21
       Storage: /var/lib/systemd/coredump/core.php-fpm.1098.504629e6e9164777bb74535d8219c1ee.1817197.1601193385000000.lz4
       Message: Process 1817197 (php-fpm) of user 1098 dumped core.
                
                Stack trace of thread 1817197:
                #0  0x000055f36a078540 ZEND_BIND_STATIC_SPEC_CV_UNUSED_HANDLER (php-fpm)
                #1  0x000055f36a0afe32 execute_ex (php-fpm)
                #2  0x000055f36a0b9661 zend_execute (php-fpm)
                #3  0x000055f36a0308fb zend_execute_scripts (php-fpm)
                #4  0x000055f369fcdc38 php_execute_script (php-fpm)
                #5  0x000055f369e9b850 main (php-fpm)
                #6  0x00007fb9677e26a3 __libc_start_main (libc.so.6)
                #7  0x000055f369e9c6be _start (php-fpm)

Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2020-11-27 06:27 UTC] sriccio at swisscenter dot com
Hi,

Just a little followup on this. We still have this issue with PHP 7.4.12 and also latest 7.3.x and 7.2.x.

After some testing it looks like if we disable file_cache_only it stops segfaulting.

So it seems to be related to file_cache ?
 [2020-11-27 09:33 UTC] nikic@php.net
File cache is plausible here. A fix that looks related based on the backtrace is https://github.com/php/php-src/commit/4609ded082fb5381bb39b408d471e88df9b81674, but that was already in 7.4.10.
 [2020-11-27 17:14 UTC] peter dot sopko at backbone dot sk
Hi, it looks like we have (on our shared hosting servers as well) hit the same bug.
We are using FreeBSD (so not CentOS), php is compiled from ports.

settings php_admin_value[opcache.file_cache] = "" seems to mitigate the problem, as well as offloading the traffic to another php server until the opcache expires (looks like the problem is in somehow corrupted cached file being created, that is afterwards used when serving requests). 


Original opcache config in pool:

php_admin_value[opcache.enable] = 1
php_admin_value[opcache.interned_strings_buffer] = 32
php_admin_value[opcache.max_accelerated_files] = 2000
php_admin_value[opcache.memory_consumption] = 512
php_admin_value[opcache.revalidate_freq] = 900
php_admin_value[opcache.fast_shutdown] = 0
php_admin_value[opcache.enable_cli] = 0
php_admin_value[opcache.revalidate_path] = 0
php_admin_value[opcache.validate_root] = 0
php_admin_value[opcache.validate_permission] = 0
php_admin_value[opcache.validate_timestamps] = 1
php_admin_value[opcache.save_comments] = 1
php_admin_value[opcache.max_file_size] = 0
php_admin_value[opcache.file_cache] = /tmp/
php_admin_value[opcache.file_cache_consistency_checks] = 1
php_admin_value[opcache.file_update_protection] = 2
php_admin_value[opcache.file_cache_only] = 0
php_admin_value[opcache.optimization_level] = 0x7FFFBFFF


Coredump backtrace:

# lldb -c php-fpm.core -- /usr/local/sbin/php-fpm
(lldb) target create "/usr/local/sbin/php-fpm" --core "php-fpm.core"
Core file '/var/tmp/php-fpm.core' (x86_64) was loaded.
(lldb) bt
* thread #1, name = 'php-fpm', stop reason = signal SIGABRT
  * frame #0: 0x0000000800d6a1ba libc.so.7`__sys_thr_kill + 10
    frame #1: 0x0000000800d685e4 libc.so.7`__raise + 52
    frame #2: 0x0000000800cdc7e9 libc.so.7`abort + 73
    frame #3: 0x0000000800d58191 libc.so.7`__assert + 81
    frame #4: 0x00000000008e1aa5 php-fpm`ZEND_BIND_STATIC_SPEC_CV_UNUSED_HANDLER(execute_data=0x0000000801814680) at zend_vm_execute.h:46891:3
    frame #5: 0x000000000085b014 php-fpm`execute_ex(ex=0x0000000801814020) at zend_vm_execute.h:53655:7
    frame #6: 0x000000000085b24b php-fpm`zend_execute(op_array=0x0000000801872100, return_value=0x0000000000000000) at zend_vm_execute.h:57957:2
    frame #7: 0x00000000007e55a2 php-fpm`zend_execute_scripts(type=8, retval=0x0000000000000000, file_count=3) at zend.c:1677:4
    frame #8: 0x000000000072db47 php-fpm`php_execute_script(primary_file=0x00007fffffffe930) at main.c:2621:14
    frame #9: 0x00000000009121d4 php-fpm`main(argc=1, argv=0x00007fffffffeb80) at fpm_main.c:1942:4
    frame #10: 0x00000000003be10f php-fpm`_start(ap=<unavailable>, cleanup=<unavailable>) at crt1.c:76:7


PHP version used (but the problem happens on previous [7.4.10, 7.4.11] as well)

# php -v
PHP 7.4.12 (cli) (built: Nov 27 2020 15:55:46) ( NTS DEBUG )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.12, Copyright (c), by Zend Technologies

FreeBSD version (but the problem happens on FreeBSD 12.2-RELEASE with latest patches as well);

# uname -a
FreeBSD nas06.backbone.intra 12.1-RELEASE-p10 FreeBSD 12.1-RELEASE-p10 GENERIC  amd64

Expected result is of course not to see those coredumps.
 [2020-12-01 11:07 UTC] nikic@php.net
I've been running PHP with file cache against a large number of projects, but didn't manage to hit this condition. (The one issue I found is PHP 8 specific and fixed in https://github.com/php/php-src/commit/5dfec886d67f01c4e7ea96ef8c26792cb1210047).

@peter dot sopko at backbone dot sk: Would it be possible to run

> p (char*)executor_globals->current_execute_data->func->op_array->filename->val
> p executor_globals->current_execute_data->opline->lineno
> p (char*)executor_globals->current_execute_data->func->op_array->scope->name->val

on the core dump, to determine in which PHP file, file and class this occurs? Mainly wondering if there's anything interesting going on (say, static variables being used inside a trait method).
 [2020-12-03 10:27 UTC] peter dot sopko at backbone dot sk
Hi Nikita,

sure thing, here is the output (folder structure obfuscated with XXXXXX to protect our customers privacy):

(lldb) p (char*)executor_globals->current_execute_data->func->op_array->filename->val
(char *) $0 = 0x0000000808224b58 "XXXXXX/wp-admin/includes/screen.php"
  Fix-it applied, fixed expression was: 
    (char*)executor_globals.current_execute_data->func->op_array.filename->val
(lldb) p executor_globals->current_execute_data->opline->lineno
(uint32_t) $1 = 22
  Fix-it applied, fixed expression was: 
    executor_globals.current_execute_data->opline->lineno
(lldb) p (char*)executor_globals->current_execute_data->func->op_array->scope->name->val
(char *) $2 = 0x00000000009a81e0 "\x0f"
  Fix-it applied, fixed expression was: 
    (char*)executor_globals.current_execute_data->func->op_array.scope->name->val


Front the output it is screen.php file that is part of one of the latest wordpress versions. The file begins with the following lines of code (except for comments), line 22 being the one that defines static variable $column_headers to an empty array.

function get_column_headers( $screen ) {
        if ( is_string( $screen ) ) {
                $screen = convert_to_screen( $screen );
        }

        static $column_headers = array();

        if ( ! isset( $column_headers[ $screen->id ] ) ) {
                $column_headers[ $screen->id ] = apply_filters( "manage_{$screen->id}_columns", array() );
        }

        return $column_headers[ $screen->id ];
}

Hope it helps in any way and thank you very much for your time.
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Sun Nov 24 05:01:32 2024 UTC