php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #80243 OPCache JIT segfaults at startup
Submitted: 2020-10-15 19:29 UTC Modified: 2020-10-21 10:41 UTC
Votes:4
Avg. Score:4.2 ± 0.8
Reproduced:4 of 4 (100.0%)
Same Version:3 (75.0%)
Same OS:0 (0.0%)
From: jens-erik dot riedel at kippdata dot de Assigned: nikic (profile)
Status: Closed Package: JIT
PHP Version: 8.0.0rc1 OS: RHEL 8.0
Private report: No CVE-ID: None
 [2020-10-15 19:29 UTC] jens-erik dot riedel at kippdata dot de
Description:
------------
When starting PHP 8.0.0rc1 with OPcache JIT enabled (i.e. opcache.jit_buffer_size is greater than zero), it segfaults just when loading and initializing the opcache extension. The segfault can be reproduced simply by calling "php -v" or "php -i" without executing any PHP script.
This segfault does not occur in 8.0.0beta3.
This segfault does not occur if the opcache extension is loaded and JIT is disabled by setting opcache.jit_buffer_size=0M.

Short analysis of core dump by "coredumpctl info":
           PID: 14163 (php)
           UID: 1200 (esuppbld)
           GID: 1200 (esupport)
        Signal: 11 (SEGV)
     Timestamp: Thu 2020-10-15 16:15:01 CEST (1min 43s ago)
  Command Line: /opt/products/php80/8.0.0rc1-1/bin/php -c /opt/instances/php80-fpm/lib/php.ini -d opcache.enable_cli=1 -i
    Executable: /opt/products/php80/8.0.0rc1-1/bin/php
 Control Group: /user.slice/user-57.slice/session-4.scope
          Unit: session-4.scope
         Slice: user-57.slice
       Session: 4
     Owner UID: 57
       Boot ID: 945e91c3455b429db5721734b2209ea8
    Machine ID: 0f3498c0144645538a1771de48e97826
      Hostname: est-rhel8-64
       Storage: /var/lib/systemd/coredump/core.php.1200.945e91c3455b429db5721734b2209ea8.14163.1602771301000000.lz4
       Message: Process 14163 (php) of user 1200 dumped core.
                
                Stack trace of thread 14163:
                #0  0x00007f604414d165 dasm_put (opcache.so)
                #1  0x00007f604416cd40 zend_jit_interrupt_handler_stub (opcache.so)
                #2  0x00007f60441ba607 zend_jit_make_stubs (opcache.so)
                #3  0x00007f60440dbf28 accel_post_startup (opcache.so)
                #4  0x00007f6047b162ed zend_post_startup (libphp80.so.0)
                #5  0x00007f6047ab6b39 php_module_startup (libphp80.so.0)
                #6  0x0000000000404b6d php_cli_startup (php)
                #7  0x000000000040453c main (php)
                #8  0x00007f60462e7813 __libc_start_main (libc.so.6)
                #9  0x00000000004047de _start (php)

php.ini settings:
memory_limit = 16M
upload_max_filesize = 20M
zend_extension=opcache
assert.active = Off
opcache.enable=1
opcache.log_verbosity_level=3
opcache.jit=1235
opcache.jit_buffer_size=1M
opcache.jit_debug=1

Expected result:
----------------
The execution of "php -v" should output:
PHP 8.0.0rc1 (cli) (built: Oct  9 2020 00:23:22)
Copyright (c) The PHP Group
Zend Engine v4.0.0-dev, Copyright (c) Zend Technologies
    with Zend OPcache v8.0.0rc1, Copyright (c), by Zend Technologies

Actual result:
--------------
The execution of "php -v" crashes with SegV.

Backtrace via gdb:
#0  dasm_put (Dst=Dst@entry=0x7ffdd80b9cc0, start=start@entry=0)
    at /bld/php80/ext/opcache/jit/dynasm/dasm_x86.h:176
#1  0x00007f604416cd40 in zend_jit_interrupt_handler_stub (Dst=0x7ffdd80b9cc0)
    at /bld/php80/ext/opcache/jit/zend_jit_x86.dasc:1712
#2  0x00007f60441ba607 in zend_jit_make_stubs ()
    at /bld/php80/ext/opcache/jit/zend_jit.c:3950
#3  zend_jit_startup (buf=<optimized out>, size=size@entry=1048576, reattached=reattached@entry=false)
    at /bld/php80/ext/opcache/jit/zend_jit.c:4200
#4  0x00007f60440dbf28 in accel_post_startup ()
    at /bld/php80/ext/opcache/ZendAccelerator.c:3039
#5  0x00007f6047b162ed in zend_post_startup () at /bld/php80/Zend/zend.c:1030
#6  0x00007f6047ab6b39 in php_module_startup (sf=<optimized out>, additional_modules=additional_modules@entry=0x0, 
    num_additional_modules=num_additional_modules@entry=0)
    at /bld/php80/main/main.c:2267
#7  0x0000000000404b6d in php_cli_startup (sapi_module=<optimized out>)
    at /bld/php80/sapi/cli/php_cli.c:407
#8  0x000000000040453c in main (argc=6, argv=0x15df6c0)
    at /bld/php80/sapi/cli/php_cli.c:1304

Patches

php80_fix_double_free.patch (last revision 2020-10-20 19:25 UTC by brainpower at mailbox dot org)

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2020-10-16 08:38 UTC] cmb@php.net
-Package: Reproducible crash +Package: JIT
 [2020-10-19 15:00 UTC] brainpower at mailbox dot org
I've got the same crash on Debian 10 buster, with 8.0.0rc1 and 8.0.0rc2,
so it still crashes with rc2.
According to gdb it seems that *sec/D->section is NULL:

(gdb) r
Starting program: /opt/php80/bin/php -dopcache.enable_cli=1 -dopcache.jit=1205 -dopcache.jit_buffer_size=32M Zend/bench.php
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
dasm_put (Dst=Dst@entry=0x7fffffffcbf0, start=start@entry=5) at /root/shmbuild/src/php-8.0.0RC2/ext/opcache/jit/dynasm/dasm_x86.h:176
176       int pos = sec->pos, ofs = sec->ofs, mrm = -1;
(gdb) bt
#0  dasm_put (Dst=Dst@entry=0x7fffffffcbf0, start=start@entry=5) at /root/shmbuild/src/php-8.0.0RC2/ext/opcache/jit/dynasm/dasm_x86.h:176
#1  0x00007ffff356a703 in zend_jit_interrupt_handler_stub (Dst=0x7fffffffcbf0) at /root/shmbuild/src/php-8.0.0RC2/ext/opcache/jit/zend_jit_x86.dasc:1726
#2  0x00007ffff35b8727 in zend_jit_make_stubs () at /root/shmbuild/src/php-8.0.0RC2/ext/opcache/jit/zend_jit.c:3999
#3  zend_jit_startup (buf=<optimized out>, size=size@entry=33554432, reattached=reattached@entry=false) at /root/shmbuild/src/php-8.0.0RC2/ext/opcache/jit/zend_jit.c:4249
#4  0x00007ffff34da66f in accel_post_startup () at /root/shmbuild/src/php-8.0.0RC2/ext/opcache/ZendAccelerator.c:3039
#5  0x0000555555b48f82 in zend_post_startup () at /root/shmbuild/src/php-8.0.0RC2/Zend/zend.c:1030
#6  0x0000555555ae8853 in php_module_startup (sf=<optimized out>, additional_modules=<optimized out>, num_additional_modules=<optimized out>)
    at /root/shmbuild/src/php-8.0.0RC2/main/main.c:2240
#7  0x0000555555bd01dd in php_cli_startup (sapi_module=<optimized out>) at /root/shmbuild/src/php-8.0.0RC2/sapi/cli/php_cli.c:406
#8  0x00005555557a061d in main (argc=5, argv=0x555556bb95e0) at /root/shmbuild/src/php-8.0.0RC2/sapi/cli/php_cli.c:1303
(gdb) list
171     {
172       va_list ap;
173       dasm_State *D = Dst_REF;
174       dasm_ActList p = D->actionlist + start;
175       dasm_Section *sec = D->section;
176       int pos = sec->pos, ofs = sec->ofs, mrm = -1;
177       int *b;
178
179       if (pos >= sec->epos) {
180         DASM_M_GROW(Dst, int, sec->buf, sec->bsize,
(gdb) print sec
$1 = (dasm_Section *) 0x0
(gdb) print D
$2 = (dasm_State *) 0x7ffff3609100
 [2020-10-19 15:48 UTC] ricardo at banak dot com
It also happens in Centos 8 and Centos 7.
 [2020-10-19 15:51 UTC] ricardo at banak dot com
And same in PHP 8.0.0RC2
 [2020-10-20 10:20 UTC] nikic@php.net
D->section is initialized by dasm_setup(), and that does get called directly before the stub is compiled. Not clear to me how it ends up being NULL.
 [2020-10-20 13:25 UTC] brainpower at mailbox dot org
Lots of other members of D are zeroed, not sure if that's ok or not:

Program received signal SIGSEGV, Segmentation fault.
dasm_put (Dst=Dst@entry=0x7fffffffcc00, start=start@entry=5) at /root/shmbuild/src/php-8.0.0RC2/ext/opcache/jit/dynasm/dasm_x86.h:176
176       int pos = sec->pos, ofs = sec->ofs, mrm = -1;
(gdb) print D->section
$1 = (dasm_Section *) 0x0
(gdb) print D->sections[0]
$2 = {rbuf = 0x0, buf = 0x0, bsize = 0, pos = 0, epos = 0, ofs = 0}
(gdb) print &D->sections[0]
$3 = (dasm_Section *) 0x7ffff3608150
(gdb) print D->maxsection
$4 = 0
(gdb) print D->status
$5 = 0
(gdb) print D->actionlist
$6 = (dasm_ActList) 0x0
(gdb) print D->lglabels
$7 = (int *) 0x0
(gdb) print D->lgsize
$8 = 0
(gdb) print D->pclabels
$9 = (int *) 0x0
(gdb) print D->pcsize
$10 = 0
(gdb) print D->globals
$11 = (void **) 0x0
(gdb) print D->psize
$12 = 0
(gdb) print D->codesize
$13 = 0
(gdb)

Maybe dasm_setup didn't run? Or some errand memset() overwrote the wrong thing(s)?
 [2020-10-20 14:10 UTC] nikic@php.net
Would it be possible for you to set a break point on dasm_setup, step through it, and check that D->section is initialized at the end?

In the meantime, I've done a php-8.0.0RC2 build on CentOS 7 (rather than Ubuntu 20.04) and wasn't able to reproduce the issue there either.
 [2020-10-20 14:24 UTC] brainpower at mailbox dot org
https://gist.github.com/brainpower/d85cc63d6820dd2fb65d12ca52f5f957

Here's stepping through dasm_setup() up to the point where D->section is broken.
As I thought, it seems something is wrong with the memsets in there...

I've added the configure command to the gist, too.
Maybe some feature I've enabled messes with things.
 [2020-10-20 14:51 UTC] brainpower at mailbox dot org
Found it: https://gist.github.com/brainpower/d85cc63d6820dd2fb65d12ca52f5f957#file-gistfile2-txt-L19

I've stepped through another time, since I thought the parameters to the memset may be important.
Well   D->lglabels  points to the same address as D, so the memset overwrites D, obviously.
(Lines 19 and 23 of gistfile2.txt)

D->lglabels is touched only in dasm_setupglobal() before dasm_setup() is called, as far as I can see.
 [2020-10-20 15:00 UTC] brainpower at mailbox dot org
I've stepped through dasm_setupglobal, too, and added that to the gist.
At the end of that function, after the "realloc", D and D->lglabels are the same.

So maybe not a Bug in JIT but in the allocation function?

Please let me know if you need more information.
 [2020-10-20 15:32 UTC] nikic@php.net
Thank you for the investigation, this is rather intriguing. I find is hard to believe that this is an allocator bug (I don't think I've seen one of those in years), it's more likely that something is corrupting the allocator state.

There's two simple things that could be tried:
1. Run "USE_ZEND_ALLOC=0 php". That will switch to using the system allocator -- which will possibly make it work, or crash harder...
2. Run "USE_ZEND_ALLOC=0 valgrind php", which might point out memory corruption. (In this case, "USE_ZEND_ALLOC=1 valgrind php" might also be interesting.)
 [2020-10-20 16:49 UTC] jens-erik dot riedel at kippdata dot de
I have tried with USE_ZEND_ALLOC=0 (using 8.0.0rc1 on RHEL 8). It crashes again; I don't know if this qualifies as "crashes harder" but at least it crashes differently.

$ USE_ZEND_ALLOC=0 /opt/products/php80/8.0.0rc1-1/bin/php -c /opt/instances/php80-fpm/lib/php.ini -d opcache.enable_cli=1 -i
free(): double free detected in tcache 2
Aborted (core dumped)

Backtrace is as follows:
Core was generated by `/opt/products/php80/8.0.0rc1-1/bin/php -c /opt/instances/php80-fpm/lib/php.ini'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f88746e893f in raise () from /lib64/libc.so.6

(gdb) bt
#0  0x00007f88746e893f in raise () from /lib64/libc.so.6
#1  0x00007f88746d2c95 in abort () from /lib64/libc.so.6
#2  0x00007f887472bd57 in __libc_message () from /lib64/libc.so.6
#3  0x00007f887473268c in malloc_printerr () from /lib64/libc.so.6
#4  0x00007f8874734155 in _int_free () from /lib64/libc.so.6
#5  0x00007f8875ea9ec1 in php_load_zend_extension_cb (arg=<optimized out>)
    at /bld/php80/main/php_ini.c:391
#6  0x00007f8875ef80ee in zend_llist_apply (l=l@entry=0x7f8876443f60 <extension_lists>, 
    func=func@entry=0x7f8875ea9e30 <php_load_zend_extension_cb>)
    at /bld/php80/Zend/zend_llist.c:182
#7  0x00007f8875eaaa47 in php_ini_register_extensions ()
    at /bld/php80/main/php_ini.c:756
#8  0x00007f8875ea3a46 in php_module_startup (sf=<optimized out>, additional_modules=additional_modules@entry=0x0, 
    num_additional_modules=num_additional_modules@entry=0)
    at /bld/php80/main/main.c:2235
#9  0x0000000000404b6d in php_cli_startup (sapi_module=<optimized out>)
    at /bld/php80/sapi/cli/php_cli.c:407
#10 0x000000000040453c in main (argc=6, argv=0x1ef3700)
    at /bld/php80/sapi/cli/php_cli.c:1304
 [2020-10-20 18:12 UTC] brainpower at mailbox dot org
Yeah, same crash in php_ini.c for me...

After a short look into that file, I think there is a return missing after line 377:

 368                 efree(orig_libpath);
369                 efree(err1);
370                 efree(libpath);
371                 efree(err2);
372                 return;
373             }
374
375             efree(orig_libpath);
376             efree(err1);
377             efree(libpath);
                >> RETURN MISSING HERE?? <<
378         }
379
380 #ifdef PHP_WIN32
381         if (!php_win32_image_compatible(handle, &err1)) {
....
387 #endif
388
389         zend_load_extension_handle(handle, libpath);
390         efree(libpath); // theese lines should probably not be reached if(!handle) above is entered...
391     }
 [2020-10-20 19:22 UTC] brainpower at mailbox dot org
Urg, brain fart.

There's not a return missing, but one `efree(libpath)` too much!
It's a double free error after all.

Curiously I wasn't able to reproduce the original issue with the crash in dasm_put() anymore after fixing the double free.
I'll attach a patch of what I did...
 [2020-10-20 19:25 UTC] brainpower at mailbox dot org
The following patch has been added/updated:

Patch Name: php80_fix_double_free.patch
Revision:   1603221928
URL:        https://bugs.php.net/patch-display.php?bug=80243&patch=php80_fix_double_free.patch&revision=1603221928
 [2020-10-20 19:41 UTC] jens-erik dot riedel at kippdata dot de
The fact that there is an extra `efree(libpath)` in main/php_ini.c in line 390 may correspond to the fact that this extra 'efree(libpath)` is not present in 8.0.0beta3 and OPCache JIT startup does not crash in 8.0.0beta3.
Perhaps the code around line 384 should be revised. IMHO there is a double free for `err1` if PHP_WIN32 is defined.
 [2020-10-20 19:43 UTC] nikic@php.net
-Assigned To: +Assigned To: nikic
 [2020-10-20 19:48 UTC] brainpower at mailbox dot org
The efree in line 390 is in the correct position, libpath is still used in line 389 after all.

The free of err1 in line 384 is not a double free,
because I believe `php_win32_image_compatible` allocates it again.
It's just a re-use of a variable name, you could name it err3 inside that PHP_WIN32 block and nothing would change...
 [2020-10-21 10:41 UTC] nikic@php.net
-Status: Assigned +Status: Closed
 [2020-10-21 10:41 UTC] nikic@php.net
Thanks everyone for the investigation! So this turned out to be completely unrelated to the JIT after all, it's just where the double-free ended up manifesting.

The double free has been fixed by https://github.com/php/php-src/commit/5998b2a3a65042dc7fc4f70945dd72e4e258500f and I've added some test coverage for zend_extension loading in https://github.com/php/php-src/commit/3966c0f8a47225486865d7cdef2552f746dd274c. All of our existing CI jobs were loading opcache either as an absolute path, or using an extension, so this was not noticed.
 
PHP Copyright © 2001-2021 The PHP Group
All rights reserved.
Last updated: Sat May 08 11:01:23 2021 UTC