PHP :: Bug #71135 :: Random memory corruption with strings

Bug #71135

Random memory corruption with strings

Submitted:

2015-12-16 12:18 UTC

Modified:

2017-09-13 04:08 UTC

Votes:	5
Avg. Score:	5.0 ± 0.0
Reproduced:	5 of 5 (100.0%)
Same Version:	1 (20.0%)
Same OS:	1 (20.0%)

From:

iquito at gmx dot net

Assigned:

laruence (profile)

Status:

Closed

Package:

opcache

PHP Version:

7.0.0

OS:

Debian Jessie

Private report:

CVE-ID:

None

View Developer Edit

Welcome! If you don't have a Git account, you can't do anything here.
If you reported this bug, you can edit this bug over here.

php.net Username: php.net Password:

Quick Fix:	(description)
	Block user comment
Status:		Assign to:
Package:
Bug Type:
Summary:
From:	iquito at gmx dot net
New email:
PHP Version:		OS:

New/Additional Comment:

[2015-12-16 12:18 UTC] iquito at gmx dot net

Description:
------------
I have been using PHP7 in production since Saturday, compiling it myself according to the instructions on https://www.howtoforge.com/tutorial/how-to-install-php-7-on-debian/ . I have about 0.5 million pageviews served per day, and two primary web servers running PHP-FPM.

First, everything worked fine - but after 12-36 hours, when everything is cached and running perfectly, at least one string in the cached PHP files seems to "flip" one character in it. One time, it changed a character in an SQL query, which generated these errors:

"Unknown column 'blocked_by_usdr' in 'field list':
SELECT 1 AS `exists`, noted, friend, friend_confirmed, friend_position, known, known_confirmed, blocked AS blocking, blocked_by_usdr AS blocked FROM userlist_new WHERE profile_id = 137137 AND friend_profile_id = 65297"

Instead of blocked_by_user, it changed to blocked_by_usdr, which suddenly generated 500 SQL-errors per minute. As soon as I restarted PHP7-FPM on the affected server, everything was fine again, and I actually checked the PHP file - the SQL query was correct, and never contained that error.

24 hours later, on the other server, this error completely broke all websites:

Warning: require(/srv_ssd/domains/website/vendor/smarty/smarty/libs/sysplugins/smarty_template_compiled-php): failed to open stream: No such file or directory in /srv_ssd/domains/website/vendor/smarty/smarty/libs/sysplugins/smarty_internal_template.php

As you can see, instead of including /srv_ssd/domains/website/vendor/smarty/smarty/libs/sysplugins/smarty_template_compiled.php the string was changed from a dot to a dash. Interestingly enough, this error appeared in all three projects running on the server, although each had its own (identical) version of smarty_internal_template.php - but maybe the PHP opcache caches them only once if they are all the same, even if they are in different directories.

This happened to me three times already, on two different production servers, and always at night when my webservers are less busy. As soon as PHP7 is restarted, everything is fine again. When the last error occured, an SQL query in another file was also broken, so in two files one character changed at the same time. Because these error occur always at night and sometimes "grouped", maybe the opcache does some garbage collection at that time, or some other internal changes?

Unfortunately, I do not know how to reproduce this bug - it always occurs within 12-36 hours on my servers, but it never occurs when starting PHP-FPM, and I do not see a clear pattern on when or why it happens. It is also possible some string changes happened but were not noticed by me, because they did not result in an immediate PHP or SQL error.

My configure line:

configure --prefix=/opt/php7 --with-pdo-pgsql --with-zlib-dir --with-freetype-dir --enable-mbstring --with-libxml-dir=/usr --enable-soap --enable-calendar --with-curl --with-mcrypt --with-zlib --with-gd --with-pgsql --disable-rpath --enable-inline-optimization --with-bz2 --with-zlib --enable-sockets --enable-sysvsem --enable-sysvshm --enable-pcntl --enable-mbregex --enable-exif --enable-bcmath --with-mhash --enable-zip --with-pcre-regex --with-pdo-mysql --with-mysqli --with-mysql-sock=/var/run/mysqld/mysqld.sock --with-jpeg-dir=/usr --with-png-dir=/usr --enable-gd-native-ttf --with-openssl --with-fpm-user=als --with-fpm-group=als --with-libdir=/lib/x86_64-linux-gnu --enable-ftp --with-imap --with-imap-ssl --with-kerberos --with-gettext --with-xmlrpc --with-xsl --with-iconv-dir --enable-intl --enable-opcache --enable-fpm

Patches

interned_strings_top (last revision 2017-09-12 12:19 UTC by mike@php.net)

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports

[2015-12-16 15:20 UTC] laruence@php.net

-Assigned To: +Assigned To: laruence

[2015-12-16 15:20 UTC] laruence@php.net

hey, you may try opcache.protect_memory=on, which will segfault if string in shared memory is changed, which will be helpful for use to find the root cause

anyway, please note, enable this will slowdown the performance visibily


thanks

[2015-12-16 15:20 UTC] laruence@php.net

-Status: Assigned +Status: Feedback

[2015-12-16 17:05 UTC] iquito at gmx dot net

Unfortunately, I do not know how to make further tests in a "safe" way - this problem only occured in production with many requests per second, and only after at least 12 hours of PHP-FPM running. If any string can have a "flipped" character in it at some point, I cannot use PHP7 for now - really bad stuff could happen in my applications.

I switched back to PHP 5.6 for now - no problem like this occurs with 5.6. PHP 7 is still running on the server with the smarty "bug" in it, but I do not know if that helps or if any information about what caused this bug can be captured from a running PHP-FPM instance which exhibits this bug.

[2015-12-27 04:22 UTC] php-bugs at lists dot php dot net

No feedback was provided. The bug is being suspended because
we assume that you are no longer experiencing the problem.
If this is not the case and you are able to provide the
information that was requested earlier, please do so and
change the status of the bug back to "Re-Opened". Thank you.

[2015-12-27 23:07 UTC] yohgaki@php.net

-Status: No Feedback +Status: Open

[2015-12-27 23:12 UTC] iquito at gmx dot net

I cannot edit the bug report / change the status, the error message

"You aren't allowed to change a bug to that state."

always appears, although I provide the correct password, and therefore I cannot submit anything with the "Edit" method or provide "official" feedback. I therefore provided feedback via Add Comment, which does not seem to count.

[2016-01-18 08:49 UTC] w1zzard at techpowerup dot com

I can confirm this issue.

Seeing it on PHP 7.0.2 from remi. Just like OP it happens roughly once a day. 

Running inside Docker container.

[2016-02-02 19:34 UTC] sroussey at gmail dot com

Try with fast_shutdown set to 0. This cleared up issues we had with string memory corruption.

[2016-02-02 21:52 UTC] iquito at gmx dot net

fast_shutdown was always set to 0 in my configuration.

[2016-02-05 13:47 UTC] vojtech dot kurka at gmail dot com

I think we're running into something related: We have a static testcase, that calls an API, that takes a result from a database and uses json_encode() to turn it into JSON and send it back. The testcase is called every 10 minutes in production to check that API is working. Randomly, that tescase starts failing on an error (json_encode reports recursion detected). The dataset is still the same. When we restart php70-php-fpm, everything is working again.

Update to PHP 7.0.3 didn't resolve this issue.

Our environment:
Centos 6, 2.6.32-573.12.1.el6.x86_64
PHP 7.0.3 (remi) with opcache
php70-php-fpm

Opcache config contains only these settings:

zend_extension=opcache.so
opcache.enable=1
opcache.memory_consumption=128
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=4000
opcache.blacklist_filename=/etc/opt/remi/php70/php.d/opcache*.blacklist


I think some fuzzy-testing for this issue would be useful. We are still trying to find a reproducible testcase, but without any success.

We are running PHP7 and PHP5.5.32 in parallel in production and PHP5 is completely OK, no such problems observed.

[2016-02-08 17:59 UTC] matej21 at matej21 dot cz

Hi, I'm experiencing similar issue. It sometimes happens to me after opcache_reset (not only manual but also automatic cache reset when opcache is out of a memory). I was able to reproduce it on my dev machine using "ab" requesting the site and running opcache_reset few times (it happens in about 10% of opcache_reset calls)

I tried it with both enabled and disabled protected_memory. And here are some errors and traces:


with opcache.protected_memory=1


Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000a5e9dd in zend_string_release (s=0x7f5937855d80) at /home/matej21/tmp/php-7.0.3/Zend/zend_string.h:270
270                     if (--GC_REFCOUNT(s) == 0) {

(gdb) bt
#0  0x0000000000a5e9dd in zend_string_release (s=0x7f5937855d80) at /home/matej21/tmp/php-7.0.3/Zend/zend_string.h:270
#1  0x0000000000a629f1 in zend_hash_destroy (ht=0x7f593599ef60) at /home/matej21/tmp/php-7.0.3/Zend/zend_hash.c:1265
#2  0x0000000000a81e45 in zend_gc_collect_cycles () at /home/matej21/tmp/php-7.0.3/Zend/zend_gc.c:1117
#3  0x0000000000a3517f in shutdown_executor () at /home/matej21/tmp/php-7.0.3/Zend/zend_execute_API.c:352
#4  0x0000000000a4cf98 in zend_deactivate () at /home/matej21/tmp/php-7.0.3/Zend/zend.c:967
#5  0x00000000009ba30d in php_request_shutdown (dummy=0x0) at /home/matej21/tmp/php-7.0.3/main/main.c:1823
#6  0x0000000000b1d4ae in main (argc=4, argv=0x7ffe1f4434a8) at /home/matej21/tmp/php-7.0.3/sapi/fpm/fpm/fpm_main.c:1972


Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000a5e9dd in zend_string_release (s=0x7f593779d6b0) at /home/matej21/tmp/php-7.0.3/Zend/zend_string.h:270
270                     if (--GC_REFCOUNT(s) == 0) {

(gdb) bt
#0  0x0000000000a5e9dd in zend_string_release (s=0x7f593779d6b0) at /home/matej21/tmp/php-7.0.3/Zend/zend_string.h:270
#1  0x0000000000a629f1 in zend_hash_destroy (ht=0x7f59360751c0) at /home/matej21/tmp/php-7.0.3/Zend/zend_hash.c:1265
#2  0x0000000000a3b042 in destroy_zend_class (zv=0x7ffe1f442c40) at /home/matej21/tmp/php-7.0.3/Zend/zend_opcode.c:289
#3  0x0000000000a6203f in _zend_hash_del_el_ex (ht=0x30df4a0, idx=700, p=0x34138c0, prev=0x0) at /home/matej21/tmp/php-7.0.3/Zend/zend_hash.c:1013
#4  0x0000000000a6210a in _zend_hash_del_el (ht=0x30df4a0, idx=700, p=0x34138c0) at /home/matej21/tmp/php-7.0.3/Zend/zend_hash.c:1037
#5  0x0000000000a639a6 in zend_hash_reverse_apply (ht=0x30df4a0, apply_func=0xa3454a <clean_non_persistent_class>) at /home/matej21/tmp/php-7.0.3/Zend/zend_hash.c:1615
#6  0x0000000000a35269 in shutdown_executor () at /home/matej21/tmp/php-7.0.3/Zend/zend_execute_API.c:367
#7  0x0000000000a4cf98 in zend_deactivate () at /home/matej21/tmp/php-7.0.3/Zend/zend.c:967
#8  0x00000000009ba30d in php_request_shutdown (dummy=0x0) at /home/matej21/tmp/php-7.0.3/main/main.c:1823
#9  0x0000000000b1d4ae in main (argc=4, argv=0x7ffe1f4434a8) at /home/matej21/tmp/php-7.0.3/sapi/fpm/fpm/fpm_main.c:1972


---------------------------------------

with opcache.protected_memory=0

first some error messages showing corrupted memory:

require(): Failed opening required '/var/www/foo-project/vendor/composer/autoload_namespaces.phpï¿œï¿œï¿œï¿œ'
require_once(): Failed opening required '/var/www-foo-project/vendor/composer/autoload_real.pfp'
require_once(): Failed opening required '/var/www/foo-project/vendor/composer/autoload_real.pgpï¿œï¿œ' 
include(): Failed opening '/var/www/foo-project/vendor/nette/utils/src/Utils/DateTime.phpï¿œ/Cube`'
Fatal Error: Class 'Solarium\Core\Configurableÿÿ' not found-obleceni/
Error: Class 'Nette\Http\SessionSectionÿÿÿ' not found
Error: Class 'Nette\Caching\Storages\FileStorageÿÿ' not found
Uncaught Error: Call to undefined method ComposerAutoloaderInit41,89608998d7c84bcc13a727394dca1::getLoader() 
Error: Class 'App\Security\UserStorageÿÿÿÿ' not found in /var/www/foo-project/temp/cache/Nette.Configurator/Container_1efc2a7985.php:7427


sometimes "only" segfault has happened, but opcache was not corrupted (IIRC), therefore other requests didn't fail:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f47ada7dfb7 in kill () at ../sysdeps/unix/syscall-template.S:81
81      ../sysdeps/unix/syscall-template.S: Adresář nebo soubor neexistuje.

(gdb) bt
#0  0x00007f47ada7dfb7 in kill () at ../sysdeps/unix/syscall-template.S:81
#1  0x0000000000a1663f in zend_mm_panic (message=0x1135a1d "zend_mm_heap corrupted") at /home/matej21/tmp/php-7.0.3/Zend/zend_alloc.c:364
#2  0x0000000000a18346 in zend_mm_free_heap (heap=0x7f47a7800040, ptr=0x7f47a35988b0, __zend_filename=0x113ce20 "/home/matej21/tmp/php-7.0.3/Zend/zend_string.h", __zend_lineno=271, __zend_orig_filename=0x0, __zend_orig_lineno=0)
    at /home/matej21/tmp/php-7.0.3/Zend/zend_alloc.c:1400
#3  0x0000000000a1ac7e in _efree (ptr=0x7f47a35988b0, __zend_filename=0x113ce20 "/home/matej21/tmp/php-7.0.3/Zend/zend_string.h", __zend_lineno=271, __zend_orig_filename=0x0, __zend_orig_lineno=0)
    at /home/matej21/tmp/php-7.0.3/Zend/zend_alloc.c:2458
#4  0x0000000000a5ea2c in zend_string_release (s=0x7f47a35988b0) at /home/matej21/tmp/php-7.0.3/Zend/zend_string.h:271
#5  0x0000000000a629f1 in zend_hash_destroy (ht=0x7f47a2920fc0) at /home/matej21/tmp/php-7.0.3/Zend/zend_hash.c:1265
#6  0x0000000000a3b042 in destroy_zend_class (zv=0x7fffbfa67e60) at /home/matej21/tmp/php-7.0.3/Zend/zend_opcode.c:289
#7  0x0000000000a6203f in _zend_hash_del_el_ex (ht=0x20884a0, idx=535, p=0x23b3360, prev=0x0) at /home/matej21/tmp/php-7.0.3/Zend/zend_hash.c:1013
#8  0x0000000000a6210a in _zend_hash_del_el (ht=0x20884a0, idx=535, p=0x23b3360) at /home/matej21/tmp/php-7.0.3/Zend/zend_hash.c:1037
#9  0x0000000000a639a6 in zend_hash_reverse_apply (ht=0x20884a0, apply_func=0xa3454a <clean_non_persistent_class>) at /home/matej21/tmp/php-7.0.3/Zend/zend_hash.c:1615
#10 0x0000000000a35269 in shutdown_executor () at /home/matej21/tmp/php-7.0.3/Zend/zend_execute_API.c:367
#11 0x0000000000a4cf98 in zend_deactivate () at /home/matej21/tmp/php-7.0.3/Zend/zend.c:967
#12 0x00000000009ba30d in php_request_shutdown (dummy=0x0) at /home/matej21/tmp/php-7.0.3/main/main.c:1823
#13 0x0000000000b1d4ae in main (argc=4, argv=0x7fffbfa686c8) at /home/matej21/tmp/php-7.0.3/sapi/fpm/fpm/fpm_main.c:1972



Program terminated with signal SIGABRT, Aborted.
#0  0x00007f47ada7dcc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56      ../nptl/sysdeps/unix/sysv/linux/raise.c: Adresář nebo soubor neexistuje.

(gdb) bt
#0  0x00007f47ada7dcc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007f47ada810d8 in __GI_abort () at abort.c:89
#2  0x00007f47ada76b86 in __assert_fail_base (fmt=0x7f47b1835029 <error: Cannot access memory at address 0x7f47b1835029>, assertion=assertion@entry=0x113d04d "*end == '\\0'", 
    file=file@entry=0x113cf08 "/home/matej21/tmp/php-7.0.3/Zend/zend_hash.c", line=line@entry=2432, function=function@entry=0x113d200 <__PRETTY_FUNCTION__.8927> "_zend_handle_numeric_str_ex") at assert.c:92
#3  0x00007f47ada76c32 in __GI___assert_fail (assertion=0x113d04d "*end == '\\0'", file=0x113cf08 "/home/matej21/tmp/php-7.0.3/Zend/zend_hash.c", line=2432, 
    function=0x113d200 <__PRETTY_FUNCTION__.8927> "_zend_handle_numeric_str_ex") at assert.c:101
#4  0x0000000000a65c03 in _zend_handle_numeric_str_ex (key=0x7f47a36645c8 "234\377", length=3, idx=0x7fffbfa65c18) at /home/matej21/tmp/php-7.0.3/Zend/zend_hash.c:2432
#5  0x0000000000aa0236 in _zend_handle_numeric_str (key=0x7f47a36645c8 "234\377", length=3, idx=0x7fffbfa65c18) at /home/matej21/tmp/php-7.0.3/Zend/zend_hash.h:264
#6  0x0000000000b07c30 in ZEND_ISSET_ISEMPTY_DIM_OBJ_SPEC_TMPVAR_CV_HANDLER () at /home/matej21/tmp/php-7.0.3/Zend/zend_vm_execute.h:44069
#7  0x0000000000aa93ff in execute_ex (ex=0x7f47a7815060) at /home/matej21/tmp/php-7.0.3/Zend/zend_vm_execute.h:414
#8  0x0000000000a36ce7 in zend_call_function (fci=0x7fffbfa65e90, fci_cache=0x7fffbfa65e60) at /home/matej21/tmp/php-7.0.3/Zend/zend_execute_API.c:860
#9  0x00000000008943f8 in zif_call_user_func_array (execute_data=0x7f47a7814fe0, return_value=0x7f47a7814fa0) at /home/matej21/tmp/php-7.0.3/ext/standard/basic_functions.c:4811
#10 0x0000000000aa9f6a in ZEND_DO_FCALL_BY_NAME_SPEC_HANDLER () at /home/matej21/tmp/php-7.0.3/Zend/zend_vm_execute.h:714
#11 0x0000000000aa93ff in execute_ex (ex=0x7f47a7814030) at /home/matej21/tmp/php-7.0.3/Zend/zend_vm_execute.h:414
#12 0x0000000000aa9511 in zend_execute (op_array=0x7f47a7865000, return_value=0x0) at /home/matej21/tmp/php-7.0.3/Zend/zend_vm_execute.h:458
#13 0x0000000000a4e394 in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /home/matej21/tmp/php-7.0.3/Zend/zend.c:1427
#14 0x00000000009bb7a1 in php_execute_script (primary_file=0x7fffbfa683d0) at /home/matej21/tmp/php-7.0.3/main/main.c:2484
#15 0x0000000000b1d368 in main (argc=4, argv=0x7fffbfa686c8) at /home/matej21/tmp/php-7.0.3/sapi/fpm/fpm/fpm_main.c:1944



Program terminated with signal SIGSEGV, Segmentation fault.
#0  memset () at ../sysdeps/x86_64/memset.S:93
93      ../sysdeps/x86_64/memset.S: Adresář nebo soubor neexistuje.

(gdb) bt
#0  memset () at ../sysdeps/x86_64/memset.S:93
#1  0x0000000000aa83e6 in i_init_code_execute_data (execute_data=0x7f47a7814a50, op_array=0x7f47a7873700, return_value=0x0) at /home/matej21/tmp/php-7.0.3/Zend/zend_execute.c:2200
#2  0x0000000000ae5790 in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER () at /home/matej21/tmp/php-7.0.3/Zend/zend_vm_execute.h:29164
#3  0x0000000000aa93ff in execute_ex (ex=0x7f47a7814930) at /home/matej21/tmp/php-7.0.3/Zend/zend_vm_execute.h:414
#4  0x0000000000a36ce7 in zend_call_function (fci=0x7fffbfa64ad0, fci_cache=0x7fffbfa64aa0) at /home/matej21/tmp/php-7.0.3/Zend/zend_execute_API.c:860
#5  0x0000000000a73f76 in zend_call_method (object=0x7f47a7802d28, obj_ce=0x7f47a7805380, fn_proxy=0x7f47a7802d20, function_name=0x7f47a792fbe8 "composer\\autoload\\classloader::loadclass\001", function_name_len=44, 
    retval_ptr=0x0, param_count=1, arg1=0x7f47a7814920, arg2=0x0) at /home/matej21/tmp/php-7.0.3/Zend/zend_interfaces.c:104
#6  0x000000000083e7fc in zif_spl_autoload_call (execute_data=0x7f47a78148c0, return_value=0x7fffbfa64da0) at /home/matej21/tmp/php-7.0.3/ext/spl/php_spl.c:429
#7  0x0000000000a36e0e in zend_call_function (fci=0x7fffbfa64de0, fci_cache=0x7fffbfa64db0) at /home/matej21/tmp/php-7.0.3/Zend/zend_execute_API.c:879
#8  0x0000000000a3758c in zend_lookup_class_ex (name=0x7f47a359b1d8, key=0x7f47a398ef20, use_autoload=1) at /home/matej21/tmp/php-7.0.3/Zend/zend_execute_API.c:1041
#9  0x0000000000a380d5 in zend_fetch_class_by_name (class_name=0x7f47a359b1d8, key=0x7f47a398ef20, fetch_type=6) at /home/matej21/tmp/php-7.0.3/Zend/zend_execute_API.c:1387
#10 0x0000000000aac681 in ZEND_ADD_TRAIT_SPEC_HANDLER () at /home/matej21/tmp/php-7.0.3/Zend/zend_vm_execute.h:1449
#11 0x0000000000aa93ff in execute_ex (ex=0x7f47a7814730) at /home/matej21/tmp/php-7.0.3/Zend/zend_vm_execute.h:414
#12 0x0000000000a36ce7 in zend_call_function (fci=0x7fffbfa65140, fci_cache=0x7fffbfa65110) at /home/matej21/tmp/php-7.0.3/Zend/zend_execute_API.c:860
#13 0x0000000000a73f76 in zend_call_method (object=0x7f47a7802d28, obj_ce=0x7f47a7805380, fn_proxy=0x7f47a7802d20, function_name=0x7f47a792fbe8 "composer\\autoload\\classloader::loadclass\001", function_name_len=44, 
    retval_ptr=0x0, param_count=1, arg1=0x7f47a7814720, arg2=0x0) at /home/matej21/tmp/php-7.0.3/Zend/zend_interfaces.c:104
#14 0x000000000083e7fc in zif_spl_autoload_call (execute_data=0x7f47a78146c0, return_value=0x7fffbfa65410) at /home/matej21/tmp/php-7.0.3/ext/spl/php_spl.c:429
#15 0x0000000000a36e0e in zend_call_function (fci=0x7fffbfa65450, fci_cache=0x7fffbfa65420) at /home/matej21/tmp/php-7.0.3/Zend/zend_execute_API.c:879
#16 0x0000000000a3758c in zend_lookup_class_ex (name=0x7f47a359b5d8, key=0x0, use_autoload=1) at /home/matej21/tmp/php-7.0.3/Zend/zend_execute_API.c:1041
#17 0x0000000000a6c0e3 in zif_class_alias (execute_data=0x7f47a7814640, return_value=0x7f47a7814620) at /home/matej21/tmp/php-7.0.3/Zend/zend_builtin_functions.c:1585
#18 0x0000000000aa99d0 in ZEND_DO_ICALL_SPEC_HANDLER () at /home/matej21/tmp/php-7.0.3/Zend/zend_vm_execute.h:586
#19 0x0000000000aa93ff in execute_ex (ex=0x7f47a7814590) at /home/matej21/tmp/php-7.0.3/Zend/zend_vm_execute.h:414
#20 0x0000000000a36ce7 in zend_call_function (fci=0x7fffbfa657e0, fci_cache=0x7fffbfa657b0) at /home/matej21/tmp/php-7.0.3/Zend/zend_execute_API.c:860
#21 0x0000000000a73f76 in zend_call_method (object=0x0, obj_ce=0x0, fn_proxy=0x7f47a78fe190, function_name=0x7f47a7901bf8 "closure::__invoke\003", function_name_len=21, retval_ptr=0x0, param_count=1, arg1=0x7f47a7814580, 
    arg2=0x0) at /home/matej21/tmp/php-7.0.3/Zend/zend_interfaces.c:104
#22 0x000000000083e7fc in zif_spl_autoload_call (execute_data=0x7f47a7814520, return_value=0x7fffbfa65ab0) at /home/matej21/tmp/php-7.0.3/ext/spl/php_spl.c:429
#23 0x0000000000a36e0e in zend_call_function (fci=0x7fffbfa65af0, fci_cache=0x7fffbfa65ac0) at /home/matej21/tmp/php-7.0.3/Zend/zend_execute_API.c:879
#24 0x0000000000a3758c in zend_lookup_class_ex (name=0x7f47a359bbc0, key=0x0, use_autoload=1) at /home/matej21/tmp/php-7.0.3/Zend/zend_execute_API.c:1041
#25 0x0000000000a37685 in zend_lookup_class (name=0x7f47a359bbc0) at /home/matej21/tmp/php-7.0.3/Zend/zend_execute_API.c:1062
#26 0x0000000000a6b557 in zif_class_exists (execute_data=0x7f47a78144b0, return_value=0x7fffbfa65e50) at /home/matej21/tmp/php-7.0.3/Zend/zend_builtin_functions.c:1437
#27 0x0000000000a36e0e in zend_call_function (fci=0x7fffbfa65ea0, fci_cache=0x7fffbfa65e70) at /home/matej21/tmp/php-7.0.3/Zend/zend_execute_API.c:879
#28 0x000000000088de93 in zif_array_map (execute_data=0x7f47a7814430, return_value=0x7f47a7814420) at /home/matej21/tmp/php-7.0.3/ext/standard/array.c:5290
#29 0x0000000000aa99d0 in ZEND_DO_ICALL_SPEC_HANDLER () at /home/matej21/tmp/php-7.0.3/Zend/zend_vm_execute.h:586
#30 0x0000000000aa93ff in execute_ex (ex=0x7f47a7814030) at /home/matej21/tmp/php-7.0.3/Zend/zend_vm_execute.h:414
#31 0x0000000000aa9511 in zend_execute (op_array=0x7f47a7873000, return_value=0x0) at /home/matej21/tmp/php-7.0.3/Zend/zend_vm_execute.h:458
#32 0x0000000000a4e394 in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /home/matej21/tmp/php-7.0.3/Zend/zend.c:1427
#33 0x00000000009bb7a1 in php_execute_script (primary_file=0x7fffbfa683d0) at /home/matej21/tmp/php-7.0.3/main/main.c:2484
#34 0x0000000000b1d368 in main (argc=4, argv=0x7fffbfa686c8) at /home/matej21/tmp/php-7.0.3/sapi/fpm/fpm/fpm_main.c:1944

[2016-02-22 22:12 UTC] matej21 at matej21 dot cz

this issue is probably fixed by https://github.com/php/php-src/commit/d2287529396539fd2cba6f449ae9679cac64c3b9

[2016-02-22 22:36 UTC] iquito at gmx dot net

Would be great if this problem was fixed, but the linked changes do not seem related to the bug here (as far as I can tell) - in my configuration it sometimes occured after a opcache_reset(), but those were not the majority cases, usually it just happened at seemingly random times when the opcache was warmed up and in full swing for many hours.

I do use opcache_invalidate quite regularly on some files, but the error occured so infrequently that I do not know if that is linked to this bug - the errors occured once a day, but within that timeframe I used opcache_reset and opcache_invalidate many times without problems. I also never had a crash, only one "flipped" character in a string, which seems an odd and hard-to-pinpoint bug anyway - almost all strings were still correct, only 1-2 strings were one character off. This makes this bug so intimidating - if strings can change a bit seemingly random, than that can be a real problem for an application and can also stay unnoticed for some time.

[2016-02-24 21:03 UTC] dmitry@php.net

The problem was caused by incorrect synchronization of processes during opcache reload. Some processes might crash or might not crash because of race-conditions. The commit mentioned above and a couple of following fixes, should solve the problem.

[2016-02-24 21:29 UTC] vojtech dot kurka at gmail dot com

Dmitry, are all these fixes included in 7.0.4? Thank you

[2016-03-05 12:06 UTC] vojtech dot kurka at gmail dot com

We have 7.0.4 in production for 48 hours and it is running without any problems with json_encode() - seems fixed for us.

Thank you, Dmitri!

[2016-03-09 15:11 UTC] iquito at gmx dot net

-Status: Assigned +Status: Closed

[2016-03-09 15:11 UTC] iquito at gmx dot net

I have also been running 7.0.4 in production for 48 hours with a lot of opcache_reset's and opcache_invalidate's, and there do not seem to be any issues anymore - everything seems to be working great. Therefore I am closing this issue.

[2016-08-16 10:19 UTC] teohhanhui at gmail dot com

We're still running into this problem in production on PHP 7.0.8

It happens every few days or so.

Examples of corruption:

Symfony -> Sylfony
Bundle -> Bumdle

(both cases happen to FQCN stored in variables)

[2017-07-03 10:38 UTC] kalyanceg at gmail dot com

We are facing this issue on debian stretch with php7.0-opcache 7.0.19-1
We see opcache getting corrupted after some days of apache running continuously and gets fixed only on restart of apache

[2017-08-28 00:06 UTC] nick at noodles dot net dot nz

I'm starting to see this problem on PHP 7.1.8. Seems the opcache is randomly being corrupted one character at a time. This is on a high traffic website with multiple web servers.

[2017-08-28 08:28 UTC] spam2 at rhsoft dot net

i face this below with 7.1.8 on the first production machine with 7.1.x randomly when under light load shared files are updated - maybe related?

Sun Aug 20 19:45:38 2017 (3066): Warning Internal error: wrong size calculation: global_mysql_ext.inc.php start=0x00007feba814c580, end=0x00007feba816fc60, real=0x00007feba816fc40

[2017-08-28 22:52 UTC] jagoba at tapatalk dot com

We do have this problem too on production, during heavy loading we got errors about sql queries that get mistery changed to "SELECT * FRPM" (notice the P), or links in our pages with www changed to vww.

We got this issues both in php 7.1.8 and 7.1.3 with opcache, any hint or workaround is welcome.

[2017-08-28 23:03 UTC] nikic@php.net

If you are still experiencing this problem or something similar to it, please try the suggestion from @laruence earlier in this thread: Enable the opcache.protect_memory ini option, which will (likely) trigger a crash instead of silently corrupting the memory. It's hard to do anything about this if we don't know where the invalid write occurs.

[2017-08-29 00:40 UTC] nick at noodles dot net dot nz

I can replicate the issue by rsync'ing our files to the production web servers over and over. This causes the timestamps to be updated each time, opcache sees these as new files and stores them again.

If I watch the wasted memory stats I can see that every time we rsync it increases by ~15MB. Once the free memory is all used up I start to get memory corruption issues (bit flipping).

I'm using the defaults for opcache, so it should be firing opcache_reset when the wasted memory is above 5%, but that doesn't look like it happens.


I'm having trouble keeping the server alive to get it to segfault when the memory corruption occurs. As soon as I enable the memory protection flag I get segfaults every second.

This is the core backtrace (or at least what I can gather):

Core was generated by `/usr/sbin/httpd -DFOREGROUND'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f47054f6d43 in s_zval_to_payload.isra.21 () from /usr/lib64/php/modules/memcached.so
Missing separate debuginfos, use: debuginfo-install apr-1.4.8-3.el7.x86_64 apr-util-1.5.2-6.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 elfutils-libelf-0.166-2.el7.x86_64 elfutils-libs-0.166-2.el7.x86_64 enchant-1.6.0-8.el7.x86_64 expat-2.1.0-10.el7_3.x86_64 fastlz-0.1.0-0.1.20070619svnrev12.el7.x86_64 freetype-2.4.11-12.el7.x86_64 glib2-2.46.2-4.el7.x86_64 glibc-2.17-157.el7_3.5.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 libX11-1.6.3-3.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXpm-3.5.11-3.el7.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libcurl-7.29.0-35.el7.centos.x86_64 libdb-5.3.21-19.el7.x86_64 libevent-2.0.21-4.el7.x86_64 libgcc-4.8.5-11.el7.x86_64 libgcrypt-1.5.3-13.el7_3.1.x86_64 libgpg-error-1.12-3.el7.x86_64 libicu-50.1.2-15.el7.x86_64 libidn-1.28-4.el7.x86_64 libjpeg-turbo-1.2.90-5.el7.x86_64 libmcrypt-2.5.8-13.el7.x86_64 libmemcached-libs-1.0.18-1.x86_64 libnghttp2-1.21.1-1.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64 libselinux-2.5-6.el7.x86_64 libssh2-1.4.3-10.el7_2.1.x86_64 libstdc++-4.8.5-11.el7.x86_64 libtool-ltdl-2.4.2-22.el7_3.x86_64 libuuid-2.23.2-33.el7_3.2.x86_64 libwebp-0.3.0-3.el7.x86_64 libxcb-1.11-4.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 libxslt-1.1.28-5.el7.x86_64 lua-5.1.4-15.el7.x86_64 mod_log_firstbyte-1.01-3.el7.x86_64 nspr-4.13.1-1.0.el7_3.x86_64 nss-3.28.4-1.2.el7_3.x86_64 nss-softokn-freebl-3.16.2.3-14.4.el7.x86_64 nss-util-3.28.4-1.0.el7_3.x86_64 openldap-2.4.40-13.el7.x86_64 openssl-libs-1.0.1e-60.el7_3.1.x86_64 pcre-8.32-15.el7_2.1.x86_64 php-pecl-igbinary-2.0.1-1.el7.x86_64 php-pecl-memcached-3.0.3-1.el7.x86_64 php-pecl-msgpack-2.0.2-1.el7.x86_64 sqlite-3.7.17-8.el7.x86_64 systemd-libs-219-30.el7_3.9.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0  0x00007f47054f6d43 in s_zval_to_payload.isra.21 () from /usr/lib64/php/modules/memcached.so
#1  0x00007f47054f7453 in s_memc_write_zval () from /usr/lib64/php/modules/memcached.so
#2  0x00007f47054f80ce in php_memc_store_impl.isra.25 () from /usr/lib64/php/modules/memcached.so
#3  0x00007f47123c1e6c in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at /usr/src/debug/php-src-php-7.1.8/Zend/zend_vm_execute.h:970
#4  0x00007f471236ffbb in execute_ex (ex=<optimized out>) at /usr/src/debug/php-src-php-7.1.8/Zend/zend_vm_execute.h:429
#5  0x00007f47123c3c74 in zend_execute (op_array=0x7f471167b000, op_array@entry=0x7f46fde42150, return_value=return_value@entry=0x7f47116115f0)
    at /usr/src/debug/php-src-php-7.1.8/Zend/zend_vm_execute.h:474
#6  0x00007f47123285b3 in zend_execute_scripts (type=type@entry=8, retval=0x7f47116115f0, retval@entry=0x0, file_count=file_count@entry=3)
    at /usr/src/debug/php-src-php-7.1.8/Zend/zend.c:1476
#7  0x00007f47122c5b18 in php_execute_script (primary_file=primary_file@entry=0x7ffd229e12a0) at /usr/src/debug/php-src-php-7.1.8/main/main.c:2537
#8  0x00007f47123c5c7d in php_handler (r=<optimized out>) at /usr/src/debug/php-src-php-7.1.8/sapi/apache2handler/sapi_apache2.c:712
#9  0x00007f47227a9550 in ap_run_handler (r=r@entry=0x7f47232b41b0) at config.c:170
#10 0x00007f47227a9a99 in ap_invoke_handler (r=r@entry=0x7f47232b41b0) at config.c:434
#11 0x00007f47227c05ea in ap_process_async_request (r=0x7f47232b41b0) at http_request.c:436
#12 0x00007f47227c08c4 in ap_process_request (r=r@entry=0x7f47232b41b0) at http_request.c:471
#13 0x00007f47227bc77d in ap_process_http_sync_connection (c=0x7f472338f7a0) at http_core.c:210
#14 ap_process_http_connection (c=0x7f472338f7a0) at http_core.c:251
#15 0x00007f47227b39c0 in ap_run_process_connection (c=c@entry=0x7f472338f7a0) at connection.c:42
#16 0x00007f47227b3f18 in ap_process_connection (c=c@entry=0x7f472338f7a0, csd=<optimized out>) at connection.c:226
#17 0x00007f4714beaa50 in child_main (child_num_arg=child_num_arg@entry=0, child_bucket=child_bucket@entry=0) at prefork.c:726
#18 0x00007f4714beacf1 in make_child (s=0x7f4722fb9420, slot=slot@entry=0, bucket=0) at prefork.c:834
#19 0x00007f4714bead5f in startup_children (number_to_start=10) at prefork.c:853
#20 0x00007f4714bebc83 in prefork_run (_pconf=<optimized out>, plog=0x7f4722fbf3a8, s=0x7f4722fb9420) at prefork.c:1020
#21 0x00007f472278b45e in ap_run_mpm (pconf=pconf@entry=0x7f4722f92188, plog=0x7f4722fbf3a8, s=0x7f4722fb9420) at mpm_common.c:94
#22 0x00007f4722783d78 in main (argc=2, argv=0x7ffd229e19e8) at main.c:783

I'm not sure if this helps, but I can do more debugging if needed.

[2017-08-29 02:52 UTC] nick at noodles dot net dot nz

For reference, this seems to be related to pecl-memcached and interned strings. This patch worked for me: https://github.com/php-memcached-dev/php-memcached/commit/5f28025c15c87d0895f39def77068c8fd66d442a

[2017-08-30 01:37 UTC] nick at noodles dot net dot nz

The pecl-memcached patch helped (at least it's not segfaulting every second). But I'm still seeing random memory corruption. I managed to get some core dumps:

Core was generated by `/usr/sbin/httpd -DFOREGROUND'.
Program terminated with signal 11, Segmentation fault.
#0  sljit_malloc_exec (size=816) at /usr/src/debug/php-src-php-7.1.8/ext/pcre/pcrelib/sljit/sljitExecAllocator.c:196
196					header->prev_size = chunk_size;
Missing separate debuginfos, use: debuginfo-install apr-1.4.8-3.el7.x86_64 apr-util-1.5.2-6.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 elfutils-libelf-0.166-2.el7.x86_64 elfutils-libs-0.166-2.el7.x86_64 enchant-1.6.0-8.el7.x86_64 expat-2.1.0-10.el7_3.x86_64 fastlz-0.1.0-0.1.20070619svnrev12.el7.x86_64 freetype-2.4.11-12.el7.x86_64 glib2-2.46.2-4.el7.x86_64 glibc-2.17-157.el7_3.5.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 libX11-1.6.3-3.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXpm-3.5.11-3.el7.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libcurl-7.29.0-35.el7.centos.x86_64 libdb-5.3.21-19.el7.x86_64 libevent-2.0.21-4.el7.x86_64 libgcc-4.8.5-11.el7.x86_64 libgcrypt-1.5.3-13.el7_3.1.x86_64 libgpg-error-1.12-3.el7.x86_64 libicu-50.1.2-15.el7.x86_64 libidn-1.28-4.el7.x86_64 libjpeg-turbo-1.2.90-5.el7.x86_64 libmcrypt-2.5.8-13.el7.x86_64 libmemcached-libs-1.0.18-1.x86_64 libnghttp2-1.21.1-1.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64 libselinux-2.5-6.el7.x86_64 libssh2-1.4.3-10.el7_2.1.x86_64 libstdc++-4.8.5-11.el7.x86_64 libtool-ltdl-2.4.2-22.el7_3.x86_64 libuuid-2.23.2-33.el7_3.2.x86_64 libwebp-0.3.0-3.el7.x86_64 libxcb-1.11-4.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 libxslt-1.1.28-5.el7.x86_64 lua-5.1.4-15.el7.x86_64 mod_log_firstbyte-1.01-3.el7.x86_64 nspr-4.13.1-1.0.el7_3.x86_64 nss-3.28.4-1.2.el7_3.x86_64 nss-softokn-freebl-3.16.2.3-14.4.el7.x86_64 nss-util-3.28.4-1.0.el7_3.x86_64 openldap-2.4.40-13.el7.x86_64 openssl-libs-1.0.1e-60.el7_3.1.x86_64 pcre-8.32-15.el7_2.1.x86_64 php-pecl-igbinary-2.0.1-1.el7.x86_64 php-pecl-memcached-3.0.3-2.el7.x86_64 php-pecl-msgpack-2.0.2-1.el7.x86_64 sqlite-3.7.17-8.el7.x86_64 systemd-libs-219-30.el7_3.9.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0  sljit_malloc_exec (size=816) at /usr/src/debug/php-src-php-7.1.8/ext/pcre/pcrelib/sljit/sljitExecAllocator.c:196
#1  sljit_generate_code (compiler=compiler@entry=0x7f228e3312f0) at /usr/src/debug/php-src-php-7.1.8/ext/pcre/pcrelib/sljit/sljitNativeX86_common.c:466
#2  0x00007f227b988a50 in _pcre_jit_compile (re=re@entry=0x7f228e11e870, extra=extra@entry=0x7f228e2fc0f0, mode=mode@entry=0)
    at /usr/src/debug/php-src-php-7.1.8/ext/pcre/pcrelib/pcre_jit_compile.c:10342
#3  0x00007f227b9639ed in php_pcre_study (external_re=external_re@entry=0x7f228e11e870, options=1, errorptr=errorptr@entry=0x7ffe7cbea888)
    at /usr/src/debug/php-src-php-7.1.8/ext/pcre/pcrelib/pcre_study.c:1628
#4  0x00007f227b98b99b in pcre_get_compiled_regex_cache (regex=0x7f2266d00728) at /usr/src/debug/php-src-php-7.1.8/ext/pcre/php_pcre.c:518
#5  0x00007f227b98e1a2 in php_pcre_replace (regex=<optimized out>, subject_str=subject_str@entry=0x7f227af11000,
    subject=subject@entry=0x7f227af11018 "<html>\n    <head>\n    \t<title>Gallstones</title>\n    \t<meta name=\"originalfile\" content=\"AZ_d0198.xml\" />\n    </head>\n    <body>\n    \n", ' ' <repeats 12 times>, "\n", ' ' <repeats 15 times>, "<h1>Gallstones </h1>\n", ' ' <repeats 15 times>, "\n "..., subject_len=12448, replace_val=replace_val@entry=0x7f227ae11830,
    is_callable_replace=is_callable_replace@entry=0, limit=limit@entry=-1, replace_count=replace_count@entry=0x7ffe7cbeaa5c) at /usr/src/debug/php-src-php-7.1.8/ext/pcre/php_pcre.c:1132
#6  0x00007f227b98e2a2 in php_replace_in_subject (regex=regex@entry=0x7f227ae11820, replace=replace@entry=0x7f227ae11830, subject=subject@entry=0x7f227ae11840, limit=limit@entry=-1,
    is_callable_replace=is_callable_replace@entry=0, replace_count=replace_count@entry=0x7ffe7cbeaa5c) at /usr/src/debug/php-src-php-7.1.8/ext/pcre/php_pcre.c:1495
#7  0x00007f227b98e6ae in preg_replace_impl (return_value=return_value@entry=0x7f227ae11770, regex=regex@entry=0x7f227ae11820, replace=0x7f227ae11830, subject=0x7f227ae11840, limit_val=-1,
    is_callable_replace=is_callable_replace@entry=0, is_filter=is_filter@entry=0) at /usr/src/debug/php-src-php-7.1.8/ext/pcre/php_pcre.c:1554
#8  0x00007f227b98f20f in zif_preg_replace (execute_data=0x7f227ae117d0, return_value=0x7f227ae11770) at /usr/src/debug/php-src-php-7.1.8/ext/pcre/php_pcre.c:1593
#9  0x00007f227bb21006 in ZEND_DO_ICALL_SPEC_RETVAL_USED_HANDLER () at /usr/src/debug/php-src-php-7.1.8/Zend/zend_vm_execute.h:675
#10 0x00007f227bb10fbb in execute_ex (ex=<optimized out>) at /usr/src/debug/php-src-php-7.1.8/Zend/zend_vm_execute.h:429
#11 0x00007f227bb64c74 in zend_execute (op_array=0x7f227ae7a000, op_array@entry=0x7f22677811f0, return_value=return_value@entry=0x7f227ae11030)
    at /usr/src/debug/php-src-php-7.1.8/Zend/zend_vm_execute.h:474
#12 0x00007f227bac95b3 in zend_execute_scripts (type=type@entry=8, retval=0x7f227ae11030, retval@entry=0x0, file_count=file_count@entry=3) at /usr/src/debug/php-src-php-7.1.8/Zend/zend.c:1476
#13 0x00007f227ba66b18 in php_execute_script (primary_file=primary_file@entry=0x7ffe7cbece90) at /usr/src/debug/php-src-php-7.1.8/main/main.c:2537
#14 0x00007f227bb66c7d in php_handler (r=<optimized out>) at /usr/src/debug/php-src-php-7.1.8/sapi/apache2handler/sapi_apache2.c:712
#15 0x00007f228bf4a550 in ap_run_handler (r=r@entry=0x7f228e1f5968) at config.c:170
#16 0x00007f228bf4aa99 in ap_invoke_handler (r=r@entry=0x7f228e1f5968) at config.c:434
#17 0x00007f228bf6094c in ap_internal_redirect (new_uri=<optimized out>, r=<optimized out>) at http_request.c:765
#18 0x00007f2280e49f1c in handler_redirect (r=0x7f228e1af910) at mod_rewrite.c:5195
#19 0x00007f228bf4a550 in ap_run_handler (r=r@entry=0x7f228e1af910) at config.c:170
#20 0x00007f228bf4aa99 in ap_invoke_handler (r=r@entry=0x7f228e1af910) at config.c:434
#21 0x00007f228bf615ea in ap_process_async_request (r=0x7f228e1af910) at http_request.c:436
#22 0x00007f228bf618c4 in ap_process_request (r=r@entry=0x7f228e1af910) at http_request.c:471
#23 0x00007f228bf5d77d in ap_process_http_sync_connection (c=0x7f228e191310) at http_core.c:210
#24 ap_process_http_connection (c=0x7f228e191310) at http_core.c:251
#25 0x00007f228bf549c0 in ap_run_process_connection (c=c@entry=0x7f228e191310) at connection.c:42
#26 0x00007f228bf54f18 in ap_process_connection (c=c@entry=0x7f228e191310, csd=<optimized out>) at connection.c:226
#27 0x00007f227e38ba50 in child_main (child_num_arg=child_num_arg@entry=10, child_bucket=child_bucket@entry=0) at prefork.c:726
#28 0x00007f227e38bcf1 in make_child (s=0x7f228ddbd420, slot=10, bucket=0) at prefork.c:834
#29 0x00007f227e38cc34 in perform_idle_server_maintenance (p=<optimized out>) at prefork.c:942
#30 prefork_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at prefork.c:1138
#31 0x00007f228bf2c45e in ap_run_mpm (pconf=pconf@entry=0x7f228dd96188, plog=0x7f228ddc33a8, s=0x7f228ddbd420) at mpm_common.c:94
#32 0x00007f228bf24d78 in main (argc=2, argv=0x7ffe7cbed658) at main.c:783

Hopefully this helps

[2017-09-08 22:37 UTC] jjones at smugmug dot com

We're running a PHP7.1.9 build on our internal testing servers, preparing for a production rollout to upgrade from PHP5.6 and are also seeing SegFaults.

Our setup is NGINX/php-fpm and the php-fpm processes sometimes running cleanly for non-trivial amounts of time, then we'll see or more processes crash before stablizing again.

We have core dumps enabled, have build the binaries with optimization disabled, meaning "-O0" for the C and C++ compiles.  Ane we are running with "opcache.protect_memory=1" already, since we were debugging another source of SegFaults before this use case appeared.

Here's a backtrace from the most recent crash, with a few curious data points that may (or may not?) be helpful.

This was the last of 5 core dumps in cluster today, so it's possible that it's the victim of corrupted shared memory rather then the one that caused it in the first place.  The previous core dump (from a memcache extension issue) was generating core dumps too frequently to save them all.  So the current setup uses a static name for the core dumps, so the last one is all I have.


Backtrace:

#0  0x0000000000b64983 in zend_inline_hash_func (len=31426464, str=0x7fcf65800001 <error: Cannot access memory at address 0x7fcf65800001>) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_string.h:331
#1  zend_hash_func (str=0x7fcf656e71a0 "", len=32577024) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_string.c:34
#2  0x0000000000b3bc9a in zend_string_hash_val (s=0x7fcf656e7188) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_string.h:86
#3  zend_hash_find_bucket (key=0x7fcf656e7188, ht=0x1ec34a0) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_hash.c:477
#4  zend_hash_find (ht=0x1ec34a0, key=0x7fcf656e7188) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_hash.c:1958
#5  0x00007fcf6536cbe5 in zend_accel_class_hash_copy (target=0x1ec34a0, source=0x7fced63878d0, pCopyConstructor=0x7fcf6536b697 <zend_class_copy_ctor>) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/ext/opcache/zend_accelerator_util_funcs.c:572
#6  0x00007fcf6536d391 in zend_accel_load_script (persistent_script=0x7fced63877c0, from_shared_memory=1) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/ext/opcache/zend_accelerator_util_funcs.c:655
#7  0x00007fcf6534ca87 in persistent_compile_file (file_handle=0x7fff7073a3b0, type=2) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/ext/opcache/ZendAccelerator.c:1948
#8  0x0000000000a987b6 in compile_filename (type=2, filename=0x7fcf65614790) at Zend/zend_language_scanner.l:662
#9  0x0000000000ba94c9 in zend_include_or_eval (inc_filename=0x7fcf65614790, type=2) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_execute.c:2839
#10 0x0000000000c88862 in ZEND_INCLUDE_OR_EVAL_SPEC_TMPVAR_HANDLER () at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:51660
#11 0x0000000000ba9a6b in execute_ex (ex=0x7fcf656146a0) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#12 0x00007fcf635e77e0 in hp_execute_ex (execute_data=0x7fcf656146a0) at /opt/xhprof/build/7.1.6/tideways.c:3603
#13 hp_execute_ex (execute_data=0x7fcf656146a0) at /opt/xhprof/build/7.1.6/tideways.c:3549
#14 0x0000000000aee90e in zend_call_function (fci=0x7fff7073ac50, fci_cache=0x7fff7073ac20) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_execute_API.c:855
#15 0x0000000000b4f153 in zend_call_method (object=0x0, obj_ce=0x0, fn_proxy=0x7fcf65663bd0, function_name=0x7fcf65602a40 "__autoload", function_name_len=10, retval_ptr=0x0, param_count=1, arg1=0x7fcf65614690, arg2=0x0)
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_interfaces.c:99
#16 0x00000000008c3769 in zif_spl_autoload_call (execute_data=0x7fcf65614640, return_value=0x7fff7073b260) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/ext/spl/php_spl.c:420
#17 0x0000000000ba612b in execute_internal (execute_data=0x7fcf65614640, return_value=0x7fff7073b260) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_execute.c:2040
#18 0x0000000000aeea21 in zend_call_function (fci=0x7fff7073b2a0, fci_cache=0x7fff7073b270) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_execute_API.c:871
#19 0x0000000000aef8d3 in zend_lookup_class_ex (name=0x7fcf65663f90, key=0x0, use_autoload=1) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_execute_API.c:1028
#20 0x0000000000af07cf in zend_fetch_class (class_name=0x7fcf65663f90, fetch_type=512) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_execute_API.c:1441
#21 0x0000000000bb5cc1 in ZEND_FETCH_CLASS_SPEC_CV_HANDLER () at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:2444
#22 0x0000000000ba9a6b in execute_ex (ex=0x7fcf656145a0) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#23 0x00007fcf635e77e0 in hp_execute_ex (execute_data=0x7fcf656145a0) at /opt/xhprof/build/7.1.6/tideways.c:3603
#24 hp_execute_ex (execute_data=0x7fcf656145a0) at /opt/xhprof/build/7.1.6/tideways.c:3549
#25 0x0000000000badfa4 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:949
#26 0x0000000000ba9a6b in execute_ex (ex=0x7fcf656140e0) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#27 0x0000000000c88ec8 in ZEND_INCLUDE_OR_EVAL_SPEC_TMPVAR_HANDLER () at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:51699
#28 0x0000000000ba9a6b in execute_ex (ex=0x7fcf65614030) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#29 0x0000000000baa42b in zend_execute (op_array=0x7fcf65672000, return_value=0x0) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:474
#30 0x0000000000b0fcc1 in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend.c:1480
#31 0x0000000000a5b290 in php_execute_script (primary_file=0x7fff7073dd60) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/main/main.c:2552
#32 0x0000000000cb836f in main (argc=2, argv=0x7fff7073e058) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/sapi/fpm/fpm/fpm_main.c:1966

Crash caused by call to compute the hash of a null string with a bogus/corrupt length given.


Data structures/fields in the code mention in the stack that might be clues.

---- Bad "refcount" in string structure

(gdb) list
81	/*---*/
82	
83	static zend_always_inline zend_ulong zend_string_hash_val(zend_string *s)
84	{
85		if (!ZSTR_H(s)) {
86			ZSTR_H(s) = zend_hash_func(ZSTR_VAL(s), ZSTR_LEN(s));
87		}
88		return ZSTR_H(s);
89	}
90	
(gdb) print *s
$11 = {gc = {refcount = 4294967295, u = {v = {type = 0 '\000', flags = 0 '\000', gc_info = 0}, type_info = 0}}, h = 0, len = 32577024, val = ""}
(gdb) printf "%x\n", s->gc->refcount
ffffffff

Meaning the "refcount" associated with the string is the maximum value that can fit in a 32-bit unsigned int.  Or it's -1 in a signed 32-bit int.  Maybe some refcount tracking code overflowed the counter, or maybe a '0' value was decremented?



And finally, in case it helps, the functions call chain suggests PHP is crashing trying to fetch something from cache, not while interpreting PHP code itself.

(gdb) print execute_data->func->common->function_name->val+0
$26 = 0x7fced3fb8248 "__autoload"
(gdb) print execute_data->prev_execute_data->func->common->function_name->val+0
$27 = 0x7fced3fc0be8 "spl_autoload_call"

[2017-09-09 05:37 UTC] jjones at smugmug dot com

Since posting the previous core dump (and info extracted from it), I found the following in the source.

ext/opcache/zend_file_cache.c:  op_array~~>refcount = (uint32_t*)(intptr_t)-1;

Zend/zend_execute.c:  #define ZEND_FAKE_OP_ARRAY ((zend_op_array*)(zend_intptr_t)-1)

which explains the 0xffffffff value of the "refcount" field in the failing call.  So scratch the question I asked about an intended decrement flipping a zero to all 0xff's.

[2017-09-10 06:31 UTC] rasmus@php.net

Is this after a cache full event? As in, when you see this happening, what are the values of these opcache vars in your phpinfo output?

    OOM restarts
    Hash keys restarts
    Manual restarts

PHP 7.1.x has been running in production on a ton of heavily hit sites with at least some of them making heavy use of memcached without ever seeing this. The ones I am involved with almost never hit a cache reset condition, so that might be a difference.

[2017-09-10 09:14 UTC] jjones at smugmug dot com

I captured another core dump which has different backtrace, but ultimately crashes with a SegFault due to a string pointer is can't dereference.

#0  0x0000000000bb6763 in ZEND_ECHO_SPEC_CONST_HANDLER ()
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:2667
#1  0x0000000000ba9a6b in execute_ex (ex=0x7f8f2f415430)
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#2  0x0000000000c88ec8 in ZEND_INCLUDE_OR_EVAL_SPEC_TMPVAR_HANDLER ()
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:51699
#3  0x0000000000ba9a6b in execute_ex (ex=0x7f8f2f4153a0)
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#4  0x0000000000baec11 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:1076
#5  0x0000000000ba9a6b in execute_ex (ex=0x7f8f2f415300)
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#6  0x0000000000baec11 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:1076
#7  0x0000000000ba9a6b in execute_ex (ex=0x7f8f2f415270)
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#8  0x0000000000baec11 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:1076
#9  0x0000000000ba9a6b in execute_ex (ex=0x7f8f2f415150)
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#10 0x0000000000badfa4 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER ()
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:949
#11 0x0000000000ba9a6b in execute_ex (ex=0x7f8f2f4150e0)
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#12 0x0000000000c88ec8 in ZEND_INCLUDE_OR_EVAL_SPEC_TMPVAR_HANDLER ()
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:51699
#13 0x0000000000ba9a6b in execute_ex (ex=0x7f8f2f415030)
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#14 0x0000000000baa42b in zend_execute (op_array=0x7f8f2f4731c0, return_value=0x0)
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:474
#15 0x0000000000b0fcc1 in zend_execute_scripts (type=8, retval=0x0, file_count=3)
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend.c:1480
#16 0x0000000000a5b290 in php_execute_script (primary_file=0x7ffc59ab28f0)
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/main/main.c:2552
#17 0x0000000000cb836f in main (argc=2, argv=0x7ffc59ab2be8)
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/sapi/fpm/fpm/fpm_main.c:1966

some info that might be helpful...

(gdb) list 2667
2662		z = EX_CONSTANT(opline->op1);
2663	
2664		if (Z_TYPE_P(z) == IS_STRING) {
2665			zend_string *str = Z_STR_P(z);
2666	
2667			if (ZSTR_LEN(str) != 0) {
2668				zend_write(ZSTR_VAL(str), ZSTR_LEN(str));
2669			}
2670		} else {
2671			zend_string *str = _zval_get_string_func(z);
(gdb) print *z
$6 = {value = {lval = 140250487923552, dval = 6.9292947895499668e-310, counted = 0x7f8e9c832360, str = 0x7f8e9c832360, 
    arr = 0x7f8e9c832360, obj = 0x7f8e9c832360, res = 0x7f8e9c832360, ref = 0x7f8e9c832360, ast = 0x7f8e9c832360, zv = 0x7f8e9c832360, 
    ptr = 0x7f8e9c832360, ce = 0x7f8e9c832360, func = 0x7f8e9c832360, ww = {w1 = 2625839968, w2 = 32654}}, u1 = {v = {type = 6 '\006', 
      type_flags = 0 '\000', const_flags = 0 '\000', reserved = 0 '\000'}, type_info = 6}, u2 = {next = 4294967295, 
    cache_slot = 4294967295, lineno = 4294967295, num_args = 4294967295, fe_pos = 4294967295, fe_iter_idx = 4294967295, 
    access_flags = 4294967295, property_guard = 4294967295, extra = 4294967295}}
(gdb) print z->value->str
$7 = (zend_string *) 0x7f8e9c832360
(gdb) print *z->value->str
Cannot access memory at address 0x7f8e9c832360

[2017-09-10 09:18 UTC] jjones at smugmug dot com

As for the values of the opcache restart stats from the crashes processes, I think this is what you're asking for.

#0  0x0000000000b64983 in zend_inline_hash_func (len=31426464, 
    str=0x7fcf65800001 <error: Cannot access memory at address 0x7fcf65800001>)
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_string.h:331
331			hash = ((hash << 5) + hash) + *str++;
(gdb) print *accel_shared_globals
$1 = {hits = 573580, misses = 94, blacklist_misses = 0, oom_restarts = 0, hash_restarts = 0, manual_restarts = 0


Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000bb6763 in ZEND_ECHO_SPEC_CONST_HANDLER ()
    at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:2667
2667			if (ZSTR_LEN(str) != 0) {
(gdb) print *accel_shared_globals
$1 = {hits = 19012587, misses = 53, blacklist_misses = 0, oom_restarts = 0, hash_restarts = 0, manual_restarts = 0

Meaning, they were all zero at the time, in both core dumps.

[2017-09-10 13:25 UTC] rasmus@php.net

Do you get a useful zbacktrace from these cores? (See https://github.com/php/php-src/blob/PHP-7.1/.gdbinit)

[2017-09-11 01:19 UTC] jjones at smugmug dot com

Thanks for the pointer to the gdbinit file.  There's some nice tools in there...

Here are the results from "zbacktrace" from the two core dumps I have.

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000b64983 in zend_inline_hash_func (len=31426464, str=0x7fcf65800001 <error: Cannot access memory at address 0x7fcf65800001>) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_string.h:331
331			hash = ((hash << 5) + hash) + *str++;
(gdb) zbacktrace
[0x7fcf656146a0] Lib\Base->__autoload("SmugMug\Router\Img") /var/www/www_inside/1504876151/include/setup.mgi:731 
[0x7fcf65614640] spl_autoload_call("SmugMug\Router\Img") [internal function]
[0x7fff7073b0a0] ??? 
[0x7fcf656145a0] SmugMug\Router\Base->setup() /var/www/www_inside/1504876151/include/classes/SmugMug/Router/Base.php:607 
[0x7fcf656140e0] (main) /var/www/www_inside/1504876151/include/setup.mgi:100 
[0x7fcf65614030] (main) /var/www/www_inside/1504876151/index.mg:8 



Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000bb6763 in ZEND_ECHO_SPEC_CONST_HANDLER () at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:2667
2667			if (ZSTR_LEN(str) != 0) {
(gdb) zbacktrace
[0x7f8f2f415430] Lib\Spit->(main) /var/www/www_inside/1504876151/include/legacy-pages/test/react-library/react-tools-browserify-bundle.js:1 
[0x7f8f2f4153a0] Lib\Spit->out("/include/legacy-pages/test/react-library/react-tools-browserify-bundle.js", reference) /var/www/www_inside/1504876151/include/classes/Lib/Spit.php:39 
[0x7f8f2f415300] Lib\Spit->swallow("/include/legacy-pages/test/react-library/react-tools-browserify-bundle.js", array(0)[0x7f8f2f415360]) /var/www/www_inside/1504876151/include/classes/Lib/Spit.php:57 
[0x7f8f2f415270] SmugMug\Spit->swallow("/include/legacy-pages/test/react-library/react-tools-browserify-bundle.js") /var/www/www_inside/1504876151/include/classes/SmugMug/Spit.php:61 
[0x7f8f2f415150] SmugMug\UI\UIExample->init(object[0x7f8f2f4151a0], array(1)[0x7f8f2f4151b0]) /var/www/www_inside/1504876151/include/classes/SmugMug/UI/UIExample.php:54 
[0x7f8f2f4150e0] (main) /var/www/www_inside/1504876151/include/legacy-pages/test/react-library/index.mg:94 
[0x7f8f2f415030] (main) /var/www/www_inside/1504876151/index.mg:21

[2017-09-11 05:16 UTC] rasmus@php.net

So two very different spots in your PHP code. Kind of expected for memory corruption. The corruption happened earlier and it is somewhat random when it actually triggers a crash.  I don't suppose you can reproduce it reliably? If you can Valgrind might be able to spot the initial corruption.

I use a this little valgrind script to do a memcheck:

#!/bin/bash
USE_ZEND_ALLOC=0 valgrind --tool=memcheck --leak-check=yes --suppressions=/home/rasmus/.suppressions --track-origins=yes --num-callers=30 --show-reachable=yes "$@"

You can skip the suppressions, although it isn't a bad idea to run it once on a trivial phpinfo() or hello world script and record your suppressions just to clean up the output a bit.

Then I just do: memcheck php script.php
Of course, it is unlikely to be that simple, but you can also run the entire php-fpm or httpd under memcheck. It runs very slowly, so don't do this on a production box, of course.

[2017-09-11 10:24 UTC] jjones at smugmug dot com

Re: coming up with a reproducible use case that triggers a crash. We're working on that, since it would be the ideal way to find/fix the problem.  But no luck so far.  So that's a "work in progress".

Re: Valgrind, I thought about that and wasn't sure how usable the system would be in terms of performance with that extra overhead added to the "php-fpm" processes.  But at this point I think it's worth a shot.

Re: disparity in the crash dump outputs when the root cause it memory corruption. Yup! I have two more core dumps so far and I haven't found anything that links them together yet aside from impossible values for various address pointers.  I'm going to dig into them more and keep searching though.

[2017-09-11 22:55 UTC] sroussey at gmail dot com

jjones: I wrote to nick at noodles up above about the memcache issue. Not sure why the issue is only in 7.1 vs 7.0. We found high corruption rates in 5.6.x where x>9 as well. Never figured them all out, but fast_shutdown=0 helped a bit. The 7.0.x series has been the most stable for us. 

That said, I really don't recommend rsync as a deployment strategy. Feel free to message me for other ideas for a high load site.

[2017-09-12 04:28 UTC] jjones at smugmug dot com

I failed mention it in my original posting, but we are running with "fast_shutdown=0" already.  That suggestion popped up in a debugging posting I read early on in the process.

We found a way to reproduce the SegFault's we're seeing on our PHP 7.1.9 servers today.  The servers running a normal configuration still see "php-fpm" crashes from time to time.  They are infrequent however so iterative debugging has been slow and tedious.

But we found that by reducing the "opcache.interned_strings_buffer" setting the time between crashes was greatly reduced.  Checking the resulting core dumps in "gdb" with,

    set $total = (accel_shared_globals->interned_strings_end - accel_shared_globals->interned_strings_start) + 1
    set $used = accel_shared_globals->interned_strings_top - accel_shared_globals->interned_strings_start
    set $left = (accel_shared_globals->interned_strings_end - accel_shared_globals->interned_strings_top) + 1
    set $prev = accel_shared_globals->interned_strings_end - accel_shared_globals->interned_strings_saved_top
    printf "Interned Strings Total Memory:  %10d\n", $total
    printf "Interned Strings Used Memory:   %10d\n", $used
    printf "Interned Strings Free Memory:   %10d\n", $left
    printf "Interned Strings Previous Free: %10d\n", $prev

shows that the buffer reserved for interned strings is exhausted when each of them crashed.  Going back to the core dumps from the "normal" configurations, we found that they showed similar stats with respect to the available interned string buffer space.

In other words, the one things the core dumps seem to have in common is that they have almost no free space in their interned string buffers when they attempt to derefernce an invalid address pointer.  The backtraces differ, the code which triggers the SegFault, and the types of structures with invalid pointers varied.  But in the six core dumps we captured today, three from a normal configuration and three from a system with very limited interned string buffers, when they crashed there was very little space available in the buffer space.

- - - Core file: php-fpm719-170911:1317.dump
[New LWP 4163]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `php-fpm: pool www              '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000b3bc76 in zend_string_hash_val (s=0x7f97754e22b0) at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_string.h:85
85              if (!ZSTR_H(s)) {
Interned Strings Total Memory:     4194305
Interned Strings Used Memory:      4194280
Interned Strings Free Memory:           25
Interned Strings Previous Free:    3856224

- - - Core file: php-fpm719-170911:1332.dump
[New LWP 15871]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `php-fpm: pool www              '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000c72bc5 in ZEND_ISSET_ISEMPTY_DIM_OBJ_SPEC_CV_CV_HANDLER () at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:47096
47096                                   if (ZEND_HANDLE_NUMERIC(str, hval)) {
Interned Strings Total Memory:     4194305
Interned Strings Used Memory:      4194296
Interned Strings Free Memory:            9
Interned Strings Previous Free:    3856224

- - - Core file: php-fpm719-170911:1336.dump
[New LWP 13330]
[Thread debugging using libthread_db enabled]
xcUsing host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `php-fpm: pool www              '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000c72bc5 in ZEND_ISSET_ISEMPTY_DIM_OBJ_SPEC_CV_CV_HANDLER () at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:47096
47096                                   if (ZEND_HANDLE_NUMERIC(str, hval)) {
Interned Strings Total Memory:     4194305
Interned Strings Used Memory:      4194296
Interned Strings Free Memory:            9
Interned Strings Previous Free:    3856224

[2017-09-12 12:18 UTC] mike@php.net

Several ingredients are needed to reproduce:

* opcache.file_cache enabled
* opcache.interned_strings_buffer set low (eg. 1)
* a reasonable large codebase to fill up the interned strings buffer
* quite a bit of luck that the remaining free buffer size is low enough to trigger

I could reproducibly, but not quite reliably produce a crash or zend_mm_panic with that recipe with eg. laravel or phan.

I've way too less knowledge of opcache to argue yet what's going on, but I'll upload a patch which /seemed/ to fix the issue for me. Maybe someone with more insight can comment whether I'm totally off.

[2017-09-12 12:19 UTC] mike@php.net

The following patch has been added/updated:

Patch Name: interned_strings_top
Revision:   1505218752
URL:        https://bugs.php.net/patch-display.php?bug=71135&patch=interned_strings_top&revision=1505218752

[2017-09-12 13:13 UTC] mike@php.net

Some valgrind logs:
https://gist.github.com/m6w6/67f71247f5325497a59d4686ba4845ac

[2017-09-12 17:21 UTC] mike@php.net

After many more tests, it seems it rather depends on the moon phase, because I cannot reproduce anymore.

Looking into the write of uninitialized bytes next.

[2017-09-13 01:22 UTC] nick at noodles dot net dot nz

I'm still seeing problems as well. So far I've disabled huge code pages and changed opcache to not check timestamps (code deploys now fire an opcache_reset() on each webserver).

Is it possible to disable interned strings?

Also, should we be adding comments to this bug or creating a new one (being that this is closed)?

[2017-09-13 01:46 UTC] jjones at smugmug dot com

Today we've been running tests with "opcache.file_cache=" disabled.  We were using that feature before.

I have not been able to crash "php-fpm" since making that change, even on a system with a tightly constrained "interned string buffer" setting.

So IMO if you're using Opcache file backed caching, it's worth trying to run with it disabled.

[2017-09-13 01:59 UTC] nick at noodles dot net dot nz

We've always had file cache disabled for what it's worth.

[2017-09-13 02:32 UTC] rasmus@php.net

You really don't want to run without the string buffer. You also don't want to fill it. You will be leaving quite a bit of performance on the table if you do. Instead of turning it off, I would jack it way up if I were you.

At Etsy we deploy dozens of times every day and do not do any sort of cache reset. But there is a nightly log rotation that does a graceful restart so everything does get cleaned up once per day. And these dozens of deploys are atomic and cache-preserving in the sense that I switch back and forth between A and B docroots so only files changed across deploys need to be recompiled. I describe the theory behind it here: https://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/

For php-fpm behind nginx see the comments of that article and the magic nginx config variable $realpath_root

[2017-09-13 03:23 UTC] nick at noodles dot net dot nz

Thanks, I've bumped it from 8MB -> 32MB and will monitor it. I can't see it hitting the 8MB limit when the corruption occurs, but it gets close.

[2017-09-13 04:08 UTC] rasmus@php.net

It depends on your code, of course, but for a large site you probably need to go to 128M or 256M for the string buffer.

[2018-01-07 21:09 UTC] jagoba at tapatalk dot com

We still have this issue. We are now on php 7.2.0, our opcache is configured to 256mb and we not using file_cache. I saw that in php master there is a commit to change how interned strings works. Is there posibility that code be merged into next version 7.2.2? will it solve the problem?

[2018-01-07 23:24 UTC] jjones at smugmug dot com

I just heard (today) about and reviewed the patches described here:

https://bugs.php.net/bug.php?id=75579

and from the description of that issue, and what the fix does, I think it's entirely possible that it will our crashes with file backed caching.  I haven't had time to test it, and we're running just fine with file based caching disabled for now.  But I wanted to connect the dots for other people following this bug.

[2019-05-14 13:04 UTC] irker@php.net

See this patch: http://git.php.net/?p=php-src.git;a=commit;h=5c4d125d4c2976236e2ecddd1d8c6e7b113ec482

It must fix some cases with broken opcache after reset.

	php.net \| support \| documentation \| report a bug \| advanced search \| search howto \| statistics \| random bug \| login
go to bug id or search bugs for


Copyright © 2001-2024 The PHP Group All rights reserved.	Last updated: Sat Nov 23 17:01:28 2024 UTC