I get these crashes with my custom site. It's not related to some specific path or time. Sometimes it's 1.8 seconds from start, sometimes 6 hours, after 1500 requests on average. I can't yet reproduce it reliably.

I suspect that something bad is happening with strings or memory allocation.

A bit before the crash, I often see ”child [PID] exited with code 1”. And sometimes, instead of a crash, my site starts producing other fatal errors with clearly corrupted code. ”Call to undefined method ClassA::ClassB::ConstInClassB()” where actually ClassA::foo() is called, or an SQL error of a missing table dbname.FOO, where FOO occurs in many other contexts but certainly not in static::$db_table.

Could some PHP core developer guess how this kind of corruption could happen? Maybe reviewing related changes between 7.2.13 and 7.3.0 could then be useful. (What is the best git command for that, considering the way PHP branches are forked?)

Assuming this only happens with opcache (based on the package), the first two things to try would be:

 * Set opcache.optimization_level=0 and see if this resolves the issue.
 * Set opcache.protect_memory=1 and see if you get reproducible segfaults. A backtrace from one of those should pinpoint the location.

I'd also recommend testing with a current snapshot of PHP 7.3, as a number of opcache-related bugs have already been fixed, but not released yet.

Sometimes php-fpm child segfaults in zend_mm_alloc_small or calls exit(1) in zend_mm_panic.

I've managed to cut it down to this weird test case:

<?php
// Run with FastCGI 100 times in a row. Look at php-fpm log.
$pdo = new PDO("mysql:host=localhost", "root", "", array(
	PDO::ATTR_PERSISTENT => true,
	PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
));
$pdo->exec("DROP DATABASE IF EXISTS tmp_database");
$pdo->exec("CREATE DATABASE tmp_database");
$pdo->exec("DROP TABLE IF EXISTS tmp_database.tmp_table");
$pdo->exec("CREATE TEMPORARY TABLE tmp_database.tmp_table (x INT)");
$pdo->exec("UPDATE tmp_database.tmp_table SET x = x");
?>

This triggers the bug in around 10 % of runs. Note that this is visible only in the php-fpm console ("child ... exited with code 1"), not in the script itself.

Disabling opcache makes the bug actually easier to reproduce.

In 7.3.0 the bug was also triggered by a slightly more complex test case containing autoload and eval (and MySQL UPDATE), and in that case, string lengths (like the length of the autoloaded class name) affected whether the bug was triggered or not. Also, my test case had originally a few hundred lines (real-life code with real-life input), and even changing some unused code or changing the length of some unused string literals (especially around 16 chars) would change the frequency of the bug, sometimes even make it disappear. That's why I'm tempted to blame either strings or memory management in general.

However, I couldn't reproduce this at all without MySQL, so it could still be a MySQL-related bug. I even bisected this to commit 4d5330fb, but reverting it didn't fix the problem.

I spent a whole day on this. What I've found out so far:

The memory manager resets between requests, reusing the same memory blocks for the next request, but some part of PHP has a "persistent" memory block which, after being freed, causes the same block to appear twice in the free_slots list.

At this point, the linked list heap->free_slot[bin_num] (slot [5] in my code) ends up with a duplicate entry, which obviously leads to the same memory slot being allocated multiple times. After the second allocation, the slot already contains user-defined data but is still used as the next pointer in the linked list, which causes the occassional SEGFAULT. This could even allow remote code execution, if ASLR didn't make it so difficult.

Currently mysqlnd is the primary suspect considering the following backtraces. (Note that numbers in zend_alloc.c are a bit off because of my debugging functions.)


Backtraces:

Allocating 0x7fc2a3277030, which will then live past zend_mm_shutdown.

Thread 2.1 "php-fpm" hit Breakpoint 1, my_break (ptr=0x7fc2a3277030) at Zend/zend_alloc.c:347
#0  my_break (ptr=0x7fc2a3277030) at Zend/zend_alloc.c:347
#1  0x000055ad3ddbbc97 in my_alloc_debug (ptr=140473937653808) at Zend/zend_alloc.c:355
#2  0x000055ad3ddc7fe2 in zend_mm_alloc_small (bin_num=5, size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1360
#3  zend_mm_alloc_heap (size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1442
#4  _emalloc (size=41) at Zend/zend_alloc.c:2618
#5  0x000055ad3dd51d97 in _mysqlnd_emalloc (size=41) at ext/mysqlnd/mysqlnd_alloc.c:99
#6  0x000055ad3dd597e8 in php_mysqlnd_rset_header_read (conn=0x55ad3eb59650, _packet=0x7fff7b65a640) at ext/mysqlnd/mysqlnd_wireprotocol.c:1106
#7  0x000055ad3dd67461 in mysqlnd_query_read_result_set_header (conn=0x55ad3eb59650, s=0x0) at ext/mysqlnd/mysqlnd_result.c:392
#8  0x000055ad3dd71f9b in mysqlnd_com_reap_result_run (cmd=0x7fff7b65a960) at ext/mysqlnd/mysqlnd_commands.c:719
#9  0x000055ad3dd72020 in mysqlnd_com_reap_result_run_command (args=0x7fff7b65a9a0) at ext/mysqlnd/mysqlnd_commands.c:736
#10 0x000055ad3dd73d3c in _mysqlnd_run_command (command=COM_REAP_RESULT) at ext/mysqlnd/mysqlnd_commands.c:1345
#11 0x000055ad3dd4bb77 in mysqlnd_mysqlnd_conn_data_reap_query_pub (conn=0x55ad3eb59650, type=MYSQLND_REAP_RESULT_IMPLICIT) at ext/mysqlnd/mysqlnd_connection.c:904
#12 0x000055ad3dd4b776 in mysqlnd_mysqlnd_conn_data_query_pub (conn=0x55ad3eb59650, query=0x7fc2a3279118 "UPDATE tmp_database.tmp_table SET x = x", query_len=39) at ext/mysqlnd/mysqlnd_connection.c:851
#13 0x000055ad3dc9163f in mysql_handle_doer (dbh=0x55ad3eb52e80, sql=<optimized out>, sql_len=<optimized out>) at ext/pdo_mysql/mysql_driver.c:259
#14 0x000055ad3dc86e85 in zim_PDO_exec (execute_data=<optimized out>, return_value=0x7fff7b65ab80) at ext/pdo/pdo_dbh.c:923
#15 0x000055ad3de7b154 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER (execute_data=0x7fc2a321d020) at Zend/zend_vm_execute.h:983
#16 0x000055ad3de3bedc in execute_ex (ex=<optimized out>) at Zend/zend_vm_execute.h:55012
#17 0x000055ad3de7b413 in zend_execute (op_array=op_array@entry=0x7fc2a32660e0, return_value=return_value@entry=0x0) at Zend/zend_vm_execute.h:60595
#18 0x000055ad3ddedecc in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3) at Zend/zend.c:1615
#19 0x000055ad3dd83f86 in php_execute_script (primary_file=<optimized out>) at main/main.c:2633
#20 0x000055ad3de88634 in main (argc=<optimized out>, argv=<optimized out>) at sapi/fpm/fpm/fpm_main.c:1945
Continuing.


Allocating 0x7fc2a3277090, which will then live past zend_mm_shutdown.

Thread 2.1 "php-fpm" hit Breakpoint 1, my_break (ptr=0x7fc2a3277090) at Zend/zend_alloc.c:347
#0  my_break (ptr=0x7fc2a3277090) at Zend/zend_alloc.c:347
#1  0x000055ad3ddbbcc1 in my_alloc_debug (ptr=140473937653904) at Zend/zend_alloc.c:358
#2  0x000055ad3ddc7fe2 in zend_mm_alloc_small (bin_num=5, size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1360
#3  zend_mm_alloc_heap (size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1442
#4  _emalloc (size=41) at Zend/zend_alloc.c:2618
#5  0x000055ad3dd53c03 in _mysqlnd_pestrndup (ptr=0x7fc2a3277030 "Rows matched: 0  Changed: 0  Warnings: 0", length=40, persistent=0 '\000') at ext/mysqlnd/mysqlnd_alloc.c:589
#6  0x000055ad3dd679b4 in mysqlnd_query_read_result_set_header (conn=0x55ad3eb59650, s=0x0) at ext/mysqlnd/mysqlnd_result.c:442
#7  0x000055ad3dd71f9b in mysqlnd_com_reap_result_run (cmd=0x7fff7b65a960) at ext/mysqlnd/mysqlnd_commands.c:719
#8  0x000055ad3dd72020 in mysqlnd_com_reap_result_run_command (args=0x7fff7b65a9a0) at ext/mysqlnd/mysqlnd_commands.c:736
#9  0x000055ad3dd73d3c in _mysqlnd_run_command (command=COM_REAP_RESULT) at ext/mysqlnd/mysqlnd_commands.c:1345
#10 0x000055ad3dd4bb77 in mysqlnd_mysqlnd_conn_data_reap_query_pub (conn=0x55ad3eb59650, type=MYSQLND_REAP_RESULT_IMPLICIT) at ext/mysqlnd/mysqlnd_connection.c:904
#11 0x000055ad3dd4b776 in mysqlnd_mysqlnd_conn_data_query_pub (conn=0x55ad3eb59650, query=0x7fc2a3279118 "UPDATE tmp_database.tmp_table SET x = x", query_len=39) at ext/mysqlnd/mysqlnd_connection.c:851
#12 0x000055ad3dc9163f in mysql_handle_doer (dbh=0x55ad3eb52e80, sql=<optimized out>, sql_len=<optimized out>) at ext/pdo_mysql/mysql_driver.c:259
#13 0x000055ad3dc86e85 in zim_PDO_exec (execute_data=<optimized out>, return_value=0x7fff7b65ab80) at ext/pdo/pdo_dbh.c:923
#14 0x000055ad3de7b154 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER (execute_data=0x7fc2a321d020) at Zend/zend_vm_execute.h:983
#15 0x000055ad3de3bedc in execute_ex (ex=<optimized out>) at Zend/zend_vm_execute.h:55012
#16 0x000055ad3de7b413 in zend_execute (op_array=op_array@entry=0x7fc2a32660e0, return_value=return_value@entry=0x0) at Zend/zend_vm_execute.h:60595
#17 0x000055ad3ddedecc in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3) at Zend/zend.c:1615
#18 0x000055ad3dd83f86 in php_execute_script (primary_file=<optimized out>) at main/main.c:2633
#19 0x000055ad3de88634 in main (argc=<optimized out>, argv=<optimized out>) at sapi/fpm/fpm/fpm_main.c:1945
Continuing.


zend_mm_shutdown is called here.


Freeing 0x7fc2a3277090, which now becomes a duplicate entry.

Thread 2.1 "php-fpm" hit Breakpoint 1, my_break (ptr=0x7fc2a3277090) at Zend/zend_alloc.c:347
#0  my_break (ptr=0x7fc2a3277090) at Zend/zend_alloc.c:347
#1  0x000055ad3ddbbd2a in heap_free_slot_5_init (p=0x7fc2a3277090) at Zend/zend_alloc.c:369
#2  0x000055ad3ddc81e9 in zend_mm_free_small (bin_num=5, ptr=0x7fc2a3277090, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1390
#3  zend_mm_free_heap (ptr=0x7fc2a3277090, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1486
#4  _efree (ptr=0x7fc2a3277090) at Zend/zend_alloc.c:2633
#5  0x000055ad3dd52cc1 in _mysqlnd_efree (ptr=0x7fc2a3277090) at ext/mysqlnd/mysqlnd_alloc.c:349
#6  0x000055ad3dd5e9fb in mysqlnd_mysqlnd_protocol_send_command_handle_OK_pub (payload_decoder_factory=0x55ad3eb5b7a0, error_info=0x55ad3eb59770, upsert_status=0x55ad3eb59738, ignore_upsert_status=1 '\001', last_message=0x55ad3eb59758) at ext/mysqlnd/mysqlnd_wireprotocol.c:2576
#7  0x000055ad3dd5ee76 in mysqlnd_mysqlnd_protocol_send_command_handle_response_pub (payload_decoder_factory=0x55ad3eb5b7a0, ok_packet=PROT_OK_PACKET, silent=1 '\001', command=COM_PING, ignore_upsert_status=1 '\001', error_info=0x55ad3eb59770, upsert_status=0x55ad3eb59738, last_message=0x55ad3eb59758) at ext/mysqlnd/mysqlnd_wireprotocol.c:2656
#8  0x000055ad3dd70dac in mysqlnd_com_ping_run (cmd=0x7fff7b65a780) at ext/mysqlnd/mysqlnd_commands.c:244
#9  0x000055ad3dd70e58 in mysqlnd_com_ping_run_command (args=0x7fff7b65a7c0) at ext/mysqlnd/mysqlnd_commands.c:268
#10 0x000055ad3dd73c86 in _mysqlnd_run_command (command=COM_PING) at ext/mysqlnd/mysqlnd_commands.c:1324
#11 0x000055ad3dd4c2fc in mysqlnd_mysqlnd_conn_data_ping_pub (conn=0x55ad3eb59650) at ext/mysqlnd/mysqlnd_connection.c:1090
#12 0x000055ad3dc9019e in pdo_mysql_check_liveness (dbh=<optimized out>) at ext/pdo_mysql/mysql_driver.c:505
#13 0x000055ad3dc87e00 in zim_PDO_dbh_constructor (execute_data=0x7fc2a321d100, return_value=<optimized out>) at ext/pdo/pdo_dbh.c:299
#14 0x000055ad3de7b154 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER (execute_data=0x7fc2a321d020) at Zend/zend_vm_execute.h:983
#15 0x000055ad3de3bedc in execute_ex (ex=<optimized out>) at Zend/zend_vm_execute.h:55012
#16 0x000055ad3de7b413 in zend_execute (op_array=op_array@entry=0x7fc2a32660e0, return_value=return_value@entry=0x0) at Zend/zend_vm_execute.h:60595
#17 0x000055ad3ddedecc in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3) at Zend/zend.c:1615
#18 0x000055ad3dd83f86 in php_execute_script (primary_file=<optimized out>) at main/main.c:2633
#19 0x000055ad3de88634 in main (argc=<optimized out>, argv=<optimized out>) at sapi/fpm/fpm/fpm_main.c:1945


Freeing 0x7fc2a3277030, which now becomes a duplicate entry.

Thread 2.1 "php-fpm" hit Breakpoint 1, my_break (ptr=0x7fc2a3277030) at Zend/zend_alloc.c:347
#0  my_break (ptr=0x7fc2a3277030) at Zend/zend_alloc.c:347
#1  0x000055ad3ddbbd2a in heap_free_slot_5_init (p=0x7fc2a3277030) at Zend/zend_alloc.c:369
#2  0x000055ad3ddc7fa5 in zend_mm_alloc_small (bin_num=5, size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1354
#3  zend_mm_alloc_heap (size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1442
#4  _emalloc (size=41) at Zend/zend_alloc.c:2618
#5  0x000055ad3dd51d97 in _mysqlnd_emalloc (size=41) at ext/mysqlnd/mysqlnd_alloc.c:99
#6  0x000055ad3dd597e8 in php_mysqlnd_rset_header_read (conn=0x55ad3eb59650, _packet=0x7fff7b65a640) at ext/mysqlnd/mysqlnd_wireprotocol.c:1106
#7  0x000055ad3dd67461 in mysqlnd_query_read_result_set_header (conn=0x55ad3eb59650, s=0x0) at ext/mysqlnd/mysqlnd_result.c:392
#8  0x000055ad3dd71f9b in mysqlnd_com_reap_result_run (cmd=0x7fff7b65a960) at ext/mysqlnd/mysqlnd_commands.c:719
#9  0x000055ad3dd72020 in mysqlnd_com_reap_result_run_command (args=0x7fff7b65a9a0) at ext/mysqlnd/mysqlnd_commands.c:736
#10 0x000055ad3dd73d3c in _mysqlnd_run_command (command=COM_REAP_RESULT) at ext/mysqlnd/mysqlnd_commands.c:1345
#11 0x000055ad3dd4bb77 in mysqlnd_mysqlnd_conn_data_reap_query_pub (conn=0x55ad3eb59650, type=MYSQLND_REAP_RESULT_IMPLICIT) at ext/mysqlnd/mysqlnd_connection.c:904
#12 0x000055ad3dd4b776 in mysqlnd_mysqlnd_conn_data_query_pub (conn=0x55ad3eb59650, query=0x7fc2a3279118 "UPDATE tmp_database.tmp_table SET x = x", query_len=39) at ext/mysqlnd/mysqlnd_connection.c:851
#13 0x000055ad3dc9163f in mysql_handle_doer (dbh=0x55ad3eb52e80, sql=<optimized out>, sql_len=<optimized out>) at ext/pdo_mysql/mysql_driver.c:259
#14 0x000055ad3dc86e85 in zim_PDO_exec (execute_data=<optimized out>, return_value=0x7fff7b65ab80) at ext/pdo/pdo_dbh.c:923
#15 0x000055ad3de7b154 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER (execute_data=0x7fc2a321d020) at Zend/zend_vm_execute.h:983
#16 0x000055ad3de3bedc in execute_ex (ex=<optimized out>) at Zend/zend_vm_execute.h:55012
#17 0x000055ad3de7b413 in zend_execute (op_array=op_array@entry=0x7fc2a32660e0, return_value=return_value@entry=0x0) at Zend/zend_vm_execute.h:60595
#18 0x000055ad3ddedecc in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3) at Zend/zend.c:1615
#19 0x000055ad3dd83f86 in php_execute_script (primary_file=<optimized out>) at main/main.c:2633
#20 0x000055ad3de88634 in main (argc=<optimized out>, argv=<optimized out>) at sapi/fpm/fpm/fpm_main.c:1945

Oops, wrong backtrace in the last piece. However, the first one led to the right place, or so it seems. I'll send a patch.

Hey Lauri,

OP here. Just want to say wow and thanks… big time! Your work is greatly appreciated! The other opcache bug I encountered seems already fixed in 7.3.1RC1. Looking fwd to the next patch releases!

last_message was changed in https://github.com/php/php-src/commit/a7305eb539596e175bd6c3ae9a20953358c5d677 to be allocated by Zend MM, with the comment that it's not supposed to be used in the next request. I'm not sure if the attached patch is right, or if some use of last_message (or lack of reset somewhere) is at fault here.

I think the problem is that the mysqlnd connection has end_psession and restart_psession handlers that are supposed to be called at the end/start of a request respectively. If end_psession were called, it would free the last_message and NULL it.

However, it seems that mysqli only calls restart_psession (which also nulls last_message, so probably prevents the worst) and PDO doesn't call either (resulting in what we see here).

@dmitry: What do you think about this? I think reverting to use persistent flag for last_message is probably the most pragmatic thing to do here. (There might be ABI concerns though...)

@nikic, thanks for pointing out those functions. That's exactly what I was looking for. I'll send another patch to use them in pdo_mysql. It works and seems like the right solution.

By the way, both functions are used in mysqli as well.

@lauri: Nice, I wasn't aware that PDO already had a handler we can use for this. I've merged your patch.

I took a snapshot from the 7.3 branch with the patch included and indeed, wp-supercache is working fine now. Thanks guys! 

However, I seem to have found another (not directly related) bug. I've reported it with at #77434.

–Christian