php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #79646 Segmentation fault in garbage collector
Submitted: 2020-05-28 08:36 UTC Modified: 2020-11-18 17:09 UTC
Votes:3
Avg. Score:3.7 ± 0.9
Reproduced:1 of 1 (100.0%)
Same Version:1 (100.0%)
Same OS:1 (100.0%)
From: enumag at gmail dot com Assigned:
Status: Open Package: Scripting Engine problem
PHP Version: 7.4.7 OS: Alpine
Private report: No CVE-ID: None
 [2020-05-28 08:36 UTC] enumag at gmail dot com
Description:
------------
Recently my PHPUnit tests started failing with segfault occasionally on CI. It only started after PHP 7.4.6 was released so I'm guessing something was broken there. Unfortunately I'm unable to reproduce the segfault reliably so the best I was capable of doing it setting up my CI to run PHPUnit through GDB to get a stacktrace the next time it happened which is how I found that the segfault is related to garbage collection. Here is the trace:

Program received signal SIGSEGV, Segmentation fault.
0x000055a7fb5ecccd in zend_gc_collect_cycles ()
#0  0x000055a7fb5ecccd in zend_gc_collect_cycles ()
#1  0x000055a7fb5ebe37 in gc_possible_root_when_full ()
#2  0x000055a7fb645f71 in execute_ex ()
#3  0x000055a7fb5f1bb2 in zend_generator_resume ()
#4  0x000055a7fb5f2632 in zim_Generator_send ()
#5  0x000055a7fb64bcae in execute_ex ()
#6  0x000055a7fb5f1bb2 in zend_generator_resume ()
#7  0x000055a7fb5f2632 in zim_Generator_send ()
#8  0x000055a7fb64bcae in execute_ex ()
#9  0x000055a7fb5f1bb2 in zend_generator_resume ()
#10 0x000055a7fb5f2632 in zim_Generator_send ()
#11 0x000055a7fb64bcae in execute_ex ()
#12 0x000055a7fb64d0ed in zend_execute ()
#13 0x000055a7fb5c7744 in zend_execute_scripts ()
#14 0x000055a7fb569428 in php_execute_script ()
#15 0x000055a7fb64f175 in do_cli ()
#16 0x000055a7fb3166a7 in main ()




Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2020-05-28 08:46 UTC] bwoebi@php.net
Is this possibly related to #79600? Can you please try with PHP 7.4.7RC1 whether the issue is still present there?
 [2020-05-28 09:01 UTC] enumag at gmail dot com
@bwoebi I don't know. I'm using the php:7.4-alpine docker image. Is there one for 7.4.7RC1? If there is I can try but I won't be able to tell reliably if it got fixed since it only happens occasionally in the first place...

Btw. bwoebi, really sorry for going off-topic here but there are number of issues with ext-uv and/or amphp where I kinda need your help. Any chance you might find some time for them in June 2020?
 [2020-06-08 20:19 UTC] alexinbeijing at gmail dot com
I suggest you use setUp() and tearDown() to run gc_collect_cycles() before and after each test. See if that makes the segfault happen more reliably. Also print out the names of the tests as they run. See if it always segfaults on the same tests (or a few of them).

That would be a huge clue pointing to what is actually wrong.
 [2020-06-11 07:08 UTC] enumag at gmail dot com
Considering the fact that the tests always failed at the same percentage of completion I do think it is always the same test - I was unable to pinpoint which one though. Is there a way to print test name for each test when it starts? `--testdox` seem to only print it when the test is finished.

However gc_collect_cycles() didn't help in making the bug more reliable to happen. It's still completely random.
 [2020-06-11 09:09 UTC] enumag at gmail dot com
Okay, not only it always fail in the same test but also in the same spot while the test is executed. Also the spot is something with rather heavy usage of Generators so if bug #79600 can indeed cause it then this is most likely the reason. I'll wait for PHP 7.4.7 to see if the problem disappears or not.
 [2020-06-15 06:59 UTC] enumag at gmail dot com
Unfortunately the segfault persists even with PHP 7.4.7. What should I do now?
 [2020-06-15 07:47 UTC] nikic@php.net
-PHP Version: 7.4.6 +PHP Version: 7.4.7
 [2020-06-15 07:47 UTC] nikic@php.net
I can't really do anything here without a way to reproduce. GC issues are always tricky, and "something related to generator GC is broken" is not enough to go on :(

You can try running PHP under valgrind, using something like

    USE_ZEND_ALLOC=0 valgrind php ...args

optionally with --suppressions=php.supp from https://gist.github.com/nikic/8d404c6799a1532b0c10280f5e57a888 to reduce noise.

That might give us a better idea of the root cause (which may have nothing to do with GC itself).
 [2020-06-15 09:00 UTC] enumag at gmail dot com
Okay, I ran `USE_ZEND_ALLOC=0 valgrind --suppressions=php.supp php vendor/bin/phpunit 2> valgrind.txt` with the php.supp file you provided. Here is the result:

https://gist.github.com/enumag/42402fcc05d9a404656f50a51fb98d2b

Please let me know if it helps.
 [2020-06-15 09:04 UTC] nikic@php.net
Thanks! The first invalid read is:


==182== Invalid read of size 16
==182==    at 0xBDF97E1: ???
==182==    by 0xA36A9C7: ???
==182==    by 0xA36A9C7: ???
==182==    by 0xA36A9F2: ???
==182==    by 0x4A8312F: ???
==182==    by 0xA36A9C7: ???
==182==  Address 0xa36a9ef is 63 bytes inside a block of size 72 alloc'd
==182==    at 0x489F72A: malloc (vg_replace_malloc.c:309)
==182==    by 0x5D30B8: __zend_malloc (in /usr/local/bin/php)
==182==    by 0x53ACA6: ??? (in /usr/local/bin/php)
==182==    by 0x540AA6: ??? (in /usr/local/bin/php)
==182==    by 0x5411E6: ??? (in /usr/local/bin/php)
==182==    by 0x67BF73: execute_ex (in /usr/local/bin/php)
==182==    by 0x5ED108: zend_call_function (in /usr/local/bin/php)
==182==    by 0x617373: zend_call_method (in /usr/local/bin/php)
==182==    by 0x4E352A: ??? (in /usr/local/bin/php)
==182==    by 0x5ED026: zend_call_function (in /usr/local/bin/php)
==182==    by 0x617373: zend_call_method (in /usr/local/bin/php)
==182==    by 0x6178BA: zend_user_it_rewind (in /usr/local/bin/php)

Unfortunately most debug symbols are missing, so we don't actually see where the invalid read occurs :(

Is it possible to install some kind of -dbg / -dbgsym package for PHP on your system?
 [2020-06-15 09:15 UTC] enumag at gmail dot com
I can use https://github.com/wodby/base-php docker image for that. Did it before to get the gdb stacktrace. Let me try...
 [2020-06-15 12:30 UTC] enumag at gmail dot com
@nikic Check this one please.
https://gist.github.com/enumag/329cbe88c5e77c5cc5be74aafa330ba8
 [2020-06-15 12:31 UTC] enumag at gmail dot com
The command I used this time:
USE_ZEND_ALLOC=0 valgrind --tool=memcheck --num-callers=30 --log-file=valgrind.txt --suppressions=php.supp php vendor/bin/phpunit
 [2020-06-15 12:43 UTC] nikic@php.net
@enumag: Thanks! From the new trace it seems clear that those "Invalid read of size 16" are false positives. They are probably caused by a vectorized implementation of some string function intentionally reading past the end of strings in a way that it knows is safe.

Unfortunately that also means that the trace ends before it gets to anything useful. Valgrind suggests using --error-limit=no to avoid that.
 [2020-06-15 12:57 UTC] enumag at gmail dot com
@nikic Alright, here is another log with no error limit:
https://gist.github.com/enumag/a5bbfbb0cfe606b77148e8b83dfd4d41
 [2020-06-23 13:01 UTC] enumag at gmail dot com
Any news on this one @nikic?
 [2020-09-15 13:55 UTC] enumag at gmail dot com
What's needed in order to get this problem fixed?
 [2020-10-22 16:36 UTC] aa dot vasilenko at gmail dot com
I've stumbled upon the same problem. Problem is reproducible only with php 7.4 (alternatively tried 7.3, it worked), using phpunit 9.1. In our case it's debian though.

Works flawlessly without code coverage, segfaults with code coverage on (100% reproducible). Trying to collect more information now
 [2020-11-18 15:02 UTC] nikic@php.net
There is some chance that this issue is fixed by https://github.com/php/php-src/commit/99a8ec6e24d2788bb533ab5be8618a958adf24cf. At least this seems like a plausible cause, but it may also be unrelated.
 [2020-11-18 15:47 UTC] enumag at gmail dot com
Considering that my app is heavily relying on Generators it is very much possible. Are you gonna backport this into 7.4.13?
 [2020-11-18 17:09 UTC] nikic@php.net
@enumag: The 7.4.13 release is already in RC, so this change will be in 7.4.14.
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Fri Oct 04 21:01:27 2024 UTC