php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #73183 Segfaults / zend_mm_heap corrupted
Submitted: 2016-09-27 12:13 UTC Modified: 2021-06-09 15:00 UTC
Votes:10
Avg. Score:4.5 ± 0.8
Reproduced:8 of 8 (100.0%)
Same Version:4 (50.0%)
Same OS:2 (25.0%)
From: schnederle at futureweb dot at Assigned: cmb (profile)
Status: Closed Package: Reproducible crash
PHP Version: 7.0.11 OS: Centos 7.2.1511
Private report: No CVE-ID: None
 [2016-09-27 12:13 UTC] schnederle at futureweb dot at
Description:
------------
Hello,

since we switched from PHP 5.3 to PHP 7.0.x (tried multiple Versions of PHP 7 - now we are on 7.0.11) we get random segfaults / zend_mm_heap corrupted with our CMS.

Unfortunately not predictable when they will occur or on which Script. (unfortunately never the same)
Sometimes a few Days no problems - sometimes 5 times a Day ...

Not 100% sure - but I believe that opcache could be the cause of the Problem - as a simple "opcache_reset();" is enough to get everything up & running again - no complete restart of Apache needed.

Error Log / Backtrace: http://temp.in.futureweb.at/php/error.log 

Hope someone can help me on this ...

Thank you
Andreas Schnederle-Wagner



Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2016-09-28 03:39 UTC] laruence@php.net
you may try to set opcache.protect_memory=1 (please note this will significantly slowdown the server).  and see if there is any unexpected write to shared memory
 [2016-09-28 08:01 UTC] schnederle at futureweb dot at
alright - I activated protect_memory and will report back.

Will I see Errors in the Logs when there are any unexpected writes to shared memory? (unfortunately there is not much documentation available what protect_memory excactly does)
 [2016-09-28 11:25 UTC] schnederle at futureweb dot at
again Segfaults with opcache.protect_memory=1:

[Wed Sep 28 13:22:25.387677 2016] [core:notice] [pid 25800] AH00052: child pid 4680 exit signal Segmentation fault (11)
...
[Wed Sep 28 13:22:46.415573 2016] [core:notice] [pid 25800] AH00052: child pid 4885 exit signal Segmentation fault (11)
 [2016-09-28 15:31 UTC] laruence@php.net
paste the backtrace out (coredump)
 [2016-09-29 08:46 UTC] schnederle at futureweb dot at
Please find BT here: http://temp.in.futureweb.at/php/core.30883.bt.txt
 [2016-09-29 12:52 UTC] schnederle at futureweb dot at
another segfault - BT:

http://temp.in.futureweb.at/php/core.23374.bt.txt
http://temp.in.futureweb.at/php/core.25625.bt.txt
 [2016-09-30 01:42 UTC] laruence@php.net
is this the coredump after you setting opcache.protect_memory?
 [2016-09-30 11:26 UTC] schnederle at futureweb dot at
They were done with opcache.protect_memory=0

This one is with opcache.protect_memory=1 
http://temp.in.futureweb.at/php/core.17683.bt.txt
 [2016-09-30 11:40 UTC] schnederle at futureweb dot at
In case it helps - here a "bt full" with opcache.protect_memory=1: http://temp.in.futureweb.at/php/core.17683.bt_full.txt
 [2016-09-30 19:40 UTC] schnederle at futureweb dot at
again segfault with opcache.protect_memory=1: 
- http://www.futureweb.at/userdata/115/temp/core.bt.29305
- http://www.futureweb.at/userdata/115/temp/core.bt.29468

bt full with opcache.protect_memory=1:
- http://www.futureweb.at/userdata/115/temp/core.bt.full.29305
 [2016-10-04 10:54 UTC] schnederle at futureweb dot at
Don't know if it helps ... some more Segfault BTs:

02.10.2016: http://temp.in.futureweb.at/php/core.25917.bt.txt
03.10.2016: http://temp.in.futureweb.at/php/core.20048.bt.txt
04.10.2016: http://temp.in.futureweb.at/php/core.27118.bt.txt
 [2016-10-20 12:43 UTC] cadonline at gmx dot net
We are facing similar Problems on our Servers - Did you already find a Solution Andreas?

thx
Markus
 [2016-10-28 08:30 UTC] schnederle at futureweb dot at
Unfortunately no News yet ... same Problems occur with 7.0.12
Still hoping for some News on this Topic from PHP Guys! ;-)
 [2016-11-23 08:58 UTC] schnederle at futureweb dot at
still no luck with 7.0.13!

Maybe this additional Information can help?

Followed by the Segfaults I often see those Kind of Errors within Logs:

----------------------------------
error_log-20161116:[Tue Nov 15 14:35:52.478805 2016] [:error] [pid 30926] [client 80.110.0.0:54073] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 140380662153216 bytes) in /var/www/path/html/templates/path/php/news_nuggets.php on line 218, referer: http://domain

----------------------------------

216 Alle
217 </div>
218 <div class="referenzen_cst_filter_right_entry<?php if(isset($_GET['filter']) && $_GET['filter'] == 2) { ?> active<?php } ?>" id="referenzen_entry_2" rel="2">
219 Tourismus
220 </div>
----------------------------------

As one can see - there is nothing which could take 127TB of Memory on Line 218 ... could it be that OpCache is trying to Output complete nonsense?
 [2016-12-01 16:23 UTC] schnederle at futureweb dot at
Some new Information - Today I set OPCache Log Level to Debug ... no Segfault so far - but some other weird Errors - maybe somehow related to the Segfaults Problem?

At first some of those:
< Thu Dec 1 11:44:34 2016 (12618): Warning Interned string buffer overflow
< Thu Dec 1 11:44:34 2016 (12618): Warning Interned string buffer overflow
< ...
< ...
< ...
< Thu Dec 1 11:44:34 2016 (12618): Warning Interned string buffer overflow
< Thu Dec 1 11:44:34 2016 (12618): Warning Interned string buffer overflow

Right after those Errors - some OOM Errors occured:
< [Thu Dec 01 11:45:52.556057 2016] [:error] [pid 11958] [client 80.110.0.0:50912] PHP Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 6935160699037703352 bytes) in /var/www/path/html/includes/path1/file.php on line 38, referer: http://www.domain.at/admin/menue/fil...111&id_usr=111

Line 38 of the File where OOM occured:
< if (!function_exists("get_galleries")) {

Did increase opcache.interned_strings_buffer from 8 to 16 now ... but there shouldn't be such OOM Troubles when reaching Limit I guess ... (not to speak about 'maybe' Segfaults ...)
 [2016-12-12 14:28 UTC] schnederle at futureweb dot at
Since I increased opcache.interned_strings_buffer from 8 to 16 there was no single SEGFAULT anymore ... guess the Problem must lie somewhere within Interned Strings Buffer Logic then?
 [2016-12-19 12:01 UTC] schnederle at futureweb dot at
I can confirm that raising opcache.interned_strings_buffer from 8 to 16 MB did solve that Segfault issues! ~3 Weeks without a single Segfault ...

Actual OPCache Stats - interned_strings_usage - more than the initial 8MB are used ...

----------------------------------------------------
Array
(
    [opcache_enabled] => 1
    [file_cache] => /var/www/ortsinfo/opcache
    [cache_full] => 
    [restart_pending] => 
    [restart_in_progress] => 
    [memory_usage] => Array
        (
            [used_memory] => 86297760
            [free_memory] => 47915104
            [wasted_memory] => 4864
            [current_wasted_percentage] => 0.0036239624023438
        )

    [interned_strings_usage] => Array
        (
            [buffer_size] => 16777216
            [used_memory] => 8840536
            [free_memory] => 7936680
            [number_of_strings] => 74538
        )

    [opcache_statistics] => Array
        (
            [num_cached_scripts] => 4397
            [num_cached_keys] => 12811
            [max_cached_keys] => 16229
            [hits] => 15118014
            [start_time] => 1482122407
            [last_restart_time] => 0
            [oom_restarts] => 0
            [hash_restarts] => 0
            [manual_restarts] => 0
            [misses] => 7
            [blacklist_misses] => 0
            [blacklist_miss_ratio] => 0
            [opcache_hit_rate] => 99.999953697643
        )

)
----------------------------------------------------

Andreas Schnederle-Wagner
 [2018-06-28 15:06 UTC] t dot motylewski at gmail dot com
similar issues:
https://bugs.php.net/bug.php?id=73598
https://bugs.php.net/bug.php?id=65590
https://bugs.php.net/bug.php?id=66569
 [2018-06-28 15:23 UTC] t dot motylewski at gmail dot com
and a few more:
https://bugs.php.net/bug.php?id=69251
https://bugs.php.net/bug.php?id=59191
 [2021-06-09 14:47 UTC] cmb@php.net
-Status: Open +Status: Feedback -Assigned To: +Assigned To: cmb
 [2021-06-09 14:47 UTC] cmb@php.net
Is this still an issue with any of the actively supported PHP
versions[1]?

[1] <https://www.php.net/supported-versions.php>
 [2021-06-09 14:57 UTC] schnederle at futureweb dot at
-Status: Feedback +Status: Assigned
 [2021-06-09 14:57 UTC] schnederle at futureweb dot at
@cmb - unfortunately I cannot test it on the affected system. The system is live & productive, I can't do a deliberately induced downtime here. Since the segfaults occurred sporadically and unfortunately I could not find a testcase with which I can trigger the segfault, unfortunately I currently have no way to test it.
The affected CMS is currently running on PHP 7.2.34, but with the option "opcache.interned_strings_buffer = 16" which also fixed the problem with V7.0.x ...
best regards
A.
 [2021-06-09 15:00 UTC] cmb@php.net
-Status: Assigned +Status: Closed
 [2021-06-09 15:00 UTC] cmb@php.net
The point is that prior to PHP 7.2.0, there have been known issues
regarding interned strings.  I assume that this issue has been
fixed, so I'm closing.  If you happen to reproduce it with any
actively supported PHP version, feel free to reopen or file a new
ticket.
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Tue Mar 19 04:01:31 2024 UTC