php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #65590 Apache segfaults and reports zend_mm_heap corrupted
Submitted: 2013-08-30 09:01 UTC Modified: 2013-09-11 21:39 UTC
Votes:64
Avg. Score:4.4 ± 0.9
Reproduced:59 of 59 (100.0%)
Same Version:17 (28.8%)
Same OS:13 (22.0%)
From: ole dot skudsvik at gmail dot com Assigned:
Status: Open Package: opcache
PHP Version: 5.4.19 OS: Linux, CentOS 6
Private report: No CVE-ID: None
Have you experienced this issue?
Rate the importance of this bug to you:

 [2013-08-30 09:01 UTC] ole dot skudsvik at gmail dot com
Description:
------------
We are experiencing regular Apache segfaults.

We are sadly not able to reproduce as this seems to happen randomly when apache 
have been running for a while. Neither can we relate the problem to any spesific 
PHP code.

Apache error.log:

[Wed Aug 28 13:00:50 2013] [notice] child pid 31638 exit signal Segmentation 
fault (11)
zend_mm_heap corrupted
zend_mm_heap corrupted
zend_mm_heap corrupted
zend_mm_heap corrupted
zend_mm_heap corrupted
zend_mm_heap corrupted
zend_mm_heap corrupted
zend_mm_heap corrupted
zend_mm_heap corrupted
zend_mm_heap corrupted
zend_mm_heap corrupted
zend_mm_heap corrupted
zend_mm_heap corrupted
[Wed Aug 28 13:43:58 2013] [notice] child pid 13750 exit signal Segmentation 
fault (11)


GDB Backtrace:

Core was generated by `/usr/sbin/httpd'.
Program terminated with signal 11, Segmentation fault.
#0  zend_mm_add_to_free_list (heap=<value optimized out>, 
mm_block=0x7f8e9aef8cc0) at /usr/src/debug/php-5.4.19/Zend/zend_alloc.c:748
748  				if (ZEND_MM_FREE_BLOCK_SIZE(prev) != size) {
(gdb) bt
#0  zend_mm_add_to_free_list (heap=<value optimized out>, 
mm_block=0x7f8e9aef8cc0) at /usr/src/debug/php-5.4.19/Zend/zend_alloc.c:748
#1  0x00007f8e8ed74412 in _zend_mm_free_int (heap=0x7f8e9a32d6a0, 
p=0x7f8e9aef8cd0) at /usr/src/debug/php-5.4.19/Zend/zend_alloc.c:2114
#2  0x00007f8e8eda6ad1 in zend_hash_destroy (ht=0x7f8e8f19ffd0) at 
/usr/src/debug/php-5.4.19/Zend/zend_hash.c:565
#3  0x00007f8e8ed8d173 in shutdown_executor () at /usr/src/debug/php-
5.4.19/Zend/zend_execute_API.c:322
#4  0x00007f8e8ed99e52 in zend_deactivate () at /usr/src/debug/php-
5.4.19/Zend/zend.c:938
#5  0x00007f8e8ed3c67c in php_request_shutdown (dummy=<value optimized out>) at 
/usr/src/debug/php-5.4.19/main/main.c:1803
#6  0x00007f8e8ee44037 in php_apache_request_dtor (r=0x7f8e9ac8d1a8) at 
/usr/src/debug/php-5.4.19/sapi/apache2handler/sapi_apache2.c:507
#7  php_handler (r=0x7f8e9ac8d1a8) at /usr/src/debug/php-
5.4.19/sapi/apache2handler/sapi_apache2.c:679
#8  0x00007f8e97ea0bb0 in ap_run_handler (r=0x7f8e9ac8d1a8) at 
/usr/src/debug/httpd-2.2.15/server/config.c:158
#9  0x00007f8e97ea446e in ap_invoke_handler (r=0x7f8e9ac8d1a8) at 
/usr/src/debug/httpd-2.2.15/server/config.c:376
#10 0x00007f8e97eafb30 in ap_process_request (r=0x7f8e9ac8d1a8) at 
/usr/src/debug/httpd-2.2.15/modules/http/http_request.c:282
#11 0x00007f8e97eac9a8 in ap_process_http_connection (c=0x7f8e9ac80c18) at 
/usr/src/debug/httpd-2.2.15/modules/http/http_core.c:190
#12 0x00007f8e97ea86b8 in ap_run_process_connection (c=0x7f8e9ac80c18) at 
/usr/src/debug/httpd-2.2.15/server/connection.c:43
#13 0x00007f8e97eb4977 in child_main (child_num_arg=<value optimized out>) at 
/usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:667
#14 0x00007f8e97eb4c8a in make_child (s=0x7f8e99ffe860, slot=6) at 
/usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:763
#15 0x00007f8e97eb590c in perform_idle_server_maintenance (_pconf=<value 
optimized out>, plog=<value optimized out>, s=<value optimized out>) at 
/usr/src/debug/httpd-2.2.15/server/mpm/prefork/prefork.c:898
#16 ap_mpm_run (_pconf=<value optimized out>, plog=<value optimized out>, 
s=<value optimized out>) at /usr/src/debug/httpd-
2.2.15/server/mpm/prefork/prefork.c:1102
#17 0x00007f8e97e8c900 in main (argc=1, argv=0x7fffb01ca148) at 
/usr/src/debug/httpd-2.2.15/server/main.c:760

A complete strace of the crash is available here: 
https://gist.github.com/olesku/6387821

Test script:
---------------
Currently not able to reproduce.


Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2013-09-09 10:24 UTC] ole dot skudsvik at gmail dot com
What i've found is that if you disable opcache.fast_shutdown in php.ini we do 
not experience any crashes.

The documentation of opcache.fast_shutdown states:

If enabled, a fast shutdown sequence is used for the accelerated code
The fast shutdown sequence doesn't free each allocated block, but lets
the Zend Engine Memory Manager do the work.

I've im also now able to reproduce the segfault by doing the following:

<?php
  opcache_reset(); 
  Zend\Mvc\Application::init(blah).run();
?>
What i think happen here is:

* We call opcache_reset() which triggers a free() on all opcache allocated 
resources.

* We start the Zend application.

* When the Zend application shuts down Zend tries to free the already free'ed 
memory since it's told to do so by the fast_shutdown flag.

Ofcourse Zend should check if the memory is already free'd before trying to free 
it, but it seems it does not ?
 [2013-09-11 21:39 UTC] aharvey@php.net
-Package: Apache2 related +Package: opcache
 [2013-10-16 10:40 UTC] arjen at react dot com
Looks like this also happens under php-fpm (5.5.4):

#0  0x000000000060dff7 in ?? ()
#1  0x000000000060e1fb in ?? ()
#2  0x0000000000642f2e in zend_hash_destroy ()
#3  0x0000000000625bc3 in ?? ()
#4  0x0000000000635412 in ?? ()
#5  0x00000000005d5c1d in php_request_shutdown ()
#6  0x0000000000426724 in ?? ()
#7  0x00007f406e548bc5 in __libc_start_main () from /usr/lib/libc.so.6
#8  0x0000000000427505 in _start ()
 [2013-12-13 09:28 UTC] raja dot govindharaj at bjss dot com
We are also experiencing the same seg fault and nasty error. The segfaults occurs when do graceful restart Apache. We use the same version of PHP (5.4.19), CentOS 6.4 and OpCache 7.0.2. It occurs always though disable fast shutdown option (opcache.fast_shutdown=0).

/httpd graceful - crashed Apache.
 [2013-12-13 16:25 UTC] raja dot govindharaj at bjss dot com
We used mistakenly two caches (opcache and apc) that cacused the apache crash when do graceful or restart. I have now disabled apc which solved the problem.
 [2015-05-15 09:23 UTC] nax_hh at hotmail dot com
I was able to reproduce this in php 5.6 with a bit of difficulty

# cat /etc/centos-release
CentOS release 6.6 (Final)

# php -v
PHP 5.6.8 (cli) (built: Apr 16 2015 20:00:59)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies
    with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2015, by Zend Technologies


script:
opcache_reset();

require_once __DIR__.'/../vendor/autoload.php'; # composer
$app = require_once __DIR__.'/../src/App.php'; # silex app
$app->run();

error.log:
zend_mm_heap corrupted
zend_mm_heap corrupted
zend_mm_heap corrupted
zend_mm_heap corrupted
zend_mm_heap corrupted
...

I normally get only that error.
If I launch an ab like:

ab command:
ab -r -n 1000 -c 2 http://localhost:8080/

I get a segfault pretty fast.
But with Apache recompiled to generate the dump I'm not able to reproduce it
 [2015-05-22 13:04 UTC] phofstetter at sensational dot ch
Reproducing this is actually quite easy: Just run any script with FPM and enabled opcache while constantly touching it in order to fill up opcache.

Once oom_restarts increments (so opcache tries to restart itself due to having used up all memory), FPM will start crashing and it will continue to do so until you restart it completely.

For us, opcache.fast_shutdown solves the problem completely.
 [2015-11-19 22:40 UTC] colin at mollenhour dot com
I am currently experiencing this same behavior on PHP 5.5.30 (ondrej/php5 PPA on Ubuntu 14.04) and the latest official Ubuntu 14.04 package (5.5.9+....). I do not have APC installed obviously but am using ZendOPcache (v7.0.6-dev and 7.0.4 respectively). It occurs with or without opcache.fast_shutdown.

Another server running the exact same code (load balanced) with PHP 5.4.45 (dotdeb.org package) does not exhibit this issue.

I disabled Zend OPcache in favor of XCache and the error disappeared.
 [2015-11-20 15:40 UTC] colin at mollenhour dot com
Sorry, I spoke too soon, setting opcache.enabled = 0 did not resolve the problem. It seems that since there is a large delay after restart before it happens that it might be related to Apache 2.4 MaxConnectionsPerChild which I have set to 2000.
 
PHP Copyright © 2001-2019 The PHP Group
All rights reserved.
Last updated: Fri Dec 13 23:01:23 2019 UTC