I have the same segfault,
PHP 5.5.19 Os Debian

GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/php...Reading symbols from /usr/lib/debug/usr/bin/php5...done.
done.
[New LWP 26995]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `php test.php'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000000006dfe18 in zend_hash_quick_find (ht=0x7fe72b0396b0, arKey=arKey@entry=0x7fe734bdeb00 "isdirty", nKeyLength=nKeyLength@entry=8, h=7572512272371981, pData=0x7fff6c1bb7e8) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_hash.c:950
950	/usr/src/php5.5/source/dotdeb-php5/Zend/zend_hash.c: No such file or directory.
(gdb) bt
#0  0x00000000006dfe18 in zend_hash_quick_find (ht=0x7fe72b0396b0, arKey=arKey@entry=0x7fe734bdeb00 "isdirty", nKeyLength=nKeyLength@entry=8, h=7572512272371981, pData=0x7fff6c1bb7e8) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_hash.c:950
#1  0x00000000006f88f9 in zend_std_get_method (object_ptr=<optimized out>, method_name=0x7fe734bdeab0 "isDirty", method_len=7, key=0x7fe72afefc70) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_object_handlers.c:1027
#2  0x0000000000708511 in ZEND_INIT_METHOD_CALL_SPEC_VAR_CONST_HANDLER (execute_data=0x7fe734c5e530) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_vm_execute.h:15346
#3  0x0000000000740308 in execute_ex (execute_data=0x7fe734c5e530) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_vm_execute.h:363
#4  0x00000000006c0add in dtrace_execute_ex (execute_data=<optimized out>) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_dtrace.c:73
#5  0x0000000000780866 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fe734c5e128) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_vm_execute.h:584
#6  0x0000000000740308 in execute_ex (execute_data=0x7fe734c5e128) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_vm_execute.h:363
#7  0x00000000006c0add in dtrace_execute_ex (execute_data=<optimized out>) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_dtrace.c:73
#8  0x00000000006c2df1 in zend_call_function (fci=fci@entry=0x7fff6c1bbc70, fci_cache=0x0, fci_cache@entry=0x7fff6c1bbc40) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_execute_API.c:937
#9  0x00000000006e86c5 in zend_call_method (object_pp=object_pp@entry=0x7fff6c1bbd28, obj_ce=<optimized out>, fn_proxy=fn_proxy@entry=0x7fff6c1bbd20, function_name=function_name@entry=0xb35eb0 "__destruct", function_name_len=function_name_len@entry=10, 
    retval_ptr_ptr=retval_ptr_ptr@entry=0x0, param_count=param_count@entry=0, arg1=arg1@entry=0x0, arg2=arg2@entry=0x0) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_interfaces.c:97
#10 0x00000000006f3b12 in zend_objects_destroy_object (object=0x7fe72af714c8, handle=<optimized out>) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_objects.c:123
#11 0x00000000006f9b50 in zend_objects_store_del_ref_by_handle_ex (handle=40, handlers=<optimized out>) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_objects_API.c:212
#12 0x00000000006f9b93 in zend_objects_store_del_ref (zobject=0x2a2d188) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_objects_API.c:178
#13 0x00000000006c0e20 in _zval_dtor (zvalue=0x2a2d188) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_variables.h:35
#14 i_zval_ptr_dtor (zval_ptr=0x2a2d188) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_execute.h:81
#15 _zval_ptr_dtor (zval_ptr=<optimized out>) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_execute_API.c:426
#16 0x00000000006f3c07 in zend_object_std_dtor (object=0x2c76de8) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_objects.c:54
#17 0x00000000006f3c39 in zend_objects_free_object_storage (object=0x2c76de8) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_objects.c:137
#18 0x00000000006f96c6 in zend_objects_store_free_object_storage (objects=objects@entry=0xea65e0) at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_objects_API.c:97
#19 0x00000000006c1473 in shutdown_executor () at /usr/src/php5.5/source/dotdeb-php5/Zend/zend_execute_API.c:293
#20 0x00000000006d0e85 in zend_deactivate () at /usr/src/php5.5/source/dotdeb-php5/Zend/zend.c:949
#21 0x000000000066f13a in php_request_shutdown (dummy=dummy@entry=0x0) at /usr/src/php5.5/source/dotdeb-php5/main/main.c:1808
#22 0x0000000000782b88 in do_cli (argc=2, argv=0x24af1b0) at /usr/src/php5.5/source/dotdeb-php5/sapi/cli/php_cli.c:1177
#23 0x000000000043236f in main (argc=2, argv=0x24af1b0) at /usr/src/php5.5/source/dotdeb-php5/sapi/cli/php_cli.c:1378

I think this is related to HTTP pipelining and Apache version 2.4

To test it, do:
echo -e "GET /test.php HTTP/1.1\nHost: localhost\n\nGET /test.php HTTP/1.1\nHost: localhost\n\n"|nc localhost 80

It does not matter what is in test.php, it can be empty.
Note that it does not always segfault. But if test.php outputs something, only the first output is delivered, empty on the second one.

I've tested this on php versions 5.4.36, 5.5.1, 5.5.20 and 5.6.4 with version 2.2.29, 2.4.2 and 2.4.10 of apache.
Tested with clean compile of both apache and php (on ubuntu 14.04).
php configure: './configure' '--prefix=/tmp/testing/php5' '--with-apxs2=/tmp/testing/apache2/bin/apxs' '--disable-all'

Segfaults for all tested version of php for apache 2.4, but never for 2.2.29. 
Also, 2.2.29 outputs all pipelined requests.


Now, I did some digging and I guess the problem is in php_handler function in sapi_apache2.c:
https://github.com/php/php-src/blob/PHP-5.5.20/sapi/apache2handler/sapi_apache2.c#L536

The server_context is reused (parent_req is found), which ends up with a corrupt zend_file_handle zfd (in line 669).
Note that php_server_context_cleanup is never called.
I assume the server_context should NOT be used again for these kind of requests.


Hopefully this helps.
I am getting a lot of segfaults these days. Seems that HTTP pipelining is enabled for safari on iOS devices.
I am also seeing some "PHP Fatal Error Allowed memory size of 268435456 bytes exhausted (tried to allocate 22455254144 bytes)". This definitely relates to this problem, and probably something that can happen if there is no segfault.

Happy to find this bug.... Ran into the same problems when trying to upgrade from an older openSUSE system with Apache 2.2, to a current one with Apache 2.4. PHP in both cases, is a self-built 5.6.6 (also tested with 5.6.7RC1 and an older 5.6.4) build.

As soon as I tried this in production, I got, on a box with about 2000 req/min, 2-3 coredumps per minute. Trials with opcache disabled, pointed to exactly the kind of backtraces reported here. Trials with opcache enabled (normal production) almost always resulted in backtraces like this:

#0  0x00007fc2c1c0a347 in i_create_execute_data_from_op_array (
    nested=<optimized out>, op_array=<optimized out>)
    at /usr/src/phb/build/release-5.6.7-Og/php-src/Zend/zend_execute.c:1676
1676			EX(prev_execute_data) = EG(current_execute_data);
(gdb) bt
#0  0x00007fc2c1c0a347 in i_create_execute_data_from_op_array (
    nested=<optimized out>, op_array=<optimized out>)
    at /usr/src/phb/build/release-5.6.7-Og/php-src/Zend/zend_execute.c:1676
#1  zend_execute (op_array=0x7fc2c06c10f8)
    at /usr/src/phb/build/release-5.6.7-Og/php-src/Zend/zend_vm_execute.h:388
#2  0x00007fc2c1b78bf3 in zend_execute_scripts (type=type@entry=2, 
    retval=retval@entry=0x0, file_count=file_count@entry=1)
    at /usr/src/phb/build/release-5.6.7-Og/php-src/Zend/zend.c:1341
#3  0x00007fc2c1c0d672 in php_handler (r=<optimized out>)
    at /usr/src/phb/build/release-5.6.7-Og/php-src/sapi/apache2handler/sapi_apache2.c:669
.....

I can fully reproduce the issue now on my development system, using the echo/netcat double request from the last comment against a trivial (only one echo) test script.

The following patch, against the 5.6.7RC1 sources, fixes the coredump, and results in both pipelined test script calls returning the expected result. I have not yet tested whether the patch has negative consequences, though. Here it is:

diff --git a/sapi/apache2handler/sapi_apache2.c b/sapi/apache2handler/sapi_apache2.c
index 088ff77..0f80aee 100644
--- a/sapi/apache2handler/sapi_apache2.c
+++ b/sapi/apache2handler/sapi_apache2.c
@@ -549,7 +549,7 @@ static int php_handler(request_rec *r)
 
 	/* apply_config() needs r in some cases, so allocate server_context early */
 	ctx = SG(server_context);
-	if (ctx == NULL || (ctx && ctx->request_processed && !strcmp(r->protocol, "INCLUDED"))) {
+	if (1 || ctx == NULL || (ctx && ctx->request_processed && !strcmp(r->protocol, "INCLUDED"))) {
 normal:
 		ctx = SG(server_context) = apr_pcalloc(r->pool, sizeof(*ctx));
 		/* register a cleanup so we clear out the SG(server_context)

Just ran a single step session (xdebug + opcache disabled), first breakpoint in apache SAPI php_handler line 669 (call to zend_execute_scripts, triggering on the second pipelined request), then second breakpoint in compile_file() like 584 where the call

zend_stack_push(&CG(context_stack), (void *) &CG(context), sizeof(CG(context)));

bombed. Stepping into zend_stack_push I find:

(gdb) step
zend_stack_push (stack=stack@entry=0x7fbd1b595478 <compiler_globals+696>, element=element@entry=0x7fbd1b595450 <compiler_globals+656>, size=size@entry=40)
    at /usr/src/phb/build/release-5.6.7-Og/php-src/Zend/zend_stack.c:34
34	{
(gdb) step
35		if (stack->top >= stack->max) {		/* we need to allocate more memory */
(gdb) print *stack
$1 = {top = 0, max = 64, elements = 0x0}

So the code thinks it does not need to allocate - and then bombs on the following assignment to stack->elements[stack->top].

Looks like CG(context_stack).max is not properly reset somehow after the previous request...

After more debugging, and comparing with my apache 2.2 setup, the root cause of the problem is pretty clear: Apache 2.4 no longer calls the pool cleanup php_server_context_cleanup() between the two requests on the pipelined connection. This makes the second, independant request run with the previous SG(server_context), like a subrequest - but the first request has already shut down the interpreter...

With apache 2.2 there is a proper call to php_server_context_cleanup() after each request.

I'm still unsure what would be the correct fix; the logic in php_handler() covers several cases, not all clear to me. Maybe somebody can clear this up: in what situation will we enter php_handler with non-NULL ctx, ctx->request_processed == 1, and r->protocol == "INCLUDED" ?

To make sure the issue is not with the somewhat old Apache 2.4.6 of openSUSE 13.1, I have now compiled a fresh Apache 2.4.12 (current release), and the problem still persists.

I also added some trace logging to the apache2handler code. Using the already described two-request echo|netcat test, I see this with a segfaulting request:

[Fri Mar 13 15:31:13.034800 2015] [:error] [pid 22003] php_handler(ctx=0, ctx->r=0, r=3f425b0) request_processed=-1
[Fri Mar 13 15:31:13.034818 2015] [:error] [pid 22003] php_handler(NEW ctx=3f47b80, ctx->r=3f425b0, r=3f425b0)
[Fri Mar 13 15:31:13.034841 2015] [:error] [pid 22003] php_apache_request_ctor(SG(server_context)=3f47b80, ctx=3f47b80, ctx->r=3f425b0, r=3f425b0)
[Fri Mar 13 15:31:13.037026 2015] [:error] [pid 22003] php_apache_request_dtor(ctx=3f47b80, ctx->r=3f425b0, r=3f425b0)
[Fri Mar 13 15:31:13.045265 2015] [:error] [pid 22003] php_handler(ctx=3f47b80, ctx->r=3f425b0, r=3f54b00) request_processed=1
[Fri Mar 13 15:31:13.045372 2015] [:error] [pid 22003] php_handler(OLD ctx=3f47b80, parent_req=3f425b0, r=3f54b00)
[Fri Mar 13 15:31:13.045407 2015] [:error] [pid 22003] php_handler(ctx=3f47b80, parent_req=3f425b0, r=3f54b00) PROCESSING SUBREQUEST

Compare this to the following trace, done with the patch I sent in [2015-03-10 09:22 UTC] applied:

[Fri Mar 13 15:44:26.381174 2015] [:error] [pid 19155] php_handler(ctx=0, ctx->r=0, r=4f7fb30) request_processed=-1
[Fri Mar 13 15:44:26.381197 2015] [:error] [pid 19155] php_handler(NEW ctx=4f830f0, ctx->r=4f7fb30, r=4f7fb30)
[Fri Mar 13 15:44:26.381218 2015] [:error] [pid 19155] php_apache_request_ctor(SG(server_context)=4f830f0, ctx=4f830f0, ctx->r=4f7fb30, r=4f7fb30)
[Fri Mar 13 15:44:26.383660 2015] [:error] [pid 19155] php_apache_request_dtor(ctx=4f830f0, ctx->r=4f7fb30, r=4f7fb30)
[Fri Mar 13 15:44:26.391144 2015] [:error] [pid 19155] php_handler(ctx=4f830f0, ctx->r=4f7fb30, r=4f69300) request_processed=1
[Fri Mar 13 15:44:26.391192 2015] [:error] [pid 19155] php_handler(NEW ctx=4f8c3b0, ctx->r=4f69300, r=4f69300)
[Fri Mar 13 15:44:26.391222 2015] [:error] [pid 19155] php_apache_request_ctor(SG(server_context)=4f8c3b0, ctx=4f8c3b0, ctx->r=4f69300, r=4f69300)
[Fri Mar 13 15:44:26.392970 2015] [:error] [pid 19155] php_apache_request_dtor(ctx=4f8c3b0, ctx->r=4f69300, r=4f69300)
[Fri Mar 13 15:44:26.398950 2015] [:error] [pid 19155] php_server_context_cleanup(4f8c3b0)
[Fri Mar 13 15:44:26.399008 2015] [:error] [pid 19155] php_server_context_cleanup(0)

So it is clear that apache 2.4 delays the call to the pool cleanup function, php_server_context_cleanup.

Now, how to solve this... I would just rip out all the attempts to handle a parent_req with interpreter reentrancy. This will make the virtual() function fail when both parent and subrequest go to the PHP handler, but some testing even on apache 2.2 suggested to me that that is broken anyway, resulting in crashes sometimes.

However, the comments / core about POST errors and 413 and so on, makes me nervous, because I have no idea how that would be related, affected, or triggered. Some testing with post requests, also overly long ones, did not show me reentrancy, but the code must have had some reason in the past.

HELP!

For completeness, here's a gist with the patch against PHP 5.6.7RC1 that I created the second trace output in the previous comment with:

https://gist.github.com/anonymous/449a2726c876d187165b

In addition to the added logging, it contains the "1 ||" change at the start of php_handler() which makes the issue go away for me in the normal, non-subrequest cases.

Bug hunting:

I am trying to reopen this bug at the Apache httpd bugzilla:
https://bz.apache.org/bugzilla/show_bug.cgi?id=56984

Thanks to the mention of php_server_context_cleanup() by *php at bof dot de* I decided to look at the source code of the php sapi module and the Apache/APR code which causes it to be called.

The PHP sapi module uses the pool register/cleanup philosophy enabled by the APR module of Apache (@see http://www.apachetutor.org/dev/pools) and stores its SG(server_context) in the request pool, depending on it to be cleaned up at the end of the request, before the Apache worker feeds it a new request.

In Apache 2.2 the code in modules/http/http_core.c is fairly straightforward:

static int ap_process_http_connection(conn_rec *c)
[...]
    while ((r = ap_read_request(c)) != NULL) {
[...]
        if (r->status == HTTP_OK)
            ap_process_request(r);

        if (ap_extended_status)
            ap_increment_counts(c->sbh, r);

        if (c->keepalive != AP_CONN_KEEPALIVE || c->aborted)
            break;

        ap_update_child_status(c->sbh, SERVER_BUSY_KEEPALIVE, r);
        apr_pool_destroy(r->pool);
[...]
    }

So before handling the next request the pool is destroyed and ergo the php_server_context_cleanup() gets called because it is registered to the pools cleanup list.

In Apache 2.4 things have been rewritten quite extensively and in a way that is completely breaking the primary functionality of the APR pools, in that they should provide an easy declarative cleanup where a client registers a cleanup and can count on the server to execute it at the moment the server sees the pool should be destroyed.

Apache 2.4.7 code (and 2.4.12)::
static int ap_process_http_sync_connection(conn_rec *c)
[...]
while ((r = ap_read_request(c)) != NULL) {

        c->keepalive = AP_CONN_UNKNOWN;
        /* process the request if it was read without error */

        ap_update_child_status(c->sbh, SERVER_BUSY_WRITE, r);
        if (r->status == HTTP_OK) {
            if (cs)
                cs->state = CONN_STATE_HANDLER;
            ap_process_request(r);
            /* After the call to ap_process_request, the
             * request pool will have been deleted.  We set
             * r=NULL here to ensure that any dereference
             * of r that might be added later in this function
             * will result in a segfault immediately instead
             * of nondeterministic failures later.
             */
            r = NULL;
        }
[...]
    }

So, there clearly is no pool cleanup anymore in this place. But surely, you say, it has just been moved to a place deeper in the stack?

Following the ap_process_request, ap_process_async_request chain, however crushes this hope slowly but surely.

There is still an attempt to let the pool be cleaned up after the request. After the handler (for example PHP) had its go at the request, the method  ap_process_request_after_handler(r) is called. (it's in modules/http/http_request.c)

The after_handler routine creates an EOR bucket and puts it in the output of the request connection. This bucket registers a eor_bucket_cleanup function to the request pool pre_cleanup list that would instruct the destruction of the request pool. However this routine can only be called by apr_pool_clear of apr_pool_destroy and the problem is exactly that as long as the connection with requests or responses doesn't end, then those methods are not called for this pool or a parent pool that might exist, at least not anywhere in the prefork worker that I could see.

So it seems that in Apache 2.4 at present the request pool has become a connection pool after all.

Apparently the problem is well masked by PHP opcaching, which has practically become the default. When the script gets cached it apparently doesn't matter that the server_context is reused (because it doesn't have to use the Zend_cache?). And also in many cases there might be multiple requests on one pipeline but just one of them would be a php script.

This could be a major problem for the PHP5 sapi module on current Apache 2.4 servers.

Here's a gist with a patch, against 5.6.7RC1, that appears to fix the issue for me, at least for the things I tested.

https://gist.github.com/bof/15173c7a11cb12a7b96f

(patch contains lots of extra debug output, none of the ap_log_error calls I added would be good for merging)

I basically rip out the whole parent_req handling attempts in php_handler(), and make ctx->request_processed be the only indicator regarding reentrancy to the interpreter. In case such a reentrancy is attempted, I error out early in php_handler() before any manipulation to interpreter state is done.

The PHP virtual() function (provided in php_functions.c) will now cleanly fail when used to "embed" another PHP script (the worst reentrancy), but it continues to work when embedding an URI handled by any other handler (the not_for_us case).

While testing I had a final issue with the ap_rflush() call in virtual() resulting in pool destruction of the in-flight request, due to the apache 2.4 EOR thing gmoniker mentioned in the previous comment. If anybody is curious, drop me an email, I've got the backtrace saved. For now I have disabled that ap_rflush() call in virtual(), and I don't know what could break by that.

Anyway, after some light testing with this gist's patch applied, my own codebase keeps working normally, and the double-echo test now works without crashes for any of these cases:

- two normal PHP requests in a row
- two PHP requests in a row, each using virtual to embed another PHP script (embedded script NOT run, PHP warning message in virtual() caller)
- two PHP requests in a row, using virtual to embed a non-PHP thing (/robots.txt tested...) - works as it should.

Reworked the patch another round, see updated gist:

https://gist.github.com/bof/15173c7a11cb12a7b96f

The pool cleanup logic is reworked, with the goal of never cleaning up the
SG(server_context) of something other than the currently running + not yet complete request - regardless of when and how delayed a pool cleanup callback comes in. This I do by 1) adding a ctx->server_context pointer to the context structure, 2) registering ctx instead of &SG(server_context) with the pool cleanup callback, and finally 3) in php_server_context_cleanup() I check that *ctx->server_context == ctx, and only in that case I NULL it.

I also rearranged the php_handler() config handling a bit, so that the xbithack-during-reentry case should now be correct, too; and there's some all-around cleanup.

The debug output of the previous patch is gone (available on request).

I tested this patch, lightly, with my testcases and our normal codebase, under both Apache 2.4.12, the openSUSE 13.1 Apache 2.4.6, and an older Apache 2.2 setup as I use it in normal production so far. No issues were found, the new logic seems to work the same everywhere. I also ran the apache 2.4.12 variant under valgrind, also without any issues flagged.

THIS STILL NEEDS REVIEW from somebody who previously worked on the apache2handler code. But I think it's solid enough now, and I'll probably test it in production on monday

Patrick,

somehow the latest patch is broken, it won't apply to any of 5.5 through master. Also, is there any piece of code to be used for the tests?

Thanks.

This is some great work.

One remark, the php_functions.c file of the apache2handler inside the virtual definition mentions that the ap_rflush is supposed to be a workaround for
http://issues.apache.org/bugzilla/show_bug.cgi?id=17629. Fortunately reading its comments seems to indicate that the underlying problem was solved in Apache 2.2 around august 2010 with https://svn.apache.org/viewvc?view=revision&revision=988400.

Sorry for the broken gist patches. Apparently gists do funny things with whitespace... I should have tested that, but didn't.

To make things easy I put the patches up on my personal webspace. Please don't link to that in any blog posts or stuff like that :)

Same patch as in the last gist, without and with added debug/trace calls:

http://bei.bof.de/php/sapi_apache2.noreentry.v2.patch
http://bei.bof.de/php/sapi_apache2.noreentry.debug.v2.patch

Since then I also added something alluded to on internals in a new thread "PHP apache2handler virtual() function" - now implemented with a new PHP function apache_tail_request(). This can be found, on top of the changes of the v2 patch, in these patches:

http://bei.bof.de/php/sapi_apache2.noreentry.v3.patch
http://bei.bof.de/php/sapi_apache2.noreentry.debug.v3.patch

All patches were made on top of PHP 5.6.7RC1, checked out by tag from git.

For about 2 hours now, I have the v3 patch actively running on one of my production servers, under apache 2.4 now, and without any coredumps / segvs in the logs or any kind of performance regression. This is with some 80 requests per second, so it gets a good workout. The patch appears to work.

ab@php.net : regarding reproducing the issue you can take any PHP script, a simple one-line echo is sufficient, and then request it twice using the echo|netcat command as shown in several of the comments here.

It seems to me that it will be hard to revalidate this handler if it is changed extensively.

Just did some testing with a one line adjustment to the code of "sapi/apache2handler/sapi_apache2.c".

Towards the end of php_handler(request_rec *r) I included the line:
                apr_pool_cleanup_run(r->pool, (void *)&SG(server_context), php_server_context_cleanup);

So it becomes:
        if (!parent_req) {
                php_apache_request_dtor(r TSRMLS_CC);
                ctx->request_processed = 1;
                bucket = apr_bucket_eos_create(r->connection->bucket_alloc);
                APR_BRIGADE_INSERT_TAIL(brigade, bucket);

                rv = ap_pass_brigade(r->output_filters, brigade);
                if (rv != APR_SUCCESS || r->connection->aborted) {
zend_first_try {
                        php_handle_aborted_connection();
} zend_end_try();
                }
                apr_brigade_cleanup(brigade);
                apr_pool_cleanup_run(r->pool, (void *)&SG(server_context), php_server_context_cleanup);
        } else {
                ctx->r = parent_req;
        }

It seems to me this solves the missing automatic request pool destruction after a request in Apache 2.4 leading the php_handler routine to consider independent follow-on requests a subrequest of the first request when they are not. The request pool can continue on, but without the php_struct in it. I am afraid the Apache 2.4 handling of these pipelines can cause memory bloat of the server process, but that belongs in Apaches corner.

I applied http://bei.bof.de/php/sapi_apache2.noreentry.v3.patch on my Centos6 development server with Apache 2.4.12 with PHP 5.6.6 as mod_php. Result: the segfaults did not happen anymore, so the patch seems to be OK.

Limesurvey, Drupal7 and Piwik ran fine after applying the patch. Especially Piwik is worth mentioning because it seems to stress PHP to the max. After patch v3 it produced an occasional 'zend_mm_heap corrupted' (like it always does), but no segfault.

Re:  [2015-03-17 23:22 UTC] gmoniker at gmail dot com

Jep, I think your one line fix to the end of php_handler(), fixes the immediate problem here. Initial testing confirms. I'll test in production later today.

I didn't realize before (but now confirmed in apache sources), that apr_pool_cleanup_run() _cancels_ an exact same previously set cleanup callback.

Reentry by virtual() still dumps core (even with ap_rflush removed there), as probably will the other funny reentry cases. But as far as I can see, all those can only be triggered by suitable PHP code, and not at will by external requests - so they are better handled separately in new bug reports.

Production test of the one-liner patch, using Apache 2.4, ran flawlessly, now for five hours straight.

Just to be sure it's good, I'll run a second build under Apache 2.2 + put that on one of my not-yet-upgraded production servers tomorrow.

But I think this patch is good to go in. I added it now as a patch to this bug report, for that reason.

@bof, Thanks for your hard work!

I still don't know exactly when the first request pool is destroyed by the EOR bucket. It might be when the output of the next request begins to run. Anyway this is a very necessary patch for using Apache 2.4.

Everything is running smoothly with the one-line patch attached yesterday (thanks again gmoniker!). That is on two different (same load) production boxes, one with Apache 2.2 and one with Apache 2.4. No segfaults, no CPU usage or memory usage or latency issues (I've got them covered pretty thoroughly in Nagios).

I think this is ready to go into the next 5.5.x, 5.6.x, and master.

I applied the 1 line patch at https://bugs.php.net/patch-display.php?bug_id=68486&patch=sapi_apache2.gmoniker.patch&revision=latest to my Centos6, Apache 2.4.12 with PHP 5.6.6 as mod_php and the segfaults vanished as well.

The same one-liner fixed segfaults in Apache 2.4.12 64-bit on Windows 2008 R2 with PHP 5.6.7 64-bit as mod_php. I could make it segfault from a remote Linux server...

I have been looking at the situations where the SAPI apache2 handler can enter the PHP handling with a PHP context set. These are:

(Apache 2.4 only)
Client connection does not drop and a previous PHP request has not had its complete output flushed by a new request on the same client connection with sufficient output to bust the buffer.

(Apache 2.2 and 2.4)
1. Running "inside" an original PHP handled client request.
   a. When entering a virtual() of the request
   b. Abnormal ending of the initial request PHP handling
      AND Apache has fired an ErrorDocument set to a PHP artefact.
2. Running inside an SSI template
   a. A virtual() of an initial PHP script call
   b. Subsequent request to a PHP script or its subrequests

BUT there may be no active SAPI yet:
This happens when an SSI includes several PHP scripts in series.
Each time the processing returns to the host SSI template itself
the SAPI is deactivated, but on entering the next PHP script call
a context remains visible that was allocated by the last PHP script.

In all cases except the specific Apache 2.4 case the SAPI will be activated before using it. But for a 413 ErrorDocument it is tried to reuse the previous SAPI instance.

Entering a virtual that has a compile error will segfault.

Using the oneline patch solves the Apache 2.4 specific case, it shortens the path of code that SSI templates run through when they have sequential PHP scripts, and it stops the handler from trying to reuse the previous PHP instance in case of a 413 handler in PHP responding to a PHP script that Apache throws a 413 error for. (Setting 413 status yourself does not call the 413 ErrorDocument).

But some more rewriting of the handler and the Virtual function is needed.

Hello All,

I have decided to try and rigorously cleanup the logic of the handler because it seems to have gotten out of hand actually. The new patch can be installed on 5.6.7. You can also get the code for sapi_apache2.c from https://github.com/gmoniker/php-apache2handler (the other files there are not changed.)

The previous code has a goto. While goto's can be benificial for your code, they can also be a spaghetti monster in the making. This goto is of the latter category. And I am glad to say that after I worked through the code paths, it became clear that it could be removed along with a lot of other lines.

Now I only did some serious testing on Apache 2.4 so far. It may be that this does not work with Apache 2.2.

This solves two problems. One is the segfaults on Apache 2.4 with pipelined requests, and the other is the problem with segfaults on 413 ErrorDocuments in PHP in both Apache 2.2 and 2.4, where the initial request is also to PHP. This can be triggered by a LimitRequestBody and a PHP ErrorDocument which tries to introspect the symbol table, by doing for example:
if (!defined('FOO')) define('BAR',1);

To pipeline requests you can use this bash script:
for var in "${@}"; do
        printf -- "GET /%s HTTP/1.1\nHost: localhost\n\n" "$var"
done | nc localhost 80

gmoniker
Gol Gol

Hello All,

Too check for any differences in output between the present SAPI handler and a handler with the second streamlining patch I made, I defined some baseline tests and executed them against the new and old on both Apache 2.2 and Apache 2.4. I used the 5.6.7 patched version unchanged in 5.3.10, the handler did not evolve in the meantime apart from quieting a compiler warning and setting a format specifier by macro.

These tests show the output of the old handler and the new simplified handler to be the same :-)

You can lookup the tests at https://github.com/gmoniker/php-apache2handler

I enabled the includes module, and added a few lines to a standard config to get a Request Body limit and 413 PHP errorhandler. There is also something interesting going on with Apache 2.4 if you include an over the limit request body on the very first request to a worker. I leave this as an exercise for the reader...

See also https://bugs.php.net/bug.php?id=69218

I have been looking into this for quite a bit as there are some interesting comments and was also checking the changes in https://github.com/gmoniker/php-apache2handler which I have got some notes for and will try to action at least some of it. I think there are some good ideas but need still to get a bit better understanding of httpd internals to be confident in merging / updating them.

I tested the mentioned flows for the virtual to see if there is still any segfault but haven't seen any of it. It should be noted that quite a few things changed in the core since this was reported. For example compile error is no longer bail outs (it's non catchable exception instead) so to get bailout I was trying to get over memory limit. The only issue that I noticed is when virtual is over the limit and then the parent gets over the limit as well. Then there is double bailout that ends the data transfer which doesn't look correct (I will create an issue for that). Otherwise I didn't manage to recreate any segfault so if anyone can still see how to segfault, please open a new GitHub issue.

As the main issue is fixed now (even though it's not probably optimal fix), I will close this. The rest will be handled separately.

Just for the record, I create the issue for the mentioned bug with double bail out: https://github.com/php/php-src/issues/17509