same on PHP 7.0

Also, this issue causes high cpu usage so in specific situation may result in denial of service.

Anyone could help us?

same problem on php 7.1.

This sounds like an interesting bug. Since it happens on FreeBSD and not on Linux it might be related to FreeBSD's accf_http "httpready" feature which buffers incoming HTTP requests before triggering the accept. Could you have a look at your /boot/loader.conf file and try disabling accf_http_load if it is enabled there and see if that makes a difference? Note that it triggers a kernel module to be loaded, so you will need to reboot, or manually "kldunload accf_http"

my /boot/loader.conf is empty. In defaults loader.conf I've got

accf_data_load="NO"
accf_nds_load="NO"
accf_http_load="NO"

While testing accf_http module is disabled

Ok, so it isn't accf_http doing it. It still feels like something along those lines. Like a socket buffer being too small and the resulting context switching causing it to slow down. Perhaps try playing around with pmcstat or dtrace and see if you can get a picture of what one of these slow requests is doing compared to a smaller fast one.

Another cause might be FreeBSD's *extremely* slow realloc() implementation. I remember this causing performance issues in other places.

This has nothing to do with realloc(). php_std_post_handler() spends 92% in memchr() and 7% in memmove().

@pstef: Thanks! That was just a wild guess, based on usage of smart_str API, which was previously reported to be slow on FreeBSD due to slow realloc(). (Thinking again that can't be, as it's used with non-persistent strings and as such uses our own allocator anyway.)

I just took a look at the php_std_post_handler() code: https://github.com/php/php-src/blob/7cba31535cbf24c0b8a24ae094afd9ed670435b0/main/php_variables.c#L339

I think the problem is that, in the case where POST form-urlencoded key-value pairs are significantly larger than the stream buffer size, the current implementation will keep appending to the buffer and rescan it in its entirety every time. This results in quadratic time complexity. (It does not rescan parts where it already detected a complete key-value pair, only if there's an unfinished one.)

This is probably caused by this commit: https://github.com/php/php-src/commit/2438490addfbfba51e12246a74588b2382caa08a#diff-28ccb3aa37e01a68f5510ad6de4ab738L231 Looks like this changed the code from buffering everything upfront to using a stream. As such, this issue could not occur previously.

What leaves me stumped here is why this issue would only occur on FreeBSD, rather than on all platforms.

Just tried this on Ubuntu. I measured using a stop-watch (REQUEST_TIME_FLOAT does not seem to give a useful value in cli-server), so rough numbers are:

8M: ~0s
16M: ~2s
32M: ~7s
64M: ~25s

Clearly the increase isn't linear, so this isn't just a FreeBSD problem. It might be that FreeBSD suffers more because it uses a smaller BUFSIZ.

PR up at https://github.com/php/php-src/pull/2362.

This time with more accurate measurements (end-to-end, request from PHP)...

Before:
 8M:  0.2s
16M:  1.0s
32M:  4.5s
64M: 18.4s

After:
 8M: 0.04s
16M: 0.06s
32M: 0.12s
64M: 0.24s

Due to DOS potential on servers with high post_max_size, marking as security issue.

Thank You very much. Could You tell me if this fix will be on php 5.6 too?

As this bug is a potential DOS vector, I believe PHP 5.6 should be fixed as well. Assigning to @stas for confirmation. (The patch should be exactly the same as for PHP 7.)

Sure, let's merge it into 5.6 too. Not sure why it's private - the fix is public and merged.

We had network and system administrators ran series of tests with the patch (https://github.com/php/php-src/pull/2362) using php, php7.0, 7.1 i 5.6  specifically under FreeBSD system. 
Alas, those tests did not yield positive results and the processing time for the POST method was still very long.Then it is quite right to assume that the  problem could lie elsewhere.

Thanks for testing. Would it be possible for you to provide a profile of where most of the time is spent? On Linux this is done using perf record + perf report, I'm not familiar with that the equivalent tool in FreeBSD would be. @pstef indicated that 90% of the time is spent in memchr(), which is the issue that the patch is supposed to solve, but maybe there is another problem here.

The change definitely helped. I tested PHP 5.4, 5.6, 7.0 (git 570a), and 7.0 (git a15b) installed from source. For a ~5 MB file it took php in server mode this much time (in seconds) to respond:
5.4: 0.4
5.6: 10.9
570a: 10.9
a15b: 1.9.

1.9 seconds is still roughly 4 times more than what it was in 5.4. I saw that most of the php's CPU time was spent in memmove(), so I followed the suggestion that it might have something to do with BUFSIZ. Under FreeBSD BUFSIZ is still 1 KB, while the Linux distribution I use (and probably every other Linux distro) has it at 8 KB. When I changed SAPI_POST_HANDLER_BUFSIZ to 8*1024, the time went further down to ~0.3 seconds.

BUFSIZ shouldn't be used here. It's the size of stdio.h buffers, for use with functions like setbuf(). php_variables.c as of PHP 5.4 didn't depend on either, later it was changed to use BUFSIZ but not setbuf().

I just played around with different buffer sizes on Ubuntu. For a 64M payload buffer sizes of 16 and 16K execute in 0.40s and 0.25s respectively. So the buffer size is not the underlying issue, it just exacerbates quadratic behavior.

Going by your observation that a lot of time is now spent in memmove(), the culprit is probably this memmove(): https://github.com/php/php-src/blob/7cba31535cbf24c0b8a24ae094afd9ed670435b0/main/php_variables.c#L329 In the case of a long POST variable, this will perform a memmove() with src==dst.

Looking at the FreeBSD implementation of bcopy(), the variant short-circuits if src==dst: https://svnweb.freebsd.org/base/head/lib/libc/string/bcopy.c?view=markup#l76 However, the AMD64 implementation does not: https://svnweb.freebsd.org/base/head/lib/libc/amd64/string/bcopy.S?view=markup

As such, we end up doing a quadratic memmove() here. We can avoid this by explicitly checking for src==dst on our side.

Just added the additional check in https://github.com/php/php-src/commit/bbfa1b64192fe3ccd2898e6e1f997387b4f210d1. Hopefully this will fix the issue on FreeBSD completely.

@nikic after applying your amendments to php 5.6 the problem appears to be solved and with no BUFSIZ altered. Thank you very much for the involvement in solving this issue. Are you able to share the information on when the official release with this fix of php 5.6 and 7 is scheduled?

@stas: Can you please apply https://gist.github.com/nikic/da218d8fbbefceda946850395bfe5f16 to PHP-5.6? I don't have commit access to that branch.

Confirmed fixed for us by applying to 5.6:
https://gist.github.com/nikic/da218d8fbbefceda946850395bfe5f16

It would be good to see this merged into 5.6 as its causes quite serious a problem, in our case a 30M POST data without the patch php_std_post_handler takes 6 mins 52 seconds and with the patch it takes just 316 ms

During this time the CPU is 100% and as its before any script its definitely an easy DOS vector.

Dtrace script for testing:
pid$target:php-fpm:php_std_post_handler:entry
{
        self->ts = timestamp
}

pid$target:php-fpm:php_std_post_handler:return
/self->ts != 0/
{
        printf("php_std_post_handler() %d ms\n", (timestamp - self->ts) / 1000000);
}

Hi.
There is still no added patch to php5.6. Is there any chance to add this patch in the near future?