php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Sec Bug #73807 Performance problem with processing post request over 2000000 chars
Submitted: 2016-12-23 12:52 UTC Modified: 2017-06-20 07:27 UTC
Votes:6
Avg. Score:3.7 ± 1.4
Reproduced:5 of 5 (100.0%)
Same Version:5 (100.0%)
Same OS:5 (100.0%)
From: pparadowski at media4u dot pl Assigned: stas
Status: Closed Package: Performance problem
PHP Version: 5.6.29, 7.0,7.1 OS: FreeBSD 9.4-11.0
Private report: No CVE-ID: needed
View Add Comment Developer Edit
Welcome! If you don't have a Git account, you can't do anything here.
You can add a comment by following this link or if you reported this bug, you can edit this bug over here.
(description)
Block user comment
Status: Assign to:
Package:
Bug Type:
Summary:
From: pparadowski at media4u dot pl
New email:
PHP Version: OS:

 

 [2016-12-23 12:52 UTC] pparadowski at media4u dot pl
Description:
------------
	 	 	
Since php 5.6.x an issue with data sending using POST method is noticeable only on FreeBSD servers. When we try to send long text post with over 2000000 characters to FreeBSD server we got a long running request

With the number of characters the processing time substantially increases as can be seen below in the tests that we carried out:

with 2000000 characters the time is as follows:
FreeBSD - 2,5s
Ubuntu/Linux - 64ms

with 10000000 characters the time is as follows:
FreeBSD - 50,71s
Ubuntu/Linux - 100ms

Please note that we used Apache 2.4 and PHP with the same configurations on the servers.

The problem was not present in the previous PHP 5.4 on FreeBSD.


Configure Command

	'./configure' '--with-layout=GNU' '--localstatedir=/var' '--with-config-file-scan-dir=/usr/local/etc/php' '--disable-all' '--enable-libxml' '--enable-mysqlnd' '--with-libxml-dir=/usr/local' '--with-pcre-regex=/usr/local' '--with-zlib-dir=/usr' '--program-prefix=' '--disable-cli' '--disable-cgi' '--with-apxs2=/usr/local/sbin/apxs' '--with-regex=php' '--with-zend-vm=CALL' '--disable-ipv6' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/info/' '--build=amd64-portbld-freebsd9.3' 'build_alias=amd64-portbld-freebsd9.3' 'CC=cc' 'CFLAGS=-O2 '-pipe' '-fstack-protector' '-fno-strict-aliasing'' 'LDFLAGS= '-fstack-protector'' 'LIBS=-lpthread' 'CPPFLAGS=' 'CPP=cpp' 'CXX=c++' 'CXXFLAGS=-O2 '-pipe' '-fstack-protector' '-fno-strict-aliasing''

Loaded Modules
core mod_so http_core mod_authn_file mod_authn_dbm mod_authn_anon mod_authn_core mod_authz_host mod_authz_groupfile mod_authz_user mod_authz_dbm mod_authz_owner mod_authz_core mod_access_compat mod_auth_basic mod_auth_digest mod_file_cache mod_cache mod_cache_disk mod_reqtimeout mod_include mod_filter mod_charset_lite mod_deflate mod_proxy_html mod_mime mod_log_config mod_logio mod_env mod_mime_magic mod_cern_meta mod_expires mod_headers mod_usertrack mod_unique_id mod_setenvif mod_version mod_proxy mod_proxy_http mod_ssl prefork mod_unixd mod_status mod_autoindex mod_asis mod_info mod_cgi mod_vhost_alias mod_negotiation mod_dir mod_imagemap mod_actions mod_speling mod_userdir mod_alias mod_rewrite mod_php5

PHP.ini changes:
post_max_size	256M	256M



Test script:
---------------
<?php
if (!empty($_POST)) {
	if (isset($_POST['data'])) {
		echo "OK!";
		die();
	} else {
		echo "NOPE!";
		die();
	}
}
$imagedata = file_get_contents("1.csv");
$base64 = base64_encode($imagedata);
?>

<form method="post">
	<input type="hidden" name="data" value="<?php echo substr($base64, 0, 10000000); ?>">
	<button>SEND</button>
</form>


Patches

Add a Patch

Pull Requests

Pull requests:

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2017-01-20 11:03 UTC] pparadowski at media4u dot pl
-PHP Version: 5.6.29 +PHP Version: 5.6.29, 7.0
 [2017-01-20 11:03 UTC] pparadowski at media4u dot pl
same on PHP 7.0
 [2017-01-25 21:42 UTC] t dot knaur at media4u dot pl
Also, this issue causes high cpu usage so in specific situation may result in denial of service.

Anyone could help us?
 [2017-01-30 10:58 UTC] pparadowski at media4u dot pl
-PHP Version: 5.6.29, 7.0 +PHP Version: 5.6.29, 7.0,7.1
 [2017-01-30 10:58 UTC] pparadowski at media4u dot pl
same problem on php 7.1.
 [2017-01-30 18:20 UTC] rasmus@php.net
-Status: Open +Status: Analyzed
 [2017-01-30 18:20 UTC] rasmus@php.net
This sounds like an interesting bug. Since it happens on FreeBSD and not on Linux it might be related to FreeBSD's accf_http "httpready" feature which buffers incoming HTTP requests before triggering the accept. Could you have a look at your /boot/loader.conf file and try disabling accf_http_load if it is enabled there and see if that makes a difference? Note that it triggers a kernel module to be loaded, so you will need to reboot, or manually "kldunload accf_http"
 [2017-01-31 06:53 UTC] pparadowski at media4u dot pl
my /boot/loader.conf is empty. In defaults loader.conf I've got

accf_data_load="NO"
accf_nds_load="NO"
accf_http_load="NO"
 [2017-01-31 07:00 UTC] pparadowski at media4u dot pl
While testing accf_http module is disabled
 [2017-01-31 07:12 UTC] rasmus@php.net
Ok, so it isn't accf_http doing it. It still feels like something along those lines. Like a socket buffer being too small and the resulting context switching causing it to slow down. Perhaps try playing around with pmcstat or dtrace and see if you can get a picture of what one of these slow requests is doing compared to a smaller fast one.
 [2017-01-31 10:43 UTC] nikic@php.net
Another cause might be FreeBSD's *extremely* slow realloc() implementation. I remember this causing performance issues in other places.
 [2017-02-02 11:50 UTC] pstef at freebsd dot org
This has nothing to do with realloc(). php_std_post_handler() spends 92% in memchr() and 7% in memmove().
 [2017-02-02 13:29 UTC] nikic@php.net
@pstef: Thanks! That was just a wild guess, based on usage of smart_str API, which was previously reported to be slow on FreeBSD due to slow realloc(). (Thinking again that can't be, as it's used with non-persistent strings and as such uses our own allocator anyway.)

I just took a look at the php_std_post_handler() code: https://github.com/php/php-src/blob/7cba31535cbf24c0b8a24ae094afd9ed670435b0/main/php_variables.c#L339

I think the problem is that, in the case where POST form-urlencoded key-value pairs are significantly larger than the stream buffer size, the current implementation will keep appending to the buffer and rescan it in its entirety every time. This results in quadratic time complexity. (It does not rescan parts where it already detected a complete key-value pair, only if there's an unfinished one.)

This is probably caused by this commit: https://github.com/php/php-src/commit/2438490addfbfba51e12246a74588b2382caa08a#diff-28ccb3aa37e01a68f5510ad6de4ab738L231 Looks like this changed the code from buffering everything upfront to using a stream. As such, this issue could not occur previously.

What leaves me stumped here is why this issue would only occur on FreeBSD, rather than on all platforms.
 [2017-02-02 14:39 UTC] nikic@php.net
Just tried this on Ubuntu. I measured using a stop-watch (REQUEST_TIME_FLOAT does not seem to give a useful value in cli-server), so rough numbers are:

8M: ~0s
16M: ~2s
32M: ~7s
64M: ~25s

Clearly the increase isn't linear, so this isn't just a FreeBSD problem. It might be that FreeBSD suffers more because it uses a smaller BUFSIZ.
 [2017-02-02 14:42 UTC] nikic@php.net
-Status: Analyzed +Status: Assigned -Assigned To: +Assigned To: nikic
 [2017-02-02 15:11 UTC] nikic@php.net
-Type: Bug +Type: Security -Private report: No +Private report: Yes
 [2017-02-02 15:11 UTC] nikic@php.net
PR up at https://github.com/php/php-src/pull/2362.

This time with more accurate measurements (end-to-end, request from PHP)...

Before:
 8M:  0.2s
16M:  1.0s
32M:  4.5s
64M: 18.4s

After:
 8M: 0.04s
16M: 0.06s
32M: 0.12s
64M: 0.24s

Due to DOS potential on servers with high post_max_size, marking as security issue.
 [2017-02-03 07:10 UTC] pparadowski at media4u dot pl
Thank You very much. Could You tell me if this fix will be on php 5.6 too?
 [2017-02-06 14:53 UTC] nikic@php.net
-Assigned To: nikic +Assigned To: stas
 [2017-02-06 14:53 UTC] nikic@php.net
As this bug is a potential DOS vector, I believe PHP 5.6 should be fixed as well. Assigning to @stas for confirmation. (The patch should be exactly the same as for PHP 7.)
 [2017-02-09 06:19 UTC] stas@php.net
-Assigned To: stas +Assigned To: nikic
 [2017-02-09 06:19 UTC] stas@php.net
Sure, let's merge it into 5.6 too. Not sure why it's private - the fix is public and merged.
 [2017-02-09 13:51 UTC] pparadowski at media4u dot pl
We had network and system administrators ran series of tests with the patch (https://github.com/php/php-src/pull/2362) using php, php7.0, 7.1 i 5.6  specifically under FreeBSD system. 
Alas, those tests did not yield positive results and the processing time for the POST method was still very long.Then it is quite right to assume that the  problem could lie elsewhere.
 [2017-02-09 15:07 UTC] nikic@php.net
Thanks for testing. Would it be possible for you to provide a profile of where most of the time is spent? On Linux this is done using perf record + perf report, I'm not familiar with that the equivalent tool in FreeBSD would be. @pstef indicated that 90% of the time is spent in memchr(), which is the issue that the patch is supposed to solve, but maybe there is another problem here.
 [2017-02-22 12:40 UTC] pstef at freebsd dot org
The change definitely helped. I tested PHP 5.4, 5.6, 7.0 (git 570a), and 7.0 (git a15b) installed from source. For a ~5 MB file it took php in server mode this much time (in seconds) to respond:
5.4: 0.4
5.6: 10.9
570a: 10.9
a15b: 1.9.

1.9 seconds is still roughly 4 times more than what it was in 5.4. I saw that most of the php's CPU time was spent in memmove(), so I followed the suggestion that it might have something to do with BUFSIZ. Under FreeBSD BUFSIZ is still 1 KB, while the Linux distribution I use (and probably every other Linux distro) has it at 8 KB. When I changed SAPI_POST_HANDLER_BUFSIZ to 8*1024, the time went further down to ~0.3 seconds.

BUFSIZ shouldn't be used here. It's the size of stdio.h buffers, for use with functions like setbuf(). php_variables.c as of PHP 5.4 didn't depend on either, later it was changed to use BUFSIZ but not setbuf().
 [2017-02-25 11:43 UTC] nikic@php.net
I just played around with different buffer sizes on Ubuntu. For a 64M payload buffer sizes of 16 and 16K execute in 0.40s and 0.25s respectively. So the buffer size is not the underlying issue, it just exacerbates quadratic behavior.

Going by your observation that a lot of time is now spent in memmove(), the culprit is probably this memmove(): https://github.com/php/php-src/blob/7cba31535cbf24c0b8a24ae094afd9ed670435b0/main/php_variables.c#L329 In the case of a long POST variable, this will perform a memmove() with src==dst.

Looking at the FreeBSD implementation of bcopy(), the variant short-circuits if src==dst: https://svnweb.freebsd.org/base/head/lib/libc/string/bcopy.c?view=markup#l76 However, the AMD64 implementation does not: https://svnweb.freebsd.org/base/head/lib/libc/amd64/string/bcopy.S?view=markup

As such, we end up doing a quadratic memmove() here. We can avoid this by explicitly checking for src==dst on our side.
 [2017-02-25 12:02 UTC] nikic@php.net
Just added the additional check in https://github.com/php/php-src/commit/bbfa1b64192fe3ccd2898e6e1f997387b4f210d1. Hopefully this will fix the issue on FreeBSD completely.
 [2017-03-03 12:49 UTC] pparadowski at media4u dot pl
@nikic after applying your amendments to php 5.6 the problem appears to be solved and with no BUFSIZ altered. Thank you very much for the involvement in solving this issue. Are you able to share the information on when the official release with this fix of php 5.6 and 7 is scheduled?
 [2017-03-09 20:26 UTC] nikic@php.net
@stas: Can you please apply https://gist.github.com/nikic/da218d8fbbefceda946850395bfe5f16 to PHP-5.6? I don't have commit access to that branch.
 [2017-03-15 12:05 UTC] killing at multiplay dot co dot uk
Confirmed fixed for us by applying to 5.6:
https://gist.github.com/nikic/da218d8fbbefceda946850395bfe5f16

It would be good to see this merged into 5.6 as its causes quite serious a problem, in our case a 30M POST data without the patch php_std_post_handler takes 6 mins 52 seconds and with the patch it takes just 316 ms

During this time the CPU is 100% and as its before any script its definitely an easy DOS vector.

Dtrace script for testing:
pid$target:php-fpm:php_std_post_handler:entry
{
        self->ts = timestamp
}

pid$target:php-fpm:php_std_post_handler:return
/self->ts != 0/
{
        printf("php_std_post_handler() %d ms\n", (timestamp - self->ts) / 1000000);
}
 [2017-03-15 15:23 UTC] rasmus@php.net
-Assigned To: nikic +Assigned To: stas
 [2017-05-12 06:51 UTC] pparadowski at media4u dot pl
Hi.
There is still no added patch to php5.6. Is there any chance to add this patch in the near future?
 [2017-06-20 07:24 UTC] stas@php.net
Automatic comment on behalf of nikita.ppv@gmail.com
Revision: http://git.php.net/?p=php-src.git;a=commit;h=0f8cf3b8497dc45c010c44ed9e96518e11e19fc3
Log: Fix bug #73807
 [2017-06-20 07:24 UTC] stas@php.net
-Status: Assigned +Status: Closed
 [2017-06-20 07:27 UTC] stas@php.net
-CVE-ID: +CVE-ID: needed
 
PHP Copyright © 2001-2017 The PHP Group
All rights reserved.
Last updated: Fri Jun 23 10:01:44 2017 UTC