php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #79143 Memory leak with streams and HTTP protocol 1.1
Submitted: 2020-01-20 20:41 UTC Modified: 2020-01-23 14:26 UTC
Votes:4
Avg. Score:4.5 ± 0.9
Reproduced:4 of 4 (100.0%)
Same Version:3 (75.0%)
Same OS:4 (100.0%)
From: curtisfarnham at gmail dot com Assigned:
Status: Duplicate Package: Streams related
PHP Version: 7.4.1 OS: Ubuntu, Alpine, MacOS
Private report: No CVE-ID: None
 [2020-01-20 20:41 UTC] curtisfarnham at gmail dot com
Description:
------------
When using the "http" stream context and protocol 1.1, there is a substantial memory leak which begins after 2.00 MB of a stream has been downloaded.  It increases linearly with time and is roughly in proportion to the amount of data downloaded.

This bug seems to have been introduced in PHP 7.2 but is still there in 7.4.1.  I have experienced it on Alpine Linux, Ubuntu Linux, and MacOS/Homebrew.  Using Protocol 1.0 successfully avoids the leak. Strangely, it also doesn't occur with MacOS Catalina's PHP 7.3.9 binary.

This discussion began at https://github.com/guzzle/guzzle/issues/2555 .

Since this test script discards the contents of the stream, the memory usage should hold constant over time.  It's initially constant, but when it gets to a cumulative 2097152 bytes downloaded (2.00 MB), the bug begins to appear and memory starts increasing steadily.

Also, there are two stepwise increases in memory usage, each of exactly 2.00 MB, prior to when the script reaches the aforementioned threshold. The increases occur at different times depending on the specific environment.

(Credit goes to Tobias Nyholm and Konstantin Kopachev for helping troubleshoot this. Thanks!)

Test script:
---------------
<?php
$context = stream_context_create(['http' => [
    'protocol_version' => '1.1',
]]);
$stream = fopen('https://kpbs.streamguys1.com/?ck=1579203064304', 'r', null, $context);

$dl = 0;
while (!feof($stream)) {
    $chunk = fread($stream, 8192);
    $dl += strlen($chunk);

    echo "downloaded $dl, memory " . memory_get_usage(true) . "\n";
}


Expected result:
----------------
downloaded 8192, memory 2097152
downloaded 16384, memory 2097152
downloaded 24576, memory 2097152
...
downloaded 1114112, memory 2097152
downloaded 1122304, memory 2097152
downloaded 1130496, memory 2097152
downloaded 1138688, memory 2097152
...
downloaded 2080768, memory 2097152
downloaded 2088960, memory 2097152
downloaded 2097152, memory 2097152
downloaded 2105344, memory 2097152
downloaded 2113536, memory 2097152
downloaded 2121728, memory 2097152

Actual result:
--------------
downloaded 8192, memory 2097152
downloaded 16384, memory 2097152
downloaded 24576, memory 2097152
...
downloaded 1114112, memory 2097152
downloaded 1122304, memory 2097152
downloaded 1130496, memory 4194304
downloaded 1138688, memory 4194304
...
downloaded 2080768, memory 4194304
downloaded 2088960, memory 4194304
downloaded 2097152, memory 6295552
downloaded 2105344, memory 6303744
downloaded 2113536, memory 6311936
downloaded 2121728, memory 6320128

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2020-01-20 20:58 UTC] nikic@php.net
Relevant massif:

->08.21% (338,643B) 0x903803: __zend_realloc (zend_alloc.c:2994)
| ->08.15% (335,955B) 0x90278E: _realloc_custom (zend_alloc.c:2434)
| | ->08.12% (334,923B) 0x9028DE: _erealloc (zend_alloc.c:2556)
| | | ->08.08% (333,347B) 0x8BEE2C: _php_stream_fill_read_buffer (streams.c:597)
| | | | ->08.08% (333,347B) 0x8BF3CE: _php_stream_read (streams.c:706)
| | | |   ->08.08% (333,347B) 0x8BF548: php_stream_read_to_str (streams.c:752)
| | | |     ->08.08% (333,347B) 0x74E5D0: zif_fread (file.c:1810)
| | | |       ->08.08% (333,347B) 0x9B0767: ZEND_DO_ICALL_SPEC_RETVAL_USED_HANDLER (zend_vm_execute.h:1314)
| | | |         ->08.08% (333,347B) 0xA164B4: execute_ex (zend_vm_execute.h:53797)
| | | |           ->08.08% (333,347B) 0xA1A5F3: zend_execute (zend_vm_execute.h:57913)
| | | |             ->08.08% (333,347B) 0x93DA8B: zend_execute_scripts (zend.c:1665)
| | | |               ->08.08% (333,347B) 0x89F734: php_execute_script (main.c:2617)
| | | |                 ->08.08% (333,347B) 0xA1D1DD: do_cli (php_cli.c:961)
| | | |                   ->08.08% (333,347B) 0xA1E358: main (php_cli.c:1352)

Possible relevant difference is the dechunking stream filter. Indeed, setting "http" => "auto_decode" => true avoids the issue.
 [2020-01-20 20:58 UTC] nikic@php.net
Sorry, I meant "auto_decode" => false of course.
 [2020-01-23 14:26 UTC] nikic@php.net
-Status: Open +Status: Duplicate
 [2020-01-23 14:26 UTC] nikic@php.net
Looks like the root cause here was the same as for bug #78902, which is now fixed.
 [2020-01-23 17:53 UTC] curtisfarnham at gmail dot com
I've verified that this fixes it. Thanks so much!
 
PHP Copyright © 2001-2020 The PHP Group
All rights reserved.
Last updated: Wed Feb 26 15:01:29 2020 UTC