php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #77308 Unbuffered queries memory leak
Submitted: 2018-12-17 09:16 UTC Modified: 2019-03-06 09:34 UTC
Votes:2
Avg. Score:4.5 ± 0.5
Reproduced:2 of 2 (100.0%)
Same Version:2 (100.0%)
Same OS:1 (50.0%)
From: sjon at hortensius dot net Assigned: dmitry (profile)
Status: Closed Package: MySQL related
PHP Version: 7.3.0 OS: archlinux
Private report: No CVE-ID: None
View Add Comment Developer Edit
Welcome! If you don't have a Git account, you can't do anything here.
You can add a comment by following this link or if you reported this bug, you can edit this bug over here.
(description)
Block user comment
Status: Assign to:
Package:
Bug Type:
Summary:
From: sjon at hortensius dot net
New email:
PHP Version: OS:

 

 [2018-12-17 09:16 UTC] sjon at hortensius dot net
Description:
------------
commit f1f1f6 broke unbuffered queries in mysqlnd; there is a memory leak causing php to run out of memory

Test script:
---------------
<?php
ini_set('memory_limit', '8M');

$pdo = new PDO('mysql:host=*;dbname=*', '*', '*');
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);

$result = $pdo->query("SELECT * FROM large_table");

foreach ($result as $i => $row)
	if ($i % 10000 === 0)
		printf("%10s %10s\n", $i, memory_get_usage());


Expected result:
----------------
reasonable constant memory usage, eg:

         0     427648
     10000     427936
     20000     427968
     30000     427968
     40000     427936


Actual result:
--------------
         0     427296
     10000    2000288
     20000    3622336
     30000    5260736
     40000    6915488

Fatal error: Allowed memory size of 8388608 bytes exhausted (tried to allocate 16384 bytes) in /php on line 10


Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2018-12-28 21:06 UTC] cweiske@php.net
This breaks Roundcube webmail, see https://bugs.archlinux.org/task/61139.
Happens here with 7.3.0-2 (Debian 9)
 [2019-01-02 10:58 UTC] nikic@php.net
Test script including (slow) table generation:

<?php
ini_set('memory_limit', '8M');

$pdo = new PDO('mysql:host=127.0.0.1;dbname=test', 'php-test', 'password');
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);

$pdo->exec("DROP TABLE IF EXISTS large_table");
$pdo->exec("CREATE TABLE large_table (id int)");
for ($j = 0; $j < 100; $j++) {
    $query = "INSERT INTO large_table (id) VALUES ";
    for ($i = 0; $i < 10000; $i++) {
        $query .= "($i), ";
    }
    $query .= "($i)";
    $pdo->exec($query);
}

$result = $pdo->query("SELECT * FROM large_table");

foreach ($result as $i => $row)
    if ($i % 10000 === 0)
        printf("%10s %10s\n", $i, memory_get_usage());

I'm also getting a

Warning: Packets out of order. Expected 231 received 48. Packet size=3422212 in Unknown on line 0

after the memory limit error.
 [2019-01-02 11:14 UTC] nikic@php.net
-Assigned To: +Assigned To: dmitry
 [2019-01-02 11:14 UTC] nikic@php.net
@dmity: Probably related to the mysqlnd arena improvements.

Massif trace:

97.88% (10,719,359B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->91.19% (9,987,510B) 0x953317: __zend_malloc (zend_alloc.c:2904)
| ->76.79% (8,409,934B) 0x9525C8: _emalloc (zend_alloc.c:2494)
| | ->73.05% (8,000,688B) 0x89116A: _mysqlnd_emalloc (mysqlnd_alloc.c:99)
| | | ->72.90% (7,984,000B) 0x8D3CDD: mysqlnd_arena_alloc (mysqlnd_block_alloc.c:66)
| | | | ->72.90% (7,984,000B) 0x8D430D: mysqlnd_mempool_get_chunk (mysqlnd_block_alloc.c:148)
| | | |   ->72.90% (7,984,000B) 0x8A119A: php_mysqlnd_read_row_ex (mysqlnd_wireprotocol.c:1384)
| | | |     ->72.90% (7,984,000B) 0x8A358B: php_mysqlnd_rowp_read (mysqlnd_wireprotocol.c:1724)
| | | |       ->72.90% (7,984,000B) 0x8BF0CE: mysqlnd_mysqlnd_result_unbuffered_fetch_row_c_pub (mysqlnd_result.c:677)
| | | |         ->72.90% (7,984,000B) 0x8C5755: mysqlnd_mysqlnd_res_fetch_row_c_pub (mysqlnd_result.c:1757)
| | | |           ->72.90% (7,984,000B) 0x65C177: pdo_mysql_stmt_fetch (mysql_statement.c:660)
| | | |             ->72.90% (7,984,000B) 0x64853F: do_fetch_common (pdo_stmt.c:676)
| | | |               ->72.90% (7,984,000B) 0x648BF1: do_fetch (pdo_stmt.c:831)
| | | |                 ->72.90% (7,984,000B) 0x64F1BF: pdo_stmt_iter_move_forwards (pdo_stmt.c:2447)
| | | |                   ->72.90% (7,984,000B) 0xA214AA: ZEND_FE_FETCH_R_SPEC_VAR_HANDLER (zend_vm_execute.h:21484)
| | | |                     ->72.90% (7,984,000B) 0xA624CD: execute_ex (zend_vm_execute.h:57707)
| | | |                       ->72.90% (7,984,000B) 0xA656DA: zend_execute (zend_vm_execute.h:60833)
| | | |                         ->72.90% (7,984,000B) 0x98AE19: zend_execute_scripts (zend.c:1568)
| | | |                           ->72.90% (7,984,000B) 0x8F0BE1: php_execute_script (main.c:2630)
| | | |                             ->72.90% (7,984,000B) 0xA6842F: do_cli (php_cli.c:997)
| | | |                               ->72.90% (7,984,000B) 0xA695A5: main (php_cli.c:1389)

Presumably freeing chunks from the arena doesn't work properly anymore.
 [2019-01-02 11:22 UTC] sjon at hortensius dot net
I've already bisected this as mentioned, caused by f1f1f6
 [2019-01-02 11:30 UTC] nikic@php.net
Uh sorry, that's what I get for not reading the description...

Link to the commit: https://github.com/php/php-src/commit/f1f1f63ce1a9f984595ce0f61e69169297df7164
 [2019-01-14 11:00 UTC] dmitry@php.net
Automatic comment on behalf of dmitry@zend.com
Revision: http://git.php.net/?p=php-src.git;a=commit;h=1a306cc9a1679a758c859731a78429429830a439
Log: Fixed bug #77308 (Unbuffered queries memory leak)
 [2019-01-14 11:00 UTC] dmitry@php.net
-Status: Assigned +Status: Closed
 [2019-03-06 09:28 UTC] alex at alex-at dot net
Still experiencing unbuffered query memory leak on 7.3.2. The test case is simple: obtaining 11 million rows with just fetch_assoc in a loop, no other code.
 [2019-03-06 09:34 UTC] nikic@php.net
@alex: If you are disabling emulated prepared statements, then you're likely hitting bug #77599, which is fixed in 7.3.3.
 [2019-03-18 11:57 UTC] alex at alex-at dot net
No, not running PDO, just new MySQLi class instantiation. Will check 7.3.3 and report though.
 [2019-03-18 12:38 UTC] alex at alex-at dot net
Still present in 7.3.3

Some details of test run:

---

1. Versions

#/opt/php71/bin/php -v
PHP 7.1.27 (cli) (built: Mar 18 2019 13:00:22) ( ZTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2018 Zend Technologies

# /opt/php72/bin/php -v
PHP 7.2.16 (cli) (built: Mar 18 2019 13:03:57) ( ZTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies

# /opt/php73/bin/php -v
PHP 7.3.3 (cli) (built: Mar 18 2019 13:07:58) ( ZTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.3, Copyright (c) 1998-2018 Zend Technologies

---

2. Top data for ~10 sec of script run

7.1: 112241 root      20   0  307892  13932   8396 R 100.0  0.2   0:11.82 /opt/php71/bin/php ./test.php

7.2: 111796 root      20   0  310480  14228   8460 R  99.7  0.2   0:10.41 /opt/php72/bin/php ./test.php

7.3: 110517 root      20   0 1148804 848480   8716 R  99.7 10.4   0:10.37 /opt/php73/bin/php ./test.php

You can easily see RES size growing from stable 14M on PHP 7.1/7.2 to 840M+ on PHP 7.3. The script fails with out of memory error eventually (Allowed memory size of 1073741824 bytes exhausted (tried to allocate 16384 bytes) in /config/scripts/test.php on line 10)

---

3. Script excerpt (it's ultimately simple, but the query returns 11+ million rows)

#!/opt/php73/bin/php
<?php

error_reporting(E_ALL);
ini_set('display_errors', 1);
ini_set('memory_limit', '1G');

$db = new MySQLi('<server>', '<login>', '<password>', 'radius');
$res = $db->query('SELECT `AcctSessionId` AS `sid`, `AcctUniqueId` AS `uid` FROM `radacct` ORDER BY `AcctSessionId`, `AcctUniqueId`', MYSQLI_USE_RESULT);
while (is_array($row = $res->fetch_assoc()));
$res->free();

---
 
PHP Copyright © 2001-2019 The PHP Group
All rights reserved.
Last updated: Tue Jul 23 18:01:25 2019 UTC