php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #77308 Unbuffered queries memory leak
Submitted: 2018-12-17 09:16 UTC Modified: 2019-11-07 08:26 UTC
Votes:2
Avg. Score:4.5 ± 0.5
Reproduced:2 of 2 (100.0%)
Same Version:2 (100.0%)
Same OS:1 (50.0%)
From: sjon at hortensius dot net Assigned: dmitry (profile)
Status: Closed Package: MySQL related
PHP Version: 7.3.0 OS: archlinux
Private report: No CVE-ID: None
Welcome back! If you're the original bug submitter, here's where you can edit the bug or add additional notes.
If you forgot your password, you can retrieve your password here.
Password:
Status:
Package:
Bug Type:
Summary:
From: sjon at hortensius dot net
New email:
PHP Version: OS:

 

 [2018-12-17 09:16 UTC] sjon at hortensius dot net
Description:
------------
commit f1f1f6 broke unbuffered queries in mysqlnd; there is a memory leak causing php to run out of memory

Test script:
---------------
<?php
ini_set('memory_limit', '8M');

$pdo = new PDO('mysql:host=*;dbname=*', '*', '*');
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);

$result = $pdo->query("SELECT * FROM large_table");

foreach ($result as $i => $row)
	if ($i % 10000 === 0)
		printf("%10s %10s\n", $i, memory_get_usage());


Expected result:
----------------
reasonable constant memory usage, eg:

         0     427648
     10000     427936
     20000     427968
     30000     427968
     40000     427936


Actual result:
--------------
         0     427296
     10000    2000288
     20000    3622336
     30000    5260736
     40000    6915488

Fatal error: Allowed memory size of 8388608 bytes exhausted (tried to allocate 16384 bytes) in /php on line 10


Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2018-12-28 21:06 UTC] cweiske@php.net
This breaks Roundcube webmail, see https://bugs.archlinux.org/task/61139.
Happens here with 7.3.0-2 (Debian 9)
 [2019-01-02 10:58 UTC] nikic@php.net
Test script including (slow) table generation:

<?php
ini_set('memory_limit', '8M');

$pdo = new PDO('mysql:host=127.0.0.1;dbname=test', 'php-test', 'password');
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);

$pdo->exec("DROP TABLE IF EXISTS large_table");
$pdo->exec("CREATE TABLE large_table (id int)");
for ($j = 0; $j < 100; $j++) {
    $query = "INSERT INTO large_table (id) VALUES ";
    for ($i = 0; $i < 10000; $i++) {
        $query .= "($i), ";
    }
    $query .= "($i)";
    $pdo->exec($query);
}

$result = $pdo->query("SELECT * FROM large_table");

foreach ($result as $i => $row)
    if ($i % 10000 === 0)
        printf("%10s %10s\n", $i, memory_get_usage());

I'm also getting a

Warning: Packets out of order. Expected 231 received 48. Packet size=3422212 in Unknown on line 0

after the memory limit error.
 [2019-01-02 11:14 UTC] nikic@php.net
-Assigned To: +Assigned To: dmitry
 [2019-01-02 11:14 UTC] nikic@php.net
@dmity: Probably related to the mysqlnd arena improvements.

Massif trace:

97.88% (10,719,359B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->91.19% (9,987,510B) 0x953317: __zend_malloc (zend_alloc.c:2904)
| ->76.79% (8,409,934B) 0x9525C8: _emalloc (zend_alloc.c:2494)
| | ->73.05% (8,000,688B) 0x89116A: _mysqlnd_emalloc (mysqlnd_alloc.c:99)
| | | ->72.90% (7,984,000B) 0x8D3CDD: mysqlnd_arena_alloc (mysqlnd_block_alloc.c:66)
| | | | ->72.90% (7,984,000B) 0x8D430D: mysqlnd_mempool_get_chunk (mysqlnd_block_alloc.c:148)
| | | |   ->72.90% (7,984,000B) 0x8A119A: php_mysqlnd_read_row_ex (mysqlnd_wireprotocol.c:1384)
| | | |     ->72.90% (7,984,000B) 0x8A358B: php_mysqlnd_rowp_read (mysqlnd_wireprotocol.c:1724)
| | | |       ->72.90% (7,984,000B) 0x8BF0CE: mysqlnd_mysqlnd_result_unbuffered_fetch_row_c_pub (mysqlnd_result.c:677)
| | | |         ->72.90% (7,984,000B) 0x8C5755: mysqlnd_mysqlnd_res_fetch_row_c_pub (mysqlnd_result.c:1757)
| | | |           ->72.90% (7,984,000B) 0x65C177: pdo_mysql_stmt_fetch (mysql_statement.c:660)
| | | |             ->72.90% (7,984,000B) 0x64853F: do_fetch_common (pdo_stmt.c:676)
| | | |               ->72.90% (7,984,000B) 0x648BF1: do_fetch (pdo_stmt.c:831)
| | | |                 ->72.90% (7,984,000B) 0x64F1BF: pdo_stmt_iter_move_forwards (pdo_stmt.c:2447)
| | | |                   ->72.90% (7,984,000B) 0xA214AA: ZEND_FE_FETCH_R_SPEC_VAR_HANDLER (zend_vm_execute.h:21484)
| | | |                     ->72.90% (7,984,000B) 0xA624CD: execute_ex (zend_vm_execute.h:57707)
| | | |                       ->72.90% (7,984,000B) 0xA656DA: zend_execute (zend_vm_execute.h:60833)
| | | |                         ->72.90% (7,984,000B) 0x98AE19: zend_execute_scripts (zend.c:1568)
| | | |                           ->72.90% (7,984,000B) 0x8F0BE1: php_execute_script (main.c:2630)
| | | |                             ->72.90% (7,984,000B) 0xA6842F: do_cli (php_cli.c:997)
| | | |                               ->72.90% (7,984,000B) 0xA695A5: main (php_cli.c:1389)

Presumably freeing chunks from the arena doesn't work properly anymore.
 [2019-01-02 11:22 UTC] sjon at hortensius dot net
I've already bisected this as mentioned, caused by f1f1f6
 [2019-01-02 11:30 UTC] nikic@php.net
Uh sorry, that's what I get for not reading the description...

Link to the commit: https://github.com/php/php-src/commit/f1f1f63ce1a9f984595ce0f61e69169297df7164
 [2019-01-14 11:00 UTC] dmitry@php.net
Automatic comment on behalf of dmitry@zend.com
Revision: http://git.php.net/?p=php-src.git;a=commit;h=1a306cc9a1679a758c859731a78429429830a439
Log: Fixed bug #77308 (Unbuffered queries memory leak)
 [2019-01-14 11:00 UTC] dmitry@php.net
-Status: Assigned +Status: Closed
 [2019-03-06 09:28 UTC] alex at alex-at dot net
Still experiencing unbuffered query memory leak on 7.3.2. The test case is simple: obtaining 11 million rows with just fetch_assoc in a loop, no other code.
 [2019-03-06 09:34 UTC] nikic@php.net
@alex: If you are disabling emulated prepared statements, then you're likely hitting bug #77599, which is fixed in 7.3.3.
 [2019-03-18 11:57 UTC] alex at alex-at dot net
No, not running PDO, just new MySQLi class instantiation. Will check 7.3.3 and report though.
 [2019-03-18 12:38 UTC] alex at alex-at dot net
Still present in 7.3.3

Some details of test run:

---

1. Versions

#/opt/php71/bin/php -v
PHP 7.1.27 (cli) (built: Mar 18 2019 13:00:22) ( ZTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.1.0, Copyright (c) 1998-2018 Zend Technologies

# /opt/php72/bin/php -v
PHP 7.2.16 (cli) (built: Mar 18 2019 13:03:57) ( ZTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies

# /opt/php73/bin/php -v
PHP 7.3.3 (cli) (built: Mar 18 2019 13:07:58) ( ZTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.3, Copyright (c) 1998-2018 Zend Technologies

---

2. Top data for ~10 sec of script run

7.1: 112241 root      20   0  307892  13932   8396 R 100.0  0.2   0:11.82 /opt/php71/bin/php ./test.php

7.2: 111796 root      20   0  310480  14228   8460 R  99.7  0.2   0:10.41 /opt/php72/bin/php ./test.php

7.3: 110517 root      20   0 1148804 848480   8716 R  99.7 10.4   0:10.37 /opt/php73/bin/php ./test.php

You can easily see RES size growing from stable 14M on PHP 7.1/7.2 to 840M+ on PHP 7.3. The script fails with out of memory error eventually (Allowed memory size of 1073741824 bytes exhausted (tried to allocate 16384 bytes) in /config/scripts/test.php on line 10)

---

3. Script excerpt (it's ultimately simple, but the query returns 11+ million rows)

#!/opt/php73/bin/php
<?php

error_reporting(E_ALL);
ini_set('display_errors', 1);
ini_set('memory_limit', '1G');

$db = new MySQLi('<server>', '<login>', '<password>', 'radius');
$res = $db->query('SELECT `AcctSessionId` AS `sid`, `AcctUniqueId` AS `uid` FROM `radacct` ORDER BY `AcctSessionId`, `AcctUniqueId`', MYSQLI_USE_RESULT);
while (is_array($row = $res->fetch_assoc()));
$res->free();

---
 [2019-11-07 07:38 UTC] kinimodmeyer at gmail dot com
why is the state closed?
the problem still exists.
see alex of 2019-03-18 12:38 UTC
 [2019-11-07 08:26 UTC] sjon@php.net
@kinimodmeyer the problem as originally reported has been fixed. Unless the exact same reproduction script causes the exact same error, you are probably running into another bug that needs a separate bug-report.
 
PHP Copyright © 2001-2025 The PHP Group
All rights reserved.
Last updated: Thu Jan 30 03:01:31 2025 UTC