php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #37929 Simple multi-dimensional array uses tons of memory, gives none back to Apache2
Submitted: 2006-06-27 14:44 UTC Modified: 2006-07-27 01:33 UTC
Votes:19
Avg. Score:4.8 ± 0.4
Reproduced:19 of 19 (100.0%)
Same Version:13 (68.4%)
Same OS:15 (78.9%)
From: joe at estara dot com Assigned:
Status: Closed Package: Apache2 related
PHP Version: 5.1.4, 4.4.0 OS: Linux
Private report: No CVE-ID: None
Anyone can comment on a bug. Have a simpler test case? Does it work for you on a different platform? Let us know!
Just going to say 'Me too!'? Don't clutter the database with that please !
Your email address:
MUST BE VALID
Solve the problem:
45 - 34 = ?
Subscribe to this entry?

 
 [2006-06-27 14:44 UTC] joe at estara dot com
Description:
------------
If you use unserialize on a large multidimensional array, each apache process it runs in ends up taking up over 68m of ram (the actual size depends on your array that you're unserializing)   This is actual ram (Resident minus shared ram), and doesn't return it to the operating system after the script exits.  Since you have say MaxClients in apache2 of 150, 150 times 68m of ram means swapping to death.  This is using prefork.

Reproduce code:
---------------
<?php

ini_set("memory_limit", "64M");
# ziplatlong is an array, with zip code as a key and a 2 element lat long array as the value
$s = file_get_contents("ziplatlong"); 
$zip = unserialize($s);
preg_match("/foo/", $zip["00601"][0]);


Expected result:
----------------
I'd expect most of the ram returned to the operating system.  When I do something similar, use the same levels of ram, apache only takes up ~10M of ram, even though it uses ~64M while processing.

<?php

ini_set("memory_limit", "64M");
for($i=0;$i<50; $i++) 
  $s[$i] = file_get_contents("ziplatlong"); 
preg_match("/foo/", $s[38]);


Actual result:
--------------
Machine swaps to death.

Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2006-06-28 08:12 UTC] tony2001@php.net
PHP releases all the memory taken during the request and it's up to the memory manager used in your OS to decide when to start using it in other processes.
Not PHP problem.
 [2006-06-28 13:09 UTC] joe at estara dot com
Tony; 
Why then when I rewrite this to be exactly the same except for the unserialize does it give all the ram back?   I've tried this with 6 different methods to grab a bunch of ram, every other method returns the ram on exit.  unserialize is the only path with a memory leak.
 [2006-06-30 18:42 UTC] joe at estara dot com
This is not the operating systems fault. This is PHP's fault.  It's present in 4.4.0-4 as well. Under debian and fedora core 4.   

This leaks 800M of ram on a single run.

<?php

ini_set("memory_limit", "800M");
for($i=0;$i<160; $i++) 
  $s[$i] = file_get_contents("ziplatlong"); 
for($i=0;$i<160; $i++) 
   preg_match("/foo/", $s[$i]);

This uses 800m but immediately returns it to the operating system under both versions and OSs. 

<?php
ini_set("memory_limit", "800M");
for($i=0;$i<260; $i++)
  $s[$i] = file_get_contents("ziplatlong");
for($i=0;$i<260; $i++)
   preg_match("/foo/", $s[$i]);

I'll email you a zip file so you can easily reproduce this PHP bug.
 [2006-06-30 20:43 UTC] joe at estara dot com
So I rewrote the unserialize function in php, and it still is happening.   Totally bizare, the leak must be somewhere deeper in PHP.   If I make copies of the array using array_merge it doesn't use more memory, but if I unserialize again it uses more memory each time, even in an ALL php version.   Please email me if you want example data file.
 [2006-06-30 22:03 UTC] tony2001@php.net
Thank you for this bug report. To properly diagnose the problem, we
need a short but complete example script to be able to reproduce
this bug ourselves. 

A proper reproducing script starts with <?php and ends with ?>,
is max. 10-20 lines long and does not require any external 
resources such as databases, etc.

If possible, make the script source available online and provide
an URL to it here. Try to avoid embedding huge scripts into the report.


 [2006-06-30 23:02 UTC] joe at estara dot com
Download an example here:
http://zeus-1.estara.com/php_mem_leak.zip
 [2006-06-30 23:17 UTC] tony2001@php.net
Please provide _SHORT_ and complete reproduce script.

A proper reproducing script starts with <?php and ends with ?>, is max. 10-20 lines long and does not require any external resources such as databases, etc.
 [2006-07-01 00:00 UTC] joe at estara dot com
It's 8 lines without comments.  You want me to write a script to make the data file instead of just downloading it?
<?php
ini_set("memory_limit", "64M");
$s = file_get_contents("ziplatlong");
$z = unserialize($s);
for($i=0;$i<10;$i++)
  $zip[$i] = unserialize($s);
for($i=0;$i<10;$i++)
   preg_match("/asdfl/", $zip[$i]["00601"][0]);
?>
 [2006-07-01 00:15 UTC] joe at estara dot com
Here, fully contained version that will make a 131M leak in 12 lines. 
<?php
ini_set("memory_limit", "64M");
function make_arr() {
   for($i=0; $i<396768; $i++)
        $a[sprintf("%05d",$i)] = array(0 => "PHPLEAKS923889239823", 1 => "PHPLEAK2349082349898");
   return $a;
}
for($i=0;$i<10;$i++)
  $zip[$i] = make_arr();
for($i=0;$i<10;$i++)
   preg_match("/asdfl/", $zip[$i]["00601"][0]);
?>
 [2006-07-06 18:27 UTC] joe at estara dot com
Arrays of arrays leak memory in many PHP versions.
 [2006-07-20 15:23 UTC] mike@php.net
Not enough information was provided for us to be able
to handle this bug. Please re-read the instructions at
http://bugs.php.net/how-to-report.php

If you can provide more information, feel free to add it
to this bug and change the status back to "Open".

Thank you for your interest in PHP.


Any evidence like valgrind etc?

The OS releases memory when it needs it.
 [2006-07-20 15:38 UTC] joe at estara dot com
Mike:  

Please just run this below script on your machine, and then use top, hit 'M' to sort by memory and apache will be the top memory user.  hit the script 10 times and your machine will start swapping to death.

This appears to be apache related: if the same script is hit, it doesn't leak MORE memory, it stays at the outrageously high level it got to the first time. 

<?php
ini_set("memory_limit", "64M");
function make_arr() {
   for($i=0; $i<396768; $i++)
        $a[sprintf("%05d",$i)] = array(0 => "PHPLEAKS923889239823", 1 =>
"PHPLEAK2349082349898");
   return $a;
}
for($i=0;$i<10;$i++)
  $zip[$i] = make_arr();
for($i=0;$i<10;$i++)
   preg_match("/asdfl/", $zip[$i]["00601"][0]);
?>
 [2006-07-20 15:40 UTC] joe at estara dot com
Meant to say if the same apache process is hit, it won't grow higher, but every apache process that is hit grows to the high level of this script (`21M, which will swap the machine to death eventually). 

It's only this data structure that leaks, if I change it slightly it doesn't leak, so I'm guessing it's the garbage collector.
 [2006-07-20 15:49 UTC] tony2001@php.net
Please try using this CVS snapshot:

  http://snaps.php.net/php5.2-latest.tar.gz
 
For Windows:
 
  http://snaps.php.net/win32/php5.2-win32-latest.zip


 [2006-07-20 17:58 UTC] joe at estara dot com
Fixed in 5.2.0-Dev (though half the time it asks me if I want to download a blank file which I guess is different bug when it's near the memory limit?)  It allocates more memory in this version but returns it all to the OS immediately on exit:


<?php

ini_set("memory_limit", "384M");

function make_arr() {
   for($i=0; $i<39678; $i++)
        $a[sprintf("%05d",$i)] = array(0=> "PHPLEAKS923889239823", 1 => "PHPLEAK2349082349898");
   return $a;
}

for($i=0;$i<15;$i++)
  $zip[$i] = make_arr();
for($i=0;$i<15;$i++)
   preg_match("/asdfl/", $zip[$i]["00601"][0]);

echo 'Script Ran<br/><br/>';

?>
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Tue Dec 03 15:01:32 2024 UTC