php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login

go to bug id or search bugs for

Bug #41053 Filling Array with integers : memory usage too high
Submitted: 2007-04-11 13:26 UTC Modified: 2007-04-11 20:24 UTC
Votes:24
Avg. Score:4.2 ± 1.0
Reproduced:16 of 18 (88.9%)
Same Version:9 (56.2%)
Same OS:10 (62.5%)
From: matthieu dot aubry at gmail dot com Assigned:
Status: Wont fix Package: Performance problem
PHP Version: 5.2.1 OS: linux
Private report: No CVE-ID: None
View Developer Edit
 [2007-04-11 13:26 UTC] matthieu dot aubry at gmail dot com 
Description:
------------
Hello,

I would like to cache big tables containing integers in PHP memory. But I notice that PHP uses a LOT of memory for variables.

I simply fill an array indexed with 1,000,000 distinct integers. I assign 0 to all elements.

Results
=======
PHP 5.1.2 result is 57.5 MB
PHP 5.2.1 result is 99.4 MB


Minimum used = 1E6 * (4bytes+4bytes) = 8MB.
I don't know much about PHP Array implementation. I don't expect it to use only 8MB of course, but I think it actually uses too much memory.

Also, why this difference between the 2 versions? 
(I've read bug reports stating memory problem in 5.2.0 but they were said to be fixed before 5.2.1)

Reproduce code:
---------------
<?php
function getMemoryUsage()
{
	return round(memory_get_usage() / (1024*1024), 1)." MB";
}
function printMemoryUsage()
{
	print("Memory = ".getMemoryUsage()." <br>\n");
}
printMemoryUsage();

$end = 1000000;
$array = array();
printMemoryUsage();
for($i = 0; $i < $end; $i++)
{
	$array[$i] = 0;
}
printMemoryUsage();
?>

Expected result:
----------------
Thank you for explanations/details ; is this behaviour expected? 
Could it be improved in future releases?


Actual result:
--------------
=================
Test PHP 5.2.1-0.dotdeb.1 with Suhosin-Patch 0.9.6.2 (cli) (built: Feb  9 2007 04:46:32)
=================
Linux 2.4.33bipiv-ipv4-32 #1 SMP lun oct 23 18:38:55 CEST 2006 i686 GNU/Linux

# time php test27.php
Memory = 0.1 MB <br>
Memory = 0.1 MB <br>
Memory = 99.4 MB <br>

real    0m0.954s

=================
Test PHP 5.1.2 (cli) (built: Mar  7 2007 21:53:56)
=================
Linux 2.6.15-23-386 #1 PREEMPT Tue May 23 13:49:40 UTC 2006 i686 GNU/Linux

$ time php test27.php
Memory = 0 MB <br>
Memory = 0 MB <br>
Memory = 57.5 MB <br>

real    0m1.520s

Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2007-04-11 14:12 UTC] tony2001@php.net 
Please try using this CVS snapshot:

  http://snaps.php.net/php5.2-latest.tar.gz
 
For Windows:
 
  http://snaps.php.net/win32/php5.2-win32-latest.zip
 [2007-04-11 17:29 UTC] stas@php.net 
Each element requires a value structure (zval) which takes 16 bytes. Also requires a hash bucket - which takes 36 bytes. That gives 52 bytes per value. Memory allocation headers take another 8 bytes*2 - which gives 68 bytes. Pretty close to what you have.
 [2007-04-11 19:03 UTC] matthieu dot aubry at gmail dot com 
Tony, I cannot risk to kill my development server yet, but as soon as possible, I'll try the latest PHP CVS.

Stats, thank you for your explanations. It is very interesting. Can we find some documentation about the internal implementation of PHP? Specifically about data structures?

68bytes is indeed pretty close; 57.5e6 / (2*4*1024*1024) ~= 68.54bytes

Do you think it would be possible to improve the memory usage of such data structures? 


Also, I don't understand "Memory allocation headers take another 8 bytes*2" : what do you mean by "memory allocation headers"?

Thanks.
 [2007-04-11 19:16 UTC] tony2001@php.net 
>68bytes is indeed pretty close; 57.5e6 / (2*4*1024*1024) ~= 68.54bytes

Right. And using PHP 5.2.2RC1 I get ~68Mb, so this is correct.

>Do you think it would be possible to improve the memory usage 
>of such data structures? 

No, that's not possible, otherwise it would have been done.
But you're encourage to help us if you know how to fit a complex struct into very limited space.
 [2007-04-11 20:24 UTC] stas@php.net 
Since PHP is a high level language, this incurs some penalties on handling values - i.e., we can't just store an integer, we have to store information about it's type, reference counts, etc. Same with arrays - we can't just put it as a block of memory, we have to enable it to work with all kinds of array functions, etc. - all of which takes its toll. 

Having said that, there are a number of ideas for reducing memory requirements that people working on, and memory management in 5.x has significantly improved, but some overhead would always exist. PHP is not built for maximum hardware efficiency as languages like C are, so if you have some scenario where absolute minimal memory usage is required you may want to look towards writing an extension that would wrap this part of the code. Writing extensions is not very hard, see: http://www.php.net/manual/en/internals.php
Also you could find internals information in this book:
http://blog.libssh2.org/index.php?/archives/24-Extending-and-Embedding-PHP.html
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved. 		Last updated: Tue Dec 17 11:01:28 2024 UTC