php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #70098 Real memory usage doesn't decrease
Submitted: 2015-07-18 20:43 UTC Modified: 2015-08-04 15:24 UTC
Votes:2
Avg. Score:4.5 ± 0.5
Reproduced:1 of 1 (100.0%)
Same Version:1 (100.0%)
Same OS:1 (100.0%)
From: robin dot kunde at recoursive dot com Assigned: dmitry
Status: Closed Package: Performance problem
PHP Version: 7.0.0beta1 OS: Ubuntu 15.04, OSX 10.10.4
Private report: No CVE-ID:
 [2015-07-18 20:43 UTC] robin dot kunde at recoursive dot com
Description:
------------
PHP 7b1 doesn't appear to be reclaiming freed memory correctly. This problem only affects real usage as reported by memory_get_usage(true). Usage as reported by memory_get_usage(false) behaves as expected.

The problem appears to be related to allocation of strings, the use of stdclass, and/or multidimensional array. I don't believe PHP is actually leaking memory, since usage is not continuously increasing. It just appear that certain actions prevent real usage from decreasing again after variables are nulled out or leave scope. The test script gist I posted can demonstrates the problem better than I can explain it.

This problem still occurs with the master git branch as of 7/17.

Test script:
---------------
https://gist.github.com/robinkunde/68f3d442aa294550ab15

Expected result:
----------------
PHP 5.6.11

Assembling 5M string
Real: 0.50, Malloc: 0.25
Storing 1M bools
Real: 0.75, Malloc: 0.25
Real: 1.00, Malloc: 0.25
Real: 1.00, Malloc: 0.25
Storing 1M ints
Real: 1.25, Malloc: 0.25
Real: 1.25, Malloc: 0.25
Real: 1.50, Malloc: 0.25
Storing 1M stringyfied ints
Real: 1.50, Malloc: 0.25
Real: 1.75, Malloc: 0.25
Real: 1.75, Malloc: 0.25
Storing 1M stringyfied ints with stringyfied int keys
Real: 1.75, Malloc: 0.25
Real: 1.75, Malloc: 0.25
Real: 1.75, Malloc: 0.25
Storing 1M empty strings
Real: 1.75, Malloc: 0.25
Real: 2.00, Malloc: 0.25
Real: 2.00, Malloc: 0.25
Storing 1M hellos
Real: 2.00, Malloc: 0.25
Real: 2.25, Malloc: 0.25
Real: 2.00, Malloc: 0.25
Storing 1M hellos with stringyfied int keys
Real: 2.00, Malloc: 0.25
Storing 1M empty arrays
Real: 1.50, Malloc: 0.25
Real: 2.00, Malloc: 0.25
Real: 2.00, Malloc: 0.25
Storing 1M arrays with empty strings
Real: 2.00, Malloc: 0.25
Real: 2.50, Malloc: 0.25
Real: 2.50, Malloc: 0.25
Storing 1M arrays with hello
Real: 2.50, Malloc: 0.25
Real: 2.75, Malloc: 0.25
Real: 2.75, Malloc: 0.25

Final Usage

Real: 2.75, Malloc: 0.25
Real peak: 436.00, Malloc peak: 435.50

Actual result:
--------------
PHP 7.0.0b1

Assembling 5M string
Real: 4.00, Malloc: 0.36
Storing 1M bools
Real: 4.00, Malloc: 0.36
Real: 32.00, Malloc: 0.36
Real: 32.00, Malloc: 0.36
Storing 1M ints
Real: 32.00, Malloc: 0.36
Real: 32.00, Malloc: 0.36
Real: 32.00, Malloc: 0.36
Storing 1M stringyfied ints
Real: 32.00, Malloc: 0.36
Real: 62.00, Malloc: 0.36
Real: 62.00, Malloc: 0.36
Storing 1M stringyfied ints with stringyfied int keys
Real: 62.00, Malloc: 0.36
Real: 62.00, Malloc: 0.36
Real: 62.00, Malloc: 0.36
Storing 1M empty strings
Real: 62.00, Malloc: 0.36
Real: 62.00, Malloc: 0.36
Real: 62.00, Malloc: 0.36
Storing 1M hellos
Real: 62.00, Malloc: 0.36
Real: 62.00, Malloc: 0.36
Real: 62.00, Malloc: 0.36
Storing 1M hellos with stringyfied int keys
Real: 62.00, Malloc: 0.36
Storing 1M empty arrays
Real: 116.00, Malloc: 0.36
Real: 116.00, Malloc: 0.36
Real: 116.00, Malloc: 0.36
Storing 1M arrays with empty strings
Real: 422.00, Malloc: 0.36
Real: 422.00, Malloc: 0.36
Real: 422.00, Malloc: 0.36
Storing 1M arrays with hello
Real: 422.00, Malloc: 0.36
Real: 422.00, Malloc: 0.36
Real: 422.00, Malloc: 0.36

Final Usage

Real: 422.00, Malloc: 0.36
Real peak: 476.00, Malloc peak: 425.46

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2015-07-19 17:45 UTC] bwoebi@php.net
-Status: Open +Status: Assigned -Assigned To: +Assigned To: dmitry
 [2015-07-19 17:45 UTC] bwoebi@php.net
PHP 7 uses a new memory allocator, which puts small allocations of 3 KB or less into buckets with well defined sizes. These small buckets though… are never freed.

@Dmitry: Is that intentional that they're never freed? I see no handling logic for unmmapping these... Sure, it's faster to not have logic which frees them, but seems we need it.
 [2015-07-20 06:12 UTC] laruence@php.net
why we need them ? this is just a particular case that triggers allocates lot's of size type.. 

and it won't affacts memory_get_usage(false)...
 [2015-07-20 11:47 UTC] bwoebi@php.net
The memory is never released back to the system. That may go bad with a few long running worker processes doing complicated calculations with a lot of data.

They'll eat all the memory and system will need to swap (or slow down in other ways…). Hence I think we should care.
 [2015-07-20 13:17 UTC] laruence@php.net
actually , it won't , let's say you have allocated a 7bytes memory , then one page of 4KB will allocated for it. and unless there is no any 8 bytes slot in this page, any request for allocating < 8 bytes won't trigger new memory allocating request to OS...

anyway, maybe we should add some gc of memory if mmap fails of OOM...
 [2015-07-20 13:28 UTC] robin dot kunde at recoursive dot com
Base on what you said, I performed another test:

I set memory_limit to 512M. At the end of the script I posted, after nulling out all the vars, and memory usage being (Real: 422.00, Malloc: 0.36), I tried to read a 200M file. This triggered the memory exhaustion fatal error. If I read the large file at the beginning of the script, and then null the var, the script runs as expected and memory usage doesn't change much since it's a large allocation.

I'm glad that the small bucket pool is counted against the memory limit and scripts can't accidentally use up to twice as much memory. However, it does make the behavior of long running, memory intensive script potentially unpredictable.

I understand the performance implications of not having to free those buckets. As a compromise, the small bucket pool could be cleaned only when absolutely necessary.
 [2015-07-20 19:36 UTC] bwoebi@php.net
@laruence: That's right. But not the issue. Issue is rather that, having allocated a lot of pages (e.g. during initialization) and then the variables all normally dtor'ed, the pages will never be returned back to the OS for the whole lifetime of the script (which may be very long)...

Sure, it's rather an edge-case case, but I'd like to see completely unused pages released back to the system

But it might involve a per-page counter and steal like a four instructions and a branch per alloc/free… Not sure how much we can afford.
 [2015-07-21 03:19 UTC] laruence@php.net
pages are allocated together in trunk(~2M), which means if a page is not used anymore it won't be able to release back to OS , unless the whole trunk is not used anymore..
 [2015-07-21 15:02 UTC] rasmus@php.net
I think the idea of freeing these on an OOM is a good one. I would hate to see additional overhead added to try to keep track of these for the common case of short-lived requests.
 [2015-07-25 07:11 UTC] alex at alex-at dot ru
Not only on OOM probably, but on forced gc_collect_cycles() call as well, to make forced cleaning possible for long running scripts.
 [2015-08-03 13:01 UTC] dmitry@php.net
New memory manager may keep pages, previously allocated to serve some "small" blocks (less than 3Kb), even if all of them were deallocated. These "small" blocks are cached (in linked list) and may be reused later.

This strategy works well for real-life apps, because all pages are reclaimed on request shut-down anyway. However, I agree, this may be not good enough for some long-running apps.

To fix this, we will need some kind of GC, that will traverse cache lists for each "small" size and determine pages where all blocks are cached. Then this pages may be freed and all the nested blocks removed from cache lists. This GC may be triggered when memory limit is reached, or before allocating each new 2M chunk.
 [2015-08-04 15:21 UTC] dmitry@php.net
Automatic comment on behalf of dmitry@zend.com
Revision: http://git.php.net/?p=php-src.git;a=commit;h=668ecaa606b3203311b3329fcbd49b59f715e1e4
Log: Fixed bug #70098 (Real memory usage doesn't decrease)
 [2015-08-04 15:21 UTC] dmitry@php.net
-Status: Assigned +Status: Closed
 [2015-08-04 15:24 UTC] dmitry@php.net
The fix doesn't change the output of the test script, because MM GC is triggered only by "Out of Memory". However, it's possible to add call to gc_mem_caches() before printing memory usage, and in this case "real" memory consumption won't grow.
 [2015-08-05 10:12 UTC] ab@php.net
Automatic comment on behalf of dmitry@zend.com
Revision: http://git.php.net/?p=php-src.git;a=commit;h=668ecaa606b3203311b3329fcbd49b59f715e1e4
Log: Fixed bug #70098 (Real memory usage doesn't decrease)
 [2016-07-20 11:37 UTC] davey@php.net
Automatic comment on behalf of dmitry@zend.com
Revision: http://git.php.net/?p=php-src.git;a=commit;h=668ecaa606b3203311b3329fcbd49b59f715e1e4
Log: Fixed bug #70098 (Real memory usage doesn't decrease)
 
PHP Copyright © 2001-2017 The PHP Group
All rights reserved.
Last updated: Sat Apr 29 07:01:45 2017 UTC