php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #74072 Memory leaks with setter causing out of memory
Submitted: 2017-02-10 10:39 UTC Modified: 2017-02-16 12:57 UTC
Votes:3
Avg. Score:5.0 ± 0.0
Reproduced:3 of 3 (100.0%)
Same Version:3 (100.0%)
Same OS:2 (66.7%)
From: ealexs at gmail dot com Assigned:
Status: Duplicate Package: Reproducible crash
PHP Version: 7.0.15 OS: Debian 8
Private report: No CVE-ID: None
 [2017-02-10 10:39 UTC] ealexs at gmail dot com
Description:
------------
We have a framework with an ORM within. The MODEL object has a setter in it that causes the problems.
We pull ~100.000 records from a DB (mysqli) in batches of 1000.
I use "memory_get_usage(true)" to monitor the memory usage 
after each batch. 
The behaiviour is random. 
After the first batches memory_get_usage(true) = 2 MB. 
Then it starts to grow until it runs out of memory (set at 32MB atm).
In rare cases (refreshing) it finishes. 
It always does the same thing! (no data changed - devel DB).

If the setter (public function __set) is removed, or we avoid beeing called,
then it always runs and memory_get_usage(true) stays at 2 MB. Always.

<?php
	public function __set($property, $value)
	{
		return $this->set($property, $value);
	}

	public function set($property, $value = null, $transform_with = null, $requester = null, $reason = null)
	{
		if ($property{0} === "_")
		{
			$this->$property = $value;
			return;
		}
		// manage saving the old property
		if ((!(isset($this->_ols[$property]) || ($this->_ols && array_key_exists($property, $this->_ols)))) && (($value === null) || ($this->$property !== $value)))
			$this->_ols[$property] = $this->$property;
		// end old
		$set = $this->getModelType()->properties[$property]->setter;
		if ($set)
			$this->$set($value);
		else
			$this->$property = $value;
	}
 
?>

I was unable to isolate the bug outside our framework atm.

Here is Valgrind's output (if the setter is removed or not called these elements vanishes)

==6720== 
==6720== 2,299,968 bytes in 7,986 blocks are possibly lost in loss record 9,815 of 9,817
==6720==    at 0x4C28C20: malloc (vg_replace_malloc.c:296)
==6720==    by 0x831CD8: __zend_malloc (zend_alloc.c:2864)
==6720==    by 0x863DF6: zend_hash_real_init_ex (zend_hash.c:140)
==6720==    by 0x863DF6: zend_hash_check_init (zend_hash.c:163)
==6720==    by 0x863DF6: _zend_hash_add_or_update_i (zend_hash.c:563)
==6720==    by 0x863DF6: _zend_hash_add (zend_hash.c:638)
==6720==    by 0x88D53C: zend_hash_add_mem (zend_hash.h:570)
==6720==    by 0x88D53C: zend_get_property_guard (zend_object_handlers.c:503)
==6720==    by 0x88FB76: zend_std_write_property (zend_object_handlers.c:667)
==6720==    by 0x8B71E3: zend_assign_to_object (zend_execute.c:1234)
==6720==    by 0x8B71E3: ZEND_ASSIGN_OBJ_SPEC_CV_CONST_HANDLER (zend_vm_execute.h:32028)
==6720==    by 0x89584A: execute_ex (zend_vm_execute.h:414)
==6720==    by 0x8E9566: zend_execute (zend_vm_execute.h:458)
==6720==    by 0x857EF3: zend_execute_scripts (zend.c:1437)
==6720==    by 0x7FB23F: php_execute_script (main.c:2492)
==6720==    by 0x8EB1F9: do_cli (php_cli.c:977)
==6720==    by 0x449796: main (php_cli.c:1347)
==6720== 

==6720== 
==6720== 18,452,736 bytes in 8,009 blocks are possibly lost in loss record 9,817 of 9,817
==6720==    at 0x4C28C20: malloc (vg_replace_malloc.c:296)
==6720==    by 0x831CD8: __zend_malloc (zend_alloc.c:2864)
==6720==    by 0x862C3F: zend_hash_real_init_ex (zend_hash.c:140)
==6720==    by 0x862C3F: zend_hash_real_init (zend_hash.c:205)
==6720==    by 0x88E192: rebuild_object_properties (zend_object_handlers.c:82)
==6720==    by 0x8900B2: zend_std_write_property (zend_object_handlers.c:709)
==6720==    by 0x8BCAF2: zend_assign_to_object (zend_execute.c:1234)
==6720==    by 0x8BCAF2: ZEND_ASSIGN_OBJ_SPEC_UNUSED_CV_HANDLER (zend_vm_execute.h:26390)
==6720==    by 0x89584A: execute_ex (zend_vm_execute.h:414)
==6720==    by 0x849650: zend_call_function (zend_execute_API.c:858)
==6720==    by 0x874A32: zend_call_method (zend_interfaces.c:104)
==6720==    by 0x88FBD9: zend_std_call_setter (zend_object_handlers.c:216)
==6720==    by 0x88FBD9: zend_std_write_property (zend_object_handlers.c:674)
==6720==    by 0x8B71E3: zend_assign_to_object (zend_execute.c:1234)
==6720==    by 0x8B71E3: ZEND_ASSIGN_OBJ_SPEC_CV_CONST_HANDLER (zend_vm_execute.h:32028)
==6720==    by 0x89584A: execute_ex (zend_vm_execute.h:414)
==6720==    by 0x8E9566: zend_execute (zend_vm_execute.h:458)
==6720==    by 0x857EF3: zend_execute_scripts (zend.c:1437)
==6720==    by 0x7FB23F: php_execute_script (main.c:2492)
==6720==    by 0x8EB1F9: do_cli (php_cli.c:977)
==6720==    by 0x449796: main (php_cli.c:1347)
==6720== 




Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2017-02-11 12:03 UTC] ealexs at gmail dot com
I have managed to create a test script that reproduces the bug
by calling "test_set_by_method($max)" the execution works fine
Using test_set_by_magic_setter($max) it will run out of memory

<?php

header("Content-type: text");
ini_set('memory_limit', '32M');

// warm up
$obj = new TestGS();
$max = 200000;

// $t1 = microtime(true);
// this will finish ok
test_set_by_method($max);
// $set_time = microtime(true) - $t1;

// $t1 = microtime(true);
// this will cause Allowed memory size ... exhausted 
test_set_by_magic_setter($max);
// $set_time = microtime(true) - $t1;

// end warm up
function test_set_by_method($max = 200000)
{
	$t1 = microtime(true);
	$val = 1;
	$bag = [];

	echo "START SET BY METHOD =======================================\n\n";
	for ($i = 0; $i < $max; $i++)
	{
		$obj = new TestGS();

		$obj->setProperty_001($val++);
		$obj->setProperty_002($val++);
		$obj->setProperty_003($val++);
		$obj->setProperty_004($val++);
		$obj->setProperty_005($val++);
		$obj->setProperty_006($val++);
		$obj->setProperty_007($val++);
		$obj->setProperty_008($val++);
		$obj->setProperty_009($val++);
		$obj->setProperty_010($val++);

		$obj->setProperty_011($val++);
		$obj->setProperty_012($val++);
		$obj->setProperty_013($val++);
		$obj->setProperty_014($val++);
		$obj->setProperty_015($val++);
		$obj->setProperty_016($val++);
		$obj->setProperty_017($val++);
		$obj->setProperty_018($val++);
		$obj->setProperty_019($val++);
		$obj->setProperty_020($val++);

		// every 10 k we dump
		if ((int)(floor($i/10000)*10000) === (int)$i)
		{
			echo "Memory usage [{$i}]: ".(memory_get_usage(true)/1024)." KB\n";
		}

		$bag[] = $obj;
	}

	echo "Memory usage [{$i}]: ".(memory_get_usage(true)/1024)." KB\n";
	echo "DONE SET BY METHOD =======================================\n\n";
	
	return [$bag, $val];
}

function test_set_by_magic_setter($max = 200000)
{
	$val = 1;
	$bag = [];

	echo "START SET BY MAGIC SETTER =======================================\n\n";
	for ($i = 0; $i < $max; $i++)
	{
		$obj = new TestGS();

		$obj->property_001 = $val++;
		$obj->property_002 = $val++;
		$obj->property_003 = $val++;
		$obj->property_004 = $val++;
		$obj->property_005 = $val++;
		$obj->property_006 = $val++;
		$obj->property_007 = $val++;
		$obj->property_008 = $val++;
		$obj->property_009 = $val++;
		$obj->property_010 = $val++;

		$obj->property_011 = $val++;
		$obj->property_012 = $val++;
		$obj->property_013 = $val++;
		$obj->property_014 = $val++;
		$obj->property_015 = $val++;
		$obj->property_016 = $val++;
		$obj->property_017 = $val++;
		$obj->property_018 = $val++;
		$obj->property_019 = $val++;
		$obj->property_020 = $val++;

		// every 10 k we dump
		if ((int)(floor($i/10000)*10000) === (int)$i)
		{
			echo "Memory usage [{$i}]: ".(memory_get_usage(true)/1024)." KB\n";
		}

		$bag[] = $obj;
	}

	echo "Memory usage [{$i}]: ".(memory_get_usage(true)/1024)." KB\n";
	echo "END SET BY MAGIC SETTER =======================================\n\n";
	
	return [$bag, $val];
}

function test_get_by_method($bag)
{
	// GET TEST
	foreach ($bag as $obj)
	{
		if ($obj->getProperty_001()) $val++;
		if ($obj->getProperty_002()) $val++;
		if ($obj->getProperty_003()) $val++;
		if ($obj->getProperty_004()) $val++;
		if ($obj->getProperty_005()) $val++;
		if ($obj->getProperty_006()) $val++;
		if ($obj->getProperty_007()) $val++;
		if ($obj->getProperty_008()) $val++;
		if ($obj->getProperty_009()) $val++;
		if ($obj->getProperty_010()) $val++;

		if ($obj->getProperty_011()) $val++;
		if ($obj->getProperty_012()) $val++;
		if ($obj->getProperty_013()) $val++;
		if ($obj->getProperty_014()) $val++;
		if ($obj->getProperty_015()) $val++;
		if ($obj->getProperty_016()) $val++;
		if ($obj->getProperty_017()) $val++;
		if ($obj->getProperty_018()) $val++;
		if ($obj->getProperty_019()) $val++;
		if ($obj->getProperty_020()) $val++;
		/*
		if ($obj->property_001) $val++;
		if ($obj->property_002) $val++;
		if ($obj->property_003) $val++;
		if ($obj->property_004) $val++;
		if ($obj->property_005) $val++;
		if ($obj->property_006) $val++;
		if ($obj->property_007) $val++;
		if ($obj->property_008) $val++;
		if ($obj->property_009) $val++;
		if ($obj->property_010) $val++;

		if ($obj->property_011) $val++;
		if ($obj->property_012) $val++;
		if ($obj->property_013) $val++;
		if ($obj->property_014) $val++;
		if ($obj->property_015) $val++;
		if ($obj->property_016) $val++;
		if ($obj->property_017) $val++;
		if ($obj->property_018) $val++;
		if ($obj->property_019) $val++;
		if ($obj->property_020) $val++;*/
	}
}

// 967 ms | 390 ms | 2.4 more time
// for compile / provisioning it's fine ... a lot of things will be internal

/*echo "\n\nGET time for {$max} objs : ", round($get_time*1000, 4), " ms";
echo "\n\nSET time for {$max} objs : ", round($set_time*1000, 4), " ms";
echo "\n\nTotal time: ", round((microtime(true) - $t1)*1000, 4), " ms";
*/

class TestGS
{
	protected $property_001;
	protected $property_002;
	protected $property_003;
	protected $property_004;
	protected $property_005;
	protected $property_006;
	protected $property_007;
	protected $property_008;
	protected $property_009;
	protected $property_010;
	
	protected $property_011;
	protected $property_012;
	protected $property_013;
	protected $property_014;
	protected $property_015;
	protected $property_016;
	protected $property_017;
	protected $property_018;
	protected $property_019;
	protected $property_020;
	
	public function getProperty_001() { return $this->property_001; }
	public function getProperty_002() { return $this->property_002; }
	public function getProperty_003() { return $this->property_003; }
	public function getProperty_004() { return $this->property_004; }
	public function getProperty_005() { return $this->property_005; }
	public function getProperty_006() { return $this->property_006; }
	public function getProperty_007() { return $this->property_007; }
	public function getProperty_008() { return $this->property_008; }
	public function getProperty_009() { return $this->property_009; }
	public function getProperty_010() { return $this->property_010; }
	
	public function getProperty_011() { return $this->property_011; }
	public function getProperty_012() { return $this->property_012; }
	public function getProperty_013() { return $this->property_013; }
	public function getProperty_014() { return $this->property_014; }
	public function getProperty_015() { return $this->property_015; }
	public function getProperty_016() { return $this->property_016; }
	public function getProperty_017() { return $this->property_017; }
	public function getProperty_018() { return $this->property_018; }
	public function getProperty_019() { return $this->property_019; }
	public function getProperty_020() { return $this->property_020; }
	
	public function setProperty_001(int $value) { $this->property_001 = $value; }
	public function setProperty_002(int $value) { $this->property_002 = $value; }
	public function setProperty_003(int $value) { $this->property_003 = $value; }
	public function setProperty_004(int $value) { $this->property_004 = $value; }
	public function setProperty_005(int $value) { $this->property_005 = $value; }
	public function setProperty_006(int $value) { $this->property_006 = $value; }
	public function setProperty_007(int $value) { $this->property_007 = $value; }
	public function setProperty_008(int $value) { $this->property_008 = $value; }
	public function setProperty_009(int $value) { $this->property_009 = $value; }
	public function setProperty_010(int $value) { $this->property_010 = $value; }
	
	public function setProperty_011(int $value) { $this->property_011 = $value; }
	public function setProperty_012(int $value) { $this->property_012 = $value; }
	public function setProperty_013(int $value) { $this->property_013 = $value; }
	public function setProperty_014(int $value) { $this->property_014 = $value; }
	public function setProperty_015(int $value) { $this->property_015 = $value; }
	public function setProperty_016(int $value) { $this->property_016 = $value; }
	public function setProperty_017(int $value) { $this->property_017 = $value; }
	public function setProperty_018(int $value) { $this->property_018 = $value; }
	public function setProperty_019(int $value) { $this->property_019 = $value; }
	public function setProperty_020(int $value) { $this->property_020 = $value; }
	
	public function __set($name, $value)
	{
		// both of these fail
		// this will cause Allowed memory size ... exhausted 
		// $this->{'set'.$name}($value);
		
		// this will also cause Allowed memory size ... exhausted 
		$this->$name = $value;
	}
}
 [2017-02-16 07:53 UTC] ealexs at gmail dot com
-Package: *General Issues +Package: Reproducible crash
 [2017-02-16 07:53 UTC] ealexs at gmail dot com
updated the "Package"
 [2017-02-16 11:22 UTC] torben at dannhauer dot info
I can reproduce this bug with the below test script.
I have a error in a production server which seems to be the same but I'm not sure.

On my side, your testscript also fails with xdebug modul loaded, however my production error does disappear with Xdebug module loaded.
 [2017-02-16 11:35 UTC] nikic@php.net
-Status: Open +Status: Duplicate
 [2017-02-16 11:35 UTC] nikic@php.net
Duplicate of bug #65340.
 [2017-02-16 12:30 UTC] ealexs at gmail dot com
@torben - less memory usage without XDEBUG, reduce memory or increase the cycles.

@nikic
Will the fix be applied for PHP 7.0.* also ?
Can you explain a bit the limitations of the fix ?
Why is the memory usage random between the cycles random ? (this worries me a bit)
 [2017-02-16 12:57 UTC] nikic@php.net
On the limitations, quoting from the other bug report:

> A magic accessor should not trigger a magic accessor on a different property name on the same object. $a->__get('x') -> $a->__set('x') is fine. $a->__get('x') -> $b->__get('y') is fine. $a->__get('x') -> $a->__get('y') may still lead to unbounded memory growth, if this is done for many distinct property names.

It is not possible to apply this change to PHP 7.0.

As to the jumping memory usage between cycles, this is an artifact of passing "true" to memory_get_usage(). If you do so, you will see internal details of the allocator. In this case, what you observe is that when the memory limit is reached, an allocator GC is triggered, which may release chunks, thus reducing memory usage below the previous value. Generally, I'd recommend calling memory_get_usage() without argument, as the values become much harder to interpret otherwise.
 [2017-02-16 13:25 UTC] ealexs at gmail dot com
Thank you very much all your points were useful.

I think this should be added to the php docs as "Caution": "A magic accessor should not trigger a magic accessor on a different property name on the same object. $a->__get('x') -> $a->__set('x') is fine. $a->__get('x') -> $b->__get('y') is fine. $a->__get('x') -> $a->__get('y') may still lead to unbounded memory growth, if this is done for many distinct property names."
> also add $a->__set('x') -> $a->__set('y') - unbounded memory growth

Suggested pages:
http://php.net/manual/en/language.oop5.magic.php
http://php.net/manual/en/language.oop5.overloading.php
 
PHP Copyright © 2001-2020 The PHP Group
All rights reserved.
Last updated: Wed Oct 28 04:01:23 2020 UTC