php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #61109 High memory overhead
Submitted: 2012-02-16 17:33 UTC Modified: 2018-05-08 15:52 UTC
Votes:4
Avg. Score:5.0 ± 0.0
Reproduced:3 of 3 (100.0%)
Same Version:2 (66.7%)
Same OS:2 (66.7%)
From: jpauli@php.net Assigned: cmb (profile)
Status: Closed Package: amqp (PECL)
PHP Version: 5.3.10 OS: Linux
Private report: No CVE-ID: None
View Add Comment Developer Edit
Welcome! If you don't have a Git account, you can't do anything here.
You can add a comment by following this link or if you reported this bug, you can edit this bug over here.
Block user comment
Status: Assign to:
Package:
Bug Type:
Summary:
From: jpauli@php.net
New email:
PHP Version: OS:

 

 [2012-02-16 17:33 UTC] jpauli@php.net
Description:
------------
For what I investigated, it's not related to ext/amqp but it's in librabbitmq; 
well I think, I'm not sure

So my script has a simple publisher, and no consumer (not needed, just fill data 
in a rabbitMQ server). It just publishes to a queue, in Direct mode, with one 
routing key. Easy.

I have a loop feeding some few data (few bytes, each publish() is fed few bytes 
). The loop has 5000 iterations. It may have more, but then I have not enough 
memory and start swaping / OOMing

IMPORTANT :
Using AMQP_IMMEDIATE in publish() triggers the leak.
Not using AMQP_IMMEDIATE in publish() is all right.

For what I noticed using callgrind, massif and memcheck, when using 
AMQP_IMMEDIATE, all the payload is delayed and actually really sent at shutdown, 
allocating lots of buffers for each data and not freeing them immediately, which 
might OOM the machine.
When not using AMQP_IMMEDIATE, the buffers seem to be allocated, sent, then 
freed immediately; thus no memory overhead.

Here are the results :
https://gist.github.com/1846541

The callgrind schemas are actually very interesting, as they clearly show that a 
memory starvation happens when PHP is shutting down using AMQP_IMMEDIATE which 
does not happen when not using AMQP_IMMEDIATE
https://gist.github.com/1846607

Test script:
---------------
class Publisher
{
    protected $channel;
    protected $connection;
    protected $name;
    protected $queue;
    
    public function __construct($exchangeName)
    {
        $this->connection = new \AMQPConnection(array('host'=>'127.0.0.1', 'port'=>5672, 'login'=>'guest', 'password'=>'guest'));
        $this->connection->connect();
        $this->channel = new \AMQPChannel($this->connection);

        $this->exchange = new \AMQPExchange($this->channel);
	$this->exchange->setName($this->name = $exchangeName);
	$this->exchange->setType(AMQP_EX_TYPE_DIRECT);
	$this->exchange->setFlags(AMQP_DURABLE);
	$this->exchange->declare();
	}
    
    public function sendSQLJob($sql)
    {
        $this->exchange->publish($sql, 'sqlkey', AMQP_IMMEDIATE);
    }

}

require __DIR__ . '/config.php';
$i=0;
$j = 5000;
$publisher = new Publisher('sql');
$fp = fopen('some/text/file', 'r');
while($sql = fgets($fp) && --$j) {
    $publisher->sendSQLJob($sql);
    printf("\r%d       \r", $i++);
}
unset($publisher);

Expected result:
----------------
memory level normal

Actual result:
--------------
memory consumption very high but not a leak as all buffers get freed, they simply 
seem to stack in a loop thus blowing up the memory

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2012-02-16 17:36 UTC] jpauli@php.net
See also #60141. Seems related
 [2012-02-28 21:15 UTC] pdezwart@php.net
Can you please be more specific about what you like to see happen? Based on the 
description, it sounds like this is normal behavior, and up to the user to not 
misuse the feature?

Thanks!
 [2012-03-05 10:26 UTC] jpauli@php.net
Aha, ok, so if this is normal behavior, then I have nothing more to add.
It's just totally inefficient as memory level really becomes too high in a huge 
loop with AMQP_IMMEDIATE activated. Thus this bug report :)
 [2012-08-30 17:15 UTC] i dot am dot brutto at gmail dot com
i have the same issue on freeBSD with 5.3.16(php-fmp) (1.0.4 amqplib) w/o AMQP_IMMEDIATE and w/o looping.

i've just published to AMQP_DURABLE exchange for one time in a script executing, 
but make itregular for a long time. 

After some little time (about 10-30 min) my scripts suddelny crushing with 
'allocate memory' issue. Also it appears in scripts where there are not any 
created amqp connections, channels or exchanges.
 [2012-09-26 15:24 UTC] ult_raa at mail dot ru
Same here with AMQPQueue::declare()

$con = new AMQPConnection();                                                                                                                                                          
$con->connect();
$ch = new AMQPChannel($con);
$q = new AMQPQueue($ch);
$q->setName('test');
while(true) { $q->declare(); }
 [2012-10-10 18:55 UTC] kitek at implix dot com
Same here even without this flag set. Just doing normal publish in a loop i.e.

for($i=0;$i<1000;$i++){
$con = new AMQPConnection();                                                                                                                                                          
$con->pconnect();
$ch = new AMQPChannel($con);
$e = new AMQPExchange($ch);
...
$e->publish('test');

}

I noticed that replacing pconnect() connect solved that.Maybe this is a different 
bug, i don't know.
 [2012-11-09 02:17 UTC] ult_raa at mail dot ru
resolved
its rabbitmq-c problem
https://github.com/alanxz/rabbitmq-c/pull/55
 [2012-11-14 01:21 UTC] bkw at codingforce dot com
this should be fixed in version 1.0.9:
https://github.com/pdezwart/php-amqp/pull/6
 [2018-05-08 15:52 UTC] cmb@php.net
-Status: Open +Status: Closed -Assigned To: +Assigned To: cmb
 [2018-05-08 15:52 UTC] cmb@php.net
> this should be fixed in version 1.0.9:

Fine, closing.
 
PHP Copyright © 2001-2021 The PHP Group
All rights reserved.
Last updated: Fri Apr 16 02:01:23 2021 UTC