php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #47940 imap_body() leaks memory
Submitted: 2009-04-09 22:31 UTC Modified: 2009-04-30 19:02 UTC
Votes:1
Avg. Score:5.0 ± 0.0
Reproduced:1 of 1 (100.0%)
Same Version:1 (100.0%)
Same OS:1 (100.0%)
From: jake dot levitt at mailtrust dot com Assigned: pajoye
Status: Closed Package: IMAP related
PHP Version: 5.2.9 OS: Ubuntu 8.04 (linux:2.6.24-21)
Private report: No CVE-ID:
 [2009-04-09 22:31 UTC] jake dot levitt at mailtrust dot com
Description:
------------
I am creating a script that migrates e-mail from one server to another.  On really large mailboxes, my script dies with "PHP Fatal error:  Out of memory (allocated 13631488) (tried to allocate 3381512 bytes)" even though memory_get_usage() function was reporting that the script was only using around 10 MBs.  My memory_limit is set to 1GB (but this isn't coming into play). When I examined the memory usage of the script using ps, I noticed that it would gradually increase until it reached ~92% of system memory (I have 2GB).  I eventually found that when I commented out imap_body, the memory usage would stay flat.  I wrote a script that can reproduce this bug.  I run the script in the background and then watch its memory using: ps -eo pid,ppid,rss,vsize,pcpu,pmem,cmd -ww --sort=pid | grep "\(memory-usage\)\|\(PID\)" | grep -v grep.  Please let me know if you need additional information.  This script is intended to be run on the cli.

Reproduce code:
---------------
<?php
$flags = '/novalidate-cert';
$host = 'mail.server.com:143';
$username = 'user';
$password = 'pass';
$folder = 'INBOX';

$base_imap_string = '{' . $host . $flags . '}';
$connect_string = $base_imap_string . $folder;

$mailbox = @imap_open($connect_string, $username, $password, 0, 3);
$reopen_success = imap_reopen($mailbox, $connect_string, 0, 3);

$message_ids = imap_search($mailbox, "ALL", SE_UID);
for ($i = 0; $i < 10000000; $i++) {
    imap_body($mailbox, $message_ids[0], FT_UID | FT_PEEK);
}

Expected result:
----------------
I would expect that memory usage as viewed by the ps command stays relatively flat.

Actual result:
--------------
Memory usage as viewed by the ps command increases over time until the script eventually dies with an "out of memory" error given.

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2009-04-14 15:43 UTC] jake dot levitt at mailtrust dot com
I have now confirmed that this bug also exists in PHP version 5.3.0RC1 on CentOs 5.1.
 [2009-04-14 18:41 UTC] jake dot levitt at mailtrust dot com
I tried closing and freeing the imap connection every 100 calls to imap_body() to see if it was the resource that was holding on to the memory.  This did not help.

Code:
if ($i % 100 === 0) {
    echo "Releasing mailbox\n";
    imap_close($mailbox);
    $mailbox = null;
    unset($mailbox);
    $mailbox = @imap_open($connect_string, $username, $password, 0, 3);
    $reopen_success = imap_reopen($mailbox, $connect_string, 0, 3);
}
 [2009-04-23 19:05 UTC] jake dot levitt at mailtrust dot com
A co-worker and I have created a diff that seems to fix this issue.  Would someone else mind checking it out and seeing if there are any problems with it?

diff php_imap.c php_imap_fixed.c 
1251a1252
>   char *body = mail_fetchtext_full (imap_le_struct->imap_stream, Z_LVAL_PP(msgno), NIL, myargc==3 ? Z_LVAL_PP(pflags) : NIL);
1253c1254,1255
<   RETVAL_STRING(mail_fetchtext_full (imap_le_struct->imap_stream, Z_LVAL_PP(msgno), NIL, myargc==3 ? Z_LVAL_PP(pflags) : NIL), 1);
---
>     RETVAL_STRING(body, 1);
>     free(body);
 [2009-04-23 21:03 UTC] scottmac@php.net
Can you provide the patch as a unified diff, just use the -u flag to cvs diff.
 [2009-04-23 21:37 UTC] jake dot levitt at mailtrust dot com
Here's a unified diff:
diff -u php-5.2.9/ext/imap/php_imap.c php-5.2.9-fixed/ext/imap/php_imap.c
--- php-5.2.9/ext/imap/php_imap.c        2008-12-31 06:17:38.000000000 -0500
+++ php-5.2.9-fixed/ext/imap/php_imap.c        2009-04-23 13:56:26.000000000 -0400
@@ -1250,7 +1250,10 @@
                RETURN_FALSE;
        }

-        RETVAL_STRING(mail_fetchtext_full (imap_le_struct->imap_stream, Z_LVAL_PP(msgno), NIL, myargc==3 ? Z_LVAL_PP(pflags) : NIL), 1);
+        char *body = mail_fetchtext_full (imap_le_struct->imap_stream, Z_LVAL_PP(msgno), NIL, myargc==3 ? Z_LVAL_PP(pflags) : NIL);
+
+        RETVAL_STRING(body, 1);
+        free(body);
}
/* }}} */
 [2009-04-23 22:35 UTC] pajoye@php.net
Slightly modified version committed in all branches. Thanks for your work!
 [2009-04-24 17:47 UTC] jake dot levitt at mailtrust dot com
Hey guys it looks like we were a little hasty in submitting this patch.  It now seems that our patch does lessen the memory leak, but does not eradicate it.  We ran our test script using the patch and it seemed to work, but running the actual migration program shows that there is still a leak.  Before our program would migrate ~1.5gb of mail now it migrates ~2.4gb before running out of memory.  We have updated our test script to better expose the memory leak:

/**
 * To watch memory usage use:
 * ps -eo pid,ppid,rss,vsize,pcpu,pmem,cmd -ww --sort=pid | grep "\(memory-usage-test\)\|\(PID\)" | grep -v grep
 */
$flags = '/novalidate-cert';
$host = 'a.host.com:143';
$username = 'a.user@a.host.com';
$password = 'password';
$folder = 'INBOX';

$base_imap_string = '{' . $host . $flags . '}';
$connect_string = $base_imap_string . $folder;

$mailbox = @imap_open($connect_string, $username, $password, 0, 3);
$reopen_success = imap_reopen($mailbox, $connect_string, 0, 3);

$message_ids = imap_search($mailbox, "ALL", SE_UID);
$num_msgs = count($message_ids);
for ($i = 0; $i < 1000000; $i++) {
    $msg_num = $i % $num_msgs;
    imap_body($mailbox, $message_ids[$msg_num], FT_UID);
}
 [2009-04-24 17:51 UTC] pajoye@php.net
Will test it using my test imap server. The last fix is correct but there is certainly other sources of leaks.
 [2009-04-24 18:52 UTC] nick at mailtrust dot com
After a little more testing we opened and closed the imap connection after every 50 messages.  This seems to release the memory and works as a temporary fix.


Also, we have confirmed the same bug in imap_fetchbody.  I tested applying the same fix as above and it seems to fix the issue.
 [2009-04-24 19:01 UTC] pajoye@php.net
There are also similar bugs with the criteria and other various imap structure which should be freed as well. fix for the criteria on its way.
 [2009-04-24 19:56 UTC] pajoye@php.net
2nd review of cclient, it is wrong to free the body as it is a simple pointer to the msg data, not a copy of its contents. However the criteria has to be freed. I'd to revert the fix for fetch_body and looks for better solution (maybe userland) to minimize the memory usage.
 [2009-04-30 19:02 UTC] pajoye@php.net
This bug has been fixed in CVS.

Snapshots of the sources are packaged every three hours; this change
will be in the next snapshot. You can grab the snapshot at
http://snaps.php.net/.
 
Thank you for the report, and for helping us make PHP better.


 
PHP Copyright © 2001-2014 The PHP Group
All rights reserved.
Last updated: Thu Apr 17 03:01:55 2014 UTC