|
|
php.net | support | documentation | report a bug | advanced search | search howto | statistics | random bug | login |
[2009-04-09 22:31 UTC] jake dot levitt at mailtrust dot com
Description:
------------
I am creating a script that migrates e-mail from one server to another. On really large mailboxes, my script dies with "PHP Fatal error: Out of memory (allocated 13631488) (tried to allocate 3381512 bytes)" even though memory_get_usage() function was reporting that the script was only using around 10 MBs. My memory_limit is set to 1GB (but this isn't coming into play). When I examined the memory usage of the script using ps, I noticed that it would gradually increase until it reached ~92% of system memory (I have 2GB). I eventually found that when I commented out imap_body, the memory usage would stay flat. I wrote a script that can reproduce this bug. I run the script in the background and then watch its memory using: ps -eo pid,ppid,rss,vsize,pcpu,pmem,cmd -ww --sort=pid | grep "\(memory-usage\)\|\(PID\)" | grep -v grep. Please let me know if you need additional information. This script is intended to be run on the cli.
Reproduce code:
---------------
<?php
$flags = '/novalidate-cert';
$host = 'mail.server.com:143';
$username = 'user';
$password = 'pass';
$folder = 'INBOX';
$base_imap_string = '{' . $host . $flags . '}';
$connect_string = $base_imap_string . $folder;
$mailbox = @imap_open($connect_string, $username, $password, 0, 3);
$reopen_success = imap_reopen($mailbox, $connect_string, 0, 3);
$message_ids = imap_search($mailbox, "ALL", SE_UID);
for ($i = 0; $i < 10000000; $i++) {
imap_body($mailbox, $message_ids[0], FT_UID | FT_PEEK);
}
Expected result:
----------------
I would expect that memory usage as viewed by the ps command stays relatively flat.
Actual result:
--------------
Memory usage as viewed by the ps command increases over time until the script eventually dies with an "out of memory" error given.
PatchesPull RequestsHistoryAllCommentsChangesGit/SVN commits
|
|||||||||||||||||||||||||||||||||||||
Copyright © 2001-2024 The PHP GroupAll rights reserved. |
Last updated: Thu Nov 21 10:01:29 2024 UTC |
I tried closing and freeing the imap connection every 100 calls to imap_body() to see if it was the resource that was holding on to the memory. This did not help. Code: if ($i % 100 === 0) { echo "Releasing mailbox\n"; imap_close($mailbox); $mailbox = null; unset($mailbox); $mailbox = @imap_open($connect_string, $username, $password, 0, 3); $reopen_success = imap_reopen($mailbox, $connect_string, 0, 3); }Here's a unified diff: diff -u php-5.2.9/ext/imap/php_imap.c php-5.2.9-fixed/ext/imap/php_imap.c --- php-5.2.9/ext/imap/php_imap.c 2008-12-31 06:17:38.000000000 -0500 +++ php-5.2.9-fixed/ext/imap/php_imap.c 2009-04-23 13:56:26.000000000 -0400 @@ -1250,7 +1250,10 @@ RETURN_FALSE; } - RETVAL_STRING(mail_fetchtext_full (imap_le_struct->imap_stream, Z_LVAL_PP(msgno), NIL, myargc==3 ? Z_LVAL_PP(pflags) : NIL), 1); + char *body = mail_fetchtext_full (imap_le_struct->imap_stream, Z_LVAL_PP(msgno), NIL, myargc==3 ? Z_LVAL_PP(pflags) : NIL); + + RETVAL_STRING(body, 1); + free(body); } /* }}} */Hey guys it looks like we were a little hasty in submitting this patch. It now seems that our patch does lessen the memory leak, but does not eradicate it. We ran our test script using the patch and it seemed to work, but running the actual migration program shows that there is still a leak. Before our program would migrate ~1.5gb of mail now it migrates ~2.4gb before running out of memory. We have updated our test script to better expose the memory leak: /** * To watch memory usage use: * ps -eo pid,ppid,rss,vsize,pcpu,pmem,cmd -ww --sort=pid | grep "\(memory-usage-test\)\|\(PID\)" | grep -v grep */ $flags = '/novalidate-cert'; $host = 'a.host.com:143'; $username = 'a.user@a.host.com'; $password = 'password'; $folder = 'INBOX'; $base_imap_string = '{' . $host . $flags . '}'; $connect_string = $base_imap_string . $folder; $mailbox = @imap_open($connect_string, $username, $password, 0, 3); $reopen_success = imap_reopen($mailbox, $connect_string, 0, 3); $message_ids = imap_search($mailbox, "ALL", SE_UID); $num_msgs = count($message_ids); for ($i = 0; $i < 1000000; $i++) { $msg_num = $i % $num_msgs; imap_body($mailbox, $message_ids[$msg_num], FT_UID); }