php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login

go to bug id or search bugs for

Bug #29922 Huge memory usage in file copy of big files
Submitted: 2004-08-31 19:07 UTC Modified: 2005-02-23 19:37 UTC
Votes:2
Avg. Score:3.0 ± 2.0
Reproduced:2 of 2 (100.0%)
Same Version:0 (0.0%)
Same OS:1 (50.0%)
From: info at pion dot xs4all dot nl Assigned: wez (profile)
Status: Closed Package: *Directory/Filesystem functions
PHP Version: 5.0.1 OS: Linux
Private report: No CVE-ID: None
View Add Comment Developer Edit
Welcome! If you don't have a Git account, you can't do anything here.
You can add a comment by following this link or if you reported this bug, you can edit this bug over here.
(description)
Block user comment
Status: Assign to:
Package:
Bug Type:
Summary:
From: info at pion dot xs4all dot nl
New email:
PHP Version: OS:

 

 [2004-08-31 19:07 UTC] info at pion dot xs4all dot nl 
Description:
------------
If you copy a file of N megabytes (N in the order of a few
hundred megs) your proces will use the same order of ammount
of memory (or less if the system doesn't have that much).

This is probably because the file is compleetly memory mapped
and than the content is written to an other file.

Probably it will be better to use the behavior of the unix
"cp" command which reads 4k and than writes the 4k.

In this way "cp" is almost a factor two faster than php
and has less impact on the system performance.

Reproduce code:
---------------
<?php
#
# This assumes that there excist a big
# file "/tmp/file_512m" which has 512mb
# of data and that there is enough space
# in "/tmp".
#
  copy ("/tmp/file_512m", "/tmp/target")
?>

Expected result:
----------------
The unix "cp" needed 1 minut 8 seconds to copy
the file and php used 2 and 18 seconds on my desktop
system with 512mb.

Actual result:
--------------
They should be in the same order

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2004-08-31 22:03 UTC] info at pion dot xs4all dot nl 
- The observation was on rhel 3 with php 4.3.2
- The test in the report was done on a FC1 system with php 4.3.3
- Now i have checked php 4.3.8 on fc2 (the default php) and php 5.0.1

Al have the same result although the speed diferences on the
fc2 system are smaller. This could be caused by different
hardware or differences in the linux vm (2.4.X versus 2.6.X)
 [2004-09-01 08:44 UTC] derick@php.net 
I'm sure there is already a bugreport for this, and afaik somebody also worked on an attempt to fix it. Can you try a PHP 5.1.0-dev snapshot please? (http://naps.php.net)
 [2004-09-01 21:55 UTC] info at pion dot xs4all dot nl 
I have checked again but i can't find the bug report.

I have tested with "5.1.0-dev" with checked out version
200409011830 but the problem is still there.
 [2004-09-01 22:19 UTC] info at pion dot xs4all dot nl 
I have a small patch for the 5.1.0-dev which solves the
problem but:
  This patch says a normal copy should be used for files
  bigger then 32 MB. But this is an rather arbitrary value
  choosen by me.

Here follows the patch:
-diff -Naur org/php5-200409011830/main/streams/streams.c php5-200409011830/main/streams/streams.c
--- org/php5-200409011830/main/streams/streams.c        2004-08-31 14:30:59.000000000 +0200
+++ php5-200409011830/main/streams/streams.c    2004-09-01 22:06:11.579747624 +0200
@@ -1276,7 +1276,12 @@
                }
        }
  
-       if (php_stream_mmap_possible(src)) {
+    /*
+    ** We shouldn't use memory mapping on big files just for copying them.
+    ** It's not an efficient sollution for the vm. So for files above
+    ** 32mb (32*1024^2=33554432) we want a normal copy
+    */
+       if ((php_stream_mmap_possible(src)) && (ssbuf.sb.st_size < 33554432)) {
                char *p;
                size_t mapped;
----------------------
 [2004-09-02 08:19 UTC] derick@php.net 
I saw a similar (but better) patch by Wez. Wez, can you have a look?
 [2004-09-02 14:53 UTC] wez@php.net 
IIRC, I added a limit for this kind of thing for readfile() (aka php_stream_passthru), which could be applied here.

Discussions on the list (with Andi) suggested mmap'ing chunks of 1MB and doing it that way.

If someone wants to cook up a patch for either of these, that's fine by me; I'm not sure how soon I'll be able to look into it.
 [2004-09-10 23:53 UTC] ben at prince dot org 
Just so we can be clear, while I understand this affects readfile() as well as copy(), does it also affect include()?
 [2005-02-23 19:37 UTC] iliaa@php.net 
This bug has been fixed in CVS.

Snapshots of the sources are packaged every three hours; this change
will be in the next snapshot. You can grab the snapshot at
http://snaps.php.net/.
 
Thank you for the report, and for helping us make PHP better.
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved. 		Last updated: Tue Apr 30 16:01:29 2024 UTC