php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #22721 _php_stream_copy_to_mem performance sucks on systems without mmap support
Submitted: 2003-03-15 00:38 UTC Modified: 2003-04-28 08:48 UTC
From: shawn at rumental dot com Assigned: wez
Status: Closed Package: Performance problem
PHP Version: 4.3.2-RC OS: win2k
Private report: No CVE-ID:
 [2003-03-15 00:38 UTC] shawn at rumental dot com
I just upgraded from 4.2.2 to 4.3.1(zip) in win2k.
apache - isapi

I replaced the ini file with newer one.

most scipts i have run fine if not faster.

I have a couple scripts which call preg-replace()on 
15 key pattern match/replace array, 

each array match occuring at most 10 times per line of a text file. testing with a foreach(5 lines).
(parsing text files to reformat).

These scripts take 10-20 x longer to execute on 4.3.1 i can switch back to 4.2.2 and execution time is back to normal(2 seconds).

tried with new and old ini.

no errors are reported other than max execution time when its not set high enough.

futher debugging is possible upon request, I dont know how at this point.

php_info() snapshots, ini's, code are available.



Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2003-03-15 01:30 UTC] shawn at rumental dot com
RETRACTION - problem seems to have to do with file() and size of the file being read.

sorry for inconveinence on my stupidities part.

will further research and post new report if necessary.
 [2003-03-17 08:31 UTC] shawn at rumental dot com
in version 4.3.1 file() seems to be extremly slow 10-100x as slow. a serious performance issue.
http://bugs.php.net/bug.php?id=19971
also refers to this problem but has been closed with incorrect information it seems.
 [2003-03-17 08:32 UTC] shawn at rumental dot com
changed summary - reopened
 [2003-03-17 09:45 UTC] sniper@php.net
Please try using this CVS snapshot:

  http://snaps.php.net/php4-STABLE-latest.tar.gz
 
For Windows:
 
  http://snaps.php.net/win32/php4-win32-STABLE-latest.zip


Also, add a short but complete example script here which 
can be used to reprodude this problem.

 [2003-03-17 22:11 UTC] shawn at rumental dot com
EXAMPLE CODE
-------------------------------------------
set_time_limit ('200');

# logfile is 4mb text file with 17370 LINES
$file="test.log";
$times=10;

$i=1;
while($i <=$times){
$mt = explode(' ', microtime()); $script_start_time = $mt[0] + $mt[1];
file($file);
$mt = explode(' ', microtime()); $script_end_time = $mt[0] + $mt[1];
$time[]=round($script_end_time - $script_start_time, 5);

echo 'file loaded in ', round($script_end_time - $script_start_time, 5), ' seconds<br>';

$i++;
}

$average=array_sum($time)/$times;
echo "average time - ".$average;

---------------------------------------------------
RESULTS:

USING 4.3.1
------------
file loaded in 0.47855 seconds
file loaded in 0.74892 seconds
file loaded in 0.7595 seconds
file loaded in 0.58205 seconds
file loaded in 0.82617 seconds
file loaded in 0.83293 seconds
file loaded in 0.86725 seconds
file loaded in 0.90333 seconds
file loaded in 0.63803 seconds
file loaded in 0.82538 seconds
average time - 0.746211

LATEST STABLE
--------------
file loaded in 16.18174 seconds
file loaded in 16.88304 seconds
file loaded in 16.68356 seconds
file loaded in 16.49004 seconds
file loaded in 16.42378 seconds
file loaded in 16.5862 seconds
file loaded in 16.80012 seconds
file loaded in 16.13519 seconds
file loaded in 25.35214 seconds
file loaded in 32.71799 seconds
average time - 19.02538

results are similar even with smaller files
 [2003-03-18 06:53 UTC] shawn at rumental dot com
typo - "USING 4.3.1 should be USING 4.2.2"
my bad
 [2003-03-18 07:56 UTC] wez@php.net
Too many reallocs makes for a slow file() implementation.
PHP5 will have much better performance here under windows with the new memory mapping support (and will actually have *faster* performance).
I am working on a fix for 4.3.x now.
 [2003-03-18 08:49 UTC] wez@php.net
This bug has been fixed in CVS.

In case this was a PHP problem, snapshots of the sources are packaged
every three hours; this change will be in the next snapshot. You can
grab the snapshot at http://snaps.php.net/.
 
In case this was a documentation problem, the fix will show up soon at
http://www.php.net/manual/.

In case this was a PHP.net website problem, the change will show
up on the PHP.net site and on the mirror sites in short time.
 
Thank you for the report, and for helping us make PHP better.

Fix will be in the next set of snapshots.

FYI, my benchmarks here have the following results (under winxp):

Benchmarking PHP 4.3.0 (without the fix)
Testing file() with 10 iterations with file size: 3323720
file_test(10): 0:29

Benchmarking PHP 5.0.0-dev (using memory mapping)
Testing file() with 10 iterations with file size: 3323720
file_test(10): 0:03

Benchmarking PHP 5.0.0-dev (NOT using memory mapping)
Testing file() with 10 iterations with file size: 3323720
file_test(10): 0:05

This last one is equivalent to the PHP 4.3.x code; I don't have 4.3.x win32 build env set up here, so this will have to do.


 [2003-03-18 19:14 UTC] shawn at rumental dot com
latest win32 Stable (4.3.x-dev)
Built On: Mar 18, 2003 17:30 GMT
http://snaps.php.net/win32/php4-win32-STABLE-200303181730.zip
still yields the same results as before. no change

although Latest CVS (5.0.x-dev) is fast as shit.
 [2003-03-18 19:39 UTC] wez@php.net
This bug has been fixed in CVS.

In case this was a PHP problem, snapshots of the sources are packaged
every three hours; this change will be in the next snapshot. You can
grab the snapshot at http://snaps.php.net/.
 
In case this was a documentation problem, the fix will show up soon at
http://www.php.net/manual/.

In case this was a PHP.net website problem, the change will show
up on the PHP.net site and on the mirror sites in short time.
 
Thank you for the report, and for helping us make PHP better.

Thanks for testing this; one line of the old code was accidentally left in the 4.3 branch; human error (mine).

I've tested this using PHP 4.3; benchmarked at 4 seconds (using same script and file as mentioned previously).

The next snapshot will have the fix.
 [2003-03-18 19:44 UTC] shawn at rumental dot com
Benchmarking PHP 4.2.2
Testing file() with 10 iterations with file size: 4096226
file_test(10): 0.7458 secs

Benchmarking PHP 4.3.2-RC
Testing file() with 10 iterations with file size: 4096226
file_test(10): 13.5399 secs
 [2003-03-18 19:47 UTC] shawn at rumental dot com
woooaahhh lots of warnings in that compile log.
think ill have to wait.
thx will test tomorrow
 [2003-03-19 19:11 UTC] shawn at rumental dot com
Benchmarking PHP 4.3.2-RC
Testing file() with 10 iterations with file size: 4096226
file_test(10): 0.8426 secs

WORKS GREAT !
 [2003-03-20 21:04 UTC] shawn at rumental dot com
BIG PROBLEM!
it seems as though opened files via file()
are staying open and preventing service from openeing new files giving errors.

I checked this with sysineternals utility called handle
 [2003-03-25 18:49 UTC] shawn at rumental dot com
IS THIS BUG STILL BEING WATCHED.
I reopened it there still seems to be a problem with the fix.
 [2003-03-25 18:59 UTC] wez@php.net
What is the problem?
We need a script to reproduce the problem, along with a description of what should happen, what does happen and why you think it is wrong.
 [2003-03-26 18:47 UTC] shawn at rumental dot com
afer running file() about a 2000 times with latest snapshot in win2k running apache 1.3.27 latest win32 binary

i get this error:
Warning: file(e:\test.txt): failed to open stream: Too many open files in e:\program files\apache group\apache\htdocs\test\testfile2.php on line 12

I used handle.exe from sysinternals to view open file handles
http://www.sysinternals.com/ntw2k/freeware/handle.shtml

and all file handles opened via file() appear to still be 
open. This is not the case when testing php 422

output from handle.exe:
Apache.exe pid: 1640 MAINFRAME\apache
  1e4: File          E:\test.txt
  1e8: File          E:\test.txt
  210: File          E:\test.txt
  ...  etc
  214: File          E:\test.txt
  
and it goes on for the number of times file() accessed the file.

CODE
----
$i=1;
while($i++ <="2000"){
file("e:\\test.txt");
}
 [2003-04-25 05:43 UTC] wez@php.net
I must verify/fix this before 4.3.2 goes gold.
 [2003-04-25 07:43 UTC] shawn at rumental dot com
Surely let me know if you need assistance verifying or need any logs or more info.

I dont beleive performance monitors reflect the files open.
But handle.exe from sysinternals http://www.sysinternals.com/ntw2k/freeware/handle.shtml
does indicate that all handles are still open.

I do not know if there is a way in php to check for that.

I am still using 4.22 because 4.23 will stop when total file handles reach a certain number.
 [2003-04-28 08:48 UTC] wez@php.net
Unable to reproduce any sign of a leak using the latest stable snapshot under XP and w2k.
Are you sure you correctly upgraded your PHP installation? (make sure you don't have any stale php dlls)
 
PHP Copyright © 2001-2014 The PHP Group
All rights reserved.
Last updated: Wed Apr 23 17:01:58 2014 UTC