There is a realpath cache that gets populated, so you need to warm up the cache 
and not strace the first request.  You shouldn't care that the first request to 
the server takes a few more stats.  The real test is whether subsequent requests 
are slow.  If you are seeing excessive stats with the caches warmed up, then you 
need to investigate the size of your realpath cache in your php.ini and possibly 
increase it.  This part has been made more efficient in PHP 5.3.

It would also be a good idea to turn off AllowOverride in your Apache config to 
get rid of the .htaccess stats if you are concerned about the number of stat 
calls.

So, do this.  

Start Apache in non-forking mode by running it with -X

Then hit a simple hello_world.php script a few times.  Don't use phpinfo because 
it does a number of evil things on its own.  After hitting your hello world 
script a couple of times, attach strace to the Apache process (strace -p <pid>) 
and then hit the server again once.  At this point you should not see any extra 
stats.

Thanks for your explanation, I followed your suggestions and there is a performance improvement in submitted case.
Anyway your suggestions are not applicable in all cases.

Please consider this problem from my point of view: in a production environment of a big web hosting provider.

In this real case there are many thousands of users that can freely write their own PHP code, there also are hundreds of thousands of pages that cannot be cached.

Please consider also that what happens when Google spiders come to crawl all pages.

That's why the setting realpath_cache_size and TTL exist. They allow you to fine tune this cache to fit your needs. On a shared host you will certainly increase the default value.

We already tuned cache size to following values:

realpath_cache_size=1024k
realpath_cache_ttl=7200

Can we increase cache size to:

realpath_cache_size=40960k
realpath_cache_ttl=72000

Do you know if memory_limit is affected by realpath_cache_size increase?
Actually our memory limit is set to:

memory_limit = 96M

But biggest problem we have at moment is when search engines spiders come to crawling all platform. 
In this case all existing pages are crawled by spiders.
Can you suggest us a workaround?

I must also to notify that looking at our production servers in some cases PHP engine tries up to 8 times before read the file.

Can you explain why PHP engine have this behavior?
There is any way to remove/change this behavior in PHP engine?

Again, it does it once and only once per path. When it does it, it checks each element of a path (and cache each of them too).

And like I said, we have made this more efficient in PHP 5.3 because we now cache 
the partial paths separately.  You should see a performance improvement going to 
5.3.

@pajoye; as is described in "Actual result" part of this bug, first time PHP engine check filepath there is a full scan of all directories at least 4 
times before cache the result, this behavior anyway cannot scale in a so large environment.

@rasmus: this is very interesting, I can consider upgrade to 5.3 in order to avoid this behavior.

I know this is an extrema ratio: may I disable symlinks support from PHP engine in order to avoid this behavior?

Is there an answer regarding the increase of realpath_cache_size realpath_cache_ttl?

The scan of each element of the paths happen anyway, whether the paths contain symlinks or not. Check the code TSRM/ for a deeper explanation.

The real problem is that such full scan are repeated more then once:

25933 lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
25933 lstat("/usr/local", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
25933 lstat("/usr/local/myspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
25933 lstat("/usr/local/myspace/webspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
25933 lstat("/usr/local/myspace/webspace/httpdocs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
25933 lstat("/usr/local/myspace/webspace/httpdocs/phpinfo.php", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0

First check

25933 lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
25933 lstat("/usr/local", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
25933 lstat("/usr/local/myspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
25933 lstat("/usr/local/myspace/webspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
25933 lstat("/usr/local/myspace/webspace/httpdocs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
25933 lstat("/usr/local/myspace/webspace/httpdocs/phpinfo.php", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0

Second check

25933 lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
25933 lstat("/usr/local", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
25933 lstat("/usr/local/myspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
25933 lstat("/usr/local/myspace/webspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

Third check (incomplete)

25933 lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
25933 lstat("/usr/local", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
25933 lstat("/usr/local/myspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
25933 lstat("/usr/local/myspace/webspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
25933 lstat("/usr/local/myspace/webspace/httpdocs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
25933 lstat("/usr/local/myspace/webspace/httpdocs/phpinfo.php", {st_mode=S_IFREG|0644, st_size=16, ...}) = 0

Final check and then read the file.

25933 open("/usr/local/myspace/webspace/httpdocs/phpinfo.php", O_RDONLY) = 16
25933 fstat(16, {st_mode=S_IFREG|0644, st_size=16, ...}) = 0
25933 read(16, "<? phpinfo() ?>\n", 8192) = 16
25933 read(16, "", 8192)                = 0
25933 read(16, "", 8192)                = 0
25933 close(16)                         = 0

phpinfo() is a special case.  Try it with a simple hello world script.

I tried again with a simple script test.php '<? echo "Ciao" ?>'
realpath_cache_size=1024k
realpath_cache_ttl=600

I have always same behavior (also second and following times I try to call curl http://damorealt.xoom.it/test.php:

13692 chdir("/usr/local/myspace/webspace/httpdocs") = 0
13692 lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr/local", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr/local/myspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr/local/myspace/webspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr/local/myspace/webspace/httpdocs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr/local/myspace/webspace/httpdocs/test.php", {st_mode=S_IFREG|0644, st_size=19, ...}) = 0
13692 lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr/local", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr/local/myspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr/local/myspace/webspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr/local/myspace/webspace/httpdocs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr/local/myspace/webspace/httpdocs/test.php", {st_mode=S_IFREG|0644, st_size=19, ...}) = 0
13692 lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr/local", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr/local/myspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr/local/myspace/webspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr/local", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr/local/myspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr/local/myspace/webspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr/local/myspace/webspace/httpdocs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
13692 lstat("/usr/local/myspace/webspace/httpdocs/test.php", {st_mode=S_IFREG|0644, st_size=19, ...}) = 0
13692 open("/usr/local/myspace/webspace/httpdocs/test.php", O_RDONLY) = 16
13692 fstat(16, {st_mode=S_IFREG|0644, st_size=19, ...}) = 0
13692 read(16, "<? echo \"Ciao!\" ?>\n", 8192) = 19
13692 read(16, "", 8192)                = 0
13692 read(16, "", 8192)                = 0
13692 close(16)                         = 0
13692 chdir("/")                        = 0

Can you try with 5.3 please?

I have already tried with 5.3 and I was thinking to open an new bug because I have same behavior, there is only a change in the order of execution of lstat from fullpath to /usr :

7339  chdir("/usr/local/myspace/webspace/httpdocs") = 0
7339  lstat("/usr/local/myspace/webspace/httpdocs/test.php", {st_mode=S_IFREG|0644, st_size=19, ...}) = 0
7339  lstat("/usr/local/myspace/webspace/httpdocs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  lstat("/usr/local/myspace/webspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  lstat("/usr/local/myspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  lstat("/usr/local", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  lstat("/usr/local/myspace/webspace/httpdocs/test.php", {st_mode=S_IFREG|0644, st_size=19, ...}) = 0
7339  lstat("/usr/local/myspace/webspace/httpdocs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  lstat("/usr/local/myspace/webspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  lstat("/usr/local/myspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  lstat("/usr/local", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  lstat("/usr/local/myspace/webspace/httpdocs/test.php", {st_mode=S_IFREG|0644, st_size=19, ...}) = 0
7339  lstat("/usr/local/myspace/webspace/httpdocs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  lstat("/usr/local/myspace/webspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  lstat("/usr/local/myspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  lstat("/usr/local", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  lstat("/usr/local/myspace/webspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  lstat("/usr/local/myspace", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  lstat("/usr/local", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  lstat("/usr", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
7339  open("/usr/local/myspace/webspace/httpdocs/test.php", O_RDONLY) = 16
7339  fstat(16, {st_mode=S_IFREG|0644, st_size=19, ...}) = 0
7339  fstat(16, {st_mode=S_IFREG|0644, st_size=19, ...}) = 0
7339  fstat(16, {st_mode=S_IFREG|0644, st_size=19, ...}) = 0
7339  mmap(NULL, 19, PROT_READ, MAP_SHARED, 16, 0) = 0x2adfec195000
7339  munmap(0x2adfec195000, 19)        = 0
7339  close(16)                         = 0
7339  chdir("/")                        = 0

I don't see that here.  Attach gdb and set a breakpoint on lstat and see who is 
calling them.

where I have attach gdb, on 5.2.13 or on 5.3.2 ?

5.3.  I haven't looked at 5.2 in 2+ years.

gdb reproduce the behavior reported in my comments, I have uploaded result of your request at 
http://damore.xoom.it/apache-2.2_php-5.3.2_break-lstat.txt

It seems like your realpath cache isn't working at all.

Could you set a breakpoint on realpath_cache_find and step into it the second time 
it hits those stat calls.  Does it go into the while(bucket) loop there at all?

Actually, try this simple test.php script:

<pre>
<?php
var_dump(realpath_cache_get());
?>
</pre>


That should tell you what is getting cached.

Please note, gdb output I sent is related to third time I have called curl http://damorealt.xoom.it/test.php

I have set break point as requested:

(gdb) break realpath_cache_find
Breakpoint 1 at 0x2b0b3c9f254d: file /usr/local/sitipersonali/sitipersonali01/NSP_SERVICE/strillo/sources/php-5.3.2/TSRM/tsrm_virtual_cwd.c, line 560.
(gdb) bt
#0  0x000000391760db00 in __accept_nocancel () from /lib64/libpthread.so.0
#1  0x00002b0b3c1f4544 in apr_socket_accept () from /usr/lib64/libapr-1.so.0
#2  0x00000000004562e5 in unixd_accept ()
#3  0x0000000000454006 in child_main ()
#4  0x000000000045416c in make_child ()
#5  0x0000000000454701 in ap_mpm_run ()
#6  0x00000000004220ef in main ()
(gdb) continue
Continuing.

During test call gdb as not broken execution.
Function realpath_cache_find is not called.

I have executed script:

<pre>
<?php
var_dump(realpath_cache_get());
?>
</pre>

[root@svilpar4 ~]# curl http://damorealt.xoom.it/test.php
<pre>
array(0) {
}
</pre>

Please also note that realpath_cache_size and realpath_cache_ttl are currently set to following values:

[root@svilpar4 ~]# curl http://damorealt.xoom.it/phpinfo.php | grep real

<tr><td class="e">realpath_cache_size</td><td class="v">1024k</td><td class="v">1024k</td></tr>
<tr><td class="e">realpath_cache_ttl</td><td class="v">600</td><td class="v">600</td></tr>

Set a bp and step through tsrm_realpath_r and figure out why it isn't getting to 
the realptath_cache_find() call there.  Seems like it should be getting there from 
the backtraces.

After having set breakpoint tsrm_realpath_r and I have execute step by step debug.
I think is interesting that after execution tsrm_virtual_cwd.c of line 681 execution continue on line 890.

gdb) break tsrm_realpath_r
Breakpoint 1 at 0x2b0b3c9f2702: file /usr/local/sitipersonali/sitipersonali01/NSP_SERVICE/strillo/sources/php-5.3.2/TSRM/tsrm_virtual_cwd.c, line 611.
(gdb) continue
Continuing.

Breakpoint 1, tsrm_realpath_r (path=0x7fffddfb32b0 "/usr/local/myspace/webspace/httpdocs/test.php", start=1, len=45, ll=0x7fffddfb32ac, t=0x7fffddfb32a0, use_realpath=2, is_dir=0, link_is_dir=0x0)
    at /usr/local/sitipersonali/sitipersonali01/NSP_SERVICE/strillo/sources/php-5.3.2/TSRM/tsrm_virtual_cwd.c:611
611		int directory = 0;
(gdb) step
624			if (len <= start) {
(gdb) step
628			i = len;
(gdb) step
629			while (i > start && !IS_SLASH(path[i-1])) {
(gdb) step
630				i--;
(gdb) step
629			while (i > start && !IS_SLASH(path[i-1])) {
(gdb) step
630				i--;
(gdb) step
629			while (i > start && !IS_SLASH(path[i-1])) {
(gdb) step
630				i--;
(gdb) step
629			while (i > start && !IS_SLASH(path[i-1])) {
(gdb) step
630				i--;
(gdb) step
629			while (i > start && !IS_SLASH(path[i-1])) {
(gdb) step
630				i--;
(gdb) step
629			while (i > start && !IS_SLASH(path[i-1])) {
(gdb) step
630				i--;
(gdb) step
629			while (i > start && !IS_SLASH(path[i-1])) {
(gdb) step
630				i--;
(gdb) step
629			while (i > start && !IS_SLASH(path[i-1])) {
(gdb) step
630				i--;
(gdb) step
629			while (i > start && !IS_SLASH(path[i-1])) {
(gdb) step
633			if (i == len ||
(gdb) step
639			} else if (i == len - 2 && path[i] == '.' && path[i+1] == '.') {
(gdb) step
677			path[len] = 0;
(gdb) step
679			save = (use_realpath != CWD_EXPAND);
(gdb) step
681			if (start && save && CWDG(realpath_cache_size_limit)) {
(gdb) watch save
Hardware watchpoint 2: save
(gdb) print save
$1 = 1
(gdb) print start
$2 = 1
(gdb) print realpath_cache_size_limit
No symbol "realpath_cache_size_limit" in current context.
(gdb) step
890			if (save && lstat(path, &st) < 0) {
(gdb)

There is a interesting update, I have found CWDG define so now we have:

(gdb) print cwd_globals.realpath_cache_size_limit
$3 = 0

Probably you should check why realpath_cache_size_limit is equal to 0

I found where the problem is, this behavior is not a bug.
Looking at main/main.c I found following lines:

1416:           /* Disable realpath cache if safe_mode or open_basedir are set */
                if (PG(safe_mode) || (PG(open_basedir) && *PG(open_basedir))) {
                        CWDG(realpath_cache_size_limit) = 0;
                }

1978:	/* Disable realpath cache if safe_mode or open_basedir are set */
        if (PG(safe_mode) || (PG(open_basedir) && *PG(open_basedir))) {
                CWDG(realpath_cache_size_limit) = 0;
        }

Could you explain why if safe_mode or open_basedir are set realpath cache is disabled?

Looking at source code main/main.c of 5.2.13 I can see:

1292:		/* Disable realpath cache if safe_mode or open_basedir are set 
*/
                if (PG(safe_mode) || (PG(open_basedir) && *PG(open_basedir))) {
                        CWDG(realpath_cache_size_limit) = 0;
                }

1769:	/* Disable realpath cache if safe_mode or open_basedir are set */
        if (PG(safe_mode) || (PG(open_basedir) && *PG(open_basedir))) {
                CWDG(realpath_cache_size_limit) = 0;
        }

So realpath cache is definitely disabled in case of safe_mode or open_basedir.
This dramatically reduce performance of PHP Engine and this behavior can bring a 
server to its knees.
Especially because there is a lack of documentation!
Can you explain why this choose?
Must I continue debugging PHP engine in order to understand what's happening?

The reason was due to a security flaw involving symbolic links and realpath cache. It allowed to bypass open_basedir when a path was cached. The cleanest way to fix it was to disable the realpath cache when open_basedir/safemode are set.

Thanks Johannes to remind us about this change.

If you take a look at debugging done in http://damore.xoom.it/apache-2.2_php-5.3.2_break-lstat.txt, you can see that tsrm_realpath_r is called in 2 different place:

- First during phar_find_in_include_path function 
- Second during php_check_specific_open_basedir function.

* During php_check_specific_open_basedir, traverse entire path at least three times, so i means 3 call to tsrm_realpath_r.

I don't discuss the need to traverse entire path, but only one time.

There should be a better way to implement such code, I'll try to write a patch if there is someone that want help me looking what I'm writing.

Please pay attention correct URL for debugging info is 
http://damore.xoom.it/apache-2.2_php-5.3.2_break-lstat.txt

changed summery in order to provide a better description

Yup, there are clearly some inefficiencies here.  We are relying too much on the 
realpath_cache being enabled.  You can eliminate the phar one by eliminating phar 
for now.  Recompile with --disable-phar until we get this cleaned up.

I want make it clear first that: in submitted test we can see only 4 directory scan, I my production environment I have seen up to 8 directory scan for php file. 

If we disable PHAR I suppose only first full scan path will be removed.

So other 3 full scan path remain there, probably we need to change php_check_specific_open_basedir in order to really try to remove inefficiencies.

Hello,

I have recompiled php commenting both in main/main.c:

/*
if (PG(safe_mode) || (PG(open_basedir) && *PG(open_basedir))) {
                        CWDG(realpath_cache_size_limit) = 0;
}
*/

I have defined into TSRM/tsrm_virtual_cwd.c

#define realpath(x,y) strcpy(y,x)

and I have disabled PHP function symlink.

Now I finally not have performance problems, but I suppose I still can have 
security problem.

Can you help me in order to understand if there is a better solution?

As a matter of fact, same performance problem occurs with PHP Version 5.3.3 on Debian Squeeze.

On our test plateform (with an SSD disk) a small bench show that the penalties of using open_basedir is more than small on real case application.

Base configuration :
ab -n200 -c20 http://benchb.xxx.yy/
Requests per second:    266.45 [#/sec] (mean)
After open_basedir activation :
Requests per second:    82.95 [#/sec] (mean)

Recompiling with --disable-phar doesn't reduce the performance gap.
Recompiling after patching main/main.c with comment on both occurences of :
//CWDG(realpath_cache_size_limit) = 0;

allows to reach an acceptable performance :
Requests per second:    178.27 [#/sec] (mean)

By increasing the realpath cache, we can at last reach a smaller penalty :
Requests per second:    220.69 [#/sec] (mean)

Our tries tu use open_basedir on NFS leads us to more dramatic situation.

Can we expect a fix for a better situation in further PHP versions ?
Maybe this realpath cache de-activation could be a default setting in case of open_basedir with the ability of re-activating without need of recompiling.

I know this can be a security hole, but maybe it's better to give an intermediate choice between on/off. 

I'm affraid that in the current situation, activating open_basedir is not really an acceptable choice due to the performance struggling, and I think that's worse for security.

You can simply use this PHP extension to bypass default security checks mentioned in comments above.

http://php.webtutor.pl/en/2011/06/02/running-php-on-nfs-huge-performance-problems-and-one-simple-solution/

Hello,

Not much more than a "me too", sorry.  Is there any plan in the works to make php 
both safe in a mass hosting setup as well as not take a big performance hit when 
running webapps with a large number of "require" and "include" functions?  I'm 
running php 5.3.6 and still seeing a huge amount of cpu time spent in "system" on 
common web apps like Joomla, Drupal and C5.  Not seeing a clear solution that 
works well on a shared hosting setup.

I really don't see a middle ground here. You are either secure or you aren't. Caching open_basedir stats is insecure and the whole point of open_basedir in a shared hosting setup is to secure these file accesses. If you don't care about security, turn it off and live with the security issue, or better yet, change your shared hosting setup to use VMs or other lower-level strategies that keep users separated from each other.

> Caching open_basedir stats is insecure

not really because the permissions are not changed the whole day

How about having concurrent cache tables for each basedir setting? For instance, when open_basedir is set to '/home/teh1234;/tmp', then the lstat populates only cache table "0", realpath_cache_* also uses exclusively this cache, and when open_basedir is set to '/home/klaczy;/tmp' then it populates and uses only cache table "1"? Are there any security considerations that I don't notice here?

This bug is extant in 5.3 and 5.4 (up to 5.4.9).  The flaw is the logic in main/main.c:php_module_startup():

	/* Disable realpath cache if safe_mode or open_basedir are set */
	if (PG(safe_mode) || (PG(open_basedir) && *PG(open_basedir))) {
		CWDG(realpath_cache_size_limit) = 0;
	}

Similar logic validly exists in code such as suEXEC and suPHP to prevent a race condition allowing and unscrupulous script author to execute a script in another UID in certain circumstances.  This simply doesn't apply in this case.  Which technically a similar race could allow a script author to break the basedir contain with miniscule probability by exploiting such a race, the performance impacts are significant, for no material increase in security strength.  Get rid of this rule or at least split open_basedir into two separate directives open_basedir_paranoid which disables the cache as above and open_basedir which does the rational checks and doesn't disable the cache.

This has been lying around for nearly 3 years.  How can we progress this?

There is no such thing as a "miniscule race" when it comes to computers. Either 
there is a race condition or there isn't. In this case there is. So if we remove 
this check open_basedir will be much less secure. Something along the lines of 
Marcin's idea of separate caches might be feasible, but this is not a small 
change.

For the purposes of this bug, let's document the advisory which triggered this change: CVE-2006-5178 see http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2006-5178.

Let's be honest, even with this patch there is still a minuscule chance that the exploit described could succeed, because we can't go into kernel mode during the recursive descent and prevent other processes with a higher nice being scheduled in and doing the dirty. So it replaces "very small" by "even smaller" at the same time killing NFS performance.

As I said don't remove this change, just make it elective by adding an extra variant of the open_basedir parameter, and openly documenting the issues so that sysadmins can make their own judgement call on security vs performance.  Let's face it on a typical shared service I can think of easier ways to do this exploit.  E.g. if exec and the proc functions aren't disabled then there is no point in having a strong open_basedir.  If you are an enterprise and just want to lock down your apps programmers from breaching good practice / infrastructure standards over a NAS infrastructure then the weak form is perfectly appropriate.

Also IMO, PHP is trying to do a better job here than the standard Linux realpath function and failing.

Incidentally I've just straced opening a require_once on 

   /home/terry/work/ext/lpc/tests/dd2.inc

and PHP 5.3.17 does the 7 x lstat walk that you indicate is necessary to meet CVE-2006-5178, but it then repeats this another 5 times and does another 3 fstats after the file is opened 45 f/lstats all in 2.5μS (on a local disk clearly).

PHP 5.4.6 is worse, it does 5x7 walks, opens the file  then does 4 fstatsand reads the file, before doing another 3x7 lstat walks and a last stat for luck, giving a grand total of 60 f/lstats in 4μS though this one includes mmapping the contents from the file cache.

OK, I can understand -- though question -- mandating one walk but why do it seven times ????

I don't know what your dd2.inc is doing, but I went through and optimized syscalls a 
couple of years ago, and in normal no-openbasedir mode things are pretty clean. Here 
are a few scenarios:

a.php does a require on ./b.php and a require_once on ./c.php

PHP 5.4/APC-HEAD/apc.stat=1/apc.include_once_override=0/open_basedir=off

stat("/var/www/a.php", {st_mode=S_IFREG|0664, st_size=49, ...}) = 0
stat("/var/www/./b.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0
open("/var/www/c.php", O_RDONLY)        = 29
fstat(29, {st_mode=S_IFREG|0664, st_size=10, ...}) = 0
fstat(29, {st_mode=S_IFREG|0664, st_size=10, ...}) = 0
fstat(29, {st_mode=S_IFREG|0664, st_size=10, ...}) = 0
stat("/var/www/c.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0

The initial stat on a.php is actually done by Apache and we ask Apache for the stat 
struct to avoid an extra stat there. You can see the effect of the open and then the 
(quick) fstat calls on fd=29. This is where APC tries to optimize things a bit using 
the include_once_override. If we turn that on we get:

stat("/var/www/a.php", {st_mode=S_IFREG|0664, st_size=49, ...}) = 0
stat("/var/www/./b.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0
stat("/var/www/./c.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0
stat("/var/www/./c.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0

Not ideal since there is still an extra stat for the require_once case, but the open() 
is gone. And you can eliminate those stats by turning off apc.stat which means only 
the initial top-page stat from Apache on a.php will be done.

Now, of course, if we turn on open_basedir we get:

www-data  1799  0.8  0.2 310232 22560 pts/6    S+   19:26   0:00 /usr/sbin/apache2 -X
root      1887  0.0  0.0  10892   916 pts/12   R+   19:26   0:00 grep apache
7:26pm x220:~> strace -o out -p 1799
Process 1799 attached - interrupt to quit
^CProcess 1799 detached
7:26pm x220:~> egrep "stat|open" out
stat("/var/www/a.php", {st_mode=S_IFREG|0664, st_size=49, ...}) = 0
lstat("/var/www/./b.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0
lstat("/var/www", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/var/www/b.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0
lstat("/var/www", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/var/www", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/var/www/./b.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0
lstat("/var/www/./c.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0
lstat("/var/www", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/var/www/c.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0
lstat("/var/www", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/var/www", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/var/www/./c.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0
lstat("/var/www/./c.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0
lstat("/var/www", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/var/www/c.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0
lstat("/var/www", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/var/www", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/var/www/./c.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0

which is far from ideal, of course, because we are not hitting our realpath cache at 
all and there is an oddity there with ./ getting resolved and re-statted and this re-
realpathed. Not sure I see where your 7 times thing is though.

With ZO+ with this config

zend_optimizerplus.revalidate_freq=0
zend_optimizerplus.enable_file_override=0
open_basedir=

we see this:

stat("/var/www/a.php", {st_mode=S_IFREG|0664, st_size=49, ...}) = 0
stat("/var/www/a.php", {st_mode=S_IFREG|0664, st_size=49, ...}) = 0
stat("/var/www/b.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0
stat("/var/www/c.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0

Here we see that ZO+ doesn't have the Apache stat optimization that APC has, which is 
something we should obviously add, but it handles the require_once better and the 
open() call and fstats are gone.

When we turn on open_basedir we get:

stat("/var/www/a.php", {st_mode=S_IFREG|0664, st_size=49, ...}) = 0
lstat("/var/www/a.php", {st_mode=S_IFREG|0664, st_size=49, ...}) = 0
lstat("/var/www", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/var/www/a.php", {st_mode=S_IFREG|0664, st_size=49, ...}) = 0
lstat("/var/www/./b.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0
lstat("/var/www", {st_mode=S_IFDIR|0777, st_size=4096, ...}) = 0
lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/var/www/b.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0
stat("/var/www/c.php", {st_mode=S_IFREG|0664, st_size=10, ...}) = 0

Better than APC, still not great, but there are no open() calls here, so the NFS 
getattr issue should be gone.

And finally, ZO+ has a TTL version of apc.stat so we can set it to only stat every 60 
seconds, so even with open_basedir enabled, with this config:

zend_optimizerplus.revalidate_freq=60
zend_optimizerplus.enable_file_override=1

we get this:

stat("/var/www/a.php", {st_mode=S_IFREG|0664, st_size=49, ...}) = 0

which is just the initial stat done by Apache which we can't do anything about. So, in 
general, assuming the 5.5 integrated opcode cache is based on ZO+ and we add the one 
missing Apache stat optimization, I really don't think the situation is all that dire, 
even for NFS-mounted scripts.

The above discussion was largely about the I/O overheads with open_basedir specified, so my figures where in that context, and dd2.inc is just an empty class, but this isnt relevant to its inclusion. Try:

   echo "">dummy.inc
   pwd
   strace -tt -o /tmp/strace.log \
      php -d  realpath_cache_ttl=9999 -d open_basedir=$(pwd) \
          -r '$x="./dummy.inc"; require_once($x);'

to see what I mean (I did this from 6 dir levels down which is pretty typical of web set ups.) since you can do this yourself I won't dump the log output but:

   for vb in lstat fstat stat open; do 
       echo -n "$vb  "
       sed -ne '/dummy.inc/,/close(/p' /tmp/strace | grep -c " $vb"
   done

   lstat  54
   fstat  5
   stat  1
   open  1   (*) I removed the open for "/etc/ld.so.cache" which isn't relevant here

whereas dropping the open_basedir directive and repeating gives:

   lstat  8
   fstat  5
   stat  1
   open  1

so the open_basedir can have a severe impact on performance. I am still at loss as to why the PHP cache is disabled if open_basedir is set. Surely the security objectives would be entirely achieved by doing the real walk only during the open routine itself?

I realise that NFS tuning can mitigate these issues -- e.g. using noatime and setting the actimeo,... parameters or even using a local fscache (in Ubuntu this is the cachefilesd package).

I agree that running mod_php5 with APC or O+ help a lot here, but as we've discussed in the past, APC and O+ do not support php-cgi or php-cli. (I know that APC has apc.enable_cli, but this seems to be a functional no-op.)  We should have acceptable PHP performance over all common usecases.

First-request cli isn't going to have a populated realpath cache no matter what 
we do since this cache is per-process and in no way shared nor persistent across 
different processes.

Yes Rasmus.  We both know that; but this won't be address without something like an LPC-style file-based cache to preserve context across image activations, but all this isn't that relevant to #52312 -- "PHP safe_mode/open_basedir - lstat performance problem".

What is relevant are my points about a require_once 6 sub-directories down taking 13 stats and 1 open with open_basedir unset and 60 stats and 1 open if it is set, and that the security requirement could still be implemented within the former stat number if done correctly. 

This isn't a material problem for single source file scripts, but MediaWiki, Wordpress and the like typically load in ~100 modules generating ~6K stats per request.  And this does become a problem.

Rasmus, picking up our 2013-02-22 23:17 / 23:26 UTC conversation, I've thought about this some further and gone through some test cases on the debugger.

Having walked through these call stacks, my view is that the PHP / Zend path scanning, file checking and opening is a tangled mess.  A typical open goes through sometimes 7 seven wrapping layers each of which can do compound path resolution.  The only reason that this doesn't end up slugging performance is because the lowest level tsrm_realparth_r() uses a resolution cache and short-circuits the actual I/O requests 95+% of the time. 

The CVE-2006-5178 advisory really to a vulerability in the open_basedir checks.  The root issue was that a key check in php_check_specific_open_basedir() was using this cached path resolution; this introduced the vulnerability because the caching enabled a race condition.  The fix was to turn of ALL realpath resolution caching.  Yes this addressed the vulnerability, but at the price of killing performance in the typical usecase where open_basedir might be used. This was unnecessary overkill.

If you think about it realpath caching itself doesn't introduce any vulnerability.  The issue is that the open itself -- or in this case the preceding php_check_specific_open_basedir() check must disable any caching for that one check alone.  Consider an example there the base dir is /a/b/c and some path needs to be checked by php_check_specific_open_basedir().  There's not vulnerability introduced by any previous resolution being used.  This will result in one of two scenarios:

  *  The resolved path is not of the form /a/b/c.... in which case the error is thrown
  *  The resolved path is of the form /a/b/c... but the actual path might contain raced symlinks, so it must be scanned (by tsrm_realparth_r) to ensure that no links are extant immediately prior to the open.  There is also a sound case for curtailing this scan on a root-owned directory, but this of second order.

Implementing this is a small code change.
  
 (i)  First drop the open_basedir predicated clearing of CWDG(realpath_cache_size_limit) = 0 in  
main/main.c:php_module_startup().  

 (ii) introduce a per-call mechanism for disabling the cache.  This could be done by adding a flag to realpath, but the two variants (ZTS and none-ZTS) and the wide use of the wrapper VCWD_REALPATH() macro might complicate this.  An alternative would be to add another flag to the PG() structure, checking this in tsrm_realparth_r() and setting it in php_check_specific_open_basedir() around the VCWD_REALPATH(path_tmp, resolved_name) call.

Reactions / Thoughts?  Is it worth me proposing a patch?

This bug is a real performance killer.

I propose this solution...
1. Modify symlink() php function so that if open_basedir or safe_mode is on, it disallows relative symlinks with .. components, instead it creates absolute symlink.
This prevents attacker from exploiting CVE-2006-5178.

2. Allow the realpath cache to work even if open_basedir or safe_mode is on.

I did this with my own php code.

Before
---------
last pid: 30437;  load averages: 32.93, 24.67, 15.62
98 processes:  43 running, 46 sleeping, 9 lock
CPU:  2.7% user,  0.0% nice, 75.3% system,  0.5% interrupt, 21.5% idle

After
---------
last pid: 30582;  load averages:  2.06,  3.57, 10.58
68 processes:  6 running, 62 sleeping
CPU:  6.8% user,  0.0% nice,  1.7% system,  0.9% interrupt, 90.6% idle

I seem to be suffering from this bug on an Apache/Linux + nfs setup. I'm using open_basedir and the performance is so poor, that I can't even really stress test the server any more.

This problem causes extreme loading on otherwise powerful servers when using PHP7/FastCGI/IIS, so bad in fact that it's unusable for busy sites on an otherwise powerful server.

Unfortunately because this is a Windows server, the workarounds are not applicable.

I think that patch proposal is a good start to getting this fixed.

We have this issue with Symfony(3.1.3), it's really poor performance with lot of stat on file...

With the same project on Windows (IIS 7.5 - PHP 5.6.1
 With open_basedir (empty) it's take ~300-350ms for the index
 With open_basedir set (~9 path) it's take 3.5-10s for the index

I understand security problem, but why not just clear cache (before and/or after risky function)

things like that : 
--------------------
diff --git a/ext/standard/link.c b/ext/standard/link.c
index 62e7295..a844f46 100644
--- a/ext/standard/link.c
+++ b/ext/standard/link.c
@@ -158,6 +158,11 @@ PHP_FUNCTION(symlink)
 		RETURN_FALSE;
 	}
 
+	/* Reset realpath_cache when open_basedir is not null to avoid security issues */
+	if(PG(open_basedir)){
+		realpath_cache_clean();
+	}
+
 	/* For the source, an expanded path must be used (in ZTS an other thread could have changed the CWD).
 	 * For the target the exact string given by the user must be used, relative or not, existing or not.
 	 * The target is relative to the link itself, not to the CWD. */
@@ -206,6 +211,11 @@ PHP_FUNCTION(link)
 		RETURN_FALSE;
 	}
 
+	/* Reset realpath_cache when open_basedir is not null to avoid security issues */
+	if(PG(open_basedir)){
+		realpath_cache_clean();
+	}
+
 #ifndef ZTS
 	ret = link(topath, frompath);
 #else
 
diff --git a/ext/standard/link_win32.c b/ext/standard/link_win32.c
index 7d43162..e47e265 100644
--- a/ext/standard/link_win32.c
+++ b/ext/standard/link_win32.c
@@ -168,6 +168,12 @@ PHP_FUNCTION(symlink)
 		php_error_docref(NULL, E_WARNING, "UTF-16 conversion failed (error %d)", GetLastError());
 		RETURN_FALSE;
 	}
+	
+	/* Reset realpath_cache when open_basedir is not null to avoid security issues */
+	if(PG(open_basedir)){
+		realpath_cache_clean();
+	}
+
 	/* For the source, an expanded path must be used (in ZTS an other thread could have changed the CWD).
 	 * For the target the exact string given by the user must be used, relative or not, existing or not.
 	 * The target is relative to the link itself, not to the CWD. */
@@ -223,6 +229,11 @@ PHP_FUNCTION(link)
 		RETURN_FALSE;
 	}
 
+	/* Reset realpath_cache when open_basedir is not null to avoid security issues */
+	if(PG(open_basedir)){
+		realpath_cache_clean();
+	}
+
 #ifndef ZTS
 	ret = CreateHardLinkA(topath, frompath, NULL);
 #else
 
diff --git a/main/main.c b/main/main.c
index bb98f27..7e5904e 100644
--- a/main/main.c
+++ b/main/main.c
@@ -2222,7 +2222,7 @@ int php_module_startup(sapi_module_struct *sf, zend_module_entry *additional_mod
 
 	/* Disable realpath cache if an open_basedir is set */
 	if (PG(open_basedir) && *PG(open_basedir)) {
-		CWDG(realpath_cache_size_limit) = 0;
+		/* CWDG(realpath_cache_size_limit) = 0; */
 	}
 
 	/* initialize stream wrappers registry
--------------

FRANKLY: why not check if "disable_functions" contains "link" and "symlink" and in that case just use the realpath-cache as if there would not be a open_basedir setting or even to avoid the overhead of check this add a "realpath_cache_openbasedir" to config options where admins which disabled the link functions can decide for themself?

At this point I think at the very least this issue should be in the docs for each of the config sections. safe_mode, open_basedir, and realpath_cache_size sections.

open_basedir disabling the realpath cache is now documented in the related ini.core sections, see https://bugs.php.net/bug.php?id=77406

and that behavior is still plain wrong which won't be changed by any documentation - hell i am tried of such compile time crazyiness and the need to patch sources

with disable_functions="link,symlink" that security problem don't exist on shared hosting and on shared hosting where people only have access to a CMS it donÄt exist at all to begin with hence make it a PIP_INI_SYSTEM option

and without a shared memory the whole cache don't work as expected at all
https://bugs.php.net/bug.php?id=73887
https://bugs.php.net/bug.php?id=73888

as alternative to patching sources (and as it wasn't mentioned yet in this bug#) there's https://github.com/Whissi/realpath_turbo

out-of-tree extensions are always a problem when it comes to minor nd major updates even even the stuff on pecl.php.net is to avoid whenever possible

"Instead of setting open_basedir you will set realpath_turbo.open_basedir" is a crude hack which requires you to touch each and every vhost and that all because stubborn decisions for no gain

----------------------------------
[builduser@testserver:/rpmbuild/SOURCES]$ cat php-72-realpath-cache-openbasedir.patch
--- php-7.1.9-original/main/main.c      2017-08-16 18:06:53.000000000 +0200
+++ php-7.1.9-patched/main/main.c       2017-08-17 14:11:05.827653357 +0200
@@ -1646,9 +1646,9 @@
                }

                /* Disable realpath cache if an open_basedir is set */
-               if (PG(open_basedir) && *PG(open_basedir)) {
-                       CWDG(realpath_cache_size_limit) = 0;
-               }
+               /* if (PG(open_basedir) && *PG(open_basedir)) { */
+               /*      CWDG(realpath_cache_size_limit) = 0; */
+               /*}*/

                if (PG(expose_php)) {
                        sapi_add_header(SAPI_PHP_VERSION_HEADER, sizeof(SAPI_PHP_VERSION_HEADER)-1, 1);
@@ -2236,9 +2236,9 @@
 #endif

        /* Disable realpath cache if an open_basedir is set */
-       if (PG(open_basedir) && *PG(open_basedir)) {
-               CWDG(realpath_cache_size_limit) = 0;
-       }
+       /*if (PG(open_basedir) && *PG(open_basedir)) { */
+       /*      CWDG(realpath_cache_size_limit) = 0;*/
+       /*}*/

        /* initialize stream wrappers registry
         * (this uses configuration parameters from php.ini)
----------------------------------
[builduser@testserver:/rpmbuild/SOURCES]$ cat php-73-realpath-cache-openbasedir.patch
--- php-7.3.0-original/main/main.c      2018-07-31 13:33:48.000000000 +0200
+++ php-7.3.0-patched/main/main.c       2018-07-31 14:01:15.144001078 +0200
@@ -1798,9 +1798,9 @@
                }

                /* Disable realpath cache if an open_basedir is set */
-               if (PG(open_basedir) && *PG(open_basedir)) {
-                       CWDG(realpath_cache_size_limit) = 0;
-               }
+               /*if (PG(open_basedir) && *PG(open_basedir)) { */
+               /*      CWDG(realpath_cache_size_limit) = 0; */
+               /*}*/

                if (PG(expose_php)) {
                        sapi_add_header(SAPI_PHP_VERSION_HEADER, sizeof(SAPI_PHP_VERSION_HEADER)-1, 1);
@@ -2292,9 +2292,9 @@
 #endif

        /* Disable realpath cache if an open_basedir is set */
-       if (PG(open_basedir) && *PG(open_basedir)) {
-               CWDG(realpath_cache_size_limit) = 0;
-       }
+       /*if (PG(open_basedir) && *PG(open_basedir)) { */
+       /*      CWDG(realpath_cache_size_limit) = 0; */
+       /*}*/

        PG(have_called_openlog) = 0;

Hm, and long and fruitless discussion about a deliberate and
documented design design decision.  Obviously, this bug tracker
isn't suitable for this.  If anybody is still interested in seeing
this improved, please pursue the RFC process[1].

[1] <https://wiki.php.net/rfc/howto>

which design?

a stubborn hardcoded "and now we disable it" don't deserve the word "design" when it could be an ini-option given that on sane hosts symlink() and link() are disabled and so the security issue don't exist

but unless someone designs the realpath cache in a way that it is useable (shared) to gain anything it don't matter too much