go to bug id or search bugs for
PHP's opcache seems to create keys for files it caches based on their filepath (including the cwd when the option opcache.use_cwd is set).
When turning on opcache in commonly used hosting environments where users are chrooted it is very easy to get key collissions as the full path of a file in a chroot can commonly be /wp-config.php. The file that was accessed on this path first will be stored by opcache and be used by any interpreters executing the same file later on.
Even without a chroot it is often easy to predict where files of another user on the same server will be located and they can still be included circumventing any file permissions set on these files even if PHP executes as the correct user (made even more trivial to figure out interesting files if access to the opcache_get_status() function is not restricted by the host).
Example is in the "test script" below which shortly shows my relevant config lines.
It'd be neat if opcache could implement a runtime config variable to give to an interpreter with a value that mashes up the key by prefixing or xor'ing it without the possibility of being overwritten from within the script.
Alternatively it might be possible to use different parts of shm based on a configuration option so the cache is per-user.
# Permissions on both directories and files are set to rwx for user only
root@debian:/home# ls -l .
drwx------ 2 one one 4096 Feb 20 12:45 one
drwx------ 2 two two 4096 Feb 20 12:45 two
root@debian:/home# ls -l one/
-rw------- 1 one one 25 Feb 20 12:42 one.php
root@debian:/home# ls -l two/
-rw------- 1 two two 46 Feb 20 12:44 two.php
# one.php just sets a single variable
root@debian:/home# cat one/one.php
$one = "one";
# This file tries to include the non-existent "/one.php" in its chroot
root@debian:/home# cat two/two.php
# FPM processes are configured to run as the user
root@debian:/home# grep -r "user =" /etc/php5/fpm/pool.d/
/etc/php5/fpm/pool.d/1.conf:user = one
/etc/php5/fpm/pool.d/2.conf:user = two
# FPM processes also run chrooted into the homedirs
root@debian:/home# grep -r "chroot " /etc/php5/fpm/pool.d/
/etc/php5/fpm/pool.d/1.conf:chroot = /home/one
/etc/php5/fpm/pool.d/2.conf:chroot = /home/two
# Request one.php on pool one
root@debian:/home# curl http://1.localhost/one.php
# Request two.php on pool two
root@debian:/home# curl http://2.localhost/two.php
# That's the content of /one.php which is owned by user one, in its own chroot
# being served by user two from a different chroot while user two doesnt even
# have read permissions on the file.
Users can only access their own files (when configured correctly).
Users can access other users' files when previously accessed by opcache cross chroots and ignoring filesystem permissions.
Add a Patch
Add a Pull Request
Correct PHP version.
I think the easiest would be to simply add an option to use a file's device+inode in addition to, or instead of, the full path. We have discussed this a few times, but nobody has gotten around to an implementation yet.
I've written a more extensive post with more examples here: https://ikanobori.jp/php55-opcache-shared-hosting.html
Inodes seem to be a good idea, I think APC does it that way?
Yes, it was one of the features we lost by dropping APC. It has been on our radar for a while to fix, but nobody has gotten to writing the code yet.
4 month for critical fix... it's bad...
I cannot seem to replicate this... But I got open basedir enabled. Can somebody verify that open basedir solve it?
noescape at centrum dot sk:
I think open_basedir is useless in chroot
I can replicate this with php5-fpm 5.5.9+dfsg-1ubuntu4.11.
I baffled how a huge security / usability problem like this can exist, yet nobody gives a poop about it.
We've got a server with multiple sites chrooted using the same directory structure inside the chrooted /.
Obviously this causes fatal problems as there are conflicts, like index.php in those hosts causing it to randomly load from any of those.
Because of this, opcache can not be used as we can't change the directory structure.
Someone, please investigate this!
I replicate this problem with two VirtualHosts (using open_basedir).
OS: Debian 8.1
PHP 5.6.14-0+deb8u1 (cli) (built: Oct 4 2015 16:13:10)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies
with XCache v3.2.0, Copyright (c) 2005-2014, by mOo
with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2015, by Zend Technologies
with Xdebug v2.2.5, Copyright (c) 2002-2014, by Derick Rethans
with XCache Optimizer v3.2.0, Copyright (c) 2005-2014, by mOo
with XCache Cacher v3.2.0, Copyright (c) 2005-2014, by mOo
with XCache Coverager v3.2.0, Copyright (c) 2005-2014, by mOo
php_admin_value open_basedir /var/www/app1
Options Indexes FollowSymLinks MultiViews
#Allow from All
Require all granted
php_admin_value open_basedir /var/www/app2
Options Indexes FollowSymLinks MultiViews
#Allow from All
Require all granted
Example - PHP apps - typical using with composer (dummy):
apps directory structures: /var/www/
web/index.php (same app1 and app2):
require_once __DIR__ . '/../vendor/autoload.php';
echo (new third\party\lib())->getName();
vendor/autoload.php (same app1 and app2):
require_once __DIR__ . '/third/party/lib.php';
public function getName()
return 'good library';
public function getName()
return 'bad hacked library';
It would be a problem in webhostings. After web server restart just run bad app2 as first.
important note to the previous example:
PHP is running as Apache module
Is this still beeing considered as a bug or should we just give up on chroot+opcache?
This is probably related:
I also find it stunning that - at the very least - PHP doesn't warn about this or even better refuse to start in this configuration. This is a serious security issue and all too easy to overlook. Took me a while to figure out whats actually happening. This can lead to very mysterious errors. Fragments or error messages of other unrelated sites on the same server suddenly showing up and so on.
Can anybody confirm that "opcache.optimization_level=0" is really a solid workaround that can be used in production? There should be no information leak whatsoever. No files. No data. No code. No nothing. That's why we have a chrooted environment in the first place. But obviously I'd rather have an optimzation level 0 than no opcache at all.
I'm concerned about this issue since it has the potential to be a security deal-breaker for Zend OPcache when used on (as an example) a shared-hosting platform.
With that said I'm unable to replicate this using michal_micko's instructions for mod_php and open_basedir configurations.
I used the same exact two VirtualHosts, directory tree, and file contents. I restarted apache, accessed http://app2 first and then http://app1 but both displayed their proper "bad" and "good" texts respectively.
Ubuntu 14.04.5 LTS (as opposed to Debian 8.1)
Apache 2.4.7 (Ubuntu) (as opposed to 2.4.10, default config plus VirtualHosts)
PHP 5.6.27-1+deb.sury.org~trusty+1 (as opposed to 5.6.14-0+deb8u1, default config)
opcache.enable = On
opcache.optimization_level = 0x7FFFBFFF
opcache.revalidate_path = Off
opcache.use_cwd = On
opcache.validate_timestamps = On
One potentially major difference between my config and michal_micko's is that I do not have any XCache modules or Xdebug loaded. I would think, with except perhaps the PHP version, that is the only major configuration difference between my set up and michal_micko's.
For me it looks safe to use PHP 5.6 OPcache + mod_php + open_basedir. When I dump the output of opcache_get_status() I see only full paths for both the "full_path" value and the array keys. I believe this is the use_cwd option doing its job. The only additional "securing" that I see is setting restrict_api to something out of reach or adding opcache_* functions to disable_functions.
I'd greatly appreciate it if anyone is able to correct or confirm what I've found.
Does this configuration of Opcache resolve the problem? See: http://stackoverflow.com/questions/20960469/php-fpm-5-5-does-opcache-run-per-domain
At this point two separate but related issues are being discussed:
1. OPCache is prone to filename collisions across chroots. Adding inode number to the key would fix this.
2. All child PHP processes have access to all scripts in the cache, regardless of file permissions. This renders OPCache unusable and dangerous on shared hosting servers, where OPCache may be used by malicious users to bypass file permissions (and open_basedir FWIW). Adding inode to the cache key would *not* fix this vulnerability, although in practice it would help in cases where parent directory is unreadable.
I'll focus on the second issue: The obvious real-world example is a shared hosting server with multiple CMS sites. Most PHP CMS's store database credentials and other sensitive information in PHP scripts. Thus one malicious user (translation: compromised CMS) typically has full access to databases for other users' CMS's if OPCache is enabled. If anyone doubts this, I can provide working proof of concept exploit scripts targeted at WordPress. Fortunately this doesn't seem to be exploited in the wild on any large scale, but I anticipate PHP malware will start incorporating this technique as soon as it's more widely known.
I'm attaching a patch against the 5.6 branch which prepends a unique user identifier (username in Windows, euid elsewhere) to the cache key. This should fix issue 2 in all situations where dl() is not allowed (PHP5 with FPM would still in theory be vulnerable unless dl() is disabled).
It's not a perfect fix because it requires the PHP process to act as gatekeeper, essentially a substitute for kernel enforcement of filesystem permissions. This is a problem if any PHP child process with a descriptor for the shared opcache can be subverted, for example with a malicious extension, hence my concern about dl()).
It will also fix issue 1 *only* in cases where the chroot environments are running PHP scripts with separate user accounts. If the chroots use a shared web server user for PHP, issue 1 is still a problem.
I agree with Rasmus that the script inode should also be added to the key, but I wasn't sure if it would be appropriate to stat the script for its inode in the key generation function due to performance concerns, and being unfamiliar with the extension and PHP itself I wasn't prepared to do this in a more appropriate place.
I started with 5.6 because I believe this is a fairly serious security vulnerability which many users are unaware of, which isn't adequately explained in the OPCache documentation, and which should IMHO be fixed in a patch release. I'm willing to port it forward to 7.x if there's interest and time allows, but I feel strongly that this patch or something similar should be included in a 5.6 patch release ASAP.
Code is only minimally tested. Use at own risk. Apologies for any indentation issues; I did my best to follow the style guide, but existing OPCache code did not. Feedback is welcome.
Excuse me but...
Wouldn't it be safer, more reliable, less hackish ... to separate the PHP pools of the website ? One pool per site, each pool listening on one port for CGI requests.
I mean, we do know for sure, that PHP cannot replace the OS and the configuration to secure webservers in a shared architecture (shared hosting).
We've tried things such as safe_mode, open_basedir etc... since the beginning, with no success.
Nowadays, with mature OS, mature stacks, and a mature FCGI handler (PHP-FPM), it is easy to build a shared hosting architecture not relying on the PHP language itself to isolate the virtualhosts.
Security - of the filesystem here - is the matter of the OS , not PHP.
Why don't you create several PHP-FPM pools, and secure them with one Unix user per pool ? Obviously, the OPCache SHM wouldn't be shared here, but that is not what you want : OPCache SHM shouldn't be crossed against several websites.
One pool per website = one SHM per website = one unix user per website = we solved every low level security problems, right ?
The suggestion as one user per site would have to apply all the way across the stack. So Apache would have to have a single user, and no multi-site, no virtual sites, no resource sharing at all? This would mean full resources for each and every site, which would be extremely wasteful. I have a client with four websites. Does the single client need four users? I have several clients but I want to run all clients and all their sites with a single configuration. Apache can do this, PHP can do this, therefore opcache should be able to support such a configuration.
I'll address jpauli's points:
Not everyone is using FPM. My patch fixes this problem on both FPM and apache2handler, where we can separate users with mod_ruid2.
My experience with FPM is limited so I may be speaking out of turn, but when I tried separate FPM pools with separate users, they were still forked from the same parent FPM master process. Correct me if I'm wrong but the OPCache SHM segment is opened in this master process and inherited by the pools as a file descriptor. The multi-user pools still share a single OPCache, and thus they actually aid in bypassing file permissions, rather than fixing the problem.
I agree 100% with your points about relying on the lower stack to isolate vhosts and enforce permissions, but the entire point of this bug report is that OPCache breaks this isolation in real-world configurations since it uses a shared cache passed from Apache parent to children, or from FPM master to pools.
You stated "Obviously, the OPCache SHM wouldn't be shared here" in a multi-pool configuration. What would such a configuration look like? My testing shows the opposite to be true. Perhaps you meant to suggest multiple FPM master daemons instead of multiple pools? If the OPCache SHM can be initialized at the pool level instead of in the master process (I don't think this is how it works today; I'd love to be proven wrong on this), this is great for FPM users but does nothing for users of other SAPI's.
The big caveat in my testing is that most of it was done under cPanel. Both mod_ruid2/mod_php and FPM have the shared OPCache problem under both EA3 and EA4. This is true for both PHP5 and PHP7. If you think this is a problem with cPanel's apache2handler and FPM configurations then I can take the issue up with cPanel, but I'd love to see how they could fix this for all SAPI's where OPCache is useful. Clearly the problem isn't limited to cPanel though, based on the other users commenting on this bug.
Hopefully this will clarify users' concerns with this bug; from your response I'm frankly not sure you fully understand the problem we're reporting. I agree with "jeff at mcneill dot io" that to isolate vhosts with PHP as it stands today, you'd need entirely separate Apache parents (for apache2handler) or FPM master processes (not just multiple pools).
php-dev - would it not be best (at least when using fpm) to run each pool under a separate user / group account as discussed here: https://www.digitalocean.com/community/tutorials/how-to-host-multiple-websites-securely-with-nginx-and-php-fpm-on-ubuntu-14-04
This would seem to satisfy the original suggestions earlier in the thread to leave as much as possible to the OS level security controls.
I mention because I was wondering if your patches (which are warmly welcomed...I've been posting on this issue for a while now on SF & SE) presume that these types of controls would also be implemented or perhaps obviate them.
bjh438-git, thank you for your comments. I had previously read that article. The setup described mitigates the security concerns I've outlined for one reason: It recommends disabling OPCache completely.
Such a multi-user setup is generally recommended, but as I mentioned in my response to jpauli, it's dangerous when combined with a single SHM cache with a simplistic hash keying scheme which makes no attempt to segregate users.
My readings of php-internals archives suggest that OPCache was indeed meant to be usable in multi-user setups so perhaps the maintainers simply haven't thought through the all implications of the key scheme, though they have discussed its other shortcomimgs. I'm working on a brief but more comprehensive advisory which I'll post here and on php-internals soon if we don't get more meaningful engagement on this issue.
jpauli, using separate pools does not fix this issue; the cache is maintained by the master process that has access to all pools regardless of permissions of those pools.
First off all thank you php-dev to bringing this issue back on the table.
I'd like to bring in my view from a shared hosting provider.
IMHO the two big hosting panel softwares Plesk and cPanel are vulnerable to this issue. We currently use cPanel. cPanel provides a different set of options to isolate the user from reading other users files. In addition to the user isolation it's also possible to isolate you processes any further by running them in a chroot environment. This could be easy leveraged by using tools/distros like Cloudlinux CageFS (https://www.cloudlinux.com/cagefs), which we use.
All this countermeasures to separate processes users on process and file system level are voided if the OpCache is accessible without the user boundaries.
Our stack currently is not vulnerable to the mentioned issue due to the combination of additional tools and server softwares that isolate users also on the OpCache level. The magic is a combination of Cloudlinix as file system isolation and Litespeed as a PHP SAPI process and OpCache isolation.
Having some insights into the hosting industry this stack is not the majority. IMHO most of the cPanel users use the built in capabilities of the EasyApache stack, which is vulnerable.
In the past we've seen some attacks targeting multiple users on the same server. The most effective was: Symlink Bypass http://www.rafayhackingarticles.net/2012/01/hack-website-on-shared-host-symlink.html
A skilled attacker could easily take over dozens of shared hosting with the help of this vulnerability. Depending on the shared hosting provider I guess an average of users per host is approx. 200-1000 users. Given the current market share of Wordpress (easy extractable secrets) of approx. 60% ( https://w3techs.com/technologies/details/cm-wordpress/all/all ) the impact of a hacked website/CMS/Plugin is very high. In automatic manner an attacker could take over half of shared hostings only by optimizing for Wordpress.
This is IMHO a pressing matter for all the other shared hosting providers and customers support staffs out there.
I'm in favor to the current solution to fix the issue by appending a contextual part to the path.
IMHO the best solution from a security point of view would be the inode information.
From a operations point of view inodes also solve the issue of deploying with symlinks. E.g. https://www.scalingphpbook.com/blog/2014/01/30/zend-opcache-and-atomic-deploys.html
dol+list, thank you for your comments. Your observations are consistent with my own experiences, with the exception that I haven't had a chance to test exploitation under CloudLinux/CageFS. I haven't bothered testing with cPanel's "Jail Apache" virtfs feature either, due to unrelated reliability problems with that feature. I'm trying to avoid too much discussion of such platform-specific mitigations because they're out of scope for a PHP bug report, and they shouldn't even be necessary. I feel strongly that this is a PHP bug which needs to be fixed in PHP.
I agree that device+inode should ideally be added to the key scheme, but they alone are insufficient. They are not necessarily private information, and can be read if permissions of the parent directory allow. This breaks user expectations; if wp-config.php is unreadable for a user, that user should not be able to execute that script, period. This is why I think EUID should be part of the key. I don't like the thought of opcache playing permissions referee and I think it suggests a fundamental design flaw with the way SHM is being used, but working within the existing SHM design I think EUID is the best way forward. Perhaps a more radical change would be better, but I'm not familiar enough with prior art in PHP opcode caching to know what it is.
Using device+inode is also less straightforward than using EUID, otherwise I'm sure the maintainers would have already done it: it seems inappropriate to stat() for this info during key generation for performance reasons. APC apparently got a stat struct passed from the SAPI so the stat() was already done outside of APC code (does this imply APC cached scripts per compilation unit and not per file, since the web server wouldn't know which files are included? I haven't reviewed APC code in enough detail yet to confirm).
For my patch I use EUID simply because there's no undue performance hit and it fixes the cross-user permissions bypass in both "out of the box" and common control panel environments. It's probably not perfect but surely we can agree it's better than the existing code.
For anyone using FPM who wants to mitigate before PHP fixes the vulnerability, 2 years ago Mattias Geniar confirmed that separate FPM master daemons is the way to go, and provided example configs. It's cumbersome and inefficient and shouldn't be necessary: https://ma.ttias.be/a-better-way-to-run-php-fpm/
Thanks for providing further insight into this issue. I'd like to take you up on your offer of a PoC for replicating under mod_php + open_basedir. FPM works as well but I'm particularly interested in the mod_php case.
kmark937, I've emailed you directly with a PoC exploit script. I'd love to hear about your results when you try it.
dol+list, one nitpick with your earlier comments; according to OPCache comments any changes to the key scheme need to be prepended, not appended, due to the way the key delimiters are parsed (I wonder what happens when script filenames contain ':'?).
To any PHP project people still reading, would it be possible to reclassify this as a bug report instead of feature request? This was opened as a feature request at the suggestion of the original bug reporter in #67481, who withdrew their bug report when they decided that the vulnerable behavior was desirable after they found a workaround by enabling use_cwd. use_cwd does not fix this issue in most cases with common CMS applications because the use_cwd logic is only applied to scripts invoked/included via relative paths, which is less common in real-world web server/CMS environments. So far I've avoided opening yet another duplicate bug tracker item for this behavior, but it absolutely needs to be treated as a bug and not a feature request since OPCache documentation does not warn about it and php-internals discussions dating back to the Optimizer+ days indicate that the maintainers do indeed intend this feature to be usable in multi-user environments.
jpauli, I do sincerely appreciate that you took the time to comment on this as a PHP project member. Did you have a chance to review my response? Do you see that multiple FPM pools with separate users to *not* have the "obvious" separate SHM behavior which you seem to think they do? Would you care to try my exploit script? I'm trying to give the PHP project every chance to address the issue before I publish the exploit script, but they've literally had years to address this and show little interest in fixing it. There's nothing novel about the exploit btw; it does nothing different from the PoC configs already given in this request and in bug #67481.
I've just tested suexec/mod_fcgid: This configuration seems to be unaffected because separate vhosts' php-cgi processes don't share a common parent PHP process. mod_fcgid plays a similar role to the FPM master, but because it's not PHP and doesn't initialize the opcache, there's no single SHM object being shared. Each vhost gets its own opcache SHM object if I understand correctly.
Thus mod_fcgid with suexec provides better compartmentization than FPM since it has no single parent PHP process from which all others are descended.
- apache/mod_ruid2/mod_php is vulnerable.
- php-fpm is vulnerable (probably regardless of webserver).
- apache/suexec/mod_fcgid/php-cgi is not vulnerable.
Perhaps php-fpm needs a long hard look at the order in which it initializes pools vs extensions?
Original reporter here, I sitll read the updates to this.
Thanks php-dev for the patch, it seems to work if just the euid is included but correct me if I am wrong for yes the euid gets set after PHP-FPM forks a new pool under a specific user but just running separate sites chrooted under the same user (say, www-data) would still cause the same issues as described in my original report.
Now, that's obviously an unwise way to run any PHP shared hosting environment but it might be something we want to change as well. I'll take a look at the patch as well but maybe we can do something closer to inodes to be used in the cache key instead of effective user id.
I'll try to replicate with your patch applied over the weekend :)
Simon, that's correct; my patch doesn't fix same-user, multiple chroot scenarios. Search the comments for "I agree with Rasmus" for my thoughts on that. I was only attempting to fix the cross-user permissions bypass, which I personally consider the more serious issue. If I knew the appropriate place to do the stat() call I would have added device+inode, but I've only had limited time for this and it's my first exposure to PHP internals so I kept the scope small. Maybe if I get more free time, and reach consensus with the maintainers, I'll have a more complete fix later.
I'd prefer something like https://github.com/SjonHortensius/php-src/commit/21b596086843f02ff0e4d937a17c80e758b27365 (untested wip) instead
It's untested; but shows the general idea. After a successful chroot (either from fpm or userspace); the path is stored in a global which is prepended to the cache keys.
The following patch has been added/updated:
Patch Name: validate_permission.diff
I've attached a patch to enable optional file permission validation.
With opcache.validate_permission=1 php.ini directive, PHP is going to revalidate readability of cached files using access() syscall. This directive is going to be disabled by default and should be enabled by shared hosting providers. Additional checks lead to ~5% slowdown on Wordpress.
The proposed manipulations with keys (e.g. including user name or root directory into key) won't work out of the box, because in some cases opcache doesn't use keys constructed by accel_make_persistent_key(), but uses full-real-name instead.
I didn't solve the chroot keys collision problem yet.
The following patch has been added/updated:
Patch Name: bug69090.diff
The new attached patch should completely fix both problems.
It modifies values of hash function XOR-ing them with a value constructed from the root inode number. This value calculated once per request, using stat("/") at request startup, if opcache.validate_root is enabled (disabled by default).
The following patch has been added/updated:
Patch Name: bug69090.diff
Automatic comment on behalf of email@example.com
Log: Fixed bug #69090 (check cached files permissions)
The issue in question appears to be fixed, thanks Dmitry for resolving this 2yo security and stability problem. Nevertheless, IMO the current solution looks more like a workaround for a larger problem: there's complex code (for which multiple bugs are fixed in half of the releases) that runs outside the chroot environment with complete visibility of the entire filesystem, making it a perfect escape route (which was actually demonstrated with the current bug).
IMO, the appropriate fix would be to initialize everything that could access the filesystem (including internal functionality & all plug-ins) AFTER performing chroot. Only the logic responsible for loading of the configs, libraries and modules should be placed in the pre-chroot portion of the daemon (to avoid placing their files in the chroot environment), but NOT their parsing (for the per-pool configs), initialization (for the modules) and management, which should be performed in the isolated chrooted environments, a separate instance for each chrooted pool.
If the current logic is to save some memory that could be shared between the chrooted pools, I do believe that security should not be sacrificed for performance or lower resource usage, especially nowadays with so cheap hardware.
PHP devs, please let me know if it's appropriate to open a new bug (type: security) for tracking this change or if this one should be reopened, or if you consider this issue has no merit to expect its implementation some day.