php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #72645 php failed with 500 error 0xfffffffe, "Cannot create mutex" in opcache.log
Submitted: 2016-07-22 07:47 UTC Modified: 2016-07-28 09:26 UTC
Votes:1
Avg. Score:3.0 ± 0.0
Reproduced:1 of 1 (100.0%)
Same Version:1 (100.0%)
Same OS:1 (100.0%)
From: pvasilevich at plesk dot com Assigned:
Status: Not a bug Package: opcache
PHP Version: 7.0.9 OS: Windows
Private report: No CVE-ID: None
 [2016-07-22 07:47 UTC] pvasilevich at plesk dot com
Description:
------------
Affected OSes: Windows 7, Windows Server 2012, 2012R2

STEPS TO REPRODUCE:
* create user phptest
* create 2 IIS application pools (pool1 and pool2), configure these pools to use this common user phptest as identity (pool->Advanced Settings->Process Model->Identity = phptest)
* create 2 different sites with php handler 7.0.9 64-bit, first one is working under pool1 and another one under pool2 users.
* try to open sample page with phpinfo() for one site - it is working fine
* try to open similar sample page with phpinfo() on another site.

Got:
HTTP Error 500.0 - Internal Server Error
D:\php-7.0.9\php-cgi.exe - The FastCGI process exited unexpectedly
Error Code 0xfffffffe

After some analysis, I have found that in opcache log (if you have enabled it before in php.ini) you can find the following entry:
Fri Jul 22 13:15:23 2016 (24780): Fatal Error Cannot create mutex

I have checked mutex created by first process php-cgi.exe
interesting thing: mutex name contains real user name (phptest) but security attributes of this mutex don't have permissions for this user, but have permissions for special internal IIS pool user (named the same way as pool named). see screenshot.

second process is working under the same user phptest, but cannot access to the mutex, because second process is working in another pool. So the error appeared.

The same issue is in php5.5 and php5.6


As fast workaround for php 5.5 (which was originally needed for me) I have tried to compile opcache that also checks env variable APP_POOL_ID (set by IIS) and if it is not empty, add this to both mmap base filename and to mutex name.


Test script:
---------------
<?php

phpinfo();


Patches

patch-opcache-use-iis-pool-name (last revision 2017-08-28 07:40 UTC by snt dot kamikaze at yahoo dot com)

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2016-07-22 07:55 UTC] pvasilevich at plesk dot com
I have uploaded screenshot from Process Explorer:
http://pasteboard.co/eszXEFkYe.png
 [2016-07-25 15:33 UTC] ab@php.net
-Status: Open +Status: Verified
 [2016-07-25 15:33 UTC] ab@php.net
Thanks for the report and analysis. Seems you also come to the root cause of the bug #72332 :) By the current implementation, the first request served will create the mutex and become its owner. With your finding it's clear, that another app pool will fail to create a mutex with the same name.

It is currently unclear, which other implications such a configuration might have. It seems, that using the identity config other than app pool will produce this issue. Probably just changing its name is the simplest solution. What i'm only concerned about is, that this would indeed split all the cache. Other variants like mmap or SHM don't do this. I guess we could solve this by setting corresponding security attributes. Checking currently impacts and possibilities.

Thanks.
 [2016-07-25 16:14 UTC] ab@php.net
Further info - it looks like phptest owns the mutex, but it's not listed by procexp explicitly.

Thanks.
 [2016-07-25 18:32 UTC] mattficken@php.net
What is fastcgi.impersonate set to in your INI?

Would you include your INI configuration?
 [2016-07-26 05:56 UTC] pvasilevich at plesk dot com
>What i'm only concerned about is, that this would indeed split all the cache. Other variants like mmap or SHM don't do this. I guess we could solve this by setting corresponding security attributes. Checking currently impacts and possibilities.
What I see in fact that both processes (running under the same user but in different pools) have written the same base address in their separate files (in my build of PHP). So mutex only protects exclusive access to the file, file contains memory address, and address is the same, so there is no cache split in fact.

From other side I think currently concept of opcache cannot be used per application. I would like to have cache explicitly dedicated for my application with my memory limit I have specified and tuned for my application. In the same time, I want to have another cache for another application, that is working on the same server. I don't want to share any data between these applications. I don't want to share memory between these applications.
I can try to configure this by having 2 different php.ini files for both my applications and explicitly selected mmap_base addresses in php.ini (guessing these addresses), but this is not user friendly to configure and you have risks that some DLLs with ASLR will be overlapped with selected address. 
There are things to improve in future versions.

Regarding current situation, you can try to fix this by explicitly specifying Security Attributes structure when creating Mutex, but in this case you need to fetch SID of current user, and this should be not primary token (which probably will be app_pool_name), but need to resolve SID of user based on GetUserName().


> What is fastcgi.impersonate set to in your INI?
> Would you include your INI configuration?

I have tried both fastcgi.impersonate = 0 and fastcgi.impersonate = 1
the problem is the same

my php.ini is default php.ini-production from php-7.0.9 with 

zend_extension=D:\php-7.0.9\ext\php_opcache.dll
opcache.error_log=c:\windows\temp\opcache.log

no other changes.
 [2016-07-26 12:49 UTC] ab@php.net
Thanks for the further investigation, Pavel. The fact processes under different pools both attach to the same memory, while using different mutexes, is dangerous. That locking functionality is used also for various cache operations, so race conditions are to expect. If we go by separate mutex, it has to be separate memory.

Yeah, the cache per app is a known crunch point regarding Opcache and shared hosting. Not Windows only, but fe on Linux with FPM. The ASLR limitation, and issues with shared memory are typical to Windows, on the other side. While ASLR is a real black box, it is somewhat easier with shared memory. On 64-bit and with a smaller cache size, memory issues are less probable. But also to mention is the file cache - it can not only help to separate sites, but also to wokaround ASLR issues.

You're right, the primary security token won't work, as it is the exact issue we're talking about. I guess some impersonation still happens if the specific IIS configuration is chosen. And with the current implementation, the token used for both CreateMutex and CreateFilemapping comes from the impersonation context.

Thanks.
 [2016-07-26 21:45 UTC] ab@php.net
I think I've more clearance on this case now, and hopefully more understanding :) Namely, I can only reproduce this behavior when using an account with certain privileges as a pool identity. When I use a non elevated custom account, or a built-in LocalService or NetworkService - this is reproducable. When I use an account with the admin membership, or built-in LocalSystem - everything works fine. Same when using AppPoolIdentity. There is also an interesting piece of reading here

http://www.iis.net/learn/manage/configuring-security/application-pool-identities
http://adopenstatic.com/cs/blogs/ken/archive/2008/01/29/15759.aspx

[quote]
On Windows 7 and Windows Server 2008 R2, and later versions of Windows, the default is to run application pools as the application pool identity. To make this happen, a new identity type with the name "AppPoolIdentity" was introduced. If the "AppPoolIdentity" identity type is selected (the default on Windows 7 and Windows Server 2008 R2, and later), IIS will run worker processes as the application pool identity. With every other identity type, the security identifier will only be injected into the access token of the process. If the identifier is injected, content can still be ACLed for the ApplicationPoolIdentity, but the owner of the token is probably not unique. Here is an article that explains this concept. 
[/quote]

This is actually matching with what I see in the practice. Fe i set both pools to run under the AppPoolIdentity, then this is what i see in the procexp

\BaseNamedObjects\ZendOPcache.SharedMemoryMutex@pool0@943d85cdd3a3751784e90ca8856128b0
\BaseNamedObjects\ZendOPcache.SharedMemoryArea@pool0@943d85cdd3a3751784e90ca8856128b0

\BaseNamedObjects\ZendOPcache.SharedMemoryMutex@pool1@943d85cdd3a3751784e90ca8856128b0
\BaseNamedObjects\ZendOPcache.SharedMemoryArea@pool1@943d85cdd3a3751784e90ca8856128b0

And this is perfectly fine. Both the mutex and the shared memory are separated. It was a bit confusing when you told same about same address is written into the base address file. But it actually doesn't matter - both the mutex and the memory handles are named. Processes, that attach to the shared memory, may indeed use same virtual address. However - the named handles do indeed point to the different memory chunks. So in the case of using the AppPoolIdentity everything is separated. IMHO it is a nice concept which works correct out of the box.

Now, to the case of using same identity for the different pools. Here's the chunk of reading that applies to the built-in accounts

https://technet.microsoft.com/library/hh831797.aspx#ApplicationPoolIdentity

LocalService and NetworkService are under the user group limitation. LocalSystem is in admins. This explains what is actually going on also with a custom pool identity. The app poll, and consequently the PHP process, will be started with under the configured custom account. If the underlying account doesn't have sufficient privilege, the injected pool identity will override the ACLs. OFC the more privileges an account has, the more security implications arise. 

My conclusion on this is, that most likely no action is required. Using AppPoolIdentity will perfectly fine separate the apps when using one app per pool. On the other hand, if different pools need to share same cache, they need to have same identity with sufficient privileges. Thus, both scenarios are possible with an appropriate server setup. In any case, the proposed patch has the same effect as if the default setting were used. And any manipulations with the security descriptors won't override the system user/group policies. I would most likely say this behavior is expected and is not a bug.

Thanks.
 [2016-07-28 09:26 UTC] ab@php.net
-Status: Verified +Status: Not a bug
 [2016-07-28 09:26 UTC] ab@php.net
Closing for now, wrt. the comment above.

Thanks.
 
PHP Copyright © 2001-2020 The PHP Group
All rights reserved.
Last updated: Tue Aug 11 01:01:25 2020 UTC