php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #59620 unable to obtain pthread lock (EDEADLK) in Unknown on line 0
Submitted: 2011-02-14 13:23 UTC Modified: 2011-05-10 19:14 UTC
From: jirwin at tagged dot com Assigned:
Status: Duplicate Package: APC (PECL)
PHP Version: 5.3.4 OS: Centos 5.4
Private report: No CVE-ID:
 [2011-02-14 13:23 UTC] jirwin at tagged dot com
Description:
------------
In production, we see an occasional deadlock in APC.
It's clearly some kind of MP hazard, as it occurs far
more often on the machines with more CPUs, and never
happens at all without a production load on the server.

It always starts like this:

Jan 19 17:47:14 web182 httpd[6439]: [error] [client 
10.0.0.18] PHP Fatal error:  include() [<a 
href='function.include'>function.include</a>]: unable to 
obtain pthread lock (EDEADLK) in /home/html/__autoload.php
Jan 19 17:47:14 web182 httpd[6439]: [error] [client 
10.0.0.15] PHP Fatal error:  Unknown: unable to obtain 
pthread lock (EDEADLK) in Unknown on line 0...

Then that last line is repeated with no other server 
activity until Apache restart.

Gdb some minutes later shows this:
(gdb) attach 6439
...
(gdb) where
#0  0x00002aea10d20920 in __accept_nocancel () from 
/lib64/libpthread.so.0
#1  0x00002aea10b061e4 in apr_socket_accept () from 
/usr/lib64/libapr-1.so.0
#2  0x00002aea0f249d01 in unixd_accept () from 
/usr/sbin/httpd
#3  0x00002aea0f2487a7 in ?? () from /usr/sbin/httpd
#4  0x00002aea0f248aea in ?? () from /usr/sbin/httpd
#5  0x00002aea0f24934d in ap_mpm_run () from /usr/sbin/httpd
#6  0x00002aea0f223e48 in main () from /usr/sbin/httpd
(gdb) p *apc_cache->header
$7 = {lock = {__data = {__lock = 0, __nr_readers = 0, 
__readers_wakeup = 998, __writer_wakeup = 39, 
__nr_readers_queued = 74, 
      __nr_writers_queued = 0, __writer = 6439, __shared = 
128, __pad1 = 0, __pad2 = 0, __flags = 1}, 
    __size = "\000\000\000\000\000\000\000\000?
\003\000\000'\000\000\000J\000\000\000\000\000\000\000'\031\
000\000\200", '\0' <repeats 19 times>, 
"\001\000\000\000\000\000\000", __align = 0}, wrlock = 
{__data = {__lock = 0, __nr_readers = 0, __readers_wakeup = 
0, __writer_wakeup = 0, 
      __nr_readers_queued = 0, __nr_writers_queued = 0, 
__writer = 0, __shared = 128, __pad1 = 0, __pad2 = 0, 
__flags = 1}, 
    __size = '\0' <repeats 28 times>, "\200", '\0' <repeats 
19 times>, "\001\000\000\000\000\000\000", __align = 0}, 
num_hits = 88732964, 
  num_misses = 1967, num_inserts = 1469, expunges = 0, 
deleted_list = 0x2aea1fb08918, start_time = 1295474742, busy 
= 0 '\0', 
  num_entries = 1376, mem_size = 62166880, lastkey = {h = 0, 
keylen = 0, mtime = 0, pid = 0}}

Since apc can die while holding the write lock,
maybe the first time a new apc invocation gets EDEADLK from
pthread_rwlock_{rd,wr}lock it should just be a warning and 
not an error?

---

APC configured with --enable-apc-pthreadrwlocks, although 
deadlock also happens with the default pthread locks.


Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2011-02-14 13:34 UTC] rasmus@php.net
Which Apache mpm? I never saw a deadlock on some of the most 
loaded sites in the world that are running this code. But, it 
was always single-threaded Apache prefork or fastcgi/fpm.
 [2011-02-14 13:58 UTC] jirwin at tagged dot com
We are using the prefork mpm on apache 2.2.3.

APC settings in php.ini:
apc.enabled=1
apc.shm_segments=1
apc.shm_size=256m
apc.num_files_hint=2000
apc.enable_cli=1
apc.stat=0

We are able to reproduce this fairly easily (failure rate is 
around 0.5% per hour per machine); please let me know if 
there's any additional gdb info we can provide.
 [2011-02-14 14:01 UTC] rasmus@php.net
apc.mmap_file_mask ?
 [2011-02-14 15:13 UTC] jirwin at tagged dot com
We are using the default mmap_file_mask (anonymous).

We've also tried mmap_file_mask="/tmp/apc.XXXXXX"
which exhibits the bug at a similar frequency as the 
anonymous setting.

One other datapoint is that we did not see this deadlock
using APC r296784 on PHP 5.3.2.
 [2011-05-10 19:14 UTC] jirwin at tagged dot com
Good news -- this is fixed in APC 3.1.8.
Looks like it was a duplicate of 
http://pecl.php.net/bugs/bug.php?id=22605
 
PHP Copyright © 2001-2014 The PHP Group
All rights reserved.
Last updated: Thu Apr 24 19:01:53 2014 UTC