php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #59391 Hung httpd processes, futex/mutex
Submitted: 2010-09-02 01:32 UTC Modified: 2016-11-18 21:50 UTC
Votes:13
Avg. Score:4.6 ± 0.7
Reproduced:11 of 11 (100.0%)
Same Version:5 (45.5%)
Same OS:3 (27.3%)
From: noodles at planetslackers dot com Assigned:
Status: Wont fix Package: APC (PECL)
PHP Version: 5.3.2 OS: RHEL 5.5
Private report: No CVE-ID: None
View Add Comment Developer Edit
Welcome! If you don't have a Git account, you can't do anything here.
You can add a comment by following this link or if you reported this bug, you can edit this bug over here.
Block user comment
Status: Assign to:
Package:
Bug Type:
Summary:
From: noodles at planetslackers dot com
New email:
PHP Version: OS:

 

 [2010-09-02 01:32 UTC] noodles at planetslackers dot com
Description:
------------
Similar problem to http://www.pecl.php.net/bugs/bug.php?id=15179 but it's happening on the latest build of APC (3.1.4).

We're seeing Apache processes hanging at random times and filling up the Apache slots causing the server to be unreachable.

So far I don't have much to go on other than strace returns this while the problem is occurring:

strace -p 8537
Process 8537 attached - interrupt to quit
futex(0xafcc1050, FUTEX_WAIT, 2, NULL <unfinished ...>
Process 8537 detached

Restarting Apache fixes the problem. I initially thought it was a memory issue so I increased apc.shm_size to 128M

APC settings are as follows:

apc.shm_size = 128M
apc.ttl = 3600



Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2010-09-02 19:11 UTC] noodles at planetslackers dot com
Just caught it again and ran gdb:

(gdb) backtrace
#0  0x00a25410 in __kernel_vsyscall ()
#1  0x00c876e9 in __lll_lock_wait () from /lib/libpthread.so.0
#2  0x00c82d9f in _L_lock_885 () from /lib/libpthread.so.0
#3  0x00c82c66 in pthread_mutex_lock () from /lib/libpthread.so.0
#4  0x00bcd646 in pthread_mutex_lock () from /lib/libc.so.6
#5  0x004d2344 in apc_pthreadmutex_lock () from /usr/lib/20090626/apc.so
#6  0x00000000 in ?? ()
 [2010-09-28 11:51 UTC] pecl at mx dot ixido dot net
I believe I've seen this same issue on a self compiled PHP-5.2.14 + PHP-FPM 0.5.14, APC-3.1.4.

I will capture the strace & gdb when I see it again.

Can't confirm the futex/mutex at this point, but the PHP processes had hung though they were responsive to the kill signal upon issuing the restart.

Cores are enabled however no core was produced.  PHP script time limit didn't trigger, memory_limit didn't trigger & php-fpm emergency_restart_interval didn't trigger, so the site was unresponsive for hours.

APC has around 10-13% free space and is configured with 128Mb available.
 [2011-01-28 09:30 UTC] mark at hell dot ne dot jp
I have reproduced this issue with APC 3.1.4 and have posted a 
backtrace on bug #15179 (somehow at that time the search 
didn't return this bug)

Anyway the problem has been reproduced with APC 3.1.7, which 
has now been disabled from our production system in the 
meantime.

A resolution to this issue would be greatly welcome.
 [2011-01-30 20:41 UTC] mark at hell dot ne dot jp
After some investigation we have found the origin of the 
problem here.

Due to a badly formatted MIME email, the imap php extension 
was crashing when trying to fetch the body of one piece of 
the mail, causing APC to be unable to release its locks.

Because of that all the other apache processes were hung on 
"waiting for a lock to be released", for an infinite time.

Maybe it could be a great idea to have APC release locks 
during script execution (I don't know exactly what happened 
since when the crash happened, no corefile was generated, 
and nothing was found in the apache log).

Anyway while the fault initially lies in imap ext, apc 
should be able to handle crashes and not lockdown the whole 
server.
 [2015-01-05 21:53 UTC] bugs dot php dot net at cboltz dot de
4 years later, I still see this bug with
php5-5.4.20-34.3.x86_64
php5-APC-3.1.14_git201309121127-3.1.x86_64
on an openSUSE 13.1 server.

I already wondered why I get lots of hanging apache processes (usually one to three times per day) and have a cronjob that regularly restarts apache as workaround.

I was able to strace one of the hanging processes - it was hanging in
    futex(0x7ff5bf79b00c, FUTEX_WAIT, 21, NULL
which looks exactly like this bug.
 [2015-04-09 21:26 UTC] bryan at revinate dot com
Anybody have a solid repro for this?  Lots of people report it, but I haven't seen piece of code which triggers the issue repeatably (thus, defining a new integration test to work off of).
 [2015-04-10 21:20 UTC] noodles at planetslackers dot com
I haven't seen this problem reoccur since I reported this bug nearly 5 years ago. Being that it was from such an old version of PHP and APC has been superseded by opcache this can probably be closed off.
 [2015-04-23 18:36 UTC] bugs dot php dot net at cboltz dot de
I still see this problem with php5-5.4.20-45.1.x86_64 and php5-APC-3.1.14_git201309121127-3.1.x86_64 on openSUSE 13.1 - but unfortunately I don't have a reproducer :-(
 [2015-11-18 10:01 UTC] valli at valli dot org
Problem still present on Ubuntu 14.04.3 LTS:
- apache2_2.4.7-1ubuntu4.8_amd64
- php5_5.5.9+dfsg-1ubuntu4.14_all
- php5-apcu_4.0.7-1build1~ubuntu14.04.1_amd64

(Zend OpCache as well as APCu is enabled on the affected VirtualHost)

This happens regularly on a Drupal8.0.0-rc1 VirtualHost
(Using APCu extensively in core/lib/Drupal/Core/Cache/ApcuBackend.php)
 [2016-03-02 16:41 UTC] maris-s at inbox dot lv
Have the same problem with APC versions apc-3.1.13 and  apc-HEAD-6a90406.

Web server configuration:

Slackware 14.1 64bit.
Apache-2.4.12.
Php-5.4.41.

Specified APC options:
apc.shm_size = 200M
apc.stat_ctime = 1

Web server is running on Virtualbox, Host computer - Windows server 2008.

Heaviest thing on web server is SugarCRM. It's quite easy to reproduce this bug by opening few SugarCRM pages (around 20) and quickly refreshing them.

It is possible to see APC statistics by refreshing apc.php simultaneously until apache service is still working. Statistic shows that there is not even half of cache used, so most probably it's not because of full cache.

There is nothing useful in apache logs too.
 [2016-11-18 21:50 UTC] kalle@php.net
-Status: Open +Status: Wont fix
 [2016-11-18 21:50 UTC] kalle@php.net
APC is no longer supported in favor of opcache that comes bundled with PHP, if you wish to use the user cache, then look at PECL/APCu.
 
PHP Copyright © 2001-2020 The PHP Group
All rights reserved.
Last updated: Wed Dec 02 22:01:23 2020 UTC