Just caught it again and ran gdb:

(gdb) backtrace
#0  0x00a25410 in __kernel_vsyscall ()
#1  0x00c876e9 in __lll_lock_wait () from /lib/libpthread.so.0
#2  0x00c82d9f in _L_lock_885 () from /lib/libpthread.so.0
#3  0x00c82c66 in pthread_mutex_lock () from /lib/libpthread.so.0
#4  0x00bcd646 in pthread_mutex_lock () from /lib/libc.so.6
#5  0x004d2344 in apc_pthreadmutex_lock () from /usr/lib/20090626/apc.so
#6  0x00000000 in ?? ()

I believe I've seen this same issue on a self compiled PHP-5.2.14 + PHP-FPM 0.5.14, APC-3.1.4.

I will capture the strace & gdb when I see it again.

Can't confirm the futex/mutex at this point, but the PHP processes had hung though they were responsive to the kill signal upon issuing the restart.

Cores are enabled however no core was produced.  PHP script time limit didn't trigger, memory_limit didn't trigger & php-fpm emergency_restart_interval didn't trigger, so the site was unresponsive for hours.

APC has around 10-13% free space and is configured with 128Mb available.

I have reproduced this issue with APC 3.1.4 and have posted a 
backtrace on bug #15179 (somehow at that time the search 
didn't return this bug)

Anyway the problem has been reproduced with APC 3.1.7, which 
has now been disabled from our production system in the 
meantime.

A resolution to this issue would be greatly welcome.

After some investigation we have found the origin of the 
problem here.

Due to a badly formatted MIME email, the imap php extension 
was crashing when trying to fetch the body of one piece of 
the mail, causing APC to be unable to release its locks.

Because of that all the other apache processes were hung on 
"waiting for a lock to be released", for an infinite time.

Maybe it could be a great idea to have APC release locks 
during script execution (I don't know exactly what happened 
since when the crash happened, no corefile was generated, 
and nothing was found in the apache log).

Anyway while the fault initially lies in imap ext, apc 
should be able to handle crashes and not lockdown the whole 
server.

4 years later, I still see this bug with
php5-5.4.20-34.3.x86_64
php5-APC-3.1.14_git201309121127-3.1.x86_64
on an openSUSE 13.1 server.

I already wondered why I get lots of hanging apache processes (usually one to three times per day) and have a cronjob that regularly restarts apache as workaround.

I was able to strace one of the hanging processes - it was hanging in
    futex(0x7ff5bf79b00c, FUTEX_WAIT, 21, NULL
which looks exactly like this bug.

Anybody have a solid repro for this?  Lots of people report it, but I haven't seen piece of code which triggers the issue repeatably (thus, defining a new integration test to work off of).

I haven't seen this problem reoccur since I reported this bug nearly 5 years ago. Being that it was from such an old version of PHP and APC has been superseded by opcache this can probably be closed off.

I still see this problem with php5-5.4.20-45.1.x86_64 and php5-APC-3.1.14_git201309121127-3.1.x86_64 on openSUSE 13.1 - but unfortunately I don't have a reproducer :-(

Problem still present on Ubuntu 14.04.3 LTS:
- apache2_2.4.7-1ubuntu4.8_amd64
- php5_5.5.9+dfsg-1ubuntu4.14_all
- php5-apcu_4.0.7-1build1~ubuntu14.04.1_amd64

(Zend OpCache as well as APCu is enabled on the affected VirtualHost)

This happens regularly on a Drupal8.0.0-rc1 VirtualHost
(Using APCu extensively in core/lib/Drupal/Core/Cache/ApcuBackend.php)

Have the same problem with APC versions apc-3.1.13 and  apc-HEAD-6a90406.

Web server configuration:

Slackware 14.1 64bit.
Apache-2.4.12.
Php-5.4.41.

Specified APC options:
apc.shm_size = 200M
apc.stat_ctime = 1

Web server is running on Virtualbox, Host computer - Windows server 2008.

Heaviest thing on web server is SugarCRM. It's quite easy to reproduce this bug by opening few SugarCRM pages (around 20) and quickly refreshing them.

It is possible to see APC statistics by refreshing apc.php simultaneously until apache service is still working. Statistic shows that there is not even half of cache used, so most probably it's not because of full cache.

There is nothing useful in apache logs too.

APC is no longer supported in favor of opcache that comes bundled with PHP, if you wish to use the user cache, then look at PECL/APCu.