What type of locking is your APC compiled with?

Looking through config.log it looks like it choose pthread mutex locking


configure:4154: checking whether we should use semaphore locking instead of 
fcntl
configure:4166: result: no
configure:4172: checking whether we should use pthread mutex locking
configure:4184: result: yes
configure:4243: gcc -o conftest -g -O2   conftest.c  -lpthread >&5
configure:4243: $? = 0
configure:4243: ./conftest
configure:4243: $? = 0
configure:4268: checking whether we should use pthread read/write locking
configure:4280: result: no

Could you try grabbing APC from svn-trunk and compiling it with semaphore 
locking instead. Not the fastest locking mechanism, but I would like to rule out 
the locking as the problem in your setup. Here are the steps:

svn checkout https://svn.php.net/repository/pecl/apc/trunk apc
cd apc
phpize
./configure --enable-apc-sem
make
make install

and restart your web server.

I've tried every type of locking.
None seem to work.  I'm using apache bench to test.  ab with only a single 
thread works fine, 2 threads seem to work for a run, but any more thread and the 
server locks up.  In our prod env it really looks like all connections are in 
the CLOSE_WAIT state.  
for my last run the apache/php server has 4 connections in CLOSE_WAIT
I see 4 connection on my machine to that server in FIN_WAIT_2.

I'll try the trunk version using semaphore locking.  does not work. only 4 
requests made it through.

Oh, you are not using the prefork Apache MPM?

correct, we're using the mpm_worker module. I have not read that that's a problem.

Ideally it shouldn't be and I have seen some others use it successfully, but the 
majority of people use the prefork MPM, so it would be another data point to know 
whether these problems go away under prefork.

I switched to prefork and the problem went away.  So the mpm_worker does not work 
with APC?  I have not seen this documented anywhere.  What is the issue?

Well, that is exactly the problem with the threaded MPM. When it works it is 
great, but when it doesn't it is extremely hard to figure out why it is failing. 
Most of the tools we use to debug issues like this don't work very well in a 
threaded environment. And since humans aren't smart enough to write threadsafe 
code, we tend to suggest to people to stick with the prefork MPM.

Are you still on it? If yes, could you please check if there is something about 
SHM or semaphores in the logs? Also on linux there is the ipcs command to check 
this kind of things, dunno what solaris analogue is. May be you see something 
weird there. When thread limit is exhausted, there should be unjoined threads 
having unclosed semaphores in the system. Testing for some longer time and 
restarting apache several times, it should complain about "too many open files", 
at least i've expierienced this.

I have the same problem on Gentoo, www-servers/apache-2.2.22-r1 with mpm_worker, dev-lang/php-5.3.15, dev-php/pecl-apc-3.1.9-r2.
After reaching MaxClients Apache becomes unresponsive. I can confirm this happens only with APC enabled. I turned off all extensions except for APC. The issue is easy to reproduce. To do that I make a bunch of connections like this:
for i in `seq 1 88`; do wget "http://myserver/somescript.php" -O /dev/null & done
I get the MaxClient message in error_log. Apache becomes unresponsive and I need to killall the remaining wget processes.

I recompiled all the relevant packages with debuginfo enabled, reproduced the problem and then attached with GDB to see what's going on:
for PID in `pgrep -x apache2`; do echo "==================== pid=$PID ===================="; gdb -ex "thread apply all bt" --batch /usr/sbin/apache2 $PID; done > apache-php-apc-lock.log

I noticed that nearly all threads are waiting on a lock, so this seems to be a deadlock. Their stacktraces look mostly like this:
#0  0xf57fe416 in __kernel_vsyscall ()
#1  0xb7462209 in __lll_lock_wait () from /lib/libpthread.so.0
#2  0xb745d5f3 in _L_lock_507 () from /lib/libpthread.so.0
#3  0xb745d422 in pthread_mutex_lock () from /lib/libpthread.so.0
#4  0xb73c06f7 in pthread_mutex_lock () from /lib/libc.so.6
#5  0xb5620abf in apc_pthreadmutex_lock (lock=0xb2e04050, tsrm_ls=0xb2d11538) at /usr/src/debug/dev-php/pecl-apc-3.1.9-r2/php5.3/apc_pthreadmutex.c:72
#6  0xb56156b8 in apc_cache_find_slot (cache=0xb8d0a490, key=..., t=1345283556, tsrm_ls=0xb2d11538) at /usr/src/debug/dev-php/pecl-apc-3.1.9-r2/php5.3/apc_cache.c:640
#7  0xb5615d6b in apc_cache_find (cache=0xb8d0a490, key=..., t=1345283556, tsrm_ls=0xb2d11538) at /usr/src/debug/dev-php/pecl-apc-3.1.9-r2/php5.3/apc_cache.c:695
#8  0xb561ee2f in my_compile_file (h=0xb2cfc6c4, type=8, tsrm_ls=0xb2d11538) at /usr/src/debug/dev-php/pecl-apc-3.1.9-r2/php5.3/apc_main.c:529
#9  0xb6762619 in phar_compile_file (file_handle=0xb2cfc6c4, type=8, tsrm_ls=0xb2d11538) at /usr/src/debug/dev-lang/php-5.3.15/sapis-build/apache2/ext/phar/phar.c:3397
#10 0xb6918e84 in compile_filename (type=8, filename=0xa08f0084, tsrm_ls=0xb2d11538) at Zend/zend_language_scanner.l:407
#11 0xb6a3ef12 in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER (execute_data=0xa72040c0, tsrm_ls=0xb2d11538) at /usr/src/debug/dev-lang/php-5.3.15/sapis-build/apache2/Zend/zend_vm_execute.h:22533
#12 0xb69b8fe8 in execute (op_array=0xa7240360, tsrm_ls=0xb2d11538) at /usr/src/debug/dev-lang/php-5.3.15/sapis-build/apache2/Zend/zend_vm_execute.h:107
#13 0xb6960de0 in zend_call_function (fci=0xb2cfca18, fci_cache=0xb2cfca3c, tsrm_ls=0xb2d11538) at /usr/src/debug/dev-lang/php-5.3.15/sapis-build/apache2/Zend/zend_execute_API.c:969
#14 0xb6961a6f in zend_lookup_class_ex (name=0xb366fb38 "comp_tools_stringTools", name_length=22, use_autoload=1, ce=0xb2cfcae0, tsrm_ls=0xb2d11538) at /usr/src/debug/dev-lang/php-5.3.15/sapis-buil
#15 0xb6962da8 in zend_fetch_class (class_name=0xb366fb38 "comp_tools_stringTools", class_name_len=22, fetch_type=0, tsrm_ls=0xb2d11538) at /usr/src/debug/dev-lang/php-5.3.15/sapis-build/apache2/Ze
#16 0xb69c1c19 in ZEND_INIT_STATIC_METHOD_CALL_SPEC_CONST_CONST_HANDLER (execute_data=0xa7202378, tsrm_ls=0xb2d11538) at /usr/src/debug/dev-lang/php-5.3.15/sapis-build/apache2/Zend/zend_vm_execute.
#17 0xb69b8fe8 in execute (op_array=0xa08e3d84, tsrm_ls=0xb2d11538) at /usr/src/debug/dev-lang/php-5.3.15/sapis-build/apache2/Zend/zend_vm_execute.h:107
#18 0xb6975606 in zend_execute_scripts (type=8, tsrm_ls=0xb2d11538, retval=0x0, file_count=3) at /usr/src/debug/dev-lang/php-5.3.15/sapis-build/apache2/Zend/zend.c:1308
#19 0xb68d53ec in php_execute_script (primary_file=0xb2cfee48, tsrm_ls=0xb2d11538) at /usr/src/debug/dev-lang/php-5.3.15/sapis-build/apache2/main/main.c:2323
#20 0xb6a6c41e in php_handler (r=0xb2d1db50) at /usr/src/debug/dev-lang/php-5.3.15/sapis-build/apache2/sapi/apache2handler/sapi_apache2.c:673
#21 0xb7786cf7 in ap_run_handler (r=0xb2d1db50) at config.c:158
#22 0xb7787299 in ap_invoke_handler (r=0xb2d1db50) at config.c:376
#23 0xb77971a4 in ap_internal_redirect (new_uri=0xb2d21a08 "/index.php?__path=kategorie/&__params=id,330,&__page=index.html&__paramsND=", r=0xb2d15778) at http_request.c:554
#24 0xb71d61c1 in handler_redirect (r=0xb2d15778) at mod_rewrite.c:4868
#25 0xb7786cf7 in ap_run_handler (r=0xb2d15778) at config.c:158
#26 0xb7787299 in ap_invoke_handler (r=0xb2d15778) at config.c:376
#27 0xb7797c30 in ap_process_request (r=0xb2d15778) at http_request.c:282
#28 0xb77943d8 in ap_process_http_connection (c=0xa5330850) at http_core.c:190
#29 0xb778f047 in ap_run_process_connection (c=0xa5330850) at connection.c:43
#30 0xb779f0f0 in process_socket (bucket_alloc=<optimized out>, my_thread_num=0, my_child_num=0, sock=0xa53306a0, p=<optimized out>) at worker.c:544
#31 worker_thread (thd=0xb8b5aed0, dummy=0xb2d00468) at worker.c:894
#32 0xb74d9452 in dummy_worker (opaque=0xb8b5aed0) at threadproc/unix/thread.c:142
#33 0xb745af38 in start_thread () from /lib/libpthread.so.0
#34 0xb73b31ae in clone () from /lib/libc.so.6

There are slight differences in frames > 9, but lower frames are always identical in these APC-related stacktraces.

The few other threads are either inside select(), accept(), read(), pthread_cond_wait() or pthread_join() and there is no notion of APC in their stacktraces.

Configuration:
<IfModule mpm_worker_module>
        StartServers            4
        MinSpareThreads         2
        MaxSpareThreads         5
        ThreadsPerChild         22
        MaxClients              88
        MaxRequestsPerChild     1000
</IfModule>
No APC configuration - just "extension=apc.so".

There doesn't seem to be a way to attach a file here, so let me know if you need the full apache-php-apc-lock.log file.

No feedback was provided. The bug is being suspended because
we assume that you are no longer experiencing the problem.
If this is not the case and you are able to provide the
information that was requested earlier, please do so and
change the status of the bug back to "Open". Thank you.