Description:
------------
Using PHP-FPM-5.3.10+APC-3.1.9.

I just discovered 30 PHP processes that's been running for 22 hours.

Further inspection revealed all of them (except one) are waiting to flock() a session file.

The process holding the flock() is doing:
futex(0x7f21238f9e40, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>

(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 0x7f2126114720 (LWP 4271) __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:97

(gdb) bt
#0  __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:97
#1  0x00007f2123614558 in _L_lock_9590 () from /lib/libc.so.6
#2  0x00007f2123612941 in *__GI___libc_free (mem=0x7f21238f9e40) at malloc.c:3737
#3  0x00007f21263f5820 in php_error_cb (type=1, error_filename=0x7f2121088fd8 "", error_lineno=154, format=<optimized out>, args=<optimized out>) at /build/php-5.3.10/main/main.c:931
#4  0x00007f2126446d7c in zend_error (type=1, format=0x7f2126902028 "Maximum execution time of %d second%s exceeded") at /build/php-5.3.10/Zend/zend.c:1127
#5  <signal handler called>
#6  0x00007f2123612a1d in *__GI___libc_malloc (bytes=50) at malloc.c:3658
#7  0x00007f2123617a22 in *__GI___strdup (s=0x7f2121088fd8 "") at strdup.c:43
#8  0x00007f21263f587a in php_error_cb (type=8, error_filename=0x7f2121088fd8 "", error_lineno=154, format=<optimized out>, args=<optimized out>) at /build/php-5.3.10/main/main.c:943
#9  0x00007f2126446d7c in zend_error (type=8, format=0x7f2126907356 "Undefined index: %s") at /build/php-5.3.10/Zend/zend.c:1127
#10 0x00007f21264b2f89 in zend_fetch_dimension_address_inner (type=<optimized out>, dim=<optimized out>, ht=<optimized out>) at /build/php-5.3.10/Zend/zend_execute.c:820
#11 zend_fetch_dimension_address_read (result=0x7f2127f17930, container_ptr=<optimized out>, dim=0x7f2126bf25c8, dim_is_tmp_var=<optimized out>, type=0) at /build/php-5.3.10/Zend/zend_execute.c:1043
#12 0x00007f21264b4059 in ZEND_FETCH_DIM_R_SPEC_CV_VAR_HANDLER (execute_data=0x7f2127f17678) at /build/php-5.3.10/Zend/zend_vm_execute.h:26962
#13 0x00007f212646ee30 in execute (op_array=0x7f2127efe020) at /build/php-5.3.10/Zend/zend_vm_execute.h:107
#14 0x00007f212644654f in zend_execute_scripts (type=8, retval=<optimized out>, file_count=3) at /build/php-5.3.10/Zend/zend.c:1308
#15 0x00007f21263f2bc7 in php_execute_script (primary_file=<optimized out>) at /build/php-5.3.10/main/main.c:2323
#16 0x00007f21264db7c8 in main (argc=669766600, argv=<optimized out>) at /build/php-5.3.10/sapi/fpm/fpm/fpm_main.c:1875

It looks like PHP caught a signal inside malloc(), causing glibc to take some lock, and that free() wants the same lock, leading to deadlock.

I'm not sure if malloc/free is safe to use in signal handlers. http://linux.derkeiler.com/Newsgroups/comp.os.linux.development.apps/2005-07/0323.html seems to suggest it's not.