Description:
------------
We've seen in our apache status script lots of hung connections:

Current Time: Wednesday, 08-Feb-2012 16:26:50 CET
Restart Time: Wednesday, 08-Feb-2012 16:11:15 CET
Parent Server Generation: 0
Server uptime: 15 minutes 35 seconds
Total accesses: 25790 - Total Traffic: 213.5 MB
CPU Usage: u1845.84 s166.55 cu0 cs0 - 215% CPU load
27.6 requests/sec - 233.8 kB/second - 8.5 kB/request
245 requests currently being processed, 9 idle workers

CCCCCCCCC_CCCCCCCCCCCCCCCC_CCCCCCCCCCCCCCC.CCCCCCCCCC.CCCCC.CCC.
CCCCCCC..CCCCCCC.CCCCCCCCCCC_CCCCCCCCCCC.CCCCCCCCCCCCCCCC_C..CCC
CCCCC.CCCC.CCC.CCCCCC.CC.CCCCCCCCCCCCCCCCCCCC.CCCCCCCCCCCCCCCCCC
CCCCC_CWCC.CC..CCCC.CCCCCCCC.C.CC.C..CCCC.C.CCC.CCCC.C.C.CC..CCC
CC.C..CCCCCCCCCC..C..C_..C.C.C.CC...C.C...._CC...C......C.......
C..............__...............................................
................................................................
..

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process

Srv	PID	Acc	M	CPU 	SS	Req	Conn	Child	Slot	Client	VHost	Request
0-0	4250	1/90/91	C 	10.23	438	70	18.3	0.86	0.86 	89.133.232.229	www.example.com	GET / HTTP/1.1
1-0	3507	1/174/174	C 	16.63	142	0	0.0	1.31	1.31 	::1	localhost	OPTIONS * HTTP/1.0
2-0	4251	1/157/158	C 	14.82	328	0	0.0	1.12	1.12 	::1	localhost	OPTIONS * HTTP/1.0
3-0	3509	1/81/81	C 	9.55	556	0	0.0	0.81	0.81 	::1	localhost	OPTIONS * HTTP/1.0
4-0	3510	1/133/133	C 	9.85	465	0	0.0	1.12	1.12 	::1	localhost	OPTIONS * HTTP/1.0
5-0	4252	1/154/155	C 	16.42	295	0	0.0	1.37	1.37 	::1	localhost	OPTIONS * HTTP/1.0
6-0	5426	1/53/84	C 	4.31	535	0	0.0	0.49	0.66 	::1	localhost	OPTIONS * HTTP/1.0
7-0	4256	1/48/49	C 	5.55	635	0	0.0	0.53	0.53 	::1	localhost	OPTIONS * HTTP/1.0

We have isolated the problem that produces this and we've found it happens when the result of the get_browser() function is saved into an static attribute, and APC is active.

It doesn't happen always, you need to do a little benchmark to get the hung processes.

Test script:
---------------
<?php

class Hanger
{
	public function getBrowserInfo()
	{
		static $browser;
		$browser = get_browser( $_SERVER['HTTP_USER_AGENT'], true );
	}
}

$hanger = new Hanger();
$hanger->getBrowserInfo();

?>

Expected result:
----------------
No "C" status in processes.

Actual result:
--------------
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_dns.so.2
0x00007f68573de08e in __lll_lock_wait_private () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install httpd-2.2.15-15.el6.centos.1.x86_64
(gdb) bt
#0  0x00007f68573de08e in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x00007f6857364c48 in _L_lock_9291 () from /lib64/libc.so.6
#2  0x00007f68573625e2 in malloc () from /lib64/libc.so.6
#3  0x00007f6858b9cc72 in local_strdup () from /lib64/ld-linux-x86-64.so.2
#4  0x00007f6858ba0636 in _dl_map_object () from /lib64/ld-linux-x86-64.so.2
#5  0x00007f6858baa934 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#6  0x00007f6858ba6106 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#7  0x00007f6858baa3ea in _dl_open () from /lib64/ld-linux-x86-64.so.2
#8  0x00007f685740c250 in do_dlopen () from /lib64/libc.so.6
#9  0x00007f6858ba6106 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#10 0x00007f685740c3a7 in __libc_dlopen_mode () from /lib64/libc.so.6
#11 0x00007f68573e4895 in init () from /lib64/libc.so.6
#12 0x00007f6857696ac3 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104
#13 0x00007f68573e4994 in backtrace () from /lib64/libc.so.6
#14 0x00007f68573587bb in __libc_message () from /lib64/libc.so.6
#15 0x00007f685735e0c6 in malloc_printerr () from /lib64/libc.so.6
#16 0x00007f685735e4f1 in malloc_consolidate () from /lib64/libc.so.6
#17 0x00007f6857360ae8 in _int_free () from /lib64/libc.so.6
#18 0x00007f6853a68031 in zend_mm_shutdown () from /etc/httpd/modules/libphp5.so
#19 0x00007f6853a31edf in php_module_shutdown () from /etc/httpd/modules/libphp5.so
#20 0x00007f6853a31f59 in php_module_shutdown_wrapper () from /etc/httpd/modules/libphp5.so
#21 0x00007f6853b0b7a1 in ?? () from /etc/httpd/modules/libphp5.so
#22 0x00007f68578be99e in run_cleanups (pool=0x7f685af49ba8) at memory/unix/apr_pools.c:2314
#23 apr_pool_destroy (pool=0x7f685af49ba8) at memory/unix/apr_pools.c:782
#24 0x00007f6858df948e in ?? ()
#25 0x00007f6858df98b9 in ?? ()
#26 0x00007f6858df9b1a in ?? ()
#27 0x00007f6858df9e4b in ap_mpm_run ()
#28 0x00007f6858dd1900 in main ()
(gdb)