Note: I dunno what catogory this bug falls under, so i put it under general issues.

This problem cannot be reproduced reliably and I have no idea what triggers it. I've had it happen once on Friday, and it happened again today (Thursday). Basically, I was clicking around developing my php application, and then I realize that things begin to get really slow. I run top and walla 5 httpd processes taking up the CPU (one of them I'm tracing via GDB):

----------

  6:27am  up 31 days,  6:45, 12 users,  load average: 4.05, 4.10, 4.17
134 processes: 126 sleeping, 6 running, 1 zombie, 1 stopped
CPU states: 98.0% user,  1.9% system,  0.0% nice,  0.0% idle
Mem:  128012K av, 121620K used,   6392K free,  23716K shrd,   2628K buff
Swap: 610428K av,  21368K used, 589060K free                 14540K cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT  LIB %CPU %MEM   TIME COMMAND
12955 nobody    20   0  5492 5320   588 R       0 26.4  4.1 191:39 httpd
12963 nobody    17   0  5800 5628   584 R       0 25.4  4.3 191:50 httpd
12960 nobody    11   0  5800 5628   588 R       0 24.2  4.3 191:30 httpd
12957 nobody    11   0  5892 5720   588 R       0 21.7  4.4 193:17 httpd
 2828 build      4   0   572  572   304 R       0  2.1  0.4   3:36 top
    1 root       0   0   124   72    52 S       0  0.0  0.0   0:03 init

--------------

One thing to note was the first time I had this problem, only one httpd process was going bonkers.

Anyways, I did the customary checks of logs, etc. Nothing seemed relavent there except for 4 lines in error_log:

<b>Fatal error</b>:  Maximum execution time of 30 seconds exceeded in <b>Unknown</b> on line <b>0</b><br>

I dunno whether that could be relavent or just coincedence because part of my development, I was doing a execs on long-running calls (just for the record, doing a while(1) or exec(sleep 100) prints that error onto the log, but does not cause httpd to spin).

Check of /proc/*/fd/* denotes no extra file descriptors are open other than the customary httpd fds, except one of the processes has a socket under close_wait to my postgres database. I doubt this is relavant, because the first time i ran into the problem, I had nothing open to postgres.

Anyways, I gdbed into it and did a bt:

-----------

0x400f6008 in chunk_free (ar_ptr=0x4018a040, p=0x825f0c8) at malloc.c:3057
3057    malloc.c: No such file or directory.
(gdb) bt
#0  0x400f6008 in chunk_free (ar_ptr=0x4018a040, p=0x825f0c8) at malloc.c:3057
#1  0x400f5d75 in __libc_free (mem=0x825f148) at malloc.c:2959
#2  0x402cacb8 in _efree () from /usr/lib/apache/libphp4.so
#3  0x402f50d6 in zend_hash_destroy () from /usr/lib/apache/libphp4.so
#4  0x402cb7d4 in shutdown_compiler () from /usr/lib/apache/libphp4.so
#5  0x402f0fc1 in zend_deactivate () from /usr/lib/apache/libphp4.so
#6  0x40302d7a in php_request_shutdown () from /usr/lib/apache/libphp4.so
#7  0x40300675 in php_apache_request_shutdown () from /usr/lib/apache/libphp4.so
#8  0x8051a2e in ap_run_cleanup ()
#9  0x805024d in ap_clear_pool ()
#10 0x80502c1 in ap_destroy_pool ()
#11 0x805023c in ap_clear_pool ()
#12 0x805fa5f in ap_child_terminate ()
#13 0x80600cc in ap_child_terminate ()
#14 0x8060179 in ap_child_terminate ()
#15 0x80607a6 in ap_child_terminate ()
#16 0x8060f39 in main ()
#17 0x400b51eb in __libc_start_main (main=0x8060bec <main>, argc=2, argv=0xbffffd84, init=0x804f1c8 <_init>, 
    fini=0x80903ec <_fini>, rtld_fini=0x4000a610 <_dl_fini>, stack_end=0xbffffd7c) at ../sysdeps/generic/libc-start.c:90
(gdb) n

-------------

Hope that helps someone. Also, I did a couple searches on deja (which should search all the mailinglists / newsgrups) for request_shutdown , and turned out with this match:

http://groups.google.com/groups?q=request_shutdown&hl=en&lr=&safe=off&rnum=3&ic=1&selm=881u57%247qm%241%40FreeBSD.csie.NCTU.edu.tw

Which shows someone checked in a patch into CVS already for this on Feb 11th. However, a look at the source for 4.04p1
or 4.05 shows that the patch is no longer there. I didn't go poking around the CVS logs (its too long) to find out what happened there.

Anyways, I right now have this box in this state and I can do whatever you want to it, but I can't keep it like this forever. Drop me an email if you want more info (do it soon if you want me to poke around the messed up process more).

-rchit