php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #56754 APC segfault in apc_cache_find() (backtrace included)
Submitted: 2005-12-28 15:46 UTC Modified: 2009-02-16 20:17 UTC
From: jd at godaddy dot com Assigned:
Status: No Feedback Package: APC (PECL)
PHP Version: APC 3.0.8 / PHP 4.4.1 / Apache 1.3.31 OS: RHEL 3
Private report: No CVE-ID: None
 [2005-12-28 15:46 UTC] jd at godaddy dot com
Description:
------------
Greetings,
  We are currently running APC in a VERY high traffic environment.  The issue we are seeing is that after a while, some of the servers will start serving up incomplete PHP pages, dropping connections perhaps 4/5ths of the way through sending the HTML.  We later found out that httpd was actually segfaulting at this point.  I was able to obtain a backtrace by attaching to an httpd process with gdb and waiting for it to crash.

  I hope this proves to be an easy fix, as APC does help this environment quite a bit otherwise.  Let me know if there's anything else you'd like me to do from gdb.

  We are running Apache/1.3.31 (Unix) mod_pointer/0.8 PHP/4.4.1, and we have the following apc related options in the php.ini:

extension="apc.so"
apc.enabled=1
apc.shm_size=32

  Here is our backtrace:

[Switching to Thread -1218527104 (LWP 30087)]
0x00307f4b in apc_cache_find (cache=0x850fa50, key=
      {data = {file = {device = 2051, inode = 3548568}, user = {identifier = 0x803 <Address 0x803 out of bounds>}}, mtime = 1135802009}, t=3548568)
    at /home/jdicioccio/APC-3.0.8/apc_cache.c:476
476             if (key_equals((*slot)->key.data.file, key.data.file)) {
(gdb) back
#0  0x00307f4b in apc_cache_find (cache=0x850fa50, key=
      {data = {file = {device = 2051, inode = 3548568}, user = {identifier = 0x803 <Address 0x803 out of bounds>}}, mtime = 1135802009}, t=3548568)
    at /home/jdicioccio/APC-3.0.8/apc_cache.c:476
#1  0x0030ac3f in my_compile_file (h=0xbffedf90, type=2)
    at /home/jdicioccio/APC-3.0.8/apc_main.c:264
#2  0x0809b28b in compile_filename (type=2, filename=0x85393c4)
    at Zend/zend_language_scanner.c:3179
#3  0x080b9ad5 in execute (op_array=0x853572c)
    at /usr/src/php-4.4.1/Zend/zend_execute.c:2242
#4  0x080b84dd in execute (op_array=0x85399e4)
    at /usr/src/php-4.4.1/Zend/zend_execute.c:1719
#5  0x080b97b9 in execute (op_array=0x853572c)
    at /usr/src/php-4.4.1/Zend/zend_execute.c:2272
#6  0x080b84dd in execute (op_array=0x857c35c)
    at /usr/src/php-4.4.1/Zend/zend_execute.c:1719
#7  0x080b97b9 in execute (op_array=0x85348f4)
    at /usr/src/php-4.4.1/Zend/zend_execute.c:2272
#8  0x080b84dd in execute (op_array=0x853484c)
    at /usr/src/php-4.4.1/Zend/zend_execute.c:1719
#9  0x080b84dd in execute (op_array=0x853208c)
    at /usr/src/php-4.4.1/Zend/zend_execute.c:1719
#10 0x080b97b9 in execute (op_array=0x8578e38)
---Type <return> to continue, or q <return> to quit---
    at /usr/src/php-4.4.1/Zend/zend_execute.c:2272
#11 0x080b84dd in execute (op_array=0x8578b8c)
    at /usr/src/php-4.4.1/Zend/zend_execute.c:1719
#12 0x080ac2e7 in zend_execute_scripts (type=8, retval=0x0, file_count=3)
    at /usr/src/php-4.4.1/Zend/zend.c:938
#13 0x0808d5ab in php_execute_script (primary_file=0xbfffb1a0)
    at /usr/src/php-4.4.1/main/main.c:1743
#14 0x080bb9c2 in apache_php_module_main (r=0x8528394, display_source_mode=0)
    at /usr/src/php-4.4.1/sapi/apache/sapi_apache.c:54
#15 0x0808619c in send_php ()
#16 0x08086206 in send_parsed_php ()
#17 0x0814f5b7 in ap_invoke_handler ()
#18 0x081643e1 in process_request_internal ()
#19 0x08164440 in ap_process_request ()
#20 0x0815b5c1 in child_main ()
#21 0x0815b821 in make_child ()
#22 0x0815bb5f in perform_idle_server_maintenance ()
#23 0x0815c196 in standalone_main ()
#24 0x0815c79c in main ()


Thanks!


Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2005-12-28 15:58 UTC] jd at godaddy dot com
Here's some additional info:

On a server where APC is currently working properly:

Breakpoint 3, apc_cache_find (cache=0xa20aa50, key=
      {data = {file = {device = 2051, inode = 3957296}, user = {identifier = 0x803 <Address 0x803 out of bounds>}}, mtime = 1131649062}, t=3957296)
    at /home/jdicioccio/APC-3.0.8/apc_cache.c:476
476             if (key_equals((*slot)->key.data.file, key.data.file)) {
(gdb) print *slot
$1 = (slot_t *) 0xb53f4880


On the broken server after the SIGSEGV was encountered:

(gdb) print slot
$1 = (slot_t **) 0xb53e69d4
(gdb) print *slot
$2 = (slot_t *) 0xffffffff
 [2005-12-28 16:48 UTC] jd at godaddy dot com
Changed version field to show APC version as well . . .
 [2005-12-29 12:58 UTC] rasmus@php.net
Are you running out of shared memory when this happens?

APC is happiest when it is running in a stable environment where it doesn't need to chop off the bottom of the cache to make room for new entries all the time.

I suggest these build flags:

./configure --enable-apc --enable-apc-mmap --with-apxs

And then crank up your apc.shm_size to 64 or 128.

If it isn't a cache-full thing, try to figure out which requests are causing this.  Is it always the same inode that shows up in the crashes?  If so, which script is this?

Something like: ls -lRi | grep 3548568
should find it for you.

You can then simply filter out this script from being cached for now and then work on creating the simplest possible version of this script that still crashes APC.  Once I have something I can use to reproduce the crash, I can fix it.
 [2005-12-29 13:09 UTC] jd at godaddy dot com
Thanks for the quick response.  Here's the file it's using:

(gdb) print (char *)*(0x85565cc)
$1 = 0x8556d5c "/var/web/htdocs/smarty/templates_c/%%7C^7CC^7CCF63B4%%radio_gd_next_show.txt.php"

This file contains:

<?php /* Smarty version 2.6.10, created on 2005-12-29 11:02:49
         compiled from radio_gd_next_show.txt */ ?>
6 days 8 hrs 58 mins

One interesting thing about the file is that the last line has no newline after it.  So when you cat the file, you don't see that last line.  I'm not sure if the lack of a newline would do anything to APC though.  I doubt it.  Depends on how you're reading in the file.

As for the stable environment.  I could try raising the limit, but even if that fixes our problem, isn't this still an issue?

Here are the options I passed to configure:

configured by ./configure, generated by GNU Autoconf 2.57,
  with options \"'--enable-apc' '--enable-apc-mmap' '--with-apxs' '--with-php-config=/usr/local/bin/php-config' '--with-apxs=/web/bin/apxs' 'CPPFLAGS=-I/web/include -DAPC_PHP4_STAT'
 [2005-12-29 13:21 UTC] rasmus@php.net
Sure, if it is a cache-full problem, it is still an issue, but it would be nice to know whether this is the case or not.  If it never happens again after increasing your cache size to 128M then this is a useful datapoint for me.
 [2005-12-29 13:44 UTC] jd at godaddy dot com
Does APC allocate shared memory in 1.4M chunks?  When I looked at ipcs -m before and after restarting apache with your 128m change, it showed something like the following:

0x00000000 294912     root      600        1474564    86         dest

I have confirmed that that segment belongs to apache.
 [2005-12-29 16:17 UTC] jd at godaddy dot com
Hello again . .  Since the issue with the crash seems to be *slot getting set to 0xffffffff, do you have any idea how it would get initialized like that?  Does the garbage collector do that?  Since you're likely a lot more familiar with the code than I, is there anywhere off the top of your head where you could see that pointer getting set to that value?
 [2005-12-29 20:32 UTC] rasmus@php.net
Nope, I don't see why something would be setting it to 0xffffffff

I guess you could put a gdb watch on it and see if you can figure out what does that if you can reproduce this reliably.
 [2006-02-24 01:58 UTC] rasmus@php.net
Is this still happening with current CVS?
 [2009-02-16 20:17 UTC] shire@php.net
No feedback was provided. The bug is being suspended because
we assume that you are no longer experiencing the problem.
If this is not the case and you are able to provide the
information that was requested earlier, please do so and
change the status of the bug back to "Open". Thank you.


 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Thu Dec 26 15:01:32 2024 UTC