php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #58148 Memcache-client makes php segfault in semi-related code
Submitted: 2008-04-08 09:23 UTC Modified: 2010-10-01 17:40 UTC
From: acm at tweakers dot net Assigned: pajoye (profile)
Status: Closed Package: memcache (PECL)
PHP Version: 5.2.5 OS: Debian Linux AMD64, sid
Private report: No CVE-ID: None
 [2008-04-08 09:23 UTC] acm at tweakers dot net
Description:
------------
I have a very weird situation where my code segfaults in a semi-related code. Basically in some weird circumstances I get an object returned by reference, rather then by value. 
That yields a segfault, possibly because the memcache-client code frees the internal value, unsetting the external one as well.

I have no real clue why this behaviour happens, but it does happen with the 3.0.0 and 3.0.1 clients, and not with 2.1.x and 2.2.0.
The situation in which the crash happens is extremely unpredictable, but once found it is reproducible.

The phpinfo of our failing server is here:


Reproduce code:
---------------
Here is the code:
http://astraeus.tweakers.net/~acm/memcachecrash.phps

Unfortunately adjusting it slightly already makes it work ok. For instance, in "getSpecificationValueFromDb" removing the completely bogus strlen or $q-concatenation makes the code work as expected, in our real application we concattenate the id to a query and then execute it...
Another step that couldn't be removed was the (in this context) useless 'call_user_func_array' in 'getSpecificationValue' or the wrapping of the MemCache object in our CacheManager-object.

If you want to load this code in your own system, you should reload it at least once to have a SpecificationValue-object in your memcached.

Expected result:
----------------
We have one server with memcached 3.0.1 and the others downgraded to 2.2.0

Here is the output on one of the servers with 2.2.0:
http://astraeus.tweakers.net/~acm/memcachecrash.php

As you can see, in the end there is a line with "Final object fetched:" and then a var_dump'ed object. Further more, just below the second "Got (#3 vs #4 is ok, object vs &object not): ", the object is written as 'object(SpecificationValue)#4'.

Actual result:
--------------
Here is the same script on our server that still has the 3.0.1-client:

http://asclepius.tweakers.net/~acm/memcachecrash.php

As you can see, the output ends with a "Got (#3 vs #4 is ok, object vs &object not): " and no "Final object fetched:". 

The last structure indicates that the object is now by reference in the array: "&object(SpecificationValue)#4"

As there are no calls to "registerSpecificationValueInLocalCache" with explicit references, this is unexpected behaviour.

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2008-04-08 09:31 UTC] acm at tweakers dot net
Here is the url for the actual phpinfo:
http://asclepius.tweakers.net/~acm/inf.php
 [2008-04-08 13:42 UTC] mikael at synd dot info
If you could distill this into a small (a few lines or so) script that demonstrates the problem I'd probably be able to fix it without much trouble. 

It sounds like a refcount/reference and double-free problem, the debug_zval_dump() function might be of some help here.

 http://www.php.net/debug_zval_dump
 [2008-04-08 15:49 UTC] acm at tweakers dot net
I managed to reduce the size of the code from 141 to 58 lines (same url). Unfortunately, I still can't reduce it to "a few lines of code". I didn't see changes between the debug_zval_dump-outputs, the output just didn't appear right after the array's value (from the first call to registerSpecificationValueInLocalCache, from the __wakeup after the memcache get) had been overwritten in the second call to registerSpecificationValueInLocalCache (from the __construct of the object's creation).
 [2008-05-19 13:35 UTC] mikael at synd dot info
I can't seem to reproduce this problem, neither with php/apache or php/cli, would you mind turning off eAccelerator and Zend Optimizer (looking at your phpinfo output this is installed) and retrying. 

If the problem still occur please attach a backtrace from the segfault and the configure line that shows how php was compiled if you've compiled it yourself, otherwise you might be able to get this info from the package info.
 [2008-06-23 16:24 UTC] mikael at synd dot info
No feedback was provided. The bug is being suspended because
we assume that you are no longer experiencing the problem.
If this is not the case and you are able to provide the
information that was requested earlier, please do so and
change the status of the bug back to "Open". Thank you.


 [2008-09-04 08:08 UTC] acm at tweakers dot net
The bug keeps occuring at several places in similar code. Mostly related to making the object known in some global context right after the unserialization and/or in the __wakeup methods.

i.e doing something like
 function __wakeup(){registerObject($this));}
and
function registerObject($object){global $list;
 $list[$object->getId()] = $object; // An error or segfault will occur once this line is reached.
}

with an object that got woken up because it was loaded from memcache and got unserialized because of that.

But it is very, very hard to produce a minimal test case or even predict under which circumstances it will occur. But once you have an occurance, you can reliably reproduce the issue with that same piece of code.

I also had the issue on yet another system that was not running eaccelerator, and on which I was able to disable most of the other modules (xcache, mysql, postgresql, etc) as well.
The funny thing was, that I discovered this particular instance because I disabled xdebug. With xdebug enabled the code worked just fine... So it is something very deep inside php where xdebug also has some hooks. I could reproduce it with both the cli and the apache-version.

The error is most commonly something like:
ALERT - canary mismatch on efree() - heap overflow detected (attacker 'REMOTE_ADDR not set', file '...', line 80)
 [2009-04-03 16:17 UTC] php at muckrake dot net
I can reproduce with php 5.2.9 patched with suhosin.

<?php
$mp = new MemcachePool();
//server is down/unreachable
$mp->addServer('127.0.0.1', 11212);
$mp->get(array('cache_key1', 'cache_key2'));
?>

produces this in the error log:
PHP Notice:  MemcachePool::get() [<a href='memcachepool.get'>memcachepool.get</a>]: Server 127.0.0.1 (tcp 11212, udp 0) failed with: Connection refused (111)
ALERT - canary mismatch on efree() - heap overflow detected

and the httpd child exits.

running gdb on httpd with MaxClients 1 and a breakpoint set on php_security_log shows this backtrace:
#0  php_security_log (loglevel=1, fmt=0x7fbd4fd828c8 "canary mismatch on efree() - heap overflow detected") at /home/dkf/tmp/php-5.2.9/main/suhosin_patch.c:112
	<locals trimmed>
#2  0x00007fbd4e7d7412 in mmc_request_free (request=0xa24ea0) at /home/dkf/tmp/memcache-3.0.4/memcache_pool.c:50
No locals.
#3  0x00007fbd4e7d779b in mmc_pool_free (pool=0xa224d8) at /home/dkf/tmp/memcache-3.0.4/memcache_pool.c:917
	i = 1
#4  0x00007fbd4fc76ee2 in list_entry_destructor (ptr=0xa247d8) at /home/dkf/tmp/php-5.2.9/Zend/zend_list.c:184
#5  0x00007fbd4fc73e70 in zend_hash_del_key_or_index (ht=0x7fbd50023148, arKey=<value optimized out>, nKeyLength=0, h=2, flag=<value optimized out>) at /home/dkf/tmp/php-5.2.9/Zend/zend_hash.c:686
#6  0x00007fbd4fc77169 in _zend_list_delete (id=<value optimized out>) at /home/dkf/tmp/php-5.2.9/Zend/zend_list.c:58
#7  0x00007fbd4fc5b0c5 in _zval_ptr_dtor (zval_ptr=0xa24900) at /home/dkf/tmp/php-5.2.9/Zend/zend_variables.h:35
#8  0x00007fbd4fc73cf8 in zend_hash_destroy (ht=0xa22318) at /home/dkf/tmp/php-5.2.9/Zend/zend_hash.c:717
#9  0x00007fbd4fc84e29 in zend_object_std_dtor (object=0xa22070) at /home/dkf/tmp/php-5.2.9/Zend/zend_objects.c:45
#10 0x00007fbd4fc84e59 in zend_objects_free_object_storage (object=0x1) at /home/dkf/tmp/php-5.2.9/Zend/zend_objects.c:122
#11 0x00007fbd4fc883db in zend_objects_store_del_ref_by_handle (handle=1) at /home/dkf/tmp/php-5.2.9/Zend/zend_objects_API.c:211
#12 0x00007fbd4fc883ff in zend_objects_store_del_ref (zobject=0xa22028) at /home/dkf/tmp/php-5.2.9/Zend/zend_objects_API.c:169
#13 0x00007fbd4fc5b0c5 in _zval_ptr_dtor (zval_ptr=0xa220d0) at /home/dkf/tmp/php-5.2.9/Zend/zend_variables.h:35
#14 0x00007fbd4fc73980 in zend_hash_apply_deleter (ht=0x7fbd50023048, p=0xa220b8) at /home/dkf/tmp/php-5.2.9/Zend/zend_hash.c:805
#15 0x00007fbd4fc73aa8 in zend_hash_reverse_apply (ht=0x7fbd50023048, apply_func=0x7fbd4fc5aa00 <zval_call_destructor>) at /home/dkf/tmp/php-5.2.9/Zend/zend_hash.c:954
#16 0x00007fbd4fc5c375 in shutdown_destructors () at /home/dkf/tmp/php-5.2.9/Zend/zend_execute_API.c:211
#17 0x00007fbd4fc69154 in zend_call_destructors () at /home/dkf/tmp/php-5.2.9/Zend/zend.c:926
#18 0x00007fbd4fc23585 in php_request_shutdown (dummy=<value optimized out>) at /home/dkf/tmp/php-5.2.9/main/main.c:1457
 [2010-01-12 05:45 UTC] lzh at yandex dot ru
We have a similar situation with FreeBSD + Apache 2.2.14 + PHP 5.2.12 and memcached extension 3.0.4

Httpd worker dies with signal 11 (segfault) when one of the memcache servers in our serverpool becomes unavailable.

We had to rollback to the previous branch of memcached extension (2.2.5) and the problem disappeared.

We can provide more additional information (httpd core dumps) to fix this bug.
 [2010-03-28 09:53 UTC] pierre dot php at gmail dot com
Can reproduce it using trunk and non blocking branch.
 [2010-03-28 10:09 UTC] pierre dot php at gmail dot com
simply create a memcache pool crashes:

$mp = new MemcachePool();

double free of the internal rsc.
 [2010-04-09 11:16 UTC] abodera at gmail dot com
Also see #17185

http://pecl.php.net/bugs/bug.php?id=17185
 [2010-05-27 19:38 UTC] ea333 at freemail dot hu
I found a (maybe) fix: dont push the pointer to queue, if the queue is still contains.

Insert the following line to mmc_queue_push (memcache_queue.c) in first line:
if (mmc_queue_contains(queue, ptr)) return;

Elbandi
 [2010-06-01 21:29 UTC] eyecue at gmail dot com
Confirming on:

libevent-1.4.13
memcached-1.4.4_2
pecl-APC-3.0.19
pecl-memcache-3.0.4
php5-5.2.12_2
php5-ctype-5.2.12_2
php5-curl-5.2.12_2
php5-dom-5.2.12_2
php5-filter-5.2.12_2
php5-ftp-5.2.12_2
php5-gd-5.2.12_2
php5-hash-5.2.12_2
php5-iconv-5.2.12_2
php5-json-5.2.12_2
php5-mysql-5.2.12_2
php5-pcre-5.2.12_2
php5-session-5.2.12_2
php5-simplexml-5.2.12_2
php5-soap-5.2.12_2
php5-spl-5.2.12_2
php5-tokenizer-5.2.12_2
php5-xml-5.2.12_2
php5-zlib-5.2.12_2

memcache support => enabled
Version => 3.0.4
Revision => $Revision: 1.83.2.36 $

Directive => Local Value => Master Value
memcache.allow_failover => 1 => 1
memcache.chunk_size => 32768 => 32768
memcache.compress_threshold => 20000 => 20000
memcache.default_port => 11211 => 11211
memcache.hash_function => crc32 => crc32
memcache.hash_strategy => consistent => consistent
memcache.lock_timeout => 15 => 15
memcache.max_failover_attempts => 20 => 20
memcache.protocol => ascii => ascii
memcache.redundancy => 1 => 1
memcache.session_redundancy => 2 => 2

Symptoms are intermittently reproducible, possibly load related:

Jun  1 17:05:17 www-002 kernel: pid 18157 (httpd), uid 80: exited on signal 11
Jun  1 17:05:17 www-002 kernel: pid 18151 (httpd), uid 80: exited on signal 10
Jun  1 17:05:18 www-002 kernel: pid 18132 (httpd), uid 80: exited on signal 11
 [2010-06-01 21:30 UTC] eyecue at gmail dot com
Missed system information in last comment:

FreeBSD www-002.dealsdirect.net 7.2-RELEASE FreeBSD 7.2-RELEASE #0: Fri May  1 07:18:07 UTC 2009     root@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
 [2010-09-16 16:45 UTC] nickg at client9 dot com
The trunk still has this problem with the testcase

<?php
$mp = new MemcachePool();
//server is down/unreachable
$mp->addServer('127.0.0.1', 11212);
$mp->get(array('cache_key1', 'cache_key2'));
?>





The problem is that the callback for error code may be 
called multiple times.  It pops items off the Q to free, in 
the bad case, does a double free.

Trying to untangle that would be very difficult, and it 
would be hard to verify correctness or that it doesn't occur 
in a different flow.  Likewise it would be hard to verify 
that things don't get multiply added.   So this patch, while 
a bit blunt, absolutely solves the issue:

ea333 at freemail dot hu
Insert the following line to mmc_queue_push 
(memcache_queue.c) in first
line:
if (mmc_queue_contains(queue, ptr)) return;

Given the size of the queues (a few items at most), this 
shouldn't be a performance problem.    The patch dropped 
100k+/day segfaults of Apache to 0 in a high volume website 
I'm working with.

What do we need to do get this change in trunk?  Happy to 
help in anyway.

thanks,

--nickg
 [2010-09-29 17:55 UTC] nickg at client9 dot com
Index: memcache_queue.c
===================================================================
--- memcache_queue.c	(revision 303241)
+++ memcache_queue.c	(working copy)
@@ -27,6 +27,8 @@
 #include "memcache_queue.h"
 
 inline void mmc_queue_push(mmc_queue_t *queue, void *ptr) {
+	if (mmc_queue_contains(queue, ptr)) return;
+
 	if (queue->len >= queue->alloc) {
 		int increase = 1 + MMC_QUEUE_PREALLOC;
 		queue->alloc += increase;
 [2010-10-01 17:40 UTC] rasmus@php.net
Thanks Nick.  I have committed the fix.
 [2010-11-09 16:01 UTC] nickg at client9 dot com
Patch was committed to wrong branch.  Not sure what it does on the trunk, but this was originally reported on the V3 NOBLOCKING_IO_CLIENT.

please re-open and reapply patch.

thanks,

nickg
 [2011-09-19 14:34 UTC] ryan dot pendergast at gmail dot com
I'm using memcache client v 3.0.6 running:
/usr/sbin/php5-fpm -v
PHP 5.3.3-1ubuntu9.5 (fpm-fcgi) (built: May  3 2011 
00:56:00)

behind nginx on ubuntu 10.10 32bit in EC2 (m1.small).  My 
site under decent load starts to 
get segfaults in both:
php5-fpm[4113]: segfault at 832e598 ip 083130bc sp bffdd310 
error 7 in php5-
fpm[8048000+70d000]

and
segfault at 1518c ip b6b9c1c9 sp bf8dbf70 error 4 in 
memcache.so

When I set the following the segfaults go away
memcache.allow_failover = 0

Here is my stacktrace:
(gdb) bt
#0  0x083130bc in _array_init ()
#1  0xb6abb163 in mmc_value_handler_multi (key=0x98facc4 "
<MY KEY HERE>", key_len=43, 
value=0x8300a70, flags=0, 
    cas=0, param=0xbffdd680) at 
/tmp/pear/temp/memcache/memcache.c:1518
#2  0xb6abef2c in mmc_unpack_value (mmc=0x98ba798, 
request=0x98fab5c, buffer=0x98fab70, 
    key=0x98facc4 "<MY KEY HERE>", key_len=43, flags=0, 
cas=0, bytes=5)
    at /tmp/pear/temp/memcache/memcache_pool.c:525
#3  0xb6ac439c in mmc_server_read_value (mmc=0x98ba798, 
request=0x98fab5c) at 
/tmp/pear/temp/memcache/memcache_ascii_protocol.c:189
#4  0xb6ac0f07 in mmc_pool_select (pool=0x98ee6f4) at 
/tmp/pear/temp/memcache/memcache_pool.c:1584
#5  0xb6ac1790 in mmc_pool_run (pool=0x98ee6f4) at 
/tmp/pear/temp/memcache/memcache_pool.c:1670
#6  0xb6abb3ba in php_mmc_store (ht=<value optimized out>, 
return_value=0x98fc954, 
return_value_ptr=0x0, this_ptr=0x98ee08c, 
    return_value_used=1, op=1) at 
/tmp/pear/temp/memcache/memcache.c:521
#7  0x08334e87 in execute_internal ()
#8  0xb686a103 in ?? () from 
/usr/lib/php5/20090626+lfs/suhosin.so
#9  0x08360d30 in ?? ()
#10 0x0833781e in execute ()
#11 0xb686a570 in ?? () from 
/usr/lib/php5/20090626+lfs/suhosin.so
#12 0x08360a1e in ?? ()
#13 0x0833781e in execute ()
#14 0xb686a570 in ?? () from 
/usr/lib/php5/20090626+lfs/suhosin.so
#15 0x08360a1e in ?? ()
#16 0x0833781e in execute ()
#17 0xb686a570 in ?? () from 
/usr/lib/php5/20090626+lfs/suhosin.so
#18 0x08360a1e in ?? ()
#19 0x0833781e in execute ()
#20 0xb686a570 in ?? () from 
/usr/lib/php5/20090626+lfs/suhosin.so
#21 0x08360a1e in ?? ()
#22 0x0833781e in execute ()
#23 0xb686a570 in ?? () from 
/usr/lib/php5/20090626+lfs/suhosin.so
#24 0x08360a1e in ?? ()
#25 0x0833781e in execute ()
#26 0xb686a570 in ?? () from 
/usr/lib/php5/20090626+lfs/suhosin.so
#27 0x08360a1e in ?? ()
#28 0x0833781e in execute ()
#29 0xb686a570 in ?? () from 
/usr/lib/php5/20090626+lfs/suhosin.so
#30 0x0830d5a6 in zend_execute_scripts ()
#31 0x082b1cb4 in php_execute_script ()

I dont want to use failover, so this workaround is OK for 
me. Any idea why this is 
happening?
 
PHP Copyright © 2001-2021 The PHP Group
All rights reserved.
Last updated: Thu May 06 07:01:25 2021 UTC