|
|
php.net | support | documentation | report a bug | advanced search | search howto | statistics | random bug | login |
[2008-04-08 09:23 UTC] acm at tweakers dot net
Description: ------------ I have a very weird situation where my code segfaults in a semi-related code. Basically in some weird circumstances I get an object returned by reference, rather then by value. That yields a segfault, possibly because the memcache-client code frees the internal value, unsetting the external one as well. I have no real clue why this behaviour happens, but it does happen with the 3.0.0 and 3.0.1 clients, and not with 2.1.x and 2.2.0. The situation in which the crash happens is extremely unpredictable, but once found it is reproducible. The phpinfo of our failing server is here: Reproduce code: --------------- Here is the code: http://astraeus.tweakers.net/~acm/memcachecrash.phps Unfortunately adjusting it slightly already makes it work ok. For instance, in "getSpecificationValueFromDb" removing the completely bogus strlen or $q-concatenation makes the code work as expected, in our real application we concattenate the id to a query and then execute it... Another step that couldn't be removed was the (in this context) useless 'call_user_func_array' in 'getSpecificationValue' or the wrapping of the MemCache object in our CacheManager-object. If you want to load this code in your own system, you should reload it at least once to have a SpecificationValue-object in your memcached. Expected result: ---------------- We have one server with memcached 3.0.1 and the others downgraded to 2.2.0 Here is the output on one of the servers with 2.2.0: http://astraeus.tweakers.net/~acm/memcachecrash.php As you can see, in the end there is a line with "Final object fetched:" and then a var_dump'ed object. Further more, just below the second "Got (#3 vs #4 is ok, object vs &object not): ", the object is written as 'object(SpecificationValue)#4'. Actual result: -------------- Here is the same script on our server that still has the 3.0.1-client: http://asclepius.tweakers.net/~acm/memcachecrash.php As you can see, the output ends with a "Got (#3 vs #4 is ok, object vs &object not): " and no "Final object fetched:". The last structure indicates that the object is now by reference in the array: "&object(SpecificationValue)#4" As there are no calls to "registerSpecificationValueInLocalCache" with explicit references, this is unexpected behaviour. PatchesPull RequestsHistoryAllCommentsChangesGit/SVN commits
|
|||||||||||||||||||||||||||
Copyright © 2001-2025 The PHP GroupAll rights reserved. |
Last updated: Fri Oct 31 11:00:01 2025 UTC |
The bug keeps occuring at several places in similar code. Mostly related to making the object known in some global context right after the unserialization and/or in the __wakeup methods. i.e doing something like function __wakeup(){registerObject($this));} and function registerObject($object){global $list; $list[$object->getId()] = $object; // An error or segfault will occur once this line is reached. } with an object that got woken up because it was loaded from memcache and got unserialized because of that. But it is very, very hard to produce a minimal test case or even predict under which circumstances it will occur. But once you have an occurance, you can reliably reproduce the issue with that same piece of code. I also had the issue on yet another system that was not running eaccelerator, and on which I was able to disable most of the other modules (xcache, mysql, postgresql, etc) as well. The funny thing was, that I discovered this particular instance because I disabled xdebug. With xdebug enabled the code worked just fine... So it is something very deep inside php where xdebug also has some hooks. I could reproduce it with both the cli and the apache-version. The error is most commonly something like: ALERT - canary mismatch on efree() - heap overflow detected (attacker 'REMOTE_ADDR not set', file '...', line 80)I can reproduce with php 5.2.9 patched with suhosin. <?php $mp = new MemcachePool(); //server is down/unreachable $mp->addServer('127.0.0.1', 11212); $mp->get(array('cache_key1', 'cache_key2')); ?> produces this in the error log: PHP Notice: MemcachePool::get() [<a href='memcachepool.get'>memcachepool.get</a>]: Server 127.0.0.1 (tcp 11212, udp 0) failed with: Connection refused (111) ALERT - canary mismatch on efree() - heap overflow detected and the httpd child exits. running gdb on httpd with MaxClients 1 and a breakpoint set on php_security_log shows this backtrace: #0 php_security_log (loglevel=1, fmt=0x7fbd4fd828c8 "canary mismatch on efree() - heap overflow detected") at /home/dkf/tmp/php-5.2.9/main/suhosin_patch.c:112 <locals trimmed> #2 0x00007fbd4e7d7412 in mmc_request_free (request=0xa24ea0) at /home/dkf/tmp/memcache-3.0.4/memcache_pool.c:50 No locals. #3 0x00007fbd4e7d779b in mmc_pool_free (pool=0xa224d8) at /home/dkf/tmp/memcache-3.0.4/memcache_pool.c:917 i = 1 #4 0x00007fbd4fc76ee2 in list_entry_destructor (ptr=0xa247d8) at /home/dkf/tmp/php-5.2.9/Zend/zend_list.c:184 #5 0x00007fbd4fc73e70 in zend_hash_del_key_or_index (ht=0x7fbd50023148, arKey=<value optimized out>, nKeyLength=0, h=2, flag=<value optimized out>) at /home/dkf/tmp/php-5.2.9/Zend/zend_hash.c:686 #6 0x00007fbd4fc77169 in _zend_list_delete (id=<value optimized out>) at /home/dkf/tmp/php-5.2.9/Zend/zend_list.c:58 #7 0x00007fbd4fc5b0c5 in _zval_ptr_dtor (zval_ptr=0xa24900) at /home/dkf/tmp/php-5.2.9/Zend/zend_variables.h:35 #8 0x00007fbd4fc73cf8 in zend_hash_destroy (ht=0xa22318) at /home/dkf/tmp/php-5.2.9/Zend/zend_hash.c:717 #9 0x00007fbd4fc84e29 in zend_object_std_dtor (object=0xa22070) at /home/dkf/tmp/php-5.2.9/Zend/zend_objects.c:45 #10 0x00007fbd4fc84e59 in zend_objects_free_object_storage (object=0x1) at /home/dkf/tmp/php-5.2.9/Zend/zend_objects.c:122 #11 0x00007fbd4fc883db in zend_objects_store_del_ref_by_handle (handle=1) at /home/dkf/tmp/php-5.2.9/Zend/zend_objects_API.c:211 #12 0x00007fbd4fc883ff in zend_objects_store_del_ref (zobject=0xa22028) at /home/dkf/tmp/php-5.2.9/Zend/zend_objects_API.c:169 #13 0x00007fbd4fc5b0c5 in _zval_ptr_dtor (zval_ptr=0xa220d0) at /home/dkf/tmp/php-5.2.9/Zend/zend_variables.h:35 #14 0x00007fbd4fc73980 in zend_hash_apply_deleter (ht=0x7fbd50023048, p=0xa220b8) at /home/dkf/tmp/php-5.2.9/Zend/zend_hash.c:805 #15 0x00007fbd4fc73aa8 in zend_hash_reverse_apply (ht=0x7fbd50023048, apply_func=0x7fbd4fc5aa00 <zval_call_destructor>) at /home/dkf/tmp/php-5.2.9/Zend/zend_hash.c:954 #16 0x00007fbd4fc5c375 in shutdown_destructors () at /home/dkf/tmp/php-5.2.9/Zend/zend_execute_API.c:211 #17 0x00007fbd4fc69154 in zend_call_destructors () at /home/dkf/tmp/php-5.2.9/Zend/zend.c:926 #18 0x00007fbd4fc23585 in php_request_shutdown (dummy=<value optimized out>) at /home/dkf/tmp/php-5.2.9/main/main.c:1457The trunk still has this problem with the testcase <?php $mp = new MemcachePool(); //server is down/unreachable $mp->addServer('127.0.0.1', 11212); $mp->get(array('cache_key1', 'cache_key2')); ?> The problem is that the callback for error code may be called multiple times. It pops items off the Q to free, in the bad case, does a double free. Trying to untangle that would be very difficult, and it would be hard to verify correctness or that it doesn't occur in a different flow. Likewise it would be hard to verify that things don't get multiply added. So this patch, while a bit blunt, absolutely solves the issue: ea333 at freemail dot hu Insert the following line to mmc_queue_push (memcache_queue.c) in first line: if (mmc_queue_contains(queue, ptr)) return; Given the size of the queues (a few items at most), this shouldn't be a performance problem. The patch dropped 100k+/day segfaults of Apache to 0 in a high volume website I'm working with. What do we need to do get this change in trunk? Happy to help in anyway. thanks, --nickgIndex: memcache_queue.c =================================================================== --- memcache_queue.c (revision 303241) +++ memcache_queue.c (working copy) @@ -27,6 +27,8 @@ #include "memcache_queue.h" inline void mmc_queue_push(mmc_queue_t *queue, void *ptr) { + if (mmc_queue_contains(queue, ptr)) return; + if (queue->len >= queue->alloc) { int increase = 1 + MMC_QUEUE_PREALLOC; queue->alloc += increase;I'm using memcache client v 3.0.6 running: /usr/sbin/php5-fpm -v PHP 5.3.3-1ubuntu9.5 (fpm-fcgi) (built: May 3 2011 00:56:00) behind nginx on ubuntu 10.10 32bit in EC2 (m1.small). My site under decent load starts to get segfaults in both: php5-fpm[4113]: segfault at 832e598 ip 083130bc sp bffdd310 error 7 in php5- fpm[8048000+70d000] and segfault at 1518c ip b6b9c1c9 sp bf8dbf70 error 4 in memcache.so When I set the following the segfaults go away memcache.allow_failover = 0 Here is my stacktrace: (gdb) bt #0 0x083130bc in _array_init () #1 0xb6abb163 in mmc_value_handler_multi (key=0x98facc4 " <MY KEY HERE>", key_len=43, value=0x8300a70, flags=0, cas=0, param=0xbffdd680) at /tmp/pear/temp/memcache/memcache.c:1518 #2 0xb6abef2c in mmc_unpack_value (mmc=0x98ba798, request=0x98fab5c, buffer=0x98fab70, key=0x98facc4 "<MY KEY HERE>", key_len=43, flags=0, cas=0, bytes=5) at /tmp/pear/temp/memcache/memcache_pool.c:525 #3 0xb6ac439c in mmc_server_read_value (mmc=0x98ba798, request=0x98fab5c) at /tmp/pear/temp/memcache/memcache_ascii_protocol.c:189 #4 0xb6ac0f07 in mmc_pool_select (pool=0x98ee6f4) at /tmp/pear/temp/memcache/memcache_pool.c:1584 #5 0xb6ac1790 in mmc_pool_run (pool=0x98ee6f4) at /tmp/pear/temp/memcache/memcache_pool.c:1670 #6 0xb6abb3ba in php_mmc_store (ht=<value optimized out>, return_value=0x98fc954, return_value_ptr=0x0, this_ptr=0x98ee08c, return_value_used=1, op=1) at /tmp/pear/temp/memcache/memcache.c:521 #7 0x08334e87 in execute_internal () #8 0xb686a103 in ?? () from /usr/lib/php5/20090626+lfs/suhosin.so #9 0x08360d30 in ?? () #10 0x0833781e in execute () #11 0xb686a570 in ?? () from /usr/lib/php5/20090626+lfs/suhosin.so #12 0x08360a1e in ?? () #13 0x0833781e in execute () #14 0xb686a570 in ?? () from /usr/lib/php5/20090626+lfs/suhosin.so #15 0x08360a1e in ?? () #16 0x0833781e in execute () #17 0xb686a570 in ?? () from /usr/lib/php5/20090626+lfs/suhosin.so #18 0x08360a1e in ?? () #19 0x0833781e in execute () #20 0xb686a570 in ?? () from /usr/lib/php5/20090626+lfs/suhosin.so #21 0x08360a1e in ?? () #22 0x0833781e in execute () #23 0xb686a570 in ?? () from /usr/lib/php5/20090626+lfs/suhosin.so #24 0x08360a1e in ?? () #25 0x0833781e in execute () #26 0xb686a570 in ?? () from /usr/lib/php5/20090626+lfs/suhosin.so #27 0x08360a1e in ?? () #28 0x0833781e in execute () #29 0xb686a570 in ?? () from /usr/lib/php5/20090626+lfs/suhosin.so #30 0x0830d5a6 in zend_execute_scripts () #31 0x082b1cb4 in php_execute_script () I dont want to use failover, so this workaround is OK for me. Any idea why this is happening?