php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #56760 incorrect "mark server as failed" on sets
Submitted: 2005-12-29 15:58 UTC Modified: 2006-01-13 11:54 UTC
From: xing at mac dot com Assigned:
Status: Closed Package: memcache (PECL)
PHP Version: 5_1 CVS-2005-12-29 OS: Centos 4.2
Private report: No CVE-ID: None
 [2005-12-29 15:58 UTC] xing at mac dot com
Description:
------------
Using this with 3 memcached servers, latest, with php 5.1.2RC1, and 2.0 package. 

With the oo interface:

$c = new memcache();
$c->addServer(server1, 11211); 
$c->addServer(server2, 11211);
$c->addServer(server3, 11211);

The following random server marked server failures errors only happen on sets and not reads and all the servers are up. It happens even when I only add one server instead of 3. 

It would throw errors on about 2 out of every 3 test pages. it would act fine, no errors, on some tries.

The cache is taking in an array (PDO fetchAll(object) result)..so an array of simple objects.


[29-Dec-2005 12:47:42] Memcache::set() [<a href='function.set'>function.set</a>]: marked server '66.180.192.246:11211' as failed (error type 8 in /xingwww/html/classes/cache.php on line 176) [context: ]

[29-Dec-2005 12:47:48] Memcache::set() [<a href='function.set'>function.set</a>]: marked server '...249:11211' as failed (error type 8 in /xingwww/html/classes/cache.php on line 176) [context: ]

[29-Dec-2005 12:47:52] Memcache::set() [<a href='function.set'>function.set</a>]: marked server '...238:11211' as failed (error type 8 in /xingwww/html/classes/cache.php on line 176) [context: ]

[29-Dec-2005 12:47:59] Memcache::set() [<a href='function.set'>function.set</a>]: marked server '...246:11211' as failed (error type 8 in /xingwww/html/classes/cache.php on line 176) [context: ]

Notice that the errors happen on all the servers configured over different requests.







Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2005-12-29 16:10 UTC] xing at mac dot com
The errors are most pronounced on pages where I'm settings upwards of 30 such array of simple objects at once. On page where with 3-4 of such write operations, frequency of write/set errors drops significantly.
 [2005-12-29 16:18 UTC] mikael at synd dot info
Do you have any example code that can reproduce the problem? For example a short script and key/values (serialized) that causes the problem to occur. Intermittant defects are generally very hard to track down.

Do you use the setCompressThreshold() method or the explicit MEMCACHE_COMPRESSED flag? Does the problem occur if you serialize() the PDO result set beforehand (eg hand a string value to memcache) instead of relying on memcache's on-the-fly serialization?
 [2005-12-29 17:01 UTC] xing at mac dot com
changing my cache api from:

memcache->set($key, $data, $cache_time);

to

memcache->set($key, serialize($data), $cache_time);

And likewise with get method for unserialize did not help to get rid of the write/set errors.

I am not using setCompressThreshold().

Let me try to dump the cache write objects to a test script for you to test out.

The 3 memcache servers I'm testing with are under load with live traffic. Sample of one server:

stats
STAT pid 25631
STAT uptime 680244
STAT time 1135896741
STAT version 1.1.12
STAT rusage_user 1942.960625
STAT rusage_system 2666.147683
STAT curr_items 3091591
STAT total_items 23103524
STAT bytes 1393144133
STAT curr_connections 56
STAT total_connections 465129
STAT connection_structures 153
STAT cmd_get 76558993
STAT cmd_set 23097542
STAT get_hits 61673675
STAT get_misses 14885318
STAT bytes_read 15490187872
STAT bytes_written 26334264253
STAT limit_maxbytes 2097152000
END
 [2005-12-29 18:34 UTC] xing at mac dot com
Pulling hair out. 

Found the write object that keeps failing 95% of the time, captured in serilized format. And writing a test script to unserlize it and write again produce no errors. 

And in the real script, I can still get it to error dump 95% of the time. I even tried to emulate the reads before the write and this is the first and only write so right now I'm going bonkers at how to simulate the real script which has tons of dependencies.
 [2005-12-30 04:28 UTC] mikael at synd dot info
If the servers are in production and heavily loaded I imagine might be some sort of timeout problem, especially if you store very large value. 

Try setting the network timeout higher like
$c->addServer(server1, 11211, true, 1, 15, 15);

Meanwhile I'll try and get memcache to trigger more verbose error messages such as including text from socket errors. I assume you haven't seen this problem with earlier versions of memcache?
 [2006-01-12 16:15 UTC] calderon dot joe at gmail dot com
i have the same problem, additionally i write the serialized mixed var to a file (in my case an array created from mysql rows) to look at its size, it seems i consistenly get this error when my var size exceeds 1Mb, my cache size is 128mb but increasing it does not help, im running memcached 1.11, will try 1.12 soon, is there a hardcoded limit of 1Mb somewhere ? slab size maybe?
 [2006-01-12 17:01 UTC] calderon dot joe at gmail dot com
ive verified this is not a bug but rather a limit of memcached, it creates 18 object size buckets ranging from 8 bytes to 1Mb , if you need larger change slabs.c and recompile memcached, i havent posted on the memcached mailing lists to see what the ramifications of doing this would be tho...


--joe
 [2006-01-13 11:54 UTC] mikael at synd dot info
This bug has been fixed in CVS.

In case this was a documentation problem, the fix will show up at the
end of next Sunday (CET) on pecl.php.net.

In case this was a pecl.php.net website problem, the change will show
up on the website in short time.
 
Thank you for the report, and for helping us make PECL better.

Default max slab size in memcached is 1048576 bytes, storing values larger than this would always result in "out of memory" errors which in turn would cause failover. Regular "out of memory" errors should not cause failover either.
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Wed May 08 20:01:36 2024 UTC