php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Request #57750 server redundancy
Submitted: 2007-07-18 15:21 UTC Modified: 2007-12-07 14:56 UTC
From: martin dot minka at gmail dot com Assigned: mikl (profile)
Status: Closed Package: memcache (PECL)
PHP Version: 5.2.1 OS: Linux
Private report: No CVE-ID: None
 [2007-07-18 15:21 UTC] martin dot minka at gmail dot com
Description:
------------
I am running huge site on a cluster of servers running PHP. The application is depending on sessions. Memcache session handler solved the speed bottleneck of storing session data to MySQL. 

The problem is that is the memcached server goes down (problem/maintenance) users loose their sessions and the site is unusable.

Everything is redundant in cluster, only memcache session handling is risky at this moment.

I think it should be easy to fix if you could add configuration option pairing servers together. 
Something like:
session.save_path=tcp://<A1>:11211+tcp://<B1>:11211, tcp://<A2>:11211+tcp://<B2>:11211

would use servers A1 and A2 to generate the storage hash and put the same value to B1 and B2. Then if A1 is not available then B1 would be used. Same if A2 is not available then B2 will be used.

What do you think ?

Best regards,
Martin Minka


Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2007-07-25 13:10 UTC] info at tamole dot net
You could still use the database for sessions and memcached just as a cache. Then you wouldn't need redundancy.
 [2007-07-25 13:33 UTC] martin dot minka at gmail dot com
Thank you for answer, but that is not solution for my situation.

I don't need the session for longer then few minutes and to store all the traffic I have to DB, will overload my MySQL server (as it did before I started to use memcache). I need the MySQL for the PHP application.

Even if I would get HW for new MySQL I don't see a way how it would work. If one of 3 memcache servers will be lost, how will the session handler know it should fall back to query DB ? I have to code own wrapper for session handling functions, right ? That will cost me another resources in compare to pure C++ solution directly in the memcache session handler.

The missing redundancy in memcache session handler makes it impossibles to reinstall/upgrade the memcache servers ... planned downtime is needed.

I believe that redundancy of memcache servers would make this session handler (almost) perfect :-).

Sincerely,
Martin Minka
 [2007-07-25 22:26 UTC] mark dot terrill at ilisys dot com dot au
Hi Martin,

We originally wrote the php session handling, and know about the issue of no data redundancy. However the issue with writing data to pairs is that write (and potentially missing key ..its been a while since I was in the code) fetch time doubles. 

Admittedly, while losing sessions is undesirable the effect is decreased by splitting your memcache instances to shards sized to 'acceptable loss'.

I'm quite open to further discussion of the issue, so feel free to contact me about progress.

Regards,
Mark
 [2007-07-30 08:42 UTC] manuel at kiessling dot net
Hi Martin,

we recently switched to memcached, and I'm currently thinking about the same problems.

My idea is: memcache is so fast, it really should not hurt that much to always connect to 2 or 3 memcache daemons and write the current session to all of them. For reading, it's of course enough to read from one server (and fail over to another one if it has gone away).

This, however, would make it necessary to write your own session functions, but this is easy with PHP.
 [2007-08-02 13:30 UTC] mikael at synd dot info
Hi,

I've before had thoughts about implementing redundancy where you specify the number of mirrors you would like (say 2 or 3) and pecl/memcache automatically sends set/incr/decr/.. to all mirrors. The proposed solution however seems to complicate an already complicated interface and config format, so how about this:

 * By selecting the servers automatically and in the same order as when failing over in the normal case, only store operations would have to be changed (ie. send the command to all mirrors) and the retrieval code could stay the same.

 * The new NON_BLOCKING_IO branch enables us to send store commands to all mirrors simultaneously, thus reducing the cost and latency of redundancy greatly.

 * Introduce two new INI settings

 memcache.redundancy = <int>
 memcache.session_redundancy = <int>

 * Optionally introduce a parameter to MemcachePool (either via a method or in the constructor) which allows you to control the redundancy on a pool/connection basis

Thoughts?

//Mikael
 [2007-08-06 12:27 UTC] carlier dot nicolas at gmail dot com
I'm very interresting about this solution. Are you begni to code this or not?
 [2007-08-06 13:25 UTC] martin dot minka at gmail dot com
Hello Mikael,
sorry to answer late, but I was traveling. This functionality is still top priority for website I am administering.

To your suggestion:
1. I don't think the actual format of session.save_path is too complicated. I am sure that who is using memcache as session handler met more difficult semantic in his life ;)
2. I don't see value of load balancing (how I understand the actual usage of more servers in memcache session handler) if all writes will be sent to all servers when I add redundancy servers
3. format like:
session.save_path=tcp://<A1>:11211 & tcp://<B1>:11211,
tcp://<A2>:11211 & tcp://<B2>:11211 | tcp://<A3>:11211 &
tcp://<B3>:11211 & tcp://<C3>:11211 
where & defines redundancy and | defines balancing would make session handler very configurable, add redundancy and keep the benefits of load balancing
4. but I have to say, that I am not good advocate for the load balancing functionality. I would be fine to have only redundancy in the session handler. I never reached the situation that I would need to use load balancing of memcache. 

Any how...I am happy that some others and also you would like to see redundancy in memcache/PECL. You are the author and you did a great job in design and codding of this extremely useful library. The most important from perspective of high availability is to get redundancy implemented.

Please, let me know what you decide and if you have time to implement it or if I should start to work on it.

To make this discussion complete, I am including below my proposal which I sent you privately before.

Sincerely,
Martin Minka

**************************************
I am interested mainly into redundancy of session data, but the changes
would make this functionality available also to the functions exported
to PHP.

Benefits why to add this to the PECL library:
A. redundant storage of data to more servers and fall back for read operation
B. upgrade to new version of memcache server without downtime
C. no need to add mysql support to achieve acceptable fault tolerance
for data like web page caching and session handling
D. no need to create session handlers written in PHP language to achieve this missing functionality

My idea how to modify the code to achieve this:
1. add to struct mmc_t list of redundancy servers:
mmc_t *backupServers;

2. add function Memcache::addBackupServer() which would fill
backupServers variable, example:
$mc = new Memcache();
$mc->addServer("<A1>", 11211);
$mc->addBackupServer("<A1>", "<B1>", 11211);
$mc->addServer("<A2>", 11211);
$mc->addBackupServer("<A2>", "<B2>", 11211);
$mc->addServer("<A3>", 11211);
$mc->addBackupServer("<A3>", "<B3>", 11211);
$mc->addBackupServer("<A3>", "<C3>", 11211);

3. in memcache.c(1098) do not stop with error, but try other servers in
mmc->backupServers

4. implement the same loop as in (3) in mmc_exec_retrieval_cmd_multi

5. add loop to mmc_server_store, mmc_delete, mmc_flush to do the same on
all backup servers

6. modify PS_OPEN_FUNC(memcache) in memcache_session.c to parse
characters ',' and '|' as new servers and parse part after character '&'
as new backup server. for example:
session.save_path=tcp://<A1>:11211 & tcp://<B1>:11211,
tcp://<A2>:11211 & tcp://<B2>:11211 | tcp://<A3>:11211 &
tcp://<B3>:11211 & tcp://<C3>:11211
Of course in case of session handler in cluster the settings needs to be
the same on all servers otherwise it will not work.

I think it is not too much code to add and will also not cost too much
speed lost if user will not define backup servers. All functions will
also keep their compatibility.

Hope you support it and that you will advice me how to proceed. I am
able to add the code by myself, but I think it should be scored by the
memcache authors.
 [2007-09-26 16:48 UTC] mikael at synd dot info
Hi,

In the NON_BLOCKING_IO CVS-branch there's now the possibility to specify redundancy (like RAID-1) so that writes are sent to n mirrors, and failed get's are retried on the mirrors. The behavior is controlled through two new INI directives whose default values are "1"

 memcache.redundancy - when used from PHP
 memcache.session_redundancy - when used as a session handler

For example setting "memcache.session_redundancy = 2" would result in sessions being written to 2 servers.

Mirrored writes are done in parallel (using non-blocking-io) so performance shouldn't go down much as the number of mirrors increases. 

It's possible to check-out and build a copy of the branch like

 cvs -d :pserver:cvsread at cvs.php.net:/repository co -r NON_BLOCKING_IO pecl/memcache
 cd pecl/memcache
 phpize
 ./configure
 make

Since the branch will soon reach beta, help with testing and profiling would be much appreciated.

//Mikael
 [2007-09-27 10:28 UTC] martin dot minka at gmail dot com
Mikael,
this is great news.

I tested the session handler with 2 different memcache servers and it worked fine. The test was basic and my plan is to test it on a fully loaded cluster next week.

I created test script for session redundancy downloadable at http://in.solit.us/archives/download/71218. Unfortunately memcache session handler does not work for me if I am using it from shell, so my test is not compatible with 'make test'. Still it could help others to test this functionality. Maybe you could improve it and add to CVS.

Sincerely,
Martin
 [2007-12-07 14:56 UTC] martin dot minka at gmail dot com
I am using this functionality on highly loaded cluster of web servers for 3 weeks now and it works without problems.

It is now easy to upgrade the memcached server to new version without interuption.

Best regards,
Martin Minka
 
PHP Copyright © 2001-2019 The PHP Group
All rights reserved.
Last updated: Wed Sep 18 03:01:27 2019 UTC