memcache-client 1.5.0 → 1.6.2

data/History.txt

@@ -1,3 +1,74 @@
+= 1.6.2 (2009-02-04)
+
+* Validate that values are less than one megabyte in size.
+
+* Refactor error handling in get_multi to handle server failures and return what values
+  we could successfully retrieve.
+
+* Add optional logging parameter for debugging and tracing.
+
+* First official release since 1.5.0.  Thanks to Eric Hodel for turning over the project to me!
+  New project home page: http://github.com/mperham/memcache-client
+
+= 1.6.1 (2009-01-28)
+
+* Add option to disable socket timeout support.  Socket timeout has a significant performance 
+  penalty (approx 3x slower than without in Ruby 1.8.6).  You can turn off the timeouts if you 
+  need absolute performance, but by default timeouts are enabled. The performance
+  penalty is much lower in Ruby 1.8.7, 1.9 and JRuby.  (mperham)
+
+* Add option to disable server failover.  Failover can lead to "split-brain" caches that 
+  return stale data. (mperham)
+
+* Implement continuum binary search in native code for performance reasons.  Pure ruby
+  is available for platforms like JRuby or Rubinius which can't use C extensions. (mperham)
+  
+* Fix #add with raw=true (iamaleksey)
+
+= 1.6.0
+
+* Implement a consistent hashing algorithm, as described in libketama.
+  This dramatically reduces the cost of adding or removing servers dynamically
+  as keys are much more likely to map to the same server.
+
+  Take a scenario where we add a fourth server.  With a naive modulo algorithm, about
+  25% of the keys will map to the same server.  In other words, 75% of your memcached
+  content suddenly becomes invalid.  With a consistent algorithm, 75% of the keys
+  will map to the same server as before - only 25% will be invalidated. (mperham)
+
+* Implement socket timeouts, should fix rare cases of very bad things happening
+  in production at 37signals and FiveRuns.  (jseirles)
+
+= 1.5.0.5
+
+* Remove native C CRC32_ITU_T extension in favor of Zlib's crc32 method.
+  memcache-client is now pure Ruby again and will work with JRuby and Rubinius.
+
+= 1.5.0.4
+
+* Get test suite working again (packagethief)
+* Ruby 1.9 compatiblity fixes (packagethief, mperham)
+* Consistently return server responses and check for errors (packagethief)
+* Properly calculate CRC in Ruby 1.9 strings (mperham)
+* Drop rspec in favor of test/unit, for 1.9 compat (mperham)
+
+= 1.5.0.3 (FiveRuns fork)
+
+* Integrated ITU-T CRC32 operation in native C extension for speed.  Thanks to Justin Balthrop!
+
+= 1.5.0.2 (FiveRuns fork)
+
+* Add support for seamless failover between servers.  If one server connection dies,
+the client will retry the operation on another server before giving up.
+
+* Merge Will Bryant's socket retry patch.
+http://willbryant.net/software/2007/12/21/ruby-memcache-client-reconnect-and-retry
+
+= 1.5.0.1 (FiveRuns fork)
+
+* Fix set not handling client disconnects.
+http://dev.twitter.com/2008/02/solving-case-of-missing-updates.html
+
 = 1.5.0
 
 * Add MemCache#flush_all command.  Patch #13019 and bug #10503.  Patches

data/LICENSE.txt

@@ -1,5 +1,5 @@
-All original code copyright 2005, 2006, 2007 Bob Cottrell, Eric Hodel,
-The Robot Co-op.  All rights reserved.
+Copyright 2005-2009 Bob Cottrell, Eric Hodel, Mike Perham.
+All rights reserved.
 
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions

data/README.rdoc

@@ -0,0 +1,43 @@
+= memcache-client
+
+A pure ruby library for accessing memcached.
+
+Source:
+
+http://github.com/mperham/memcache-client
+
+== Installing memcache-client
+
+Just install the gem:
+
+  $ sudo gem install memcache-client
+
+== Using memcache-client
+
+With one server:
+
+  CACHE = MemCache.new 'localhost:11211', :namespace => 'my_namespace'
+
+Or with multiple servers:
+
+  CACHE = MemCache.new %w[one.example.com:11211 two.example.com:11211],
+                       :namespace => 'my_namespace'
+
+See MemCache.new for details.  Please note memcache-client is not thread-safe
+by default.  You should create a separate instance for each thread in your
+process.
+
+== Using memcache-client with Rails
+
+There's no need to use memcache-client in a Rails application.  Rails 2.1+ includes
+a basic caching library which can be used with memcached.  See ActiveSupport::Cache::Store
+for more details.
+
+== Questions?
+
+memcache-client is maintained by Mike Perham and was originally written by Bob Cottrell, 
+Eric Hodel and the seattle.rb crew.
+
+Email::        mailto:mperham@gmail.com
+Twitter::      mperham[http://twitter.com/mperham]
+WWW::          http://mikeperham.com

data/Rakefile

@@ -1,19 +1,26 @@
 # vim: syntax=Ruby
+require 'rubygems'
+require 'rake/rdoctask'
+require 'rake/testtask'
 
-require 'hoe'
-
-$:.unshift 'lib'
-require 'memcache'
+task :gem do
+	sh "gem build memcache-client.gemspec"
+end
 
-hoe = Hoe.new 'memcache-client', MemCache::VERSION do |p|
-  p.summary = 'A Ruby memcached client'
-  p.description = p.paragraphs_of('README.txt', 8).first
-  p.author = ['Eric Hodel', 'Robert Cottrell']
-  p.email = 'drbrain@segment7.net'
-  p.url = p.paragraphs_of('README.txt', 6).first
-  p.changes = File.read('History.txt').scan(/\A(=.*?)^=/m).first.first
+task :install => [:gem] do
+	sh "sudo gem install memcache-client-*.gem"
+end
 
-  p.rubyforge_name = 'seattlerb'
-  p.extra_deps << ['ZenTest', '>= 3.4.2']
+Rake::RDocTask.new do |rd|
+	rd.main = "README.rdoc"
+	rd.rdoc_files.include("README.rdoc", "lib/**/*.rb")
+	rd.rdoc_dir = 'doc'
 end
 
+Rake::TestTask.new
+
+task :default => :test
+
+task :rcov do
+  `rcov -Ilib test/*.rb`
+end

data/ext/memcache/binary_search.c

@@ -0,0 +1,54 @@
+#include "ruby.h"
+#include "stdio.h"
+
+/*
+def binary_search(ary, value)
+  upper = ary.size - 1
+  lower = 0
+  idx = 0
+
+  while(lower <= upper) do
+    idx = (lower + upper) / 2
+    comp = ary[idx].value <=> value
+
+    if comp == 0
+      return idx
+    elsif comp > 0
+      upper = idx - 1
+    else
+      lower = idx + 1
+    end
+  end
+  return upper
+end
+*/ 
+static VALUE binary_search(VALUE self, VALUE ary, VALUE number) {
+    int upper = RARRAY_LEN(ary) - 1;
+    int lower = 0;
+    int idx = 0;
+    unsigned int r = NUM2UINT(number);
+    ID value = rb_intern("value");
+    
+    while (lower <= upper) {
+        idx = (lower + upper) / 2;
+        
+        VALUE continuumValue = rb_funcall(RARRAY_PTR(ary)[idx], value, 0);
+        unsigned int l = NUM2UINT(continuumValue);
+        if (l == r) {
+            return INT2FIX(idx);
+        }
+        else if (l > r) {
+            upper = idx - 1;
+        }
+        else {
+            lower = idx + 1;
+        }
+    }
+    return INT2FIX(upper);
+}
+ 
+VALUE cContinuum;
+void Init_binary_search() {
+  cContinuum = rb_define_module("Continuum");
+  rb_define_module_function(cContinuum, "binary_search", binary_search, 2);
+}

data/ext/memcache/extconf.rb

@@ -0,0 +1,5 @@
+require 'mkmf'
+ 
+dir_config("binary_search")
+ 
+create_makefile("binary_search")

data/lib/continuum.rb

@@ -0,0 +1,46 @@
+module Continuum
+  POINTS_PER_SERVER = 160 # this is the default in libmemcached
+
+  begin
+    require 'binary_search' # try to load native extension
+  rescue LoadError => e
+    puts "Unable to load fast binary search, falling back to pure Ruby: #{e.message}"
+
+    # slow but pure ruby version
+    # Find the closest index in Continuum with value <= the given value
+    def self.binary_search(ary, value, &block)
+      upper = ary.size - 1
+      lower = 0
+      idx = 0
+
+      while(lower <= upper) do
+        idx = (lower + upper) / 2
+        comp = ary[idx].value <=> value
+ 
+        if comp == 0
+          return idx
+        elsif comp > 0
+          upper = idx - 1
+        else
+          lower = idx + 1
+        end
+      end
+      return upper
+    end
+  end
+
+
+  class Entry
+    attr_reader :value
+    attr_reader :server
+
+    def initialize(val, srv)
+      @value = val
+      @server = srv
+    end
+
+    def inspect
+      "<#{value}, #{server.host}:#{server.port}>"
+    end
+  end
+end

data/lib/memcache.rb

@@ -3,46 +3,21 @@ $TESTING = defined?($TESTING) && $TESTING
 require 'socket'
 require 'thread'
 require 'timeout'
-require 'rubygems'
+require 'zlib'
+require 'digest/sha1'
 
-class String
-
-  ##
-  # Uses the ITU-T polynomial in the CRC32 algorithm.
-
-  def crc32_ITU_T
-    n = length
-    r = 0xFFFFFFFF
-
-    n.times do |i|
-      r ^= self[i]
-      8.times do
-        if (r & 1) != 0 then
-          r = (r>>1) ^ 0xEDB88320
-        else
-          r >>= 1
-        end
-      end
-    end
-
-    r ^ 0xFFFFFFFF
-  end
-
-end
+require 'continuum'
 
 ##
 # A Ruby client library for memcached.
 #
-# This is intended to provide access to basic memcached functionality.  It
-# does not attempt to be complete implementation of the entire API, but it is
-# approaching a complete implementation.
 
 class MemCache
 
   ##
   # The version of MemCache you are using.
 
-  VERSION = '1.5.0'
+  VERSION = '1.6.2'
 
   ##
   # Default options for the cache object.
@@ -51,6 +26,9 @@ class MemCache
     :namespace   => nil,
     :readonly    => false,
     :multithread => false,
+    :failover    => true,
+    :timeout     => 0.5,
+    :logger      => nil,
   }
 
   ##
@@ -63,13 +41,6 @@ class MemCache
 
   DEFAULT_WEIGHT = 1
 
-  ##
-  # The amount of time to wait for a response from a memcached server.  If a
-  # response is not completed within this time, the connection to the server
-  # will be closed and an error will be raised.
-
-  attr_accessor :request_timeout
-
   ##
   # The namespace for this instance
 
@@ -85,6 +56,23 @@ class MemCache
 
   attr_reader :servers
 
+  ##
+  # Socket timeout limit with this client, defaults to 0.25 sec.
+  # Set to nil to disable timeouts.
+
+  attr_reader :timeout
+
+  ##
+  # Should the client try to failover to another server if the
+  # first server is down?  Defaults to true.
+
+  attr_reader :failover
+
+  ##
+  # Log debug/info/warn/error to the given Logger, defaults to nil.
+
+  attr_reader :logger
+
   ##
   # Accepts a list of +servers+ and a list of +opts+.  +servers+ may be
   # omitted.  See +servers=+ for acceptable server list arguments.
@@ -92,9 +80,13 @@ class MemCache
   # Valid options for +opts+ are:
   #
   #   [:namespace]   Prepends this value to all keys added or retrieved.
-  #   [:readonly]    Raises an exeception on cache writes when true.
+  #   [:readonly]    Raises an exception on cache writes when true.
   #   [:multithread] Wraps cache access in a Mutex for thread safety.
-  #
+  #   [:failover]    Should the client try to failover to another server if the
+  #                  first server is down?  Defaults to true.
+  #   [:timeout]     Time to use as the socket read timeout.  Defaults to 0.25 sec,
+  #                  set to nil to disable timeouts (this is a major performance penalty in Ruby 1.8).
+  #   [:logger]      Logger to use for info/debug output, defaults to nil
   # Other options are ignored.
 
   def initialize(*args)
@@ -121,8 +113,13 @@ class MemCache
     @namespace   = opts[:namespace]
     @readonly    = opts[:readonly]
     @multithread = opts[:multithread]
+    @timeout     = opts[:timeout]
+    @failover    = opts[:failover]
+    @logger      = opts[:logger]
     @mutex       = Mutex.new if @multithread
-    @buckets     = []
+
+    logger.info { "memcache-client #{VERSION} #{Array(servers).inspect}" } if logger
+
     self.servers = servers
   end
 
@@ -130,8 +127,8 @@ class MemCache
   # Returns a string representation of the cache object.
 
   def inspect
-    "<MemCache: %d servers, %d buckets, ns: %p, ro: %p>" %
-      [@servers.length, @buckets.length, @namespace, @readonly]
+    "<MemCache: %d servers, ns: %p, ro: %p>" %
+      [@servers.length, @namespace, @readonly]
   end
 
   ##
@@ -152,48 +149,44 @@ class MemCache
   # Set the servers that the requests will be distributed between.  Entries
   # can be either strings of the form "hostname:port" or
   # "hostname:port:weight" or MemCache::Server objects.
-
+  #
   def servers=(servers)
     # Create the server objects.
-    @servers = servers.collect do |server|
+    @servers = Array(servers).collect do |server|
       case server
       when String
         host, port, weight = server.split ':', 3
         port ||= DEFAULT_PORT
         weight ||= DEFAULT_WEIGHT
         Server.new self, host, port, weight
-      when Server
-        if server.memcache.multithread != @multithread then
+      else
+        if server.multithread != @multithread then
           raise ArgumentError, "can't mix threaded and non-threaded servers"
         end
         server
-      else
-        raise TypeError, "cannot convert #{server.class} into MemCache::Server"
       end
     end
 
-    # Create an array of server buckets for weight selection of servers.
-    @buckets = []
-    @servers.each do |server|
-      server.weight.times { @buckets.push(server) }
-    end
+    logger.debug { "Servers now: #{@servers.inspect}" } if logger
+
+    # There's no point in doing this if there's only one server
+    @continuum = create_continuum_for(@servers) if @servers.size > 1
+
+    @servers
   end
 
   ##
-  # Deceremets the value for +key+ by +amount+ and returns the new value.
+  # Decrements the value for +key+ by +amount+ and returns the new value.
   # +key+ must already exist.  If +key+ is not an integer, it is assumed to be
   # 0.  +key+ can not be decremented below 0.
 
   def decr(key, amount = 1)
-    server, cache_key = request_setup key
-
-    if @multithread then
-      threadsafe_cache_decr server, cache_key, amount
-    else
+    raise MemCacheError, "Update of readonly cache" if @readonly
+    with_server(key) do |server, cache_key|
       cache_decr server, cache_key, amount
     end
-  rescue TypeError, SocketError, SystemCallError, IOError => err
-    handle_error server, err
+  rescue TypeError => err
+    handle_error nil, err
   end
 
   ##
@@ -201,21 +194,15 @@ class MemCache
   # unmarshalled.
 
   def get(key, raw = false)
-    server, cache_key = request_setup key
-
-    value = if @multithread then
-              threadsafe_cache_get server, cache_key
-            else
-              cache_get server, cache_key
-            end
-
-    return nil if value.nil?
-
-    value = Marshal.load value unless raw
-
-    return value
-  rescue TypeError, SocketError, SystemCallError, IOError => err
-    handle_error server, err
+    with_server(key) do |server, cache_key|
+      value = cache_get server, cache_key
+      logger.debug { "GET #{key} from #{server.inspect}: #{value ? value.to_s.size : 'nil'}" } if logger
+      return nil if value.nil?
+      value = Marshal.load value unless raw
+      return value
+    end
+  rescue TypeError => err
+    handle_error nil, err
   end
 
   ##
@@ -251,38 +238,36 @@ class MemCache
 
     results = {}
 
-    server_keys.each do |server, keys|
-      keys = keys.join ' '
-      values = if @multithread then
-                 threadsafe_cache_get_multi server, keys
-               else
-                 cache_get_multi server, keys
-               end
-      values.each do |key, value|
-        results[cache_keys[key]] = Marshal.load value
+    server_keys.each do |server, keys_for_server|
+      keys_for_server_str = keys_for_server.join ' '
+      begin
+        values = cache_get_multi server, keys_for_server_str
+        values.each do |key, value|
+          results[cache_keys[key]] = Marshal.load value
+        end
+      rescue IndexError => e
+        # Ignore this server and try the others
+        logger.warn { "Unable to retrieve #{keys_for_server.size} elements from #{server.inspect}: #{e.message}"} if logger
       end
     end
 
     return results
-  rescue TypeError, SocketError, SystemCallError, IOError => err
-    handle_error server, err
+  rescue TypeError => err
+    handle_error nil, err
   end
 
   ##
-  # Increments the value for +key+ by +amount+ and retruns the new value.
+  # Increments the value for +key+ by +amount+ and returns the new value.
   # +key+ must already exist.  If +key+ is not an integer, it is assumed to be
   # 0.
 
   def incr(key, amount = 1)
-    server, cache_key = request_setup key
-
-    if @multithread then
-      threadsafe_cache_incr server, cache_key, amount
-    else
+    raise MemCacheError, "Update of readonly cache" if @readonly
+    with_server(key) do |server, cache_key|
       cache_incr server, cache_key, amount
     end
-  rescue TypeError, SocketError, SystemCallError, IOError => err
-    handle_error server, err
+  rescue TypeError => err
+    handle_error nil, err
   end
 
   ##
@@ -292,24 +277,32 @@ class MemCache
   # Warning: Readers should not call this method in the event of a cache miss;
   # see MemCache#add.
 
+  ONE_MB = 1024 * 1024
+
   def set(key, value, expiry = 0, raw = false)
     raise MemCacheError, "Update of readonly cache" if @readonly
-    server, cache_key = request_setup key
-    socket = server.socket
+    with_server(key) do |server, cache_key|
 
-    value = Marshal.dump value unless raw
-    command = "set #{cache_key} 0 #{expiry} #{value.size}\r\n#{value}\r\n"
+      value = Marshal.dump value unless raw
+      logger.debug { "SET #{key} to #{server.inspect}: #{value ? value.to_s.size : 'nil'}" } if logger
 
-    begin
-      @mutex.lock if @multithread
-      socket.write command
-      result = socket.gets
-      raise MemCacheError, $1.strip if result =~ /^SERVER_ERROR (.*)/
-    rescue SocketError, SystemCallError, IOError => err
-      server.close
-      raise MemCacheError, err.message
-    ensure
-      @mutex.unlock if @multithread
+      data = value.to_s
+      raise MemCacheError, "Value too large, memcached can only store 1MB of data per key" if data.size > ONE_MB
+
+      command = "set #{cache_key} 0 #{expiry} #{data.size}\r\n#{data}\r\n"
+
+      with_socket_management(server) do |socket|
+        socket.write command
+        result = socket.gets
+        raise_on_error_response! result
+
+        if result.nil?
+          server.close
+          raise MemCacheError, "lost connection to #{server.host}:#{server.port}"
+        end
+
+        result
+      end
     end
   end
 
@@ -323,21 +316,17 @@ class MemCache
 
   def add(key, value, expiry = 0, raw = false)
     raise MemCacheError, "Update of readonly cache" if @readonly
-    server, cache_key = request_setup key
-    socket = server.socket
-
-    value = Marshal.dump value unless raw
-    command = "add #{cache_key} 0 #{expiry} #{value.size}\r\n#{value}\r\n"
-
-    begin
-      @mutex.lock if @multithread
-      socket.write command
-      socket.gets
-    rescue SocketError, SystemCallError, IOError => err
-      server.close
-      raise MemCacheError, err.message
-    ensure
-      @mutex.unlock if @multithread
+    with_server(key) do |server, cache_key|
+      value = Marshal.dump value unless raw
+      logger.debug { "ADD #{key} to #{server}: #{value ? value.to_s.size : 'nil'}" } if logger
+      command = "add #{cache_key} 0 #{expiry} #{value.to_s.size}\r\n#{value}\r\n"
+
+      with_socket_management(server) do |socket|
+        socket.write command
+        result = socket.gets
+        raise_on_error_response! result
+        result
+      end
     end
   end
 
@@ -345,24 +334,15 @@ class MemCache
   # Removes +key+ from the cache in +expiry+ seconds.
 
   def delete(key, expiry = 0)
-    @mutex.lock if @multithread
-
-    raise MemCacheError, "No active servers" unless active?
-    cache_key = make_cache_key key
-    server = get_server_for_key cache_key
-
-    sock = server.socket
-    raise MemCacheError, "No connection to server" if sock.nil?
-
-    begin
-      sock.write "delete #{cache_key} #{expiry}\r\n"
-      sock.gets
-    rescue SocketError, SystemCallError, IOError => err
-      server.close
-      raise MemCacheError, err.message
+    raise MemCacheError, "Update of readonly cache" if @readonly
+    with_server(key) do |server, cache_key|
+      with_socket_management(server) do |socket|
+        socket.write "delete #{cache_key} #{expiry}\r\n"
+        result = socket.gets
+        raise_on_error_response! result
+        result
+      end
     end
-  ensure
-    @mutex.unlock if @multithread
   end
 
   ##
@@ -371,20 +351,19 @@ class MemCache
   def flush_all
     raise MemCacheError, 'No active servers' unless active?
     raise MemCacheError, "Update of readonly cache" if @readonly
+
     begin
       @mutex.lock if @multithread
       @servers.each do |server|
-        begin
-          sock = server.socket
-          raise MemCacheError, "No connection to server" if sock.nil?
-          sock.write "flush_all\r\n"
-          result = sock.gets
-          raise MemCacheError, $2.strip if result =~ /^(SERVER_)?ERROR(.*)/
-        rescue SocketError, SystemCallError, IOError => err
-          server.close
-          raise MemCacheError, err.message
+        with_socket_management(server) do |socket|
+          socket.write "flush_all\r\n"
+          result = socket.gets
+          raise_on_error_response! result
+          result
         end
       end
+    rescue IndexError => err
+      handle_error nil, err
     ensure
       @mutex.unlock if @multithread
     end
@@ -436,16 +415,16 @@ class MemCache
     server_stats = {}
 
     @servers.each do |server|
-      sock = server.socket
-      raise MemCacheError, "No connection to server" if sock.nil?
+      next unless server.alive?
 
-      value = nil
-      begin
-        sock.write "stats\r\n"
+      with_socket_management(server) do |socket|
+        value = nil
+        socket.write "stats\r\n"
         stats = {}
-        while line = sock.gets do
+        while line = socket.gets do
+          raise_on_error_response! line
           break if line == "END\r\n"
-          if line =~ /^STAT ([\w]+) ([\w\.\:]+)/ then
+          if line =~ /\ASTAT ([\S]+) ([\w\.\:]+)/ then
             name, value = $1, $2
             stats[name] = case name
                           when 'version'
@@ -455,7 +434,7 @@ class MemCache
                             microseconds ||= 0
                             Float(seconds) + (Float(microseconds) / 1_000_000)
                           else
-                            if value =~ /^\d+$/ then
+                            if value =~ /\A\d+\Z/ then
                               value.to_i
                             else
                               value
@@ -464,12 +443,10 @@ class MemCache
           end
         end
         server_stats["#{server.host}:#{server.port}"] = stats
-      rescue SocketError, SystemCallError, IOError => err
-        server.close
-        raise MemCacheError, err.message
       end
     end
 
+    raise MemCacheError, "No active servers" if server_stats.empty?
     server_stats
   end
 
@@ -500,45 +477,49 @@ class MemCache
     end
   end
 
+  ##
+  # Returns an interoperable hash value for +key+.  (I think, docs are
+  # sketchy for down servers).
+
+  def hash_for(key)
+    Zlib.crc32(key)
+  end
+
   ##
   # Pick a server to handle the request based on a hash of the key.
 
-  def get_server_for_key(key)
+  def get_server_for_key(key, options = {})
     raise ArgumentError, "illegal character in key #{key.inspect}" if
       key =~ /\s/
     raise ArgumentError, "key too long #{key.inspect}" if key.length > 250
     raise MemCacheError, "No servers available" if @servers.empty?
     return @servers.first if @servers.length == 1
 
-    hkey = hash_for key
+    hkey = hash_for(key)
 
     20.times do |try|
-      server = @buckets[hkey % @buckets.nitems]
+      entryidx = Continuum.binary_search(@continuum, hkey)
+      server = @continuum[entryidx].server
       return server if server.alive?
-      hkey += hash_for "#{try}#{key}"
+      break unless failover
+      hkey = hash_for "#{try}#{key}"
     end
-
+    
     raise MemCacheError, "No servers available"
   end
 
-  ##
-  # Returns an interoperable hash value for +key+.  (I think, docs are
-  # sketchy for down servers).
-
-  def hash_for(key)
-    (key.crc32_ITU_T >> 16) & 0x7fff
-  end
-
   ##
   # Performs a raw decr for +cache_key+ from +server+.  Returns nil if not
   # found.
 
   def cache_decr(server, cache_key, amount)
-    socket = server.socket
-    socket.write "decr #{cache_key} #{amount}\r\n"
-    text = socket.gets
-    return nil if text == "NOT_FOUND\r\n"
-    return text.to_i
+    with_socket_management(server) do |socket|
+      socket.write "decr #{cache_key} #{amount}\r\n"
+      text = socket.gets
+      raise_on_error_response! text
+      return nil if text == "NOT_FOUND\r\n"
+      return text.to_i
+    end
   end
 
   ##
@@ -546,50 +527,54 @@ class MemCache
   # miss.
 
   def cache_get(server, cache_key)
-    socket = server.socket
-    socket.write "get #{cache_key}\r\n"
-    keyline = socket.gets # "VALUE <key> <flags> <bytes>\r\n"
+    with_socket_management(server) do |socket|
+      socket.write "get #{cache_key}\r\n"
+      keyline = socket.gets # "VALUE <key> <flags> <bytes>\r\n"
 
-    if keyline.nil? then
-      server.close
-      raise MemCacheError, "lost connection to #{server.host}:#{server.port}"
-    end
+      if keyline.nil? then
+        server.close
+        raise MemCacheError, "lost connection to #{server.host}:#{server.port}"
+      end
 
-    return nil if keyline == "END\r\n"
+      raise_on_error_response! keyline
+      return nil if keyline == "END\r\n"
 
-    unless keyline =~ /(\d+)\r/ then
-      server.close
-      raise MemCacheError, "unexpected response #{keyline.inspect}"
+      unless keyline =~ /(\d+)\r/ then
+        server.close
+        raise MemCacheError, "unexpected response #{keyline.inspect}"
+      end
+      value = socket.read $1.to_i
+      socket.read 2 # "\r\n"
+      socket.gets   # "END\r\n"
+      return value
     end
-    value = socket.read $1.to_i
-    socket.read 2 # "\r\n"
-    socket.gets   # "END\r\n"
-    return value
   end
 
   ##
   # Fetches +cache_keys+ from +server+ using a multi-get.
 
   def cache_get_multi(server, cache_keys)
-    values = {}
-    socket = server.socket
-    socket.write "get #{cache_keys}\r\n"
+    with_socket_management(server) do |socket|
+      values = {}
+      socket.write "get #{cache_keys}\r\n"
 
-    while keyline = socket.gets do
-      return values if keyline == "END\r\n"
+      while keyline = socket.gets do
+        return values if keyline == "END\r\n"
+        raise_on_error_response! keyline
 
-      unless keyline =~ /^VALUE (.+) (.+) (.+)/ then
-        server.close
-        raise MemCacheError, "unexpected response #{keyline.inspect}"
+        unless keyline =~ /\AVALUE (.+) (.+) (.+)/ then
+          server.close
+          raise MemCacheError, "unexpected response #{keyline.inspect}"
+        end
+
+        key, data_length = $1, $3
+        values[$1] = socket.read data_length.to_i
+        socket.read(2) # "\r\n"
       end
 
-      key, data_length = $1, $3
-      values[$1] = socket.read data_length.to_i
-      socket.read(2) # "\r\n"
+      server.close
+      raise MemCacheError, "lost connection to #{server.host}:#{server.port}" # TODO: retry here too
     end
-
-    server.close
-    raise MemCacheError, "lost connection to #{server.host}:#{server.port}"
   end
 
   ##
@@ -597,17 +582,79 @@ class MemCache
   # found.
 
   def cache_incr(server, cache_key, amount)
-    socket = server.socket
-    socket.write "incr #{cache_key} #{amount}\r\n"
-    text = socket.gets
-    return nil if text == "NOT_FOUND\r\n"
-    return text.to_i
+    with_socket_management(server) do |socket|
+      socket.write "incr #{cache_key} #{amount}\r\n"
+      text = socket.gets
+      raise_on_error_response! text
+      return nil if text == "NOT_FOUND\r\n"
+      return text.to_i
+    end
+  end
+
+  ##
+  # Gets or creates a socket connected to the given server, and yields it
+  # to the block, wrapped in a mutex synchronization if @multithread is true.
+  #
+  # If a socket error (SocketError, SystemCallError, IOError) or protocol error
+  # (MemCacheError) is raised by the block, closes the socket, attempts to
+  # connect again, and retries the block (once).  If an error is again raised,
+  # reraises it as MemCacheError.
+  #
+  # If unable to connect to the server (or if in the reconnect wait period),
+  # raises MemCacheError.  Note that the socket connect code marks a server
+  # dead for a timeout period, so retrying does not apply to connection attempt
+  # failures (but does still apply to unexpectedly lost connections etc.).
+
+  def with_socket_management(server, &block)
+    @mutex.lock if @multithread
+    retried = false
+
+    begin
+      socket = server.socket
+
+      # Raise an IndexError to show this server is out of whack. If were inside
+      # a with_server block, we'll catch it and attempt to restart the operation.
+
+      raise IndexError, "No connection to server (#{server.status})" if socket.nil?
+
+      block.call(socket)
+
+    rescue SocketError => err
+      logger.warn { "Socket failure: #{err.message}" } if logger
+      server.mark_dead(err)
+      handle_error(server, err)
+
+    rescue MemCacheError, SystemCallError, IOError => err
+      logger.warn { "Generic failure: #{err.class.name}: #{err.message}" } if logger
+      handle_error(server, err) if retried || socket.nil?
+      retried = true
+      retry
+    end
+  ensure
+    @mutex.unlock if @multithread
+  end
+
+  def with_server(key)
+    retried = false
+    begin
+      server, cache_key = request_setup(key)
+      yield server, cache_key
+    rescue IndexError => e
+      logger.warn { "Server failed: #{e.class.name}: #{e.message}" } if logger
+      if !retried && @servers.size > 1
+        logger.info { "Connection to server #{server.inspect} DIED! Retrying operation..." } if logger
+        retried = true
+        retry
+      end
+      handle_error(nil, e)
+    end
   end
 
   ##
   # Handles +error+ from +server+.
 
   def handle_error(server, error)
+    raise error if error.is_a?(MemCacheError)
     server.close if server
     new_error = MemCacheError.new error.message
     new_error.set_backtrace error.backtrace
@@ -622,36 +669,32 @@ class MemCache
     raise MemCacheError, 'No active servers' unless active?
     cache_key = make_cache_key key
     server = get_server_for_key cache_key
-    raise MemCacheError, 'No connection to server' if server.socket.nil?
     return server, cache_key
   end
 
-  def threadsafe_cache_decr(server, cache_key, amount) # :nodoc:
-    @mutex.lock
-    cache_decr server, cache_key, amount
-  ensure
-    @mutex.unlock
+  def raise_on_error_response!(response)
+    if response =~ /\A(?:CLIENT_|SERVER_)?ERROR(.*)/
+      raise MemCacheError, $1.strip
+    end
   end
 
-  def threadsafe_cache_get(server, cache_key) # :nodoc:
-    @mutex.lock
-    cache_get server, cache_key
-  ensure
-    @mutex.unlock
-  end
+  def create_continuum_for(servers)
+    total_weight = servers.inject(0) { |memo, srv| memo + srv.weight }
+    continuum = []
 
-  def threadsafe_cache_get_multi(socket, cache_keys) # :nodoc:
-    @mutex.lock
-    cache_get_multi socket, cache_keys
-  ensure
-    @mutex.unlock
+    servers.each do |server|
+      entry_count_for(server, servers.size, total_weight).times do |idx|
+        hash = Digest::SHA1.hexdigest("#{server.host}:#{server.port}:#{idx}")
+        value = Integer("0x#{hash[0..7]}")
+        continuum << Continuum::Entry.new(value, server)
+      end
+    end
+
+    continuum.sort { |a, b| a.value <=> b.value }
   end
 
-  def threadsafe_cache_incr(server, cache_key, amount) # :nodoc:
-    @mutex.lock
-    cache_incr server, cache_key, amount
-  ensure
-    @mutex.unlock
+  def entry_count_for(server, total_servers, total_weight)
+    ((total_servers * Continuum::POINTS_PER_SERVER * server.weight) / Float(total_weight)).floor
   end
 
   ##
@@ -697,6 +740,9 @@ class MemCache
 
     attr_reader :status
 
+    attr_reader :multithread
+    attr_reader :logger
+
     ##
     # Create a new MemCache::Server object for the memcached instance
     # listening on the given host and port, weighted by the given weight.
@@ -705,17 +751,18 @@ class MemCache
       raise ArgumentError, "No host specified" if host.nil? or host.empty?
       raise ArgumentError, "No port specified" if port.nil? or port.to_i.zero?
 
-      @memcache = memcache
       @host   = host
       @port   = port.to_i
       @weight = weight.to_i
 
-      @multithread = @memcache.multithread
+      @multithread = memcache.multithread
       @mutex = Mutex.new
 
       @sock   = nil
       @retry  = nil
       @status = 'NOT CONNECTED'
+      @timeout = memcache.timeout
+      @logger = memcache.logger
     end
 
     ##
@@ -750,16 +797,16 @@ class MemCache
 
       # Attempt to connect if not already connected.
       begin
-        @sock = timeout CONNECT_TIMEOUT do
-          TCPSocket.new @host, @port
-        end
+        @sock = @timeout ? TCPTimeoutSocket.new(@host, @port, @timeout) : TCPSocket.new(@host, @port)
+
         if Socket.constants.include? 'TCP_NODELAY' then
           @sock.setsockopt Socket::IPPROTO_TCP, Socket::TCP_NODELAY, 1
         end
         @retry  = nil
         @status = 'CONNECTED'
       rescue SocketError, SystemCallError, IOError, Timeout::Error => err
-        mark_dead err.message
+        logger.warn { "Unable to open socket: #{err.class.name}, #{err.message}" } if logger
+        mark_dead err
       end
 
       return @sock
@@ -781,17 +828,17 @@ class MemCache
       @mutex.unlock if @multithread
     end
 
-    private
-
     ##
     # Mark the server as dead and close its socket.
 
-    def mark_dead(reason = "Unknown error")
+    def mark_dead(error)
       @sock.close if @sock && !@sock.closed?
       @sock   = nil
       @retry  = Time.now + RETRY_DELAY
 
-      @status = sprintf "DEAD: %s, will retry at %s", reason, @retry
+      reason = "#{error.class.name}: #{error.message}"
+      @status = sprintf "%s:%s DEAD (%s), will retry at %s", @host, @port, reason, @retry
+      @logger.info { @status } if @logger
     end
 
   end
@@ -803,3 +850,47 @@ class MemCache
 
 end
 
+# TCPSocket facade class which implements timeouts.
+class TCPTimeoutSocket
+  
+  def initialize(host, port, timeout)
+    Timeout::timeout(MemCache::Server::CONNECT_TIMEOUT, SocketError) do
+      @sock = TCPSocket.new(host, port)
+      @len = timeout
+    end
+  end
+  
+  def write(*args)
+    Timeout::timeout(@len, SocketError) do
+      @sock.write(*args)
+    end
+  end
+  
+  def gets(*args)
+    Timeout::timeout(@len, SocketError) do
+      @sock.gets(*args)
+    end
+  end
+  
+  def read(*args)
+    Timeout::timeout(@len, SocketError) do
+      @sock.read(*args)
+    end
+  end
+  
+  def _socket
+    @sock
+  end
+  
+  def method_missing(meth, *args)
+    @sock.__send__(meth, *args)
+  end
+  
+  def closed?
+    @sock.closed?
+  end
+  
+  def close
+    @sock.close
+  end
+end