memcache-client 1.7.4 → 1.7.5

data/FAQ.rdoc

@@ -23,9 +23,9 @@ You can increase the timeout or disable them completely with the following confi
   MemCache.new ['server1', 'server2'], { :timeout => 1.0 } # 1 second timeout
 
 
-== The new memcache-client is slower than 1.5.0!?
+== Isn't Evan Weaver's memcached gem faster?
 
-Yes, in the simplest case, 1.6.x+ is slower than 1.5.0.  If you just have a single memcached
-server, the latest memcache-client will be slower.  This is because 1.5.0 does not
-timeout network operations.  For reliability purposes, memcache-client 1.6.x+ enables 
-timeouts by default.  Most of this performance penalty is now gone in 1.7.3 and later.
+The latest version of memcached-client is anywhere from 33% to 100% slower than memcached in various benchmarks.  Keep in mind this means that 10,000 get requests take 1.8 sec instead of 1.2 seconds.
+In practice, memcache-client is unlikely to be a bottleneck in your system but there is always going
+to be an overhead to pure Ruby.  Evan's memcached gem is a thin wrapper around a C library, which
+makes it very fast but also difficult to install for some people.

data/History.rdoc

@@ -1,3 +1,12 @@
+= 1.7.5 (2009-09-09)
+
+* Fix ruby warnings (josh)
+* Make megabyte value size limit optional since Tokyo Tyrant can accept values larger than 1MB.
+  Use :check_size => false to disable the size check. (jsl)
+* Ruby 1.9 support for recent I/O changes.
+* Fix duplicate value marshalling on server error. (rajiv)
+* Added option :autofix_keys (disabled by default) to replace long keys with md5 hashes (sd)
+
 = 1.7.4 (2009-06-09)
 
 * Fix issue with raising timeout errors.

data/README.rdoc

@@ -32,8 +32,9 @@ multiple memcached servers, you should install the RubyInline gem for ultimate p
 
 == Using memcache-client with Rails
 
-Rails 2.1+ includes memcache-client out of the box.  See ActiveSupport::Cache::MemCacheStore
-and the Rails.cache method for more details.
+Rails 2.1+ includes memcache-client 1.5.0 out of the box.  See ActiveSupport::Cache::MemCacheStore
+and the Rails.cache method for more details.  Rails 2.3+ will use the latest memcache-client
+gem installed.
 
 
 == Questions?

data/Rakefile

@@ -26,7 +26,9 @@ Rake::RDocTask.new do |rd|
 	rd.rdoc_dir = 'doc'
 end
 
-Rake::TestTask.new
+Rake::TestTask.new do |t|
+  t.warning = true
+end
 
 task :default => :test
 

data/lib/memcache.rb

@@ -15,19 +15,21 @@ class MemCache
   ##
   # The version of MemCache you are using.
 
-  VERSION = '1.7.4'
+  VERSION = '1.7.5'
 
   ##
   # Default options for the cache object.
 
   DEFAULT_OPTIONS = {
-    :namespace   => nil,
-    :readonly    => false,
-    :multithread => true,
-    :failover    => true,
-    :timeout     => 0.5,
-    :logger      => nil,
-    :no_reply    => false,
+    :namespace    => nil,
+    :readonly     => false,
+    :multithread  => true,
+    :failover     => true,
+    :timeout      => 0.5,
+    :logger       => nil,
+    :no_reply     => false,
+    :check_size   => true,
+    :autofix_keys => false
   }
 
   ##
@@ -50,6 +52,19 @@ class MemCache
 
   attr_reader :multithread
 
+  ##
+  # Whether to try to fix keys that are too long and will be truncated by
+  # using their SHA1 hash instead.
+  # The hash is only used on keys longer than 250 characters, or containing spaces,
+  # to avoid impacting performance unnecesarily.
+  #
+  # In theory, your code should generate correct keys when calling memcache, 
+  # so it's your responsibility and you should try to fix this problem at its source.
+  #
+  # But if that's not possible, enable this option and memcache-client will give you a hand.
+  
+  attr_reader :autofix_keys
+  
   ##
   # The servers this client talks to.  Play at your own peril.
 
@@ -84,19 +99,21 @@ class MemCache
   #
   # Valid options for +opts+ are:
   #
-  #   [:namespace]   Prepends this value to all keys added or retrieved.
-  #   [:readonly]    Raises an exception on cache writes when true.
-  #   [:multithread] Wraps cache access in a Mutex for thread safety. Defaults to true.
-  #   [:failover]    Should the client try to failover to another server if the
-  #                  first server is down?  Defaults to true.
-  #   [:timeout]     Time to use as the socket read timeout.  Defaults to 0.5 sec,
-  #                  set to nil to disable timeouts (this is a major performance penalty in Ruby 1.8,
-  #                  "gem install SystemTimer' to remove most of the penalty).
-  #   [:logger]      Logger to use for info/debug output, defaults to nil
-  #   [:no_reply]    Don't bother looking for a reply for write operations (i.e. they
-  #                  become 'fire and forget'), memcached 1.2.5 and later only, speeds up
-  #                  set/add/delete/incr/decr significantly.
-  #
+  #   [:namespace]    Prepends this value to all keys added or retrieved.
+  #   [:readonly]     Raises an exception on cache writes when true.
+  #   [:multithread]  Wraps cache access in a Mutex for thread safety. Defaults to true.
+  #   [:failover]     Should the client try to failover to another server if the
+  #                   first server is down?  Defaults to true.
+  #   [:timeout]      Time to use as the socket read timeout.  Defaults to 0.5 sec,
+  #                   set to nil to disable timeouts.
+  #   [:logger]       Logger to use for info/debug output, defaults to nil
+  #   [:no_reply]     Don't bother looking for a reply for write operations (i.e. they
+  #                   become 'fire and forget'), memcached 1.2.5 and later only, speeds up
+  #                   set/add/delete/incr/decr significantly.
+  #   [:check_size]   Raises a MemCacheError if the value to be set is greater than 1 MB, which
+  #                   is the maximum key size for the standard memcached server.  Defaults to true.
+  #   [:autofix_keys] If a key is longer than 250 characters or contains spaces, 
+  #                   use an SHA1 hash instead, to prevent collisions on truncated keys.
   # Other options are ignored.
 
   def initialize(*args)
@@ -120,14 +137,16 @@ class MemCache
     end
 
     opts = DEFAULT_OPTIONS.merge opts
-    @namespace   = opts[:namespace]
-    @readonly    = opts[:readonly]
-    @multithread = opts[:multithread]
-    @timeout     = opts[:timeout]
-    @failover    = opts[:failover]
-    @logger      = opts[:logger]
-    @no_reply    = opts[:no_reply]
-    @mutex       = Mutex.new if @multithread
+    @namespace    = opts[:namespace]
+    @readonly     = opts[:readonly]
+    @multithread  = opts[:multithread]
+    @autofix_keys = opts[:autofix_keys]
+    @timeout      = opts[:timeout]
+    @failover     = opts[:failover]
+    @logger       = opts[:logger]
+    @no_reply     = opts[:no_reply]
+    @check_size   = opts[:check_size]
+    @mutex        = Mutex.new if @multithread
 
     logger.info { "memcache-client #{VERSION} #{Array(servers).inspect}" } if logger
 
@@ -312,12 +331,14 @@ class MemCache
 
   def set(key, value, expiry = 0, raw = false)
     raise MemCacheError, "Update of readonly cache" if @readonly
-    with_server(key) do |server, cache_key|
 
-      value = Marshal.dump value unless raw
+    value = Marshal.dump value unless raw
+    with_server(key) do |server, cache_key|
       logger.debug { "set #{key} to #{server.inspect}: #{value.to_s.size}" } if logger
 
-      raise MemCacheError, "Value too large, memcached can only store 1MB of data per key" if value.to_s.size > ONE_MB
+      if @check_size && value.to_s.size > ONE_MB
+        raise MemCacheError, "Value too large, memcached can only store 1MB of data per key"
+      end
 
       command = "set #{cache_key} 0 #{expiry} #{value.to_s.size}#{noreply}\r\n#{value}\r\n"
 
@@ -359,10 +380,9 @@ class MemCache
     (value, token) = gets(key, raw)
     return nil unless value
     updated = yield value
+    value = Marshal.dump updated unless raw
 
     with_server(key) do |server, cache_key|
-
-      value = Marshal.dump updated unless raw
       logger.debug { "cas #{key} to #{server.inspect}: #{value.to_s.size}" } if logger
       command = "cas #{cache_key} 0 #{expiry} #{value.to_s.size} #{token}#{noreply}\r\n#{value}\r\n"
 
@@ -392,8 +412,8 @@ class MemCache
 
   def add(key, value, expiry = 0, raw = false)
     raise MemCacheError, "Update of readonly cache" if @readonly
+    value = Marshal.dump value unless raw
     with_server(key) do |server, cache_key|
-      value = Marshal.dump value unless raw
       logger.debug { "add #{key} to #{server}: #{value ? value.to_s.size : 'nil'}" } if logger
       command = "add #{cache_key} 0 #{expiry} #{value.to_s.size}#{noreply}\r\n#{value}\r\n"
 
@@ -413,8 +433,8 @@ class MemCache
   # If +raw+ is true, +value+ will not be Marshalled.
   def replace(key, value, expiry = 0, raw = false)
     raise MemCacheError, "Update of readonly cache" if @readonly
+    value = Marshal.dump value unless raw
     with_server(key) do |server, cache_key|
-      value = Marshal.dump value unless raw
       logger.debug { "replace #{key} to #{server}: #{value ? value.to_s.size : 'nil'}" } if logger
       command = "replace #{cache_key} 0 #{expiry} #{value.to_s.size}#{noreply}\r\n#{value}\r\n"
 
@@ -621,6 +641,10 @@ class MemCache
   # requested.
 
   def make_cache_key(key)
+    if @autofix_keys and (key =~ /\s/ or (key.length + (namespace.nil? ? 0 : namespace.length)) > 250)
+      key = "#{Digest::SHA1.hexdigest(key)}-autofixed"
+    end
+
     if namespace.nil? then
       key
     else
@@ -1040,23 +1064,23 @@ class MemCache
   class BufferedIO < Net::BufferedIO # :nodoc:
     BUFSIZE = 1024 * 16
 
-    # An implementation similar to this is in *trunk* for 1.9.  When it
-    # gets released, this method can be removed when using 1.9
-    def rbuf_fill
-      begin
-        @rbuf << @io.read_nonblock(BUFSIZE)
-      rescue Errno::EWOULDBLOCK
-        retry unless @read_timeout
-        if IO.select([@io], nil, nil, @read_timeout)
-          retry
-        else
-          raise Timeout::Error, 'IO timeout'
+    if RUBY_VERSION < '1.9.1'
+      def rbuf_fill
+        begin
+          @rbuf << @io.read_nonblock(BUFSIZE)
+        rescue Errno::EWOULDBLOCK
+          retry unless @read_timeout
+          if IO.select([@io], nil, nil, @read_timeout)
+            retry
+          else
+            raise Timeout::Error, 'IO timeout'
+          end
         end
       end
     end
 
-    def setsockopt *args
-      @io.setsockopt *args
+    def setsockopt(*args)
+      @io.setsockopt(*args)
     end
 
     def gets

data/test/test_mem_cache.rb

@@ -18,7 +18,8 @@ require File.dirname(__FILE__) + '/../lib/memcache' if not defined?(MemCache)
 class MemCache
 
   attr_writer :namespace
-
+  attr_writer :autofix_keys
+  
 end
 
 class FakeSocket
@@ -269,7 +270,7 @@ class TestMemCache < Test::Unit::TestCase
       @cache.cache_get(server, 'my_namespace:key')
     end
 
-    assert_match /#{Regexp.quote 'No connection to server (NOT CONNECTED)'}/, e.message
+    assert_match(/#{Regexp.quote 'No connection to server (NOT CONNECTED)'}/, e.message)
 
     assert !server.alive?
   end
@@ -327,7 +328,7 @@ class TestMemCache < Test::Unit::TestCase
       @cache.cache_get_multi server, 'my_namespace:key'
     end
 
-    assert_match /#{Regexp.quote 'No connection to server (NOT CONNECTED)'}/, e.message
+    assert_match(/#{Regexp.quote 'No connection to server (NOT CONNECTED)'}/, e.message)
 
     assert !server.alive?
   end
@@ -563,7 +564,7 @@ class TestMemCache < Test::Unit::TestCase
       @cache.get 'key'
     end
 
-    assert_match /^No connection to server/, e.message
+    assert_match(/^No connection to server/, e.message)
   end
 
   def test_get_no_servers
@@ -717,6 +718,36 @@ class TestMemCache < Test::Unit::TestCase
     assert_equal 'key', @cache.make_cache_key('key')
   end
 
+  def test_make_cache_key_without_autofix
+    @cache.autofix_keys = false
+
+    key = "keys with more than two hundred and fifty characters can cause problems, because they get truncated and start colliding with each other. It's not a common occurrence, but when it happens is very hard to debug. the autofix option takes care of that for you"
+    hash = Digest::SHA1.hexdigest(key)
+    @cache.namespace = nil
+    assert_equal key, @cache.make_cache_key(key)
+
+    @cache.autofix_keys = true
+
+    @cache.namespace = "my_namespace"
+    assert_equal 'my_namespace:key', @cache.make_cache_key('key')
+    @cache.namespace = nil
+    assert_equal 'key', @cache.make_cache_key('key')
+    
+    key = "keys with more than two hundred and fifty characters can cause problems, because they get truncated and start colliding with each other. It's not a common occurrence, but when it happens is very hard to debug. the autofix option takes care of that for you"
+    hash = Digest::SHA1.hexdigest(key)
+    @cache.namespace = "my_namespace"
+    assert_equal "my_namespace:#{hash}-autofixed", @cache.make_cache_key(key)
+    @cache.namespace = nil
+    assert_equal "#{hash}-autofixed", @cache.make_cache_key(key)
+    
+    key = "a short key with spaces"
+    hash = Digest::SHA1.hexdigest(key)
+    @cache.namespace = "my_namespace"
+    assert_equal "my_namespace:#{hash}-autofixed", @cache.make_cache_key(key)
+    @cache.namespace = nil
+    assert_equal "#{hash}-autofixed", @cache.make_cache_key(key)
+  end
+  
   def test_servers
     server = FakeServer.new
     @cache.servers = []
@@ -776,6 +807,38 @@ class TestMemCache < Test::Unit::TestCase
     assert_equal 'Update of readonly cache', e.message
   end
 
+  def test_check_size_on
+    cache = MemCache.new :check_size => true
+
+    server = FakeServer.new
+    server.socket.data.write "STORED\r\n"
+    server.socket.data.rewind
+
+    cache.servers = []
+    cache.servers << server
+
+    e = assert_raise MemCache::MemCacheError do
+      cache.set 'key', 'v' * 1048577
+    end
+
+    assert_equal 'Value too large, memcached can only store 1MB of data per key', e.message
+  end
+
+  def test_check_size_off
+    cache = MemCache.new :check_size => false
+
+    server = FakeServer.new
+    server.socket.data.write "STORED\r\n"
+    server.socket.data.rewind
+
+    cache.servers = []
+    cache.servers << server
+
+    assert_nothing_raised do
+      cache.set 'key', 'v' * 1048577
+    end
+  end
+
   def test_set_too_big
     server = FakeServer.new
 
@@ -790,7 +853,7 @@ class TestMemCache < Test::Unit::TestCase
       @cache.set 'key', 'v'
     end
 
-    assert_match /object too large for cache/, e.message
+    assert_match(/object too large for cache/, e.message)
   end
 
   def test_prepend
@@ -977,7 +1040,7 @@ class TestMemCache < Test::Unit::TestCase
       @cache.flush_all
     end
 
-    assert_match /flush_all\r\n/, socket.written.string
+    assert_match(/flush_all\r\n/, socket.written.string)
   end
   
   def test_flush_all_for_real
@@ -1033,8 +1096,8 @@ class TestMemCache < Test::Unit::TestCase
     end
 
     output = server.socket.written.string
-    assert_match /set my_namespace:test/, output
-    assert_match /test value/, output
+    assert_match(/set my_namespace:test/, output)
+    assert_match(/test value/, output)
   end
 
   def test_basic_unthreaded_operations_should_work
@@ -1056,8 +1119,8 @@ class TestMemCache < Test::Unit::TestCase
     end
 
     output = server.socket.written.string
-    assert_match /set my_namespace:test/, output
-    assert_match /test value/, output
+    assert_match(/set my_namespace:test/, output)
+    assert_match(/test value/, output)
   end
 
   def util_setup_fake_server
@@ -1122,9 +1185,9 @@ class TestMemCache < Test::Unit::TestCase
             assert cache.decr('c', 5) > 14
             assert_equal 11, cache.get('b')
             d = cache.get('d', true)
-            assert_match /\Aab*\Z/, d
+            assert_match(/\Aab*\Z/, d)
             e = cache.get('e', true)
-            assert_match /\Ay*x\Z/, e
+            assert_match(/\Ay*x\Z/, e)
           end
         end
       end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: memcache-client
 version: !ruby/object:Gem::Version 
-  version: 1.7.4
+  version: 1.7.5
 platform: ruby
 authors: 
 - Eric Hodel
@@ -11,7 +11,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2009-06-09 00:00:00 -05:00
+date: 2009-09-09 00:00:00 -05:00
 default_executable: 
 dependencies: []
 
@@ -31,7 +31,7 @@ files:
 - Rakefile
 - lib/memcache.rb
 - lib/continuum_native.rb
-has_rdoc: false
+has_rdoc: true
 homepage: http://github.com/mperham/memcache-client
 licenses: []
 
@@ -55,7 +55,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
 requirements: []
 
 rubyforge_project: seattlerb
-rubygems_version: 1.3.2
+rubygems_version: 1.3.5
 signing_key: 
 specification_version: 3
 summary: A Ruby library for accessing memcached.