memcache 1.2.0 → 1.2.1

data/.gitignore

@@ -1 +1,12 @@
-pkg
+pkg
+*.bundle
+*.so
+*.o
+Makefile
+mkmf.log
+*.gem
+*.swp
+ext/bin
+ext/lib
+ext/share
+ext/include

data/README.rdoc

@@ -2,8 +2,8 @@
 
 This is the Geni memcached client. It started out as a fork of fiveruns/memcache-client,
 which was a fork of seattle.rb's memcache-client, but over time, our client has diverged,
-and I've rewritten most of the code. Of course, a lot of credit is due to those whose code
-served as a starting point for this code.
+and I've rewritten the majority of the code. Of course, a lot of credit is due to those
+whose code served as a starting point for this code.
 
 == Usage
 
@@ -16,11 +16,11 @@ served as a starting point for this code.
  cache['things']
  => {:foo => "1", :bar => [1,2,3]}
 
-== How is this different from memcache-client?
+== How is this different from other memcache clients?
 
-Like memcache-client, _memcache_ (shown in italics when I am referring to this
-library) is a memcached client, but it differs significantly from memcache-client in
-several important ways.
+Like fiveruns/memcache-client and fauna/memcached, _memcache_ (shown in italics when I am
+referring to this library) is a memcached client, but it differs significantly from
+these clients in several important ways.
 
 === Interface
 
@@ -80,8 +80,8 @@ changes:
    cache.add('foo', 0)
    cache.get('foo')
    => 1
-  
-   cache.replace('foo', 2) 
+
+   cache.replace('foo', 2)
    cache.get('foo')
    => 2
 
@@ -111,13 +111,13 @@ changes:
 
 The underlying architechture of _memcache_ is more modular than memcache-client.
 A given +Memcache+ instance has a group of servers, just like before, but much more of the
-functionality in encapsulated inside the <tt>Memcache::Server</tt> object. Really, a +Server+
+functionality is encapsulated inside the <tt>Memcache::Server</tt> object. Really, a +Server+
 object is a thin wrapper around an remote memcached server that takes care of the socket
 and protocol details along with basic error handling. The +Memcache+ class handles the
 partitioning algorithm, marshaling of ruby objects and various higher-level methods.
 
 By encapsulating the protocol inside the +Server+ object, it becomes very easy to plug-in
-alternate server implementations. Right now, there are two basic, alternate servers:
+alternate backend server implementations. Right now, there are three basic, alternate servers:
 
 [+LocalServer+] This is an in-process server for storing keys and values in local
                 memory. It is good for testing, when you don't want to spin up an instance
@@ -131,6 +131,9 @@ alternate server implementations. Right now, there are two basic, alternate serv
              are very large. It can also be used in a multi-level cache setup with
              <tt>Memcache::Server</tt> to provide persistence without sacrificing speed.
 
+[+NativeServer+] This implementation uses native bindings to libmemcached. It is described
+                 in more detail in the "Native Bindings" section below.
+
 === Very Large Values
 
 Memcached limits the size of values to 1MB. This is done to reduce memory usage, but it
@@ -170,10 +173,61 @@ finally marking a server as dead and raising an exception. We will not attempt t
 from a dead server for 5 seconds, but a write will always attempt to revive a dead server
 by attempting to connect.
 
+=== Keys, Namespaces, and Prefixes
+
+Unlike the other ruby memcache clients, keys in _memcache_ can contain spaces. This is
+possible because the backend transparently enscapes all space characters, and is
+especially important if you are using method_cache[http://github.com/ninjudd/method_cache]
+or record_cache[http://github.com/ninjudd/record_cache]. <tt>Memcache::Server</tt> implements
+this escaping using gsub and it adds a slight performance penalty when escaping is
+necessary. +NativeServer+ implements this escaping directly in C, and the performance
+overhead is negligible.
+
+You can also partition your keys into different namespaces for convenience. This is done
+by prefixing all keys in the backend server with "namespace:". However, the hash keys
+returned by multi gets do not contain the prefix. In this way, the namespace can be
+totally transparent to your code. You can also determine whether the prefix is used for
+hashing with the following option:
+
+[+hash_with_prefix+] Determines whether the prefix/namespace is used when hashing keys to
+                     determine which server to use. Defaults to true.
+
+== Native Bindings
+
+The <tt>Memcache::NativeServer</tt> backend provides native bindings to libmecached. This is
+significantly faster than using <tt>Memcache::Server</tt> as demonstrated by runnning
+bench/benchmark.rb. NativeServer encapsulates a set of remote servers and allows you to
+use the various hashing methods in libmemcached.
+
+You can use native bindings either by passing +NativeServer+ objects to +Memcache+, or you
+can use the +native+ option. Native bindings are compatible with segmented values through
+the +SegmentedNativeServer+ object or by combining the +native+ option with
++segment_large_values+.
+
+ server = Memcache::NativeServer.new(:servers => ['localhost:11211', 'localhost:11212'])
+ cache = Memcache.new(:server => server)
+
+ cache = Memcache.new(:servers => ['localhost:11211', 'localhost:11212'], :native => true)
+
+NativeServer also accepts a few other options:
+
+[+hash+] The libmemcached hashing method. See http://docs.tangent.org/libmemcached/index.html
+         for more detail. One of:
+
+         <tt>:default :md5 :crc :fnv1_64 :fnv1a_64 :fnv1_32 :fnv1a_32 :jenkins
+         :hsieh :murmur</tt>.
+
+         NOTE: Even though there is a libmemcached method named <tt>:default</tt> (which
+         is actually Jenkins's one-at-a-time hash), the default hashing method if you
+         don't specify one is <tt>:crc</tt>.
+
+[+binary+] A boolean value specifying whether to use memcached's binary protocol instead
+           of the default ascii protocol. This is slightly slower, but should allow you to use unicode keys.
+
 == Installation
 
   $ sudo gem install memcache --source http://gemcutter.org
 
 == License:
 
-Copyright (c) 2009 Justin Balthrop, Geni.com; Published under The MIT License, see LICENSE
+Copyright (c) 2010 Justin Balthrop, Geni.com; Published under The MIT License, see LICENSE

data/Rakefile

@@ -41,6 +41,10 @@ task :test => :check_dependencies
 
 task :default => :test
 
+task :clean do
+  `rm -rf ext/lib ext/bin ext/share ext/include`
+end
+
 require 'rake/rdoctask'
 Rake::RDocTask.new do |rdoc|
   if File.exist?('VERSION')
@@ -54,3 +58,13 @@ Rake::RDocTask.new do |rdoc|
   rdoc.rdoc_files.include('README*')
   rdoc.rdoc_files.include('lib/**/*.rb')
 end
+
+namespace :test do
+  Rake::TestTask.new(:native) do |t|
+    `cd ext && make && cp native_server.bundle native_server.o ../lib/memcache/`
+    t.libs << 'test'
+    t.pattern = 'test/memcache_native_server_test.rb'
+    t.verbose
+  end
+end
+  

data/VERSION.yml

@@ -1,5 +1,5 @@
 --- 
 :build: 
-:patch: 0
-:major: 1
 :minor: 2
+:patch: 1
+:major: 1

data/bench/benchmark.rb

@@ -0,0 +1,126 @@
+require 'rubygems'
+require 'benchmark'
+require 'pp'
+
+$:.unshift(File.dirname(__FILE__) + '/../lib')
+require  'memcache'
+
+def array(*args)
+  arr = []
+  args.each do |arg|
+    if arg.kind_of?(Enumerable)
+      arr.concat(arg.to_a)
+    else
+      arr << arg
+    end
+  end
+  arr
+end
+
+CHARS = array('a'..'z', 'A'..'Z', '0'..'9', '_', '+', '-')
+def rand_string(len)
+  str = ''
+  len.times do
+    str << pick_rand(CHARS)
+  end
+  str
+end
+
+def pick_rand(items)
+  i = rand(items.size)
+  items[i]
+end
+
+def pick_mod(items, i)
+  i = i % items.size
+  items[i]
+end
+
+class MemcacheBench
+  attr_reader :num_items, :key_length, :val_length, :keys, :vals, :n
+
+  def initialize(opts = {})
+    @n          = opts[:n] || 100_000
+    @num_items  = opts[:num_items] || 5000
+    @key_length = array(opts[:key_length] || 10)
+    @val_length = array(opts[:val_length] || 100)
+
+    puts "N = #{@n}"
+    puts "key_length: #{@key_length.join(' or ')}"
+    puts "val_length: #{@val_length.join(' or ')}"
+    puts "Generating #{@num_items} random keys and values..."
+    @keys = []
+    @vals = []
+    @num_items.times do
+      @keys << rand_string( pick_rand(@key_length) )
+      @vals << rand_string( pick_rand(@val_length) )
+    end
+
+    Benchmark.bm(36) do |x|
+      @bench = x
+    end
+  end
+
+  def bench(name, nkeys = 1, &block)
+    if nkeys > 1
+      keyseq = keys + keys
+      block  = lambda do |i|
+        i = i % keys.size
+        yield(keyseq[i, nkeys])
+      end
+      name = "#{name}-#{nkeys}"
+    elsif block.arity == 1
+      block = lambda {|i| yield(pick_mod(keys, i))}
+    else
+      block = lambda {|i| yield(pick_mod(keys, i), pick_mod(vals, i))}
+    end
+
+    @bench.report(name) do
+      (n/nkeys).times do |i|
+        block.call(i)
+      end
+    end
+  end
+end
+
+def init_servers(*ports)
+  servers = []
+  ports.each do |port|
+    system("memcached -p #{port} -U 0 -d -P /tmp/memcached_#{port}.pid")
+    servers << "127.0.0.1:#{port}"
+    sleep 0.3
+  end
+  memcache = yield(servers)
+  memcache.flush_all
+  memcache
+end
+
+def ___
+  puts('=' * 81)
+end
+
+puts `uname -a`
+puts "Ruby #{RUBY_VERSION}p#{RUBY_PATCHLEVEL}"
+
+ns = 'namespace'
+memcache      = init_servers(10000,10001) {|s| Memcache.new(:servers => s, :namespace => ns)}
+native        = init_servers(10002,10003) {|s| Memcache.new(:servers => s, :namespace => ns, :native => true)}
+native_nowrap = init_servers(10004,10005) {|s| Memcache::NativeServer.new(:servers => s, :prefix => "#{ns}:")}
+
+b = MemcacheBench.new(:num_items => 5000, :n => 100_000, :key_length => 20, :val_length => 100)
+
+2.times do
+  ___
+  b.bench( 'set:native-nowrap'      ) {|key, val| native_nowrap.set(key, val) }
+  b.bench( 'get:native-nowrap'      ) {|key     | native_nowrap.get(key)      }
+  b.bench( 'get:native-nowrap', 100 ) {|keys    | native_nowrap.get(keys)     }
+  ___
+  b.bench( 'set:native'      ) {|key, val| native.set(key, val, :raw => true) }
+  b.bench( 'get:native'      ) {|key     | native.get(key,      :raw => true) }
+  b.bench( 'get:native', 100 ) {|keys    | native.get(keys,     :raw => true) }
+  ___
+  b.bench( 'set:ruby'      ) {|key, val| memcache.set(key, val, :raw => true) }
+  b.bench( 'get:ruby'      ) {|key     | memcache.get(key,      :raw => true) }
+  b.bench( 'get:ruby', 100 ) {|keys    | memcache.get(keys,     :raw => true) }
+  ___
+end

data/ext/extconf.rb

@@ -1,3 +1,52 @@
 require 'mkmf'
-have_library('memcached')
-create_makefile('native_server')
+require 'rbconfig'
+
+# Code taken from Evan's Weaver memcached library: http://github.com/fauna/memcached
+
+HERE        = File.expand_path(File.dirname(__FILE__))
+BUNDLE      = Dir.glob("libmemcached-*.tar.gz").first
+BUNDLE_PATH = BUNDLE.sub(".tar.gz", "")
+
+$CXXFLAGS = " -std=gnu++98"
+
+if !ENV["EXTERNAL_LIB"]
+  $includes    = " -I#{HERE}/include"
+  $libraries   = " -L#{HERE}/lib"
+  $CFLAGS      = "#{$includes} #{$libraries} #{$CFLAGS}"
+  $LDFLAGS     = "#{$libraries} #{$LDFLAGS}"
+  $LIBPATH     = ["#{HERE}/lib"]
+  $DEFLIBPATH  = []
+
+  Dir.chdir(HERE) do
+    if false and File.exist?("lib")
+      puts "Libmemcached already built; run 'rake clean' first if you need to rebuild."
+    else
+      puts "Building libmemcached."
+      puts(cmd = "tar xzf #{BUNDLE} 2>&1")
+      raise "'#{cmd}' failed" unless system(cmd)
+      
+      Dir.chdir(BUNDLE_PATH) do        
+        puts(cmd = "env CFLAGS='-fPIC' ./configure --prefix=#{HERE} --without-memcached --disable-shared --disable-utils --disable-dependency-tracking #{$EXTRA_CONF} 2>&1")
+        raise "'#{cmd}' failed" unless system(cmd)
+
+        puts(cmd = "make CXXFLAGS='#{$CXXFLAGS}' || true 2>&1")
+        raise "'#{cmd}' failed" unless system(cmd)
+
+        puts(cmd = "make install || true 2>&1")
+        raise "'#{cmd}' failed" unless system(cmd)
+      end
+
+      system("rm -rf #{BUNDLE_PATH}") unless ENV['DEBUG'] or ENV['DEV']
+    end
+  end
+  
+  # Absolutely prevent the linker from picking up any other libmemcached
+  Dir.chdir("#{HERE}/lib") do
+    system('cp -f libmemcached.a  libmemcached_gem.a') 
+    system('cp -f libmemcached.la libmemcached_gem.la') 
+  end
+
+  $LIBS << " -lmemcached_gem"
+end
+
+create_makefile('memcache/native_server')