memcache 1.0.0

data/.gitignore

@@ -0,0 +1 @@
+pkg

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 Justin Balthrop
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,179 @@
+= memcache
+
+This is the Geni memcached client. It started out as a fork of fiveruns/memcache-client,
+which was a fork of seattle.rb's memcache-client, but over time, our client has diverged,
+and I've rewritten most of the code. Of course, a lot of credit is due to those whose code
+served as a starting point for this code.
+
+== Usage
+
+ cache = Memcache.new(:server => "localhost:11211")
+ cache.set('stuff', [:symbol, 'String', 1, {:bar => 5}])
+ cache.get('stuff')
+ => [:symbol, "String", 1, {:bar => 5}]
+
+ cache['things'] = {:foo => '1', :bar => [1,2,3]}
+ cache['things']
+ => {:foo => "1", :bar => [1,2,3]}
+
+== How is this different from memcache-client?
+
+Like memcache-client, _memcache_ (shown in italics when I am referring to this
+library) is a memcached client, but it differs significantly from memcache-client in
+several important ways.
+
+=== Interface
+
+I tried to keep the basic interface as similar as I could to memcache-client. In some
+cases, _memcache_ can be a near drop-in replacement for memcache-client. However, I did
+rename the main class from +MemCache+ to +Memcache+ to prevent confusion and to force
+those switching to _memcache_ to update their code. Here are the notable interface
+changes:
+
+- +expiry+ and +raw+ are specified as options in a hash now, instead of as unnamed parameters.
+
+   cache.set('foo', :a,  :expiry => 10.minutes)
+   cache.set('bar', :b,  :expiry => Time.parse('5:51pm Nov 24, 2018'))
+   cache.set('baz', 'c', :expiry => 30.minutes, :raw => true)
+
+- +get_multi+ has been replaced by a more versatile +get+ interface. If the first argument is
+  an array, then a hash of key/value pairs is returned. If the first argument is not an
+  array, then the value alone is returned.
+
+   cache.get('foo')          # => :a
+   cache.get(['foo', 'bar']) # => {"foo"=>:a, "bar"=>:b}
+   cache.get(['foo'])        # => {"foo"=>:a}
+
+- +get+ also supports updating the expiry for a single key. this can be used to keep
+  frequently accessed data in cache longer than less accessed data, though usually the
+  memcached LRU algorithm will be sufficient.
+
+   cache.get('foo', :expiry => 1.day)
+
+- Support for flags has been added to all methods. So you can store additional metadata on
+  each value. Depending on which server version you are using, flags can be 16 bit or 32
+  bit unsigned integers (though it seems that memcache 1.4.1 returns signed values if the
+  upper bit is set).
+
+   cache.set('foo', :aquatic, :flags => 0b11101111)
+   value = cache.get('foo')
+   => :aquatic
+   value.memcache_flags.to_s(2)
+   => "11101111"
+
+   cache.set('foo', 'aquatic', :raw => true, :flags => 0xff08)
+   cache.get('foo', :raw => true).memcache_flags.to_s(2)
+   => "1111111100001000"
+
+- +incr+ and +decr+ automatically initialize the value to 0 if the key doesn't
+  exist. The +count+ method returns the integer count associated with a given key.
+
+   cache.count('hits')    # => 0
+   cache.incr('hits', 52) # => 52
+   cache.decr('hits', 9)  # => 43
+   cache.count('hits')    # => 43
+
+- In addition to +add+, which was already supported, support has been added for +replace+,
+  +append+ and +prepend+ from the memcached protocol.
+
+   cache.add('foo', 1)
+   cache.add('foo', 0)
+   cache.get('foo')
+   => 1
+  
+   cache.replace('foo', 2) 
+   cache.get('foo')
+   => 2
+
+   cache.write('foo', 'bar')     ## shortcut for cache.set('foo', 'bar', :raw => true)
+   cache.append('foo', 'none')   ## append and prepend only works on raw values
+   cache.prepend('foo', 'foo')   ##
+   cache.read('foo')             ## shortcut for cache.get('foo', :raw => true)
+   => "foobarnone"
+
+- Support has also been added for +cas+ (compare-and-set).
+
+   value = cache.get('foo', :cas => true)
+   cache.cas('foo', value.upcase, :cas => value.memcache_cas)
+   cache.get('foo')
+   => "FOOBARNONE"
+
+   value = cache.get('foo', :cas => true)
+   cache.set('foo', 'modified')
+   cache.cas('foo', value.downcase, :cas => value.memcache_cas)
+   cache.get('foo')
+   => "modified"
+
+- Several additional convenience methods have been added including +get_or_add+,
+  +get_or_set+, +update+, +get_some+, +lock+, +unlock+, and +with_lock+.
+
+=== Implementation
+
+The underlying architechture of _memcache_ is more modular than memcache-client.
+A given +Memcache+ instance has a group of servers, just like before, but much more of the
+functionality in encapsulated inside the <tt>Memcache::Server</tt> object. Really, a +Server+
+object is a thin wrapper around an remote memcached server that takes care of the socket
+and protocol details along with basic error handling. The +Memcache+ class handles the
+partitioning algorithm, marshaling of ruby objects and various higher-level methods.
+
+By encapsulating the protocol inside the +Server+ object, it becomes very easy to plug-in
+alternate server implementations. Right now, there are two basic, alternate servers:
+
+[+LocalServer+] This is an in-process server for storing keys and values in local
+                memory. It is good for testing, when you don't want to spin up an instance
+                of memcached, and also as a second level of caching. For example, in a web
+                application, you can use this as a quick cache which lasts for the
+                duration of a request.
+
+[+PGServer+] This is an implementation of memcached functionality using SQL. It stores all
+             data in a single postgres table and uses +PGConn+ to select and update this
+             table. This works well as a permanent cache or in the case when your objects
+             are very large. It can also be used in a multi-level cache setup with
+             <tt>Memcache::Server</tt> to provide persistence without sacrificing speed.
+
+=== Very Large Values
+
+Memcached limits the size of values to 1MB. This is done to reduce memory usage, but it
+means that large data structures, which are also often costly to compute, cannot be stored
+easily. We solve this problem by providing an additional server called
+<tt>Memcache::SegmentedServer</tt>. It inherits from <tt>Memcache::Server</tt>, but
+includes code to segment and reassemble large values. Mike Stangel at Geni originally
+wrote this code as an extension to memcache-client and I adapted it for the new
+architecture.
+
+You can use segmented values either by passing +SegmentedServer+ objects to +Memcache+, or
+you can use the +segment_large_values+ option.
+
+ server = Memcache::SegmentedServer.new(:host => 'localhost', :port => 11211)
+ cache = Memcache.new(:server => server)
+
+ cache = Memcache.new(:server => 'localhost:11211', :segment_large_values => true)
+
+=== Error Handling and Recovery
+
+We handle errors differently in _memcache_ than memcache-client does. Whenever there is a
+connection error or other fatal error, memcache-client marks the offending server as dead
+for 30 seconds, and all calls that require that server fail for the next 30 seconds. This
+was unacceptable for us in a production environment. We tried changing the retry timeout
+to 1 second, but still found our exception logs filling up with failed web requests
+whenever a network connection was broken.
+
+So, the default behavior in _memcache_ is for reads to be stable even if the underlying
+server is unavailable. This means, that instead of raising an exception, a read will just
+return nil if the server is down. Of course, you need to monitor your memcached servers to
+make sure they aren't down for long, but this allows your site to be resilient to minor
+network blips. Any error that occurs while unmarshalling a stored object will also return nil.
+
+Writes, on the other hand, cannot just be ignored when the server is down. For this reason,
+every write operation is retried once by closing and reopening the connection before
+finally marking a server as dead and raising an exception. We will not attempt to read
+from a dead server for 5 seconds, but a write will always attempt to revive a dead server
+by attempting to connect.
+
+== Installation
+
+  $ sudo gem install memcache --source http://gemcutter.org
+
+== License:
+
+Copyright (c) 2009 Justin Balthrop, Geni.com; Published under The MIT License, see LICENSE

data/Rakefile

@@ -0,0 +1,56 @@
+require 'rubygems'
+require 'rake'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    gem.name = "memcache"
+    gem.summary = %Q{Advanced ruby memcache client}
+    gem.description = %Q{Ruby client for memcached supporting advanced protocol features and pluggable architecture.}
+    gem.email = "code@justinbalthrop.com"
+    gem.homepage = "http://github.com/ninjudd/memcache"
+    gem.authors = ["Justin Balthrop"]
+    # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
+  end
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts "Jeweler (or a dependency) not available. Install it with: sudo gem install jeweler"
+end
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/*_test.rb'
+  test.verbose = true
+end
+
+begin
+  require 'rcov/rcovtask'
+  Rcov::RcovTask.new do |test|
+    test.libs << 'test'
+    test.pattern = 'test/**/*_test.rb'
+    test.verbose = true
+  end
+rescue LoadError
+  task :rcov do
+    abort "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
+  end
+end
+
+task :test => :check_dependencies
+
+task :default => :test
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+  if File.exist?('VERSION')
+    version = File.read('VERSION')
+  else
+    version = ""
+  end
+
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = "memcache #{version}"
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end

data/VERSION.yml

@@ -0,0 +1,4 @@
+--- 
+:patch: 0
+:major: 1
+:minor: 0

data/ext/extconf.rb

@@ -0,0 +1,3 @@
+require 'mkmf'
+have_library('memcached')
+create_makefile('native_server')

data/lib/memcache/local_server.rb

@@ -0,0 +1,107 @@
+class Memcache
+  class LocalServer
+    def initialize
+      @data = {}
+      @expiry = {}
+    end
+
+    def name
+      "local:#{hash}"
+    end
+
+    def stats
+      { # curr_items may include items that have expired.
+        'curr_items'   => @data.size,
+        'expiry_count' => @expiry.size,
+      }
+    end
+
+    def flush_all(delay = 0)
+      raise 'flush_all not supported with delay' if delay != 0
+      @data.clear
+      @expiry.clear
+    end
+
+    def gets(keys)
+      get(keys, true)
+    end
+
+    def get(keys, cas = false)
+      if keys.kind_of?(Array)
+        hash = {}
+        keys.each do |key|
+          key = key.to_s
+          val = get(key)
+          hash[key] = val if val
+        end
+        hash
+      else
+        key = keys.to_s
+        if @expiry[key] and Time.now > @expiry[key]
+          @data[key]   = nil
+          @expiry[key] = nil
+        end
+        @data[key]
+      end
+    end
+
+    def incr(key, amount = 1)
+      key = key.to_s
+      value = get(key)
+      return unless value
+      return unless value =~ /^\d+$/
+
+      value = value.to_i + amount
+      value = 0 if value < 0
+      @data[key] = value.to_s
+      value
+    end
+
+    def decr(key, amount = 1)
+      incr(key, -amount)
+    end
+
+    def delete(key)
+      @data.delete(key.to_s)
+    end
+
+    def set(key, value, expiry = 0, flags = 0)
+      key = key.to_s
+      @data[key] = value.to_s
+      if expiry.kind_of?(Time)
+        @expiry[key] = expiry
+      else  
+        expiry = expiry.to_i
+        @expiry[key] = expiry == 0 ? nil : Time.now + expiry
+      end
+      value
+    end
+
+    def cas(key, value, cas, expiry = 0, flags = 0)
+      # No cas implementation yet, just do a set for now.
+      set(key, value, expiry, flags)
+    end
+
+    def add(key, value, expiry = 0, flags = 0)
+      return nil if get(key)
+      set(key, value, expiry)
+    end
+
+    def replace(key, value, expiry = 0, flags = 0)
+      return nil if get(key).nil?
+      set(key, value, expiry)
+    end
+
+    def append(key, value)
+      existing = get(key)
+      return nil if existing.nil?
+      set(key, existing + value)
+    end
+
+    def prepend(key, value)
+      existing = get(key)
+      return nil if existing.nil?
+      set(key, value + existing)
+    end
+  end
+end

data/lib/memcache/migration.rb

@@ -0,0 +1,23 @@
+class Memcache
+  class Migration < ActiveRecord::Migration
+    class << self
+      attr_accessor :table
+    end
+    
+    def self.up
+      create_table table, :id => false do |t|
+        t.string    :key
+        t.text      :value
+        t.timestamp :expires_at
+        t.timestamp :updated_at
+      end
+      
+      add_index table, [:key], :unique => true
+      add_index table, [:expires_at]
+    end
+    
+    def self.down
+      drop_table table
+    end
+  end
+end

data/lib/memcache/null_server.rb

@@ -0,0 +1,30 @@
+class Memcache
+  class NullServer
+    def name
+      "null"
+    end
+
+    def flush_all(delay = nil)
+    end
+
+    def get(keys)
+      keys.kind_of?(Array) ? {} : nil
+    end
+
+    def incr(key, amount = nil)
+      nil
+    end
+
+    def delete(key, expiry = nil)
+      nil
+    end
+
+    def set(key, value, expiry = nil)
+      nil
+    end
+
+    def add(key, value, expiry = nil)
+      nil
+    end
+  end
+end

data/lib/memcache/pg_server.rb

@@ -0,0 +1,163 @@
+require 'active_record'
+require 'memcache/migration'
+
+class Memcache
+  class PGServer
+    attr_reader :db, :table
+
+    def initialize(opts)
+      @table = opts[:table]
+      @db    = opts[:db] || ActiveRecord::Base.connection.raw_connection
+    end
+
+    def name
+      @name ||= begin
+        db_config = db.instance_variable_get(:@config)
+        "#{db_config[:host]}:#{db_config[:database]}:#{table}"
+      end
+    end
+
+    def flush_all(delay = nil)
+      db.exec("TRUNCATE #{table}")
+    end
+
+    def get(keys)
+      return get([keys])[keys.to_s] unless keys.kind_of?(Array)
+
+      keys = keys.collect {|key| quote(key.to_s)}.join(',')
+      sql = %{
+        SELECT key, value FROM #{table}
+          WHERE key IN (#{keys}) AND #{expiry_clause}
+      }
+      results = {}
+      db.query(sql).each do |key, value|
+        results[key] = value
+      end
+      results
+    end
+
+    def incr(key, amount = 1)
+      transaction do
+        value = get(key)
+        return unless value        
+        return unless value =~ /^\d+$/
+        
+        value = value.to_i + amount
+        value = 0 if value < 0
+        db.exec %{
+          UPDATE #{table} SET value = #{quote(value)}, updated_at = NOW()
+            WHERE key = #{quote(key)}
+        }
+        value
+      end
+    end
+
+    def decr(key, amount = 1)
+      incr(key, -amount)
+    end
+
+    def delete(key)
+      result = db.exec %{
+        DELETE FROM #{table}
+          WHERE key = #{quote(key)}
+      }
+    end
+
+    def set(key, value, expiry = 0)
+      transaction do
+        delete(key)
+        insert(key, value, expiry)
+      end
+      value
+    end
+
+    def add(key, value, expiry = 0)
+      delete_expired(key)
+      insert(key, value, expiry)
+      value
+    rescue PGError => e
+      nil
+    end
+
+    def replace(key, value, expiry = 0)      
+      delete_expired(key)
+      result = update(key, value, expiry)
+      result.cmdtuples == 1 ? value : nil
+    end
+
+    def append(key, value)      
+      delete_expired(key)
+      result = db.exec %{
+        UPDATE #{table}
+          SET value = value || #{quote(value)}, updated_at = NOW()
+          WHERE key = #{quote(key)}
+      }
+      result.cmdtuples == 1
+    end
+
+    def prepend(key, value)      
+      delete_expired(key)
+      result = db.exec %{
+        UPDATE #{table}
+          SET value = #{quote(value)} || value, updated_at = NOW()
+          WHERE key = #{quote(key)}
+      }
+      result.cmdtuples == 1
+    end
+
+  private
+ 
+    def insert(key, value, expiry = 0)
+      db.exec %{
+        INSERT INTO #{table} (key, value, updated_at, expires_at)
+          VALUES (#{quote(key)}, #{quote(value)}, NOW(), #{expiry_sql(expiry)})
+      }
+    end
+
+    def update(key, value, expiry = 0)
+      db.exec %{
+        UPDATE #{table}
+          SET value = #{quote(value)}, updated_at = NOW(), expires_at = #{expiry_sql(expiry)}
+          WHERE key = #{quote(key)}
+      }
+    end
+
+    def transaction
+      return yield if @in_transaction
+      
+      begin 
+        @in_transaction = true
+        db.exec('BEGIN')
+        value = yield
+        db.exec('COMMIT')
+        value
+      rescue Exception => e
+        db.exec('ROLLBACK')
+        raise e
+      ensure
+        @in_transaction = false
+      end
+    end
+      
+    def quote(string)
+      string.to_s.gsub(/'/,"\'")
+      "'#{string}'"
+    end
+
+    def delete_expired(key)
+      db.exec "DELETE FROM #{table} WHERE key = #{quote(key)} AND NOT (#{expiry_clause})"
+    end
+    
+    def expiry_clause
+      "expires_at IS NULL OR expires_at > NOW()"
+    end
+
+    def expiry_sql(expiry)
+      if expiry.kind_of?(Time)
+        quote(expiry.to_s(:db))
+      else
+        expiry == 0 ? 'NULL' : "NOW() + interval '#{expiry} seconds'"
+      end
+    end
+  end
+end

data/lib/memcache/segmented_server.rb

@@ -0,0 +1,97 @@
+require 'digest/sha1'
+
+class Memcache
+  class SegmentedServer < Server
+    MAX_SIZE = 1000000 # bytes
+    PARTIAL_VALUE = 0x40000000
+    
+    def get(keys, opts = {})
+      return get([keys], opts)[keys.to_s] unless keys.kind_of?(Array)
+      return {} if keys.empty?
+
+      results = super
+      keys = {}
+      keys_to_fetch = []
+      results.each do |key, value|
+        next unless segmented?(value)
+        hash, num = value.split(':')
+        keys[key] = []
+        num.to_i.times do |i|
+          hash_key = "#{hash}:#{i}"
+          keys_to_fetch << hash_key
+          keys[key]     << hash_key 
+        end
+      end
+      
+      parts = super(keys_to_fetch)
+      keys.each do |key, hashes|
+        value = ''
+        hashes.each do |hash_key|          
+          if part = parts[hash_key]
+            value << part
+          else
+            value = nil
+            break
+          end
+        end
+
+        if value
+          value.memcache_cas   = results[key].memcache_cas
+          value.memcache_flags = results[key].memcache_flags ^ PARTIAL_VALUE
+          results[key] = value
+        end
+      end
+      results
+    end
+
+    def set(key, value, expiry = 0, flags = 0)
+      value, flags = store_segments(key, value, expiry, flags)
+      super(key, value, expiry, flags) && value
+    end
+
+    def cas(key, value, cas, expiry = 0, flags = 0)
+      value, flags = store_segments(key, value, expiry, flags)
+      super(key, value, cas, expiry, flags)
+    end
+
+    def add(key, value, expiry = 0, flags = 0)
+      value, flags = store_segments(key, value, expiry, flags)
+      super(key, value, expiry, flags)
+    end
+
+    def replace(key, value, expiry = 0, flags = 0)
+      value, flags = store_segments(key, value, expiry, flags)
+      super(key, value, expiry, flags)
+    end
+
+  private
+
+    def segmented?(value)
+      value.memcache_flags & PARTIAL_VALUE == PARTIAL_VALUE
+    end
+
+    def segment(key, value)
+      hash  = Digest::SHA1.hexdigest("#{key}:#{Time.now}:#{rand}")
+      parts = {}
+      i = 0; offset = 0
+      while offset < value.size
+        parts["#{hash}:#{i}"] = value[offset, MAX_SIZE]
+        offset += MAX_SIZE; i += 1
+      end
+      master_key = "#{hash}:#{parts.size}"
+      [master_key, parts]
+    end
+    
+    def store_segments(key, value, expiry = 0, flags = 0)
+      if value and value.size > MAX_SIZE
+        master_key, parts = segment(key, value)
+        parts.each do |hash, data|
+          set(hash, data, expiry)
+        end
+        [master_key, flags | PARTIAL_VALUE]
+      else
+        [value, flags]
+      end
+    end
+  end
+end