remix-stash 0.9.6 → 1.0.0

data/AUTHORS

@@ -0,0 +1 @@
+Brian Mitchell (@binary42) <brian@streampunks.com>

data/README.markdown

@@ -1,32 +1,45 @@
-Real docs coming soon! Check out the examples directory for more.
+# Quick & Dirty Specs
 
-# Quick Specs
-
-New API! I've rethought a lot of the API and this comes with a lot of new capabilities. More work is being done on making it as expressive as possible without terrible overhead. This includes vectorized keys which allow emulation of partial cache clearing as well as nice shortcuts like eval and gate for expressions. Options, clusters, and implicit scope are easy to manage on a stash-by-stash basis. Keys are also easy to pass in as it will create composite keys from whatever you pass in (as long as it has to_s) so no more ugly string interpolation all over the place.
+New API that doesn't actually suck! I've rethought a lot of the API and this comes with a lot of new capabilities. More work is being done on making it as expressive as possible without terrible overhead. This includes vectorized keys which allow emulation of partial cache clearing as well as nice shortcuts like eval and gate for expressions. Options, clusters, and implicit scope are easy to manage on a stash-by-stash basis. Keys are also easy to pass in as it will create composite keys from whatever you pass in (as long as it has to_s) so no more ugly string interpolation all over the place.
 
 It's fast (faster than memcache-client). It's simple (pure ruby and only a few hundred lines). It's tested (shoulda). Of course, because it's pure ruby it will run almost anywhere as well unlike many other clients.
 
 It does require memcached 1.4+ but you should be running that anyway (if you aren't, upgrade already).
 
-Take a look and let me know what you think!
+Take a look at the examples and let me know what you think!
+
+# Installation
+
+Right now remix-stash is designed to be run as a gem. I've published it to both github and gemcutter (preferred). You can also safely use this as a rails plugin, just check out the source in your plugin directory and it will be automatically loaded.
+
+## Install via GemCutter
+
+    gem install remix-stash --source=http://gemcutter.org/
+
+## Install via GitHub
+
+Warning: THIS IS CURRENTLY LIMITED TO VERSION 0.9.6 (until github reenables gem building).
 
-# TODO
+    gem install binary42-remix-stash --source=http://gems.github.com/
 
-* namespacing
-* implement the rest of the memcached 1.4 binary API (replace, append, prepend)
+# Using Remix::Stash
+
+The examples directory has some simple code snippets that introduce general features of the library. In general, everything should just work after `require 'remix/stash'`. This includes integration with Rails and automatic pick-up of Heroku style memcached environment variables. Of course, this library is completely independent of Rails and does not need environment variables to function or be used.
+
+# Specifications
+
+This project is tested with shoulda (install via the thoughtbot-shoulda gem on github) and takes the philosophy that fewer moving parts is better. So to avoid complex runners just run `spec/spec.rb` or the spec you are interested in directly. In order for the specs to function, you should have memcached 1.4+ running locally on port 11211.
+
+# Future Work
+
+* add block form to access response headers directly
+* get/set add/replace read/write should allow a CAS flag to be passed
 * allow swappable cluster types for consistent hashing, ketama, etc...
-* failsafe marshal load
 * support non-marshal value dumps configured per stash
 * support multi vector sets
 * thread safe cluster
-* add block form
 * quiet/multi command forms (will require a protocol refactoring most likely)
-* server pings
-* complete stats API
-* incr/decr should take default value flags
-* get/set add/replace read/write should allow a CAS flag to be passed
 * accelerated binary API implementation with Ruby fallback
 * redis support for vectors and/or value
-* large key handling support
 * UDP support (more experimentation on the tradeoffs)
 * EventMachine integration (non-blocking?)

data/VERSION

@@ -1 +1 @@
-0.9.6
+1.0.0

data/init.rb

@@ -0,0 +1 @@
+require 'remix/stash'

data/lib/remix/stash/auto_detection.rb

@@ -0,0 +1,27 @@
+if defined?(Rails)
+  stash.default(:logger => Rails.logger)
+
+  module Remix::Stash::RailsSupport
+  private
+
+    def cycle_action_vectors
+      Remix::Stash.cycle_action
+    end
+
+  end
+
+  class ActionController::Base
+    include Remix::Stash::RailsSupport
+    after_filter :cycle_action_vectors
+  end
+
+  if servers = ENV['MEMCACHED_SERVERS']
+    Remix::Stash.define_cluster(:environment => servers.split(','))
+    stash.default(:cluster => :environment)
+  end
+
+  if namespace = ENV['MEMCACHED_NAMESPACE']
+    stash.default(:namespace => namespace)
+  end
+
+end

data/lib/remix/stash/cluster.rb

@@ -2,6 +2,7 @@ require 'socket'
 require 'digest/md5'
 
 class Remix::Stash::Cluster
+  include Enumerable
   include Socket::Constants
 
   @@connections = {}

data/lib/remix/stash/protocol.rb

@@ -58,10 +58,23 @@ module Remix::Stash::Protocol
     resp[:status] == NO_ERROR
   end
 
-  DECR_PACKET = HEADER_FORMAT + 'NNQNa*'
-  def decr(io, key, step)
+  APPEND_PACKET = HEADER_FORMAT + 'a*a*'
+  def append(io, key, data)
+    header = [REQUEST, APPEND, key.size, 0, 0, 0, data.size + key.size, 0, 0, key, data].pack(APPEND_PACKET)
+    io.write(header)
+    resp = read_resp(io)
+    resp[:status] == NO_ERROR
+  end
+
+  DECR_PACKET = HEADER_FORMAT + 'NNNNNa*'
+  def decr(io, key, step, default = nil, ttl = nil)
     low, high = split64(step)
-    header = [REQUEST, DECREMENT, key.size, 20, 0, 0, key.size + 20, 0, 0, high, low, 0, COUNTER_FAULT_EXPIRATION, key].pack(DECR_PACKET)
+    if default
+      default_low, default_high = split64(default)
+      header = [REQUEST, DECREMENT, key.size, 20, 0, 0, key.size + 20, 0, 0, high, low, default_high, default_low, ttl, key].pack(DECR_PACKET)
+    else
+      header = [REQUEST, DECREMENT, key.size, 20, 0, 0, key.size + 20, 0, 0, high, low, 0, 0, COUNTER_FAULT_EXPIRATION, key].pack(DECR_PACKET)
+    end
     io.write(header)
     resp = read_resp(io)
     if resp[:status] == NO_ERROR
@@ -94,10 +107,15 @@ module Remix::Stash::Protocol
     resp[:status] == NO_ERROR ? resp[:body][GET_BODY] : nil
   end
 
-  INCR_PACKET = HEADER_FORMAT + 'NNQNa*'
-  def incr(io, key, step)
+  INCR_PACKET = HEADER_FORMAT + 'NNNNNa*'
+  def incr(io, key, step, default = nil, ttl = nil)
     low, high = split64(step)
-    header = [REQUEST, INCREMENT, key.size, 20, 0, 0, key.size + 20, 0, 0, high, low, 0, COUNTER_FAULT_EXPIRATION, key].pack(INCR_PACKET)
+    if default
+      default_low, default_high = split64(default)
+      header = [REQUEST, INCREMENT, key.size, 20, 0, 0, key.size + 20, 0, 0, high, low, default_high, default_low, ttl, key].pack(DECR_PACKET)
+    else
+      header = [REQUEST, INCREMENT, key.size, 20, 0, 0, key.size + 20, 0, 0, high, low, 0, 0, COUNTER_FAULT_EXPIRATION, key].pack(INCR_PACKET)
+    end
     io.write(header)
     resp = read_resp(io)
     if resp[:status] == NO_ERROR
@@ -105,6 +123,30 @@ module Remix::Stash::Protocol
     end
   end
 
+  PING_PACKET = HEADER_FORMAT
+  def ping(io)
+    header = [REQUEST, NO_OP, 0, 0, 0, 0, 0, 0, 0].pack(PING_PACKET)
+    io.write(header)
+    resp = read_resp(io)
+    resp[:status] == NO_ERROR
+  end
+
+  PREPEND_PACKET = HEADER_FORMAT + 'a*a*'
+  def prepend(io, key, data)
+    header = [REQUEST, PREPEND, key.size, 0, 0, 0, data.size + key.size, 0, 0, key, data].pack(PREPEND_PACKET)
+    io.write(header)
+    resp = read_resp(io)
+    resp[:status] == NO_ERROR
+  end
+
+  REPLACE_PACKET = HEADER_FORMAT + 'NNa*a*'
+  def replace(io, key, data, ttl = 0)
+    header = [REQUEST, REPLACE, key.size, 8, 0, 0, data.size + key.size + 8, 0, 0, 0, ttl, key, data].pack(REPLACE_PACKET)
+    io.write(header)
+    resp = read_resp(io)
+    resp[:status] == NO_ERROR
+  end
+
   SET_PACKET = HEADER_FORMAT + 'NNa*a*'
   def set(io, key, data, ttl = 0)
     header = [REQUEST, SET, key.size, 8, 0, 0, data.size + key.size + 8, 0, 0, 0, ttl, key, data].pack(SET_PACKET)
@@ -113,6 +155,17 @@ module Remix::Stash::Protocol
     resp[:status] == NO_ERROR
   end
 
+  STAT_PACKET = HEADER_FORMAT + 'a*'
+  def stat(io, info = '')
+    header = [REQUEST, STAT, info.size, 0, 0, 0, info.size, 0, 0, info].pack(STAT_PACKET)
+    io.write(header)
+    loop do
+      stat = read_stat(io, info)
+      break unless stat[:key]
+      yield *stat.values_at(:key, :value)
+    end
+  end
+
 private
 
   COUNTER_SPLIT = 'NN'
@@ -132,6 +185,18 @@ private
       {:status => status, :body => io.read(body_length)}
   end
 
+  STAT_HEADER = '@2n@6nN'
+  def read_stat(io, info)
+    header = io.read(24)
+    header or raise Remix::Stash::ProtocolError,
+    "No data in response header"
+    key_length, status, body_length = *header.unpack(STAT_HEADER)
+    stat = {:status => status}
+    stat[:key] = io.read(key_length) if key_length > 0
+    stat[:value] = io.read(body_length - key_length) if body_length - key_length > 0
+    stat
+  end
+
   def split64(n)
     [0xFFFFFFFF & n, n >> 32]
   end

data/lib/remix/stash.rb

@@ -40,13 +40,6 @@ class Remix::Stash
     end
   end
 
-  def add(*keys)
-    opts = default_opts(keys)
-    value = keys.pop
-    key = canonical_key(keys, opts)
-    cluster(opts).select(key) {|io| Protocol.add(io, key, value, opts[:ttl])}
-  end
-
   def clear(*keys)
     opts = default_opts(keys)
     if keys.empty?
@@ -81,7 +74,7 @@ class Remix::Stash
     opts = default_opts(keys)
     step = keys.pop
     key = canonical_key(keys, opts)
-    cluster(opts).select(key) {|io| Protocol.decr(io, key, step)}
+    cluster(opts).select(key) {|io| Protocol.decr(io, key, step, opts[:default], opts[:ttl])}
   end
 
   def default(opts = nil)
@@ -101,12 +94,6 @@ class Remix::Stash
     @opts
   end
 
-  def delete(*keys)
-    opts = default_opts(keys)
-    key = canonical_key(keys, opts)
-    cluster(opts).select(key) {|io| Protocol.delete(io, key)}
-  end
-
   def eval(*keys)
     opts = default_opts(keys)
     key = canonical_key(keys, opts)
@@ -146,7 +133,11 @@ class Remix::Stash
     opts = default_opts(keys)
     step = keys.pop
     key = canonical_key(keys, opts)
-    cluster(opts).select(key) {|io| Protocol.incr(io, key, step)}
+    cluster(opts).select(key) {|io| Protocol.incr(io, key, step, opts[:default], opts[:ttl])}
+  end
+
+  def ping(name = default[:cluster])
+    cluster(:cluster => name).map {|io| Protocol.ping(io)}
   end
 
   def read(*keys)
@@ -168,10 +159,51 @@ class Remix::Stash
     opts = default_opts(keys)
     value = keys.pop
     key = canonical_key(keys, opts)
-    cluster(opts).select(key) {|io| Protocol.set(io, key, dump_value(value), opts[:ttl])}
+    cluster(opts).select(key) {|io|
+      case opts[:op]
+      when :add
+        Protocol.add(io, key, dump_value(value), opts[:ttl])
+      when :replace
+        Protocol.replace(io, key, dump_value(value), opts[:ttl])
+      else
+        Protocol.set(io, key, dump_value(value), opts[:ttl])
+      end
+    }
   end
   alias []= set
 
+  def stats(name = default[:cluster])
+    cluster(:cluster => name).map do |io|
+      stats = {:settings => {}, :slabs => [], :foo => {}}
+      Protocol.stat(io) {|key, value|
+        stats[key.to_sym] = normalize_stat(value)
+      }
+      Protocol.stat(io, 'settings') {|key, value|
+        stats[:settings][key.to_sym] = normalize_stat(value)
+      }
+      prefix = stats[:settings][:stat_key_prefix]
+      Protocol.stat(io, 'items') {|key, value|
+        part, slab_index, subkey = *key.split(prefix)
+        slab_index = slab_index.to_i
+        stats[:slabs][slab_index] ||= {:index => slab_index}
+        stats[:slabs][slab_index][subkey.to_sym] = normalize_stat(value)
+      }
+      Protocol.stat(io, 'slabs') {|key, value|
+        parts = key.split(prefix)
+        if parts.size == 1
+          stats["slabs_#{key}".to_sym] = normalize_stat(value)
+        else
+          slab_index, subkey = parts.values_at(-2, -1)
+          slab_index = slab_index.to_i
+          stats[:slabs][slab_index] ||= {:index => slab_index}
+          stats[:slabs][slab_index][subkey.to_sym] = normalize_stat(value)
+        end
+      }
+      stats[:slabs].compact!
+      stats
+    end
+  end
+
   def transaction
     yield self
   ensure
@@ -182,7 +214,20 @@ class Remix::Stash
     opts = default_opts(keys)
     value = keys.pop
     key = canonical_key(keys, opts)
-    cluster(opts).select(key) {|io| Protocol.set(io, key, value, opts[:ttl])}
+    cluster(opts).select(key) {|io|
+      case opts[:op]
+      when :add
+        Protocol.add(io, key, value, opts[:ttl])
+      when :replace
+        Protocol.replace(io, key, value, opts[:ttl])
+      when :append
+        Protocol.append(io, key, value)
+      when :prepend
+        Protocol.prepend(io, key, value)
+      else
+        Protocol.set(io, key, value, opts[:ttl])
+      end
+    }
   end
 
 protected
@@ -196,13 +241,16 @@ private
   KEY_SEPARATOR = '/'
   def canonical_key(keys, opts)
     v = vector(opts)
-    if @scope
-      "#{implicit_scope}#{keys.join(KEY_SEPARATOR)}#{vector(opts)}"
-    elsif v
-      keys.join(KEY_SEPARATOR) << v
-    else
-      keys.join(KEY_SEPARATOR)
-    end
+    namespace = opts[:namespace].to_s
+    key = namespace +
+      if @scope
+        "#{implicit_scope}#{keys.join(KEY_SEPARATOR)}#{vector(opts)}"
+      elsif v
+        keys.join(KEY_SEPARATOR) << v
+      else
+        keys.join(KEY_SEPARATOR)
+      end
+    key.size > 250 ? Digest::MD5.hexdigest(key) : key
   end
 
   def cluster(opts = {})
@@ -223,12 +271,23 @@ private
 
   def load_value(data)
     Marshal.load(data) if data
-  rescue TypeError, ArgumentError
+  rescue TypeError, ArgumentError => e
+    if e.message =~ /undefined class\/module (.*)/
+      retry if begin
+        $1.split('::').inject(Object) {|m,x|
+          m.const_get(x)}
+      rescue Exception
+      end
+    end
     logger = default_opts[:logger]
     logger && logger.error("[stash] Unable to load marshal stream: #{data.inspect}")
     nil
   end
 
+  def normalize_stat(value)
+    value == "NULL" ? nil : (Integer(value) rescue Float(value)) rescue value
+  end
+
   def vector(opts)
     return if @name == :root
     return @vector if @vector && opts[:coherency] != :dynamic
@@ -254,3 +313,5 @@ end
 
 class Object; include Remix::Stash::Extension end
 module Remix; extend Remix::Stash::Extension end
+
+require 'remix/stash/auto_detection'

data/remix-stash.gemspec

@@ -5,11 +5,11 @@
 
 Gem::Specification.new do |s|
   s.name = %q{remix-stash}
-  s.version = "0.9.6"
+  s.version = "1.0.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Brian Mitchell"]
-  s.date = %q{2009-09-22}
+  s.date = %q{2009-10-03}
   s.email = %q{binary42@gmail.com}
   s.extra_rdoc_files = [
     "LICENSE",
@@ -17,6 +17,7 @@ Gem::Specification.new do |s|
   ]
   s.files = [
     ".gitignore",
+     "AUTHORS",
      "LICENSE",
      "README.markdown",
      "Rakefile",
@@ -29,14 +30,65 @@ Gem::Specification.new do |s|
      "examples/scope.rb",
      "examples/stash.rb",
      "harness.rb",
+     "init.rb",
      "lib/remix/stash.rb",
+     "lib/remix/stash/auto_detection.rb",
      "lib/remix/stash/cluster.rb",
      "lib/remix/stash/extension.rb",
      "lib/remix/stash/protocol.rb",
      "remix-stash.gemspec",
+     "spec/auto_detection_spec.rb",
      "spec/extension_spec.rb",
      "spec/spec.rb",
-     "spec/stash_spec.rb"
+     "spec/stash_spec.rb",
+     "spec/support/bar.rb",
+     "spec/support/rails/README",
+     "spec/support/rails/Rakefile",
+     "spec/support/rails/app/controllers/application_controller.rb",
+     "spec/support/rails/app/helpers/application_helper.rb",
+     "spec/support/rails/config/boot.rb",
+     "spec/support/rails/config/database.yml",
+     "spec/support/rails/config/environment.rb",
+     "spec/support/rails/config/environments/development.rb",
+     "spec/support/rails/config/environments/production.rb",
+     "spec/support/rails/config/environments/test.rb",
+     "spec/support/rails/config/initializers/backtrace_silencers.rb",
+     "spec/support/rails/config/initializers/inflections.rb",
+     "spec/support/rails/config/initializers/mime_types.rb",
+     "spec/support/rails/config/initializers/new_rails_defaults.rb",
+     "spec/support/rails/config/initializers/session_store.rb",
+     "spec/support/rails/config/locales/en.yml",
+     "spec/support/rails/config/routes.rb",
+     "spec/support/rails/db/seeds.rb",
+     "spec/support/rails/doc/README_FOR_APP",
+     "spec/support/rails/log/development.log",
+     "spec/support/rails/log/production.log",
+     "spec/support/rails/log/server.log",
+     "spec/support/rails/log/test.log",
+     "spec/support/rails/public/404.html",
+     "spec/support/rails/public/422.html",
+     "spec/support/rails/public/500.html",
+     "spec/support/rails/public/favicon.ico",
+     "spec/support/rails/public/images/rails.png",
+     "spec/support/rails/public/index.html",
+     "spec/support/rails/public/javascripts/application.js",
+     "spec/support/rails/public/javascripts/controls.js",
+     "spec/support/rails/public/javascripts/dragdrop.js",
+     "spec/support/rails/public/javascripts/effects.js",
+     "spec/support/rails/public/javascripts/prototype.js",
+     "spec/support/rails/public/robots.txt",
+     "spec/support/rails/script/about",
+     "spec/support/rails/script/console",
+     "spec/support/rails/script/dbconsole",
+     "spec/support/rails/script/destroy",
+     "spec/support/rails/script/generate",
+     "spec/support/rails/script/performance/benchmarker",
+     "spec/support/rails/script/performance/profiler",
+     "spec/support/rails/script/plugin",
+     "spec/support/rails/script/runner",
+     "spec/support/rails/script/server",
+     "spec/support/rails/test/performance/browsing_test.rb",
+     "spec/support/rails/test/test_helper.rb"
   ]
   s.has_rdoc = true
   s.homepage = %q{http://github.com/binary42/remix-stash}
@@ -45,9 +97,27 @@ Gem::Specification.new do |s|
   s.rubygems_version = %q{1.3.1}
   s.summary = %q{Remix your memcache}
   s.test_files = [
-    "spec/extension_spec.rb",
+    "spec/auto_detection_spec.rb",
+     "spec/extension_spec.rb",
      "spec/spec.rb",
      "spec/stash_spec.rb",
+     "spec/support/bar.rb",
+     "spec/support/rails/app/controllers/application_controller.rb",
+     "spec/support/rails/app/helpers/application_helper.rb",
+     "spec/support/rails/config/boot.rb",
+     "spec/support/rails/config/environment.rb",
+     "spec/support/rails/config/environments/development.rb",
+     "spec/support/rails/config/environments/production.rb",
+     "spec/support/rails/config/environments/test.rb",
+     "spec/support/rails/config/initializers/backtrace_silencers.rb",
+     "spec/support/rails/config/initializers/inflections.rb",
+     "spec/support/rails/config/initializers/mime_types.rb",
+     "spec/support/rails/config/initializers/new_rails_defaults.rb",
+     "spec/support/rails/config/initializers/session_store.rb",
+     "spec/support/rails/config/routes.rb",
+     "spec/support/rails/db/seeds.rb",
+     "spec/support/rails/test/performance/browsing_test.rb",
+     "spec/support/rails/test/test_helper.rb",
      "examples/eval.rb",
      "examples/gate.rb",
      "examples/getset.rb",