redis-objects 0.2.1 → 0.2.4

data/ChangeLog

@@ -0,0 +1,23 @@
+
+*0.2.3 [Final] (18 February 2010)*
+
+* Added lock expiration to Redis::Lock [Ben VandenBos]
+
+* Fixed some bugs [Ben VandenBos]
+
+* Added lock tests and test helpers [Ben VandenBos]
+
+*0.2.2 [Final] (14 December 2009)*
+
+* Added @set.diff(@set2) with "^" and "-" synonyms (oversight). [Nate Wiger]
+
+* Implemented Redis core commands in all data types, such as rename. [Nate Wiger]
+
+* Renamed Redis::Serialize to Redis::Helpers::Serialize to keep Redis:: cleaner. [Nate Wiger]
+
+* More spec coverage. [Nate Wiger]
+
+*0.2.1 [Final] (27 November 2009)*
+
+* First worthwhile public release, with good spec coverage and functionality. [Nate Wiger]
+

data/README.rdoc

@@ -1,48 +1,142 @@
 = Redis::Objects - Map Redis types directly to Ruby objects
 
-This is *not* an ORM.  People that are wrapping ORM's around Redis are missing
-the point.
+This is *not* an ORM. People that are wrapping ORM’s around Redis are missing the point.
 
-The killer feature of Redis that it allows you to perform atomic operations
-on _individual_ data structures, like counters, lists, and sets. You can then use
-these *with* your existing ActiveRecord/DataMapper/etc models, or in classes that have
-nothing to do with an ORM or even a database.  This gem maps Ezra's +redis+ library
-to Ruby objects to make use seamless.
+The killer feature of Redis is that it allows you to perform _atomic_ operations
+on _individual_ data structures, like counters, lists, and sets.  The *atomic* part is HUGE.
+Using an ORM wrapper that retrieves a "record", updates values, then sends those values back,
+_removes_ the atomicity, cutting the nuts off the major advantage of Redis.  Just use MySQL, k?
+
+This gem provides a Rubyish interface to Redis, by mapping {Redis types}[http://code.google.com/p/redis/wiki/CommandReference]
+to Ruby objects, via a thin layer over Ezra's +redis+ gem.  It offers several advantages
+over the lower-level redis-rb API:
+
+1. Easy to integrate directly with existing ORMs - ActiveRecord, DataMapper, etc.  Add counters to your model!
+2. Complex data structures are automatically Marshaled
+3. Integers are returned as integers, rather than '17'
+4. Higher-level types are provided, such as Locks, that wrap multiple calls
 
 This gem originally arose out of a need for high-concurrency atomic operations;
-for a fun rant on the topic, see
-{ATOMICITY}[http://github.com/nateware/redis-objects/blob/master/ATOMICITY.rdoc],
+for a fun rant on the topic, see {An Atomic Rant}[http://nateware.com/2010/02/18/an-atomic-rant],
 or scroll down to "Atomicity" in this README.
 
-There are two ways to use Redis::Objects, either as an +include+ in a model class,
-or by using +new+ with the type of data structure you want to create.
+There are two ways to use Redis::Objects, either as an include in a model class (to
+integrate with ORMs or other classes), or by using new with the type of data structure
+you want to create. 
 
 == Installation
 
-    gem install gemcutter
-    gem tumble
     gem install redis-objects
 
-== Example 1: Individual Usage
+== Example 1: Model Class Usage
+
+Using Redis::Objects this way makes it trivial to integrate Redis types with an
+existing ActiveRecord model, DataMapper resource, or other class.  Redis::Objects
+will work with _any_ class that provides an +id+ method that returns a unique
+value.  Redis::Objects will automatically create keys that are unique to
+each object, in the format:
 
-There is a Ruby object that maps to each Redis type.
+    model_name:id:field_name
 
 === Initialization
 
-You can either set the $redis global variable to your Redis handle:
+Redis::Objects needs a handle created by Redis.new.  (If you're on Rails,
+config/initializers/redis.rb is a good place for this.)
+
+    require 'redis'
+    require 'redis/objects'
+    Redis::Objects.redis = Redis.new(:host => 127.0.0.1, :port => 6379)
+
+Remember you can use Redis::Objects in any Ruby code.  There are *no* dependencies
+on Rails.  Standalone, Sinatra, Resque - no problem.
+
+=== Model Class
+
+You can include Redis::Objects in any type of class:
+
+    class Team < ActiveRecord::Base
+      include Redis::Objects
+
+      lock :trade_players, :expiration => 15  # sec
+      counter :hits
+      counter :runs
+      counter :outs
+      counter :inning, :start => 1
+      list :on_base
+      set :outfielders
+      value :at_bat
+    end
+
+Familiar Ruby array operations Just Work (TM):
+
+    @team = Team.find_by_name('New York Yankees')
+    @team.on_base << 'player1'
+    @team.on_base << 'player2'
+    @team.on_base << 'player3'
+    @team.on_base    # ['player1', 'player2', 'player3']
+    @team.on_base.pop
+    @team.on_base.shift
+    @team.on_base.length  # 1
+    @team.on_base.delete('player2')
+
+Sets work too:
+
+    @team.outfielders << 'outfielder1'
+    @team.outfielders << 'outfielder2'
+    @team.outfielders << 'outfielder1'   # dup ignored
+    @team.outfielders  # ['outfielder1', 'outfielder2']
+    @team.outfielders.each do |player|
+      puts player
+    end
+    player = @team.outfielders.detect{|of| of == 'outfielder2'}
+
+And you can do intersections between objects (kinda cool):
+
+    @team1.outfielders | @team2.outfielders   # outfielders on both teams
+    @team1.outfielders & @team2.outfielders   # in baseball, should be empty :-)
+
+Counters can be atomically incremented/decremented (but not assigned):
+
+    @team.hits.increment  # or incr
+    @team.hits.decrement  # or decr
+    @team.hits.incr(3)    # add 3
+    @team.runs = 4        # exception
+
+Finally, for free, you get a +redis+ method that points directly to a Redis connection:
+
+    Team.redis.get('somekey')
+    @team = Team.new
+    @team.redis.get('somekey')
+    @team.redis.smembers('someset')
+
+You can use the +redis+ handle to directly call any {Redis API command}[http://code.google.com/p/redis/wiki/CommandReference].
+
+== Example 2: Standalone Usage
+
+There is a Ruby class that maps to each Redis type, with methods for each
+{Redis API command}[http://code.google.com/p/redis/wiki/CommandReference].
+Note that calling +new+ does not imply it's actually a "new" value - it just
+creates a mapping between that object and the corresponding Redis data structure,
+which may already exist on the redis-server.
+
+=== Initialization
+
+Redis::Objects needs a handle to the +redis+ server.  For standalone use, you
+can either set the $redis global variable:
 
     $redis = Redis.new(:host => 'localhost', :port => 6379)
-    @value = Redis::Value.new('myvalue')
+    @list  = Redis::List.new('mylist')
 
-Or you can pass the Redis handle into the new method:
+Or you can pass the Redis handle into the new method for each type:
 
     redis  = Redis.new(:host => 'localhost', :port => 6379)
-    @value = Redis::Value.new('myvalue', redis)
+    @list  = Redis::List.new('mylist', redis)
 
 === Counters
 
 Create a new counter. The +counter_name+ is the key stored in Redis.
 
+    require 'redis/counter'
     @counter = Redis::Counter.new('counter_name')
     @counter.increment
     @counter.decrement
@@ -56,10 +150,39 @@ This gem provides a clean way to do atomic blocks as well:
 
 See the section on "Atomicity" for cool uses of atomic counter blocks.
 
+=== Locks
+
+A convenience class that wraps the pattern of {using +setnx+ to perform locking}[http://code.google.com/p/redis/wiki/SetnxCommand].
+
+    require 'redis/lock'
+    @lock = Redis::Lock.new('image_resizing', :expiration => 15, :timeout => 0.1)
+    @lock.lock do
+      # do work
+    end
+
+This can be especially useful if you're running batch jobs spread across multiple hosts.
+
+=== Values
+
+Simple values are easy as well:
+
+    require 'redis/value'
+    @value = Redis::Value.new('value_name')
+    @value.value = 'a'
+    @value.delete
+
+Of course complex data is no problem:
+
+    @account = Account.create!(params[:account])
+    @newest  = Redis::Value.new('newest_account')
+    @newest.value = @account.attributes
+    puts @newest.value['username']
+
 === Lists
 
 Lists work just like Ruby arrays:
 
+    require 'redis/list'
     @list = Redis::List.new('list_name')
     @list << 'a'
     @list << 'b'
@@ -82,11 +205,14 @@ Complex data types are no problem:
     @list.each do |el|
       puts "#{el[:name]} lives in #{el[:city]}"
     end
+    
+This gem will automatically marshal complex data, similar to how session stores work.
 
 === Sets
 
 Sets work like the Ruby {Set}[http://ruby-doc.org/core/classes/Set.html] class:
 
+    require 'redis/set'
     @set = Redis::Set.new('set_name')
     @set << 'a'
     @set << 'b'
@@ -99,7 +225,7 @@ Sets work like the Ruby {Set}[http://ruby-doc.org/core/classes/Set.html] class:
     @set.clear
     # etc
 
-You can perform Redis intersections/unions easily:
+You can perform Redis intersections/unions/diffs easily:
 
     @set1 = Redis::Set.new('set1')
     @set2 = Redis::Set.new('set2')
@@ -107,8 +233,11 @@ You can perform Redis intersections/unions easily:
     members = @set1 & @set2   # intersection
     members = @set1 | @set2   # union
     members = @set1 + @set2   # union
+    members = @set1 ^ @set2   # difference
+    members = @set1 - @set2   # difference
     members = @set1.intersection(@set2, @set3)  # multiple
     members = @set1.union(@set2, @set3)         # multiple
+    members = @set1.difference(@set2, @set3)    # multiple
 
 Or store them in Redis:
 
@@ -116,6 +245,8 @@ Or store them in Redis:
     members = @set1.redis.get('intername')
     @set1.unionstore('unionname', @set2, @set3)
     members = @set1.redis.get('unionname')
+    @set1.diffstore('diffname', @set2, @set3)
+    members = @set1.redis.get('diffname')
 
 And use complex data types too:
 
@@ -125,95 +256,43 @@ And use complex data types too:
     @set2 << {:name => "Jeff", :city => "Del Mar"}
 
     @set1 & @set2  # Nate
+    @set1 - @set2  # Peter
     @set1 | @set2  # all 3 people
 
-=== Values
-
-Simple values are easy as well:
-
-    @value = Redis::Value.new('value_name')
-    @value.value = 'a'
-    @value.delete
-
-Of course complex data is no problem:
-
-    @account = Account.create!(params[:account])
-    @newest  = Redis::Value.new('newest_account')
-    @newest.value = @account
-
-== Example 2: Class Usage
-
-Using Redis::Objects this way makes it trivial to integrate Redis types with an
-existing ActiveRecord model, DataMapper resource, or other class.  Redis::Objects
-will work with any class that provides an +id+ method that returns a unique
-value.  Redis::Objects will automatically create keys that are unique to
-each object.
-
-=== Initialization
+=== Sorted Sets
 
-If on Rails, config/initializers/redis.rb is a good place for this:
+Due to their unique properties, Sorted Sets work like a hybrid between
+a Hash and an Array.  You assign like a Hash, but retrieve like an Array:
 
-    require 'redis'
-    require 'redis/objects'
-    Redis::Objects.redis = Redis.new(:host => 127.0.0.1, :port => 6379)
-
-=== Model Class
-
-Include Redis::Objects in any type of class:
-
-    class Team < ActiveRecord::Base
-      include Redis::Objects
-      
-      counter :hits
-      counter :runs
-      counter :outs
-      counter :inning, :start => 1
-      list :on_base
-      set :outfielders
-      value :at_bat
-    end
-
-Familiar Ruby array operations Just Work (TM):
-
-    @team = Team.find_by_name('New York Yankees')
-    @team.on_base << 'player1'
-    @team.on_base << 'player2'
-    @team.on_base << 'player3'
-    @team.on_base    # ['player1', 'player2']
-    @team.on_base.pop
-    @team.on_base.shift
-    @team.on_base.length  # 1
-    @team.on_base.delete('player3')
-
-Sets operations work too:
+    require 'redis/sorted_set'
+    @sorted_set = Redis::SortedSet.new('number_of_posts')
+    @sorted_set['Nate']  = 15
+    @sorted_set['Peter'] = 75
+    @sorted_set['Jeff']  = 24
     
-    @team.outfielders << 'outfielder1' << 'outfielder1'
-    @team.outfielders << 'outfielder2'
-    @team.outfielders  # ['outfielder1', 'outfielder2']
-    @team.outfielders.each do |player|
-      puts player
-    end
-    player = @team.outfielders.detect{|of| of == 'outfielder2'}
-
-Counters can be atomically incremented/decremented (but not assigned):
+    # atomic increment
+    @sorted_set.increment('Nate')
+    @sorted_set.incr('Peter')   # shorthand
+    @sorted_set.incr('Jeff', 4)
 
-    @team.hits.increment  # or incr
-    @team.hits.decrement  # or decr
-    @team.hits.incr(3)    # add 3
-    @team.runs = 4        # exception
+    @sorted_set[0,2]            # => ["Nate", "Jeff", "Peter"]
+    @sorted_set.first           # => "Nate"
+    @sorted_set.last            # => "Nate"
+    @sorted_set.revrange(0,2)   # => ["Peter", "Jeff", "Nate"]
 
-For free, you get a +redis+ handle usable in your class:
+    @sorted_set['Newbie'] = 1
+    @sorted_set.members         # => ["Newbie", "Nate", "Jeff", "Peter"]
 
-    @team.redis.get('somekey')
-    @team.redis.smembers('someset')
+    @sorted_set.rangebyscore(10, 100, :limit => 2)   # => ["Nate", "Jeff"]
+    @sorted_set.members(:withscores => true)         # => [["Newbie", 1], ["Nate", 16], ["Jeff", 28], ["Peter", 76]]
 
-You can call any operation supported by {Redis}[http://code.google.com/p/redis/wiki/CommandReference]
+The other Redis Sorted Set commands are supported as well; see {SortedSets API}[http://code.google.com/p/redis/wiki/SortedSets].
 
-== Atomicity
+== Atomic Counters and Locks
 
 You are probably not handling atomicity correctly in your app.  For a fun rant
 on the topic, see
-{ATOMICITY}[http://github.com/nateware/redis-objects/blob/master/ATOMICITY.rdoc].
+{An Atomic Rant}[http://nateware.com/2010/02/18/an-atomic-rant].
 
 Atomic counters are a good way to handle concurrency:
 
@@ -260,6 +339,10 @@ Class-level atomic block (may save a DB fetch depending on your app):
 
 Locks with Redis. On completion or exception the lock is released:
 
+    class Team < ActiveRecord::Base
+      lock :reorder # declare a lock
+    end
+
     @team.reorder_lock.lock do
       @team.reorder_all_players
     end
@@ -270,9 +353,19 @@ Class-level lock (same concept)
       Team.reorder_all_players(team_id)
     end
 
+Lock expiration.  Sometimes you want to make sure your locks are cleaned up should
+the unthinkable happen (server failure).  You can set lock expirations to handle
+this.  Expired locks are released by the next process to attempt lock.  Just
+make sure you expiration value is sufficiently large compared to your expected
+lock time.
+
+    class Team < ActiveRecord::Base
+      lock :reorder, :expiration => 15.minutes
+    end
+
 
 == Author
 
-Copyright (c) 2009 {Nate Wiger}[http://nate.wiger.org].  All Rights Reserved.
+Copyright (c) 2009-2010 {Nate Wiger}[http://nate.wiger.org].  All Rights Reserved.
 Released under the {Artistic License}[http://www.opensource.org/licenses/artistic-license-2.0.php].
 

data/lib/redis/counter.rb

@@ -7,11 +7,14 @@ class Redis
   # class to define a counter.
   #
   class Counter
+    require 'redis/helpers/core_commands'
+    include Redis::Helpers::CoreCommands
+
     attr_reader :key, :options, :redis
-    def initialize(key, redis=$redis, options={})
+    def initialize(key, *args)
       @key = key
-      @redis = redis
-      @options = options
+      @options = args.last.is_a?(Hash) ? args.pop : {}
+      @redis = args.first || $redis
       @options[:start] ||= 0
       @redis.setnx(key, @options[:start]) unless @options[:start] == 0 || @options[:init] === false
     end
@@ -21,7 +24,8 @@ class Redis
     # with a parent and starting over (for example, restarting a game and
     # disconnecting all players).
     def reset(to=options[:start])
-      redis.set(key, to.to_i)
+      redis.set key, to.to_i
+      true  # hack for redis-rb regression
     end
 
     # Returns the current value of the counter.  Normally just calling the
@@ -33,19 +37,13 @@ class Redis
     end
     alias_method :get, :value
 
-    # Delete a counter.  Usage discouraged.  Consider +reset+ instead.
-    def delete
-      redis.del(key)
-    end
-    alias_method :del, :delete
-
     # Increment the counter atomically and return the new value.  If passed
     # a block, that block will be evaluated with the new value of the counter
     # as an argument. If the block returns nil or throws an exception, the
     # counter will automatically be decremented to its previous value.  This
     # method is aliased as incr() for brevity.
     def increment(by=1, &block)
-      val = redis.incr(key, by).to_i
+      val = redis.incrby(key, by).to_i
       block_given? ? rewindable_block(:decrement, val, &block) : val
     end
     alias_method :incr, :increment
@@ -56,7 +54,7 @@ class Redis
     # counter will automatically be incremented to its previous value.  This
     # method is aliased as incr() for brevity.
     def decrement(by=1, &block)
-      val = redis.decr(key, by).to_i
+      val = redis.decrby(key, by).to_i
       block_given? ? rewindable_block(:increment, val, &block) : val
     end
     alias_method :decr, :decrement

data/lib/redis/helpers/core_commands.rb

@@ -0,0 +1,54 @@
+class Redis
+  module Helpers
+    # These are core commands that all types share (rename, etc)
+    module CoreCommands
+      def exists?
+        redis.exists key
+      end
+      
+      def delete
+        redis.del key
+      end
+      alias_method :del, :delete
+      alias_method :clear, :delete
+      
+      def type
+        redis.type key
+      end
+
+      def rename(name, setkey=true)
+        dest = name.is_a?(self.class) ? name.key : name
+        ret  = redis.rename key, dest
+        @key = dest if ret && setkey
+        ret
+      end
+
+      def renamenx(name, setkey=true)
+        dest = name.is_a?(self.class) ? name.key : name
+        ret  = redis.renamenx key, dest
+        @key = dest if ret && setkey
+        ret
+      end
+    
+      def expire(seconds)
+        redis.expire key, seconds
+      end
+
+      def expireat(unixtime)
+        redis.expire key, unixtime
+      end
+    
+      def move(dbindex)
+        redis.move key, dbindex
+      end
+      
+      # See the documentation for SORT: http://code.google.com/p/redis/wiki/SortCommand
+      # TODO
+      # def sort(options)
+      #   args = []
+      #   args += ['sort']
+      #   from_redis redis.sort key
+      # end
+    end
+  end
+end

data/lib/redis/helpers/serialize.rb

@@ -0,0 +1,25 @@
+class Redis
+  module Helpers
+    module Serialize
+      include Marshal
+
+      def to_redis(value)
+        case value
+        when String, Fixnum, Bignum, Float
+          value
+        else
+          dump(value)
+        end
+      end
+    
+      def from_redis(value)
+        case value
+        when Array
+          value.collect{|v| from_redis(v)}
+        else
+          restore(value) rescue value
+        end
+      end
+    end
+  end
+end

data/lib/redis/list.rb

@@ -6,14 +6,16 @@ class Redis
   class List
     require 'enumerator'
     include Enumerable
-    require 'redis/serialize'
-    include Redis::Serialize
+    require 'redis/helpers/core_commands'
+    include Redis::Helpers::CoreCommands
+    require 'redis/helpers/serialize'
+    include Redis::Helpers::Serialize
 
     attr_reader :key, :options, :redis
-    def initialize(key, redis=$redis, options={})
+    def initialize(key, *args)
       @key = key
-      @redis = redis
-      @options = options
+      @options = args.last.is_a?(Hash) ? args.pop : {}
+      @redis = args.first || $redis
     end
     
     # Works like push.  Can chain together: list << 'a' << 'b'
@@ -63,6 +65,7 @@ class Redis
     
     # Delete the element(s) from the list that match name. If count is specified,
     # only the first-N (if positive) or last-N (if negative) will be removed.
+    # Use .del to completely delete the entire key.
     # Redis: LREM
     def delete(name, count=0)
       redis.lrem(key, count, name)  # weird api
@@ -96,11 +99,6 @@ class Redis
       at(-1)
     end
 
-    # Clear the list entirely. Redis: DEL
-    def clear
-      redis.del(key)
-    end
-
     # Return the length of the list. Aliased as size. Redis: LLEN
     def length
       redis.llen(key)

data/lib/redis/lock.rb

@@ -10,11 +10,12 @@ class Redis
     class LockTimeout < StandardError; end #:nodoc:
 
     attr_reader :key, :options, :redis
-    def initialize(key, redis=$redis, options={})
+    def initialize(key, *args)
       @key = key
-      @redis = redis
-      @options = options
+      @options = args.last.is_a?(Hash) ? args.pop : {}
+      @redis = args.first || $redis
       @options[:timeout] ||= 5
+      @options[:init] = false if @options[:init].nil? # default :init to false
       @redis.setnx(key, @options[:start]) unless @options[:start] == 0 || @options[:init] === false
     end
 
@@ -31,17 +32,53 @@ class Redis
     def lock(&block)
       start = Time.now
       gotit = false
+      expiration = nil
       while Time.now - start < @options[:timeout]
-        gotit = redis.setnx(key, 1)
+        expiration = generate_expiration
+        # Use the expiration as the value of the lock.
+        gotit = redis.setnx(key, expiration)
         break if gotit
+
+        # Lock is being held.  Now check to see if it's expired (if we're using
+        # lock expiration).
+        # See "Handling Deadlocks" section on http://code.google.com/p/redis/wiki/SetnxCommand
+        if !@options[:expiration].nil?
+          old_expiration = redis.get(key).to_f
+
+          if old_expiration < Time.now.to_f
+            # If it's expired, use GETSET to update it.
+            expiration = generate_expiration
+            old_expiration = redis.getset(key, expiration).to_f
+
+            # Since GETSET returns the old value of the lock, if the old expiration
+            # is still in the past, we know no one else has expired the locked
+            # and we now have it.
+            if old_expiration < Time.now.to_f
+              gotit = true
+              break
+            end
+          end
+        end
+
         sleep 0.1
       end
       raise LockTimeout, "Timeout on lock #{key} exceeded #{@options[:timeout]} sec" unless gotit
       begin
         yield
       ensure
-        redis.del(key)
+        # We need to be careful when cleaning up the lock key.  If we took a really long
+        # time for some reason, and the lock expired, someone else may have it, and
+        # it's not safe for us to remove it.  Check how much time has passed since we
+        # wrote the lock key and only delete it if it hasn't expired (or we're not using
+        # lock expiration)
+        if @options[:expiration].nil? || expiration > Time.now.to_f
+          redis.del(key)
+        end
       end
     end
+
+    def generate_expiration
+      @options[:expiration].nil? ? 1 : (Time.now + @options[:expiration].to_f + 1).to_f
+    end
   end
 end