redis-objects 0.2.2 → 0.2.4

data/ChangeLog

@@ -1,4 +1,12 @@
 
+*0.2.3 [Final] (18 February 2010)*
+
+* Added lock expiration to Redis::Lock [Ben VandenBos]
+
+* Fixed some bugs [Ben VandenBos]
+
+* Added lock tests and test helpers [Ben VandenBos]
+
 *0.2.2 [Final] (14 December 2009)*
 
 * Added @set.diff(@set2) with "^" and "-" synonyms (oversight). [Nate Wiger]

data/README.rdoc

@@ -1,44 +1,136 @@
 = Redis::Objects - Map Redis types directly to Ruby objects
 
-This is *not* an ORM.  People that are wrapping ORM's around Redis are missing
-the point.
+This is *not* an ORM. People that are wrapping ORM’s around Redis are missing the point.
 
-The killer feature of Redis that it allows you to perform atomic operations
-on _individual_ data structures, like counters, lists, and sets. You can then use
-these *with* your existing ActiveRecord/DataMapper/etc models, or in classes that have
-nothing to do with an ORM or even a database.  This gem maps {Redis types}[http://code.google.com/p/redis/wiki/CommandReference]
-to Ruby objects, via a thin layer over Ezra's +redis+ gem.
+The killer feature of Redis is that it allows you to perform _atomic_ operations
+on _individual_ data structures, like counters, lists, and sets.  The *atomic* part is HUGE.
+Using an ORM wrapper that retrieves a "record", updates values, then sends those values back,
+_removes_ the atomicity, cutting the nuts off the major advantage of Redis.  Just use MySQL, k?
+
+This gem provides a Rubyish interface to Redis, by mapping {Redis types}[http://code.google.com/p/redis/wiki/CommandReference]
+to Ruby objects, via a thin layer over Ezra's +redis+ gem.  It offers several advantages
+over the lower-level redis-rb API:
+
+1. Easy to integrate directly with existing ORMs - ActiveRecord, DataMapper, etc.  Add counters to your model!
+2. Complex data structures are automatically Marshaled
+3. Integers are returned as integers, rather than '17'
+4. Higher-level types are provided, such as Locks, that wrap multiple calls
 
 This gem originally arose out of a need for high-concurrency atomic operations;
-for a fun rant on the topic, see
-{ATOMICITY}[http://github.com/nateware/redis-objects/blob/master/ATOMICITY.rdoc],
+for a fun rant on the topic, see {An Atomic Rant}[http://nateware.com/2010/02/18/an-atomic-rant],
 or scroll down to "Atomicity" in this README.
 
-There are two ways to use Redis::Objects, either as an +include+ in a model class,
-or by using +new+ with the type of data structure you want to create.
+There are two ways to use Redis::Objects, either as an include in a model class (to
+integrate with ORMs or other classes), or by using new with the type of data structure
+you want to create. 
 
 == Installation
 
-    gem install gemcutter
-    gem tumble
     gem install redis-objects
 
-== Example 1: Standalone Usage
+== Example 1: Model Class Usage
+
+Using Redis::Objects this way makes it trivial to integrate Redis types with an
+existing ActiveRecord model, DataMapper resource, or other class.  Redis::Objects
+will work with _any_ class that provides an +id+ method that returns a unique
+value.  Redis::Objects will automatically create keys that are unique to
+each object, in the format:
 
-There is a Ruby object that maps to each Redis type.
+    model_name:id:field_name
 
 === Initialization
 
-This gem needs a handle to the +redis+ server.  For standalone use, you can
-either set the $redis global variable to your Redis.new handle:
+Redis::Objects needs a handle created by Redis.new.  (If you're on Rails,
+config/initializers/redis.rb is a good place for this.)
+
+    require 'redis'
+    require 'redis/objects'
+    Redis::Objects.redis = Redis.new(:host => 127.0.0.1, :port => 6379)
+
+Remember you can use Redis::Objects in any Ruby code.  There are *no* dependencies
+on Rails.  Standalone, Sinatra, Resque - no problem.
+
+=== Model Class
+
+You can include Redis::Objects in any type of class:
+
+    class Team < ActiveRecord::Base
+      include Redis::Objects
+
+      lock :trade_players, :expiration => 15  # sec
+      counter :hits
+      counter :runs
+      counter :outs
+      counter :inning, :start => 1
+      list :on_base
+      set :outfielders
+      value :at_bat
+    end
+
+Familiar Ruby array operations Just Work (TM):
+
+    @team = Team.find_by_name('New York Yankees')
+    @team.on_base << 'player1'
+    @team.on_base << 'player2'
+    @team.on_base << 'player3'
+    @team.on_base    # ['player1', 'player2', 'player3']
+    @team.on_base.pop
+    @team.on_base.shift
+    @team.on_base.length  # 1
+    @team.on_base.delete('player2')
+
+Sets work too:
+
+    @team.outfielders << 'outfielder1'
+    @team.outfielders << 'outfielder2'
+    @team.outfielders << 'outfielder1'   # dup ignored
+    @team.outfielders  # ['outfielder1', 'outfielder2']
+    @team.outfielders.each do |player|
+      puts player
+    end
+    player = @team.outfielders.detect{|of| of == 'outfielder2'}
+
+And you can do intersections between objects (kinda cool):
+
+    @team1.outfielders | @team2.outfielders   # outfielders on both teams
+    @team1.outfielders & @team2.outfielders   # in baseball, should be empty :-)
+
+Counters can be atomically incremented/decremented (but not assigned):
+
+    @team.hits.increment  # or incr
+    @team.hits.decrement  # or decr
+    @team.hits.incr(3)    # add 3
+    @team.runs = 4        # exception
+
+Finally, for free, you get a +redis+ method that points directly to a Redis connection:
+
+    Team.redis.get('somekey')
+    @team = Team.new
+    @team.redis.get('somekey')
+    @team.redis.smembers('someset')
+
+You can use the +redis+ handle to directly call any {Redis API command}[http://code.google.com/p/redis/wiki/CommandReference].
+
+== Example 2: Standalone Usage
+
+There is a Ruby class that maps to each Redis type, with methods for each
+{Redis API command}[http://code.google.com/p/redis/wiki/CommandReference].
+Note that calling +new+ does not imply it's actually a "new" value - it just
+creates a mapping between that object and the corresponding Redis data structure,
+which may already exist on the redis-server.
+
+=== Initialization
+
+Redis::Objects needs a handle to the +redis+ server.  For standalone use, you
+can either set the $redis global variable:
 
     $redis = Redis.new(:host => 'localhost', :port => 6379)
-    @value = Redis::Value.new('myvalue')
+    @list  = Redis::List.new('mylist')
 
-Or you can pass the Redis handle into the new method:
+Or you can pass the Redis handle into the new method for each type:
 
     redis  = Redis.new(:host => 'localhost', :port => 6379)
-    @value = Redis::Value.new('myvalue', redis)
+    @list  = Redis::List.new('mylist', redis)
 
 === Counters
 
@@ -58,6 +150,34 @@ This gem provides a clean way to do atomic blocks as well:
 
 See the section on "Atomicity" for cool uses of atomic counter blocks.
 
+=== Locks
+
+A convenience class that wraps the pattern of {using +setnx+ to perform locking}[http://code.google.com/p/redis/wiki/SetnxCommand].
+
+    require 'redis/lock'
+    @lock = Redis::Lock.new('image_resizing', :expiration => 15, :timeout => 0.1)
+    @lock.lock do
+      # do work
+    end
+
+This can be especially useful if you're running batch jobs spread across multiple hosts.
+
+=== Values
+
+Simple values are easy as well:
+
+    require 'redis/value'
+    @value = Redis::Value.new('value_name')
+    @value.value = 'a'
+    @value.delete
+
+Of course complex data is no problem:
+
+    @account = Account.create!(params[:account])
+    @newest  = Redis::Value.new('newest_account')
+    @newest.value = @account.attributes
+    puts @newest.value['username']
+
 === Lists
 
 Lists work just like Ruby arrays:
@@ -85,6 +205,8 @@ Complex data types are no problem:
     @list.each do |el|
       puts "#{el[:name]} lives in #{el[:city]}"
     end
+    
+This gem will automatically marshal complex data, similar to how session stores work.
 
 === Sets
 
@@ -111,8 +233,8 @@ You can perform Redis intersections/unions/diffs easily:
     members = @set1 & @set2   # intersection
     members = @set1 | @set2   # union
     members = @set1 + @set2   # union
-    members = @set1 ^ @set2   # union
-    members = @set1 - @set2   # union
+    members = @set1 ^ @set2   # difference
+    members = @set1 - @set2   # difference
     members = @set1.intersection(@set2, @set3)  # multiple
     members = @set1.union(@set2, @set3)         # multiple
     members = @set1.difference(@set2, @set3)    # multiple
@@ -137,100 +259,40 @@ And use complex data types too:
     @set1 - @set2  # Peter
     @set1 | @set2  # all 3 people
 
-=== Values
-
-Simple values are easy as well:
-
-    require 'redis/value'
-    @value = Redis::Value.new('value_name')
-    @value.value = 'a'
-    @value.delete
-
-Of course complex data is no problem:
-
-    @account = Account.create!(params[:account])
-    @newest  = Redis::Value.new('newest_account')
-    @newest.value = @account
-
-== Example 2: Model Class Usage
+=== Sorted Sets
 
-Using Redis::Objects this way makes it trivial to integrate Redis types with an
-existing ActiveRecord model, DataMapper resource, or other class.  Redis::Objects
-will work with _any_ class that provides an +id+ method that returns a unique
-value.  Redis::Objects will automatically create keys that are unique to
-each object.
-
-=== Initialization
-
-If on Rails, config/initializers/redis.rb is a good place for this:
-
-    require 'redis'
-    require 'redis/objects'
-    Redis::Objects.redis = Redis.new(:host => 127.0.0.1, :port => 6379)
-
-=== Model Class
+Due to their unique properties, Sorted Sets work like a hybrid between
+a Hash and an Array.  You assign like a Hash, but retrieve like an Array:
 
-Include Redis::Objects in any type of class:
-
-    class Team < ActiveRecord::Base
-      include Redis::Objects
-      
-      counter :hits
-      counter :runs
-      counter :outs
-      counter :inning, :start => 1
-      list :on_base
-      set :outfielders
-      value :at_bat
-    end
-
-Familiar Ruby array operations Just Work (TM):
-
-    @team = Team.find_by_name('New York Yankees')
-    @team.on_base << 'player1'
-    @team.on_base << 'player2'
-    @team.on_base << 'player3'
-    @team.on_base    # ['player1', 'player2']
-    @team.on_base.pop
-    @team.on_base.shift
-    @team.on_base.length  # 1
-    @team.on_base.delete('player3')
-
-Sets work too:
+    require 'redis/sorted_set'
+    @sorted_set = Redis::SortedSet.new('number_of_posts')
+    @sorted_set['Nate']  = 15
+    @sorted_set['Peter'] = 75
+    @sorted_set['Jeff']  = 24
     
-    @team.outfielders << 'outfielder1' << 'outfielder1'
-    @team.outfielders << 'outfielder2'
-    @team.outfielders  # ['outfielder1', 'outfielder2']
-    @team.outfielders.each do |player|
-      puts player
-    end
-    player = @team.outfielders.detect{|of| of == 'outfielder2'}
+    # atomic increment
+    @sorted_set.increment('Nate')
+    @sorted_set.incr('Peter')   # shorthand
+    @sorted_set.incr('Jeff', 4)
 
-And you can do intersections between ORM objects (kinda cool):
+    @sorted_set[0,2]            # => ["Nate", "Jeff", "Peter"]
+    @sorted_set.first           # => "Nate"
+    @sorted_set.last            # => "Nate"
+    @sorted_set.revrange(0,2)   # => ["Peter", "Jeff", "Nate"]
 
-    @team1.outfielders | @team2.outfielders   # all outfielders
-    @team1.outfielders & @team2.outfielders   # should be empty
+    @sorted_set['Newbie'] = 1
+    @sorted_set.members         # => ["Newbie", "Nate", "Jeff", "Peter"]
 
-Counters can be atomically incremented/decremented (but not assigned):
-
-    @team.hits.increment  # or incr
-    @team.hits.decrement  # or decr
-    @team.hits.incr(3)    # add 3
-    @team.runs = 4        # exception
+    @sorted_set.rangebyscore(10, 100, :limit => 2)   # => ["Nate", "Jeff"]
+    @sorted_set.members(:withscores => true)         # => [["Newbie", 1], ["Nate", 16], ["Jeff", 28], ["Peter", 76]]
 
-Finally, for free, you get a +redis+ handle usable in your class that
-points directly to a Redis API object:
-
-    @team.redis.get('somekey')
-    @team.redis.smembers('someset')
-
-You can use the +redis+ handle to directly call any {Redis command}[http://code.google.com/p/redis/wiki/CommandReference]
+The other Redis Sorted Set commands are supported as well; see {SortedSets API}[http://code.google.com/p/redis/wiki/SortedSets].
 
 == Atomic Counters and Locks
 
 You are probably not handling atomicity correctly in your app.  For a fun rant
 on the topic, see
-{ATOMICITY}[http://github.com/nateware/redis-objects/blob/master/ATOMICITY.rdoc].
+{An Atomic Rant}[http://nateware.com/2010/02/18/an-atomic-rant].
 
 Atomic counters are a good way to handle concurrency:
 
@@ -277,6 +339,10 @@ Class-level atomic block (may save a DB fetch depending on your app):
 
 Locks with Redis. On completion or exception the lock is released:
 
+    class Team < ActiveRecord::Base
+      lock :reorder # declare a lock
+    end
+
     @team.reorder_lock.lock do
       @team.reorder_all_players
     end
@@ -287,9 +353,19 @@ Class-level lock (same concept)
       Team.reorder_all_players(team_id)
     end
 
+Lock expiration.  Sometimes you want to make sure your locks are cleaned up should
+the unthinkable happen (server failure).  You can set lock expirations to handle
+this.  Expired locks are released by the next process to attempt lock.  Just
+make sure you expiration value is sufficiently large compared to your expected
+lock time.
+
+    class Team < ActiveRecord::Base
+      lock :reorder, :expiration => 15.minutes
+    end
+
 
 == Author
 
-Copyright (c) 2009 {Nate Wiger}[http://nate.wiger.org].  All Rights Reserved.
+Copyright (c) 2009-2010 {Nate Wiger}[http://nate.wiger.org].  All Rights Reserved.
 Released under the {Artistic License}[http://www.opensource.org/licenses/artistic-license-2.0.php].
 

data/lib/redis/counter.rb

@@ -11,10 +11,10 @@ class Redis
     include Redis::Helpers::CoreCommands
 
     attr_reader :key, :options, :redis
-    def initialize(key, redis=$redis, options={})
+    def initialize(key, *args)
       @key = key
-      @redis = redis
-      @options = options
+      @options = args.last.is_a?(Hash) ? args.pop : {}
+      @redis = args.first || $redis
       @options[:start] ||= 0
       @redis.setnx(key, @options[:start]) unless @options[:start] == 0 || @options[:init] === false
     end
@@ -25,6 +25,7 @@ class Redis
     # disconnecting all players).
     def reset(to=options[:start])
       redis.set key, to.to_i
+      true  # hack for redis-rb regression
     end
 
     # Returns the current value of the counter.  Normally just calling the
@@ -42,7 +43,7 @@ class Redis
     # counter will automatically be decremented to its previous value.  This
     # method is aliased as incr() for brevity.
     def increment(by=1, &block)
-      val = redis.incr(key, by).to_i
+      val = redis.incrby(key, by).to_i
       block_given? ? rewindable_block(:decrement, val, &block) : val
     end
     alias_method :incr, :increment
@@ -53,7 +54,7 @@ class Redis
     # counter will automatically be incremented to its previous value.  This
     # method is aliased as incr() for brevity.
     def decrement(by=1, &block)
-      val = redis.decr(key, by).to_i
+      val = redis.decrby(key, by).to_i
       block_given? ? rewindable_block(:increment, val, &block) : val
     end
     alias_method :decr, :decrement

data/lib/redis/helpers/core_commands.rb

@@ -15,7 +15,7 @@ class Redis
       def type
         redis.type key
       end
-      
+
       def rename(name, setkey=true)
         dest = name.is_a?(self.class) ? name.key : name
         ret  = redis.rename key, dest
@@ -41,6 +41,14 @@ class Redis
       def move(dbindex)
         redis.move key, dbindex
       end
+      
+      # See the documentation for SORT: http://code.google.com/p/redis/wiki/SortCommand
+      # TODO
+      # def sort(options)
+      #   args = []
+      #   args += ['sort']
+      #   from_redis redis.sort key
+      # end
     end
   end
 end

data/lib/redis/list.rb

@@ -12,10 +12,10 @@ class Redis
     include Redis::Helpers::Serialize
 
     attr_reader :key, :options, :redis
-    def initialize(key, redis=$redis, options={})
+    def initialize(key, *args)
       @key = key
-      @redis = redis
-      @options = options
+      @options = args.last.is_a?(Hash) ? args.pop : {}
+      @redis = args.first || $redis
     end
     
     # Works like push.  Can chain together: list << 'a' << 'b'

data/lib/redis/lock.rb

@@ -10,11 +10,12 @@ class Redis
     class LockTimeout < StandardError; end #:nodoc:
 
     attr_reader :key, :options, :redis
-    def initialize(key, redis=$redis, options={})
+    def initialize(key, *args)
       @key = key
-      @redis = redis
-      @options = options
+      @options = args.last.is_a?(Hash) ? args.pop : {}
+      @redis = args.first || $redis
       @options[:timeout] ||= 5
+      @options[:init] = false if @options[:init].nil? # default :init to false
       @redis.setnx(key, @options[:start]) unless @options[:start] == 0 || @options[:init] === false
     end
 
@@ -31,17 +32,53 @@ class Redis
     def lock(&block)
       start = Time.now
       gotit = false
+      expiration = nil
       while Time.now - start < @options[:timeout]
-        gotit = redis.setnx(key, 1)
+        expiration = generate_expiration
+        # Use the expiration as the value of the lock.
+        gotit = redis.setnx(key, expiration)
         break if gotit
+
+        # Lock is being held.  Now check to see if it's expired (if we're using
+        # lock expiration).
+        # See "Handling Deadlocks" section on http://code.google.com/p/redis/wiki/SetnxCommand
+        if !@options[:expiration].nil?
+          old_expiration = redis.get(key).to_f
+
+          if old_expiration < Time.now.to_f
+            # If it's expired, use GETSET to update it.
+            expiration = generate_expiration
+            old_expiration = redis.getset(key, expiration).to_f
+
+            # Since GETSET returns the old value of the lock, if the old expiration
+            # is still in the past, we know no one else has expired the locked
+            # and we now have it.
+            if old_expiration < Time.now.to_f
+              gotit = true
+              break
+            end
+          end
+        end
+
         sleep 0.1
       end
       raise LockTimeout, "Timeout on lock #{key} exceeded #{@options[:timeout]} sec" unless gotit
       begin
         yield
       ensure
-        redis.del(key)
+        # We need to be careful when cleaning up the lock key.  If we took a really long
+        # time for some reason, and the lock expired, someone else may have it, and
+        # it's not safe for us to remove it.  Check how much time has passed since we
+        # wrote the lock key and only delete it if it hasn't expired (or we're not using
+        # lock expiration)
+        if @options[:expiration].nil? || expiration > Time.now.to_f
+          redis.del(key)
+        end
       end
     end
+
+    def generate_expiration
+      @options[:expiration].nil? ? 1 : (Time.now + @options[:expiration].to_f + 1).to_f
+    end
   end
 end

data/lib/redis/objects/counters.rb

@@ -56,7 +56,7 @@ class Redis
         def increment_counter(name, id=nil, by=1, &block)
           verify_counter_defined!(name, id)
           initialize_counter!(name, id)
-          value = redis.incr(field_key(name, id), by).to_i
+          value = redis.incrby(field_key(name, id), by).to_i
           block_given? ? rewindable_block(:decrement_counter, name, id, value, &block) : value
         end
 
@@ -65,7 +65,7 @@ class Redis
         def decrement_counter(name, id=nil, by=1, &block)
           verify_counter_defined!(name, id)
           initialize_counter!(name, id)
-          value = redis.decr(field_key(name, id), by).to_i
+          value = redis.decrby(field_key(name, id), by).to_i
           block_given? ? rewindable_block(:increment_counter, name, id, value, &block) : value
         end
 
@@ -73,7 +73,8 @@ class Redis
         def reset_counter(name, id=nil, to=nil)
           verify_counter_defined!(name, id)
           to = @redis_objects[name][:start] if to.nil?
-          redis.set(field_key(name, id), to)
+          redis.set(field_key(name, id), to.to_i)
+          true
         end
         
         private

data/lib/redis/objects/locks.rb

@@ -35,9 +35,6 @@ class Redis
               end
             EndMethods
           end
-
-          
-          
         end
 
         # Obtain a lock, and execute the block synchronously.  Any other code
@@ -47,7 +44,7 @@ class Redis
           verify_lock_defined!(name)
           raise ArgumentError, "Missing block to #{self.name}.obtain_lock" unless block_given?
           lock_name = field_key("#{name}_lock", id)
-          Redis::Lock.new(redis, lock_name, self.class.redis_objects[name]).lock(&block)
+          Redis::Lock.new(lock_name, redis, self.redis_objects[name]).lock(&block)
         end
 
         # Clear the lock.  Use with care - usually only in an Admin page to clear

data/lib/redis/objects/sorted_sets.rb

@@ -0,0 +1,45 @@
+# This is the class loader, for use as "include Redis::Objects::Sets"
+# For the object itself, see "Redis::Set"
+require 'redis/sorted_set'
+class Redis
+  module Objects
+    module SortedSets
+      def self.included(klass)
+        klass.send :include, InstanceMethods
+        klass.extend ClassMethods
+      end
+
+      # Class methods that appear in your class when you include Redis::Objects.
+      module ClassMethods
+        # Define a new list.  It will function like a regular instance
+        # method, so it can be used alongside ActiveRecord, DataMapper, etc.
+        def sorted_set(name, options={})
+          @redis_objects[name] = options.merge(:type => :sorted_set)
+          if options[:global]
+            instance_eval <<-EndMethods
+              def #{name}
+                @#{name} ||= Redis::SortedSet.new(field_key(:#{name}, ''), redis, @redis_objects[:#{name}])
+              end
+            EndMethods
+            class_eval <<-EndMethods
+              def #{name}
+                self.class.#{name}
+              end
+            EndMethods
+          else
+            class_eval <<-EndMethods
+              def #{name}
+                @#{name} ||= Redis::SortedSet.new(field_key(:#{name}), redis, self.class.redis_objects[:#{name}])
+              end
+            EndMethods
+          end
+          
+        end
+      end
+
+      # Instance methods that appear in your class when you include Redis::Objects.
+      module InstanceMethods
+      end
+    end
+  end
+end

data/lib/redis/objects.rb

@@ -6,7 +6,7 @@ class Redis
   # Redis::Objects enables high-performance atomic operations in your app
   # by leveraging the atomic features of the Redis server.  To use Redis::Objects,
   # first include it in any class you want.  (This example uses an ActiveRecord
-  # subclass, but that is *not* required.) Then, use +counter+ and +lock+
+  # subclass, but that is *not* required.) Then, use +counter+, +lock+, +set+, etc
   # to define your primitives:
   #
   #   class Game < ActiveRecord::Base
@@ -14,8 +14,8 @@ class Redis
   #
   #     counter :joined_players
   #     counter :active_players
-  #     set :player_ids
   #     lock :archive_game
+  #     set :player_ids
   #   end
   # 
   # The, you can use these counters both for bookeeping and as atomic actions:
@@ -39,10 +39,11 @@ class Redis
     dir = File.expand_path(__FILE__.sub(/\.rb$/,''))
 
     autoload :Counters, File.join(dir, 'counters')
-    autoload :Values, File.join(dir, 'values')
     autoload :Lists, File.join(dir, 'lists')
-    autoload :Sets, File.join(dir, 'sets')
     autoload :Locks, File.join(dir, 'locks')
+    autoload :Sets, File.join(dir, 'sets')
+    autoload :SortedSets, File.join(dir, 'sorted_sets')
+    autoload :Values, File.join(dir, 'values')
 
     class NotConnected  < StandardError; end
 
@@ -61,10 +62,11 @@ class Redis
         
         # Pull in each object type
         klass.send :include, Redis::Objects::Counters
-        klass.send :include, Redis::Objects::Values
         klass.send :include, Redis::Objects::Lists
-        klass.send :include, Redis::Objects::Sets
         klass.send :include, Redis::Objects::Locks
+        klass.send :include, Redis::Objects::Sets
+        klass.send :include, Redis::Objects::SortedSets
+        klass.send :include, Redis::Objects::Values
       end
     end