redis-objects 0.2.2 → 0.3.0

data/CHANGELOG.rdoc

@@ -0,0 +1,37 @@
+= Changelog for Redis::Objects
+
+== 0.3.0 [Final] (14 April 2010)
+
+* Due to Ruby 1.9 bugs and performance considerations, marshaling of
+data types is now OFF by default.  You must say :marshal => true for
+any objects that you want serialization enabled on. [Nate Wiger]
+
+* Sorted Set class changed slightly due to feedback. You can now get
+an individual element back via @set['item'] since it acts like a Hash.
+
+== 0.2.4 [Final] (9 April 2010)*
+
+* Added sorted set support via Redis::SortedSet [Nate Wiger]
+
+== 0.2.3 [Final] (18 February 2010)*
+
+* Added lock expiration to Redis::Lock [Ben VandenBos]
+
+* Fixed some bugs [Ben VandenBos]
+
+* Added lock tests and test helpers [Ben VandenBos]
+
+== 0.2.2 [Final] (14 December 2009)*
+
+* Added @set.diff(@set2) with "^" and "-" synonyms (oversight). [Nate Wiger]
+
+* Implemented Redis core commands in all data types, such as rename. [Nate Wiger]
+
+* Renamed Redis::Serialize to Redis::Helpers::Serialize to keep Redis:: cleaner. [Nate Wiger]
+
+* More spec coverage. [Nate Wiger]
+
+== 0.2.1 [Final] (27 November 2009)*
+
+* First worthwhile public release, with good spec coverage and functionality. [Nate Wiger]
+

data/README.rdoc

@@ -1,44 +1,136 @@
 = Redis::Objects - Map Redis types directly to Ruby objects
 
-This is *not* an ORM.  People that are wrapping ORM's around Redis are missing
-the point.
+This is *not* an ORM. People that are wrapping ORM’s around Redis are missing the point.
 
-The killer feature of Redis that it allows you to perform atomic operations
-on _individual_ data structures, like counters, lists, and sets. You can then use
-these *with* your existing ActiveRecord/DataMapper/etc models, or in classes that have
-nothing to do with an ORM or even a database.  This gem maps {Redis types}[http://code.google.com/p/redis/wiki/CommandReference]
-to Ruby objects, via a thin layer over Ezra's +redis+ gem.
+The killer feature of Redis is that it allows you to perform _atomic_ operations
+on _individual_ data structures, like counters, lists, and sets.  The *atomic* part is HUGE.
+Using an ORM wrapper that retrieves a "record", updates values, then sends those values back,
+_removes_ the atomicity, cutting the nuts off the major advantage of Redis.  Just use MySQL, k?
+
+This gem provides a Rubyish interface to Redis, by mapping {Redis types}[http://code.google.com/p/redis/wiki/CommandReference]
+to Ruby objects, via a thin layer over Ezra's +redis+ gem.  It offers several advantages
+over the lower-level redis-rb API:
+
+1. Easy to integrate directly with existing ORMs - ActiveRecord, DataMapper, etc.  Add counters to your model!
+2. Complex data structures are automatically Marshaled (if you set :marshal => true)
+3. Integers are returned as integers, rather than '17'
+4. Higher-level types are provided, such as Locks, that wrap multiple calls
 
 This gem originally arose out of a need for high-concurrency atomic operations;
-for a fun rant on the topic, see
-{ATOMICITY}[http://github.com/nateware/redis-objects/blob/master/ATOMICITY.rdoc],
+for a fun rant on the topic, see {An Atomic Rant}[http://nateware.com/2010/02/18/an-atomic-rant],
 or scroll down to "Atomicity" in this README.
 
-There are two ways to use Redis::Objects, either as an +include+ in a model class,
-or by using +new+ with the type of data structure you want to create.
+There are two ways to use Redis::Objects, either as an include in a model class (to
+integrate with ORMs or other classes), or by using new with the type of data structure
+you want to create. 
 
 == Installation
 
-    gem install gemcutter
-    gem tumble
     gem install redis-objects
 
-== Example 1: Standalone Usage
+== Example 1: Model Class Usage
+
+Using Redis::Objects this way makes it trivial to integrate Redis types with an
+existing ActiveRecord model, DataMapper resource, or other class.  Redis::Objects
+will work with _any_ class that provides an +id+ method that returns a unique
+value.  Redis::Objects will automatically create keys that are unique to
+each object, in the format:
+
+    model_name:id:field_name
+
+=== Initialization
+
+Redis::Objects needs a handle created by Redis.new.  (If you're on Rails,
+config/initializers/redis.rb is a good place for this.)
+
+    require 'redis'
+    require 'redis/objects'
+    Redis::Objects.redis = Redis.new(:host => 127.0.0.1, :port => 6379)
+
+Remember you can use Redis::Objects in any Ruby code.  There are *no* dependencies
+on Rails.  Standalone, Sinatra, Resque - no problem.
+
+=== Model Class
+
+You can include Redis::Objects in any type of class:
+
+    class Team < ActiveRecord::Base
+      include Redis::Objects
+
+      lock :trade_players, :expiration => 15  # sec
+      counter :hits
+      counter :runs
+      counter :outs
+      counter :inning, :start => 1
+      list :on_base
+      set :outfielders
+      value :at_bat
+    end
+
+Familiar Ruby array operations Just Work (TM):
+
+    @team = Team.find_by_name('New York Yankees')
+    @team.on_base << 'player1'
+    @team.on_base << 'player2'
+    @team.on_base << 'player3'
+    @team.on_base    # ['player1', 'player2', 'player3']
+    @team.on_base.pop
+    @team.on_base.shift
+    @team.on_base.length  # 1
+    @team.on_base.delete('player2')
+
+Sets work too:
+
+    @team.outfielders << 'outfielder1'
+    @team.outfielders << 'outfielder2'
+    @team.outfielders << 'outfielder1'   # dup ignored
+    @team.outfielders  # ['outfielder1', 'outfielder2']
+    @team.outfielders.each do |player|
+      puts player
+    end
+    player = @team.outfielders.detect{|of| of == 'outfielder2'}
+
+And you can do intersections between objects (kinda cool):
+
+    @team1.outfielders | @team2.outfielders   # outfielders on both teams
+    @team1.outfielders & @team2.outfielders   # in baseball, should be empty :-)
+
+Counters can be atomically incremented/decremented (but not assigned):
+
+    @team.hits.increment  # or incr
+    @team.hits.decrement  # or decr
+    @team.hits.incr(3)    # add 3
+    @team.runs = 4        # exception
+
+Finally, for free, you get a +redis+ method that points directly to a Redis connection:
+
+    Team.redis.get('somekey')
+    @team = Team.new
+    @team.redis.get('somekey')
+    @team.redis.smembers('someset')
+
+You can use the +redis+ handle to directly call any {Redis API command}[http://code.google.com/p/redis/wiki/CommandReference].
+
+== Example 2: Standalone Usage
 
-There is a Ruby object that maps to each Redis type.
+There is a Ruby class that maps to each Redis type, with methods for each
+{Redis API command}[http://code.google.com/p/redis/wiki/CommandReference].
+Note that calling +new+ does not imply it's actually a "new" value - it just
+creates a mapping between that object and the corresponding Redis data structure,
+which may already exist on the redis-server.
 
 === Initialization
 
-This gem needs a handle to the +redis+ server.  For standalone use, you can
-either set the $redis global variable to your Redis.new handle:
+Redis::Objects needs a handle to the +redis+ server.  For standalone use, you
+can either set the $redis global variable:
 
     $redis = Redis.new(:host => 'localhost', :port => 6379)
-    @value = Redis::Value.new('myvalue')
+    @list  = Redis::List.new('mylist')
 
-Or you can pass the Redis handle into the new method:
+Or you can pass the Redis handle into the new method for each type:
 
     redis  = Redis.new(:host => 'localhost', :port => 6379)
-    @value = Redis::Value.new('myvalue', redis)
+    @list  = Redis::List.new('mylist', redis)
 
 === Counters
 
@@ -58,6 +150,34 @@ This gem provides a clean way to do atomic blocks as well:
 
 See the section on "Atomicity" for cool uses of atomic counter blocks.
 
+=== Locks
+
+A convenience class that wraps the pattern of {using +setnx+ to perform locking}[http://code.google.com/p/redis/wiki/SetnxCommand].
+
+    require 'redis/lock'
+    @lock = Redis::Lock.new('image_resizing', :expiration => 15, :timeout => 0.1)
+    @lock.lock do
+      # do work
+    end
+
+This can be especially useful if you're running batch jobs spread across multiple hosts.
+
+=== Values
+
+Simple values are easy as well:
+
+    require 'redis/value'
+    @value = Redis::Value.new('value_name')
+    @value.value = 'a'
+    @value.delete
+
+Complex data is no problem with :marshal => true:
+
+    @account = Account.create!(params[:account])
+    @newest  = Redis::Value.new('newest_account', :marshal => true)
+    @newest.value = @account.attributes
+    puts @newest.value['username']
+
 === Lists
 
 Lists work just like Ruby arrays:
@@ -78,14 +198,15 @@ Lists work just like Ruby arrays:
     @list.clear
     # etc
 
-Complex data types are no problem:
+Complex data types are no problem with :marshal => true:
 
+    @list = Redis::List.new('list_name', :marshal => true)
     @list << {:name => "Nate", :city => "San Diego"}
     @list << {:name => "Peter", :city => "Oceanside"}
     @list.each do |el|
       puts "#{el[:name]} lives in #{el[:city]}"
     end
-
+    
 === Sets
 
 Sets work like the Ruby {Set}[http://ruby-doc.org/core/classes/Set.html] class:
@@ -111,8 +232,8 @@ You can perform Redis intersections/unions/diffs easily:
     members = @set1 & @set2   # intersection
     members = @set1 | @set2   # union
     members = @set1 + @set2   # union
-    members = @set1 ^ @set2   # union
-    members = @set1 - @set2   # union
+    members = @set1 ^ @set2   # difference
+    members = @set1 - @set2   # difference
     members = @set1.intersection(@set2, @set3)  # multiple
     members = @set1.union(@set2, @set3)         # multiple
     members = @set1.difference(@set2, @set3)    # multiple
@@ -126,8 +247,10 @@ Or store them in Redis:
     @set1.diffstore('diffname', @set2, @set3)
     members = @set1.redis.get('diffname')
 
-And use complex data types too:
+And use complex data types too, with :marshal => true:
 
+    @set1 = Redis::Set.new('set1', :marshal => true)
+    @set2 = Redis::Set.new('set2', :marshal => true)
     @set1 << {:name => "Nate", :city => "San Diego"}
     @set1 << {:name => "Peter", :city => "Oceanside"}
     @set2 << {:name => "Nate", :city => "San Diego"}
@@ -137,100 +260,50 @@ And use complex data types too:
     @set1 - @set2  # Peter
     @set1 | @set2  # all 3 people
 
-=== Values
-
-Simple values are easy as well:
-
-    require 'redis/value'
-    @value = Redis::Value.new('value_name')
-    @value.value = 'a'
-    @value.delete
-
-Of course complex data is no problem:
-
-    @account = Account.create!(params[:account])
-    @newest  = Redis::Value.new('newest_account')
-    @newest.value = @account
-
-== Example 2: Model Class Usage
-
-Using Redis::Objects this way makes it trivial to integrate Redis types with an
-existing ActiveRecord model, DataMapper resource, or other class.  Redis::Objects
-will work with _any_ class that provides an +id+ method that returns a unique
-value.  Redis::Objects will automatically create keys that are unique to
-each object.
-
-=== Initialization
-
-If on Rails, config/initializers/redis.rb is a good place for this:
-
-    require 'redis'
-    require 'redis/objects'
-    Redis::Objects.redis = Redis.new(:host => 127.0.0.1, :port => 6379)
-
-=== Model Class
-
-Include Redis::Objects in any type of class:
-
-    class Team < ActiveRecord::Base
-      include Redis::Objects
-      
-      counter :hits
-      counter :runs
-      counter :outs
-      counter :inning, :start => 1
-      list :on_base
-      set :outfielders
-      value :at_bat
-    end
+=== Sorted Sets
 
-Familiar Ruby array operations Just Work (TM):
+Due to their unique properties, Sorted Sets work like a hybrid between
+a Hash and an Array.  You assign like a Hash, but retrieve like an Array:
 
-    @team = Team.find_by_name('New York Yankees')
-    @team.on_base << 'player1'
-    @team.on_base << 'player2'
-    @team.on_base << 'player3'
-    @team.on_base    # ['player1', 'player2']
-    @team.on_base.pop
-    @team.on_base.shift
-    @team.on_base.length  # 1
-    @team.on_base.delete('player3')
+    require 'redis/sorted_set'
+    @sorted_set = Redis::SortedSet.new('number_of_posts')
+    @sorted_set['Nate']  = 15
+    @sorted_set['Peter'] = 75
+    @sorted_set['Jeff']  = 24
 
-Sets work too:
-    
-    @team.outfielders << 'outfielder1' << 'outfielder1'
-    @team.outfielders << 'outfielder2'
-    @team.outfielders  # ['outfielder1', 'outfielder2']
-    @team.outfielders.each do |player|
-      puts player
-    end
-    player = @team.outfielders.detect{|of| of == 'outfielder2'}
+    # Array access to get sorted order
+    @sorted_set[0..2]           # => ["Nate", "Jeff", "Peter"]
+    @sorted_set[0,2]            # => ["Nate", "Jeff"]
 
-And you can do intersections between ORM objects (kinda cool):
+    @sorted_set['Peter']        # => 75
+    @sorted_set['Jeff']         # => 24
+    @sorted_set.score('Jeff')   # same thing (24)
 
-    @team1.outfielders | @team2.outfielders   # all outfielders
-    @team1.outfielders & @team2.outfielders   # should be empty
+    @sorted_set.rank('Peter')   # => 2
+    @sorted_set.rank('Jeff')    # => 1
 
-Counters can be atomically incremented/decremented (but not assigned):
+    @sorted_set.first           # => "Nate"
+    @sorted_set.last            # => "Peter"
+    @sorted_set.revrange(0,2)   # => ["Peter", "Jeff", "Nate"]
 
-    @team.hits.increment  # or incr
-    @team.hits.decrement  # or decr
-    @team.hits.incr(3)    # add 3
-    @team.runs = 4        # exception
+    @sorted_set['Newbie'] = 1
+    @sorted_set.members         # => ["Newbie", "Nate", "Jeff", "Peter"]
 
-Finally, for free, you get a +redis+ handle usable in your class that
-points directly to a Redis API object:
+    @sorted_set.rangebyscore(10, 100, :limit => 2)   # => ["Nate", "Jeff"]
+    @sorted_set.members(:withscores => true)         # => [["Newbie", 1], ["Nate", 16], ["Jeff", 28], ["Peter", 76]]
 
-    @team.redis.get('somekey')
-    @team.redis.smembers('someset')
+    # atomic increment
+    @sorted_set.increment('Nate')
+    @sorted_set.incr('Peter')   # shorthand
+    @sorted_set.incr('Jeff', 4)
 
-You can use the +redis+ handle to directly call any {Redis command}[http://code.google.com/p/redis/wiki/CommandReference]
+The other Redis Sorted Set commands are supported as well; see {Sorted Sets API}[http://code.google.com/p/redis/wiki/SortedSets].
 
 == Atomic Counters and Locks
 
 You are probably not handling atomicity correctly in your app.  For a fun rant
 on the topic, see
-{ATOMICITY}[http://github.com/nateware/redis-objects/blob/master/ATOMICITY.rdoc].
+{An Atomic Rant}[http://nateware.com/2010/02/18/an-atomic-rant].
 
 Atomic counters are a good way to handle concurrency:
 
@@ -277,6 +350,10 @@ Class-level atomic block (may save a DB fetch depending on your app):
 
 Locks with Redis. On completion or exception the lock is released:
 
+    class Team < ActiveRecord::Base
+      lock :reorder # declare a lock
+    end
+
     @team.reorder_lock.lock do
       @team.reorder_all_players
     end
@@ -287,9 +364,19 @@ Class-level lock (same concept)
       Team.reorder_all_players(team_id)
     end
 
+Lock expiration.  Sometimes you want to make sure your locks are cleaned up should
+the unthinkable happen (server failure).  You can set lock expirations to handle
+this.  Expired locks are released by the next process to attempt lock.  Just
+make sure you expiration value is sufficiently large compared to your expected
+lock time.
+
+    class Team < ActiveRecord::Base
+      lock :reorder, :expiration => 15.minutes
+    end
+
 
 == Author
 
-Copyright (c) 2009 {Nate Wiger}[http://nate.wiger.org].  All Rights Reserved.
+Copyright (c) 2009-2010 {Nate Wiger}[http://nate.wiger.org].  All Rights Reserved.
 Released under the {Artistic License}[http://www.opensource.org/licenses/artistic-license-2.0.php].
 

data/lib/redis/counter.rb

@@ -11,10 +11,10 @@ class Redis
     include Redis::Helpers::CoreCommands
 
     attr_reader :key, :options, :redis
-    def initialize(key, redis=$redis, options={})
+    def initialize(key, *args)
       @key = key
-      @redis = redis
-      @options = options
+      @options = args.last.is_a?(Hash) ? args.pop : {}
+      @redis = args.first || $redis
       @options[:start] ||= 0
       @redis.setnx(key, @options[:start]) unless @options[:start] == 0 || @options[:init] === false
     end
@@ -25,6 +25,7 @@ class Redis
     # disconnecting all players).
     def reset(to=options[:start])
       redis.set key, to.to_i
+      true  # hack for redis-rb regression
     end
 
     # Returns the current value of the counter.  Normally just calling the
@@ -42,7 +43,7 @@ class Redis
     # counter will automatically be decremented to its previous value.  This
     # method is aliased as incr() for brevity.
     def increment(by=1, &block)
-      val = redis.incr(key, by).to_i
+      val = redis.incrby(key, by).to_i
       block_given? ? rewindable_block(:decrement, val, &block) : val
     end
     alias_method :incr, :increment
@@ -53,7 +54,7 @@ class Redis
     # counter will automatically be incremented to its previous value.  This
     # method is aliased as incr() for brevity.
     def decrement(by=1, &block)
-      val = redis.decr(key, by).to_i
+      val = redis.decrby(key, by).to_i
       block_given? ? rewindable_block(:increment, val, &block) : val
     end
     alias_method :decr, :decrement

data/lib/redis/helpers/core_commands.rb

@@ -15,7 +15,7 @@ class Redis
       def type
         redis.type key
       end
-      
+
       def rename(name, setkey=true)
         dest = name.is_a?(self.class) ? name.key : name
         ret  = redis.rename key, dest
@@ -41,6 +41,14 @@ class Redis
       def move(dbindex)
         redis.move key, dbindex
       end
+      
+      # See the documentation for SORT: http://code.google.com/p/redis/wiki/SortCommand
+      # TODO
+      # def sort(options)
+      #   args = []
+      #   args += ['sort']
+      #   from_redis redis.sort key
+      # end
     end
   end
 end

data/lib/redis/helpers/serialize.rb

@@ -4,6 +4,7 @@ class Redis
       include Marshal
 
       def to_redis(value)
+        return value unless options[:marshal]
         case value
         when String, Fixnum, Bignum, Float
           value
@@ -13,6 +14,7 @@ class Redis
       end
     
       def from_redis(value)
+        return value unless options[:marshal]
         case value
         when Array
           value.collect{|v| from_redis(v)}

data/lib/redis/list.rb

@@ -12,10 +12,10 @@ class Redis
     include Redis::Helpers::Serialize
 
     attr_reader :key, :options, :redis
-    def initialize(key, redis=$redis, options={})
+    def initialize(key, *args)
       @key = key
-      @redis = redis
-      @options = options
+      @options = args.last.is_a?(Hash) ? args.pop : {}
+      @redis = args.first || $redis
     end
     
     # Works like push.  Can chain together: list << 'a' << 'b'
@@ -62,7 +62,7 @@ class Redis
         at(index)
       end
     end
-    
+
     # Delete the element(s) from the list that match name. If count is specified,
     # only the first-N (if positive) or last-N (if negative) will be removed.
     # Use .del to completely delete the entire key.

data/lib/redis/lock.rb

@@ -10,11 +10,12 @@ class Redis
     class LockTimeout < StandardError; end #:nodoc:
 
     attr_reader :key, :options, :redis
-    def initialize(key, redis=$redis, options={})
+    def initialize(key, *args)
       @key = key
-      @redis = redis
-      @options = options
+      @options = args.last.is_a?(Hash) ? args.pop : {}
+      @redis = args.first || $redis
       @options[:timeout] ||= 5
+      @options[:init] = false if @options[:init].nil? # default :init to false
       @redis.setnx(key, @options[:start]) unless @options[:start] == 0 || @options[:init] === false
     end
 
@@ -31,17 +32,53 @@ class Redis
     def lock(&block)
       start = Time.now
       gotit = false
+      expiration = nil
       while Time.now - start < @options[:timeout]
-        gotit = redis.setnx(key, 1)
+        expiration = generate_expiration
+        # Use the expiration as the value of the lock.
+        gotit = redis.setnx(key, expiration)
         break if gotit
+
+        # Lock is being held.  Now check to see if it's expired (if we're using
+        # lock expiration).
+        # See "Handling Deadlocks" section on http://code.google.com/p/redis/wiki/SetnxCommand
+        if !@options[:expiration].nil?
+          old_expiration = redis.get(key).to_f
+
+          if old_expiration < Time.now.to_f
+            # If it's expired, use GETSET to update it.
+            expiration = generate_expiration
+            old_expiration = redis.getset(key, expiration).to_f
+
+            # Since GETSET returns the old value of the lock, if the old expiration
+            # is still in the past, we know no one else has expired the locked
+            # and we now have it.
+            if old_expiration < Time.now.to_f
+              gotit = true
+              break
+            end
+          end
+        end
+
         sleep 0.1
       end
       raise LockTimeout, "Timeout on lock #{key} exceeded #{@options[:timeout]} sec" unless gotit
       begin
         yield
       ensure
-        redis.del(key)
+        # We need to be careful when cleaning up the lock key.  If we took a really long
+        # time for some reason, and the lock expired, someone else may have it, and
+        # it's not safe for us to remove it.  Check how much time has passed since we
+        # wrote the lock key and only delete it if it hasn't expired (or we're not using
+        # lock expiration)
+        if @options[:expiration].nil? || expiration > Time.now.to_f
+          redis.del(key)
+        end
       end
     end
+
+    def generate_expiration
+      @options[:expiration].nil? ? 1 : (Time.now + @options[:expiration].to_f + 1).to_f
+    end
   end
 end

data/lib/redis/objects/counters.rb

@@ -56,7 +56,7 @@ class Redis
         def increment_counter(name, id=nil, by=1, &block)
           verify_counter_defined!(name, id)
           initialize_counter!(name, id)
-          value = redis.incr(field_key(name, id), by).to_i
+          value = redis.incrby(field_key(name, id), by).to_i
           block_given? ? rewindable_block(:decrement_counter, name, id, value, &block) : value
         end
 
@@ -65,7 +65,7 @@ class Redis
         def decrement_counter(name, id=nil, by=1, &block)
           verify_counter_defined!(name, id)
           initialize_counter!(name, id)
-          value = redis.decr(field_key(name, id), by).to_i
+          value = redis.decrby(field_key(name, id), by).to_i
           block_given? ? rewindable_block(:increment_counter, name, id, value, &block) : value
         end
 
@@ -73,7 +73,8 @@ class Redis
         def reset_counter(name, id=nil, to=nil)
           verify_counter_defined!(name, id)
           to = @redis_objects[name][:start] if to.nil?
-          redis.set(field_key(name, id), to)
+          redis.set(field_key(name, id), to.to_i)
+          true
         end
         
         private

data/lib/redis/objects/locks.rb

@@ -35,9 +35,6 @@ class Redis
               end
             EndMethods
           end
-
-          
-          
         end
 
         # Obtain a lock, and execute the block synchronously.  Any other code
@@ -47,7 +44,7 @@ class Redis
           verify_lock_defined!(name)
           raise ArgumentError, "Missing block to #{self.name}.obtain_lock" unless block_given?
           lock_name = field_key("#{name}_lock", id)
-          Redis::Lock.new(redis, lock_name, self.class.redis_objects[name]).lock(&block)
+          Redis::Lock.new(lock_name, redis, self.redis_objects[name]).lock(&block)
         end
 
         # Clear the lock.  Use with care - usually only in an Admin page to clear