redis-objects 0.1.0

data/ATOMICITY.rdoc

@@ -0,0 +1,154 @@
+= An Atomic Rant
+
+== Brush Up Your Resume
+
+You are probably not handling atomic operations properly in your app, and 
+probably have some nasty lurking race conditions.  The worst part is these
+will get worse as your user count increases, are difficult to reproduce,
+and usually happen to your most critical pieces of code.  (And no, your
+rspec tests can't catch them either.)
+
+Let's assume you're writing an app to enable students to enroll in courses.
+You need to ensure that no more than 30 students can sign up for a given course.
+In your enrollment code, you have something like this:
+
+    @course = Course.find(1)
+    if @course.num_students < 30
+      @course.course_students.create!(:student_id => 101)
+      @course.num_students += 1
+      @course.save!
+    else
+      # course is full
+    end
+    
+You're screwed.  You now have 32 people in your 30 person class, and you have
+no idea what happened.
+
+"Well no duh," you're saying, "even the {ActiveRecord docs mention locking}[http://api.rubyonrails.org/classes/ActiveRecord/Locking/Pessimistic.html],
+so I'll just use that."
+
+    @course = Course.find(1, :lock => true)
+    if @course.num_students < 30
+      # ...
+
+Nice try, but now you've introduced other issues.  Any other piece of code
+in your entire app that needs to update _anything_ about the course - maybe
+the course name, or start date, or location - is now serialized.  If you need high
+concurrency, you're still screwed.
+
+You think, "ah-ha, the problem is having a separate counter!"
+
+    @course = Course.find(1)
+    if @course.course_students.count < 30
+      @course.course_students.create!(:student_id => 101)
+    else
+      # course is full
+    end
+
+Nope.  Still screwed.
+
+== The Root Down
+
+It's worth understanding the root issue, and how to address it.
+
+Race conditions arise from the difference in time between *evaluating* and *altering*
+a value.  In our example, we fetched the record, then checked the value, then
+changed it.  The more lines of code between those operations, and the higher your user
+count, the bigger the window of opportunity for other clients to get the data in an
+inconsistent state.
+
+Sometimes race conditions don't matter in practice, since often a user is
+only operating their own data.  This has a race condition, but is probably ok:
+
+    @post = Post.create(:user_id => @user.id, :title => "Whattup", ...)
+    @user.total_posts += 1  # update my post count
+
+But this _would_ be problematic:
+
+    @post = Post.create(:user_id => @user.id, :title => "Whattup", ...)
+    @blog.total_posts += 1  # update post count across all users
+
+As multiple users could be adding posts concurrently.
+
+In a traditional RDBMS, you can increment counters atomically (but not return them)
+by firing off an update statement that self-references the column:
+
+    update users set total_posts = total_posts + 1 where id = 372
+
+You may have seen {ActiveRecord's increment_counter class method}[http://api.rubyonrails.org/classes/ActiveRecord/Base.html#M002278],
+which wraps this functionality.  But outside of being cumbersome, this has
+the side effect that your object is no longer in sync with the DB, so you
+get other issues:
+
+    Blog.increment_counter :total_posts, @blog.id
+    if @blog.total_posts == 1000
+      # the 1000th poster - award them a gold star!
+
+The DB says 1000, but your @blog object still says 999, and the right person
+doesn't get their gold star.  Sad faces all around.
+
+== A Better Way
+
+Bottom line: Any operation that could alter a value *must* return that value in
+the _same_ _operation_ for it to be atomic.  If you do a separate get then set,
+or set then get, you're open to a race condition.  There are very few systems that
+support an "increment and return" type operation, and Redis is one of them
+(Oracle sequences are another).
+
+When you think of the specific things that you need to ensure, many of these will
+reduce to numeric operations:
+
+* Ensuring there are no more than 30 students in a course
+* Getting more than 2 but less than 6 people in a game
+* Keeping a chat room to a max of 50 people
+* Correctly recording the total number of blog posts
+* Only allowing one piece of code to reorder a large dataset at a time
+
+All except the last one can be implemented with counters.  The last one 
+will need a carefully placed lock.
+
+The best way I've found to balance atomicity and concurrency is, for each value,
+actually create two counters:
+
+* A counter you base logic on (eg, +slots_taken+)
+* A counter users see (eg, +current_students+)
+
+The reason you want two counters is you'll need to change the value of the logic
+counter *first*, _before_ checking it, to address any race conditions.  This means
+the value can get wonky momentarily (eg, there could be 32 +slots_taken+ for a 30-person
+course).  This doesn't affect its function - indeed, it's part of what makes it work - but
+does mean you don't want to display it.
+
+So, taking our +Course+ example:
+
+    class Course < ActiveRecord::Base
+      include Redis::Atoms
+      
+      counter :slots_taken
+      counter :current_students
+    end
+
+Then:
+
+    @course = Course.find(1)
+    @course.slots_taken.increment do |val|
+      if val <= @course.max_students
+        @course.course_students.create!(:student_id => 101)
+        @course.current_students.increment
+      end
+    end
+
+Race-condition free.  And, with the separate +current_students+ counter, your
+views get consistent information about the course, since it will only be
+incremented on success.  There is still a race condition where +current_students+
+could be less than the real number of +CourseStudent+ records, but since you'll be
+displaying these values in a view (after that block completes) you shouldn't see
+this manifest in real-world usage.
+
+Now you can sleep soundly, without fear of getting fired at 3am via an angry
+phone call from your boss.  (At least, not about this...)
+
+== Author
+
+Copyright (c) 2009 {Nate Wiger}[http://nate.wiger.org].  All Rights Reserved.
+Rant released under {Creative Commons}[http://creativecommons.org/licenses/by/3.0/legalcode].

data/README.rdoc

@@ -0,0 +1,144 @@
+= Redis::Objects - Lightweight, atomic object layer around redis-rb
+
+This is *not* an ORM.  People that are wrapping ORM's around Redis are missing
+the point.
+
+The killer feature of Redis that it allows you to perform atomic operations
+on _individual_ data structures, like counters, lists, and sets, that you can use
+*with* your existing ActiveRecord/DataMapper/etc models, or in classes that have
+nothing to do with an ORM or even a database. That's where this gem comes in.
+
+
+This gem originally arose out of a need for high-concurrency operations for online
+games; for a fun rant on the topic, see
+{ATOMICITY}[http://github.com/nateware/redis-objects/blob/master/ATOMICITY.rdoc],
+or scroll down to "Atomicity" in this README.
+
+There are two ways to use Redis::Objects, either as an +include+ in a class, or
+by using +new+ with the type of data structure you want to create.
+
+== Installation
+
+    gem install gemcutter
+    gem tumble
+    gem install redis-objects
+   
+=== Initialization
+
+    # If on Rails, config/initializers/redis.rb is a good place for this
+    require 'redis'
+    require 'redis/objects'
+    Redis::Objects.redis = Redis.new(:host => 127.0.0.1, :port => 6379)
+
+== Examples
+
+=== Model Class Usage
+
+    class Team < ActiveRecord::Base
+      include Redis::Objects
+      
+      counter :drafted_players
+      counter :active_players
+      counter :total_online_players, :global => true
+      list :on_base
+      set :outfielders
+      value :coach
+    end
+    
+Familiar Ruby operations Just Work:
+
+    @team = Team.find(1)
+    @team.on_base << 'player1'
+    @team.on_base << 'player2'
+    puts @team.on_base   # ['player1', 'player2']
+    @team.on_base.pop
+
+With one purposeful exception - counters cannot be set, only incremented/decremented:
+
+    @team.drafted_players.increment  # or incr
+    @team.drafted_players.decrement  # or decr
+    @team.drafted_players = 4   # exception
+    @team.drafted_players += 1  # exception
+
+=== Instance Usage
+
+=== Counters
+
+    @counter = Redis::Counter.new('counter_name')
+    @counter.increment
+    puts @counter
+    puts @counter.get  # force re-fetch
+
+=== Lists
+
+    @list = Redis::List.new('list_name')
+    @list << 'a'
+    @list << 'b'
+    puts @list
+
+== Atomicity
+
+You are probably not handling atomicity correctly in your app.  For a fun rant
+on the topic, see {ATOMICITY}[ATOMICITY.doc]
+
+Atomic counters are a good way to handle concurrency:
+
+    @team = Team.find(1)
+    if @team.drafted_players.increment <= @team.max_players
+      # do stuff
+      @team.team_players.create!(:player_id => 221)
+      @team.active_players.increment
+    else
+      # reset counter state
+      @team.drafted_players.decrement
+    end
+
+Atomic block - a cleaner way to do the above. Exceptions or return nil
+rewind counter back to previous state:
+
+    @team.drafted_players.increment do |val|
+      raise Team::TeamFullError if val > @team.max_players
+      @team.team_players.create!(:player_id => 221)
+      @team.active_players.increment
+    end
+
+Similar approach, using an if block (failure rewinds counter):
+
+    @team.drafted_players.increment do |val|
+      if val <= @team.max_players
+        @team.team_players.create!(:player_id => 221)
+        @team.active_players.increment
+      end
+    end
+
+Class methods work too - notice we override ActiveRecord counters:
+
+    Team.increment_counter :drafted_players, team_id
+    Team.decrement_counter :drafted_players, team_id, 2
+    Team.increment_counter :total_online_players  # no ID on global counter
+
+Class-level atomic block (may save a DB fetch depending on your app):
+
+    Team.increment_counter(:drafted_players, team_id) do |val|
+      TeamPitcher.create!(:team_id => team_id, :pitcher_id => 181)
+      Team.increment_counter(:active_players, team_id)
+    end
+
+Locks with Redis. On completion or exception the lock is released:
+
+    @team.reorder_lock.lock do
+      @team.reorder_all_players
+    end
+
+Class-level lock (same concept)
+
+    Team.obtain_lock(:reorder, team_id) do
+      Team.reorder_all_players(team_id)
+    end
+
+
+== Author
+
+Copyright (c) 2009 {Nate Wiger}[http://nate.wiger.org].  All Rights Reserved.
+Released under the {Artistic License}[http://www.opensource.org/licenses/artistic-license-2.0.php].
+

data/lib/redis/counter.rb

@@ -0,0 +1,105 @@
+class Redis
+  #
+  # Class representing a Redis counter.  This functions like a proxy class, in
+  # that you can say @object.counter_name to get the value and then
+  # @object.counter_name.increment to operate on it.  You can use this
+  # directly, or you can use the counter :foo class method in your
+  # class to define a counter.
+  #
+  class Counter
+    attr_reader :key, :options, :redis
+    def initialize(key, options={})
+      @key = key
+      @options = options
+      @redis = options[:redis] || $redis || Redis::Objects.redis
+      @options[:start] ||= 0
+      @options[:type]  ||= @options[:start] == 0 ? :increment : :decrement
+      @redis.setnx(key, @options[:start]) unless @options[:start] == 0 || @options[:init] === false
+    end
+
+    # Reset the counter to its starting value.  Not atomic, so use with care.
+    # Normally only useful if you're discarding all sub-records associated
+    # with a parent and starting over (for example, restarting a game and
+    # disconnecting all players).
+    def reset(to=options[:start])
+      redis.set(key, to.to_i)
+      @value = to.to_i
+    end
+
+    # Re-gets the current value of the counter.  Normally just calling the
+    # counter will lazily fetch the value, and only update it if increment
+    # or decrement is called.  This forces a network call to redis-server
+    # to get the current value.
+    def get
+      @value = redis.get(key).to_i
+    end
+
+    # Delete a counter.  Usage discouraged.  Consider +reset+ instead.
+    def delete
+      redis.del(key)
+      @value = nil
+    end
+    alias_method :del, :delete
+
+    # Returns the (possibly cached) value of the counter.  Use +get+ to
+    # force a re-get from the Redis server.
+    def value
+      @value ||= get
+    end
+
+    # Increment the counter atomically and return the new value.  If passed
+    # a block, that block will be evaluated with the new value of the counter
+    # as an argument. If the block returns nil or throws an exception, the
+    # counter will automatically be decremented to its previous value.  This
+    # method is aliased as incr() for brevity.
+    def increment(by=1, &block)
+      @value = redis.incr(key, by).to_i
+      block_given? ? rewindable_block(:decrement, @value, &block) : @value
+    end
+    alias_method :incr, :increment
+
+    # Decrement the counter atomically and return the new value.  If passed
+    # a block, that block will be evaluated with the new value of the counter
+    # as an argument. If the block returns nil or throws an exception, the
+    # counter will automatically be incremented to its previous value.  This
+    # method is aliased as incr() for brevity.
+    def decrement(by=1, &block)
+      @value = redis.decr(key, by).to_i
+      block_given? ? rewindable_block(:increment, @value, &block) : @value
+    end
+    alias_method :decr, :decrement
+
+    ##
+    # Proxy methods to help make @object.counter == 10 work
+    def to_s; value.to_s; end
+    alias_method :to_str, :to_s
+    alias_method :to_i, :value
+    def nil?; value.nil? end
+
+    # Math ops
+    %w(== < > <= >=).each do |m|
+      class_eval <<-EndOverload
+        def #{m}(x)
+          value #{m} x
+        end
+      EndOverload
+    end
+   
+    private
+
+    # Implements atomic increment/decrement blocks
+    def rewindable_block(rewind, value, &block)
+      raise ArgumentError, "Missing block to rewindable_block somehow" unless block_given?
+      ret = nil
+      begin
+        ret = yield value
+      rescue
+        send(rewind)
+        raise
+      end
+      send(rewind) if ret.nil?
+      ret
+    end
+  end
+end
+

data/lib/redis/data_types.rb

@@ -0,0 +1,84 @@
+class Redis
+  module DataTypes
+    TYPES = %w(String Integer Float EpochTime DateTime Json Yaml IPAddress FilePath Uri Slug)
+    def self.included(klass)
+      TYPES.each do |data_type|
+        if Object.const_defined?(data_type)
+          klass = Object.const_get(data_type)
+        else
+          klass = Object.const_set(data_type, Class.new)
+        end
+        if const_defined?(data_type) 
+          klass.extend const_get(data_type)
+        end
+      end
+    end
+
+    module String
+      def to_redis; to_s; end
+    end
+
+    module Integer
+      def from_redis(value); value && value.to_i end
+    end
+
+    module Float
+      def from_redis(value); value && value.to_f end
+    end
+
+    module EpochTime
+      def to_redis(value)
+        value.is_a?(DateTime) ? value.to_time.to_i : value.to_i
+      end
+    
+      def from_redis(value) Time.at(value.to_i) end
+    end
+
+    module DateTime
+      def to_redis(value); value.strftime('%FT%T%z')                        end
+      def from_redis(value); value && ::DateTime.strptime(value, '%FT%T%z') end
+    end
+
+    module Json
+      def to_redis(value); Yajl::Encoder.encode(value)          end
+      def from_redis(value); value && Yajl::Parser.parse(value) end
+    end
+  
+    module Yaml
+      def to_redis(value); Yaml.dump(value)    end
+      def from_redis(value); Yaml.load(value)  end
+    end
+  
+    module IPAddress      
+      def from_redis(value)
+        return nil if value.nil?
+        if value.is_a?(String)
+          IPAddr.new(value.empty? ? '0.0.0.0' : value)
+        else
+          raise "+value+ must be nil or a String"
+        end
+      end
+    end
+  
+    module FilePath
+      require 'pathname'
+      def from_redis(value)
+        value.blank? ? nil : Pathname.new(value)
+      end
+    end
+  
+    module Uri
+      require 'addressable/uri'
+      def from_redis(value)
+        Addressable::URI.parse(value)
+      end
+    end
+  
+    module Slug
+      require 'addressable/uri'
+      def to_redis(value)
+        Addressable::URI.parse(value).display_uri
+      end
+    end
+  end
+end