redis-semaphore 0.1.5 → 0.1.6

data/README.md

@@ -3,14 +3,14 @@ redis-semaphore
 
 Implements a mutex and semaphore using Redis and the neat BLPOP command.
 
-The mutex and semaphore is blocking, not polling, and has a fair queue serving processes on a first-come, first-serve basis.
+The mutex and semaphore is blocking, not polling, and has a fair queue serving processes on a first-come, first-serve basis. It can also have an optional timeout after which a lock is unlocked automatically, to protect against dead clients.
 
 For more info see [Wikipedia](http://en.wikipedia.org/wiki/Semaphore_(programming\)).
 
 Usage
 -----
 
-First let's see how to create a mutex:
+Create a mutex:
 
 ```ruby
 s = Redis::Semaphore.new(:semaphore_name, :connection => "localhost")
@@ -18,37 +18,37 @@ s.lock do
   # We're now in a mutex protected area
   # No matter how many processes are running this program,
   # there will be only one running this code block at a time.
-  do_something_speshiul()
+  work
 end
 ```
 
 While our application is inside the code block given to ```s.lock```, other calls to use the mutex with the same name will block until our code block is finished. Once our mutex unlocks, the next process will unblock and be able to execute the code block. The blocking processes get unblocked in order of arrival, creating a fair queue.
 
-You can also allow a set number of processes inside the semaphore-protected block:
+You can also allow a set number of processes inside the semaphore-protected block, in case you have a well-defined number of resources available:
 
 ```ruby
-s = Redis::Semaphore.new(:semaphore_name, 5, :connection => "localhost")
+s = Redis::Semaphore.new(:semaphore_name, :resources => 5, :connection => "localhost")
 s.lock do
   # Up to five processes at a time will be able to get inside this code
   # block simultaneously.
-  do_something_less_speshiul()
+  work
 end
 ```
 
-You don't need to use code blocks, you can also use linear code:
+You're not obligated to use code blocks, linear calls work just fine:
 
 ```ruby
 s = Redis::Semaphore.new(:semaphore_name, :connection => "localhost")
 s.lock
-do_something_speshiul()
-s.unlock  # Don't forget this, or the mutex will be locked forever!
+work
+s.unlock  # Don't forget this, or the mutex will stay locked!
 ```
 
-If you don't want to wait forever until the mutex or semaphore release, you can use a timeout, in seconds:
+If you don't want to wait forever until the semaphore releases, you can pass in a timeout of seconds you want to wait:
 
 ```ruby
-if s.lock(5) # This will only block for at most 5 seconds if the mutex stays locked.
-  do_something_speshiul()
+if s.lock(5) # This will only block for at most 5 seconds if the semaphore stays locked.
+  work
   s.unlock
 else
   puts "Aborted."
@@ -58,15 +58,23 @@ end
 You can check if the mutex or semaphore already exists, or how many resources are left in the semaphore:
 
 ```ruby
-puts "Someone already initialized this mutex or semaphore!" if s.exists?
-puts "There are #{s.available} resources available right now."
+puts "This semaphore does exist." if s.exists?
+puts "There are #{s.available_count} resources available right now."
+```
+
+When calling ```unlock```, the new number of available resources is returned:
+
+```ruby
+sem.lock
+sem.unlock # returns 1
+sem.available_count # also returns 1
 ```
 
 In the constructor you can pass in any arguments that you would pass to a regular Redis constructor. You can even pass in your custom Redis client:
 
 ```ruby
 r = Redis.new(:connection => "localhost", :db => 222)
-s = Redis::Semaphore.new(:another_name, r)
+s = Redis::Semaphore.new(:another_name, :redis => r)
 #...
 ```
 
@@ -83,6 +91,73 @@ end
 ```
 
 
+Staleness
+---------
+
+To allow for clients to die, and the token returned to the list, a stale-check was added. As soon as a lock is started, the time of the lock is set. If another process detects that the timeout has passed since the lock was set, it can force unlock the lock itself.
+
+There are two ways to take advantage of this. You can either define a :stale\_client\_timeout upon initialization. This will check for stale locks everytime your program wants to lock the semaphore:
+
+```ruby
+s = Redis::Semaphore(:stale_semaphore, :redis = r, :stale_client_timeout => 1000) # in ms
+```
+
+Or you could start a different thread or program that frequently checks for stale locks. This has the advantage of unblocking blocking calls to Semaphore#lock as well:
+
+```ruby
+normal_sem = Redis::Semaphore(:semaphore, :connection => "localhost")
+
+Thread.new do
+  watchdog = Redis::Semaphore(:semaphore, :connection => "localhost", :stale_client_timeout => 1000)
+  
+  while(true) do
+    watchdog.release_stale_locks!
+    sleep 1
+  end
+end
+
+normal_sem.lock
+sleep 5
+normal_sem.locked? # returns false
+
+normal_sem.lock
+normal_sem.lock(5) # will block until the watchdog releases the previous lock after 1 second
+```
+
+
+Advanced
+--------
+
+The methods ```wait``` and ```signal```, the traditional method names of a Semaphore, are also implemented. ```wait``` is aliased to lock, while ```signal``` puts the specified token back on the semaphore, or generates a unique new token and puts that back if none is passed:
+
+```ruby
+# Retrieve 2 resources
+token1 = sem.wait
+token2 = sem.wait
+
+work
+
+# Put 3 resources back
+sem.signal(token1)
+sem.signal(token2)
+sem.signal
+
+sem.available_count # returns 3
+```
+
+This can be used to create a semaphore where the process that consumes resources, and the process that generates resources, are not the same. An example is a dynamic queue system with a consumer process and a producer process:
+
+```ruby
+# Consumer process
+job = semaphore.wait
+
+# Producer process
+semaphore.signal(new_job) # Job can be any string, it will be passed unmodified to the consumer process
+```
+
+Used in this fashion, a timeout does not make sense. Using the :stale\_client\_timeout here is not recommended.
+
+
 Installation
 ------------
 
@@ -97,6 +172,10 @@ Testing
 Changelog
 ---------
 
+###0.1.6 March 31, 2013
+- Add non-ownership of tokens
+- Add stale client timeout (thanks timgaleckas!)
+
 ###0.1.5 October 1, 2012
 - Add detection of Redis::Namespace definition to avoid potential bug (thanks ruud!).
 
@@ -125,6 +204,6 @@ Contributors
 
 Thanks to these awesome fellas for their contributions:
 
-- (Rimas Silkaitis)[https://github.com/neovintage]
-- (Tim Galeckas)[https://github.com/timgaleckas]
-- (Ruurd Pels)[https://github.com/ruurd]
+- [Rimas Silkaitis](https://github.com/neovintage)
+- [Tim Galeckas](https://github.com/timgaleckas)
+- [Ruurd Pels](https://github.com/ruurd)

data/Rakefile

@@ -3,5 +3,5 @@ task :test    => :spec
 
 desc "Run specs"
 task :spec do
-  exec "rspec spec/redis_spec.rb"
+  exec "rspec spec/semaphore_spec.rb"
 end

data/lib/redis/semaphore.rb

@@ -2,88 +2,185 @@ require 'redis'
 
 class Redis
   class Semaphore
-
-    attr_reader :resources
-
-    # RedisSempahore.new(:my_semaphore, 5, myRedis)
-    # RedisSemaphore.new(:my_semaphore, myRedis)
-    # RedisSemaphore.new(:my_semaphore, :connection => "", :port => "")
-    # RedisSemaphore.new(:my_semaphore, :path => "bla")
-    def initialize(*args)
-      raise "Need at least two arguments" if args.size < 2
-
-      @locked = false
-      @name = args.shift.to_s
-      @redis = args.pop
-      if !(@redis.is_a?(Redis) || (defined?(Redis::Namespace) && @redis.is_a?(Redis::Namespace)))
-        @redis = Redis.new(@redis)
-      end
-      @resources = args.pop || 1
-
+    API_VERSION = "1"
+
+    #stale_client_timeout is the threshold of time before we assume
+    #that something has gone terribly wrong with a client and we
+    #invalidate it's lock.
+    # Default is nil for which we don't check for stale clients
+    # Redis::Semaphore.new(:my_semaphore, :stale_client_timeout => 30, :redis => myRedis)
+    # Redis::Semaphore.new(:my_semaphore, :redis => myRedis)
+    # Redis::Semaphore.new(:my_semaphore, :resources => 1, :redis => myRedis)
+    # Redis::Semaphore.new(:my_semaphore, :connection => "", :port => "")
+    # Redis::Semaphore.new(:my_semaphore, :path => "bla")
+    def initialize(name, opts = {})
+      @name = name
+      @resource_count = opts.delete(:resources) || 1
+      @stale_client_timeout = opts.delete(:stale_client_timeout)
+      @redis = opts.delete(:redis) || Redis.new(opts)
+      @tokens = []
     end
 
-    def available
-      @redis.llen(list_name)
+    def exists_or_create!
+      token = @redis.getset(exists_key, API_VERSION)
+
+      if token.nil?
+        create!
+      elsif token != API_VERSION
+        raise "Semaphore exists but running as wrong version (version #{version} vs #{API_VERSION})."
+      else
+        true
+      end
     end
 
-    def exists?
-      @redis.exists(exists_name)
+    def available_count
+      @redis.llen(available_key)
     end
 
     def delete!
-      @redis.del(list_name)
-      @redis.del(exists_name)
+      @redis.del(available_key)
+      @redis.del(grabbed_key)
+      @redis.del(exists_key)
     end
 
     def lock(timeout = 0)
       exists_or_create!
+      release_stale_locks! if check_staleness?
 
+      token_pair = @redis.blpop(available_key, timeout)
+      return false if token_pair.nil?
 
-      resource_index = @redis.blpop(list_name, timeout)
-      return false if resource_index.nil?
-
-      @locked = resource_index[1].to_i
+      current_token = token_pair[1]
+      @tokens.push(current_token)
+      @redis.hset(grabbed_key, current_token, Time.now.to_i)
+      
       if block_given?
         begin
-          yield @locked
+          yield current_token
         ensure
-          unlock
+          signal(current_token)
         end
       end
 
-      true
+      current_token
     end
+    alias_method :wait, :lock
 
     def unlock
       return false unless locked?
+      signal(@tokens.pop)[1]
+    end
 
-      @redis.lpush(list_name, @locked)
-      @locked = false
+    def locked?(token = nil)
+      if token
+        @redis.hexists(grabbed_key, token)
+      else
+        @tokens.each do |token|
+          return true if locked?(token)
+        end
+        
+        false
+      end
+    end
+
+    def signal(token = 1)
+      token ||= generate_unique_token
+
+      @redis.multi do
+        @redis.hdel grabbed_key, token
+        @redis.lpush available_key, token
+      end
     end
 
-    def locked?
-      !!@locked
+    def exists?
+      @redis.exists(exists_key)
+    end
+
+    def all_tokens
+      @redis.multi do
+        @redis.lrange(available_key, 0, -1)
+        @redis.lrange(grabbed_key, 0, -1)
+      end.flatten
     end
 
+    def generate_unique_token
+      tokens = all_tokens
+      token = Random.rand.to_s
+
+      while(tokens.include? token)
+        token = Random.rand.to_s
+      end
+    end
 
   private
-    def list_name
-      (defined?(Redis::Namespace) && @redis.is_a?(Redis::Namespace)) ? "#{@name}:LIST" : "SEMAPHORE:#{@name}:LIST"
+    def simple_mutex(key_name, expires = nil)
+      key_name = namespaced_key(key_name) if key_name.kind_of? Symbol
+      token = @redis.getset(key_name, API_VERSION)
+
+      return false unless token.nil?
+      @redis.expire(key_name, expires) unless expires.nil?
+
+      begin
+        yield token
+      ensure
+        @redis.del(key_name)
+      end
     end
 
-    def exists_name
-      (defined?(Redis::Namespace) && @redis.is_a?(Redis::Namespace)) ? "#{@name}:EXISTS" : "SEMAPHORE:#{@name}:EXISTS"
+    def release_stale_locks!
+      simple_mutex(:release_locks, 10) do
+        @redis.hgetall(grabbed_key).each do |token, locked_at|
+          timed_out_at = locked_at.to_i + @stale_client_timeout
+
+          if timed_out_at < Time.now.to_i
+            signal(token)
+          end
+        end
+      end
     end
 
-    def exists_or_create!
-      exists = @redis.getset(exists_name, 1)
+    def create!
+      @redis.expire(exists_key, 10)
 
-      if "1" != exists
-        @resources.times do |index|
-          @redis.rpush(list_name, index)
+      @redis.multi do
+        @redis.del(grabbed_key)
+        @redis.del(available_key)
+        @resource_count.times do |index|
+          @redis.rpush(available_key, index)
         end
+
+        # Persist key
+        @redis.del(exists_key)
+        @redis.set(exists_key, API_VERSION)
       end
     end
 
+    def check_staleness?
+      !@stale_client_timeout.nil?
+    end
+
+    def redis_namespace?
+      (defined?(Redis::Namespace) && @redis.is_a?(Redis::Namespace))
+    end
+
+    def namespaced_key(variable)
+      if redis_namespace?
+        "#{@name}:#{variable}"
+      else
+        "SEMAPHORE:#{@name}:#{variable}"
+      end
+    end
+
+    def available_key
+      @available_key ||= namespaced_key('AVAILABLE')
+    end
+
+    def exists_key
+      @exists_key ||= namespaced_key('EXISTS')
+    end
+
+    def grabbed_key
+      @grabbed_key ||= namespaced_key('GRABBED')
+    end
   end
 end

data/spec/semaphore_spec.rb

@@ -0,0 +1,137 @@
+require File.dirname(__FILE__) + '/spec_helper'
+
+describe "redis" do
+  before(:all) do
+    # use database 15 for testing so we dont accidentally step on you real data
+    @redis = Redis.new :db => 15
+  end
+
+  before(:each) do
+    @redis.flushdb
+  end
+
+  after(:all) do
+    @redis.quit
+  end
+
+  shared_examples_for "a semaphore" do
+
+    it "has the correct amount of available resources" do
+      semaphore.lock
+      semaphore.unlock.should == 1
+      semaphore.available_count.should == 1
+    end
+
+    it "should not exist from the start" do
+      semaphore.exists?.should == false
+      semaphore.lock
+      semaphore.exists?.should == true
+    end
+
+    it "should be unlocked from the start" do
+      semaphore.locked?.should == false
+    end
+
+    it "should lock and unlock" do
+      semaphore.lock(1)
+      semaphore.locked?.should == true
+      semaphore.unlock
+      semaphore.locked?.should == false
+    end
+
+    it "should not lock twice as a mutex" do
+      semaphore.lock(1).should_not == false
+      semaphore.lock(1).should == false
+    end
+
+    it "should not lock three times when only two available" do
+      multisem.lock(1).should_not == false
+      multisem.lock(1).should_not == false
+      multisem.lock(1).should == false
+    end
+
+    it "should always have the correct lock-status" do
+      multisem.lock(1)
+      multisem.lock(1)
+
+      multisem.locked?.should == true
+      multisem.unlock
+      multisem.locked?.should == true
+      multisem.unlock
+      multisem.locked?.should == false
+    end
+
+    it "should get all different tokens when saturating" do
+      ids = []
+      2.times do 
+        ids << multisem.lock(1)
+      end
+
+      ids.should == %w(0 1)
+    end
+
+    it "should execute the given code block" do
+      code_executed = false
+      semaphore.lock(1) do
+        code_executed = true
+      end
+      code_executed.should == true
+    end
+
+    it "should pass an exception right through" do
+      lambda do
+        semaphore.lock(1) do
+          raise Exception, "redis semaphore exception"
+        end
+      end.should raise_error(Exception, "redis semaphore exception")
+    end
+
+    it "should not leave the semaphore locked after raising an exception" do
+      lambda do
+        semaphore.lock(1) do
+          raise Exception
+        end
+      end.should raise_error
+
+      semaphore.locked?.should == false
+    end
+  end
+
+  describe "semaphore without staleness checking" do
+    let(:semaphore) { Redis::Semaphore.new(:my_semaphore, :redis => @redis) }
+    let(:multisem) { Redis::Semaphore.new(:my_semaphore_2, :resources => 2, :redis => @redis) }
+
+    it_behaves_like "a semaphore"
+
+    it "can dynamically add resources" do
+      semaphore.exists_or_create!
+
+      3.times do
+        semaphore.signal
+      end
+
+      semaphore.available_count.should == 4
+
+      semaphore.wait(1)
+      semaphore.wait(1)
+      semaphore.wait(1)
+
+      semaphore.available_count.should == 1
+    end
+  end
+
+  describe "semaphore with staleness checking" do
+    let(:semaphore) { Redis::Semaphore.new(:my_semaphore, :redis => @redis, :stale_client_timeout => 5) }
+    let(:multisem) { Redis::Semaphore.new(:my_semaphore_2, :resources => 2, :redis => @redis, :stale_client_timeout => 5) }
+
+    it_behaves_like "a semaphore"
+
+    it "should restore resources of stale clients" do
+      hyper_aggressive_sem = Redis::Semaphore.new(:hyper_aggressive_sem, :resources => 1, :redis => @redis, :stale_client_timeout => 1)
+
+      hyper_aggressive_sem.lock(1).should_not == false
+      hyper_aggressive_sem.lock(1).should == false
+      hyper_aggressive_sem.lock(1).should_not == false
+    end
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: redis-semaphore
 version: !ruby/object:Gem::Version
-  version: 0.1.5
+  version: 0.1.6
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-10-01 00:00:00.000000000 Z
+date: 2013-03-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: redis
@@ -40,7 +40,7 @@ files:
 - LICENSE
 - lib/redis/semaphore.rb
 - lib/redis-semaphore.rb
-- spec/redis_spec.rb
+- spec/semaphore_spec.rb
 - spec/spec_helper.rb
 homepage: http://github.com/dv/redis-semaphore
 licenses: []

data/spec/redis_spec.rb

@@ -1,106 +0,0 @@
-require File.dirname(__FILE__) + '/spec_helper'
-
-describe "redis" do
-  before(:all) do
-    # use database 15 for testing so we dont accidentally step on you real data
-    @redis = Redis.new :db => 15
-    @semaphore = Redis::Semaphore.new(:my_semaphore, @redis)
-  end
-
-  before(:each) do
-    @redis.flushdb
-  end
-
-  after(:each) do
-    @redis.flushdb
-  end
-
-  after(:all) do
-    @redis.quit
-  end
-
-  it "should be unlocked from the start" do
-    @semaphore.locked?.should == false
-  end
-
-  it "should lock and unlock" do
-    @semaphore.lock
-    @semaphore.locked?.should == true
-    @semaphore.unlock
-    @semaphore.locked?.should == false
-  end
-
-  it "should not lock twice as a mutex" do
-    @semaphore.lock
-    @semaphore.lock(1).should == false
-  end
-
-  it "should not lock three times when only two available" do
-    multisem = Redis::Semaphore.new(:my_semaphore2, 2, @redis)
-    multisem.lock.should == true
-    multisem.lock(1).should == true
-    multisem.lock(1).should == false
-  end
-
-  it "should reuse the same index for 5 calls in serial" do
-    multisem = Redis::Semaphore.new(:my_semaphore5_serial, 5, @redis)
-    ids = []
-    5.times do
-      multisem.lock(1) do |i|
-        ids << i
-      end
-    end
-    ids.size.should == 5
-    ids.uniq.size.should == 1
-  end
-
-  it "should have 5 different indexes for 5 parallel calls" do
-    multisem = Redis::Semaphore.new(:my_semaphore5_parallel, 5, @redis)
-    ids = []
-    multisem.lock(1) do |i|
-      ids << i
-      multisem.lock(1) do |i|
-        ids << i
-        multisem.lock(1) do |i|
-          ids << i
-          multisem.lock(1) do |i|
-            ids << i
-            multisem.lock(1) do |i|
-              ids << i
-              multisem.lock(1) do |i|
-                ids << i
-              end.should == false
-            end
-          end
-        end
-      end
-    end
-    (0..4).to_a.should == ids
-  end
-
-  it "should execute the given code block" do
-    code_executed = false
-    @semaphore.lock do
-      code_executed = true
-    end
-    code_executed.should == true
-  end
-
-  it "should pass an exception right through" do
-    lambda do
-      @semaphore.lock do
-        raise Exception, "redis semaphore exception"
-      end
-    end.should raise_error(Exception, "redis semaphore exception")
-  end
-
-  it "should not leave the semaphore locked after raising an exception" do
-    lambda do
-      @semaphore.lock do
-        raise Exception
-      end
-    end.should raise_error
-
-    @semaphore.locked?.should == false
-  end
-end