resque-lock-timeout 0.3.3 → 0.4.0

data/HISTORY.md

@@ -1,7 +1,11 @@
+## 0.4.0 (2012-11-09)
+
+* Add `@loner` boolean option to prevent job being enqueued if already
+  running/enqueued. (Thanks to @ssaunier)
+
 ## 0.3.3 (2012-03-09)
-## 0.3.2 (2012-03-09)
 
-* Release changes nothing of importance, tested against v1.20.0 of resque.
+* Tested against v1.20.0 of resque.
 
 ## 0.3.1 (2011-07-16)
 
@@ -23,4 +27,4 @@
 ## 0.2.0 (2010-05-05)
 
 * Initial release as `resque-lock-timeout`, forked from Chris Wanstrath'
-`resque-lock`.
+  `resque-lock`.

data/README.md

@@ -6,7 +6,7 @@ A [Resque][rq] plugin. Requires Resque >= v1.8.0.
 resque-lock-timeout adds locking, with optional timeout/deadlock handling to
 resque jobs.
 
-Using a `lock_timeout` allows you to re-aquire the lock should your worker
+Using a `lock_timeout` allows you to re-acquire the lock should your worker
 fail, crash, or is otherwise unable to relase the lock. **i.e.** Your server
 unexpectedly looses power. Very handy for jobs that are recurring or may be
 retried.
@@ -37,7 +37,23 @@ Default behaviour...
 
 Please see below for more information about the identifer/lock key.
 
-### With Lock Expiry/Timeout
+### Enqueued Exclusivity (Loner Option)
+
+Setting the `@loner` boolean to `true` will ensure the job is not enqueued if
+the job (identified by the `identifier` method) is already running/enqueued.
+
+    class LonelyJob
+      extend Resque::Plugins::LockTimeout
+      @queue = :loners
+
+      @loner = true
+
+      def self.perform(repo_id)
+        heavy_lifting
+      end
+    end
+
+### Lock Expiry/Timeout
 
 The locking algorithm used can be found in the [Redis SETNX][redis-setnx]
 documentation.
@@ -158,11 +174,19 @@ Several callbacks are available to override and implement your own logic, e.g.
       # Lock may be held for upto an hour.
       @lock_timeout = 3600
 
+      # No same job get enqueued if one already running/enqueued
+      @loner = true
+
       # Job failed to acquire lock. You may implement retry or other logic.
       def self.lock_failed(repo_id)
         raise LockFailed
       end
 
+      # Unable to enqueue job because its running or already enqueued.
+      def self.loner_enqueue_failed(repo_id)
+        raise EnqueueFailed
+      end
+
       # Job has complete; but the lock expired before we could relase it.
       # The lock wasn't released; as its *possible* the lock is now held
       # by another job.

data/lib/resque/plugins/lock_timeout.rb

@@ -3,32 +3,51 @@ module Resque
     # If you want only one instance of your job running at a time,
     # extend it with this module:
     #
-    # require 'resque-lock-timeout'
+    #   require 'resque-lock-timeout'
     #
-    # class UpdateNetworkGraph
-    #   extend Resque::Plugins::LockTimeout
-    #   @queue = :network_graph
+    #   class UpdateNetworkGraph
+    #     extend Resque::Plugins::LockTimeout
+    #     @queue = :network_graph
     #
-    #   def self.perform(repo_id)
-    #     heavy_lifting
+    #     def self.perform(repo_id)
+    #       heavy_lifting
+    #     end
     #   end
-    # end
     #
-    # If you wish to limit the durati on a lock may be held for, you can
+    # If you wish to limit the duration a lock may be held for, you can
     # set/override `lock_timeout`. e.g.
     #
-    # class UpdateNetworkGraph
-    #   extend Resque::Plugins::LockTimeout
-    #   @queue = :network_graph
+    #   class UpdateNetworkGraph
+    #     extend Resque::Plugins::LockTimeout
+    #     @queue = :network_graph
     #
-    #   # lock may be held for upto an hour.
-    #   @lock_timeout = 3600
+    #     # lock may be held for upto an hour.
+    #     @lock_timeout = 3600
     #
-    #   def self.perform(repo_id)
-    #     heavy_lifting
+    #     def self.perform(repo_id)
+    #       heavy_lifting
+    #     end
     #   end
-    # end
     #
+    # If you wish that only one instance of the job defined by #identifier may be
+    # enqueued or running, you can set/override `loner`. e.g.
+    #
+    #   class PdfExport
+    #     extend Resque::Plugins::LockTimeout
+    #     @queue = :exports
+    #
+    #     # only one job can be running/enqueued at a time. For instance a button
+    #     # to run a PDF export. If the user clicks several times on it, enqueue
+    #     # the job if and only if
+    #     #   - the same export is not currently running
+    #     #   - the same export is not currently queued.
+    #     # ('same' being defined by `identifier`)
+    #     @loner = true
+    #
+    #     def self.perform(repo_id)
+    #       heavy_lifting
+    #     end
+    #   end
     module LockTimeout
       # @abstract You may override to implement a custom identifier,
       #           you should consider doing this if your job arguments
@@ -56,8 +75,7 @@ module Resque
       # Override to fully control the lock key used. It is passed
       # the job arguments.
       #
-      # The default looks like this:
-      # `resque-lock-timeout:<class name>:<identifier>`
+      # The default looks like this: `lock:<class name>:<identifier>`
       #
       # @param [Array] args job arguments
       # @return [String] redis key
@@ -65,6 +83,16 @@ module Resque
         ['lock', name, identifier(*args)].compact.join(':')
       end
 
+      # Builds lock key used by `@loner` option. Passed job arguments.
+      #
+      # The default looks like this: `loner:lock:<class name>:<identifier>`
+      #
+      # @param [Array] args job arguments
+      # @return [String] redis key
+      def redis_loner_lock_key(*args)
+        ['loner', redis_lock_key(*args)].compact.join(':')
+      end
+
       # Number of seconds the lock may be held for.
       # A value of 0 or below will lock without a timeout.
       #
@@ -74,6 +102,14 @@ module Resque
         @lock_timeout ||= 0
       end
 
+      # Whether one instance of the job should be running or enqueued.
+      #
+      # @param [Array] args job arguments
+      # @return [TrueClass || FalseClass]
+      def loner(*args)
+        @loner ||= false
+      end
+
       # Convenience method, not used internally.
       #
       # @return [Boolean] true if the job is locked by someone
@@ -82,12 +118,19 @@ module Resque
       end
 
       # @abstract
-      # Hook method; called when a were unable to aquire the lock.
+      # Hook method; called when a were unable to acquire the lock.
       #
       # @param [Array] args job arguments
       def lock_failed(*args)
       end
 
+      # @abstract
+      # Hook method; called when a were unable to enqueue loner job.
+      #
+      # @param [Array] args job arguments
+      def loner_enqueue_failed(*args)
+      end
+
       # @abstract
       # Hook method; called when the lock expired before we released it.
       #
@@ -95,17 +138,49 @@ module Resque
       def lock_expired_before_release(*args)
       end
 
-      # Try to acquire a lock.
+      # @abstract
+      # if the job is a `loner`, enqueue only if no other same job
+      # is already running/enqueued
       #
-      # * Returns false; when unable to acquire the lock.
+      # @param [Array] args job arguments
+      def before_enqueue_lock(*args)
+        if loner
+          if locked?(*args) 
+            # Same job is currently running
+            loner_enqueue_failed(*args)
+            false
+          else
+            acquire_loner_lock!(*args)
+          end
+        end
+      end
+
+      # Try to acquire a lock for running the job.
+      # @return [Boolean, Fixnum]
+      def acquire_lock!(*args)
+        acquire_lock_impl!(:redis_lock_key, :lock_failed, *args)
+      end
+
+      # Try to acquire a lock to enqueue a loner job.
+      # @return [Boolean, Fixnum]
+      def acquire_loner_lock!(*args)
+        acquire_lock_impl!(:redis_loner_lock_key, :loner_enqueue_failed, *args)
+      end
+
+      # Generic implementation of the locking logic
+      #
+      # Returns false; when unable to acquire the lock.
       # * Returns true; when lock acquired, without a timeout.
       # * Returns timestamp; when lock acquired with a timeout, timestamp is
       #   when the lock timeout expires.
       #
+      # @param [Symbol] lock_key_method the method returning redis key to lock
+      # @param [Symbol] failed_hook the method called if lock failed
+      # @param [Array] args job arguments
       # @return [Boolean, Fixnum]
-      def acquire_lock!(*args)
+      def acquire_lock_impl!(lock_key_method, failed_hook, *args)
         acquired = false
-        lock_key = redis_lock_key(*args)
+        lock_key = self.send lock_key_method, *args
 
         unless lock_timeout(*args) > 0
           # Acquire without using a timeout.
@@ -115,11 +190,11 @@ module Resque
           acquired, lock_until = acquire_lock_algorithm!(lock_key, *args)
         end
 
-        lock_failed(*args) if !acquired
+        self.send(failed_hook, *args) if !acquired
         lock_until && acquired ? lock_until : acquired
       end
 
-      # Attempts to aquire the lock using a timeout / deadlock algorithm.
+      # Attempts to acquire the lock using a timeout / deadlock algorithm.
       #
       # Locking algorithm: http://code.google.com/p/redis/wiki/SetnxCommand
       #
@@ -154,6 +229,13 @@ module Resque
         lock_redis.del(redis_lock_key(*args))
       end
 
+      # Release the enqueue lock for loner jobs
+      #
+      # @param [Array] args job arguments
+      def release_loner_lock!(*args)
+        lock_redis.del(redis_loner_lock_key(*args))
+      end
+
       # Refresh the lock.
       #
       # @param [Array] args job arguments
@@ -167,8 +249,13 @@ module Resque
       #
       # @param [Array] args job arguments
       def around_perform_lock(*args)
+        lock_until = acquire_lock!(*args)
+
+        # Release loner lock as job has been dequeued
+        release_loner_lock!(*args) if loner
+
         # Abort if another job holds the lock.
-        return unless lock_until = acquire_lock!(*args)
+        return unless lock_until
 
         begin
           yield

data/test/lock_test.rb

@@ -2,7 +2,7 @@ require File.dirname(__FILE__) + '/test_helper'
 
 class LockTest < MiniTest::Unit::TestCase
   def setup
-    $success = $lock_failed = $lock_expired = 0
+    $success = $lock_failed = $lock_expired = $enqueue_failed = 0
     Resque.redis.flushall
     @worker = Resque::Worker.new(:test)
   end
@@ -155,4 +155,41 @@ class LockTest < MiniTest::Unit::TestCase
     diff = latest_lock - initial_lock
     assert diff >= 1, 'diff between initial lock and refreshed lock should be at least 1 second'
   end
+
+  def test_cannot_enqueue_two_loner_jobs
+    Resque.enqueue(LonelyJob)
+    Resque.enqueue(LonelyJob)
+    assert_equal 1, Resque.size(:test), '1 job should be enqueued'
+
+    Resque.enqueue(LonelyTimeoutJob)
+    Resque.enqueue(LonelyTimeoutJob)
+    assert_equal 2, Resque.size(:test), '2 jobs should be enqueued'
+  end
+
+  def test_loner_job_should_not_be_enqued_if_already_running
+    Resque.enqueue(LonelyJob)
+    thread = Thread.new { @worker.process }
+
+    sleep 0.1 # The LonelyJob should be running (perfom is 0.2 seconds long)
+    Resque.enqueue(LonelyJob)
+    assert_equal 0, Resque.size(:test)
+    assert_equal 1, $enqueue_failed, 'One job callback should increment enqueue_failed'
+
+    thread.join
+    assert_equal 1, $success, 'One job should increment success'
+  end
+
+  def test_loner_job_with_timeout_should_not_be_enqued_if_already_running
+    Resque.enqueue(LonelyTimeoutJob)
+    thread = Thread.new { @worker.process }
+
+    sleep 0.1 # Job should be running (perfom is 0.2 seconds long)
+    Resque.enqueue(LonelyTimeoutJob)
+    assert_equal 0, Resque.size(:test)
+    assert_equal 1, $enqueue_failed, 'One job callback should increment enqueue_failed'
+
+    thread.join
+    assert_equal 1, $success, 'One job should increment success'
+  end
+
 end

data/test/redis-test.conf

@@ -6,10 +6,10 @@ daemonize yes
 
 # When run as a daemon, Redis write a pid file in /var/run/redis.pid by default.
 # You can specify a custom pid file location here.
-pidfile ./test/redis-test.pid
+#pidfile ./test/redis-test.pid
 
 # Accept connections on the specified port, default is 6379
-port 9736
+port 9737
 
 # If you want you can bind a single interface, if the bind option is not
 # specified all the interfaces will listen for connections.
@@ -30,16 +30,16 @@ timeout 300
 #   after 900 sec (15 min) if at least 1 key changed
 #   after 300 sec (5 min) if at least 10 keys changed
 #   after 60 sec if at least 10000 keys changed
-save 900 1
-save 300 10
-save 60 10000
+#save 900 1
+#save 300 10
+#save 60 10000
 
 # The filename where to dump the DB
-dbfilename dump.rdb
+#dbfilename dump.rdb
 
 # For default save/load DB in/from the working directory
 # Note that you must specify a directory not a file name.
-dir ./test/
+#dir ./test/
 
 # Set server verbosity to 'debug'
 # it can be one of:
@@ -112,4 +112,4 @@ databases 16
 # Glue small output buffers together in order to send small replies in a
 # single TCP packet. Uses a bit more CPU but most of the times it is a win
 # in terms of number of queries per second. Use 'yes' if unsure.
-glueoutputbuf yes
+# glueoutputbuf yes

data/test/test_helper.rb

@@ -5,36 +5,39 @@ $TESTING = true
 require 'rubygems'
 require 'minitest/unit'
 require 'minitest/pride'
-require 'simplecov'
 
-SimpleCov.start do
-  add_filter '/test/'
-end unless RUBY_PLATFORM == 'java'
+# Run code coverage in MRI 1.9 only.
+if RUBY_VERSION >= '1.9' && RUBY_ENGINE == 'ruby'
+  require 'simplecov'
+  SimpleCov.start do
+    add_filter '/test/'
+  end
+end
 
 require 'resque-lock-timeout'
 require dir + '/test_jobs'
 
-# make sure we can run redis
+# make sure we can run redis-server
 if !system('which redis-server')
-  puts '', "** can't find `redis-server` in your path"
-  puts '** try running `sudo rake install`'
+  puts '', "** `redis-server` was not found in your PATH"
   abort ''
 end
 
-# start our own redis when the tests start,
-# kill it when they end
+# make sure we can shutdown the server using cli.
+if !system('which redis-cli')
+  puts '', "** `redis-cli` was not found in your PATH"
+  abort ''
+end
+
+# This code is run `at_exit` to setup everything before running the tests.
+# Redis server is started before this code block runs.
 at_exit do
   next if $!
 
   exit_code = MiniTest::Unit.new.run(ARGV)
-
-  pid = `ps -e -o pid,command | grep [r]edis-test`.split(' ')[0]
-  puts 'Killing test redis server...'
-  `rm -f #{dir}/dump.rdb`
-  `kill -9 #{pid}`
-  exit(exit_code)
+  `redis-cli -p 9737 shutdown nosave`
 end
 
-puts 'Starting redis for testing at localhost:9736...'
+puts "Starting redis for testing at localhost:9737..."
 `redis-server #{dir}/redis-test.conf`
-Resque.redis = '127.0.0.1:9736'
+Resque.redis = '127.0.0.1:9737'

data/test/test_jobs.rb

@@ -112,4 +112,37 @@ class RefreshLockJob
   extend Resque::Plugins::LockTimeout
   @queue = :test
   @lock_timeout = 60
-end
+end
+
+# Job that prevents the job being enqueued if already enqueued/running.
+class LonelyJob
+  extend Resque::Plugins::LockTimeout
+  @queue = :test
+  @loner = true
+
+  def self.perform
+    $success += 1
+    sleep 0.2
+  end
+
+  def self.loner_enqueue_failed(*args)
+    $enqueue_failed += 1
+  end
+end
+
+# Exclusive job (only one queued/running) with a timeout.
+class LonelyTimeoutJob
+  extend Resque::Plugins::LockTimeout
+  @queue = :test
+  @loner = true
+  @lock_timeout = 60
+
+  def self.perform
+    $success += 1
+    sleep 0.2
+  end
+
+  def self.loner_enqueue_failed(*args)
+    $enqueue_failed += 1
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: resque-lock-timeout
 version: !ruby/object:Gem::Version
-  version: 0.3.3
+  version: 0.4.0
   prerelease: 
 platform: ruby
 authors:
@@ -11,11 +11,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-03-09 00:00:00.000000000Z
+date: 2012-11-09 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: resque
-  requirement: &2155990660 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -23,10 +23,15 @@ dependencies:
         version: 1.8.0
   type: :runtime
   prerelease: false
-  version_requirements: *2155990660
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 1.8.0
 - !ruby/object:Gem::Dependency
   name: rake
-  requirement: &2155972620 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -34,10 +39,15 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2155972620
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: minitest
-  requirement: &2155947800 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -45,10 +55,15 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2155947800
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: json
-  requirement: &2155922920 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -56,10 +71,15 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2155922920
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: yard
-  requirement: &2155919780 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -67,10 +87,15 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2155919780
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rdiscount
-  requirement: &2155916580 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -78,10 +103,15 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *2155916580
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: simplecov
-  requirement: &2155910660 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -89,9 +119,14 @@ dependencies:
         version: 0.3.0
   type: :development
   prerelease: false
-  version_requirements: *2155910660
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 0.3.0
 description: ! "  A Resque plugin. Adds locking, with optional timeout/deadlock handling
-  to\n  resque jobs.\n\n  Using a `lock_timeout` allows you to re-aquire the lock
+  to\n  resque jobs.\n\n  Using a `lock_timeout` allows you to re-acquire the lock
   should your worker\n  fail, crash, or is otherwise unable to relase the lock.\n
   \ \n  i.e. Your server unexpectedly looses power. Very handy for jobs that are\n
   \ recurring or may be retried.\n"
@@ -130,7 +165,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.10
+rubygems_version: 1.8.24
 signing_key: 
 specification_version: 3
 summary: A Resque plugin adding locking, with optional timeout/deadlock handling to