resque-workers-lock 1.6 → 1.7

data/README.md

@@ -12,48 +12,74 @@ If resque jobs have the same lock applied this means that those jobs cannot be p
 By default the lock is the instance name + arguments (just like the classic resque-lock). Override this lock to lock on specific arguments.
 
 ## How does it differ from resque-lock?
-Resque-lock will not let you queue jobs when you locked them. Resque-workers-lock locks on a workers-level and will requeue the locked jobs. When the enqueue_lock is set to false, this plugin will not prevent you to queue jobs. If a worker takes on a job that is already being processed by another worker it will put the job back up in the queue! Also, this plugin will still offer you the ability to lock at enqueue-level just like resque-lock does (see example).
+Resque-lock will not let you enqueue jobs when you locked them. Resque-workers-lock locks on a workers-level and will requeue the locked jobs. If a worker takes on a job that is already being processed by another worker it will put the job back up in the queue!
 
 ## Example
 This example shows how you can use the workers-lock to prevent two jobs with the same domain to be processed simultaneously.
+
 ``` ruby
 require 'resque/plugins/workers/lock'
 
 class Parser
   extend Resque::Plugins::Workers::Lock
 
-	# Lock method has the same arguments as the self.perform
-	def self.lock_workers(domain, arg2, arg3)
-		return domain
-	end
-	
-	# Turn off standard resque-lock functionality
-	def self.lock_enqueue(domain, arg2, arg3)
-		false
-	end
-
-	# Perform method with some arguments
+  # Lock method has the same arguments as the self.perform
+  def self.lock_workers(domain, arg2, arg3)
+    return domain
+  end
+
+  # This is the time in seconds that the worker lock should be considered valid.
+  # The default is one hour (3600 seconds).
+  def self.worker_lock_timeout(domain, arg2, arg3)
+    3600
+  end
+
+  # Perform method with some arguments
   def self.perform(domain, arg2, arg3)
     # do the work
   end
 end
 ```
+
 In this example `domain` is used to specify certain types of jobs that are not allowed to run at the same time. For example: if you create three jobs with the domain argument google.com, google.com and yahoo.com, the two google.com jobs will never run at the same time.
 
 ## One queue
 Best results with one big queue instead of multiple queues.
 
 ## Requeue loop
-When a job is requeue'ed there is a small delay (1 second by default) before the worker places the job actually back in the queue. Let's say you have two jobs left, and one job is taking 15 seconds on the first worker and the other similar job is being blocked by the second worker. The second worker will continuously try to put the job back in the queue and it will try to process it again (racing for 15 seconds untill the other job has finished). This only happens when there are no other (not locked) jobs in the queue.
+When a job is requeued there is a small delay (1 second by default) before the worker places the job back in the queue. Let's say you have two jobs left, and one job is taking 15 seconds on the first worker and the other similar job is being blocked by the second worker. The second worker will continuously try to put the job back in the queue and it will try to process it again (racing for 15 seconds untill the other job has finished). This only happens when there are no other (not locked) jobs in the queue.
 
 To overwrite this delay in your class:
 ``` ruby
 def self.requeue_perform_delay
-	5.0
+  5.0
 end
 ```
 
 Please note that setting this value to 5 seconds will keep the worker idle for 5 seconds when the job is locked.
 
-## Possibilities to prevent the loop 
+## Possibilities to prevent the loop
 Do a delayed resque (re)queue. However, this will have approximately the same results and will require a large extra chunk of code and rake configurations.
+
+## Run workers for the test
+To run the tests using `rake test` properly, make sure there are a few workers running:
+```
+$ redis-server
+$ VVERBOSE=1 COUNT=4 QUEUE=* rake resque:work
+```
+
+```
+➜  resque-workers-lock git:(master) ✗ rake test
+Run options: 
+
+# Running tests:
+
+...
+
+Finished tests in 10.426519s, 0.2877 tests/s, 0.3836 assertions/s.
+
+3 tests, 4 assertions, 0 failures, 0 errors, 0 skips
+```
+
+## Authors/Contributors
+[nicholaides](https://github.com/nicholaides)

data/Rakefile

@@ -8,4 +8,8 @@ Rake::TestTask.new do |t|
   t.libs << 'lib'
   t.pattern = 'test/**/*_test.rb'
   t.verbose = false
+end
+
+task "resque:setup" do
+  require_relative "test/unique_job"
 end

data/lib/resque/plugins/workers/lock.rb

@@ -4,9 +4,7 @@ module Resque
   alias_method :orig_remove_queue, :remove_queue
 
   def remove_queue(queue)
-    Resque.redis.keys('enqueuelock:*').collect { |x| Resque.redis.del(x) }.count
-    Resque.redis.keys('workerslock:*').collect { |x| Resque.redis.del(x) }.count
-
+    Resque.redis.keys('workerslock:*').each{ |x| Resque.redis.del(x) }
     orig_remove_queue(queue)
   end
 
@@ -14,13 +12,11 @@ module Resque
     module Workers
       module Lock
 
-        # Override in your job to control the queue lock key
-        def lock_enqueue(*args)
-          "#{name}-#{args.to_s}"
-        end
-
-        def get_lock_enqueue(*args)
-          "enqueuelock:"+lock_enqueue(*args).to_s
+        # Override in your job to control the worker lock experiation time. This
+        # is the time in seconds that the lock should be considered valid. The
+        # default is one hour (3600 seconds).
+        def worker_lock_timeout(*)
+          3600
         end
 
         # Override in your job to control the workers lock key.
@@ -37,50 +33,34 @@ module Resque
           1.0
         end
 
-
-        # Called with the job args before a job is placed on the queue.
-        # If the hook returns false, the job will not be placed on the queue.
-        def before_enqueue_lock(*args)
-          if lock_enqueue(*args) == false
-            return true
-          else
-            return Resque.redis.setnx(get_lock_enqueue(*args), true)
-          end
-        end
-
         # Called with the job args before perform.
         # If it raises Resque::Job::DontPerform, the job is aborted.
-        def before_perform_lock(*args)
+        def before_perform_workers_lock(*args)
           if lock_workers(*args)
-            nx = Resque.redis.setnx(get_lock_workers(*args), true)
-            if nx == false
+            if Resque.redis.setnx(get_lock_workers(*args), true)
+              Resque.redis.expire(get_lock_workers(*args), worker_lock_timeout(*args))
+            else
               sleep(requeue_perform_delay)
-              Resque.redis.del(get_lock_enqueue(*args))
               Resque.enqueue(self, *args)
               raise Resque::Job::DontPerform
             end
           end
         end
 
-        def after_dequeue_lock(*args)
-          # Clear the lock when dequeueed
-          Resque.redis.del(get_lock_enqueue(*args))
+        def clear_workers_lock(*args)
+          Resque.redis.del(get_lock_workers(*args))
         end
 
-        def around_perform_lock(*args)
-          begin
-            yield
-          ensure
-            # Clear the lock. (even with errors)
-            Resque.redis.del(get_lock_workers(*args))
-            Resque.redis.del(get_lock_enqueue(*args))
-          end
+        def around_perform_workers_lock(*args)
+          yield
+        ensure
+          # Clear the lock. (even with errors)
+          clear_workers_lock(*args)
         end
 
-        def on_failure_lock(exception, *args)
+        def on_failure_workers_lock(exception, *args)
           # Clear the lock on DirtyExit
-          Resque.redis.del(get_lock_workers(*args))
-          Resque.redis.del(get_lock_enqueue(*args))
+          clear_workers_lock(*args)
         end
 
       end

data/test/lock_test.rb

@@ -1,24 +1,15 @@
 require 'test/unit'
-require 'resque/plugins/workers/lock'
+require File.expand_path('../../lib/resque/plugins/workers/lock', __FILE__)
 require 'tempfile'
+require 'timeout'
 
-class LockTest < Test::Unit::TestCase
-  class UniqueJob
-    extend Resque::Plugins::Workers::Lock
-    @queue = :lock_test
+require_relative 'unique_job'
 
-    def self.lock_workers(*)
-      self.name
-    end
+class LockTest < Test::Unit::TestCase
+  
 
-    def self.perform params
-      File.open(params['output_file'], 'a') do |output_file|
-        output_file.puts params['job']
-        output_file.flush
-        sleep 1
-        output_file.puts params['job']
-      end
-    end
+  def setup
+    Resque.redis.del(UniqueJob.get_lock_workers)
   end
 
   def test_lint
@@ -31,8 +22,49 @@ class LockTest < Test::Unit::TestCase
     assert_locking_works_with jobs: 2, workers: 2
   end
 
+  def test_worker_locks_timeout
+    output_file = Tempfile.new 'output_file'
+
+    Resque.enqueue UniqueJob, job: 'interrupted-job', output_file: output_file.path, sleep: 1000
+
+    worker_pid = start_worker
+    wait_until(10){ lock_has_been_acquired }
+    kill_worker(worker_pid)
+
+    Resque.enqueue UniqueJob, job: 'completing-job', output_file: output_file.path, sleep: 0
+    process_jobs workers: 1, timeout: UniqueJob.worker_lock_timeout + 2
+
+    lines = File.readlines(output_file).map(&:chomp)
+    assert_equal ['starting interrupted-job', 'starting completing-job', 'finished completing-job'], lines
+  end
+
   private
 
+  def lock_has_been_acquired
+    Resque.redis.exists(UniqueJob.get_lock_workers)
+  end
+
+  def kill_worker(worker_pid)
+    Process.kill("TERM", worker_pid)
+    Process.waitpid(worker_pid)
+  end
+
+  def start_worker
+    fork.tap do |pid|
+      if !pid
+        worker = Resque::Worker.new('*')
+        worker.term_child = true
+        worker.reconnect
+        worker.work(0.5)
+        exit!
+      end
+    end
+  end
+
+  def assert_worker_lock_exists(job_class, *args)
+    assert Resque.redis.exists(job_class.get_lock_workers(*args), "lock does not exist")
+  end
+
   def assert_locking_works_with options
     jobs = (1..options[:jobs]).map{|job| "Job #{job}" }
     output_file = Tempfile.new 'output_file'
@@ -45,10 +77,8 @@ class LockTest < Test::Unit::TestCase
 
     lines = File.readlines(output_file).map(&:chomp)
     lines.each_slice(2) do |a,b|
-      assert_equal a,b, "#{a} was interrupted by #{b}"
+      assert_equal a.split.last,b.split.last, "#{a} was interrupted by #{b}"
     end
-  ensure
-    output_file.close
   end
 
   def process_jobs options
@@ -67,6 +97,7 @@ class LockTest < Test::Unit::TestCase
       else
         pids = [] # Don't kill from child's ensure
         worker = Resque::Worker.new('*')
+        worker.term_child = true
         worker.reconnect
         worker.work(0.5)
         exit!
@@ -94,11 +125,11 @@ class LockTest < Test::Unit::TestCase
   end
 
   def wait_until(timeout)
-    timeout.times do
-      return if yield
-      sleep 1
+    Timeout::timeout(timeout) do
+      loop do
+        return if yield
+        sleep 1
+      end
     end
-
-    raise "Timout occured"
   end
 end

data/test/unique_job.rb

@@ -0,0 +1,26 @@
+require File.expand_path('../../lib/resque/plugins/workers/lock', __FILE__)
+
+class UniqueJob
+  extend Resque::Plugins::Workers::Lock
+  @queue = :lock_test
+
+  def self.worker_lock_timeout(*)
+    5
+  end
+
+  def self.lock_workers(*)
+    self.name
+  end
+
+  def self.append_output filename, string
+    File.open(filename, 'a') do |output_file|
+      output_file.puts string
+    end
+  end
+
+  def self.perform params
+    append_output params['output_file'], "starting #{params['job']}"
+    sleep(params['sleep'] || 1)
+    append_output params['output_file'], "finished #{params['job']}"
+  end
+end

metadata

@@ -1,15 +1,16 @@
 --- !ruby/object:Gem::Specification
 name: resque-workers-lock
 version: !ruby/object:Gem::Version
-  version: '1.6'
+  version: '1.7'
   prerelease: 
 platform: ruby
 authors:
 - Bart Olsthoorn
+- Mike Nicholaides
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-11-26 00:00:00.000000000 Z
+date: 2012-12-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: resque
@@ -59,6 +60,7 @@ files:
 - LICENSE
 - lib/resque/plugins/workers/lock.rb
 - test/lock_test.rb
+- test/unique_job.rb
 homepage: http://github.com/bartolsthoorn/resque-workers-lock
 licenses: []
 post_install_message: