resque-locket 0.1.0

data/.gitignore

@@ -0,0 +1,18 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+vendor/bundle

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in resque-locket.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,20 @@
+The MIT License (MIT)
+
+Copyright (c) 2013 AcademicWorks, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2013 AcademicWorks
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,66 @@
+# resque-locket
+
+A Resque plugin to ensure unique workers while preventing queue starvation. While a job is being processed, duplicate jobs are locked from being simultaneously processed, and Locket intelligently avoids starvation in priority queueing situations.
+
+### Usage
+
+Here is the simplest possible Locket configuration:
+
+    # in your Gemfile
+    gem "resque-locket"
+
+    # somewhere in your application initialization:
+    Resque.unique_queues!
+    Resque.locketed_queues = ["unique", "queues"] # optional configuration, and Locket defaults to all queues
+
+Easy enough. All queues will be guaranteed unique across workers, locks will expire after 35 seconds of inactivity, while a job is working, its key will be refreshed every 30 seconds and the entire payload will be used as the lock key. In the next section, we'll introduce other options as we discuss them.
+
+### How It Works
+
+At a high level, before processing a job, Locket will attempt to obtain a **job lock** for said job. If it *can* obtain a lock, it will spawn a child thread that continually extends the lock while the job is being processed. If it *can't* obtain a lock, it will requeue the job and increment a **queue lock counter** to determine if all jobs in that queue are locked. If they are, that queue will be temporarily removed from the worker's queues list.
+
+So a lock is implemented at both the job level and the queue level—if a job is being processed, it is locked. If all jobs in a queue are being processed, the queue is locked. But whether a job is locked is the starting point for both types of lock.
+
+#### When A Job is Not Locked
+
+When Resque forks a child process, Locket will check (in the parent process) if a lock exists in Redis for the job that's about to be processed. It will find one doesn't, then it will:
+
+1. *Set an expiring lock for the current job.* This should be unique for a given job, and once it is put in place, if a workers attempts to perform an identical job, the lock will prevent that from being possible. Both the key name and the expiration are configurable, as detailed below, and it's the expiration that protects us in the event of a worker dying a quick death without calling its failure hooks.
+2. *Destroy the queue lock queue counter.* When obtaining a lock fails, a hash key is incremented that is a counter for that queue. If the number of locked jobs reaches the queue's size, the queue is no longer fit for pulling jobs from. We have to do this in the parent process (as opposed to in an `after_*` hook in the job process) because the job process could have died abruptly and left its queue's counter in place. So when work can be done, we set right the state of the world by clearing the counter.
+3. *Spawn a child thread that will continually extend the expiration of the lock key.* As shown below, this frequency is also configurable. But this serves as a heartbeat so as long as a Resque process lives and performs a job, its lock is continually extended.
+4. *Attach `after_perform` and `after_failure` [hooks](https://github.com/resque/resque/blob/master/docs/HOOKS.md) to clean up after the job.* These will destroy a job's lock and the queue lock counter upon completion of a job, which ensures a new identical job could be processed quickly and that a queue will not remain locked if a job from a locked queue completes and its queue grows before jobs from unlocked queues are performed.
+
+Knowing what we know now, we can discuss new configuration options:
+
+    # how often the spawned child thread will extend the lock key
+    Resque.heartbeat_frequency = 30
+
+    # how long the lock key will remain good for in the absence of explicit extension/deletion
+    Resque.job_lock_duration = 35
+
+    # a proc that will be passed the job to create the key that will be used as the lock for
+    # a job. the default is the entire payload, but let's say you inserted a timestamp in your
+    # payload that was the start time of a job. you'd need to exclude this from your lock, as
+    # that would result in every lock being unique, and no lock check would ever return false
+    Resque.job_lock_key = Proc.new { |job| "locket:job_locks:#{job.payload.to_s}" }
+
+#### When a Job is Locked
+
+The process is much simpler if Locket cannot obtain a lock for a job.
+
+1. *The job is requeued.* Fairly obvious, we don't want lost jobs.
+2. *A lock counter is increased for that queue.* A hash key is incremented that tracks the number of locked jobs in a queue.
+
+That lock counter works in tandem with another mechanism for determining what a worker should do next:
+
+When a worker attempts to pull a job off of the queues it is watching, it will first do a simple operation: check the queue lock counter hash, and if that queue's count is equal to or greater than its length, skip it.
+
+This prevents lower-priority queues from being starved, but it also means a queue could be locked, the job that caused the queue lock could abruptly die, and the lock would remain in place without clearing the counter hash. So any time no job can be reserved, we go ahead and and clear the queue lock counter.
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request

data/Rakefile

@@ -0,0 +1 @@
+require "bundler/gem_tasks"

data/lib/resque-locket.rb

@@ -0,0 +1 @@
+require "#{File.dirname(__FILE__)}/resque/plugins/locket"

data/lib/resque/plugins/locket.rb

@@ -0,0 +1,8 @@
+require "resque"
+
+require "resque/plugins/locket/version"
+require "resque/plugins/locket/locket"
+require "resque/plugins/locket/worker"
+
+Resque.send(:extend, Resque::Plugins::Locket)
+Resque::Worker.send(:include, Resque::Plugins::Locket::Worker)

data/lib/resque/plugins/locket/locket.rb

@@ -0,0 +1,184 @@
+module Resque
+  module Plugins
+    module Locket
+
+      # Check if a queue's jobs should be unique across workers.
+      def locketed_queue?(queue)
+        case
+        when !locket_enabled?     then false
+        when locketed_queues.nil? then true
+        else                           locketed_queues.include?(queue)
+        end
+      end
+
+      # List all locketed queues.
+      def locketed_queues
+        @locketed_queues
+      end
+
+      # Set which queues jobs should be unique across workers.
+      def locketed_queues=(queues)
+        @locketed_queues = queues
+      end
+
+      # Has resque-locket been enabled?
+      def locket_enabled?
+        @locket_enabled
+      end
+
+      # Enable locket. Set all queues to be watched, and register the after_fork hook.
+      def locket!
+        Resque.after_fork { |job| locket_or_requeue(job) } unless locket_enabled?
+
+        @locket_enabled = true
+      end
+
+      # When a queue is removed, we also need to remove its lock counters and tell locket
+      # to stop tracking it.
+      def remove_queue(queue)
+        super(queue)
+        redis.hdel("locket:queue_lock_counters", queue)
+      end
+
+      # Adjust how often the job will call to redis to extend the job lock.
+      def heartbeat_frequency=(seconds)
+        if seconds <= 0
+          raise ArgumentError, "The heartbeat frequency cannot be 0 seconds"
+        end
+
+        @heartbeat_frequency = seconds
+      end
+
+      def heartbeat_frequency
+        @heartbeat_frequency || 30
+      end
+
+      # Adjust how long the duration of the lock will be set to. The heartbeat should
+      # refresh the lock at a rate faster than its expiration.
+      def job_lock_duration=(seconds)
+        if !seconds.is_a?(Integer) || seconds <= 0
+          raise ArgumentError, "The job lock duration must be an integer greater than 0"
+        end
+
+        @job_lock_duration = seconds
+      end
+
+      def job_lock_duration
+        @job_lock_duration || 35
+      end
+
+      def job_lock_key=(job_lock_proc)
+        @job_lock_proc = job_lock_proc
+      end
+
+    private
+
+      # Check if a queue is locketed, and if so, validate the job of that queue's
+      # availability for locking.
+      def locket_or_requeue(job)
+        return unless locketed_queue?(job.queue)
+
+        obtain_job_lock(job) ? retain_job_lock(job) : requeue_job(job)
+      end
+
+      # If a lock doesn't exist for a job, set an expiring lock. If it does, we can't
+      # obtain the lock, and this will return nil.
+      def obtain_job_lock(job)
+        lock_key = job_lock_key(job)
+
+        set_expiring_key(job) unless redis.get(lock_key)
+      end
+
+      # WHEN A JOB IS LOCKED --------------------------------------------------------------------------------
+      #
+      # Requeue the locked job and increment our lock counter.
+
+      def requeue_job(job)
+        attach_before_perform_exception(job)
+        job.recreate
+        increment_queue_lock(job)
+      end
+
+      def increment_queue_lock(job)
+        redis.hincrby("locket:queue_lock_counters", job.queue, 1)
+      end
+
+      def attach_before_perform_exception(job)
+        job.payload_class.singleton_class.class_eval do
+          define_method(:before_perform_raise_exception) do |*args|
+            raise Resque::Job::DontPerform
+          end
+        end
+      end
+
+      # WHEN A JOB IS NOT LOCKED ----------------------------------------------------------------------------
+      #
+      # Clear our queue lock counters, begin a thread to start a heartbeat to redis that
+      # will hold the lock as long as we're active, and dynamically attach an
+      # after_perform hook that will manually remove the lock.
+
+      def retain_job_lock(job)
+        validate_timing
+        destroy_queue_lock_counters
+        spawn_heartbeat_thread(job)
+        attach_job_expirations(job)
+      end
+
+      def spawn_heartbeat_thread(job)
+        Thread.new do
+          loop do
+            sleep(heartbeat_frequency)
+            set_expiring_key(job)
+          end
+        end
+      end
+
+      def attach_job_expirations(job)
+        lock_key = job_lock_key(job)
+
+        job.payload_class.singleton_class.class_eval do
+          # TODO : should we use around_perform with begin/ensure/end so we expire this on failure?
+          define_method(:after_perform_remove_lock) do |*args|
+            Resque.redis.del(lock_key)
+            Resque.redis.del("locket:queue_lock_counters")
+          end
+
+          define_method(:on_failure_remove_lock) do |*args|
+            Resque.redis.del(lock_key)
+            Resque.redis.del("locket:queue_lock_counters")
+          end
+        end
+      end
+
+      def validate_timing
+        if job_lock_duration < heartbeat_frequency
+          raise "A job's heartbeat must be more frequent than its lock expiration."
+        end
+      end
+
+      def destroy_queue_lock_counters
+        redis.del("locket:queue_lock_counters")
+      end
+
+      # INDIVIDUAL JOB LOCK CONVENIENCES --------------------------------------------------------------------
+      #
+      # A couple quickies to make our life easier when dealing with setting a lock for a
+      # job that is currently being processed.
+
+      def set_expiring_key(job)
+        lock_key = job_lock_key(job)
+        redis.setex(lock_key, job_lock_duration, "")
+      end
+
+      def job_lock_key(job)
+        if @job_lock_proc
+          @job_lock_proc.call(job)
+        else
+          "locket:job_locks:#{job.payload.to_s}"
+        end
+      end
+
+    end
+
+  end
+end

data/lib/resque/plugins/locket/version.rb

@@ -0,0 +1,7 @@
+module Resque
+  module Plugins
+    module Locket
+      VERSION = "0.1.0"
+    end
+  end
+end

data/lib/resque/plugins/locket/worker.rb

@@ -0,0 +1,52 @@
+module Resque
+  module Plugins
+    module Locket
+      module Worker
+
+        def self.included(receiver)
+          receiver.class_eval do
+            alias queues_without_lock queues
+            alias queues queues_with_lock
+
+            alias reserve_and_clear_without_counter reserve
+            alias reserve reserve_and_clear_counter
+          end
+        end
+
+        # overwrite our original queues method with a new method that will check which
+        # of said queues were locked and not reserve jobs from those
+        def queues_with_lock
+          return queues_without_lock unless Resque.locket_enabled?
+
+          queues_without_lock - locked_queues
+        end
+
+        def reserve_and_clear_counter
+          return reserve_and_clear_without_counter unless Resque.locket_enabled?
+
+          job = reserve_and_clear_without_counter
+
+          redis.del("locket:queue_lock_counters") if job == nil
+
+          job
+        end
+
+      private
+
+        def locked_queues
+          locked_queues = Resque.redis.hkeys("locket:queue_lock_counters")
+
+          return [] if locked_queues.nil?
+
+          locked_queues.to_a.map do |key|
+            locked_count = Resque.redis.hget("locket:queue_lock_counters", key).to_i
+            queue_size   = Resque.size(key)
+
+            key if locked_count >= queue_size
+          end.compact
+        end
+
+      end
+    end
+  end
+end

data/resque-locket.gemspec

@@ -0,0 +1,25 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'resque/plugins/locket/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "resque-locket"
+  spec.version       = Resque::Plugins::Locket::VERSION
+  spec.authors       = ["Joshua Cody"]
+  spec.email         = ["josh@joshuacody.net"]
+  spec.summary       = "A Resque plugin to ensure unique workers while preventing queue starvation"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files`.split($/)
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "resque"
+
+  spec.add_development_dependency "bundler", "~> 1.3"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "rspec"
+  spec.add_development_dependency "pry"
+end

data/spec/resque/plugins/locket/worker_spec.rb

@@ -0,0 +1,66 @@
+require "spec_helper.rb"
+
+describe Resque::Plugins::Locket::Worker do
+
+  let(:all_queues) { %w(1 2) }
+  let(:worker){ Resque::Worker.new("*") }
+
+  before(:all) { class GoodJob; def self.perform; end; end }
+  before(:each){ all_queues.map { |queue| Resque.watch_queue(queue) }}
+  after(:each) {
+    Resque.redis.flushdb
+    Resque.instance_variable_set :@locket_enabled,      nil
+    Resque.instance_variable_set :@locketed_queues,     nil
+    Resque.instance_variable_set :@job_lock_duration,   nil
+    Resque.instance_variable_set :@job_lock_proc,       nil
+    Resque.instance_variable_set :@heartbeat_frequency, nil
+    Resque.after_fork = nil
+  }
+
+  describe "#reserve" do
+
+    context "enabled" do
+      before(:each) { Resque.locket! }
+
+      it "clears the lock counter if no job can be reserved" do
+        Resque.redis.hset("locket:queue_lock_counters", all_queues.first, 1)
+
+        worker.stub(:queues).and_return([])
+        worker.reserve
+
+        Resque.redis.exists("locket:queue_lock_counters").should be_false
+      end
+    end
+
+  end
+
+  describe "#queues" do
+
+    context "disabled" do
+      it "returns all known queues immediately" do
+        worker.should_not_receive(:locked_queues)
+        worker.queues.should eq all_queues
+      end
+    end
+
+    context "enabled" do
+      before(:each) { Resque.locket! }
+
+      it "returns all known queues if none are locked" do
+        worker.should_receive(:locked_queues).and_call_original
+        worker.queues.should eq all_queues
+      end
+
+      it "does not return a queue if its lock counter is equivalent to its size" do
+        Resque.redis.hset("locket:queue_lock_counters", all_queues.first, 2)
+
+        Resque.enqueue_to(all_queues[0], GoodJob, "stuffs")
+        Resque.enqueue_to(all_queues[0], GoodJob, "more_stuffs")
+
+        worker.queues.should eq (all_queues - [all_queues.first])
+      end
+    end
+
+  end
+
+end

data/spec/resque/plugins/locket_spec.rb

@@ -0,0 +1,275 @@
+require "spec_helper.rb"
+
+describe Resque::Plugins::Locket do
+
+  after(:each) {
+    Resque.redis.flushdb
+    Resque.instance_variable_set :@locket_enabled,      nil
+    Resque.instance_variable_set :@locketed_queues,     nil
+    Resque.instance_variable_set :@job_lock_duration,   nil
+    Resque.instance_variable_set :@job_lock_proc,       nil
+    Resque.instance_variable_set :@heartbeat_frequency, nil
+    Resque.after_fork = nil
+  }
+
+  it "passes the plugin linter" do
+    Resque::Plugin.lint(Resque::Plugins::Locket)
+  end
+
+  context "#locket_enabled?" do
+
+    it "is false by default" do
+      Resque.locket_enabled?.should be_false
+    end
+
+    it "is true when resque-locket has been manually enabled" do
+      Resque.locket!
+      Resque.locket_enabled?.should be_true
+    end
+
+  end
+
+  context "not enabled" do
+    context "#locketed_queue?" do
+      it "returns false" do
+        Resque.locketed_queue?("5").should be_false
+      end
+    end
+  end
+
+  context "enabled" do
+    let(:all_queues) { %w(1 2 3 4 5) }
+    before(:each) do
+      all_queues.map { |queue| Resque.watch_queue(queue) }
+      Resque.locket!
+    end
+
+    context "#heartbeat_frequency" do
+      it "has a default value that can be overridden" do
+        Resque.heartbeat_frequency.should_not be_nil
+        Resque.heartbeat_frequency = 30
+        Resque.heartbeat_frequency.should be 30
+      end
+
+      it "validates a heartbeat frequency is set" do
+        expect { Resque.heartbeat_frequency = 0 }.to raise_exception
+      end
+    end
+
+    context "#job_lock_duration" do
+      it "has a default value that can be overridden" do
+        Resque.job_lock_duration.should_not be_nil
+        Resque.job_lock_duration = 30
+        Resque.job_lock_duration.should be 30
+      end
+
+      it "doesn't allow a non-integer or <= 0 duration" do
+        Resque.job_lock_duration.should_not be_nil
+        expect { Resque.job_lock_duration = "fun!" }.to raise_exception
+        expect { Resque.job_lock_duration = -1 }.to raise_exception
+      end
+
+      it "validates a job lock expiration is set" do
+        expect { Resque.job_lock_duration = 0 }.to raise_exception
+        expect { Resque.job_lock_duration = 4.3 }.to raise_exception
+      end
+    end
+
+    context "#job_lock_key" do
+      it "takes a proc that will be evaluated to determine the job lock key" do
+        Resque.job_lock_key = Proc.new { |job| job.payload_class_name }
+
+        my_job = Resque::Job.new(:jobs, "class" => "GoodJob", "args" => "stuffs")
+
+        Resque.send(:job_lock_key, my_job).should eq "GoodJob"
+      end
+    end
+
+    context "#locketed_queues" do
+      it "accepts a list of queues that should be locked" do
+        locked_queues = %w(1 2 3)
+        Resque.locketed_queues = locked_queues
+        Resque.locketed_queues.should eq locked_queues
+      end
+    end
+
+    context "#locketed_queue?" do
+      it "returns true if locketed_queues were not set" do
+        Resque.locketed_queue?("5").should be_true
+      end
+
+      it "knows when a queue was manually locketed" do
+        Resque.locketed_queues = %w(1 2)
+        Resque.locketed_queue?("1").should be_true
+      end
+
+      it "knows when a queue was not manually locketed" do
+        Resque.locketed_queues = %w(1 2 3)
+        Resque.locketed_queue?("5").should be_false
+      end
+    end
+
+    context "#remove_queue" do
+      it "removes the queue" do
+        Resque.remove_queue(all_queues.first)
+
+        Resque.redis.smembers(:queues).should_not include(all_queues.first)
+      end
+
+      it "removes the locked job counter for a given queue" do
+        Resque.redis.hincrby "locket:queue_lock_counters", all_queues.first, 1
+
+        Resque.remove_queue(all_queues.first)
+
+        Resque.redis.hexists("locket:queue_lock_counters", all_queues.first).should be_false
+      end
+    end
+
+    context "#locket!" do
+      it "only registers a single after_fork hook" do
+        Resque.locket!
+        Resque.locket!
+        Resque.locket!
+        Resque.locket!
+
+        job    = Resque::Job.new(all_queues.first, {"class" => "GoodJob", "args" => "stuffs"})
+        worker = Resque::Worker.new(all_queues.first)
+
+        worker.run_hook :after_fork, job
+
+        job.after_hooks.length.should be 1
+      end
+    end
+
+    context "#after_fork" do
+
+      context "in an unlocketed queue" do
+
+        let(:job) { Resque::Job.new(:jobs, "class" => "BadJob", "args" => "stuffs") }
+        let(:worker){ Resque::Worker.new(:jobs) }
+
+        it "does not attempt to obtain a lock for a non-locketed queue" do
+          Resque.locketed_queues = %w(1 2 3)
+          Resque.should_not_receive(:obtain_job_lock)
+
+          worker.run_hook :after_fork, job
+        end
+
+      end
+
+      context "in a locketed queue" do
+
+        let(:payload) { {"class" => "GoodJob", "args" => "stuffs"} }
+        let(:job)     { Resque::Job.new(all_queues.first, payload) }
+        let(:worker)  { Resque::Worker.new(all_queues.first) }
+
+        before(:all) {
+          class GoodJob
+            @queue = "1" # TODO : hate that i have this hard-coded
+
+            def self.perform(*args); end
+          end
+        }
+
+        it "sets an expiring lock key for a job if one doesn't already exist" do
+          worker.run_hook :after_fork, job
+
+          Resque.redis.get("locket:job_locks:#{payload.to_s}").should_not be_nil
+          Resque.redis.ttl("locket:job_locks:#{payload.to_s}").should be > 0
+        end
+
+        context "with an unlocked job" do
+
+          class GoodJob; end
+
+          before(:each){ Resque.stub(:obtain_job_lock) { true }}
+
+          it "validates the job's heartbeat is shorter than its lock's expiration" do
+            Resque.job_lock_duration   = 40
+            Resque.heartbeat_frequency = 60
+
+            expect { worker.run_hook :after_fork, job }.to raise_exception
+          end
+
+          it "destroys the locked job counter" do
+            Resque.redis.hincrby "locket:queue_lock_counters", all_queues.first, 1
+            Resque.redis.exists("locket:queue_lock_counters").should be_true
+
+            worker.run_hook :after_fork, job
+
+            Resque.redis.exists("locket:queue_lock_counters").should be_false
+          end
+
+          it "spawns a thread that extends the lock repeatedly" do
+            lock_duration = 5
+
+            Resque.heartbeat_frequency = 0.01
+            Resque.job_lock_duration   = lock_duration
+
+            Resque.should_receive(:sleep).with(0.01).twice.and_call_original
+            Resque.redis.should_receive(:setex).with("locket:job_locks:#{job.payload.to_s}", lock_duration, "").twice
+
+            worker.run_hook :after_fork, job
+            sleep(0.025)
+          end
+
+          it "deletes the lock key and lock counter after the job completes" do
+            Resque.redis.setex("locket:job_locks:#{payload.to_s}", 35, "")
+            Resque.redis.hset("locket:queue_lock_counters", job.queue, 1)
+
+            job.after_hooks.each { |hook| job.payload_class.send(hook, job.args || []) }
+
+            Resque.redis.exists("locket:job_locks:#{payload.to_s}").should be_false
+            Resque.redis.exists("locket:queue_lock_counters").should be_false
+          end
+
+          it "deletes the lock key and lock counter if the job explodes" do
+            Resque.redis.setex("locket:job_locks:#{payload.to_s}", 35, "")
+            Resque.redis.hset("locket:queue_lock_counters", job.queue, 1)
+
+            job.failure_hooks.each { |hook| job.payload_class.send(hook, job.args || []) }
+
+            Resque.redis.exists("locket:job_locks:#{payload.to_s}").should be_false
+            Resque.redis.exists("locket:queue_lock_counters").should be_false
+          end
+        end
+
+        context "with a locked job" do
+
+          before(:each){ Resque.stub(:obtain_job_lock) { false }}
+
+          it "doesn't actually run the job" do
+            worker.run_hook :after_fork, job
+
+            job.perform.should be_false
+          end
+
+          it "requeues a job if it cannot obtain a look for it" do
+            worker.run_hook :after_fork, job
+
+            last_payload = Resque.decode(Resque.redis.rpop("queue:#{job.queue}"))
+
+            last_payload["args"].should eq [job.args]
+            last_payload["class"].should eq job.payload_class_name
+          end
+
+          it "increments a locked job counter" do
+            job_2 = Resque::Job.new(all_queues.first, payload)
+
+            Resque.redis.hget("locket:queue_lock_counters", job.queue).should be_nil
+
+            worker.run_hook :after_fork, job
+            Resque.redis.hget("locket:queue_lock_counters", job.queue).should eq "1"
+
+            worker.run_hook :after_fork, job_2
+            Resque.redis.hget("locket:queue_lock_counters", job.queue).should eq "2"
+          end
+        end
+
+      end
+
+    end
+
+  end
+
+end

data/spec/spec_helper.rb

@@ -0,0 +1,11 @@
+require "rubygems"
+require "bundler/setup"
+require "pry"
+
+require "resque"
+require "resque-locket"
+
+RSpec.configure do |config|
+  config.filter_run :focus => true
+  config.run_all_when_everything_filtered = true
+end

metadata

@@ -0,0 +1,144 @@
+--- !ruby/object:Gem::Specification
+name: resque-locket
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+  prerelease: 
+platform: ruby
+authors:
+- Joshua Cody
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2013-10-11 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: resque
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.3'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: pry
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: 
+email:
+- josh@joshuacody.net
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- Gemfile
+- LICENSE
+- LICENSE.txt
+- README.md
+- Rakefile
+- lib/resque-locket.rb
+- lib/resque/plugins/locket.rb
+- lib/resque/plugins/locket/locket.rb
+- lib/resque/plugins/locket/version.rb
+- lib/resque/plugins/locket/worker.rb
+- resque-locket.gemspec
+- spec/resque/plugins/locket/worker_spec.rb
+- spec/resque/plugins/locket_spec.rb
+- spec/spec_helper.rb
+homepage: 
+licenses:
+- MIT
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.23
+signing_key: 
+specification_version: 3
+summary: A Resque plugin to ensure unique workers while preventing queue starvation
+test_files:
+- spec/resque/plugins/locket/worker_spec.rb
+- spec/resque/plugins/locket_spec.rb
+- spec/spec_helper.rb