ryansch-resque-loner 1.0.1.1

data/README.markdown

@@ -0,0 +1,110 @@
+Resque-Loner
+======
+
+Resque-Loner is a plugin for defunkt/resque which adds unique jobs to resque: Only one job with the same payload per queue.
+
+
+Installation
+-------------
+
+First install the gem:
+
+    $ gem install resque-loner 
+
+Then include it in your app:
+
+    require 'resque-loner'
+
+
+Tests
+-----------
+To make sure this plugin works on your installation, you should run the tests. resque-loner is tested in RSpec, but it also includes resque's original testsuite. You can run all tests specific to resque-loner with `rake spec`.
+
+To make sure the plugin did not break resque, you can run `rake test` (the standard resque test suite). This runs all tests from the 1.10.0 version of resque, so make sure you have that version of resque installed, when you run the resque-tests.
+
+Example
+--------
+
+Unique jobs can be useful in situations where running the same job multiple times issues the same results. Let's say you have a job called CacheSweeper that refreshes some cache. A user has edited some_article, so you put a job on the queue to refresh the cache for that article.
+
+    >> Resque.enqueue CacheSweeper, some_article.id
+    => "OK"
+
+Your queue is really full, so the job does not get executed right away. But the user editing the article has noticed another error, and updates the article again, and your app kindly queues another job to update that article's cache.
+
+    >> Resque.enqueue CacheSweeper, some_article.id
+    => "OK"
+
+At this point you will have two jobs in the queue, the second of which has no effect: You don't have to run it, once the cache has been updated for the first time. This is where resque-loner's UniqueJobs come in. If you define CacheSweeper like this:
+
+    class CacheSweeper
+      include Resque::Plugins::UniqueJob
+      @queue = :cache_sweeps
+
+      def self.perform(article_id)
+        # Cache Me If You Can...
+      end
+    end
+
+Just like that you've assured that on the :cache_sweeps queue, there can only be one CacheSweeper job for each article. Let's see what happens when you try to enqueue a couple of these jobs now:
+
+    >> Resque.enqueue CacheSweeper, 1
+    => "OK"
+    >> Resque.enqueue CacheSweeper, 1
+    => "EXISTED"
+    >> Resque.enqueue CacheSweeper, 1
+    => "EXISTED"
+    >> Resque.size :cache_sweeps
+    => 1
+
+Since resque-loner keeps track of which jobs are queued in a way that allows for finding jobs very quickly, you can also query if a job is currently in a queue:
+
+    >> Resque.enqueue CacheSweeper, 1
+    => "OK"
+    >> Resque.enqueued? CacheSweeper, 1
+    => true
+    >> Resque.enqueued? CacheSweeper, 2
+    => false
+    >> Resque.enqueued_in? :another_queue, CacheSweeper, 1
+    => false
+
+If you want the same type of job in different queues, resque-loner lets you enqueue/dequeue to a queue of your choice:
+
+    >> Resque.enqueue_to :another_queue, CacheSweeper, 1
+    => "OK"
+    >> Resqueue.dequeue_from :another_queue, CacheSweeper, 1
+    => 1
+
+How it works
+--------
+
+### Keeping track of queued unique jobs
+
+For each created UniqueJob, resque-loner sets a redis key to 1. This key remains set until the job has either been fetched from the queue or destroyed through the Resque::Job.destroy method. As long as the key is set, the job is considered queued and consequent queue adds are being rejected.
+
+Here's how these keys are constructed:
+
+    resque:loners:queue:cache_sweeps:job:5ac5a005253450606aa9bc3b3d52ea5b
+    |          |        |                |
+    |          |        |                `---- Job's ID (#redis_key method)
+    |          |        `--------------------- Name of the queue
+    |          `------------------------------ Prefix for this plugin
+    `----------------------------------------- Your redis namespace
+
+The last part of this key is the job's ID, which is pretty much your queue item's payload. For our CacheSweeper job, the payload would be:
+
+    { 'class': 'CacheSweeper', 'args': [1] }`
+
+The default method to create a job ID from these parameters  is to do some normalization on the payload and then md5'ing it (defined in `Resque::Plugins::UniqueJob#redis_key`).
+
+You could also use the whole payload or anything else as a redis key, as long as you make sure these requirements are met:
+
+1. Two jobs of the same class with the same parameters/arguments/workload must produce the same redis_key
+2. Two jobs with either a different class or different parameters/arguments/workloads must not produce the same redis key 
+3. The key must not be binary, because this restriction applies to redis keys: *Keys are not binary safe strings in Redis, but just strings not containing a space or a newline character. For instance "foo" or "123456789" or "foo_bar" are valid keys, while "hello world" or "hello\n" are not.* (see http://code.google.com/p/redis/wiki/IntroductionToRedisDataTypes)
+
+So when your job overwrites the #redis_key method, make sure these requirements are met. And all should be good.
+
+### Resque integration
+
+Unfortunately not everything could be done as a plugin, so I overwrote three methods of Resque::Job: create, reserve and destroy (I found no hooks for these events). All the logic is in `module Resque::Plugins::Loner` though, so it should be fairly easy to make this a *pure* plugin once the hooks are known.

data/lib/resque-ext/job.rb

@@ -0,0 +1,55 @@
+#
+#  Since there were not enough hooks to hook into, I have to overwrite
+#  3 methods of Resque::Job - the rest of the implementation is in the
+#  proper Plugin namespace.
+# 
+module Resque
+  class Job
+
+
+    #
+    #  Overwriting original create method to mark an item as queued
+    #  after Resque::Job.create has called Resque.push
+    #
+    def self.create_with_loner(queue, klass, *args)
+      return create_without_loner(queue, klass, *args) if Resque.inline?
+      item = { :class => klass.to_s, :args => args }
+      return "EXISTED" if Resque::Plugins::Loner::Helpers.loner_queued?(queue, item)
+      job = create_without_loner(queue, klass, *args)
+      Resque::Plugins::Loner::Helpers.mark_loner_as_queued(queue, item)
+      job
+    end
+
+    #
+    #  Overwriting original reserve method to mark an item as unqueued
+    #
+    def self.reserve_with_loner(queue)
+      item = reserve_without_loner(queue)
+      Resque::Plugins::Loner::Helpers.mark_loner_as_unqueued( queue, item ) if item && !Resque.inline?
+      item
+    end
+
+    #
+    #  Overwriting original destroy method to mark all destroyed jobs as unqueued.
+    #  Because the original method only returns the amount of jobs destroyed, but not 
+    #  the jobs themselves. Hence Resque::Plugins::Loner::Helpers.job_destroy looks almost
+    #  as the original method Resque::Job.destroy. Couldn't make it any dry'er.
+    #
+    def self.destroy_with_loner(queue, klass, *args)
+      Resque::Plugins::Loner::Helpers.job_destroy(queue, klass, *args) unless Resque.inline?
+      destroy_without_loner(queue, klass, *args)
+    end
+
+    #
+    # Chain..
+    #
+    class << self
+      alias_method :create_without_loner, :create
+      alias_method :create, :create_with_loner
+      alias_method :reserve_without_loner, :reserve
+      alias_method :reserve, :reserve_with_loner
+      alias_method :destroy_without_loner, :destroy
+      alias_method :destroy, :destroy_with_loner
+    end
+  end
+end

data/lib/resque-ext/resque.rb

@@ -0,0 +1,26 @@
+module Resque
+
+  def self.enqueued?( klass, *args)
+    enqueued_in?(queue_from_class(klass), klass, *args )
+  end
+
+  def self.enqueued_in?(queue, klass, *args)
+    item = { :class => klass.to_s, :args => args }
+    return nil unless Resque::Plugins::Loner::Helpers.item_is_a_unique_job?(item)
+    Resque::Plugins::Loner::Helpers.loner_queued?(queue, item)
+  end
+
+  def self.remove_queue_with_loner_cleanup(queue)
+    self.remove_queue_without_loner_cleanup(queue)
+    Resque::Plugins::Loner::Helpers.cleanup_loners(queue)
+  end
+
+
+  class << self
+
+    alias_method :remove_queue_without_loner_cleanup, :remove_queue
+    alias_method :remove_queue, :remove_queue_with_loner_cleanup
+
+  end
+
+end

data/lib/resque-loner.rb

@@ -0,0 +1,6 @@
+require 'rubygems'
+require 'resque'
+require 'resque-loner/unique_job'
+require 'resque-loner/helpers'
+require 'resque-ext/job'
+require 'resque-ext/resque'

data/lib/resque-loner/helpers.rb

@@ -0,0 +1,61 @@
+module Resque
+  module Plugins
+    module Loner
+      class Helpers
+        extend Resque::Helpers
+
+        def self.loner_queued?(queue, item)
+          return false unless item_is_a_unique_job?(item)
+          redis.get(unique_job_queue_key(queue, item)) == "1"
+        end
+
+        def self.mark_loner_as_queued(queue, item)
+          return unless item_is_a_unique_job?(item)
+          redis.set(unique_job_queue_key(queue, item), 1)
+        end
+
+        def self.mark_loner_as_unqueued(queue, job)
+          item = job.is_a?(Resque::Job) ? job.payload : job
+          return unless item_is_a_unique_job?(item)
+          redis.del(unique_job_queue_key(queue, item))
+        end
+
+        def self.unique_job_queue_key(queue, item)
+          job_key = constantize(item[:class] || item["class"]).redis_key(item)
+          "loners:queue:#{queue}:job:#{job_key}"
+        end
+
+        def self.item_is_a_unique_job?(item)
+          begin
+            klass = constantize(item[:class] || item["class"])
+            klass.included_modules.include?(::Resque::Plugins::UniqueJob)
+          rescue
+            false # Resque testsuite also submits strings as job classes while Resque.enqueue'ing,
+          end     # so resque-loner should not start throwing up when that happens.
+        end
+
+        def self.job_destroy(queue, klass, *args)
+          klass = klass.to_s
+          redis_queue = "queue:#{queue}"
+
+          redis.lrange(redis_queue, 0, -1).each do |string|
+            json   = decode(string)
+
+            match  = json['class'] == klass
+            match &= json['args'] == args unless args.empty?
+
+            if match
+             Resque::Plugins::Loner::Helpers.mark_loner_as_unqueued( queue, json )
+            end
+          end
+        end
+
+        def self.cleanup_loners(queue)
+          keys = redis.keys("loners:queue:#{queue}:job:*")
+          redis.del(*keys) unless keys.empty?
+        end
+
+      end
+    end
+  end
+end

data/lib/resque-loner/unique_job.rb

@@ -0,0 +1,59 @@
+require 'digest/md5'
+
+#
+#  If you want your job to be unique, include this module in it. If you wish,
+#  you can overwrite this implementation of redis_key to fit your needs
+#
+module Resque
+  module Plugins
+    module UniqueJob
+
+      def self.included(base)
+        base.extend         ClassMethods
+        base.class_eval do
+          base.send(:extend, Resque::Helpers)
+        end
+      end # self.included
+
+      module ClassMethods
+
+
+        #
+        #  Payload is what Resque stored for this job along with the job's class name.
+        #  On a Resque with no plugins installed, this is a hash containing :class and :args
+        #
+        def redis_key(payload)
+          payload = decode(encode(payload)) # This is the cycle the data goes when being enqueued/dequeued
+          job  = payload[:class] || payload["class"]
+          args = (payload[:args]  || payload["args"])
+          args.map! do |arg|
+            arg.is_a?(Hash) ? arg.sort : arg
+          end
+
+          digest = Digest::MD5.hexdigest encode(:class => job, :args => args)
+          digest
+        end
+      end # ClassMethods
+
+
+    end
+  end
+end
+
+module Resque
+  module Plugins
+    module Loner
+      class UniqueJob
+
+        include Resque::Plugins::UniqueJob
+
+        def self.inherited(host)
+          super(host)
+          return  if @__unique_job_warned
+          warn "Inherit Resque::Plugins::Loner::UniqueJob is deprecated. Include Resque::Plugins::UniqueJob module instead."
+          @__unique_job_warned = true
+        end
+      end
+    end
+  end
+end

data/lib/resque-loner/version.rb

@@ -0,0 +1,7 @@
+module Resque
+  module Plugins
+    module Loner
+      VERSION = "1.0.1.1"
+    end
+  end
+end

metadata

@@ -0,0 +1,169 @@
+--- !ruby/object:Gem::Specification 
+name: ryansch-resque-loner
+version: !ruby/object:Gem::Version 
+  hash: 89
+  prerelease: 
+  segments: 
+  - 1
+  - 0
+  - 1
+  - 1
+  version: 1.0.1.1
+platform: ruby
+authors: 
+- Jannis Hermanns
+- Ryan Schlesinger
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2011-11-04 00:00:00 -07:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: resque
+  prerelease: false
+  requirement: &id001 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        hash: 15
+        segments: 
+        - 1
+        - 0
+        version: "1.0"
+  type: :runtime
+  version_requirements: *id001
+- !ruby/object:Gem::Dependency 
+  name: rake
+  prerelease: false
+  requirement: &id002 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        hash: 49
+        segments: 
+        - 0
+        - 8
+        - 7
+        version: 0.8.7
+  type: :development
+  version_requirements: *id002
+- !ruby/object:Gem::Dependency 
+  name: rspec
+  prerelease: false
+  requirement: &id003 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        hash: 27
+        segments: 
+        - 2
+        - 5
+        - 0
+        version: 2.5.0
+  type: :development
+  version_requirements: *id003
+- !ruby/object:Gem::Dependency 
+  name: bundler
+  prerelease: false
+  requirement: &id004 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        hash: 23
+        segments: 
+        - 1
+        - 0
+        - 0
+        version: 1.0.0
+  type: :development
+  version_requirements: *id004
+- !ruby/object:Gem::Dependency 
+  name: rack-test
+  prerelease: false
+  requirement: &id005 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ~>
+      - !ruby/object:Gem::Version 
+        hash: 5
+        segments: 
+        - 0
+        - 5
+        - 7
+        version: 0.5.7
+  type: :development
+  version_requirements: *id005
+description: |
+  Makes sure that for special jobs, there can be only one job with the same workload in one queue.
+  
+  Example:
+      class CacheSweeper 
+  
+         include Resque::Plugins::UniqueJob
+  
+         @queue = :cache_sweeps
+  
+         def self.perform(article_id)
+           # Cache Me If You Can...
+         end
+      end
+
+email: 
+- jannis@moviepilot.com
+- ryan@instanceinc.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: []
+
+files: 
+- lib/resque-ext/job.rb
+- lib/resque-ext/resque.rb
+- lib/resque-loner/helpers.rb
+- lib/resque-loner/unique_job.rb
+- lib/resque-loner/version.rb
+- lib/resque-loner.rb
+- README.markdown
+has_rdoc: true
+homepage: http://github.com/ryansch/resque-loner
+licenses: []
+
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+requirements: []
+
+rubyforge_project: resque-loner
+rubygems_version: 1.6.2
+signing_key: 
+specification_version: 3
+summary: Adds unique jobs to resque
+test_files: []
+