nogara-resque-loner 1.2.1

data/.gitignore

@@ -0,0 +1,5 @@
+*.gem
+**.DS_Store
+**.swp
+Gemfile.lock
+.rvmrc

data/CHANGELOG.markdown

@@ -0,0 +1,27 @@
+1.2.1
+--------------------------------
+Merged @aerodynamik's pull request. Enqueuing and marking as
+enqueued as an atomic operation. 
+
+1.2.0
+--------------------------------
+Thanks @unclebilly for your pull request. Resque-loner now supports
+a maximum time for which a job should be unique. Just define @loner_ttl
+in your job (or leave it at -1 to never expire) and after @loner_ttl
+seconds your job can be enqueued again, even if an older one is still
+marked as running.
+
+1.1.0
+--------------------------------
+Merged in @ryansch's pull requests to clean up things a bit.
+This removed the `enqueue_to` and `dequeue_from` methods from
+resque-loner because it caused troubles with resque's own 
+`enqueue_to`.
+
+1.0.1
+--------------------------------
+Pulled in #8 and #9 so that removing empty queues
+does not fail
+
+1.0
+---------------------------------

data/Gemfile

@@ -0,0 +1,8 @@
+source 'http://rubygems.org'
+gemspec
+
+group :development do
+  gem 'gemcutter'
+  gem 'ruby-debug', :platform => :mri_18
+  gem 'ruby-debug19', :platform => :mri_19
+end

data/LICENSE

@@ -0,0 +1,19 @@
+Copyright (c) 2010 Moviepilot GmbH http://moviepilot.com
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,103 @@
+Resque-Loner
+======
+
+Resque-Loner is a plugin for defunkt/resque which adds unique jobs to resque: Only one job with the same payload per queue.
+
+
+Installation
+-------------
+
+First install the gem:
+
+    $ gem install resque-loner 
+
+Then include it in your app:
+
+    require 'resque-loner'
+
+
+Tests
+-----------
+To make sure this plugin works on your installation, you should run the tests. resque-loner is tested in RSpec, but it also includes resque's original testsuite. You can run all tests specific to resque-loner with `rake spec`.
+
+To make sure the plugin did not break resque, you can run `rake test` (the standard resque test suite). This runs all tests from the 1.10.0 version of resque, so make sure you have that version of resque installed, when you run the resque-tests.
+
+Example
+--------
+
+Unique jobs can be useful in situations where running the same job multiple times issues the same results. Let's say you have a job called CacheSweeper that refreshes some cache. A user has edited some_article, so you put a job on the queue to refresh the cache for that article.
+
+    >> Resque.enqueue CacheSweeper, some_article.id
+    => "OK"
+
+Your queue is really full, so the job does not get executed right away. But the user editing the article has noticed another error, and updates the article again, and your app kindly queues another job to update that article's cache.
+
+    >> Resque.enqueue CacheSweeper, some_article.id
+    => "OK"
+
+At this point you will have two jobs in the queue, the second of which has no effect: You don't have to run it, once the cache has been updated for the first time. This is where resque-loner's UniqueJobs come in. If you define CacheSweeper like this:
+
+    class CacheSweeper
+      include Resque::Plugins::UniqueJob
+      @queue = :cache_sweeps
+
+      def self.perform(article_id)
+        # Cache Me If You Can...
+      end
+    end
+
+Just like that you've assured that on the :cache_sweeps queue, there can only be one CacheSweeper job for each article. Let's see what happens when you try to enqueue a couple of these jobs now:
+
+    >> Resque.enqueue CacheSweeper, 1
+    => "OK"
+    >> Resque.enqueue CacheSweeper, 1
+    => "EXISTED"
+    >> Resque.enqueue CacheSweeper, 1
+    => "EXISTED"
+    >> Resque.size :cache_sweeps
+    => 1
+
+Since resque-loner keeps track of which jobs are queued in a way that allows for finding jobs very quickly, you can also query if a job is currently in a queue:
+
+    >> Resque.enqueue CacheSweeper, 1
+    => "OK"
+    >> Resque.enqueued? CacheSweeper, 1
+    => true
+    >> Resque.enqueued? CacheSweeper, 2
+    => false
+    >> Resque.enqueued_in? :another_queue, CacheSweeper, 1
+    => false
+
+How it works
+--------
+
+### Keeping track of queued unique jobs
+
+For each created UniqueJob, resque-loner sets a redis key to 1. This key remains set until the job has either been fetched from the queue or destroyed through the Resque::Job.destroy method. As long as the key is set, the job is considered queued and consequent queue adds are being rejected.
+
+Here's how these keys are constructed:
+
+    resque:loners:queue:cache_sweeps:job:5ac5a005253450606aa9bc3b3d52ea5b
+    |          |        |                |
+    |          |        |                `---- Job's ID (#redis_key method)
+    |          |        `--------------------- Name of the queue
+    |          `------------------------------ Prefix for this plugin
+    `----------------------------------------- Your redis namespace
+
+The last part of this key is the job's ID, which is pretty much your queue item's payload. For our CacheSweeper job, the payload would be:
+
+    { 'class': 'CacheSweeper', 'args': [1] }`
+
+The default method to create a job ID from these parameters  is to do some normalization on the payload and then md5'ing it (defined in `Resque::Plugins::UniqueJob#redis_key`).
+
+You could also use the whole payload or anything else as a redis key, as long as you make sure these requirements are met:
+
+1. Two jobs of the same class with the same parameters/arguments/workload must produce the same redis_key
+2. Two jobs with either a different class or different parameters/arguments/workloads must not produce the same redis key 
+3. The key must not be binary, because this restriction applies to redis keys: *Keys are not binary safe strings in Redis, but just strings not containing a space or a newline character. For instance "foo" or "123456789" or "foo_bar" are valid keys, while "hello world" or "hello\n" are not.* (see http://code.google.com/p/redis/wiki/IntroductionToRedisDataTypes)
+
+So when your job overwrites the #redis_key method, make sure these requirements are met. And all should be good.
+
+### Resque integration
+
+Unfortunately not everything could be done as a plugin, so I overwrote three methods of Resque::Job: create, reserve and destroy (I found no hooks for these events). All the logic is in `module Resque::Plugins::Loner` though, so it should be fairly easy to make this a *pure* plugin once the hooks are known.

data/Rakefile

@@ -0,0 +1,83 @@
+#
+# Setup
+#
+
+$LOAD_PATH.unshift 'lib'
+
+require "rubygems"
+require "bundler"
+Bundler.setup
+
+require 'rspec/core/rake_task'
+
+load 'tasks/redis.rake'
+require 'rake/testtask'
+
+require 'resque/tasks'
+
+require 'bundler/gem_tasks'
+
+def command?(command)
+  system("type #{command} > /dev/null 2>&1")
+end
+
+
+#
+# Tests
+#
+
+task :default => :spec
+
+desc "Run specs for resque-loner"
+RSpec::Core::RakeTask.new(:spec) do |t|
+  t.pattern = "spec/**/*_spec.rb"
+  t.rspec_opts = %w(-fd -c)
+end
+
+# desc "Run resque's test suite to make sure we did not break anything"
+# task :test do
+#   rg = command?(:rg)
+#   Dir['test/**/*_test.rb'].each do |f|
+#     rg ? sh("rg #{f}") : ruby(f)
+#   end
+# end
+
+if command?(:rg)
+  desc "Run the test suite with rg"
+  task :test do
+    Dir['test/**/*_test.rb'].each do |f|
+      sh("rg #{f}")
+    end
+  end
+else
+  Rake::TestTask.new do |test|
+    test.libs << "test"
+    test.test_files = FileList['test/**/*_test.rb']
+  end
+end
+
+if command? :kicker
+  desc "Launch Kicker (like autotest)"
+  task :kicker do
+    puts "Kicking... (ctrl+c to cancel)"
+    exec "kicker -e rake test lib examples"
+  end
+end
+
+
+#
+# Install
+#
+
+task :install => [ 'redis:install', 'dtach:install' ]
+
+
+#
+# Documentation
+#
+
+begin
+  require 'sdoc_helpers'
+rescue LoadError
+end
+

data/init.rb

@@ -0,0 +1 @@
+require 'rails/init.rb'

data/lib/resque-ext/job.rb

@@ -0,0 +1,57 @@
+#
+#  Since there were not enough hooks to hook into, I have to overwrite
+#  3 methods of Resque::Job - the rest of the implementation is in the
+#  proper Plugin namespace.
+# 
+module Resque
+  class Job
+
+
+    #
+    #  Overwriting original create method to mark an item as queued
+    #  after Resque::Job.create has called Resque.push
+    #
+    def self.create_with_loner(queue, klass, *args)
+      return create_without_loner(queue, klass, *args) if Resque.inline?
+      item = { :class => klass.to_s, :args => args }
+      return "EXISTED" if Resque::Plugins::Loner::Helpers.loner_queued?(queue, item)
+      # multi block returns array of keys
+      Resque.redis.multi do
+        create_without_loner(queue, klass, *args)
+        Resque::Plugins::Loner::Helpers.mark_loner_as_queued(queue, item)
+      end.first
+    end
+
+    #
+    #  Overwriting original reserve method to mark an item as unqueued
+    #
+    def self.reserve_with_loner(queue)
+      item = reserve_without_loner(queue)
+      Resque::Plugins::Loner::Helpers.mark_loner_as_unqueued( queue, item ) if item && !Resque.inline?
+      item
+    end
+
+    #
+    #  Overwriting original destroy method to mark all destroyed jobs as unqueued.
+    #  Because the original method only returns the amount of jobs destroyed, but not 
+    #  the jobs themselves. Hence Resque::Plugins::Loner::Helpers.job_destroy looks almost
+    #  as the original method Resque::Job.destroy. Couldn't make it any dry'er.
+    #
+    def self.destroy_with_loner(queue, klass, *args)
+      Resque::Plugins::Loner::Helpers.job_destroy(queue, klass, *args) unless Resque.inline?
+      destroy_without_loner(queue, klass, *args)
+    end
+
+    #
+    # Chain..
+    #
+    class << self
+      alias_method :create_without_loner, :create
+      alias_method :create, :create_with_loner
+      alias_method :reserve_without_loner, :reserve
+      alias_method :reserve, :reserve_with_loner
+      alias_method :destroy_without_loner, :destroy
+      alias_method :destroy, :destroy_with_loner
+    end
+  end
+end

data/lib/resque-ext/resque.rb

@@ -0,0 +1,26 @@
+module Resque
+
+  def self.enqueued?( klass, *args)
+    enqueued_in?(queue_from_class(klass), klass, *args )
+  end
+
+  def self.enqueued_in?(queue, klass, *args)
+    item = { :class => klass.to_s, :args => args }
+    return nil unless Resque::Plugins::Loner::Helpers.item_is_a_unique_job?(item)
+    Resque::Plugins::Loner::Helpers.loner_queued?(queue, item)
+  end
+
+  def self.remove_queue_with_loner_cleanup(queue)
+    self.remove_queue_without_loner_cleanup(queue)
+    Resque::Plugins::Loner::Helpers.cleanup_loners(queue)
+  end
+
+
+  class << self
+
+    alias_method :remove_queue_without_loner_cleanup, :remove_queue
+    alias_method :remove_queue, :remove_queue_with_loner_cleanup
+
+  end
+
+end

data/lib/resque-loner.rb

@@ -0,0 +1,6 @@
+require 'rubygems'
+require 'resque'
+require 'resque-loner/unique_job'
+require 'resque-loner/helpers'
+require 'resque-ext/job'
+require 'resque-ext/resque'

data/lib/resque-loner/helpers.rb

@@ -0,0 +1,73 @@
+module Resque
+  module Plugins
+    module Loner
+      class Helpers
+        extend Resque::Helpers
+
+        def self.loner_queued?(queue, item)
+          return false unless item_is_a_unique_job?(item)
+          redis.get(unique_job_queue_key(queue, item)) == "1"
+        end
+
+        def self.mark_loner_as_queued(queue, item)
+          return unless item_is_a_unique_job?(item)
+          key = unique_job_queue_key(queue, item)
+          redis.set(key, 1)
+          unless(ttl=item_ttl(item)) == -1 # no need to incur overhead for default value
+            redis.expire(key, ttl)
+          end
+        end
+
+        def self.mark_loner_as_unqueued(queue, job)
+          item = job.is_a?(Resque::Job) ? job.payload : job
+          return unless item_is_a_unique_job?(item)
+          redis.del(unique_job_queue_key(queue, item))
+        end
+
+        def self.unique_job_queue_key(queue, item)
+          job_key = constantize(item[:class] || item["class"]).redis_key_loner(item)
+          "loners:queue:#{queue}:job:#{job_key}"
+        end
+
+        def self.item_is_a_unique_job?(item)
+          begin
+            klass = constantize(item[:class] || item["class"])
+            klass.included_modules.include?(::Resque::Plugins::UniqueJob)
+          rescue
+            false # Resque testsuite also submits strings as job classes while Resque.enqueue'ing,
+          end     # so resque-loner should not start throwing up when that happens.
+        end
+
+        def self.item_ttl(item)
+          begin
+            constantize(item[:class] || item["class"]).loner_ttl
+          rescue
+            -1
+          end
+        end
+
+        def self.job_destroy(queue, klass, *args)
+          klass = klass.to_s
+          redis_queue = "queue:#{queue}"
+
+          redis.lrange(redis_queue, 0, -1).each do |string|
+            json   = decode(string)
+
+            match  = json['class'] == klass
+            match &= json['args'] == args unless args.empty?
+
+            if match
+             Resque::Plugins::Loner::Helpers.mark_loner_as_unqueued( queue, json )
+            end
+          end
+        end
+
+        def self.cleanup_loners(queue)
+          keys = redis.keys("loners:queue:#{queue}:job:*")
+          redis.del(*keys) unless keys.empty?
+        end
+
+      end
+    end
+  end
+end

data/lib/resque-loner/unique_job.rb

@@ -0,0 +1,75 @@
+require 'digest/md5'
+
+#
+#  If you want your job to be unique, include this module in it. If you wish,
+#  you can overwrite this implementation of redis_key to fit your needs
+#
+module Resque
+  module Plugins
+    module UniqueJob
+
+      def self.included(base)
+        base.extend         ClassMethods
+        base.class_eval do
+          base.send(:extend, Resque::Helpers)
+        end
+      end # self.included
+
+      module ClassMethods
+
+
+        #
+        #  Payload is what Resque stored for this job along with the job's class name.
+        #  On a Resque with no plugins installed, this is a hash containing :class and :args
+        #
+        def redis_key_loner(payload)
+          payload = decode(encode(payload)) # This is the cycle the data goes when being enqueued/dequeued
+          job  = payload[:class] || payload["class"]
+          args = (payload[:args]  || payload["args"])
+          args.map! do |arg|
+            arg.is_a?(Hash) ? arg.sort : arg
+          end
+
+          digest = Digest::MD5.hexdigest encode(:class => job, :args => args)
+          digest
+        end
+
+        #
+        # The default ttl of a locking key is -1, i.e. forever.  If for some reason you only
+        # want the lock to be in place after a certain amount of time, just set a ttl for
+        # for your job.  For example:
+        #
+        # class FooJob
+        #   include Resque::Plugins::UniqueJob
+        #   @loner_ttl = 40
+        #   end
+        # end
+        #
+        def loner_ttl
+          @loner_ttl || -1
+        end
+
+      end # ClassMethods
+
+
+    end
+  end
+end
+
+module Resque
+  module Plugins
+    module Loner
+      class UniqueJob
+
+        include Resque::Plugins::UniqueJob
+
+        def self.inherited(host)
+          super(host)
+          return  if @__unique_job_warned
+          warn "Inherit Resque::Plugins::Loner::UniqueJob is deprecated. Include Resque::Plugins::UniqueJob module instead."
+          @__unique_job_warned = true
+        end
+      end
+    end
+  end
+end