resque 1.6.1 → 1.7.0

data/HISTORY.md

@@ -1,3 +1,10 @@
+## 1.7.0 (2010-03-31)
+
+* Job hooks API. See HOOKS.md.
+* web: Hovering over dates shows a timestamp
+* web: AJAXify retry action for failed jobs
+* web bugfix: Fix pagination bug
+
 ## 1.6.1 (2010-03-25)
 
 * Bugfix: Workers may not be clearing their state correctly on

data/HOOKS.md

@@ -0,0 +1,120 @@
+Resque Hooks
+============
+
+You can customize Resque or write plugins using its hook API. In many
+cases you can use a hook rather than mess with Resque's internals.
+
+For a list of available plugins see
+<http://wiki.github.com/defunkt/resque/plugins>.
+
+
+Worker Hooks
+------------
+
+If you wish to have a Proc called before the worker forks for the
+first time, you can add it in the initializer like so:
+
+    Resque.before_first_fork do
+      puts "Call me once before the worker forks the first time"
+    end
+
+You can also run a hook before _every_ fork:
+
+    Resque.before_fork do |job|
+      puts "Call me before the worker forks"
+    end
+
+The `before_fork` hook will be run in the **parent** process. So, be
+careful - any changes you make will be permanent for the lifespan of
+the worker.
+
+And after forking:
+
+    Resque.after_fork do |job|
+      puts "Call me after the worker forks"
+    end
+
+The `after_fork` hook will be run in the child process and is passed
+the current job. Any changes you make, therefor, will only live as
+long as the job currently being processes.
+
+All worker hooks can also be set using a setter, e.g.
+
+    Resque.after_fork = proc { puts "called" }
+
+
+Job Hooks
+---------
+
+Plugins can utilize job hooks to provide additional behavior. A job
+hook is a method name in the following format:
+
+    HOOKNAME_IDENTIFIER
+
+For example, a `before_perform` hook which adds locking may be defined
+like this:
+
+    def before_perform_with_lock(*args)
+      set_lock!
+    end
+
+Once this hook is made available to your job (either by way of
+inheritence or `extend`), it will be run before the job's `perform`
+method is called.
+
+The available hooks are:
+
+* `before_perform`: Called with the job args before perform. If it raises
+  `Resque::Job::DontPerform`, the job is aborted. If other exceptions
+  are raised, they will be propagated up the the `Resque::Failure`
+  backend.
+
+* `after_perform`: Called with the job args after it performs. Uncaught
+  exceptions will propagate up to the `Resque::Failure` backend.
+
+* `around_perform`: Called with the job args. It is expected to yield in order
+  to perform the job (but is not required to do so). It may handle exceptions
+  thrown by `perform`, but any that are not caught will propagate up to the
+  `Resque::Failure` backend.
+
+* `on_failure`: Called with the exception and job args if any exception occurs
+  while performing the job (or hooks).
+
+Hooks are easily implemented with superclasses or modules. A superclass could
+look something like this.
+
+    class LoggedJob
+      def self.before_perform_log_job(*args)
+        Logger.info "About to perform #{self} with #{args.inspect}"
+      end
+    end
+
+    class MyJob < LoggedJob
+      def self.perform(*args)
+        ...
+      end
+    end
+
+Modules are even better because jobs can use many of them.
+
+    module LoggedJob
+      def before_perform_log_job(*args)
+        Logger.info "About to perform #{self} with #{args.inspect}"
+      end
+    end
+
+    module RetriedJob
+      def on_failure_retry(e, *args)
+        Logger.info "Performing #{self} caused an exception (#{e}). Retrying..."
+        Resque.enqueue self, *args
+      end
+    end
+
+    class MyJob
+      extend LoggedJob
+      extend RetriedJob
+      def self.perform(*args)
+        ...
+      end
+    end
+

data/README.markdown

@@ -485,6 +485,8 @@ tool that's best for your app.
 Installing Redis
 ----------------
 
+Resque requires Redis 0.900 or higher.
+
 Resque uses Redis' lists for its queues. It also stores worker state
 data in Redis.
 
@@ -529,6 +531,9 @@ Resque Dependencies
 If you cannot install `yajl-ruby` (JRuby?), you can install the `json`
 gem and Resque will use it instead.
 
+When problems arise, make sure you have the newest versions of the
+`redis` and `redis-namespace` gems.
+
 
 Installing Resque
 -----------------
@@ -648,39 +653,16 @@ this way we can tell our Sinatra app about the config file:
 
 Now everyone is on the same page.
 
-Worker Hooks
-------------
-
-If you wish to have a Proc called before the worker forks for the
-first time, you can add it in the initializer like so:
-
-    Resque.before_first_fork do
-      puts "CALL ME ONCE BEFORE THE WORKER FORKS THE FIRST TIME"
-    end
-
-You can also run a hook before _every_ fork:
-
-    Resque.before_fork do |job|
-      puts "CALL ME BEFORE THE WORKER FORKS"
-    end
 
-The `before_fork` hook will be run in the **parent** process. So, be
-careful - any changes you make will be permanent for the lifespan of
-the worker.
-
-And after forking:
-
-    Resque.after_fork do |job|
-      puts "CALL ME AFTER THE WORKER FORKS"
-    end
-
-The `after_fork` hook will be run in the child process and is passed
-the current job. Any changes you make, therefor, will only live as
-long as the job currently being processes.
+Plugins and Hooks
+-----------------
 
-All hooks can also be set using a setter, e.g.
+For a list of available plugins see
+<http://wiki.github.com/defunkt/resque/plugins>.
 
-    Resque.after_fork = proc { puts "called" }
+If you'd like to write your own plugin, or want to customize Resque
+using hooks (such as `Resque.after_fork`), see
+[HOOKS.md](http://github.com/defunkt/resque/blob/master/HOOKS.md).
 
 
 Namespaces
@@ -714,10 +696,18 @@ Try it out by looking at the README, found at `examples/demo/README.markdown`.
 Monitoring
 ----------
 
+### god
+
 If you're using god to monitor Resque, we have provided example
 configs in `examples/god/`. One is for starting / stopping workers,
 the other is for killing workers that have been running too long.
 
+### monit
+
+If you're using monit, `examples/monit/resque.monit` is provided free
+of charge. This is **not** used by GitHub in production, so please
+send patches for any tweaks or improvements you can make to it.
+
 
 Development
 -----------

data/Rakefile

@@ -7,6 +7,11 @@ task :default => :test
 
 desc "Run tests"
 task :test do
+  begin
+    require 'redgreen'
+  rescue LoadError
+  end
+
   # Don't use the rake/testtask because it loads a new
   # Ruby interpreter - we want to run tests with the current
   # `rake` so our library manager still works

data/lib/resque/job.rb

@@ -15,6 +15,10 @@ module Resque
     include Helpers
     extend Helpers
 
+    # Raise Resque::Job::DontPerform from a before_perform hook to
+    # abort the job.
+    DontPerform = Class.new(StandardError)
+
     # The worker object which is currently processing this job.
     attr_accessor :worker
 
@@ -101,7 +105,64 @@ module Resque
     # Calls #perform on the class given in the payload with the
     # arguments given in the payload.
     def perform
-      args ? payload_class.perform(*args) : payload_class.perform
+      job = payload_class
+      job_args = args || []
+      job_was_performed = false
+
+      before_hooks  = job.methods.grep(/^before_perform/)
+      around_hooks  = job.methods.grep(/^around_perform/)
+      after_hooks   = job.methods.grep(/^after_perform/)
+      failure_hooks = job.methods.grep(/^on_failure/)
+
+      begin
+        # Execute before_perform hook. Abort the job gracefully if
+        # Resque::DontPerform is raised.
+        begin
+          before_hooks.each do |hook|
+            job.send(hook, *job_args)
+          end
+        rescue DontPerform
+          return false
+        end
+
+        # Execute the job. Do it in an around_perform hook if available.
+        if around_hooks.empty?
+          job.perform(*job_args)
+          job_was_performed = true
+        else
+          # We want to nest all around_perform plugins, with the last one
+          # finally calling perform
+          stack = around_hooks.inject(nil) do |last_hook, hook|
+            if last_hook
+              lambda do
+                job.send(hook, *job_args) { last_hook.call }
+              end
+            else
+              lambda do
+                job.send(hook, *job_args) do
+                  job.perform(*job_args)
+                  job_was_performed = true
+                end
+              end
+            end
+          end
+          stack.call
+        end
+
+        # Execute after_perform hook
+        after_hooks.each do |hook|
+          job.send(hook, *job_args)
+        end
+
+        # Return true if the job was performed
+        return job_was_performed
+
+      # If an exception occurs during the job execution, look for an
+      # on_failure hook then re-raise.
+      rescue Object => e
+        failure_hooks.each { |hook| job.send(hook, e, *job_args) }
+        raise e
+      end
     end
 
     # Returns the actual class constant represented in this job's payload.

data/lib/resque/server.rb

@@ -20,7 +20,7 @@ module Resque
       end
 
       def current_page
-        url request.path_info.sub('/','').downcase
+        url request.path_info.sub('/','')
       end
 
       def url(*path_parts)
@@ -153,7 +153,11 @@ module Resque
     
     get "/failed/requeue/:index" do
       Resque::Failure.requeue(params[:index])
-      redirect u('failed')
+      if request.xhr?
+        return Resque::Failure.all(params[:index])['retried_at']
+      else
+        redirect u('failed')
+      end
     end
 
     get "/stats" do

data/lib/resque/server/public/ranger.js

@@ -1,24 +1,67 @@
-var poll_interval = 2;
-
 $(function() {
+  var poll_interval = 2
+
+  var relatizer = function(){
+    var dt = $(this).text(), relatized = $.relatizeDate(this)
+    if ($(this).parents("a").length > 0 || $(this).is("a")) {
+      $(this).relatizeDate()
+      if (!$(this).attr('title')) {
+        $(this).attr('title', dt)
+      }
+    } else {
+      $(this)
+        .text('')
+        .append( $('<a href="#" class="toggle_format" title="' + dt + '" />')
+        .append('<span class="date_time">' + dt +
+                '</span><span class="relatized_time">' +
+                relatized + '</span>') )
+    }
+  };
+
+  $('.time').each(relatizer);
+
+  $('.time a.toggle_format .date_time').hide()
+
+  var format_toggler = function(){
+    $('.time a.toggle_format span').toggle()
+    $(this).attr('title', $('span:hidden',this).text())
+    return false
+  };
+
+  $('.time a.toggle_format').click(format_toggler);
 
-  $('.time').relatizeDate()
   $('.backtrace').click(function() {
     $(this).next().toggle()
     return false
   })
-  
+
   $('a[rel=poll]').click(function() {
     var href = $(this).attr('href')
     $(this).parent().text('Starting...')
     $("#main").addClass('polling')
+
     setInterval(function() {
-      $.ajax({dataType:'text', type:'get', url:href, success:function(data) { 
-        $('#main').html(data) 
+      $.ajax({dataType: 'text', type: 'get', url: href, success: function(data) {
+        $('#main').html(data)
         $('#main .time').relatizeDate()
       }})
     }, poll_interval * 1000)
+
     return false
   })
-    
+
+  $('ul.failed a[rel=retry]').click(function() {
+    var href = $(this).attr('href');
+    $(this).text('Retrying...');
+    var parent = $(this).parent();
+    $.ajax({dataType: 'text', type: 'get', url: href, success: function(data) {
+      parent.html('Retried <b><span class="time">' + data + '</span></b>');
+      relatizer.apply($('.time', parent));
+      $('.date_time', parent).hide();
+      $('a.toggle_format span', parent).click(format_toggler);
+    }});
+    return false;
+  })
+
+
 })

data/lib/resque/server/public/style.css

@@ -77,3 +77,5 @@ body { padding:0; margin:0; }
 #main p.pagination a.more { float:right;}
 
 #main form {float:right; margin-top:-10px;}
+
+#main .time a.toggle_format {text-decoration:none;}

data/lib/resque/server/views/failed.erb

@@ -23,7 +23,7 @@
             <% if job['retried_at'] %>
               Retried <b><span class="time"><%= job['retried_at'] %></span></b>
             <% else %>
-              <a href="<%= u "failed/requeue/#{start + index - 1}" %>">Retry</a>
+              <a href="<%= u "failed/requeue/#{start + index - 1}" %>" rel="retry">Retry</a>
             <% end %>
           </div>
         </dd>
@@ -46,3 +46,4 @@
 </ul>
 
 <%= partial :next_more, :start => start, :size => size %>
+

data/lib/resque/version.rb

@@ -1,3 +1,3 @@
 module Resque
-  Version = '1.6.1'
+  Version = '1.7.0'
 end

data/lib/resque/worker.rb

@@ -143,6 +143,7 @@ module Resque
       return unless job ||= reserve
 
       begin
+        job.worker = self
         run_hook :after_fork, job
         working_on job
         job.perform
@@ -332,7 +333,6 @@ module Resque
     # Given a job, tells Redis we're working on it. Useful for seeing
     # what workers are doing and when.
     def working_on(job)
-      job.worker = self
       data = encode \
         :queue   => job.queue,
         :run_at  => Time.now.to_s,

data/test/job_hooks_test.rb

@@ -0,0 +1,302 @@
+require File.dirname(__FILE__) + '/test_helper'
+
+context "Resque::Job before_perform" do
+  include PerformJob
+
+  class BeforePerformJob
+    def self.before_perform_record_history(history)
+      history << :before_perform
+    end
+
+    def self.perform(history)
+      history << :perform
+    end
+  end
+
+  test "it runs before_perform before perform" do
+    result = perform_job(BeforePerformJob, history=[])
+    assert_equal true, result, "perform returned true"
+    assert_equal history, [:before_perform, :perform]
+  end
+
+  class BeforePerformJobFails
+    def self.before_perform_fail_job(history)
+      history << :before_perform
+      raise StandardError
+    end
+    def self.perform(history)
+      history << :perform
+    end
+  end
+
+  test "raises an error and does not perform if before_perform fails" do
+    history = []
+    assert_raises StandardError do
+      perform_job(BeforePerformJobFails, history)
+    end
+    assert_equal history, [:before_perform], "Only before_perform was run"
+  end
+
+  class BeforePerformJobAborts
+    def self.before_perform_abort(history)
+      history << :before_perform
+      raise Resque::Job::DontPerform
+    end
+    def self.perform(history)
+      history << :perform
+    end
+  end
+
+  test "does not perform if before_perform raises Resque::Job::DontPerform" do
+    result = perform_job(BeforePerformJobAborts, history=[])
+    assert_equal false, result, "perform returned false"
+    assert_equal history, [:before_perform], "Only before_perform was run"
+  end
+end
+
+context "Resque::Job after_perform" do
+  include PerformJob
+
+  class AfterPerformJob
+    def self.perform(history)
+      history << :perform
+    end
+    def self.after_perform_record_history(history)
+      history << :after_perform
+    end
+  end
+
+  test "it runs after_perform after perform" do
+    result = perform_job(AfterPerformJob, history=[])
+    assert_equal true, result, "perform returned true"
+    assert_equal history, [:perform, :after_perform]
+  end
+
+  class AfterPerformJobFails
+    def self.perform(history)
+      history << :perform
+    end
+    def self.after_perform_fail_job(history)
+      history << :after_perform
+      raise StandardError
+    end
+  end
+
+  test "raises an error but has already performed if after_perform fails" do
+    history = []
+    assert_raises StandardError do
+      perform_job(AfterPerformJobFails, history)
+    end
+    assert_equal history, [:perform, :after_perform], "Only after_perform was run"
+  end
+end
+
+context "Resque::Job around_perform" do
+  include PerformJob
+
+  class AroundPerformJob
+    def self.perform(history)
+      history << :perform
+    end
+    def self.around_perform_record_history(history)
+      history << :start_around_perform
+      yield
+      history << :finish_around_perform
+    end
+  end
+
+  test "it runs around_perform then yields in order to perform" do
+    result = perform_job(AroundPerformJob, history=[])
+    assert_equal true, result, "perform returned true"
+    assert_equal history, [:start_around_perform, :perform, :finish_around_perform]
+  end
+
+  class AroundPerformJobFailsBeforePerforming
+    def self.perform(history)
+      history << :perform
+    end
+    def self.around_perform_fail(history)
+      history << :start_around_perform
+      raise StandardError
+      yield
+      history << :finish_around_perform
+    end
+  end
+
+  test "raises an error and does not perform if around_perform fails before yielding" do
+    history = []
+    assert_raises StandardError do
+      perform_job(AroundPerformJobFailsBeforePerforming, history)
+    end
+    assert_equal history, [:start_around_perform], "Only part of around_perform was run"
+  end
+
+  class AroundPerformJobFailsWhilePerforming
+    def self.perform(history)
+      history << :perform
+      raise StandardError
+    end
+    def self.around_perform_fail_in_yield(history)
+      history << :start_around_perform
+      begin
+        yield
+      ensure
+        history << :ensure_around_perform
+      end
+      history << :finish_around_perform
+    end
+  end
+
+  test "raises an error but may handle exceptions if perform fails" do
+    history = []
+    assert_raises StandardError do
+      perform_job(AroundPerformJobFailsWhilePerforming, history)
+    end
+    assert_equal history, [:start_around_perform, :perform, :ensure_around_perform], "Only part of around_perform was run"
+  end
+
+  class AroundPerformJobDoesNotHaveToYield
+    def self.perform(history)
+      history << :perform
+    end
+    def self.around_perform_dont_yield(history)
+      history << :start_around_perform
+      history << :finish_around_perform
+    end
+  end
+
+  test "around_perform is not required to yield" do
+    history = []
+    result = perform_job(AroundPerformJobDoesNotHaveToYield, history)
+    assert_equal false, result, "perform returns false"
+    assert_equal history, [:start_around_perform, :finish_around_perform], "perform was not run"
+  end
+end
+
+context "Resque::Job on_failure" do
+  include PerformJob
+
+  class FailureJobThatDoesNotFail
+    def self.perform(history)
+      history << :perform
+    end
+    def self.on_failure_record_failure(exception, history)
+      history << exception.message
+    end
+  end
+
+  test "it does not call on_failure if no failures occur" do
+    result = perform_job(FailureJobThatDoesNotFail, history=[])
+    assert_equal true, result, "perform returned true"
+    assert_equal history, [:perform]
+  end
+
+  class FailureJobThatFails
+    def self.perform(history)
+      history << :perform
+      raise StandardError, "oh no"
+    end
+    def self.on_failure_record_failure(exception, history)
+      history << exception.message
+    end
+  end
+
+  test "it calls on_failure with the exception and then re-raises the exception" do
+    history = []
+    assert_raises StandardError do
+      perform_job(FailureJobThatFails, history)
+    end
+    assert_equal history, [:perform, "oh no"]
+  end
+
+  class FailureJobThatFailsBadly
+    def self.perform(history)
+      history << :perform
+      raise SyntaxError, "oh no"
+    end
+    def self.on_failure_record_failure(exception, history)
+      history << exception.message
+    end
+  end
+
+  test "it calls on_failure even with bad exceptions" do
+    history = []
+    assert_raises SyntaxError do
+      perform_job(FailureJobThatFailsBadly, history)
+    end
+    assert_equal history, [:perform, "oh no"]
+  end
+end
+
+context "Resque::Job all hooks" do
+  include PerformJob
+
+  class VeryHookyJob
+    def self.before_perform_record_history(history)
+      history << :before_perform
+    end
+    def self.around_perform_record_history(history)
+      history << :start_around_perform
+      yield
+      history << :finish_around_perform
+    end
+    def self.perform(history)
+      history << :perform
+    end
+    def self.after_perform_record_history(history)
+      history << :after_perform
+    end
+    def self.on_failure_record_history(exception, history)
+      history << exception.message
+    end
+  end
+
+  test "the complete hook order" do
+    result = perform_job(VeryHookyJob, history=[])
+    assert_equal true, result, "perform returned true"
+    assert_equal history, [
+      :before_perform,
+      :start_around_perform,
+      :perform,
+      :finish_around_perform,
+      :after_perform
+    ]
+  end
+
+  class VeryHookyJobThatFails
+    def self.before_perform_record_history(history)
+      history << :before_perform
+    end
+    def self.around_perform_record_history(history)
+      history << :start_around_perform
+      yield
+      history << :finish_around_perform
+    end
+    def self.perform(history)
+      history << :perform
+    end
+    def self.after_perform_record_history(history)
+      history << :after_perform
+      raise StandardError, "oh no"
+    end
+    def self.on_failure_record_history(exception, history)
+      history << exception.message
+    end
+  end
+
+  test "the complete hook order with a failure at the last minute" do
+    history = []
+    assert_raises StandardError do
+      perform_job(VeryHookyJobThatFails, history)
+    end
+    assert_equal history, [
+      :before_perform,
+      :start_around_perform,
+      :perform,
+      :finish_around_perform,
+      :after_perform,
+      "oh no"
+    ]
+  end
+end
+

data/test/test_helper.rb

@@ -63,6 +63,15 @@ def context(*args, &block)
   klass.class_eval &block
 end
 
+##
+# Helper to perform job classes
+#
+module PerformJob
+  def perform_job(klass, *args)
+    resque_job = Resque::Job.new(:testqueue, 'class' => klass, 'args' => args)
+    resque_job.perform
+  end
+end
 
 #
 # fixture classes

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: resque
 version: !ruby/object:Gem::Version 
-  version: 1.6.1
+  version: 1.7.0
 platform: ruby
 authors: 
 - Chris Wanstrath
@@ -9,7 +9,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2010-03-25 00:00:00 -07:00
+date: 2010-03-31 00:00:00 -07:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -77,6 +77,7 @@ files:
 - .kick
 - CONTRIBUTORS
 - HISTORY.md
+- HOOKS.md
 - LICENSE
 - README.markdown
 - Rakefile
@@ -132,6 +133,7 @@ files:
 - lib/resque/worker.rb
 - tasks/redis.rake
 - tasks/resque.rake
+- test/job_hooks_test.rb
 - test/redis-test.conf
 - test/resque-web_test.rb
 - test/resque_test.rb
@@ -166,6 +168,7 @@ signing_key:
 specification_version: 3
 summary: ""
 test_files: 
+- test/job_hooks_test.rb
 - test/resque-web_test.rb
 - test/resque_test.rb
 - test/test_helper.rb