resque-cleaner 0.0.1 → 0.0.2

data/README.markdown

@@ -1,6 +1,223 @@
-Resque-Cleaner
+ResqueCleaner
 ==============
 
-Resque-Cleaner is a Resque plugin which helps you to deal with failure jobs.
+[github.com/ono/resque-cleaner](https://github.com/ono/resque-cleaner)
+
+
+Description
+-----------
+
+ResqueCleaner is a [Resque](https://github.com/defunkt/resque) plugin which
+helps you to deal with failed jobs of Resque by:
+
+* Showing stats of failed jobs
+* Retrying failed jobs
+* Removing failed jobs
+* Filtering failed jobs
+
+Although ResqueCleaner hasn't integrated with Resque's web-based interface yet,
+it is pretty easy to use on irb(console).
+
+
+Installation
+------------
+
+Install as a gem:
+    $ gem install resque-cleaner
+
+
+Usage
+-----
+
+**Create Instance**
+
+    > cleaner = Resque::Plugins::ResqueCleaner.new
+
+**Show Stats**
+
+Shows stats of failed jobs grouped by date.
+
+    > cleaner.stats_by_date
+    2009/03/13:    6
+    2009/11/13:   14
+    2010/08/13:   22
+         total:   42
+    => {'2009/03/10' => 6, ...}
+
+You could also group them by class.
+
+    > cleaner.stats_by_class
+         BadJob:    3
+    HorribleJob:    7
+          total:   10
+    => {'BadJob' => 3, ...}
+
+You can get the ones filtered with a block: it targets only jobs which the block
+evaluetes true. e.g. Show stats only of jobs entried with some arguments.
+
+    > cleaner.stats_by_date{|job| job["payload"]["args"].size > 0}
+    2009/03/13:    3
+    2009/11/13:    7
+    2010/08/13:   11
+         total:   22
+    => {'2009/03/10' => 3, ...}
+
+**Retry(Requeue) Jobs**
+
+You can retry all failed jobs with this method.
+
+    > cleaner.requeue
+
+Of course, you can filter jobs with a block; it requeues only jobs which the
+block evaluates true. e.g. Retry only jobs with some arguments.
+
+    > cleaner.requeue{ |job| job["payload"]["args"].size > 0}
+
+The job hash is extended with a module which defines some useful methods. You
+can use it in the blcok. e.g. Retry only jobs entried within a day.
+
+    > cleaner.requeue {|j| j.after?(1.day.ago)}
+
+e.g. Retry EmailJob entried with arguments within 3 days:
+
+    > cleaner.requeue {|j| j["payload"]["args"]>0 && j.after?(3.days.ago) && j.klass?(EmailJob)}
+
+See Helper Methods bellow for more.
+
+NOTE:
+[1.day.ago](https://github.com/rails/rails/blob/master/activesupport/lib/active_support/core_ext/numeric/time.rb)
+is not in standard library. It is equivalent to `Time.now - 60*60*24*3`.
+
+**Clear Jobs**
+
+You can clear all failed jobs with this method.
+
+    > cleaner.clear
+
+Like you can do with the retry method, the clear metod takes a block. Here are
+some examples:
+
+    > cleaner.clear {|j| j.retried?}
+    => clears all jobs already retried and returns number of the jobs.
+
+    > cleaner.clear {|j| j.queue?(:low) && j.before?('2010-10-10')}
+    => clears all jobs entried in :low queue before 10th October, 2010.
+
+    > cleaner.clear {|j| j["exception"]=="RuntimeError" && j.queue?(:low)}
+    => clears all jobs raised RuntimeError and queued :low queue
+
+**Retry and Clear Jobs**
+
+You can retry(requeue) and clear failed jobs at the same time; just pass true
+as an argument. e.g. Retry EmailJob and remove from failed jobs.
+
+    > cleaner.requeue(true) {|j| j.klass?(EmailJob)}
+
+**Select Jobs**
+
+You can just select the jobs of course. Here are some examples:
+
+    > cleaner.select {|j| j["exception"]=="RuntimeError"}
+    > cleaner.select {|j| j.after?(2.days.ago)}
+    > cleaner.select #=> returns all jobs
+
+**Helper Methods**
+
+Here is a list of methods a job extended.
+
+    retried?: returns true if the job has already been retried.
+    requeued?: alias of retried?.
+    before?(time): returns true if the job failed before the time.
+    after?(time): returns true if the job failed after the time.
+    klass?(klass_or_name): returns true if the class of job matches.
+    queue?(queue_name): returns true if the queue of job matches.
+
+
+Failed Job
+-----------
+
+I show a sample of failed job here; it might help you when you write a block for
+filtering failed jobs.
+
+    {"failed_at": "2009/03/13 00:00:00",
+     "payload": {"args": ["Johnson"], "class": "BadJob"},
+     "queue": "jobs",
+     "worker": "localhost:7327:jobs,jobs2",
+     "exception": "RuntimeError",
+     "error": "Bad job, Johnson",
+     "backtrace": 
+      ["./test/test_helper.rb:108:in `perform'",
+       "/opt/local/lib/ruby/gems/1.8/gems/resque-1.10.0/lib/resque/job.rb:133:in `perform'",
+       "/opt/local/lib/ruby/gems/1.8/gems/resque-1.10.0/lib/resque/worker.rb:157:in `perform'",
+       "/opt/local/lib/ruby/gems/1.8/gems/resque-1.10.0/lib/resque/worker.rb:124:in `work'",
+       "....(omitted)....",
+       "./test/test_helper.rb:41",
+       "test/resque_cleaner_test.rb:3"]
+    }
+
+
+Limiter
+-------
+
+ResqueCleaner expects a disaster situation like a huge number of failed jobs are
+out there. Since ResqueCleaner's filter function is running on your application
+process but on your Redis, it would not respond ages if you try to deal with all
+of those jobs.
+
+ResqueCleaner supposes recent jobs are more important than old jobs. Therefore
+ResqueCleaner deals with **ONLY LAST X(default=1000)** jobs. In that way, you
+don't have to worry that your block for filtering might be stuck. You can change
+its setting through `limiter` attribute. I will show you how it works with a
+sample situation.
+
+**Sample Situation**
+
+* Number of failed jobs: 100,000
+
+Default limiter is 1000 so that it returns 1000 as a count.
+
+    > cleaner.limiter.count
+    => 1,000
+    > cleaner.failure.count
+    => 100,000
+
+You could know if the limiter is on with on? method.
+
+    > cleaner.limiter.on?
+    => true
+
+You can change the maximum number of the limiter with maximum attribute.
+
+    > cleaner.limiter.maxmum = 3000
+    => 3,000
+    > cleaner.limiter.count
+    => 3,000
+    > cleaner.limiter.on?
+    => true
+
+With limiter, ResqueClener's filtering targets only the last X(3000 in this
+sampe) failed jobs.
+
+    > cleaner.select.size
+    => 3,000
+
+The clear\_stale method deletes all jobs entried prior to the last X(3000 in
+this sample) failed jobs. This calls Redis API and no iteration occurs on Ruby
+application; it should be quick even if there are huge number of failed jobs.
+
+    > cleaner.clear_stale
+    > cleaner.failure.count
+    => 3,000
+    > cleaner.limiter.count
+    => 3,000
+    > cleaner.limiter.on?
+    => false
+
+TODO
+----
+
+* Integration with Resque's sinatra based front end.
+* More stats.
+
+Any suggestion or idea are welcomed.
 
-(This README will be update in days.)

data/lib/resque-cleaner.rb

@@ -0,0 +1 @@
+require 'resque_cleaner'

data/lib/resque_cleaner.rb

@@ -35,42 +35,37 @@ module Resque
 
       # Stats by date.
       def stats_by_date(&block)
-        jobs = select(&block)
-        summary = {}
+        jobs, stats = select(&block), {}
         jobs.each do |job|
           date = job["failed_at"][0,10]
-          summary[date] ||= 0
-          summary[date] += 1
+          stats[date] ||= 0
+          stats[date] += 1
         end
 
-        if print?
-          log too_many_message if @limiter.on?
-          summary.keys.sort.each do |k|
-            log "%s: %4d" % [k,summary[k]]
-          end
-          log "%10s: %4d" % ["total", @limiter.count]
-        end
-        summary
+        print_stats(stats) if print?
+        stats 
       end
 
       # Stats by class.
       def stats_by_class(&block)
-        jobs = select(&block)
-        summary = {}
+        jobs, stats = select(&block), {}
         jobs.each do |job|
           klass = job["payload"]["class"]
-          summary[klass] ||= 0
-          summary[klass] += 1
+          stats[klass] ||= 0
+          stats[klass] += 1
         end
 
-        if print?
-          log too_many_message if @limiter.on?
-          summary.keys.sort.each do |k|
-            log "%15s: %4d" % [k,summary[k]]
-          end
-          log "%15s: %4d" % ["total", @limiter.count]
+        print_stats(stats) if print?
+        stats 
+      end
+
+      # Print stats
+      def print_stats(stats)
+        log too_many_message if @limiter.on?
+        stats.keys.sort.each do |k|
+          log "%15s: %4d" % [k,stats[k]]
         end
-        summary
+        log "%15s: %4d" % ["total", @limiter.count]
       end
 
       # Returns every jobs for which block evaluates to true.
@@ -132,49 +127,39 @@ module Resque
         c
       end
 
-      # Returns Proc which you can add a useful condition easily.
-      # e.g.
-      # cleaner.clear &cleaner.proc.retried
-      #   #=> Clears all jobs retried.
-      # cleaner.select &cleaner.proc.after(10.days.ago).klass(EmailJob)
-      #   #=> Selects all EmailJob failed within 10 days.
-      # cleaner.select &cleaner.proc{|j| j["exception"]=="RunTimeError"}.klass(EmailJob)
-      #   #=> Selects all EmailJob failed with RunTimeError.
-      def proc(&block)
-        FilterProc.new(&block)
-      end
-
-      # Provides typical proc you can filter jobs.
-      class FilterProc < Proc
-        def retried
-          FilterProc.new {|job| self.call(job) && job['retried_at'].blank?}
+      # Exntends job(Hash instance) with some helper methods.
+      module FailedJobEx
+        # Returns true if the job has been already retried. Otherwise returns
+        # false.
+        def retried?
+          self['retried_at'].blank?
         end
-        alias :requeued :retried
+        alias :requeued? :retried?
 
-        def before(time)
+        # Returns true if the job processed(failed) before the given time.
+        # Otherwise returns false.
+        # You can pass Time object or String.
+        def before?(time)
           time = Time.parse(time) if time.is_a?(String)
-          FilterProc.new {|job| self.call(job) && Time.parse(job['failed_at']) <= time}
+          Time.parse(self['failed_at']) < time
         end
 
-        def after(time)
+        # Returns true if the job processed(failed) after the given time.
+        # Otherwise returns false.
+        # You can pass Time object or String.
+        def after?(time)
           time = Time.parse(time) if time.is_a?(String)
-          FilterProc.new {|job| self.call(job) && Time.parse(job['failed_at']) >= time}
-        end
-
-        def klass(klass_or_name)
-          FilterProc.new {|job| self.call(job) && job["payload"]["class"] == klass_or_name.to_s}
+          Time.parse(self['failed_at']) >= time
         end
 
-        def queue(queue)
-          FilterProc.new {|job| self.call(job) && job["queue"] == queue.to_s}
+        # Returns true if the class of the job matches. Otherwise returns false.
+        def klass?(klass_or_name)
+          self["payload"]["class"] == klass_or_name.to_s
         end
 
-        def self.new(&block)
-          if block
-            super
-          else
-            super {|job| true}
-          end
+        # Returns true if the queue of the job matches. Otherwise returns false.
+        def queue?(queue)
+          self["queue"] == queue.to_s
         end
       end
 
@@ -214,11 +199,12 @@ module Resque
           end
         end
 
-        # wraps Resque's all and returns always array.
+        # Wraps Resque's all and returns always array.
         def all(index=0,count=1)
           jobs = @cleaner.failure.all( index, count)
           jobs = [] unless jobs
           jobs = [jobs] unless jobs.is_a?(Array)
+          jobs.each{|j| j.extend FailedJobEx}
           jobs
         end
 

data/test/resque_cleaner_test.rb

@@ -104,29 +104,29 @@ context "ResqueCleaner" do
     assert_equal Time.parse(@cleaner.failure_jobs[0]['failed_at']), Time.parse('2010-08-13')
   end
 
-  test "#proc gives you handy proc definitions" do
+  test "FailedJobEx module extends job and provides some useful methods" do
     # before 2009-04-01
-    ret = @cleaner.select &@cleaner.proc.before('2009-04-01')
+    ret = @cleaner.select {|j| j.before?('2009-04-01')}
     assert_equal 6, ret.size
 
     # after 2010-01-01
-    ret = @cleaner.select &@cleaner.proc.after('2010-01-01')
+    ret = @cleaner.select {|j| j.after?('2010-01-01')}
     assert_equal 22, ret.size
 
     # filter by class
-    ret = @cleaner.select &@cleaner.proc.klass(BadJobWithSyntaxError)
+    ret = @cleaner.select {|j| j.klass?(BadJobWithSyntaxError)}
     assert_equal 7, ret.size
 
     # filter by queue
-    ret = @cleaner.select &@cleaner.proc.queue(:jobs2)
+    ret = @cleaner.select {|j| j.queue?(:jobs2)}
     assert_equal 20, ret.size
 
-    # you can chain
-    ret = @cleaner.select &@cleaner.proc.queue(:jobs2).before('2009-12-01')
+    # combination
+    ret = @cleaner.select {|j| j.queue?(:jobs2) && j.before?('2009-12-01')}
     assert_equal 9, ret.size
 
-    # you can chain with your custom block
-    ret = @cleaner.select &@cleaner.proc{|j| j['payload']['args']==['Jason']}.queue(:jobs2)
+    # combination 2
+    ret = @cleaner.select {|j| j['payload']['args']==['Jason'] && j.queue?(:jobs2)}
     assert_equal 13, ret.size
   end
 

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: resque-cleaner
 version: !ruby/object:Gem::Version 
-  hash: 29
+  hash: 27
   prerelease: false
   segments: 
   - 0
   - 0
-  - 1
-  version: 0.0.1
+  - 2
+  version: 0.0.2
 platform: ruby
 authors: 
 - Tatsuya Ono
@@ -15,7 +15,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2010-11-18 00:00:00 +00:00
+date: 2010-11-19 00:00:00 +00:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -46,6 +46,7 @@ files:
 - README.markdown
 - Rakefile
 - LICENSE
+- lib/resque-cleaner.rb
 - lib/resque_cleaner.rb
 - test/redis-test.conf
 - test/resque_cleaner_test.rb