resque-cleanerer 0.5.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 169e0b85f5b87c2b4fe7a19a763ab2b8a2db96f5625021b2a1888b71dd670e26
+  data.tar.gz: b3246d131619b8f8ff3612aa87b1958fe81df8b31f117a94681804dca3d56e35
+SHA512:
+  metadata.gz: a4c8a7104f3251aa8dc3e25823265cb8d7c4cef6b070e9f9e3cdef652792928f95d237fd19f6f61c8c412f43ec4f09db4745a4c2e1f19917f43ac781adb25ce7
+  data.tar.gz: 5fb2ca512f0b3f06628e6a4f9b7a5044257a36c717ba107d06bd6ff45ef43b05a3c495e51cc02f2d1b5867018fdc496be968b7b2df0aa54b4a7f67cdf56c2fac

data/CHANGELOG.md

@@ -0,0 +1,92 @@
+## 0.4.1 (2018-01-25)
+
+* Remove Requeue version lock (#46)
+
+## 0.4.0 (2017-11-20)
+
+* Render CSS as ERB partial to make Rails 5 compatible (#45)
+
+## 0.3.2 (2016-03-09)
+
+* Handle failure JSON with nil on payload safely (#40)
+
+## 0.3.1 (2016-01-28)
+
+* Bug fix: fix the issue regex is not applied on bulk clearance (#36)
+
+## 0.3.0 (2014-05-27)
+
+* Search by regex (#27)
+* Show stats by exception (#29)
+* Stop supporting ruby 1.8.x and 1.9.2
+* Escape query parameters (#30)
+
+## 0.2.12 (2013-12-03)
+
+* Remove Resque::Helpers include (#23)
+* Tweak Gemspec (#24)
+* Don't use yaml format to show arguments
+* Tweak README
+
+## 0.2.11 (2013-07-19)
+
+* Use transaction for retry-and-clear (#22).
+* Fix for CI
+
+## 0.2.10 (2012-10-15)
+
+* Bug fix: use URL helper to support resque-web which is not hosted under '/'.
+
+## 0.2.9 (2012-05-10)
+
+* Make limiter configurable on resque-web.
+
+## 0.2.8 (2012-03-19)
+
+* UI tweak (#15)
+
+## 0.2.7 (2012-01-17)
+
+* Support Travis CI
+* Support failure jobs without payload information (#11).
+
+## 0.2.6 (2011-08-23)
+
+* Follow the latest resque for date format.
+
+## 0.2.5 (2011-05-05)
+
+* BUGFIX: Pagination has been broken.
+* Dump a list as JSON format.
+
+## 0.2.4 (2011-04-19)
+
+* BUGFIX: "Select All" ignores exception filter.
+
+## 0.2.3 (2011-04-11)
+
+* Exception filter.
+
+## 0.2.2 (2011-04-07)
+
+* Changed a way to load yajl/json\_gem.
+
+## 0.2.1 (2011-04-06)
+
+* BUGFIX: Bulk action didn't work properly when you select all jobs.
+* Removed absolute paths from html.
+
+## 0.2.0 (2011-04-06)
+
+* Extended with resque-web
+
+## 0.1.1 (2010-12-30)
+
+* Fixed for ruby 1.9.2
+* Fixed a bug on #retried? method
+
+## 0.1.0 (2010-11-24)
+
+* First official release
+
+

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2010 Tatsuya Ono
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,333 @@
+ResqueCleaner
+=============
+
+[github.com/ono/resque-cleaner](https://github.com/ono/resque-cleaner)
+
+This project is not active.
+Please read [this](https://github.com/ono/resque-cleaner/issues/47) before you use the library.
+
+Description
+-----------
+
+ResqueCleaner is a [Resque](https://github.com/defunkt/resque) plugin which
+aims to help you to clean up failed jobs on Resque by:
+
+* Showing stats of failed jobs
+* Retrying failed jobs
+* Removing failed jobs
+* Filtering failed jobs
+
+
+Installation
+------------
+
+Install as a gem:
+
+    $ gem install resque-cleaner
+
+
+Resque-Web integration
+----------------------
+
+![Screen 1](https://github.com/ono/resque-cleaner/raw/master/misc/resque-cleaner-main.png)
+![Screen 2](misc/resque-cleaner-list-with-regex.png)
+
+
+Configuration
+-------------
+
+At first, you have to load ResqueCleaner to enable the Cleaner tab. Here is
+an example step.
+
+1. Create a configuration file for resque-web
+<br/>```touch [app_dir]/config/resque-web.rb```
+
+2. Add the following line into the file
+<br/>```require 'resque-cleaner'```
+
+3. Then pass the file when you start resque-web
+<br/>```% resque-web [app_dir]/config/resque-web.rb```
+
+You can also configure [limiter](https://github.com/ono/resque-cleaner#limiter)
+in the file.
+
+e.g.
+
+```ruby
+require 'resque-cleaner'
+module Resque::Plugins
+  ResqueCleaner::Limiter.default_maximum = 10_000
+end
+```
+
+Console
+-------
+
+Hopefully a situation of your failed jobs is simple enough to get figured out through
+the web interface. But, if not, a powerful filtering feature of ResqueCleaner may help
+you to understand what is going on with your console(irb).
+
+**Create Instance**
+
+```ruby
+    > cleaner = Resque::Plugins::ResqueCleaner.new
+```
+
+**Show Stats**
+
+Shows stats of failed jobs grouped by date.
+
+```ruby
+    > cleaner.stats_by_date
+    2009/03/13:    6
+    2009/11/13:   14
+    2010/08/13:   22
+         total:   42
+    => {'2009/03/10' => 6, ...}
+```
+
+You could also group them by class.
+
+```ruby
+    > cleaner.stats_by_class
+         BadJob:    3
+    HorribleJob:    7
+          total:   10
+    => {'BadJob' => 3, ...}
+```
+
+Or you could also group them by exception.
+
+```ruby
+    > cleaner.stats_by_exception
+	 RuntimeError:   35
+    SyntaxError:    7
+          total:   42
+    => {'RuntimeError' => 35, ...}
+```
+
+You can get the ones filtered with a block: it targets only jobs which the block
+evaluates true.
+
+e.g. Show stats only of jobs entered with some arguments:
+
+```ruby
+    > cleaner.stats_by_date {|j| j["payload"]["args"].size > 0}
+    2009/03/13:    3
+    2009/11/13:    7
+    2010/08/13:   11
+         total:   22
+    => {'2009/03/10' => 3, ...}
+```
+
+A convenience helper for searching for failed jobs which satisfy a
+regular expression:
+
+```ruby
+cleaner.select_by_regex(/Jason/) # => all failed jobs that have "Jason" in them
+```
+
+**Retry(Requeue) Jobs**
+
+You can retry all failed jobs with this method.
+
+```ruby
+    > cleaner.requeue
+```
+
+Of course, you can filter jobs with a block; it requeues only jobs which the
+block evaluates true. 
+
+e.g. Retry only jobs with some arguments:
+
+```ruby
+    > cleaner.requeue {|j| j["payload"]["args"].size > 0}
+```
+
+The job hash is extended with a module which defines some useful methods. You
+can use it in the block.
+
+e.g. Retry only jobs entered within a day:
+
+```ruby
+    > cleaner.requeue {|j| j.after?(1.day.ago)}
+```
+
+e.g. Retry EmailJob entered with arguments within 3 days:
+
+```ruby
+    > cleaner.requeue {|j| j.after?(3.days.ago) && j.klass?(EmailJob) && j["payload"]["args"].size>0}
+```
+
+See Helper Methods section bellow for more information.
+
+NOTE:
+[1.day.ago](https://github.com/rails/rails/blob/master/activesupport/lib/active_support/core_ext/numeric/time.rb)
+is not in standard library. Using it for making explanation more understandable. It is equivalent to `Time.now - 60*60*24*3`.
+
+**Clear Jobs**
+
+You can clear all failed jobs with this method:
+
+```ruby
+    > cleaner.clear
+```
+
+Like you can do with the retry method, the clear method takes a block. Here are
+some examples:
+
+```ruby
+    > cleaner.clear {|j| j.retried?}
+    => clears all jobs already retried and returns number of the jobs.
+
+    > cleaner.clear {|j| j.queue?(:low) && j.before?('2010-10-10')}
+    => clears all jobs entried in :low queue before 10th October, 2010.
+
+    > cleaner.clear {|j| j.exception?("RuntimeError") && j.queue?(:low)}
+    => clears all jobs raised RuntimeError and queued :low queue
+```
+
+**Retry and Clear Jobs**
+
+You can retry(requeue) and clear failed jobs at the same time; just pass true
+as an argument. 
+
+e.g. Retry EmailJob and remove from failed jobs:
+
+```ruby
+    > cleaner.requeue(true) {|j| j.klass?(EmailJob)}
+```
+
+**Retry with other queue**
+
+You can requeue failed jobs into other queue. In this way, you can retry failed
+jobs without blocking jobs being entered by your service running in the live.
+
+e.g. Retry failed jobs on :retry queue
+
+```ruby
+    > cleaner.requeue(false, :queue => :retry)
+```
+
+Don't forget to launch resque worker for the queue.
+
+    % QUEUE=retry rake resque:work
+
+**Select Jobs**
+
+You can just select the jobs of course. Here are some examples:
+
+```ruby
+    > cleaner.select {|j| j["payload"]["args"][0]=="Johonson"}
+    > cleaner.select {|j| j.after?(2.days.ago)}
+    > cleaner.select #=> returns all jobs
+```
+
+**Helper Methods**
+
+Here is a list of methods a failed job retained through ResqueCleaner has:
+
+    retried?: returns true if the job has already been retried.
+    requeued?: alias of retried?.
+    before?(time): returns true if the job failed before the time.
+    after?(time): returns true if the job failed after the time.
+    klass?(klass_or_name): returns true if the class of job matches.
+    queue?(queue_name): returns true if the queue of job matches.
+    exception?(exception_name): returns true if the exception matches.
+
+
+Failed Job
+-----------
+
+Here is a sample of failed jobs:
+
+    {"failed_at": "2009/03/13 00:00:00",
+     "payload": {"args": ["Johnson"], "class": "BadJob"},
+     "queue": "jobs",
+     "worker": "localhost:7327:jobs,jobs2",
+     "exception": "RuntimeError",
+     "error": "Bad job, Johnson",
+     "backtrace": 
+      ["./test/test_helper.rb:108:in `perform'",
+       "/opt/local/lib/ruby/gems/1.8/gems/resque-1.10.0/lib/resque/job.rb:133:in `perform'",
+       "/opt/local/lib/ruby/gems/1.8/gems/resque-1.10.0/lib/resque/worker.rb:157:in `perform'",
+       "/opt/local/lib/ruby/gems/1.8/gems/resque-1.10.0/lib/resque/worker.rb:124:in `work'",
+       "....(omitted)....",
+       "./test/test_helper.rb:41",
+       "test/resque_cleaner_test.rb:3"]
+    }
+
+
+Limiter
+-------
+
+ResqueCleaner expects a disaster situation like a huge number of failed jobs are
+out there. Since ResqueCleaner's filter function is running on your application
+process but on your Redis, it would not respond ages if you try to deal with all
+of those jobs.
+
+ResqueCleaner supposes recent jobs are more important than old jobs. Therefore
+ResqueCleaner deals with **ONLY LAST X(default=1000) JOBS**. In this way, you
+could avoid slow responses. You can change the number through `limiter` attribute.
+
+Let's see how it works with an following example.
+
+**Sample Situation**
+
+* Number of failed jobs: 100,000
+
+Default limiter is 1000 so that the limiter returns 1000 as a count.
+
+```ruby
+    > cleaner.limiter.count
+    => 1,000
+    > cleaner.failure.count
+    => 100,000
+```
+
+You could know if the limiter is on with on? method.
+
+```ruby
+    > cleaner.limiter.on?
+    => true
+```
+
+You can change the maximum number of the limiter with maximum attribute.
+
+```ruby
+    > cleaner.limiter.maximum = 3000
+    => 3,000
+    > cleaner.limiter.count
+    => 3,000
+    > cleaner.limiter.on?
+    => true
+```
+
+With limiter, ResqueCleaner's filtering targets only the last X(3000 in this
+sample) failed jobs.
+
+```ruby
+    > cleaner.select.size
+    => 3,000
+```
+
+The clear\_stale method deletes all jobs entered prior to the last X(3000 in
+this sample) failed jobs. This calls Redis API and no iteration occurs on Ruby
+application; it should be quick even if there are huge number of failed jobs.
+
+```ruby
+    > cleaner.clear_stale
+    > cleaner.failure.count
+    => 3,000
+    > cleaner.limiter.count
+    => 3,000
+    > cleaner.limiter.on?
+    => false
+```
+
+Many Thanks!
+------------
+
+To our [Contributors](https://github.com/ono/resque-cleaner/contributors)
+
+

data/Rakefile

@@ -0,0 +1,38 @@
+#
+# Setup
+#
+
+# load 'tasks/redis.rake'
+#require 'rake/testtask'
+
+$LOAD_PATH.unshift 'lib'
+#require 'resque/tasks'
+
+def command?(command)
+  system("type #{command} > /dev/null 2>&1")
+end
+
+
+#
+# Tests
+#
+
+task :default => :test
+
+desc "Run the test suite"
+task :test do
+  rg = command?(:rg)
+  Dir['test/**/*_test.rb'].each do |f|
+    rg ? sh("rg #{f}") : ruby(f)
+  end
+end
+
+if command? :kicker
+  desc "Launch Kicker (like autotest)"
+  task :kicker do
+    puts "Kicking... (ctrl+c to cancel)"
+    exec "kicker -e rake test lib examples"
+  end
+end
+
+

data/lib/resque-cleaner.rb

@@ -0,0 +1 @@
+require 'resque_cleaner'

data/lib/resque_cleaner/server/views/_cleaner_styles.erb

@@ -0,0 +1,63 @@
+<style type="text/css" media="screen">
+  #main .cleaner .title h1 { float: left }
+
+  #main .cleaner .sub_header {color: #888; font-size: 80%; font-weight: bold}
+
+  #main .cleaner .control_panel {
+    margin: 5px 0 10px; padding: 5px; background: #eee; display: inline-block;
+    -webkit-border-radius:5px; border:1px solid #ccc;
+  }
+
+  #main .cleaner #exec { color: black; }
+  #main .cleaner #exec select { margin-right: 20px }
+  #main .cleaner #exec a { margin-right: 5px}
+  #main .cleaner #exec a.disabled { color: #aaa}
+
+  #main .cleaner table.class_list { width: auto }
+  #main .cleaner td.number { text-align: right }
+  #main .cleaner tr.total { border-top: 3px double #ccc}
+
+  #main .cleaner form { float: none; margin: 0}
+
+  #main .cleaner .warning { margin: 30px 0; padding: 10px; background: #fdd; display: inline-block; }
+
+  #main .clearfix:after {
+    content: ".";
+    display: block;
+    height: 0;
+    clear: both;
+    visibility: hidden;
+  }
+  #main .cleaner ul.failed {
+    margin-bottom: 10px
+  }
+
+  #main .cleaner .list_info{
+    color: #999; text-align: center;
+  }
+  #main .cleaner .list_info a{
+    color: #666
+  }
+  #main .cleaner .list_info .list_summary{
+    vertical-align: baseline;
+    display: inline-block;
+    margin: 0; padding:0;
+  }
+  #main .cleaner ul.pagination {
+    display: inline-block;
+    vertical-align: baseline;
+    margin:0; padding:0;
+  }
+  #main .cleaner ul.pagination li {
+    list-style:none; float: left; padding: 0 3px;
+  }
+  #main .cleaner ul.pagination li.current {
+    color: #333; font-weight: bold;
+  }
+  #main .cleaner ul.pagination li.summary{
+    padding-right: 10px
+  }
+  #main .cleaner div.dump{
+    float: right
+  }
+</style>

data/lib/resque_cleaner/server/views/_limiter.erb

@@ -0,0 +1,13 @@
+  <div class="warning">
+    <p>
+    There are more than <%= @cleaner.limiter.maximum%> jobs. ResqueCleaner handles only recent <%= @cleaner.limiter.maximum %> jobs. See the Limiter section on <a href="https://github.com/ono/resque-cleaner" target="_blank">README</a> for more detail.
+    </p>
+    <p>
+      <form method="post" action="cleaner_stale">
+        <input type="hidden" name="action" value="clear_stale" />
+        <input type="submit" onclick="return confirm('Are you sure?')" value="Clear all jobs older than <%= @cleaner.limiter.count %>th job."/>
+        (<%= @cleaner.failure.count - @cleaner.limiter.maximum %> jobs will be cleared)
+      </form>
+    </p>
+  </div>
+

data/lib/resque_cleaner/server/views/_paginate.erb

@@ -0,0 +1,54 @@
+<div class="list_info clearfix">
+
+  <ul class="pagination">
+    <li class="summary">
+      <%= @paginate.first_index%> - <%= @paginate.last_index%> / <%= @paginate.total_size%>
+    </li>
+    <% if @paginate.max_page>1 %>
+      <% if @paginate.first_page? %>
+        <li class="prev off">&lt;&lt;newer</li>
+      <% else %>
+        <li class="prev"><a href="<%= @paginate.page_url(:prev)%>">&lt;&lt;newer</a></li>
+      <% end %>
+
+      <% 1.upto(6<@paginate.max_page ? 6 : @paginate.max_page) do |pg| %>
+        <% if @paginate.page==pg %>
+          <li class="current"><%=pg%></li>
+        <% else %>
+          <li><a href="<%= @paginate.page_url(pg)%>"><%=pg%></a></li>
+        <% end %>
+      <% end %>
+
+      <% if @paginate.max_page > 6 %>
+        <% if @paginate.max_page!=@paginate.page %>
+          <% if @paginate.page>7 %>
+            <li class="off">...</li>
+          <% end %>
+          <% if @paginate.page>6 %>
+            <li class="current"><%=@paginate.page%></li>
+          <% end %>
+        <% end %>
+        <% if @paginate.max_page>7 && @paginate.page+1!=@paginate.max_page %>
+          <li class="off">...</li>
+        <% end %>
+
+        <% if @paginate.max_page==@paginate.page %>
+          <li class="current"><%=@paginate.max_page%></li>
+        <% else %>
+          <li><a href="<%= @paginate.page_url(@paginate.max_page)%>"><%=@paginate.max_page%></a></li>
+        <% end %>
+      <% end %>
+
+      <% if @paginate.last_page? %>
+        <li class="next off">older&gt;&gt;</li>
+      <% else %>
+        <li class="next"><a href="<%= @paginate.page_url(:next)%>">older&gt;&gt;</a></li>
+      <% end %>
+    <% end %>
+
+  </ul>
+    <div class="dump">
+        <a href="<%= @dump_url%>" target="_blank">Dump</a>
+    </div>
+</div>
+

data/lib/resque_cleaner/server/views/_stats.erb

@@ -0,0 +1,44 @@
+<table class="class_list">
+  <tr>
+    <th><%= type.gsub('klass','class').capitalize %></th>
+    <th>Failed</th>
+    <th>In last 1 hour</th>
+    <th>In last 3 hours</th>
+    <th>In last 24 hours</th>
+    <th>In last 3 days</th>
+    <th>In last 7 days</th>
+  </tr>
+  <% @stats[type.to_sym].each do |field,count| %>
+    <tr>
+      <% filter = "#{q}=#{CGI.escape(field)}" %>
+      <td><%= field %></td>
+      <td class="number">
+        <a href="cleaner_list?<%=filter%>"><%= count[:total] %></a>
+      </td>
+      <td class="number">
+        <a href="cleaner_list?<%=filter%>&f=1"><%= count[:h1] %></a>
+      </td>
+      <td class="number">
+        <a href="cleaner_list?<%=filter%>&f=3"><%= count[:h3] %></a>
+      </td>
+      <td class="number">
+        <a href="cleaner_list?<%=filter%>&f=24"><%= count[:d1] %></a>
+      </td>
+      <td class="number">
+        <a href="cleaner_list?<%=filter%>&f=72"><%= count[:d3] %></a>
+      </td>
+      <td class="number">
+        <a href="cleaner_list?<%=filter%>&f=168"><%= count[:d7] %></a>
+      </td>
+    </tr>
+  <% end %>
+  <tr class="total">
+    <td>Total</td>
+    <td class="number"><a href="cleaner_list"><%= @total[:total] %></a></td>
+    <td class="number"><a href="cleaner_list?f=1"><%= @total[:h1] %></a></td>
+    <td class="number"><a href="cleaner_list?f=3"><%= @total[:h3] %></a></td>
+    <td class="number"><a href="cleaner_list?f=24"><%= @total[:d1] %></a></td>
+    <td class="number"><a href="cleaner_list?f=72"><%= @total[:d3] %></a></td>
+    <td class="number"><a href="cleaner_list?f=168"><%= @total[:d7] %></a></td>
+  </tr>
+</table>

data/lib/resque_cleaner/server/views/cleaner.erb

@@ -0,0 +1,25 @@
+<%= erb File.read(ResqueCleaner::Server.erb_path("_cleaner_styles.erb")) %>
+
+<div class="cleaner">
+  <div class="title clearfix">
+    <h1>Group by class</h1>
+  </div>
+
+  <%= erb(
+        File.read(ResqueCleaner::Server.erb_path("_stats.erb")),
+        locals: {q: 'c', type: 'klass'}
+      ) %>
+
+  <div class="title clearfix">
+    <h1>Group by exception</h1>
+  </div>
+
+  <%= erb(
+        File.read(ResqueCleaner::Server.erb_path("_stats.erb")),
+        locals: {q: 'ex', type: 'exception'}
+      ) %>
+
+  <% if @cleaner.limiter.on? %>
+    <%= erb File.read(ResqueCleaner::Server.erb_path("_limiter.erb")) %>
+  <% end %>
+</div>