resque-cleaner 0.0.2 → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/CHANGELOG.md +5 -0
- data/README.markdown +41 -20
- data/lib/resque_cleaner.rb +3 -2
- data/test/resque_cleaner_test.rb +9 -0
- metadata +7 -5
data/CHANGELOG.md
ADDED
data/README.markdown
CHANGED
@@ -8,14 +8,14 @@ Description
|
|
8
8
|
-----------
|
9
9
|
|
10
10
|
ResqueCleaner is a [Resque](https://github.com/defunkt/resque) plugin which
|
11
|
-
helps you to deal with failed jobs
|
11
|
+
helps you to deal with failed jobs on Resque by:
|
12
12
|
|
13
13
|
* Showing stats of failed jobs
|
14
14
|
* Retrying failed jobs
|
15
15
|
* Removing failed jobs
|
16
16
|
* Filtering failed jobs
|
17
17
|
|
18
|
-
Although ResqueCleaner
|
18
|
+
Although ResqueCleaner has not integrated with Resque's web-based interface yet,
|
19
19
|
it is pretty easy to use on irb(console).
|
20
20
|
|
21
21
|
|
@@ -23,6 +23,7 @@ Installation
|
|
23
23
|
------------
|
24
24
|
|
25
25
|
Install as a gem:
|
26
|
+
|
26
27
|
$ gem install resque-cleaner
|
27
28
|
|
28
29
|
|
@@ -53,9 +54,11 @@ You could also group them by class.
|
|
53
54
|
=> {'BadJob' => 3, ...}
|
54
55
|
|
55
56
|
You can get the ones filtered with a block: it targets only jobs which the block
|
56
|
-
evaluetes true.
|
57
|
+
evaluetes true.
|
58
|
+
|
59
|
+
e.g. Show stats only of jobs entried with some arguments:
|
57
60
|
|
58
|
-
> cleaner.stats_by_date{|
|
61
|
+
> cleaner.stats_by_date {|j| j["payload"]["args"].size > 0}
|
59
62
|
2009/03/13: 3
|
60
63
|
2009/11/13: 7
|
61
64
|
2010/08/13: 11
|
@@ -69,28 +72,32 @@ You can retry all failed jobs with this method.
|
|
69
72
|
> cleaner.requeue
|
70
73
|
|
71
74
|
Of course, you can filter jobs with a block; it requeues only jobs which the
|
72
|
-
block evaluates true.
|
75
|
+
block evaluates true.
|
76
|
+
|
77
|
+
e.g. Retry only jobs with some arguments:
|
73
78
|
|
74
|
-
> cleaner.requeue{
|
79
|
+
> cleaner.requeue {|j| j["payload"]["args"].size > 0}
|
75
80
|
|
76
81
|
The job hash is extended with a module which defines some useful methods. You
|
77
|
-
can use it in the blcok.
|
82
|
+
can use it in the blcok.
|
83
|
+
|
84
|
+
e.g. Retry only jobs entried within a day:
|
78
85
|
|
79
86
|
> cleaner.requeue {|j| j.after?(1.day.ago)}
|
80
87
|
|
81
88
|
e.g. Retry EmailJob entried with arguments within 3 days:
|
82
89
|
|
83
|
-
> cleaner.requeue {|j| j
|
90
|
+
> cleaner.requeue {|j| j.after?(3.days.ago) && j.klass?(EmailJob) && j["payload"]["args"].size>0}
|
84
91
|
|
85
|
-
See Helper Methods bellow for more.
|
92
|
+
See Helper Methods section bellow for more information.
|
86
93
|
|
87
94
|
NOTE:
|
88
95
|
[1.day.ago](https://github.com/rails/rails/blob/master/activesupport/lib/active_support/core_ext/numeric/time.rb)
|
89
|
-
is not in standard library. It is equivalent to `Time.now - 60*60*24*3`.
|
96
|
+
is not in standard library. Using it for making explanation more understandable. It is equivalent to `Time.now - 60*60*24*3`.
|
90
97
|
|
91
98
|
**Clear Jobs**
|
92
99
|
|
93
|
-
You can clear all failed jobs with this method
|
100
|
+
You can clear all failed jobs with this method:
|
94
101
|
|
95
102
|
> cleaner.clear
|
96
103
|
|
@@ -109,10 +116,25 @@ some examples:
|
|
109
116
|
**Retry and Clear Jobs**
|
110
117
|
|
111
118
|
You can retry(requeue) and clear failed jobs at the same time; just pass true
|
112
|
-
as an argument.
|
119
|
+
as an argument.
|
120
|
+
|
121
|
+
e.g. Retry EmailJob and remove from failed jobs:
|
113
122
|
|
114
123
|
> cleaner.requeue(true) {|j| j.klass?(EmailJob)}
|
115
124
|
|
125
|
+
**Retry with other queue**
|
126
|
+
|
127
|
+
You can requeue failed jobs into other queue. In this way, you can retry failed
|
128
|
+
jobs without blocking jobs being entried by your service running in the live.
|
129
|
+
|
130
|
+
e.g. Retry failed jobs on :retry queue
|
131
|
+
|
132
|
+
> cleaner.requeue(false, :queue => :retry)
|
133
|
+
|
134
|
+
Don't forget to launch resque worker for the queue.
|
135
|
+
|
136
|
+
% QUEUE=retry rake resque:work
|
137
|
+
|
116
138
|
**Select Jobs**
|
117
139
|
|
118
140
|
You can just select the jobs of course. Here are some examples:
|
@@ -123,7 +145,7 @@ You can just select the jobs of course. Here are some examples:
|
|
123
145
|
|
124
146
|
**Helper Methods**
|
125
147
|
|
126
|
-
Here is a list of methods a job
|
148
|
+
Here is a list of methods a failed job ratained through ResqueCleaner has:
|
127
149
|
|
128
150
|
retried?: returns true if the job has already been retried.
|
129
151
|
requeued?: alias of retried?.
|
@@ -136,8 +158,7 @@ Here is a list of methods a job extended.
|
|
136
158
|
Failed Job
|
137
159
|
-----------
|
138
160
|
|
139
|
-
|
140
|
-
filtering failed jobs.
|
161
|
+
Here is a sample of failed jobs:
|
141
162
|
|
142
163
|
{"failed_at": "2009/03/13 00:00:00",
|
143
164
|
"payload": {"args": ["Johnson"], "class": "BadJob"},
|
@@ -165,16 +186,16 @@ process but on your Redis, it would not respond ages if you try to deal with all
|
|
165
186
|
of those jobs.
|
166
187
|
|
167
188
|
ResqueCleaner supposes recent jobs are more important than old jobs. Therefore
|
168
|
-
ResqueCleaner deals with **ONLY LAST X(default=1000)
|
169
|
-
|
170
|
-
|
171
|
-
|
189
|
+
ResqueCleaner deals with **ONLY LAST X(default=1000) JOBS**. In this way, you
|
190
|
+
could avoid slow responses. You can change the number through `limiter` attribute.
|
191
|
+
|
192
|
+
Let's see how it works with an follwing example.
|
172
193
|
|
173
194
|
**Sample Situation**
|
174
195
|
|
175
196
|
* Number of failed jobs: 100,000
|
176
197
|
|
177
|
-
Default limiter is 1000 so that
|
198
|
+
Default limiter is 1000 so that the limiter returns 1000 as a count.
|
178
199
|
|
179
200
|
> cleaner.limiter.count
|
180
201
|
=> 1,000
|
data/lib/resque_cleaner.rb
CHANGED
@@ -94,8 +94,9 @@ module Resque
|
|
94
94
|
end
|
95
95
|
|
96
96
|
# Retries every jobs for which block evaluates to true.
|
97
|
-
def requeue(clear_after_requeue=false, &block)
|
97
|
+
def requeue(clear_after_requeue=false, options={}, &block)
|
98
98
|
requeued = 0
|
99
|
+
queue = options["queue"] || options[:queue]
|
99
100
|
@limiter.lock do
|
100
101
|
@limiter.jobs.each_with_index do |job,i|
|
101
102
|
if !block_given? || block.call(job)
|
@@ -111,7 +112,7 @@ module Resque
|
|
111
112
|
redis.lset(:failed, @limiter.start_index+i, Resque.encode(job))
|
112
113
|
end
|
113
114
|
|
114
|
-
Job.create(job['queue'], job['payload']['class'], *job['payload']['args'])
|
115
|
+
Job.create(queue||job['queue'], job['payload']['class'], *job['payload']['args'])
|
115
116
|
requeued += 1
|
116
117
|
end
|
117
118
|
end
|
data/test/resque_cleaner_test.rb
CHANGED
@@ -97,6 +97,15 @@ context "ResqueCleaner" do
|
|
97
97
|
assert_equal 0, @cleaner.select.size
|
98
98
|
end
|
99
99
|
|
100
|
+
test "#requeue with :queue option requeues the jobs to the queue" do
|
101
|
+
assert_equal 0, queue_size(:jobs,:jobs2,:retry)
|
102
|
+
requeued = @cleaner.requeue false, :queue => :retry
|
103
|
+
assert_equal 42, requeued
|
104
|
+
assert_equal 42, @cleaner.select.size # it doesn't clear jobs
|
105
|
+
assert_equal 0, queue_size(:jobs,:jobs2)
|
106
|
+
assert_equal 42, queue_size(:retry)
|
107
|
+
end
|
108
|
+
|
100
109
|
test "#clear_stale deletes failure jobs which is queued before the last x enqueued" do
|
101
110
|
@cleaner.limiter.maximum = 10
|
102
111
|
@cleaner.clear_stale
|
metadata
CHANGED
@@ -5,9 +5,9 @@ version: !ruby/object:Gem::Version
|
|
5
5
|
prerelease: false
|
6
6
|
segments:
|
7
7
|
- 0
|
8
|
+
- 1
|
8
9
|
- 0
|
9
|
-
|
10
|
-
version: 0.0.2
|
10
|
+
version: 0.1.0
|
11
11
|
platform: ruby
|
12
12
|
authors:
|
13
13
|
- Tatsuya Ono
|
@@ -15,7 +15,7 @@ autorequire:
|
|
15
15
|
bindir: bin
|
16
16
|
cert_chain: []
|
17
17
|
|
18
|
-
date: 2010-11-
|
18
|
+
date: 2010-11-24 00:00:00 +00:00
|
19
19
|
default_executable:
|
20
20
|
dependencies:
|
21
21
|
- !ruby/object:Gem::Dependency
|
@@ -33,7 +33,7 @@ dependencies:
|
|
33
33
|
version: "1.0"
|
34
34
|
type: :runtime
|
35
35
|
version_requirements: *id001
|
36
|
-
description: "
|
36
|
+
description: " Resque helper plugin cleaning up failed jobs with some neat features such as filtering, retrying, removing and showing stats.\n"
|
37
37
|
email: ononoma@gmail.com
|
38
38
|
executables: []
|
39
39
|
|
@@ -42,8 +42,10 @@ extensions: []
|
|
42
42
|
extra_rdoc_files:
|
43
43
|
- LICENSE
|
44
44
|
- README.markdown
|
45
|
+
- CHANGELOG.md
|
45
46
|
files:
|
46
47
|
- README.markdown
|
48
|
+
- CHANGELOG.md
|
47
49
|
- Rakefile
|
48
50
|
- LICENSE
|
49
51
|
- lib/resque-cleaner.rb
|
@@ -84,6 +86,6 @@ rubyforge_project:
|
|
84
86
|
rubygems_version: 1.3.7
|
85
87
|
signing_key:
|
86
88
|
specification_version: 3
|
87
|
-
summary:
|
89
|
+
summary: Resque plugin cleaning up failed jobs.
|
88
90
|
test_files: []
|
89
91
|
|