resque-cleaner 0.0.2 → 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- data/CHANGELOG.md +5 -0
- data/README.markdown +41 -20
- data/lib/resque_cleaner.rb +3 -2
- data/test/resque_cleaner_test.rb +9 -0
- metadata +7 -5
data/CHANGELOG.md
ADDED
data/README.markdown
CHANGED
@@ -8,14 +8,14 @@ Description
|
|
8
8
|
-----------
|
9
9
|
|
10
10
|
ResqueCleaner is a [Resque](https://github.com/defunkt/resque) plugin which
|
11
|
-
helps you to deal with failed jobs
|
11
|
+
helps you to deal with failed jobs on Resque by:
|
12
12
|
|
13
13
|
* Showing stats of failed jobs
|
14
14
|
* Retrying failed jobs
|
15
15
|
* Removing failed jobs
|
16
16
|
* Filtering failed jobs
|
17
17
|
|
18
|
-
Although ResqueCleaner
|
18
|
+
Although ResqueCleaner has not integrated with Resque's web-based interface yet,
|
19
19
|
it is pretty easy to use on irb(console).
|
20
20
|
|
21
21
|
|
@@ -23,6 +23,7 @@ Installation
|
|
23
23
|
------------
|
24
24
|
|
25
25
|
Install as a gem:
|
26
|
+
|
26
27
|
$ gem install resque-cleaner
|
27
28
|
|
28
29
|
|
@@ -53,9 +54,11 @@ You could also group them by class.
|
|
53
54
|
=> {'BadJob' => 3, ...}
|
54
55
|
|
55
56
|
You can get the ones filtered with a block: it targets only jobs which the block
|
56
|
-
evaluetes true.
|
57
|
+
evaluetes true.
|
58
|
+
|
59
|
+
e.g. Show stats only of jobs entried with some arguments:
|
57
60
|
|
58
|
-
> cleaner.stats_by_date{|
|
61
|
+
> cleaner.stats_by_date {|j| j["payload"]["args"].size > 0}
|
59
62
|
2009/03/13: 3
|
60
63
|
2009/11/13: 7
|
61
64
|
2010/08/13: 11
|
@@ -69,28 +72,32 @@ You can retry all failed jobs with this method.
|
|
69
72
|
> cleaner.requeue
|
70
73
|
|
71
74
|
Of course, you can filter jobs with a block; it requeues only jobs which the
|
72
|
-
block evaluates true.
|
75
|
+
block evaluates true.
|
76
|
+
|
77
|
+
e.g. Retry only jobs with some arguments:
|
73
78
|
|
74
|
-
> cleaner.requeue{
|
79
|
+
> cleaner.requeue {|j| j["payload"]["args"].size > 0}
|
75
80
|
|
76
81
|
The job hash is extended with a module which defines some useful methods. You
|
77
|
-
can use it in the blcok.
|
82
|
+
can use it in the blcok.
|
83
|
+
|
84
|
+
e.g. Retry only jobs entried within a day:
|
78
85
|
|
79
86
|
> cleaner.requeue {|j| j.after?(1.day.ago)}
|
80
87
|
|
81
88
|
e.g. Retry EmailJob entried with arguments within 3 days:
|
82
89
|
|
83
|
-
> cleaner.requeue {|j| j
|
90
|
+
> cleaner.requeue {|j| j.after?(3.days.ago) && j.klass?(EmailJob) && j["payload"]["args"].size>0}
|
84
91
|
|
85
|
-
See Helper Methods bellow for more.
|
92
|
+
See Helper Methods section bellow for more information.
|
86
93
|
|
87
94
|
NOTE:
|
88
95
|
[1.day.ago](https://github.com/rails/rails/blob/master/activesupport/lib/active_support/core_ext/numeric/time.rb)
|
89
|
-
is not in standard library. It is equivalent to `Time.now - 60*60*24*3`.
|
96
|
+
is not in standard library. Using it for making explanation more understandable. It is equivalent to `Time.now - 60*60*24*3`.
|
90
97
|
|
91
98
|
**Clear Jobs**
|
92
99
|
|
93
|
-
You can clear all failed jobs with this method
|
100
|
+
You can clear all failed jobs with this method:
|
94
101
|
|
95
102
|
> cleaner.clear
|
96
103
|
|
@@ -109,10 +116,25 @@ some examples:
|
|
109
116
|
**Retry and Clear Jobs**
|
110
117
|
|
111
118
|
You can retry(requeue) and clear failed jobs at the same time; just pass true
|
112
|
-
as an argument.
|
119
|
+
as an argument.
|
120
|
+
|
121
|
+
e.g. Retry EmailJob and remove from failed jobs:
|
113
122
|
|
114
123
|
> cleaner.requeue(true) {|j| j.klass?(EmailJob)}
|
115
124
|
|
125
|
+
**Retry with other queue**
|
126
|
+
|
127
|
+
You can requeue failed jobs into other queue. In this way, you can retry failed
|
128
|
+
jobs without blocking jobs being entried by your service running in the live.
|
129
|
+
|
130
|
+
e.g. Retry failed jobs on :retry queue
|
131
|
+
|
132
|
+
> cleaner.requeue(false, :queue => :retry)
|
133
|
+
|
134
|
+
Don't forget to launch resque worker for the queue.
|
135
|
+
|
136
|
+
% QUEUE=retry rake resque:work
|
137
|
+
|
116
138
|
**Select Jobs**
|
117
139
|
|
118
140
|
You can just select the jobs of course. Here are some examples:
|
@@ -123,7 +145,7 @@ You can just select the jobs of course. Here are some examples:
|
|
123
145
|
|
124
146
|
**Helper Methods**
|
125
147
|
|
126
|
-
Here is a list of methods a job
|
148
|
+
Here is a list of methods a failed job ratained through ResqueCleaner has:
|
127
149
|
|
128
150
|
retried?: returns true if the job has already been retried.
|
129
151
|
requeued?: alias of retried?.
|
@@ -136,8 +158,7 @@ Here is a list of methods a job extended.
|
|
136
158
|
Failed Job
|
137
159
|
-----------
|
138
160
|
|
139
|
-
|
140
|
-
filtering failed jobs.
|
161
|
+
Here is a sample of failed jobs:
|
141
162
|
|
142
163
|
{"failed_at": "2009/03/13 00:00:00",
|
143
164
|
"payload": {"args": ["Johnson"], "class": "BadJob"},
|
@@ -165,16 +186,16 @@ process but on your Redis, it would not respond ages if you try to deal with all
|
|
165
186
|
of those jobs.
|
166
187
|
|
167
188
|
ResqueCleaner supposes recent jobs are more important than old jobs. Therefore
|
168
|
-
ResqueCleaner deals with **ONLY LAST X(default=1000)
|
169
|
-
|
170
|
-
|
171
|
-
|
189
|
+
ResqueCleaner deals with **ONLY LAST X(default=1000) JOBS**. In this way, you
|
190
|
+
could avoid slow responses. You can change the number through `limiter` attribute.
|
191
|
+
|
192
|
+
Let's see how it works with an follwing example.
|
172
193
|
|
173
194
|
**Sample Situation**
|
174
195
|
|
175
196
|
* Number of failed jobs: 100,000
|
176
197
|
|
177
|
-
Default limiter is 1000 so that
|
198
|
+
Default limiter is 1000 so that the limiter returns 1000 as a count.
|
178
199
|
|
179
200
|
> cleaner.limiter.count
|
180
201
|
=> 1,000
|
data/lib/resque_cleaner.rb
CHANGED
@@ -94,8 +94,9 @@ module Resque
|
|
94
94
|
end
|
95
95
|
|
96
96
|
# Retries every jobs for which block evaluates to true.
|
97
|
-
def requeue(clear_after_requeue=false, &block)
|
97
|
+
def requeue(clear_after_requeue=false, options={}, &block)
|
98
98
|
requeued = 0
|
99
|
+
queue = options["queue"] || options[:queue]
|
99
100
|
@limiter.lock do
|
100
101
|
@limiter.jobs.each_with_index do |job,i|
|
101
102
|
if !block_given? || block.call(job)
|
@@ -111,7 +112,7 @@ module Resque
|
|
111
112
|
redis.lset(:failed, @limiter.start_index+i, Resque.encode(job))
|
112
113
|
end
|
113
114
|
|
114
|
-
Job.create(job['queue'], job['payload']['class'], *job['payload']['args'])
|
115
|
+
Job.create(queue||job['queue'], job['payload']['class'], *job['payload']['args'])
|
115
116
|
requeued += 1
|
116
117
|
end
|
117
118
|
end
|
data/test/resque_cleaner_test.rb
CHANGED
@@ -97,6 +97,15 @@ context "ResqueCleaner" do
|
|
97
97
|
assert_equal 0, @cleaner.select.size
|
98
98
|
end
|
99
99
|
|
100
|
+
test "#requeue with :queue option requeues the jobs to the queue" do
|
101
|
+
assert_equal 0, queue_size(:jobs,:jobs2,:retry)
|
102
|
+
requeued = @cleaner.requeue false, :queue => :retry
|
103
|
+
assert_equal 42, requeued
|
104
|
+
assert_equal 42, @cleaner.select.size # it doesn't clear jobs
|
105
|
+
assert_equal 0, queue_size(:jobs,:jobs2)
|
106
|
+
assert_equal 42, queue_size(:retry)
|
107
|
+
end
|
108
|
+
|
100
109
|
test "#clear_stale deletes failure jobs which is queued before the last x enqueued" do
|
101
110
|
@cleaner.limiter.maximum = 10
|
102
111
|
@cleaner.clear_stale
|
metadata
CHANGED
@@ -5,9 +5,9 @@ version: !ruby/object:Gem::Version
|
|
5
5
|
prerelease: false
|
6
6
|
segments:
|
7
7
|
- 0
|
8
|
+
- 1
|
8
9
|
- 0
|
9
|
-
|
10
|
-
version: 0.0.2
|
10
|
+
version: 0.1.0
|
11
11
|
platform: ruby
|
12
12
|
authors:
|
13
13
|
- Tatsuya Ono
|
@@ -15,7 +15,7 @@ autorequire:
|
|
15
15
|
bindir: bin
|
16
16
|
cert_chain: []
|
17
17
|
|
18
|
-
date: 2010-11-
|
18
|
+
date: 2010-11-24 00:00:00 +00:00
|
19
19
|
default_executable:
|
20
20
|
dependencies:
|
21
21
|
- !ruby/object:Gem::Dependency
|
@@ -33,7 +33,7 @@ dependencies:
|
|
33
33
|
version: "1.0"
|
34
34
|
type: :runtime
|
35
35
|
version_requirements: *id001
|
36
|
-
description: "
|
36
|
+
description: " Resque helper plugin cleaning up failed jobs with some neat features such as filtering, retrying, removing and showing stats.\n"
|
37
37
|
email: ononoma@gmail.com
|
38
38
|
executables: []
|
39
39
|
|
@@ -42,8 +42,10 @@ extensions: []
|
|
42
42
|
extra_rdoc_files:
|
43
43
|
- LICENSE
|
44
44
|
- README.markdown
|
45
|
+
- CHANGELOG.md
|
45
46
|
files:
|
46
47
|
- README.markdown
|
48
|
+
- CHANGELOG.md
|
47
49
|
- Rakefile
|
48
50
|
- LICENSE
|
49
51
|
- lib/resque-cleaner.rb
|
@@ -84,6 +86,6 @@ rubyforge_project:
|
|
84
86
|
rubygems_version: 1.3.7
|
85
87
|
signing_key:
|
86
88
|
specification_version: 3
|
87
|
-
summary:
|
89
|
+
summary: Resque plugin cleaning up failed jobs.
|
88
90
|
test_files: []
|
89
91
|
|