onyx-resque-retry 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,33 @@
1
+ ## 0.1.0 (2010-08-29)
2
+
3
+ * Feature: Multiple failure backend with retry suppression.
4
+ * Feature: resque-web tab showing retry information.
5
+ * Improved README documentation, added a 'Quick Start' section.
6
+
7
+ ## 0.0.6 (2010-07-12)
8
+
9
+ * Feature: Added support for custom retry criteria check callbacks.
10
+
11
+ ## 0.0.5 (2010-06-27)
12
+
13
+ * Handle our own dependancies.
14
+
15
+ ## 0.0.4 (2010-06-16)
16
+
17
+ * Relax gemspec dependancies.
18
+
19
+ ## 0.0.3 (2010-06-02)
20
+
21
+ * Bugfix: Make sure that `redis_retry_key` has no whitespace.
22
+
23
+ ## 0.0.2 (2010-05-06)
24
+
25
+ * Bugfix: Were calling non-existent method to delete redis key.
26
+ * Delay no-longer falls back to `sleep`. resque-scheduler is a required
27
+ dependancy.
28
+ * Redis key doesn't include ending colon `:` if no args were passed
29
+ to the job.
30
+
31
+ ## 0.0.1 (2010-04-27)
32
+
33
+ * First release.
data/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ Copyright (c) 2010 Luke Antins
2
+ Copyright (c) 2010 Ryan Carver
3
+
4
+ Permission is hereby granted, free of charge, to any person obtaining
5
+ a copy of this software and associated documentation files (the
6
+ Software), to deal in the Software without restriction, including
7
+ without limitation the rights to use, copy, modify, merge, publish,
8
+ distribute, sublicense, and/or sell copies of the Software, and to
9
+ permit persons to whom the Software is furnished to do so, subject to
10
+ the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be
13
+ included in all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND,
16
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
17
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
18
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
19
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
20
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
21
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,289 @@
1
+ resque-retry
2
+ ============
3
+
4
+ A [Resque][rq] plugin. Requires Resque 1.8.0 & [resque-scheduler][rqs].
5
+
6
+ resque-retry provides retry, delay and exponential backoff support for
7
+ resque jobs.
8
+
9
+ * Redis backed retry count/limit.
10
+ * Retry on all or specific exceptions.
11
+ * Exponential backoff (varying the delay between retrys).
12
+ * Multiple failure backend with retry suppression & resque-web tab.
13
+ * Small & Extendable - plenty of places to override retry logic/settings.
14
+
15
+ Install & Quick Start
16
+ ---------------------
17
+
18
+ To install:
19
+
20
+ $ gem install resque-retry
21
+
22
+ Add this to your `Rakefile`:
23
+
24
+ require 'resque/tasks'
25
+ require 'resque_scheduler/tasks'
26
+
27
+ The delay between retry attempts is provided by [resque-scheduler][rqs].
28
+ You'll want to run the scheduler process, otherwise delayed retry attempts
29
+ will never perform:
30
+
31
+ $ rake resque:scheduler
32
+
33
+ Use the plugin:
34
+
35
+ require 'resque-retry'
36
+
37
+ class ExampleRetryJob
38
+ extend Resque::Plugins::Retry
39
+ @queue = :example_queue
40
+
41
+ @retry_limit = 3
42
+ @retry_delay = 60
43
+
44
+ def self.perform(*args)
45
+ # your magic/heavy lifting goes here.
46
+ end
47
+ end
48
+
49
+ Then start up a resque worker as normal:
50
+
51
+ $ QUEUE=* rake resque:work
52
+
53
+ Now if you ExampleRetryJob fails, it will be retried 3 times, with a 60 second
54
+ delay between attempts.
55
+
56
+ For more explanation and examples, please see the remaining documentation.
57
+
58
+ Failure Backend & Resque Web Additions
59
+ --------------------------------------
60
+
61
+ Lets say your using the Redis failure backend of resque (the default).
62
+ Every time a job fails, the failure queue is populated with the job and
63
+ exception details.
64
+
65
+ Normally this is useful, but if your jobs retry... it can cause a bit of a mess.
66
+
67
+ For example: given a job that retried 4 times before completing successful.
68
+ You'll have a lot of failures for the same job and you wont be sure if it
69
+ actually completed successfully just by just using the resque-web interface.
70
+
71
+ ### Failure Backend
72
+
73
+ `MultipleWithRetrySuppression` is a multiple failure backend, with retry suppression.
74
+
75
+ Here's an example, using the Redis failure backend:
76
+
77
+ require 'resque-retry'
78
+ require 'resque/failure/redis'
79
+
80
+ # require your jobs & application code.
81
+
82
+ Resque::Failure::MultipleWithRetrySuppression.classes = [Resque::Failure::Redis]
83
+ Resque::Failure.backend = Resque::Failure::MultipleWithRetrySuppression
84
+
85
+ If a job fails, but **can and will** retry, the failure details wont be
86
+ logged in the Redis failed queue *(visible via resque-web)*.
87
+
88
+ If the job fails, but **can't or won't** retry, the failure will be logged in
89
+ the Redis failed queue, like a normal failure *(without retry)* would.
90
+
91
+ ### Resque Web Additions
92
+
93
+ If your using the `MultipleWithRetrySuppression` failure backend, you should
94
+ also checkout the resque-web additions!
95
+
96
+ The new Retry tab displays delayed jobs with retry information; the number of
97
+ attempts and the exception details from the last failure.
98
+
99
+ Make sure you include this in your `config.ru` or similar file:
100
+
101
+ require 'resque-retry'
102
+ require 'resque-retry/server'
103
+
104
+ # require your jobs & application code.
105
+
106
+ run Resque::Server.new
107
+
108
+ Retry Options & Logic
109
+ ---------------------
110
+
111
+ Please take a look at the yardoc/code for more details on methods you may
112
+ wish to override.
113
+
114
+ Customisation is pretty easy, the below examples should give you
115
+ some ideas =), adapt for your own usage and feel free to pick and mix!
116
+
117
+ ### Retry Defaults
118
+
119
+ Retry the job **once** on failure, with zero delay.
120
+
121
+ require 'resque-retry'
122
+
123
+ class DeliverWebHook
124
+ extend Resque::Plugins::Retry
125
+ @queue = :web_hooks
126
+
127
+ def self.perform(url, hook_id, hmac_key)
128
+ heavy_lifting
129
+ end
130
+ end
131
+
132
+ When a job runs, the number of retry attempts is checked and incremented
133
+ in Redis. If your job fails, the number of retry attempts is used to
134
+ determine if we can requeue the job for another go.
135
+
136
+ ### Custom Retry
137
+
138
+ class DeliverWebHook
139
+ extend Resque::Plugins::Retry
140
+ @queue = :web_hooks
141
+
142
+ @retry_limit = 10
143
+ @retry_delay = 120
144
+
145
+ def self.perform(url, hook_id, hmac_key)
146
+ heavy_lifting
147
+ end
148
+ end
149
+
150
+ The above modification will allow your job to retry upto 10 times, with
151
+ a delay of 120 seconds, or 2 minutes between retry attempts.
152
+
153
+ Alternatively you could override the `retry_delay` method to do something
154
+ more special.
155
+
156
+ ### Exponential Backoff
157
+
158
+ Use this if you wish to vary the delay between retry attempts:
159
+
160
+ class DeliverSMS
161
+ extend Resque::Plugins::ExponentialBackoff
162
+ @queue = :mt_messages
163
+
164
+ def self.perform(mt_id, mobile_number, message)
165
+ heavy_lifting
166
+ end
167
+ end
168
+
169
+ **Default Settings**
170
+
171
+ key: m = minutes, h = hours
172
+
173
+ no delay, 1m, 10m, 1h, 3h, 6h
174
+ @backoff_strategy = [0, 60, 600, 3600, 10800, 21600]
175
+
176
+ The first delay will be 0 seconds, the 2nd will be 60 seconds, etc...
177
+ Again, tweak to your own needs.
178
+
179
+ The number if retrys is equal to the size of the `backoff_strategy`
180
+ array, unless you set `retry_limit` yourself.
181
+
182
+ ### Retry Specific Exceptions
183
+
184
+ The default will allow a retry for any type of exception. You may change
185
+ it so only specific exceptions are retried using `retry_exceptions`:
186
+
187
+ class DeliverSMS
188
+ extend Resque::Plugins::Retry
189
+ @queue = :mt_messages
190
+
191
+ @retry_exceptions = [NetworkError]
192
+
193
+ def self.perform(mt_id, mobile_number, message)
194
+ heavy_lifting
195
+ end
196
+ end
197
+
198
+ The above modification will **only** retry if a `NetworkError` (or subclass)
199
+ exception is thrown.
200
+
201
+ ### Custom Retry Criteria Check Callbacks
202
+
203
+ You may define custom retry criteria callbacks:
204
+
205
+ class TurkWorker
206
+ extend Resque::Plugins::Retry
207
+ @queue = :turk_job_processor
208
+
209
+ @retry_exceptions = [NetworkError]
210
+
211
+ retry_criteria_check do |exception, *args|
212
+ if exception.message =~ /InvalidJobId/
213
+ false # don't retry if we got passed a invalid job id.
214
+ else
215
+ true # its okay for a retry attempt to continue.
216
+ end
217
+ end
218
+
219
+ def self.perform(job_id)
220
+ heavy_lifting
221
+ end
222
+ end
223
+
224
+ Similar to the previous example, this job will retry if either a
225
+ `NetworkError` (or subclass) exception is thrown **or** any of the callbacks
226
+ return true.
227
+
228
+ Use `@retry_exceptions = []` to **only** use callbacks, to determine if the
229
+ job should retry.
230
+
231
+ ### Retry Arguments
232
+
233
+ You may override `args_for_retry`, which is passed the current
234
+ job arguments, to modify the arguments for the next retry attempt.
235
+
236
+ class DeliverViaSMSC
237
+ extend Resque::Plugins::Retry
238
+ @queue = :mt_smsc_messages
239
+
240
+ # retry using the emergency SMSC.
241
+ def self.args_for_retry(smsc_id, mt_message)
242
+ [999, mt_message]
243
+ end
244
+
245
+ self.perform(smsc_id, mt_message)
246
+ heavy_lifting
247
+ end
248
+ end
249
+
250
+ ### Job Identifier/Key
251
+
252
+ The retry attempt is incremented and stored in a Redis key. The key is
253
+ built using the `identifier`. If you have a lot of arguments or really long
254
+ ones, you should consider overriding `identifier` to define a more precise
255
+ or loose custom identifier.
256
+
257
+ The default identifier is just your job arguments joined with a dash `-`.
258
+
259
+ By default the key uses this format:
260
+ `resque-retry:<job class name>:<identifier>`.
261
+
262
+ Or you can define the entire key by overriding `redis_retry_key`.
263
+
264
+ class DeliverSMS
265
+ extend Resque::Plugins::Retry
266
+ @queue = :mt_messages
267
+
268
+ def self.identifier(mt_id, mobile_number, message)
269
+ "#{mobile_number}:#{mt_id}"
270
+ end
271
+
272
+ self.perform(mt_id, mobile_number, message)
273
+ heavy_lifting
274
+ end
275
+ end
276
+
277
+ Contributing/Pull Requests
278
+ --------------------------
279
+
280
+ * Yes please!
281
+ * Fork the project.
282
+ * Make your feature addition or bug fix.
283
+ * Add tests for it.
284
+ * Commit.
285
+ * Send me a pull request. Bonus points for topic branches.
286
+ * If you edit the gemspec/version etc, do it in another commit please.
287
+
288
+ [rq]: http://github.com/defunkt/resque
289
+ [rqs]: http://github.com/bvandenbos/resque-scheduler
@@ -0,0 +1,25 @@
1
+ $LOAD_PATH.unshift 'lib'
2
+
3
+ require 'rake/testtask'
4
+ require 'fileutils'
5
+ require 'yard'
6
+ require 'yard/rake/yardoc_task'
7
+
8
+ task :default => :test
9
+
10
+ ##
11
+ # Test task.
12
+ Rake::TestTask.new(:test) do |task|
13
+ task.test_files = FileList['test/*_test.rb']
14
+ task.verbose = true
15
+ end
16
+
17
+ ##
18
+ # docs task.
19
+ YARD::Rake::YardocTask.new :yardoc do |t|
20
+ t.files = ['lib/**/*.rb']
21
+ t.options = ['--output-dir', "doc/",
22
+ '--files', 'LICENSE',
23
+ '--readme', 'README.md',
24
+ '--title', 'resque-retry documentation']
25
+ end
@@ -0,0 +1,6 @@
1
+ require 'resque'
2
+ require 'resque_scheduler'
3
+
4
+ require 'resque/plugins/retry'
5
+ require 'resque/plugins/exponential_backoff'
6
+ require 'resque/failure/multiple_with_retry_suppression'
@@ -0,0 +1,51 @@
1
+ # Extend Resque::Server to add tabs.
2
+ module ResqueRetry
3
+ module Server
4
+
5
+ def self.included(base)
6
+ base.class_eval {
7
+ helpers do
8
+ # builds a retry key for the specified job.
9
+ def retry_key_for_job(job)
10
+ klass = Resque.constantize(job['class'])
11
+ if klass.respond_to?(:redis_retry_key)
12
+ klass.redis_retry_key(job['args'])
13
+ else
14
+ nil
15
+ end
16
+ end
17
+
18
+ # gets the number of retry attempts for a job.
19
+ def retry_attempts_for_job(job)
20
+ Resque.redis.get(retry_key_for_job(job))
21
+ end
22
+
23
+ # gets the failure details hash for a job.
24
+ def retry_failure_details(retry_key)
25
+ Resque.decode(Resque.redis["failure_#{retry_key}"])
26
+ end
27
+
28
+ # reads a 'local' template file.
29
+ def local_template(path)
30
+ # Is there a better way to specify alternate template locations with sinatra?
31
+ File.read(File.join(File.dirname(__FILE__), "server/views/#{path}"))
32
+ end
33
+ end
34
+
35
+ get '/retry' do
36
+ erb local_template('retry.erb')
37
+ end
38
+
39
+ get '/retry/:timestamp' do
40
+ erb local_template('retry_timestamp.erb')
41
+ end
42
+ }
43
+ end
44
+
45
+ end
46
+ end
47
+
48
+ Resque::Server.tabs << 'Retry'
49
+ Resque::Server.class_eval do
50
+ include ResqueRetry::Server
51
+ end
@@ -0,0 +1,48 @@
1
+ <h1>Delayed Jobs with Retry Information</h1>
2
+
3
+ <p class="intro">
4
+ This list below contains the timestamps for scheduled delayed jobs with
5
+ retry information.
6
+ </p>
7
+
8
+ <p class="sub">
9
+ Showing <%= start = params[:start].to_i %> to <%= start + 20 %> of
10
+ <b><%= size = resque.delayed_queue_schedule_size %></b> timestamps
11
+ </p>
12
+
13
+ <table>
14
+ <tr>
15
+ <th></th>
16
+ <th>Timestamp</th>
17
+ <th>Job count</th>
18
+ <th>Class</th>
19
+ <th>Args</th>
20
+ <th>Retry Attempts</th>
21
+ </tr>
22
+ <% timestamps = resque.delayed_queue_peek(start, start+20) %>
23
+ <% timestamps.each do |timestamp| %>
24
+ <% job = resque.delayed_timestamp_peek(timestamp, 0, 1).first %>
25
+ <% retry_key = retry_key_for_job(job) %>
26
+ <tr>
27
+ <td>
28
+ <form action="<%= url "/delayed/queue_now" %>" method="post">
29
+ <input type="hidden" name="timestamp" value="<%= timestamp.to_i %>">
30
+ <input type="submit" value="Queue now">
31
+ </form>
32
+ </td>
33
+ <td><a href="<%= url "retry/#{timestamp}" %>"><%= format_time(Time.at(timestamp)) %></a></td>
34
+ <td><%= delayed_timestamp_size = resque.delayed_timestamp_size(timestamp) %></td>
35
+ <% if job && delayed_timestamp_size == 1 %>
36
+ <td><%= h job['class'] %></td>
37
+ <td><%= h job['args'].inspect %></td>
38
+ <td><%= retry_attempts_for_job(job) || '<i>n/a</i>' %></td>
39
+ <% else %>
40
+ <td><a href="<%= url "retry/#{timestamp}" %>">see details</a></td>
41
+ <td></td>
42
+ <td></td>
43
+ <% end %>
44
+ </tr>
45
+ <% end %>
46
+ </table>
47
+
48
+ <%= partial :next_more, :start => start, :size => size %>