sneakers_handlers 0.0.2 → 0.0.3
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +111 -50
- data/docs/backoff.png +0 -0
- data/lib/sneakers_handlers/exponential_backoff_handler.rb +8 -9
- data/lib/sneakers_handlers/version.rb +1 -1
- metadata +4 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: a9d628c97cd9755d31e310c6fdc8094b0d815d2f
|
4
|
+
data.tar.gz: f739480da18963717a07196e7cfb390acf9f0252
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: fc631d12e28639232b9312de45e3f37195b9ab776a56e80ece1d935f5bb6967dfeabb34c99c06ded86c2657ad2225403576e20b70eb96ee9362b7b12fa286662
|
7
|
+
data.tar.gz: 44cb5f6321ec4c1f66203b486e6468bd470f8d1776da538f793370c12f78c118852fd18787e46ea71550e52d9868b1ea0e21e4783a8c4a18108163ee4db7aefd
|
data/README.md
CHANGED
@@ -1,90 +1,151 @@
|
|
1
1
|
# SneakersHandlers
|
2
2
|
|
3
|
-
|
3
|
+
The gem introduces three handlers you can use as part of your [`Sneakers`](https://github.com/jondot/sneakers) workers:
|
4
4
|
|
5
|
-
|
5
|
+
* `SneakersHandlers::DeadLetterHandler`
|
6
|
+
* `SneakersHandlers::RetryHandler`
|
7
|
+
* `SneakersHandlers::ExponentialBackoffHandler`.
|
6
8
|
|
7
|
-
|
8
|
-
|
9
|
-
Add this line to your application's Gemfile:
|
9
|
+
`Sneakers` handlers are used to define custom behaviours to different scenarios (e.g. a success, error, timeout, etc.).
|
10
10
|
|
11
|
-
|
12
|
-
|
13
|
-
|
11
|
+
By default `Sneakers` uses a handler called [`OneShot`](https://github.com/jondot/sneakers/blob/41883dd0df8b360c8d6e2f29101c960d5650f711/lib/sneakers/handlers/oneshot.rb) that,
|
12
|
+
as the name indicates, will try to execute the message only once, and `reject` it if something goes wrong. That can be fine for some workers, but we usually need something that will be able
|
13
|
+
to handle failed messages in a better way, either by sending them to a [dead-letter exchange](https://www.rabbitmq.com/dlx.html) or by trying to execute them again.
|
14
14
|
|
15
|
-
|
15
|
+
## Using the `DeadLetterHandler`
|
16
16
|
|
17
|
-
|
17
|
+
The `DeadLetterHandler` is an extension of the default `OneShot` handler. It will try to process the message only once, and when something goes wrong it will publish this message to the dead letter exchange.
|
18
18
|
|
19
|
-
|
19
|
+
When defining your worker, you have to define these extra arguments:
|
20
20
|
|
21
|
-
|
21
|
+
`x-dead-letter-exchange`: The name of the dead-letter exchange where failed messages will be published to.
|
22
22
|
|
23
|
-
|
23
|
+
`x-dead-letter-routing-key`: The routing key that will be used when dead-lettering a failed message. This value needs to be unique to your
|
24
|
+
application to avoid having the same message delivered to multiple queues. The recommendation is to use the queue name, although that's not mandatory.
|
24
25
|
|
25
|
-
|
26
|
+
Here's an example:
|
26
27
|
|
27
|
-
|
28
|
+
```diff
|
29
|
+
class DeadLetterWorker
|
30
|
+
include Sneakers::Worker
|
28
31
|
|
29
|
-
|
32
|
+
from_queue "sneakers_handlers.my_queue",
|
33
|
+
ack: true,
|
34
|
+
exchange: "sneakers_handlers",
|
35
|
+
exchange_type: :topic,
|
36
|
+
routing_key: "sneakers_handlers.dead_letter_test",
|
37
|
+
+ handler: SneakersHandlers::DeadLetterHandler,
|
38
|
+
+ arguments: { "x-dead-letter-exchange" => "sneakers_handlers.dlx",
|
39
|
+
+ "x-dead-letter-routing-key" => "sneakers_handlers.my_queue" }
|
40
|
+
|
41
|
+
def work(*args)
|
42
|
+
ack!
|
43
|
+
end
|
44
|
+
end
|
45
|
+
```
|
30
46
|
|
31
|
-
|
47
|
+
## Using the `RetryHandler`
|
32
48
|
|
33
|
-
`
|
49
|
+
The `RetryHandler` will try to execute the message `max_retry` times before dead-lettering it. The setup is very similar to the `DeadLetterHandler`, the only difference if that you can
|
50
|
+
also provide a `max_retry` argument, that will specify how many times the handler should try to execute this message.
|
34
51
|
|
35
|
-
```
|
36
|
-
class
|
52
|
+
```diff
|
53
|
+
class RetryWorker
|
37
54
|
include Sneakers::Worker
|
38
|
-
|
39
|
-
|
55
|
+
|
56
|
+
from_queue "sneakers_handlers.my_queue",
|
40
57
|
ack: true,
|
41
|
-
exchange: "
|
58
|
+
exchange: "sneaker_handlers",
|
42
59
|
exchange_type: :topic,
|
43
|
-
routing_key: "
|
44
|
-
|
45
|
-
|
46
|
-
|
47
|
-
|
48
|
-
|
49
|
-
def work(
|
50
|
-
|
60
|
+
routing_key: "sneakers_handlers.retry_test",
|
61
|
+
+ handler: SneakersHandlers::RetryHandler,
|
62
|
+
+ max_retry: 50,
|
63
|
+
+ arguments: { "x-dead-letter-exchange" => "sneakers_handlers.dlx",
|
64
|
+
+ "x-dead-letter-routing-key" => "sneakers_handlers.my_queue" }
|
65
|
+
|
66
|
+
def work(*args)
|
67
|
+
ack!
|
51
68
|
end
|
52
69
|
end
|
53
70
|
```
|
54
71
|
|
55
|
-
|
72
|
+
When a message fails, it will be published back to the end of the queue, so, assuming the queue is empty, there will be no delay (other than the network latency) between these retries.
|
56
73
|
|
57
|
-
|
58
|
-
static configurable backoff. Plan on updating the name in the future.
|
74
|
+
## Using the `ExponentialBackoffHandler`
|
59
75
|
|
60
|
-
|
76
|
+
With this handler every retry is delayed by a power of 2 on the attempt number. The retry attempt is inserted into a new queue with a naming convention of `<queue name>.retry.<delay>`.
|
77
|
+
After exhausting the maximum number of retries (`max_retries`), the message will be moved into the dead letter exchange.
|
61
78
|
|
62
|
-
|
79
|
+
![backoff](https://github.com/alphasights/sneakers_handlers/blob/master/docs/backoff.png)
|
63
80
|
|
64
|
-
|
65
|
-
|
81
|
+
The setup is also very similar to the other handlers:
|
82
|
+
|
83
|
+
```diff
|
84
|
+
class ExponentialBackoffWorker
|
85
|
+
include Sneakers::Worker
|
86
|
+
|
87
|
+
from_queue "sneakers_handlers.my_queue",
|
88
|
+
ack: true,
|
89
|
+
exchange: "sneaker_handlers",
|
90
|
+
exchange_type: :topic,
|
91
|
+
routing_key: "sneakers_handlers.backoff_test",
|
92
|
+
+ handler: SneakersHandlers::ExponentialBackoffHandler,
|
93
|
+
+ max_retries: 50,
|
94
|
+
+ arguments: { "x-dead-letter-exchange" => "sneakers_handlers.dlx",
|
95
|
+
+ "x-dead-letter-routing-key" => "sneakers_handlers.my_queue" }
|
96
|
+
|
97
|
+
def work(*args)
|
98
|
+
ack!
|
99
|
+
end
|
100
|
+
end
|
101
|
+
```
|
102
|
+
|
103
|
+
You can also customize the backoff function defining the `backoff_function` option, that can be any `call`able object (a lambda, a method, a class that responds to `call`, etc.)
|
104
|
+
that will receive the current attempt count and should return in how many seconds the message will be retried.
|
105
|
+
|
106
|
+
```diff
|
107
|
+
class ExponentialBackoffWorker
|
66
108
|
include Sneakers::Worker
|
67
|
-
|
68
|
-
|
109
|
+
|
110
|
+
from_queue "sneakers_handlers.my_queue",
|
69
111
|
ack: true,
|
70
|
-
exchange: "
|
112
|
+
exchange: "sneaker_handlers",
|
71
113
|
exchange_type: :topic,
|
72
|
-
routing_key: "
|
114
|
+
routing_key: "sneakers_handlers.backoff_test",
|
73
115
|
handler: SneakersHandlers::ExponentialBackoffHandler,
|
74
|
-
|
116
|
+
+ backoff_function: ->(attempt_number) { attempt_number ** 3 },
|
117
|
+
max_retries: 50,
|
118
|
+
arguments: { "x-dead-letter-exchange" => "sneakers_handlers.dlx",
|
119
|
+
"x-dead-letter-routing-key" => "sneakers_handlers.my_queue" }
|
75
120
|
|
76
|
-
def work(
|
77
|
-
|
121
|
+
def work(*args)
|
122
|
+
ack!
|
78
123
|
end
|
79
124
|
end
|
80
125
|
```
|
81
126
|
|
82
|
-
|
127
|
+
For a more detailed explanation of how the backoff handler works, check out the [blog post](https://m.alphasights.com/exponential-backoff-with-rabbitmq-78386b9bec81) we wrote about it.
|
128
|
+
|
129
|
+
## Installation
|
83
130
|
|
84
|
-
|
131
|
+
Add this line to your application's Gemfile:
|
85
132
|
|
86
|
-
|
133
|
+
```ruby
|
134
|
+
gem 'sneakers_handlers'
|
135
|
+
```
|
87
136
|
|
88
|
-
|
137
|
+
And then execute:
|
138
|
+
|
139
|
+
$ bundle
|
140
|
+
|
141
|
+
Or install it yourself as:
|
142
|
+
|
143
|
+
$ gem install sneakers_handlers
|
144
|
+
|
145
|
+
For a more detailed explanation of how the backoff handler works, check out the [blog post](https://m.alphasights.com/exponential-backoff-with-rabbitmq-78386b9bec81) we wrote about it.
|
146
|
+
|
147
|
+
## Development
|
148
|
+
|
149
|
+
After checking out the repository, run `bin/setup` to install dependencies. Then, run `rake` to run the tests (you will need to have a real `RabbitMQ` instance running). You can also run `bin/console` for an interactive prompt that will allow you to experiment.
|
89
150
|
|
90
|
-
|
151
|
+
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
data/docs/backoff.png
ADDED
Binary file
|
@@ -23,15 +23,17 @@
|
|
23
23
|
|
24
24
|
module SneakersHandlers
|
25
25
|
class ExponentialBackoffHandler
|
26
|
-
attr_reader :queue, :channel, :options, :max_retries
|
26
|
+
attr_reader :queue, :channel, :options, :max_retries, :backoff_function
|
27
27
|
|
28
28
|
DEFAULT_MAX_RETRY_ATTEMPTS = 25
|
29
|
+
DEFAULT_BACKOFF_FUNCTION = -> (attempt_number) { (attempt_number + 1) ** 2 }
|
29
30
|
|
30
31
|
def initialize(channel, queue, options)
|
31
32
|
@queue = queue
|
32
33
|
@channel = channel
|
33
34
|
@options = options
|
34
35
|
@max_retries = options[:max_retries] || DEFAULT_MAX_RETRY_ATTEMPTS
|
36
|
+
@backoff_function = options[:backoff_function] || DEFAULT_BACKOFF_FUNCTION
|
35
37
|
|
36
38
|
create_error_exchange!
|
37
39
|
|
@@ -47,7 +49,7 @@ module SneakersHandlers
|
|
47
49
|
end
|
48
50
|
|
49
51
|
def error(delivery_info, properties, message, err)
|
50
|
-
retry_message(delivery_info, properties, message, err)
|
52
|
+
retry_message(delivery_info, properties, message, err.inspect)
|
51
53
|
end
|
52
54
|
|
53
55
|
def timeout(delivery_info, properties, message)
|
@@ -63,7 +65,7 @@ module SneakersHandlers
|
|
63
65
|
attempt_number = death_count(properties[:headers])
|
64
66
|
|
65
67
|
if attempt_number < max_retries
|
66
|
-
delay =
|
68
|
+
delay = backoff_function.call(attempt_number)
|
67
69
|
|
68
70
|
log("msg=retrying, delay=#{delay}, count=#{attempt_number}, properties=#{properties}, reason=#{reason}")
|
69
71
|
|
@@ -72,7 +74,8 @@ module SneakersHandlers
|
|
72
74
|
retry_queue = create_retry_queue!(delay)
|
73
75
|
retry_queue.bind(primary_exchange, routing_key: routing_key)
|
74
76
|
|
75
|
-
|
77
|
+
headers = (properties[:headers] || {}).merge(rejection_reason: reason.to_s)
|
78
|
+
primary_exchange.publish(message, routing_key: routing_key, headers: headers)
|
76
79
|
acknowledge(delivery_info, properties, message)
|
77
80
|
else
|
78
81
|
log("msg=erroring, count=#{attempt_number}, properties=#{properties}")
|
@@ -118,7 +121,7 @@ module SneakersHandlers
|
|
118
121
|
|
119
122
|
def create_retry_queue!(delay)
|
120
123
|
clear_queues_cache
|
121
|
-
channel.queue(
|
124
|
+
channel.queue("#{queue.name}.retry.#{delay}",
|
122
125
|
durable: options[:queue_options][:durable],
|
123
126
|
arguments: {
|
124
127
|
:"x-dead-letter-exchange" => options[:exchange],
|
@@ -129,10 +132,6 @@ module SneakersHandlers
|
|
129
132
|
)
|
130
133
|
end
|
131
134
|
|
132
|
-
def seconds_to_delay(count)
|
133
|
-
(count + 1) ** 2
|
134
|
-
end
|
135
|
-
|
136
135
|
# When we create a new queue, `Bunny` stores its name in an internal cache.
|
137
136
|
# The problem is that as we are creating ephemeral queues that can expire shortly
|
138
137
|
# after they are created, this cached queue may not exist anymore when we try to
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: sneakers_handlers
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.3
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- John Bohn, Abe Petrillo, Brian Storti
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2017-01-10 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: sneakers
|
@@ -110,6 +110,7 @@ files:
|
|
110
110
|
- bin/rake
|
111
111
|
- bin/setup
|
112
112
|
- circle.yml
|
113
|
+
- docs/backoff.png
|
113
114
|
- lib/sneakers_handlers.rb
|
114
115
|
- lib/sneakers_handlers/dead_letter_handler.rb
|
115
116
|
- lib/sneakers_handlers/exponential_backoff_handler.rb
|
@@ -136,7 +137,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
136
137
|
version: '0'
|
137
138
|
requirements: []
|
138
139
|
rubyforge_project:
|
139
|
-
rubygems_version: 2.
|
140
|
+
rubygems_version: 2.5.1
|
140
141
|
signing_key:
|
141
142
|
specification_version: 4
|
142
143
|
summary: Adds Handlers to use with Sneakers
|