sneakers_handlers 0.0.2 → 0.0.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f8282f82b86f3812de85b874ad7f000ff38cc382
-  data.tar.gz: eac3f28db8e8f77678d48e621d446bc986615216
+  metadata.gz: a9d628c97cd9755d31e310c6fdc8094b0d815d2f
+  data.tar.gz: f739480da18963717a07196e7cfb390acf9f0252
 SHA512:
-  metadata.gz: 8618303e16ee9775d7c8e60e71928e55cfad85a7840f53bd6d912aaa8ca5f148bf011a2148ed22412ee3458a21b8cf8b89e6f96b8590f2e613d977f8e90ef88c
-  data.tar.gz: 08408381176a5f5e420aa2b3115ba6ab6559e89361c84c27423e2c322699c9a29dfc535affc9a5f67ccfffb6f9047b60b2d2c063961f91f68471641cc2665af6
+  metadata.gz: fc631d12e28639232b9312de45e3f37195b9ab776a56e80ece1d935f5bb6967dfeabb34c99c06ded86c2657ad2225403576e20b70eb96ee9362b7b12fa286662
+  data.tar.gz: 44cb5f6321ec4c1f66203b486e6468bd470f8d1776da538f793370c12f78c118852fd18787e46ea71550e52d9868b1ea0e21e4783a8c4a18108163ee4db7aefd

data/README.md

@@ -1,90 +1,151 @@
 # SneakersHandlers
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/sneakers_handlers`. To experiment with that code, run `bin/console` for an interactive prompt.
+The gem introduces three handlers you can use as part of your [`Sneakers`](https://github.com/jondot/sneakers) workers: 
 
-TODO: Delete this and the text above, and describe your gem
+* `SneakersHandlers::DeadLetterHandler`
+* `SneakersHandlers::RetryHandler` 
+* `SneakersHandlers::ExponentialBackoffHandler`.
 
-## Installation
-
-Add this line to your application's Gemfile:
+`Sneakers` handlers are used to define custom behaviours to different scenarios (e.g. a success, error, timeout, etc.). 
 
-```ruby
-gem 'sneakers_handlers'
-```
+By default `Sneakers` uses a handler called [`OneShot`](https://github.com/jondot/sneakers/blob/41883dd0df8b360c8d6e2f29101c960d5650f711/lib/sneakers/handlers/oneshot.rb) that,
+as the name indicates, will try to execute the message only once, and `reject` it if something goes wrong. That can be fine for some workers, but we usually need something that will be able
+to handle failed messages in a better way, either by sending them to a [dead-letter exchange](https://www.rabbitmq.com/dlx.html) or by trying to execute them again.
 
-And then execute:
+## Using the `DeadLetterHandler`
 
-    $ bundle
+The `DeadLetterHandler` is an extension of the default `OneShot` handler. It will try to process the message only once, and when something goes wrong it will publish this message to the dead letter exchange.
 
-Or install it yourself as:
+When defining your worker, you have to define these extra arguments:
 
-    $ gem install sneakers_handlers
+`x-dead-letter-exchange`: The name of the dead-letter exchange where failed messages will be published to.
 
-## Usage
+`x-dead-letter-routing-key`: The routing key that will be used when dead-lettering a failed message. This value needs to be unique to your
+application to avoid having the same message delivered to multiple queues. The recommendation is to use the queue name, although that's not mandatory.
 
-The gem introduces two handlers you can use as part of your sneaker workers: `SneakersHandlers::DeadLetterHandler` and `SneakersHandlers::RetryHandler`
+Here's an example:
 
-## Using the `SneakersHandlers::RetryHandler` handler
+```diff
+class DeadLetterWorker
+  include Sneakers::Worker
 
-When defining your worker, you have the following extra options:
+  from_queue "sneakers_handlers.my_queue",
+    ack: true,
+    exchange: "sneakers_handlers",
+    exchange_type: :topic,
+    routing_key: "sneakers_handlers.dead_letter_test",
++   handler: SneakersHandlers::DeadLetterHandler,
++   arguments: { "x-dead-letter-exchange" => "sneakers_handlers.dlx",
++                "x-dead-letter-routing-key" => "sneakers_handlers.my_queue" }
+
+  def work(*args)
+    ack!
+  end
+end
+```
 
-`max_retry` [optional] : The number of times a message will be processed after a rejection
+## Using the `RetryHandler`
 
-`x-dead-letter-routing-key` [mandatory] : The routing key that the retry queue will be bound to, usually it is the name of the original queue, for example:
+The `RetryHandler` will try to execute the message `max_retry` times before dead-lettering it. The setup is very similar to the `DeadLetterHandler`, the only difference if that you can
+also provide a `max_retry` argument, that will specify how many times the handler should try to execute this message.
 
-```ruby
-class MyWorker
+```diff
+class RetryWorker
   include Sneakers::Worker
-  from_queue "my-app.resource_processor",
-      durable: true,
+
+  from_queue "sneakers_handlers.my_queue",
       ack: true,
-      exchange: "domain_events",
+      exchange: "sneaker_handlers",
       exchange_type: :topic,
-      routing_key: "resources.lifecycle.*",
-      handler: Sneakers::Handlers::RetryHandler,
-      max_retry: 6,
-      arguments: { "x-dead-letter-exchange" => "domain_events.dlx",
-                   "x-dead-letter-routing-key" => "my-app.resource_processor" }
-
-  def work(payload)
-    ...
+      routing_key: "sneakers_handlers.retry_test",
++     handler: SneakersHandlers::RetryHandler,
++     max_retry: 50,
++     arguments: { "x-dead-letter-exchange" => "sneakers_handlers.dlx",
++                  "x-dead-letter-routing-key" => "sneakers_handlers.my_queue" }
+
+  def work(*args)
+    ack!
   end
 end
 ```
 
-## Using the `SneakersHandlers::ExponentialBackoffHandler` handler
+When a message fails, it will be published back to the end of the queue, so, assuming the queue is empty, there will be no delay (other than the network latency) between these retries.
 
-Exponential Backoff isn't really the right phrase for this. It's more
-static configurable backoff. Plan on updating the name in the future.
+## Using the `ExponentialBackoffHandler`
 
-When defining your worker, you have the following extra options:
+With this handler every retry is delayed by a power of 2 on the attempt number. The retry attempt is inserted into a new queue with a naming convention of `<queue name>.retry.<delay>`.
+After exhausting the maximum number of retries (`max_retries`), the message will be moved into the dead letter exchange.
 
-`delay` [required] : An array containing the number of seconds to pause between retries
+![backoff](https://github.com/alphasights/sneakers_handlers/blob/master/docs/backoff.png)
 
-```ruby
-class MyWorker
+The setup is also very similar to the other handlers:
+
+```diff
+class ExponentialBackoffWorker
+  include Sneakers::Worker
+
+  from_queue "sneakers_handlers.my_queue",
+      ack: true,
+      exchange: "sneaker_handlers",
+      exchange_type: :topic,
+      routing_key: "sneakers_handlers.backoff_test",
++     handler: SneakersHandlers::ExponentialBackoffHandler,
++     max_retries: 50,
++     arguments: { "x-dead-letter-exchange" => "sneakers_handlers.dlx",
++                  "x-dead-letter-routing-key" => "sneakers_handlers.my_queue" }
+
+  def work(*args)
+    ack!
+  end
+end
+```
+
+You can also customize the backoff function defining the `backoff_function` option, that can be any `call`able object (a lambda, a method, a class that responds to `call`, etc.)
+that will receive the current attempt count and should return in how many seconds the message will be retried. 
+
+```diff
+class ExponentialBackoffWorker
   include Sneakers::Worker
-  from_queue "my-app.resource_processor",
-      durable: true,
+
+  from_queue "sneakers_handlers.my_queue",
       ack: true,
-      exchange: "domain_events",
+      exchange: "sneaker_handlers",
       exchange_type: :topic,
-      routing_key: "resources.lifecycle.*",
+      routing_key: "sneakers_handlers.backoff_test",
       handler: SneakersHandlers::ExponentialBackoffHandler,
-      delay: [1.second, 10.seconds, 1.minute, 10.minutes]
++     backoff_function: ->(attempt_number) { attempt_number ** 3 },
+      max_retries: 50,
+      arguments: { "x-dead-letter-exchange" => "sneakers_handlers.dlx",
+                   "x-dead-letter-routing-key" => "sneakers_handlers.my_queue" }
 
-  def work(payload)
-    ...
+  def work(*args)
+    ack!
   end
 end
 ```
 
-## Development
+For a more detailed explanation of how the backoff handler works, check out the [blog post](https://m.alphasights.com/exponential-backoff-with-rabbitmq-78386b9bec81) we wrote about it.
+
+## Installation
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+Add this line to your application's Gemfile:
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+```ruby
+gem 'sneakers_handlers'
+```
 
-## Contributing
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install sneakers_handlers
+
+For a more detailed explanation of how the backoff handler works, check out the [blog post](https://m.alphasights.com/exponential-backoff-with-rabbitmq-78386b9bec81) we wrote about it.
+
+## Development
+
+After checking out the repository, run `bin/setup` to install dependencies. Then, run `rake` to run the tests (you will need to have a real `RabbitMQ` instance running). You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/sneakers_handlers.
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).

data/lib/sneakers_handlers/exponential_backoff_handler.rb

@@ -23,15 +23,17 @@
 
 module SneakersHandlers
   class ExponentialBackoffHandler
-    attr_reader :queue, :channel, :options, :max_retries
+    attr_reader :queue, :channel, :options, :max_retries, :backoff_function
 
     DEFAULT_MAX_RETRY_ATTEMPTS = 25
+    DEFAULT_BACKOFF_FUNCTION = -> (attempt_number) { (attempt_number + 1) ** 2 }
 
     def initialize(channel, queue, options)
       @queue = queue
       @channel = channel
       @options = options
       @max_retries = options[:max_retries] || DEFAULT_MAX_RETRY_ATTEMPTS
+      @backoff_function  = options[:backoff_function] || DEFAULT_BACKOFF_FUNCTION
 
       create_error_exchange!
 
@@ -47,7 +49,7 @@ module SneakersHandlers
     end
 
     def error(delivery_info, properties, message, err)
-      retry_message(delivery_info, properties, message, err)
+      retry_message(delivery_info, properties, message, err.inspect)
     end
 
     def timeout(delivery_info, properties, message)
@@ -63,7 +65,7 @@ module SneakersHandlers
       attempt_number = death_count(properties[:headers])
 
       if attempt_number < max_retries
-        delay = seconds_to_delay(attempt_number)
+        delay = backoff_function.call(attempt_number)
 
         log("msg=retrying, delay=#{delay}, count=#{attempt_number}, properties=#{properties}, reason=#{reason}")
 
@@ -72,7 +74,8 @@ module SneakersHandlers
         retry_queue = create_retry_queue!(delay)
         retry_queue.bind(primary_exchange, routing_key: routing_key)
 
-        primary_exchange.publish(message, routing_key: routing_key, headers: properties[:headers])
+        headers = (properties[:headers] || {}).merge(rejection_reason: reason.to_s)
+        primary_exchange.publish(message, routing_key: routing_key, headers: headers)
         acknowledge(delivery_info, properties, message)
       else
         log("msg=erroring, count=#{attempt_number}, properties=#{properties}")
@@ -118,7 +121,7 @@ module SneakersHandlers
 
     def create_retry_queue!(delay)
       clear_queues_cache
-      channel.queue( "#{queue.name}.retry.#{delay}",
+      channel.queue("#{queue.name}.retry.#{delay}",
          durable: options[:queue_options][:durable],
          arguments: {
            :"x-dead-letter-exchange" => options[:exchange],
@@ -129,10 +132,6 @@ module SneakersHandlers
         )
     end
 
-    def seconds_to_delay(count)
-      (count + 1) ** 2
-    end
-
     # When we create a new queue, `Bunny` stores its name in an internal cache.
     # The problem is that as we are creating ephemeral queues that can expire shortly
     # after they are created, this cached queue may not exist anymore when we try to

data/lib/sneakers_handlers/version.rb

@@ -1,3 +1,3 @@
 module SneakersHandlers
-  VERSION = "0.0.2"
+  VERSION = "0.0.3"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sneakers_handlers
 version: !ruby/object:Gem::Version
-  version: 0.0.2
+  version: 0.0.3
 platform: ruby
 authors:
 - John Bohn, Abe Petrillo, Brian Storti
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-06-03 00:00:00.000000000 Z
+date: 2017-01-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sneakers
@@ -110,6 +110,7 @@ files:
 - bin/rake
 - bin/setup
 - circle.yml
+- docs/backoff.png
 - lib/sneakers_handlers.rb
 - lib/sneakers_handlers/dead_letter_handler.rb
 - lib/sneakers_handlers/exponential_backoff_handler.rb
@@ -136,7 +137,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.4.5.1
+rubygems_version: 2.5.1
 signing_key: 
 specification_version: 4
 summary: Adds Handlers to use with Sneakers