pwwka 0.12.0 → 0.13.0.RC1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 73aae4703124cb6373853ed75313f9d009edb86e
-  data.tar.gz: 0c88c9dcff661db1d434999fd8e2231717c6187e
+  metadata.gz: c804b87530f9d0b29e22b2a10131c8e7ef122cf2
+  data.tar.gz: 0a7b10536a20463505d055ac9797664320a58dbf
 SHA512:
-  metadata.gz: 728c2006b6a317656a23154132ac440d39218174ab8e655312ac069064941cb557606f0d6fc7329df1c0b3480701798c7deb02db81b3ae7ca3442de7c3e89e60
-  data.tar.gz: d71062618705b340a75b378ca8af5b4ab0168bd90a24b62320d23592cc863a2a749877f5fce16d0a11558ffe318d4205046d64a9770b0ef348d321c71d347730
+  metadata.gz: 3c68568f681edce5f795676d3ed340f1d2ae5f52f333c1546c967cc3fa788f71b568b7abfe94b670013c825f0a8cbe8b6399574e9ebc7eddaada252c98f045fc
+  data.tar.gz: 0b61ce8f1fa5c070d11f09b41cf494aaa4e3e32a1793395f498237dfea94feedc267f449ce4b6a31d97bc0fd34592e029cd4b89c4e24849f534fbd28542998b3

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    pwwka (0.12.0)
+    pwwka (0.13.0.RC1)
       activemodel
       activesupport
       bunny
@@ -10,9 +10,9 @@ PATH
 GEM
   remote: https://www.rubygems.org/
   specs:
-    activemodel (5.1.3)
-      activesupport (= 5.1.3)
-    activesupport (5.1.3)
+    activemodel (5.1.4)
+      activesupport (= 5.1.4)
+    activesupport (5.1.4)
       concurrent-ruby (~> 1.0, >= 1.0.2)
       i18n (~> 0.7)
       minitest (~> 5.1)

data/README.md

@@ -282,6 +282,81 @@ these messages.  Some day Pwwka might have code to allow that.  Today is not tha
 
 **You should configure `requeue_on_error`**. It's not the default for backwards compatibility.
 
+#### Advanced Error Handling
+
+The underlying implementation of how errors are handled is via a [chain of responsibility-ish](https://en.wikipedia.org/wiki/Chain-of-responsibility_pattern) implementation.  When an unhandled exception occurs, pwwka's `Receiver`
+defers to the configurations `error_handling_chain`, which is a list of classes that can handle errors.  `requeue_on_error` and `keep_alive_on_handler_klass_exceptions` control which classes are in the chain.
+
+If you want to handle errors differently, for example crashing on some exceptions, but not others, or requeing messages on failures always (instead of just once), you can do that by subclassing `Pwwka::ErrorHandlers::BaseHandler`.
+It defines a method `handle_error` that is given the `Receiver` instance, queue name, payload, delivery info, and the uncaught exception.  If the method returns `true`, Pwwka calls the remaining handlers.  If false, it stops processing.
+
+Your subclass can be inserted into the chain in two ways.  Way #1 is to override the entire chain by setting `Pwwka.configuration.error_handling_chain` to an array of handlers, including yours.  Way #2 is to have your specific
+message handler implement `self.error_handler` to return the class to be used for just that message handler.
+
+**When you do this**, be careful to ensure you ack or nack.  If you fail to do either, your messages will build up and bad things will happen.
+
+For example, suppose you want to catch an ActiveRecord error, unwrap it to see if it's a problem with the connection, and reconnect before trying again.
+
+First, implement your custom error handler:
+
+```ruby
+class PostgresReconnectHandler < Pwwka::ErrorHandlers::BaseHandler
+  def handle_error(receiver,queue_name,payload,delivery_info,exception)
+    if exception.cause.is_a?(PG::ConnectionBad)
+      ActiveRecord::Base.connection.reconnect!
+    end
+    keep_going
+  end
+end
+```
+
+In your pwwka initializer:
+
+```ruby
+require 'pwwka'
+Pwwka.configure do |config|
+  config.rabbit_mq_host        = ENV['RABBITMQ_URL']
+  config.topic_exchange_name   = "mycompany-topics-#{Rails.env}"
+  config.delayed_exchange_name = "mycompany-topics-#{Rails.env}"
+  config.options               = {allow_delayed: true}
+  config.error_handling_chain = [
+    PostgresReconnectHandler,
+    Pwwka::ErrorHandlers::NackAndRequeueOnce,
+    Pwwka::ErrorHandlers::Crash
+  ]
+end
+```
+
+This says:
+
+* If the error was a `PG::ConnectionBad`, reconnect
+* If the message has not been retried, nack it and requeue it, otherwise ignore it (`NackAndRequeueOnce`)
+* Crash the handler
+
+You might not want to crash the handler in the case of `PG::ConnectionBad`.  And, you might want to always retry the job, even if it's been retried before so you don't lose it.
+
+In that case, your handler could work like this:
+
+
+```ruby
+class PostgresReconnectHandler < Pwwka::ErrorHandlers::BaseHandler
+  def handle_error(receiver,queue_name,payload,delivery_info,exception)
+    if exception.cause.is_a?(PG::ConnectionBad)
+      ActiveRecord::Base.connection.reconnect!
+      log("Retrying an Error Processing Message",queue_name,payload,delivery_info,exception)
+      receiver.nack_requeue(delivery_info.delivery_tag)
+      abort_chain
+    else
+      keep_going
+    end
+  end
+end
+```
+
+Now, if we get a `PG::ConnectionBad`, we reconnect, nack with requeue and stop processing the error (`abort_chain` is an alias for `false`, and `keep_going` is an alias for `true`, but they keep you from having to remember what to return).
+
+**When making your own handlers** it's important to make sure that the message is nacked or acked.**
+
 #### Handling Messages with Resque
 
 If you use [Resque][resque], and you wish to handle messages in a resque job, you can use `Pwwka::QueueResqueJobHandler`, which is an adapter between the standard `handle!` method provided by pwwka and your Resque job.

data/lib/pwwka.rb

@@ -28,7 +28,7 @@ require 'pwwka/handling'
 require 'pwwka/receiver'
 require 'pwwka/transmitter'
 require 'pwwka/message_queuer'
+require 'pwwka/error_handlers'
 require 'pwwka/configuration'
 require 'pwwka/send_message_async_job'
 
-

data/lib/pwwka/configuration.rb

@@ -11,9 +11,18 @@ module Pwwka
     attr_accessor :options
     attr_accessor :async_job_klass
     attr_accessor :send_message_resque_backoff_strategy
-    attr_accessor :requeue_on_error
+    attr_reader   :requeue_on_error
     attr_writer   :app_id
-    attr_writer   :keep_alive_on_handler_klass_exceptions
+    attr_writer   :error_handling_chain
+
+    def keep_alive_on_handler_klass_exceptions=(val)
+      @keep_alive_on_handler_klass_exceptions = val
+      @error_handling_chain = nil
+    end
+    def requeue_on_error=(val)
+      @requeue_on_error = val
+      @error_handling_chain = nil
+    end
 
     def initialize
       @rabbit_mq_host        = nil
@@ -61,5 +70,20 @@ module Pwwka
       options[:allow_delayed]
     end
 
+    def error_handling_chain
+      @error_handling_chain ||= begin
+                                  klasses = []
+                                  if requeue_on_error
+                                    klasses << Pwwka::ErrorHandlers::NackAndRequeueOnce
+                                  else
+                                    klasses << Pwwka::ErrorHandlers::NackAndIgnore
+                                  end
+                                  unless Pwwka.configuration.keep_alive_on_handler_klass_exceptions?
+                                    klasses << Pwwka::ErrorHandlers::Crash
+                                  end
+                                  klasses
+                                end
+    end
+
   end
 end

data/lib/pwwka/error_handlers.rb

@@ -0,0 +1,5 @@
+require_relative "error_handlers/chain"
+require_relative "error_handlers/base_error_handler"
+require_relative "error_handlers/crash"
+require_relative "error_handlers/nack_and_requeue_once"
+require_relative "error_handlers/nack_and_ignore"

data/lib/pwwka/error_handlers/base_error_handler.rb

@@ -0,0 +1,33 @@
+module Pwwka
+  module ErrorHandlers
+    class BaseErrorHandler
+      include Pwwka::Logging
+      def handle_error(receiver,queue_name,payload,delivery_info,exception)
+        raise "subclass must implement"
+      end
+
+    private
+
+      def log(message,queue_name,payload,delivery_info,exception)
+        logf "%{message} on %{queue_name} -> %{payload}, %{routing_key}: %{exception}: %{backtrace}", {
+          message: message,
+          queue_name: queue_name,
+          payload: payload,
+          routing_key: delivery_info.routing_key,
+          exception: exception,
+          backtrace: exception.backtrace.join(";"),
+        }
+      end
+
+      # Subclasses can call these methods instead of
+      # using true/false to more clearly indicate their intent
+      def keep_going
+        true
+      end
+
+      def abort_chain
+        false
+      end
+    end
+  end
+end

data/lib/pwwka/error_handlers/chain.rb

@@ -0,0 +1,28 @@
+module Pwwka
+  module ErrorHandlers
+    # Given a chain of error handlers, calls them until either
+    # one returns false/aborts or we exhaust the chain of handlers
+    class Chain
+      include Pwwka::Logging
+      def initialize(default_handler_chain=[])
+        @error_handlers = default_handler_chain
+      end
+      def handle_error(message_handler_klass,receiver,queue_name,payload,delivery_info,exception)
+        if message_handler_klass.respond_to?(:error_handler)
+          @error_handlers.unshift(message_handler_klass.send(:error_handler))
+        end
+        @error_handlers.reduce(true) { |keep_going,error_handler|
+          if keep_going
+            keep_going = error_handler.new.handle_error(receiver,queue_name,payload,delivery_info,exception)
+            unless keep_going
+              logf "%{error_handler_class} has halted to error-handling chain", error_handler_class: error_handler.class
+            end
+          else
+            logf "Skipping %{error_handler_class} as we were asked to abort previously", error_handler_class: error_handler.class
+          end
+          keep_going
+        }
+      end
+    end
+  end
+end

data/lib/pwwka/error_handlers/crash.rb

@@ -0,0 +1,11 @@
+require_relative "base_error_handler"
+module Pwwka
+  module ErrorHandlers
+    class Crash < BaseErrorHandler
+      def handle_error(receiver,queue_name,payload,delivery_info,e)
+        raise Interrupt,"Exiting due to exception #{e.inspect}"
+        abort_chain
+      end
+    end
+  end
+end

data/lib/pwwka/error_handlers/nack_and_ignore.rb

@@ -0,0 +1,13 @@
+require_relative "base_error_handler"
+module Pwwka
+  module ErrorHandlers
+    class NackAndIgnore < BaseErrorHandler
+      def handle_error(receiver,queue_name,payload,delivery_info,exception)
+        log("Error Processing Message",queue_name,payload,delivery_info,exception)
+        receiver.nack(delivery_info.delivery_tag)
+        keep_going
+      end
+    end
+
+  end
+end

data/lib/pwwka/error_handlers/nack_and_requeue_once.rb

@@ -0,0 +1,17 @@
+require_relative "base_error_handler"
+module Pwwka
+  module ErrorHandlers
+    class NackAndRequeueOnce < BaseErrorHandler
+      def handle_error(receiver,queue_name,payload,delivery_info,exception)
+        if delivery_info.redelivered
+          log("Error Processing Message",queue_name,payload,delivery_info,exception)
+          receiver.nack(delivery_info.delivery_tag)
+        else
+          log("Retrying an Error Processing Message",queue_name,payload,delivery_info,exception)
+          receiver.nack_requeue(delivery_info.delivery_tag)
+        end
+        keep_going
+      end
+    end
+  end
+end

data/lib/pwwka/receiver.rb

@@ -28,23 +28,16 @@ module Pwwka
             handler_klass.handle!(delivery_info, properties, payload)
             receiver.ack(delivery_info.delivery_tag)
             logf "Processed Message on %{queue_name} -> %{payload}, %{routing_key}", queue_name: queue_name, payload: payload, routing_key: delivery_info.routing_key
-          rescue => e
-            error_options = {
-               queue_name: queue_name,
-                  payload: payload,
-              routing_key: delivery_info.routing_key,
-                exception: e
-            }
-            if Pwwka.configuration.requeue_on_error && !delivery_info.redelivered
-              log_error "Retrying an Error Processing Message", error_options
-              receiver.nack_requeue(delivery_info.delivery_tag)
-            else
-              log_error "Error Processing Message", error_options
-              receiver.nack(delivery_info.delivery_tag)
-            end
-            unless Pwwka.configuration.keep_alive_on_handler_klass_exceptions?
-              raise Interrupt,"Exiting due to exception #{e.inspect}"
-            end
+          rescue => exception
+            Pwwka::ErrorHandlers::Chain.new(
+              Pwwka.configuration.error_handling_chain
+            ).handle_error(
+              handler_klass,
+              receiver,
+              queue_name,
+              payload,
+              delivery_info,
+              exception)
           end
         end
       rescue Interrupt => _
@@ -86,13 +79,5 @@ module Pwwka
       channel_connector.connection_close
     end
 
-  private
-
-    def self.log_error(message,options)
-      options[:message] = message
-      options[:backtrace] = options.fetch(:exception).backtrace.join(';')
-      logf "%{message} on %{queue_name} -> %{payload}, %{routing_key}: %{exception}: %{backtrace}", options
-    end
-
   end
 end

data/lib/pwwka/version.rb

@@ -1,3 +1,3 @@
 module Pwwka
-  VERSION = '0.12.0'
+  VERSION = '0.13.0.RC1'
 end

data/spec/integration/unhandled_errors_in_receivers_spec.rb

@@ -8,88 +8,126 @@ describe "receivers with unhandled errors", :integration do
 
   before do
     @testing_setup = IntegrationTestSetup.new
-    setup_receivers
     Pwwka.configure do |c|
       c.requeue_on_error = false
       c.keep_alive_on_handler_klass_exceptions = false
     end
   end
 
-  before :each do
-    WellBehavedReceiver.reset!
-    ExceptionThrowingReceiver.reset!
-    IntermittentErrorReceiver.reset!
-  end
-
   after do
     @testing_setup.kill_threads_and_clear_queues
   end
 
-  it "an error in one receiver doesn't prevent others from getting messages" do
-    Pwwka::Transmitter.send_message!({ sample: "payload", has: { deeply: true, nested: 4 }},
-                                     "pwwka.testing.foo")
-    allow_receivers_to_process_queues
+  context "default configuration to crash on errors" do
+    before do
+      setup_receivers
+      WellBehavedReceiver.reset!
+      ExceptionThrowingReceiver.reset!
+      IntermittentErrorReceiver.reset!
+      ExceptionThrowingReceiverWithErrorHook.reset!
+    end
 
-    expect(WellBehavedReceiver.messages_received.size).to eq(1)
-    expect(ExceptionThrowingReceiver.messages_received.size).to eq(1)
-  end
+    it "an error in one receiver doesn't prevent others from getting messages" do
+      Pwwka::Transmitter.send_message!({ sample: "payload", has: { deeply: true, nested: 4 }},
+                                       "pwwka.testing.foo")
+      allow_receivers_to_process_queues
 
-  it "when configured to requeue failed messages, the message is requeued exactly once" do
-    Pwwka.configure do |c|
-      c.requeue_on_error = true
-      c.keep_alive_on_handler_klass_exceptions = true # only so we can check that the requeued message got sent; otherwise the receiver crashes and we can't test that
+      expect(WellBehavedReceiver.messages_received.size).to eq(1)
+      expect(ExceptionThrowingReceiver.messages_received.size).to eq(1)
     end
-    Pwwka::Transmitter.send_message!({ sample: "payload", has: { deeply: true, nested: 4 }},
-                                     "pwwka.testing.foo")
-    allow_receivers_to_process_queues
+    it "crashes the receiver that received an error" do
+      Pwwka::Transmitter.send_message!({ sample: "payload", has: { deeply: true, nested: 4 }},
+                                       "pwwka.testing.foo")
+      allow_receivers_to_process_queues
 
-    expect(WellBehavedReceiver.messages_received.size).to eq(1)
-    expect(ExceptionThrowingReceiver.messages_received.size).to eq(2)
-    expect(ExceptionThrowingReceiver.messages_received[1][0].redelivered).to eq(true)
-    expect(ExceptionThrowingReceiver.messages_received[1][2]).to eq(ExceptionThrowingReceiver.messages_received[0][2])
-  end
+      expect(@testing_setup.threads[ExceptionThrowingReceiver].alive?).to eq(false)
+    end
+    it "does not crash the receiver that successfully processed a message" do
+      Pwwka::Transmitter.send_message!({ sample: "payload", has: { deeply: true, nested: 4 }},
+                                       "pwwka.testing.foo")
+      allow_receivers_to_process_queues
+
+      expect(@testing_setup.threads[WellBehavedReceiver].alive?).to  eq(true)
+    end
 
-  it "crashes the receiver that received an error" do
-    Pwwka::Transmitter.send_message!({ sample: "payload", has: { deeply: true, nested: 4 }},
-                                     "pwwka.testing.foo")
-    allow_receivers_to_process_queues
+    it "crashes the receiver if it gets a failure that we retry" do
+      Pwwka.configure do |c|
+        c.requeue_on_error = true
+      end
+      Pwwka::Transmitter.send_message!({ sample: "payload", has: { deeply: true, nested: 4 }},
+                                       "pwwka.testing.foo")
+      allow_receivers_to_process_queues
 
-    expect(@testing_setup.threads[ExceptionThrowingReceiver].alive?).to eq(false)
+      expect(@testing_setup.threads[IntermittentErrorReceiver].alive?).to eq(false)
+    end
   end
 
-  it "does not crash the receiver that received an error when we configure it not to" do
-    Pwwka.configure do |c|
-      c.keep_alive_on_handler_klass_exceptions = true
+  context "configured not to crash on error" do
+    before do
+      setup_receivers
+      WellBehavedReceiver.reset!
+      ExceptionThrowingReceiver.reset!
+      IntermittentErrorReceiver.reset!
+      ExceptionThrowingReceiverWithErrorHook.reset!
     end
-    Pwwka::Transmitter.send_message!({ sample: "payload", has: { deeply: true, nested: 4 }},
-                                     "pwwka.testing.foo")
-    allow_receivers_to_process_queues
+    it "does not crash the receiver that received an error" do
+      Pwwka.configure do |c|
+        c.keep_alive_on_handler_klass_exceptions = true
+      end
+      Pwwka::Transmitter.send_message!({ sample: "payload", has: { deeply: true, nested: 4 }},
+                                       "pwwka.testing.foo")
+      allow_receivers_to_process_queues
 
-    expect(@testing_setup.threads[ExceptionThrowingReceiver].alive?).to eq(true)
+      expect(@testing_setup.threads[ExceptionThrowingReceiver].alive?).to eq(true)
+    end
   end
 
-  it "does not crash the receiver that successfully processed a message" do
-    Pwwka::Transmitter.send_message!({ sample: "payload", has: { deeply: true, nested: 4 }},
-                                     "pwwka.testing.foo")
-    allow_receivers_to_process_queues
-
-    expect(@testing_setup.threads[WellBehavedReceiver].alive?).to  eq(true)
+  context "configured to requeue failed messages" do
+    before do
+      setup_receivers
+      WellBehavedReceiver.reset!
+      ExceptionThrowingReceiver.reset!
+      IntermittentErrorReceiver.reset!
+      ExceptionThrowingReceiverWithErrorHook.reset!
+    end
+    it "requeues the message exactly once" do
+      Pwwka.configure do |c|
+        c.requeue_on_error = true
+        c.keep_alive_on_handler_klass_exceptions = true # only so we can check that the requeued message got sent; otherwise the receiver crashes and we can't test that
+      end
+      Pwwka::Transmitter.send_message!({ sample: "payload", has: { deeply: true, nested: 4 }},
+                                       "pwwka.testing.foo")
+      allow_receivers_to_process_queues
+
+      expect(WellBehavedReceiver.messages_received.size).to eq(1)
+      expect(ExceptionThrowingReceiver.messages_received.size).to eq(2)
+      expect(ExceptionThrowingReceiver.messages_received[1][0].redelivered).to eq(true)
+      expect(ExceptionThrowingReceiver.messages_received[1][2]).to eq(ExceptionThrowingReceiver.messages_received[0][2])
+    end
   end
 
-  it "crashes the receiver if it gets a failure that we retry" do
-    Pwwka.configure do |c|
-      c.requeue_on_error = true
+  context "handler with a custom error handler that ignores the exception" do
+    before do
+      setup_receivers(ExceptionThrowingReceiverWithErrorHook)
+      WellBehavedReceiver.reset!
+      ExceptionThrowingReceiver.reset!
+      IntermittentErrorReceiver.reset!
+      ExceptionThrowingReceiverWithErrorHook.reset!
     end
-    Pwwka::Transmitter.send_message!({ sample: "payload", has: { deeply: true, nested: 4 }},
-                                     "pwwka.testing.foo")
-    allow_receivers_to_process_queues
 
-    expect(@testing_setup.threads[IntermittentErrorReceiver].alive?).to eq(false)
+    it "does not crash the receiver" do
+      Pwwka::Transmitter.send_message!({ sample: "payload", has: { deeply: true, nested: 4 }},
+                                       "pwwka.testing.foo")
+      allow_receivers_to_process_queues
+
+      expect(ExceptionThrowingReceiverWithErrorHook.messages_received.size).to eq(1)
+      expect(@testing_setup.threads[ExceptionThrowingReceiverWithErrorHook].alive?).to eq(true)
+    end
   end
 
-  def setup_receivers
+  def setup_receivers(exception_throwing_receiver_klass=ExceptionThrowingReceiver)
     [
-      [ExceptionThrowingReceiver, "exception_throwing_receiver_pwwkatesting"],
+      [exception_throwing_receiver_klass, "exception_throwing_receiver_pwwkatesting"],
       [WellBehavedReceiver, "well_behaved_receiver_pwwkatesting"],
       [IntermittentErrorReceiver, "intermittent_error_receiver_pwwkatesting"],
     ].each do |(klass, queue_name)|
@@ -102,6 +140,24 @@ describe "receivers with unhandled errors", :integration do
       raise "OH NOES!"
     end
   end
+  class NoOpHandler < Pwwka::ErrorHandlers::BaseErrorHandler
+    def initialize(*)
+    end
+    def handle_error(receiver,queue_name,payload,delivery_info,exception)
+      receiver.nack(delivery_info.delivery_tag)
+      abort_chain
+    end
+  end
+  class ExceptionThrowingReceiverWithErrorHook < LoggingReceiver
+    def self.error_handler
+      NoOpHandler
+    end
+
+    def self.handle!(delivery_info,properties,payload)
+      super(delivery_info,properties,payload)
+      raise "OH NOES!"
+    end
+  end
   class IntermittentErrorReceiver < LoggingReceiver
     def self.handle!(delivery_info,properties,payload)
       super(delivery_info,properties,payload)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pwwka
 version: !ruby/object:Gem::Version
-  version: 0.12.0
+  version: 0.13.0.RC1
 platform: ruby
 authors:
 - Stitch Fix Engineering
@@ -15,7 +15,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-09-06 00:00:00.000000000 Z
+date: 2017-09-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bunny
@@ -206,6 +206,12 @@ files:
 - lib/pwwka.rb
 - lib/pwwka/channel_connector.rb
 - lib/pwwka/configuration.rb
+- lib/pwwka/error_handlers.rb
+- lib/pwwka/error_handlers/base_error_handler.rb
+- lib/pwwka/error_handlers/chain.rb
+- lib/pwwka/error_handlers/crash.rb
+- lib/pwwka/error_handlers/nack_and_ignore.rb
+- lib/pwwka/error_handlers/nack_and_requeue_once.rb
 - lib/pwwka/handling.rb
 - lib/pwwka/logging.rb
 - lib/pwwka/message_queuer.rb
@@ -254,9 +260,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
+  - - ">"
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 1.3.1
 requirements: []
 rubyforge_project: 
 rubygems_version: 2.6.13