waterdrop 2.4.11 → 2.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c5190e7c6d460afe6d9a55cb177220e23cd5e89262430f361c3e29ab7df685e6
-  data.tar.gz: 8cf622a618610bc1bdd283ba2046e0d233d660c0e71dfc70c78ecea2f41db22e
+  metadata.gz: 8aa4c6d8e6d5364a4a1a93739a0292410d664df16f7d93545bb478b5659ee443
+  data.tar.gz: 2bf8a36f8332be75eef108b90b2179ecc18f7301750f551eadce9d90eec6c5a9
 SHA512:
-  metadata.gz: 48b3d383c175c392acbdd4a3a41a543192c0ca0e404d755f7fc8eae862350e8d62c9855fb8fea8dc354e358162574f4ff052c850c2ae0fc13a763f951330b97d
-  data.tar.gz: 7da204b9493122e752933dac8856057c80c4ca640ccd31c8c5787cb9f20fbd8add2ef1851a447ef04010a72e70722d54689978697c4ef1def6ee5c3f2dc23fbe
+  metadata.gz: 35269366d659f9dcada5fafd5120f2434784c29417c021a6d78fad722496c37acde6617c4e4e6fee3132052248e5b935c958cb45a1113c8510d995cfd6136ccb
+  data.tar.gz: 8ff77a6a0c61118cf5dcad89f7d7414c510107bb559683fd8b0c133798c686cf6bbf5eef95fb14ecb5bf5762354148eb545b6835e597730457125c69cd996a41

data/CHANGELOG.md

@@ -1,5 +1,36 @@
 # WaterDrop changelog
 
+## 2.5.0 (2023-03-04)
+- [Feature] Pipe **all** the errors including synchronous errors via the `error.occurred`.
+- [Improvement] Pipe delivery errors that occurred not via the error callback using the `error.occurred` channel.
+- [Improvement] Introduce `WaterDrop::Errors::ProduceError` and `WaterDrop::Errors::ProduceManyError` for any inline raised errors that occur. You can get the original error by using the `#cause`.
+- [Improvement] Include `#dispatched` messages handler in the `WaterDrop::Errors::ProduceManyError` error, to be able to understand which of the messages were delegated to `librdkafka` prior to the failure.
+- [Maintenance] Remove the `WaterDrop::Errors::FlushFailureError` in favour of correct error that occurred to unify the error handling.
+- [Maintenance] Rename `Datadog::Listener` to `Datadog::MetricsListener` to align with Karafka (#329).
+- [Fix] Do **not** flush when there is no data to flush in the internal buffer.
+- [Fix] Wait on the final data flush for short-lived producers to make sure, that the message is actually dispatched by `librdkafka` or timeout.
+
+### Upgrade notes
+
+Please note, this **is** a **breaking** release, hence `2.5.0`.
+
+1. If you used to catch `WaterDrop::Errors::FlushFailureError` now you need to catch `WaterDrop::Errors::ProduceError`. `WaterDrop::Errors::ProduceManyError` is based on the `ProduceError`, hence it should be enough.
+2. Prior to `2.5.0` there was always a chance of partial dispatches via `produce_many_` methods. Now you can get the info on all the errors via `error.occurred`.
+3. Inline `Rdkafka::RdkafkaError` are now re-raised via `WaterDrop::Errors::ProduceError` and available under `#cause`. Async `Rdkafka::RdkafkaError` errors are still directly available and you can differentiate between errors using the event `type`.
+4. If you are using the Datadog listener, you need to:
+
+```ruby
+# Replace require:
+require 'waterdrop/instrumentation/vendors/datadog/listener'
+# With
+require 'waterdrop/instrumentation/vendors/datadog/metrics_listener'
+
+# Replace references of
+::WaterDrop::Instrumentation::Vendors::Datadog::Listener.new
+# With
+::WaterDrop::Instrumentation::Vendors::Datadog::MetricsListener.new
+```
+
 ## 2.4.11 (2023-02-24)
 - Replace the local rspec locator with generalized core one.
 - Make `::WaterDrop::Instrumentation::Notifications::EVENTS` list public for anyone wanting to re-bind those into a different notification bus.

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    waterdrop (2.4.11)
+    waterdrop (2.5.0)
       karafka-core (>= 2.0.12, < 3.0.0)
       zeitwerk (~> 2.3)
 
@@ -14,7 +14,7 @@ GEM
       minitest (>= 5.1)
       tzinfo (~> 2.0)
     byebug (11.1.3)
-    concurrent-ruby (1.2.0)
+    concurrent-ruby (1.2.2)
     diff-lcs (1.5.0)
     docile (1.4.0)
     factory_bot (6.2.1)
@@ -67,4 +67,4 @@ DEPENDENCIES
   waterdrop!
 
 BUNDLED WITH
-   2.4.6
+   2.4.7

data/README.md

@@ -387,6 +387,8 @@ end
 # WaterDrop error occurred: Local: Broker transport failure (transport)
 ```
 
+**Note:** `error.occurred` will also include any errors originating from `librdkafka` for synchronous operations, including those that are raised back to the end user.
+
 ### Acknowledgment notifications
 
 WaterDrop allows you to listen to Kafka messages' acknowledgment events. This will enable you to monitor deliveries of messages from WaterDrop even when using asynchronous dispatch methods.
@@ -423,7 +425,7 @@ WaterDrop comes with (optional) full Datadog and StatsD integration that you can
 ```ruby
 # require datadog/statsd and the listener as it is not loaded by default
 require 'datadog/statsd'
-require 'waterdrop/instrumentation/vendors/datadog/listener'
+require 'waterdrop/instrumentation/vendors/datadog/metrics_listener'
 
 # initialize your producer with statistics.interval.ms enabled so the metrics are published
 producer = WaterDrop::Producer.new do |config|
@@ -435,7 +437,7 @@ producer = WaterDrop::Producer.new do |config|
 end
 
 # initialize the listener with statsd client
-listener = ::WaterDrop::Instrumentation::Vendors::Datadog::Listener.new do |config|
+listener = ::WaterDrop::Instrumentation::Vendors::Datadog::MetricsListener.new do |config|
   config.client = Datadog::Statsd.new('localhost', 8125)
   # Publish host as a tag alongside the rest of tags
   config.default_tags = ["host:#{Socket.gethostname}"]

data/lib/waterdrop/errors.rb

@@ -29,15 +29,18 @@ module WaterDrop
     # contact us as it is an error.
     StatusInvalidError = Class.new(BaseError)
 
-    # Raised when during messages flushing something bad happened
-    class FlushFailureError < BaseError
-      attr_reader :dispatched_messages
+    # Raised when there is an inline error during single message produce operations
+    ProduceError = Class.new(BaseError)
 
-      # @param dispatched_messages [Array<Rdkafka::Producer::DeliveryHandle>] handlers of the
+    # Raised when during messages producing something bad happened inline
+    class ProduceManyError < ProduceError
+      attr_reader :dispatched
+
+      # @param dispatched [Array<Rdkafka::Producer::DeliveryHandle>] handlers of the
       #   messages that we've dispatched
-      def initialize(dispatched_messages)
+      def initialize(dispatched)
         super()
-        @dispatched_messages = dispatched_messages
+        @dispatched = dispatched
       end
     end
   end

data/lib/waterdrop/instrumentation/callbacks/delivery.rb

@@ -17,6 +17,30 @@ module WaterDrop
         # Emits delivery details to the monitor
         # @param delivery_report [Rdkafka::Producer::DeliveryReport] delivery report
         def call(delivery_report)
+          if delivery_report.error.to_i.positive?
+            instrument_error(delivery_report)
+          else
+            instrument_acknowledged(delivery_report)
+          end
+        end
+
+        private
+
+        # @param delivery_report [Rdkafka::Producer::DeliveryReport] delivery report
+        def instrument_error(delivery_report)
+          @monitor.instrument(
+            'error.occurred',
+            caller: self,
+            error: ::Rdkafka::RdkafkaError.new(delivery_report.error),
+            producer_id: @producer_id,
+            offset: delivery_report.offset,
+            partition: delivery_report.partition,
+            type: 'librdkafka.dispatch_error'
+          )
+        end
+
+        # @param delivery_report [Rdkafka::Producer::DeliveryReport] delivery report
+        def instrument_acknowledged(delivery_report)
           @monitor.instrument(
             'message.acknowledged',
             producer_id: @producer_id,

data/lib/waterdrop/instrumentation/callbacks/error.rb

@@ -18,6 +18,8 @@ module WaterDrop
         # @param client_name [String] rdkafka client name
         # @param error [Rdkafka::Error] error that occurred
         # @note It will only instrument on errors of the client of our producer
+        # @note When there is a particular message produce error (not internal error), the error
+        #   is shipped via the delivery callback, not via error callback.
         def call(client_name, error)
           # Emit only errors related to our client
           # Same as with statistics (mor explanation there)

data/lib/waterdrop/instrumentation/vendors/datadog/{listener.rb → metrics_listener.rb}

@@ -10,7 +10,7 @@ module WaterDrop
         # and/or Datadog
         #
         # @note You need to setup the `dogstatsd-ruby` client and assign it
-        class Listener
+        class MetricsListener
           include ::Karafka::Core::Configurable
           extend Forwardable
 

data/lib/waterdrop/patches/rdkafka/client.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+module WaterDrop
+  module Patches
+    module Rdkafka
+      # Patches for the producer client
+      module Client
+        # @param _object_id [nil] rdkafka API compatibility argument
+        # @param timeout_ms [Integer] final flush timeout in ms
+        def close(_object_id = nil, timeout_ms = 5_000)
+          return unless @native
+
+          # Indicate to polling thread that we're closing
+          @polling_thread[:closing] = true
+          # Wait for the polling thread to finish up
+          @polling_thread.join
+
+          ::Rdkafka::Bindings.rd_kafka_flush(@native, timeout_ms)
+          ::Rdkafka::Bindings.rd_kafka_destroy(@native)
+
+          @native = nil
+        end
+      end
+    end
+  end
+end
+
+::Rdkafka::Bindings.attach_function(
+  :rd_kafka_flush,
+  %i[pointer int],
+  :void
+)
+
+Rdkafka::Producer::Client.prepend WaterDrop::Patches::Rdkafka::Client

data/lib/waterdrop/patches/rdkafka/producer.rb

@@ -70,6 +70,14 @@ module WaterDrop
 
           @_inner_kafka
         end
+
+        # Closes our librdkafka instance with the flush patch
+        # @param timeout_ms [Integer] flush timeout
+        def close(timeout_ms = 5_000)
+          ObjectSpace.undefine_finalizer(self)
+
+          @client.close(nil, timeout_ms)
+        end
       end
     end
   end

data/lib/waterdrop/producer/async.rb

@@ -23,7 +23,19 @@ module WaterDrop
           'message.produced_async',
           producer_id: id,
           message: message
-        ) { client.produce(**message) }
+        ) { produce(message) }
+      rescue *SUPPORTED_FLOW_ERRORS
+        re_raised = Errors::ProduceError.new
+
+        @monitor.instrument(
+          'error.occurred',
+          producer_id: id,
+          message: message,
+          error: re_raised,
+          type: 'message.produce_async'
+        )
+
+        raise re_raised
       end
 
       # Produces many messages to Kafka and does not wait for them to be delivered
@@ -39,6 +51,7 @@ module WaterDrop
       def produce_many_async(messages)
         ensure_active!
 
+        dispatched = []
         messages = middleware.run_many(messages)
         messages.each { |message| validate_message!(message) }
 
@@ -47,8 +60,25 @@ module WaterDrop
           producer_id: id,
           messages: messages
         ) do
-          messages.map { |message| client.produce(**message) }
+          messages.each do |message|
+            dispatched << produce(message)
+          end
+
+          dispatched
         end
+      rescue *SUPPORTED_FLOW_ERRORS
+        re_raised = Errors::ProduceManyError.new(dispatched)
+
+        @monitor.instrument(
+          'error.occurred',
+          producer_id: id,
+          messages: messages,
+          dispatched: dispatched,
+          error: re_raised,
+          type: 'messages.produce_many_async'
+        )
+
+        raise re_raised
       end
     end
   end

data/lib/waterdrop/producer/buffer.rb

@@ -4,14 +4,6 @@ module WaterDrop
   class Producer
     # Component for buffered operations
     module Buffer
-      # Exceptions we catch when dispatching messages from a buffer
-      RESCUED_ERRORS = [
-        Rdkafka::RdkafkaError,
-        Rdkafka::Producer::DeliveryHandle::WaitTimeoutError
-      ].freeze
-
-      private_constant :RESCUED_ERRORS
-
       # Adds given message into the internal producer buffer without flushing it to Kafka
       #
       # @param message [Hash] hash that complies with the {Contracts::Message} contract
@@ -85,39 +77,21 @@ module WaterDrop
       # @param sync [Boolean] should it flush in a sync way
       # @return [Array<Rdkafka::Producer::DeliveryHandle, Rdkafka::Producer::DeliveryReport>]
       #   delivery handles for async or delivery reports for sync
-      # @raise [Errors::FlushFailureError] when there was a failure in flushing
+      # @raise [Errors::ProduceManyError] when there was a failure in flushing
       # @note We use this method underneath to provide a different instrumentation for sync and
       #   async flushing within the public API
       def flush(sync)
         data_for_dispatch = nil
-        dispatched = []
 
         @buffer_mutex.synchronize do
           data_for_dispatch = @messages
           @messages = Concurrent::Array.new
         end
 
-        dispatched = data_for_dispatch.map { |message| client.produce(**message) }
-
-        return dispatched unless sync
-
-        dispatched.map do |handler|
-          handler.wait(
-            max_wait_timeout: @config.max_wait_timeout,
-            wait_timeout: @config.wait_timeout
-          )
-        end
-      rescue *RESCUED_ERRORS => e
-        @monitor.instrument(
-          'error.occurred',
-          caller: self,
-          error: e,
-          producer_id: id,
-          dispatched: dispatched,
-          type: sync ? 'buffer.flushed_sync.error' : 'buffer.flush_async.error'
-        )
+        # Do nothing if nothing to flush
+        return data_for_dispatch if data_for_dispatch.empty?
 
-        raise Errors::FlushFailureError.new(dispatched)
+        sync ? produce_many_sync(data_for_dispatch) : produce_many_async(data_for_dispatch)
       end
     end
   end

data/lib/waterdrop/producer/sync.rb

@@ -26,13 +26,20 @@ module WaterDrop
           producer_id: id,
           message: message
         ) do
-          client
-            .produce(**message)
-            .wait(
-              max_wait_timeout: @config.max_wait_timeout,
-              wait_timeout: @config.wait_timeout
-            )
+          wait(produce(message))
         end
+      rescue *SUPPORTED_FLOW_ERRORS
+        re_raised = Errors::ProduceError.new
+
+        @monitor.instrument(
+          'error.occurred',
+          producer_id: id,
+          message: message,
+          error: re_raised,
+          type: 'message.produce_sync'
+        )
+
+        raise re_raised
       end
 
       # Produces many messages to Kafka and waits for them to be delivered
@@ -48,21 +55,37 @@ module WaterDrop
       # @raise [Errors::MessageInvalidError] When any of the provided messages details are invalid
       #   and the message could not be sent to Kafka
       def produce_many_sync(messages)
-        ensure_active!
+        ensure_active! unless @closing_thread_id && @closing_thread_id == Thread.current.object_id
 
         messages = middleware.run_many(messages)
         messages.each { |message| validate_message!(message) }
 
+        dispatched = []
+
         @monitor.instrument('messages.produced_sync', producer_id: id, messages: messages) do
-          messages
-            .map { |message| client.produce(**message) }
-            .map! do |handler|
-              handler.wait(
-                max_wait_timeout: @config.max_wait_timeout,
-                wait_timeout: @config.wait_timeout
-              )
-            end
+          messages.each do |message|
+            dispatched << produce(message)
+          end
+
+          dispatched.map! do |handler|
+            wait(handler)
+          end
+
+          dispatched
         end
+      rescue *SUPPORTED_FLOW_ERRORS
+        re_raised = Errors::ProduceManyError.new(dispatched)
+
+        @monitor.instrument(
+          'error.occurred',
+          producer_id: id,
+          messages: messages,
+          dispatched: dispatched,
+          error: re_raised,
+          type: 'messages.produce_many_sync'
+        )
+
+        raise re_raised
       end
     end
   end

data/lib/waterdrop/producer.rb

@@ -8,6 +8,14 @@ module WaterDrop
     include Async
     include Buffer
 
+    # Which of the inline flow errors do we want to intercept and re-bind
+    SUPPORTED_FLOW_ERRORS = [
+      Rdkafka::RdkafkaError,
+      Rdkafka::Producer::DeliveryHandle::WaitTimeoutError
+    ].freeze
+
+    private_constant :SUPPORTED_FLOW_ERRORS
+
     def_delegators :config, :middleware
 
     # @return [String] uuid of the current producer
@@ -117,6 +125,10 @@ module WaterDrop
           # This should be used only in case a producer was not closed properly and forgotten
           ObjectSpace.undefine_finalizer(id)
 
+          # We save this thread id because we need to bypass the activity verification on the
+          # producer for final flush of buffers.
+          @closing_thread_id = Thread.current.object_id
+
           # Flush has its own buffer mutex but even if it is blocked, flushing can still happen
           # as we close the client after the flushing (even if blocked by the mutex)
           flush(true)
@@ -125,7 +137,7 @@ module WaterDrop
           # It is safe to run it several times but not exactly the same moment
           # We also mark it as closed only if it was connected, if not, it would trigger a new
           # connection that anyhow would be immediately closed
-          client.close if @client
+          client.close(@config.max_wait_timeout) if @client
 
           # Remove callbacks runners that were registered
           ::Karafka::Core::Instrumentation.statistics_callbacks.delete(@id)
@@ -155,5 +167,22 @@ module WaterDrop
     def validate_message!(message)
       @contract.validate!(message, Errors::MessageInvalidError)
     end
+
+    # Runs the client produce method with a given message
+    #
+    # @param message [Hash] message we want to send
+    def produce(message)
+      client.produce(**message)
+    end
+
+    # Waits on a given handler
+    #
+    # @param handler [Rdkafka::Producer::DeliveryHandle]
+    def wait(handler)
+      handler.wait(
+        max_wait_timeout: @config.max_wait_timeout,
+        wait_timeout: @config.wait_timeout
+      )
+    end
   end
 end

data/lib/waterdrop/version.rb

@@ -3,5 +3,5 @@
 # WaterDrop library
 module WaterDrop
   # Current WaterDrop version
-  VERSION = '2.4.11'
+  VERSION = '2.5.0'
 end

data.tar.gz.sig

@@ -1,3 +1,4 @@
-�y��~[�7�����m��%)�"�:�hc��?�����L₃�'�5?n��)��A	�
�ix?Ԅ��Bq�Ц~�j�{�܍�&3O՗�~5��;���o��SKюL%�ospW\�}�סL�v��q�II!��y��4+}$X��(�֝_NT�����bD�l�514c@�3���p:��ˍ��������O��%{N >�=-@��'�@b�:�W��
6+��G���Z.(|��]�NZ^S�<u���?�Y8��񧅳�����"�4�}�b
-d߿|��GȽ`qs�R�6{� \q��;��3{�C8i–��'�8��A�:�����Du(2F����oBP��2ē�1�g�
-E=��%D�Djq;M�M[
+�b�k+%���zv4�5��@S��Z�s��d`<��~sd�^$w��0����
+p�����"1�_-��y��Գ7�
Ӭ4EI��fm�g��,7�����
����X��6u�1%P<���F5�i����'Z%���掜�R46����_�y��w�j ᕆ쫙*����4� ���)��:��ʭBMDB����*�� ��q�Q~�D
+R</�~�w�[,�0��qì�C�\	�Z3���Z���C�v
+9���-��o�~�Z�ݯd��'��:��5v������T�]Da@ySR�6u|c$L_ �����j�Q�'�܀��pٚ+�$�Y�I-9rq���K�	*�?8++r[Y���?k

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: waterdrop
 version: !ruby/object:Gem::Version
-  version: 2.4.11
+  version: 2.5.0
 platform: ruby
 authors:
 - Maciej Mensfeld
@@ -35,7 +35,7 @@ cert_chain:
   Qf04B9ceLUaC4fPVEz10FyobjaFoY4i32xRto3XnrzeAgfEe4swLq8bQsR3w/EF3
   MGU0FeSV2Yj7Xc2x/7BzLK8xQn5l7Yy75iPF+KP3vVmDHnNl
   -----END CERTIFICATE-----
-date: 2023-02-24 00:00:00.000000000 Z
+date: 2023-03-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: karafka-core
@@ -107,8 +107,9 @@ files:
 - lib/waterdrop/instrumentation/monitor.rb
 - lib/waterdrop/instrumentation/notifications.rb
 - lib/waterdrop/instrumentation/vendors/datadog/dashboard.json
-- lib/waterdrop/instrumentation/vendors/datadog/listener.rb
+- lib/waterdrop/instrumentation/vendors/datadog/metrics_listener.rb
 - lib/waterdrop/middleware.rb
+- lib/waterdrop/patches/rdkafka/client.rb
 - lib/waterdrop/patches/rdkafka/metadata.rb
 - lib/waterdrop/patches/rdkafka/producer.rb
 - lib/waterdrop/producer.rb