waterdrop 2.10.8 → 2.10.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7681d5f6c3333372a6349ddd24a1e1214a3474135755f918aa554064689976be
-  data.tar.gz: 2f9d99fff24a56acaf5575e539d647682b305f303206a75bbf10b128dd8112c8
+  metadata.gz: 0e6efd47232839d14e444c45df4cc5fae39661dd7e8ef646639f00d41f62d3b3
+  data.tar.gz: edd8560d05bb3742d8592a1b90f2dc784410e7150045e8a8d4a0ebf1c0d82e81
 SHA512:
-  metadata.gz: 63fcee865ef2c371bea98931f0b0ae1b5a9c118462acfb56746195547201741bba4cbdc6b6efc5feb1ec100e3c65fea8e6879da5774668f4d83691f4865cebe6
-  data.tar.gz: dc4c8e69e8ed7e6312a0aa4b4ccd7eeec3d233113b2b6b80b4308dd6602e40a7a55c3a29842ef80e258d850b3ad93482be5b68f328feebccc36d6809e5509092
+  metadata.gz: 1532bf1ecbb02ec192bdd8a410b5e7e7579337e11515fe828fb88bd8cf2a2168e31ef669e9d3ba1c334e64720c148069d26b5259d7c3f913d80258d5ddbb711b
+  data.tar.gz: 87178e647ac42309e0cdbef6c48af8c0f7a902ee32238c3c5cf8cbca2ee7ae80396895bb3a70417c912f98f669834e19ce1066c596a7a5432c7245b29a537248

data/.ruby-version

@@ -1 +1 @@
-4.0.7
+4.0.9

data/CHANGELOG.md

@@ -1,13 +1,23 @@
 # WaterDrop changelog
 
+## 2.10.9 (2026-11-02)
+- [Maintenance] Rename the positional topic name parameters of `Producer#partition_count`, `Clients::Buffered#messages_for` and the Datadog metrics listener to `topic_name`.
+- [Maintenance] Rename the positional topic name parameter of `Clients::Dummy::Handle#initialize` to `topic_name`.
+- [Maintenance] Declare `rake` in the `Gemfile`. It came only through `karafka-rdkafka`, which will stop depending on it for precompiled platform gems.
+- [Fix] Copy the buffer in `Producer#messages` (and so the `buffer.flushed_async`/`buffer.flushed_sync` payloads) under a dedicated lock, so reading it while other threads buffer no longer raises `ConcurrencyError` on JRuby.
+- [Fix] Re-buffer the undispatched rest of a `#flush` batch when a concurrent `#close` stops it, and expose the handles dispatched before the close as `ProducerClosedError#dispatched`.
+- [Fix] Copy the buffer in `Producer#messages` (and so the `buffer.flushed_async`/`buffer.flushed_sync` payloads) under a dedicated lock, so reading it while other threads buffer no longer raises `ConcurrencyError` on JRuby.
+
 ## 2.10.8 (2026-10-25)
 - [Maintenance] Document that a transaction timeout also raises `:fenced`, indistinguishable from a takeover, and when removing `:fenced` from `non_reloadable_errors` is safe.
+- [Maintenance] Drop unused `byebug` from the test bundle.
+- [Maintenance] Fix test races and skip the fork test on JRuby, resolving unstable behavior in `test/` only.
 - [Fix] Stop `Producer#close` from raising `Errno::EPIPE` on JRuby when its close signal races the poller closing the pipe.
-- [Fix] Abort a failed transaction only once: raise an abortable commit error as is instead of a follow-up `Operation not valid in state Ready` error, and do not abort after a failed `begin_transaction`.
+- [Fix] Abort a failed transaction only once: raise the abortable commit error instead of a follow-up `Operation not valid in state Ready` error, and do not abort after a failed `begin_transaction`.
 
 ## 2.10.7 (2026-10-19)
 - [Change] Require `karafka-rdkafka` `>=` `0.31.0` so delivery reports carry the message persistence status, latency and broker id.
-- [Maintenance] Stop the transaction purge instrumentation spec from flaking when a slow runner lets librdkafka deliver the message before the abort, which leaves nothing to purge.
+- [Maintenance] Fix a flaky transaction purge instrumentation spec on slow runners.
 
 ## 2.10.6 (2026-10-12)
 - **[EOL]** Drop Ruby 3.2 support; require Ruby 3.3+.
@@ -20,10 +30,10 @@
 - [Fix] Restore both buffers when `#flush` fails before anything is dispatched, instead of discarding the whole batch.
 
 ## 2.10.4 (2026-08-26)
+- [Maintenance] Stop the `#partition_count when topic does not exist` spec from flaking on slow CI runners by waiting for authoritative broker metadata before asserting the count.
 - [Fix] Avoid a `FrozenError` on `Producer#close` when the producer is configured with a frozen string id (for example a frozen string literal, `config.id = "rspec"`).
 - [Fix] Emit `connection_pool.shutdown` only once when shutting down the global connection pool.
 - [Fix] Retry a produce that races a concurrent idempotent fatal-error reload instead of leaking a raw `Rdkafka::ClosedProducerError`/`Rdkafka::ClosedInnerError`.
-- [Maintenance] Stop the `#partition_count when topic does not exist` spec from flaking on slow CI runners by waiting for authoritative broker metadata before asserting the count.
 
 ## 2.10.3 (2026-07-15)
 - [Feature] Add `wait_timeout_on_transaction_abort` (disabled by default) to avoid a fatal `INVALID_TXN_STATE` on abort ([librdkafka#4849](https://github.com/confluentinc/librdkafka/issues/4849)). When enabled, the awaited message is delivered as aborted instead of purged.

data/Gemfile

@@ -4,6 +4,7 @@ source "https://rubygems.org"
 
 gemspec
 
+gem "rake"
 gem "zeitwerk", "~> 2.8.0"
 
 group :test do

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    waterdrop (2.10.8)
+    waterdrop (2.10.9)
       karafka-core (>= 2.5.12, < 3.0.0)
       karafka-rdkafka (>= 0.31.0)
       zeitwerk (~> 2.3)
@@ -45,6 +45,7 @@ DEPENDENCIES
   minitest
   mocha
   ostruct
+  rake
   simplecov
   warning
   waterdrop!
@@ -67,7 +68,7 @@ CHECKSUMS
   ruby2_keywords (0.0.5) sha256=ffd13740c573b7301cf7a2e61fc857b2a8e3d3aff32545d6f8300d8bae10e3ef
   simplecov (1.3.2) sha256=2a22ec1759f1b8db60881a2e038758956fc69ef93ed669be0bde0678de6d0e85
   warning (1.6.0) sha256=a49cdfae19fb77d19afff2efbe45f8ab759e9cd25b4e4ce2c79dbaf46bdb6c9e
-  waterdrop (2.10.8)
+  waterdrop (2.10.9)
   zeitwerk (2.8.3) sha256=2c85125a8467ce069e20123d1e709a08955c9d29c118c25b46b7b7fafdbb92e5
 
 BUNDLED WITH

data/lib/waterdrop/clients/buffered.rb

@@ -75,9 +75,9 @@ module WaterDrop
       end
 
       # Returns messages produced to a given topic
-      # @param topic [String]
-      def messages_for(topic)
-        @topics[topic]
+      # @param topic_name [String]
+      def messages_for(topic_name)
+        @topics[topic_name]
       end
 
       # Clears internal buffer. Used in between specs so messages do not leak out

data/lib/waterdrop/clients/dummy.rb

@@ -10,11 +10,11 @@ module WaterDrop
     class Dummy
       # `::Rdkafka::Producer::DeliveryHandle` object API compatible dummy object
       class Handle < ::Rdkafka::Producer::DeliveryHandle
-        # @param topic [String] topic where we want to dispatch message
+        # @param topic_name [String] topic where we want to dispatch message
         # @param partition [Integer] target partition
         # @param offset [Integer] offset assigned by our fake "Kafka"
-        def initialize(topic, partition, offset)
-          @topic = topic
+        def initialize(topic_name, partition, offset)
+          @topic = topic_name
           @partition = partition
           @offset = offset
         end

data/lib/waterdrop/errors.rb

@@ -23,7 +23,17 @@ module WaterDrop
     ProducerUsedInParentProcess = Class.new(BaseError)
 
     # Raised when there was an attempt to use a closed producer
-    ProducerClosedError = Class.new(BaseError)
+    class ProducerClosedError < BaseError
+      attr_reader :dispatched
+
+      # @param message [String, nil] error message
+      # @param dispatched [Array<Rdkafka::Producer::DeliveryHandle>] handlers of the messages of a
+      #   batch that we've dispatched before the producer got closed. Empty outside of a batch.
+      def initialize(message = nil, dispatched: [])
+        super(message)
+        @dispatched = dispatched
+      end
+    end
 
     # Raised if you attempt to close the producer from within a transaction. This is not allowed.
     ProducerTransactionalCloseAttemptError = Class.new(BaseError)

data/lib/waterdrop/instrumentation/vendors/datadog/metrics_listener.rb

@@ -190,10 +190,10 @@ module WaterDrop
           end
 
           # Report that a message has been produced to a topic.
-          # @param topic [String] Kafka topic
+          # @param topic_name [String] Kafka topic
           # @param method_name [Symbol] method from which this message operation comes
-          def report_message(topic, method_name)
-            increment(method_name, tags: default_tags + ["topic:#{topic}"])
+          def report_message(topic_name, method_name)
+            increment(method_name, tags: default_tags + ["topic:#{topic_name}"])
           end
 
           # Wraps metric name in listener's namespace

data/lib/waterdrop/producer/async.rb

@@ -53,6 +53,8 @@ module WaterDrop
       # @raise [Rdkafka::RdkafkaError] When adding the messages to rdkafka's queue failed
       # @raise [Errors::MessageInvalidError] When any of the provided messages details are invalid
       #   and the message could not be sent to Kafka
+      # @raise [Errors::ProducerClosedError] When the producer got closed mid-batch. Its
+      #   `#dispatched` holds the handlers of the messages dispatched before that
       def produce_many_async(messages, run_middleware: true)
         dispatched = []
         messages = middleware.run_many(messages) if run_middleware
@@ -84,6 +86,8 @@ module WaterDrop
         )
 
         raise re_raised
+      rescue Errors::ProducerClosedError => e
+        raise Errors::ProducerClosedError.new(e.message, dispatched: dispatched)
       end
     end
   end

data/lib/waterdrop/producer/buffer.rb

@@ -28,7 +28,7 @@ module WaterDrop
         ) do
           @buffer_mutex.synchronize do
             ensure_active!
-            @messages << message
+            @messages_mutex.synchronize { @messages << message }
           end
         end
       end
@@ -55,7 +55,7 @@ module WaterDrop
         ) do
           @buffer_mutex.synchronize do
             ensure_active!
-            @messages.concat(messages)
+            @messages_mutex.synchronize { @messages.concat(messages) }
           end
 
           messages
@@ -91,6 +91,7 @@ module WaterDrop
       # @return [Array<Rdkafka::Producer::DeliveryHandle>] delivery handles (in final state for
       #   sync, pending for async)
       # @raise [Errors::ProduceManyError] when there was a failure in flushing
+      # @raise [Errors::ProducerClosedError] when the producer got closed during flushing
       # @note We use this method underneath to provide a different instrumentation for sync and
       #   async flushing within the public API
       def flush(sync)
@@ -99,10 +100,12 @@ module WaterDrop
         data_for_dispatch = []
 
         @buffer_mutex.synchronize do
-          requeued = @requeued
-          @requeued = []
-          fresh = @messages
-          @messages = []
+          @messages_mutex.synchronize do
+            requeued = @requeued
+            @requeued = []
+            fresh = @messages
+            @messages = []
+          end
         end
 
         # Middleware runs exactly once per message: on the fresh messages here, never again on the
@@ -141,12 +144,13 @@ module WaterDrop
         else
           produce_many_async(data_for_dispatch, run_middleware: false)
         end
-      rescue Errors::ProduceManyError => e
-        # A dispatch failed partway through the batch. Re-buffer the messages that never reached
-        # librdkafka so a partial failure does not silently drop valid buffered messages. For a
-        # transactional producer the whole batch is rolled back (nothing is visible to consumers),
-        # so all of it is restored; for a regular producer `e.dispatched` holds the handles already
-        # created, so only the remainder is restored.
+      rescue Errors::ProduceManyError, Errors::ProducerClosedError => e
+        # A dispatch failed partway through the batch, either on an inline error or because a
+        # concurrent close stopped it. Re-buffer the messages that never reached librdkafka so a
+        # partial failure does not silently drop valid buffered messages. For a transactional
+        # producer the whole batch is rolled back (nothing is visible to consumers), so all of it is
+        # restored; for a regular producer `e.dispatched` holds the handles already created, so only
+        # the remainder is restored.
         requeue_unflushed(transactional? ? data_for_dispatch : data_for_dispatch.drop(e.dispatched.size))
 
         raise
@@ -166,8 +170,10 @@ module WaterDrop
       # @param requeued [Array<Hash>] already middleware-processed messages
       def restore_undispatched(fresh, requeued)
         @buffer_mutex.synchronize do
-          @messages.unshift(*fresh)
-          @requeued.unshift(*requeued)
+          @messages_mutex.synchronize do
+            @messages.unshift(*fresh)
+            @requeued.unshift(*requeued)
+          end
         end
       end
 
@@ -178,7 +184,9 @@ module WaterDrop
       def requeue_unflushed(messages)
         return if messages.empty?
 
-        @buffer_mutex.synchronize { @requeued.unshift(*messages) }
+        @buffer_mutex.synchronize do
+          @messages_mutex.synchronize { @requeued.unshift(*messages) }
+        end
       end
     end
   end

data/lib/waterdrop/producer/sync.rb

@@ -60,6 +60,8 @@ module WaterDrop
       #   handles are still pending
       # @raise [Errors::MessageInvalidError] When any of the provided messages details are invalid
       #   and the message could not be sent to Kafka
+      # @raise [Errors::ProducerClosedError] When the producer got closed mid-batch. Its
+      #   `#dispatched` holds the handlers of the messages dispatched before that
       #
       # @note We return delivery handles instead of delivery reports to ensure consistent return
       #   types for both success and error flows. In case of partial failures (e.g., mid-batch
@@ -67,10 +69,10 @@ module WaterDrop
       #   allowing uniform error handling. Each handle is in its final state after this method
       #   returns, so you can call `handle.create_result` to obtain the delivery report if needed.
       def produce_many_sync(messages, run_middleware: true)
+        dispatched = []
         messages = middleware.run_many(messages) if run_middleware
         messages.each { |message| validate_message!(message) }
 
-        dispatched = []
         inline_error = nil
 
         @monitor.instrument("messages.produced_sync", producer_id: id, messages: messages) do
@@ -128,6 +130,8 @@ module WaterDrop
         )
 
         raise re_raised
+      rescue Errors::ProducerClosedError => e
+        raise Errors::ProducerClosedError.new(e.message, dispatched: dispatched)
       end
     end
   end

data/lib/waterdrop/producer.rb

@@ -50,6 +50,11 @@ module WaterDrop
     def initialize(&block)
       @operations_in_progress = Helpers::Counter.new
       @buffer_mutex = Mutex.new
+      # Guards only the in-place changes of @messages and @requeued and the copy in #messages. It
+      # is always taken inside @buffer_mutex and held for one array operation, never around
+      # callbacks, so #messages does not wait for a @buffer_mutex holder (e.g. #disconnect waiting
+      # on the poller thread whose callbacks may call #messages).
+      @messages_mutex = Mutex.new
       @connecting_mutex = Mutex.new
       @operating_mutex = Mutex.new
       @transaction_mutex = Mutex.new
@@ -182,6 +187,12 @@ module WaterDrop
     # @return [Array] internal messages buffer awaiting a flush. Includes both freshly buffered
     #   messages and any messages re-buffered after a failed flush that are waiting to be retried.
     def messages
+      @messages_mutex.synchronize { @requeued + @messages }
+    rescue ThreadError => e
+      # Mutex#synchronize raises in a signal trap, so there the copy is taken without the lock.
+      # It is not safe against concurrent buffering without a GVL (JRuby).
+      raise unless e.message == "can't be called from trap context"
+
       @requeued + @messages
     end
 
@@ -222,13 +233,13 @@ module WaterDrop
 
     # Fetches and caches the partition count of a topic
     #
-    # @param topic [String] topic for which we want to get the number of partitions
+    # @param topic_name [String] topic for which we want to get the number of partitions
     # @return [Integer] number of partitions of the requested topic or -1 if number could not be
     #   retrieved.
     #
     # @note It uses the underlying `rdkafka-ruby` partition count fetch and cache.
-    def partition_count(topic)
-      client.partition_count(topic.to_s)
+    def partition_count(topic_name)
+      client.partition_count(topic_name.to_s)
     end
 
     # Purges data from both the buffer queue as well as the librdkafka queue.
@@ -239,8 +250,10 @@ module WaterDrop
     def purge
       @monitor.instrument("buffer.purged", producer_id: id) do
         @buffer_mutex.synchronize do
-          @messages = []
-          @requeued = []
+          @messages_mutex.synchronize do
+            @messages = []
+            @requeued = []
+          end
         end
 
         # We should not purge if there is no client initialized

data/lib/waterdrop/version.rb

@@ -2,5 +2,5 @@
 
 module WaterDrop
   # Current WaterDrop version
-  VERSION = "2.10.8"
+  VERSION = "2.10.9"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: waterdrop
 version: !ruby/object:Gem::Version
-  version: 2.10.8
+  version: 2.10.9
 platform: ruby
 authors:
 - Maciej Mensfeld