waterdrop 2.10.7 → 2.10.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6a34bd3898b2807134dc7941c0b22e66933a242de7ecf1e7cc7e1c76a5f23267
-  data.tar.gz: 8ca1ca970bc65ea3d27ca8c529f493c33b95388304dd936402d1562e2f151c79
+  metadata.gz: 0e6efd47232839d14e444c45df4cc5fae39661dd7e8ef646639f00d41f62d3b3
+  data.tar.gz: edd8560d05bb3742d8592a1b90f2dc784410e7150045e8a8d4a0ebf1c0d82e81
 SHA512:
-  metadata.gz: 5357203176e130d237e79fe7e936ac547874d16a9bb6f3d96e30d57b0c24523508ebc6754086c01fdeb7ec33542d48bc95c2cccd74887e6ced30e8484625a5f7
-  data.tar.gz: 6e6ec7ad65759afb4a0aa923a372e1a4e00cf447dd7c9ed00805f8ddf7e59d0dea92b7edc819d8d86c570128e0997faf23cae3c9c9ea97d8706866b5d0baccde
+  metadata.gz: 1532bf1ecbb02ec192bdd8a410b5e7e7579337e11515fe828fb88bd8cf2a2168e31ef669e9d3ba1c334e64720c148069d26b5259d7c3f913d80258d5ddbb711b
+  data.tar.gz: 87178e647ac42309e0cdbef6c48af8c0f7a902ee32238c3c5cf8cbca2ee7ae80396895bb3a70417c912f98f669834e19ce1066c596a7a5432c7245b29a537248

data/.ruby-version

@@ -1 +1 @@
-4.0.7
+4.0.9

data/CHANGELOG.md

@@ -1,7 +1,23 @@
 # WaterDrop changelog
 
+## 2.10.9 (2026-11-02)
+- [Maintenance] Rename the positional topic name parameters of `Producer#partition_count`, `Clients::Buffered#messages_for` and the Datadog metrics listener to `topic_name`.
+- [Maintenance] Rename the positional topic name parameter of `Clients::Dummy::Handle#initialize` to `topic_name`.
+- [Maintenance] Declare `rake` in the `Gemfile`. It came only through `karafka-rdkafka`, which will stop depending on it for precompiled platform gems.
+- [Fix] Copy the buffer in `Producer#messages` (and so the `buffer.flushed_async`/`buffer.flushed_sync` payloads) under a dedicated lock, so reading it while other threads buffer no longer raises `ConcurrencyError` on JRuby.
+- [Fix] Re-buffer the undispatched rest of a `#flush` batch when a concurrent `#close` stops it, and expose the handles dispatched before the close as `ProducerClosedError#dispatched`.
+- [Fix] Copy the buffer in `Producer#messages` (and so the `buffer.flushed_async`/`buffer.flushed_sync` payloads) under a dedicated lock, so reading it while other threads buffer no longer raises `ConcurrencyError` on JRuby.
+
+## 2.10.8 (2026-10-25)
+- [Maintenance] Document that a transaction timeout also raises `:fenced`, indistinguishable from a takeover, and when removing `:fenced` from `non_reloadable_errors` is safe.
+- [Maintenance] Drop unused `byebug` from the test bundle.
+- [Maintenance] Fix test races and skip the fork test on JRuby, resolving unstable behavior in `test/` only.
+- [Fix] Stop `Producer#close` from raising `Errno::EPIPE` on JRuby when its close signal races the poller closing the pipe.
+- [Fix] Abort a failed transaction only once: raise the abortable commit error instead of a follow-up `Operation not valid in state Ready` error, and do not abort after a failed `begin_transaction`.
+
 ## 2.10.7 (2026-10-19)
 - [Change] Require `karafka-rdkafka` `>=` `0.31.0` so delivery reports carry the message persistence status, latency and broker id.
+- [Maintenance] Fix a flaky transaction purge instrumentation spec on slow runners.
 
 ## 2.10.6 (2026-10-12)
 - **[EOL]** Drop Ruby 3.2 support; require Ruby 3.3+.
@@ -14,10 +30,10 @@
 - [Fix] Restore both buffers when `#flush` fails before anything is dispatched, instead of discarding the whole batch.
 
 ## 2.10.4 (2026-08-26)
+- [Maintenance] Stop the `#partition_count when topic does not exist` spec from flaking on slow CI runners by waiting for authoritative broker metadata before asserting the count.
 - [Fix] Avoid a `FrozenError` on `Producer#close` when the producer is configured with a frozen string id (for example a frozen string literal, `config.id = "rspec"`).
 - [Fix] Emit `connection_pool.shutdown` only once when shutting down the global connection pool.
 - [Fix] Retry a produce that races a concurrent idempotent fatal-error reload instead of leaking a raw `Rdkafka::ClosedProducerError`/`Rdkafka::ClosedInnerError`.
-- [Maintenance] Stop the `#partition_count when topic does not exist` spec from flaking on slow CI runners by waiting for authoritative broker metadata before asserting the count.
 
 ## 2.10.3 (2026-07-15)
 - [Feature] Add `wait_timeout_on_transaction_abort` (disabled by default) to avoid a fatal `INVALID_TXN_STATE` on abort ([librdkafka#4849](https://github.com/confluentinc/librdkafka/issues/4849)). When enabled, the awaited message is delivered as aborted instead of purged.
@@ -48,7 +64,7 @@
 ## 2.10.0 (2026-05-07)
 - [Fix] Clean up the native client, instrumentation callbacks and poller registration when `init_transactions` fails during producer construction, so failed attempts no longer leak threads and file descriptors.
 - **[Breaking]** Skip librdkafka statistics when nothing subscribes to `statistics.emitted` before the rdkafka client is built, to save allocations. To use statistics, subscribe a listener BEFORE the first producer use.
-- **[Breaking]** Raise `WaterDrop::Errors::StatisticsNotEnabledError` when attempting to subscribe to `statistics.emitted` on a monitor where librdkafka statistics have been disabled at client build time, replacing the previous "silent nothing" failure mode.
+- **[Breaking]** Raise `WaterDrop::Errors::StatisticsNotEnabledError` when subscribing to `statistics.emitted` on a monitor whose librdkafka statistics were disabled at client build time, instead of silently doing nothing.
 - [Feature] Add tombstone API (`#tombstone_sync`, `#tombstone_async`, `#tombstone_many_sync`, `#tombstone_many_async`) for producing tombstone records (nil-payload messages) with required key and partition validation. Works with variants.
 - [Fix] Add `ensure_same_process!` to `Poller#unregister` for fork safety, so a child that inherited a pre-fork producer no longer deadlocks on `producer.close`.
 
@@ -57,7 +73,7 @@
 - [Fix] Exclude test files, `.github/`, and `log/` directories from gem releases to reduce package size.
 - **[Breaking]** Switch default polling mode from `:thread` to `:fd`. If you experience any issues, you can revert to the previous behavior by setting `config.polling.mode = :thread`. The `:thread` mode will be deprecated in 2.10 and removed in 2.11.
 - **[Breaking]** Decorate only the statistics keys used by the built-in Datadog listener (`tx`, `txretries`, `txerrs`, `rxerrs`). If you use other `_d` or `_fd` keys, set a custom `config.statistics_decorator`.
-- [Feature] Add `config.statistics_decorator` setting to allow full control over the `StatisticsDecorator` instance used for statistics decoration. Users can provide a custom decorator with different `only_keys` and `excluded_keys` to match their instrumentation needs.
+- [Feature] Add a `config.statistics_decorator` setting to use a custom `StatisticsDecorator` (e.g. with different `only_keys` and `excluded_keys`).
 - [Change] Upscale default timeout values 3x closer to librdkafka defaults to prevent intermediate timeouts during node recovery (`message.timeout.ms`: 50s → 150s, `transaction.timeout.ms`: 55s → 165s, `max_wait_timeout`: 60s → 180s).
 
 **Upgrade Notes**: https://karafka.io/docs/Upgrades-WaterDrop-2.9/
@@ -65,7 +81,7 @@
 ## 2.8.16 (2026-02-25)
 - [Feature] Add FD-based polling (`config.polling.mode = :fd`): a single Ruby thread with `IO.select` instead of the default thread mode, with 39-54% higher throughput, lower memory usage and fewer threads.
 - [Feature] Add `config.polling.fd.max_time` setting (default: 100ms) to control maximum polling time per producer per cycle. This enables per-producer priority differentiation.
-- [Feature] Add `Producer#queue_size` (alias: `#queue_length`) to return the count of messages pending in the librdkafka queue. This counts messages that have been dispatched to librdkafka but not yet transmitted to the Kafka broker.
+- [Feature] Add `Producer#queue_size` (alias: `#queue_length`) to return the count of messages dispatched to librdkafka but not yet sent to the Kafka broker.
 - [Enhancement] Add `poll_nb` and `poll_drain` methods to rdkafka Producer for efficient non-blocking polling without GVL release overhead.
 - [Enhancement] Add `poller.producer_registered` and `poller.producer_unregistered` instrumentation events for FD mode.
 - [Change] Update to use `max_wait_timeout_ms` parameter in rdkafka `AbstractHandle#wait` for consistency with librdkafka conventions.
@@ -108,10 +124,10 @@
 
 ## 2.8.10 (2025-09-25)
 - [Enhancement] Add `#close` alias for `WaterDrop::ConnectionPool#shutdown` to align with producer API for consistent interface across both individual producers and connection pools.
-- [Enhancement] Add `WaterDrop.monitor` method as the preferred alias for `WaterDrop.instrumentation` to align with per-producer API naming convention. The `instrumentation` method remains available as a deprecated alias for backward compatibility.
+- [Enhancement] Add `WaterDrop.monitor` as the preferred alias for `WaterDrop.instrumentation`, matching the per-producer API. `instrumentation` remains as a deprecated alias.
 
 ## 2.8.9 (2025-09-23)
-- [Enhancement] Add connection pool lifecycle events to global instrumentation for improved observability. Events include `connection_pool.created`, `connection_pool.setup`, `connection_pool.shutdown`, `connection_pool.reload`, and `connection_pool.reloaded`.
+- [Enhancement] Add connection pool lifecycle events to global instrumentation: `connection_pool.created`, `connection_pool.setup`, `connection_pool.shutdown`, `connection_pool.reload` and `connection_pool.reloaded`.
 - [Enhancement] Add default connection pool transactional direct API.
 
 ## 2.8.8 (2025-09-23)

data/Gemfile

@@ -4,12 +4,9 @@ source "https://rubygems.org"
 
 gemspec
 
+gem "rake"
 gem "zeitwerk", "~> 2.8.0"
 
-group :development do
-  gem "byebug"
-end
-
 group :test do
   gem "connection_pool"
   gem "ostruct"

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    waterdrop (2.10.7)
+    waterdrop (2.10.9)
       karafka-core (>= 2.5.12, < 3.0.0)
       karafka-rdkafka (>= 0.31.0)
       zeitwerk (~> 2.3)
@@ -9,12 +9,9 @@ PATH
 GEM
   remote: https://rubygems.org/
   specs:
-    byebug (13.0.0)
-      reline (>= 0.6.0)
     connection_pool (3.0.2)
     drb (2.2.3)
     ffi (1.17.4)
-    io-console (0.9.4)
     json (3.0.2)
     karafka-core (2.6.4)
       karafka-rdkafka (>= 0.20.0)
@@ -35,8 +32,6 @@ GEM
     ostruct (0.6.3)
     prism (1.9.0)
     rake (13.4.2)
-    reline (0.7.0)
-      io-console (~> 0.5)
     ruby2_keywords (0.0.5)
     simplecov (1.3.2)
     warning (1.6.0)
@@ -46,22 +41,20 @@ PLATFORMS
   ruby
 
 DEPENDENCIES
-  byebug
   connection_pool
   minitest
   mocha
   ostruct
+  rake
   simplecov
   warning
   waterdrop!
   zeitwerk (~> 2.8.0)
 
 CHECKSUMS
-  byebug (13.0.0) sha256=d2263efe751941ca520fa29744b71972d39cbc41839496706f5d9b22e92ae05d
   connection_pool (3.0.2) sha256=33fff5ba71a12d2aa26cb72b1db8bba2a1a01823559fb01d29eb74c286e62e0a
   drb (2.2.3) sha256=0b00d6fdb50995fe4a45dea13663493c841112e4068656854646f418fda13373
   ffi (1.17.4) sha256=bcd1642e06f0d16fc9e09ac6d49c3a7298b9789bcb58127302f934e437d60acf
-  io-console (0.9.4) sha256=a403304f32928ee85e56d457b4612ca43f2703b561a3258844992bd24085fbb1
   json (3.0.2) sha256=8e6d7e7b11384c21230430cef90b71f14849a34a1f4452796670f7c981bd19df
   karafka-core (2.6.4) sha256=e08166ca7ef12fadf0500f2a3aba63b5ba918a98533c3feaac9f43c90f159363
   karafka-rdkafka (0.31.0) sha256=7d21274122d66ee8d08b8e56a0d06084933deb2306cf2a6e4bd0d073c14e7f9f
@@ -72,11 +65,10 @@ CHECKSUMS
   ostruct (0.6.3) sha256=95a2ed4a4bd1d190784e666b47b2d3f078e4a9efda2fccf18f84ddc6538ed912
   prism (1.9.0) sha256=7b530c6a9f92c24300014919c9dcbc055bf4cdf51ec30aed099b06cd6674ef85
   rake (13.4.2) sha256=cb825b2bd5f1f8e91ca37bddb4b9aaf345551b4731da62949be002fa89283701
-  reline (0.7.0) sha256=5b012d8e55dbf9d450f12bde2cf7d15ff546ae80b3f8f3b30e570d431815583d
   ruby2_keywords (0.0.5) sha256=ffd13740c573b7301cf7a2e61fc857b2a8e3d3aff32545d6f8300d8bae10e3ef
   simplecov (1.3.2) sha256=2a22ec1759f1b8db60881a2e038758956fc69ef93ed669be0bde0678de6d0e85
   warning (1.6.0) sha256=a49cdfae19fb77d19afff2efbe45f8ab759e9cd25b4e4ce2c79dbaf46bdb6c9e
-  waterdrop (2.10.7)
+  waterdrop (2.10.9)
   zeitwerk (2.8.3) sha256=2c85125a8467ce069e20123d1e709a08955c9d29c118c25b46b7b7fafdbb92e5
 
 BUNDLED WITH

data/lib/waterdrop/clients/buffered.rb

@@ -75,9 +75,9 @@ module WaterDrop
       end
 
       # Returns messages produced to a given topic
-      # @param topic [String]
-      def messages_for(topic)
-        @topics[topic]
+      # @param topic_name [String]
+      def messages_for(topic_name)
+        @topics[topic_name]
       end
 
       # Clears internal buffer. Used in between specs so messages do not leak out

data/lib/waterdrop/clients/dummy.rb

@@ -10,11 +10,11 @@ module WaterDrop
     class Dummy
       # `::Rdkafka::Producer::DeliveryHandle` object API compatible dummy object
       class Handle < ::Rdkafka::Producer::DeliveryHandle
-        # @param topic [String] topic where we want to dispatch message
+        # @param topic_name [String] topic where we want to dispatch message
         # @param partition [Integer] target partition
         # @param offset [Integer] offset assigned by our fake "Kafka"
-        def initialize(topic, partition, offset)
-          @topic = topic
+        def initialize(topic_name, partition, offset)
+          @topic = topic_name
           @partition = partition
           @offset = offset
         end

data/lib/waterdrop/config.rb

@@ -130,13 +130,18 @@ module WaterDrop
     #   WARNING: Modifying this setting can cause infinite reload loops if not properly understood.
     #   The default includes :fenced errors because:
     #   - Fencing occurs when another producer with the same transactional.id takes over
-    #   - This is an unrecoverable state - reloading won't help as the other producer is active
+    #   - Fencing also occurs when the broker aborts a transaction that exceeded
+    #     `transaction.timeout.ms`. Both cases raise the same :fenced error, so WaterDrop cannot
+    #     tell them apart
+    #   - After a takeover this is an unrecoverable state - reloading won't help as the other
+    #     producer is active
     #   - Attempting to reload on fenced errors creates: produce -> fenced -> reload -> produce ->
     #     fenced -> reload (infinite loop)
     #
-    #   Only remove :fenced from this list if you have explicit logic to handle producer fencing
-    #   in your application (e.g., coordinated transactional.id assignment, manual intervention).
-    #   In most cases, you should keep the default value.
+    #   Only remove :fenced from this list if each transactional.id is used by one producer at a
+    #   time (e.g., coordinated transactional.id assignment). Otherwise a reload fences the newer
+    #   producer and the two can keep fencing each other for as long as both run. In most cases,
+    #   you should keep the default value.
     setting :non_reloadable_errors, default: %i[fenced]
     # option [Integer] Idle disconnect timeout in milliseconds. When set to 0, idle disconnection
     #   is disabled. When set to a positive value, WaterDrop will automatically disconnect

data/lib/waterdrop/errors.rb

@@ -23,7 +23,17 @@ module WaterDrop
     ProducerUsedInParentProcess = Class.new(BaseError)
 
     # Raised when there was an attempt to use a closed producer
-    ProducerClosedError = Class.new(BaseError)
+    class ProducerClosedError < BaseError
+      attr_reader :dispatched
+
+      # @param message [String, nil] error message
+      # @param dispatched [Array<Rdkafka::Producer::DeliveryHandle>] handlers of the messages of a
+      #   batch that we've dispatched before the producer got closed. Empty outside of a batch.
+      def initialize(message = nil, dispatched: [])
+        super(message)
+        @dispatched = dispatched
+      end
+    end
 
     # Raised if you attempt to close the producer from within a transaction. This is not allowed.
     ProducerTransactionalCloseAttemptError = Class.new(BaseError)

data/lib/waterdrop/instrumentation/vendors/datadog/metrics_listener.rb

@@ -190,10 +190,10 @@ module WaterDrop
           end
 
           # Report that a message has been produced to a topic.
-          # @param topic [String] Kafka topic
+          # @param topic_name [String] Kafka topic
           # @param method_name [Symbol] method from which this message operation comes
-          def report_message(topic, method_name)
-            increment(method_name, tags: default_tags + ["topic:#{topic}"])
+          def report_message(topic_name, method_name)
+            increment(method_name, tags: default_tags + ["topic:#{topic_name}"])
           end
 
           # Wraps metric name in listener's namespace

data/lib/waterdrop/polling/queue_pipe.rb

@@ -29,8 +29,8 @@ module WaterDrop
       # Thread-safe and non-blocking; silently ignores errors
       def signal
         @writer.write_nonblock("W", exception: false)
-      rescue IOError, Errno::EBADF
-        # Pipe closed
+      rescue IOError, Errno::EBADF, Errno::EPIPE
+        # Pipe closed. EPIPE: the reader is already closed, so there is no one left to wake
       end
 
       # Drains all pending bytes from the pipe

data/lib/waterdrop/producer/async.rb

@@ -53,6 +53,8 @@ module WaterDrop
       # @raise [Rdkafka::RdkafkaError] When adding the messages to rdkafka's queue failed
       # @raise [Errors::MessageInvalidError] When any of the provided messages details are invalid
       #   and the message could not be sent to Kafka
+      # @raise [Errors::ProducerClosedError] When the producer got closed mid-batch. Its
+      #   `#dispatched` holds the handlers of the messages dispatched before that
       def produce_many_async(messages, run_middleware: true)
         dispatched = []
         messages = middleware.run_many(messages) if run_middleware
@@ -84,6 +86,8 @@ module WaterDrop
         )
 
         raise re_raised
+      rescue Errors::ProducerClosedError => e
+        raise Errors::ProducerClosedError.new(e.message, dispatched: dispatched)
       end
     end
   end

data/lib/waterdrop/producer/buffer.rb

@@ -28,7 +28,7 @@ module WaterDrop
         ) do
           @buffer_mutex.synchronize do
             ensure_active!
-            @messages << message
+            @messages_mutex.synchronize { @messages << message }
           end
         end
       end
@@ -55,7 +55,7 @@ module WaterDrop
         ) do
           @buffer_mutex.synchronize do
             ensure_active!
-            @messages.concat(messages)
+            @messages_mutex.synchronize { @messages.concat(messages) }
           end
 
           messages
@@ -91,6 +91,7 @@ module WaterDrop
       # @return [Array<Rdkafka::Producer::DeliveryHandle>] delivery handles (in final state for
       #   sync, pending for async)
       # @raise [Errors::ProduceManyError] when there was a failure in flushing
+      # @raise [Errors::ProducerClosedError] when the producer got closed during flushing
       # @note We use this method underneath to provide a different instrumentation for sync and
       #   async flushing within the public API
       def flush(sync)
@@ -99,10 +100,12 @@ module WaterDrop
         data_for_dispatch = []
 
         @buffer_mutex.synchronize do
-          requeued = @requeued
-          @requeued = []
-          fresh = @messages
-          @messages = []
+          @messages_mutex.synchronize do
+            requeued = @requeued
+            @requeued = []
+            fresh = @messages
+            @messages = []
+          end
         end
 
         # Middleware runs exactly once per message: on the fresh messages here, never again on the
@@ -141,12 +144,13 @@ module WaterDrop
         else
           produce_many_async(data_for_dispatch, run_middleware: false)
         end
-      rescue Errors::ProduceManyError => e
-        # A dispatch failed partway through the batch. Re-buffer the messages that never reached
-        # librdkafka so a partial failure does not silently drop valid buffered messages. For a
-        # transactional producer the whole batch is rolled back (nothing is visible to consumers),
-        # so all of it is restored; for a regular producer `e.dispatched` holds the handles already
-        # created, so only the remainder is restored.
+      rescue Errors::ProduceManyError, Errors::ProducerClosedError => e
+        # A dispatch failed partway through the batch, either on an inline error or because a
+        # concurrent close stopped it. Re-buffer the messages that never reached librdkafka so a
+        # partial failure does not silently drop valid buffered messages. For a transactional
+        # producer the whole batch is rolled back (nothing is visible to consumers), so all of it is
+        # restored; for a regular producer `e.dispatched` holds the handles already created, so only
+        # the remainder is restored.
         requeue_unflushed(transactional? ? data_for_dispatch : data_for_dispatch.drop(e.dispatched.size))
 
         raise
@@ -166,8 +170,10 @@ module WaterDrop
       # @param requeued [Array<Hash>] already middleware-processed messages
       def restore_undispatched(fresh, requeued)
         @buffer_mutex.synchronize do
-          @messages.unshift(*fresh)
-          @requeued.unshift(*requeued)
+          @messages_mutex.synchronize do
+            @messages.unshift(*fresh)
+            @requeued.unshift(*requeued)
+          end
         end
       end
 
@@ -178,7 +184,9 @@ module WaterDrop
       def requeue_unflushed(messages)
         return if messages.empty?
 
-        @buffer_mutex.synchronize { @requeued.unshift(*messages) }
+        @buffer_mutex.synchronize do
+          @messages_mutex.synchronize { @requeued.unshift(*messages) }
+        end
       end
     end
   end

data/lib/waterdrop/producer/sync.rb

@@ -60,6 +60,8 @@ module WaterDrop
       #   handles are still pending
       # @raise [Errors::MessageInvalidError] When any of the provided messages details are invalid
       #   and the message could not be sent to Kafka
+      # @raise [Errors::ProducerClosedError] When the producer got closed mid-batch. Its
+      #   `#dispatched` holds the handlers of the messages dispatched before that
       #
       # @note We return delivery handles instead of delivery reports to ensure consistent return
       #   types for both success and error flows. In case of partial failures (e.g., mid-batch
@@ -67,10 +69,10 @@ module WaterDrop
       #   allowing uniform error handling. Each handle is in its final state after this method
       #   returns, so you can call `handle.create_result` to obtain the delivery report if needed.
       def produce_many_sync(messages, run_middleware: true)
+        dispatched = []
         messages = middleware.run_many(messages) if run_middleware
         messages.each { |message| validate_message!(message) }
 
-        dispatched = []
         inline_error = nil
 
         @monitor.instrument("messages.produced_sync", producer_id: id, messages: messages) do
@@ -128,6 +130,8 @@ module WaterDrop
         )
 
         raise re_raised
+      rescue Errors::ProducerClosedError => e
+        raise Errors::ProducerClosedError.new(e.message, dispatched: dispatched)
       end
     end
   end

data/lib/waterdrop/producer/transactions.rb

@@ -65,7 +65,12 @@ module WaterDrop
 
           transactional_instrument(:finished) do
             with_transactional_error_handling(:begin) do
-              transactional_instrument(:started) { client.begin_transaction }
+              transactional_instrument(:started) do
+                client.begin_transaction
+                # Set before the `started` listeners run, so an error raised by one of them still
+                # aborts the transaction that has begun
+                @transaction_open = true
+              end
             end
 
             result = nil
@@ -100,6 +105,15 @@ module WaterDrop
           # We need to handle any interrupt including critical in order not to have the transaction
           # running. This will also handle things like `IRB::Abort`
           rescue Exception => e
+            # Neither a transaction that never began nor one the error handling already aborted
+            # (abortable error) has anything left to abort: aborting again would fail with an
+            # invalid state error that replaces the original one
+            unless @transaction_open
+              transactional_reload_client_if_needed(e)
+
+              raise
+            end
+
             # This code is a bit tricky. We have an error and when it happens we try to rollback
             # the transaction. However we may end up in a state where transaction aborting itself
             # produces error. In such case we also want to handle it as fatal and reload client.
@@ -124,6 +138,8 @@ module WaterDrop
 
             raise unless e.is_a?(WaterDrop::Errors::AbortTransaction)
           ensure
+            @transaction_open = false
+
             # The first delivery handle is scoped to the transaction that produced it and is only
             # ever read by the abort path (which runs in the rescue above, before this). Clearing it
             # here - on commit, on abort and on a re-raise alike - keeps it from outliving its
@@ -273,6 +289,8 @@ module WaterDrop
           with_transactional_error_handling(:abort, allow_abortable: false) do
             transactional_instrument(:aborted) { client.abort_transaction }
           end
+
+          @transaction_open = false
         end
 
         raise

data/lib/waterdrop/producer.rb

@@ -50,6 +50,11 @@ module WaterDrop
     def initialize(&block)
       @operations_in_progress = Helpers::Counter.new
       @buffer_mutex = Mutex.new
+      # Guards only the in-place changes of @messages and @requeued and the copy in #messages. It
+      # is always taken inside @buffer_mutex and held for one array operation, never around
+      # callbacks, so #messages does not wait for a @buffer_mutex holder (e.g. #disconnect waiting
+      # on the poller thread whose callbacks may call #messages).
+      @messages_mutex = Mutex.new
       @connecting_mutex = Mutex.new
       @operating_mutex = Mutex.new
       @transaction_mutex = Mutex.new
@@ -66,6 +71,7 @@ module WaterDrop
       @idempotent_fatal_error_attempts = 0
       @transaction_fatal_error_attempts = 0
       @transaction_first_handle = nil
+      @transaction_open = false
 
       @status = Status.new
       @messages = []
@@ -181,6 +187,12 @@ module WaterDrop
     # @return [Array] internal messages buffer awaiting a flush. Includes both freshly buffered
     #   messages and any messages re-buffered after a failed flush that are waiting to be retried.
     def messages
+      @messages_mutex.synchronize { @requeued + @messages }
+    rescue ThreadError => e
+      # Mutex#synchronize raises in a signal trap, so there the copy is taken without the lock.
+      # It is not safe against concurrent buffering without a GVL (JRuby).
+      raise unless e.message == "can't be called from trap context"
+
       @requeued + @messages
     end
 
@@ -221,13 +233,13 @@ module WaterDrop
 
     # Fetches and caches the partition count of a topic
     #
-    # @param topic [String] topic for which we want to get the number of partitions
+    # @param topic_name [String] topic for which we want to get the number of partitions
     # @return [Integer] number of partitions of the requested topic or -1 if number could not be
     #   retrieved.
     #
     # @note It uses the underlying `rdkafka-ruby` partition count fetch and cache.
-    def partition_count(topic)
-      client.partition_count(topic.to_s)
+    def partition_count(topic_name)
+      client.partition_count(topic_name.to_s)
     end
 
     # Purges data from both the buffer queue as well as the librdkafka queue.
@@ -238,8 +250,10 @@ module WaterDrop
     def purge
       @monitor.instrument("buffer.purged", producer_id: id) do
         @buffer_mutex.synchronize do
-          @messages = []
-          @requeued = []
+          @messages_mutex.synchronize do
+            @messages = []
+            @requeued = []
+          end
         end
 
         # We should not purge if there is no client initialized

data/lib/waterdrop/version.rb

@@ -2,5 +2,5 @@
 
 module WaterDrop
   # Current WaterDrop version
-  VERSION = "2.10.7"
+  VERSION = "2.10.9"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: waterdrop
 version: !ruby/object:Gem::Version
-  version: 2.10.7
+  version: 2.10.9
 platform: ruby
 authors:
 - Maciej Mensfeld