waterdrop 2.10.7 → 2.10.8

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6a34bd3898b2807134dc7941c0b22e66933a242de7ecf1e7cc7e1c76a5f23267
-  data.tar.gz: 8ca1ca970bc65ea3d27ca8c529f493c33b95388304dd936402d1562e2f151c79
+  metadata.gz: 7681d5f6c3333372a6349ddd24a1e1214a3474135755f918aa554064689976be
+  data.tar.gz: 2f9d99fff24a56acaf5575e539d647682b305f303206a75bbf10b128dd8112c8
 SHA512:
-  metadata.gz: 5357203176e130d237e79fe7e936ac547874d16a9bb6f3d96e30d57b0c24523508ebc6754086c01fdeb7ec33542d48bc95c2cccd74887e6ced30e8484625a5f7
-  data.tar.gz: 6e6ec7ad65759afb4a0aa923a372e1a4e00cf447dd7c9ed00805f8ddf7e59d0dea92b7edc819d8d86c570128e0997faf23cae3c9c9ea97d8706866b5d0baccde
+  metadata.gz: 63fcee865ef2c371bea98931f0b0ae1b5a9c118462acfb56746195547201741bba4cbdc6b6efc5feb1ec100e3c65fea8e6879da5774668f4d83691f4865cebe6
+  data.tar.gz: dc4c8e69e8ed7e6312a0aa4b4ccd7eeec3d233113b2b6b80b4308dd6602e40a7a55c3a29842ef80e258d850b3ad93482be5b68f328feebccc36d6809e5509092

data/CHANGELOG.md

@@ -1,7 +1,13 @@
 # WaterDrop changelog
 
+## 2.10.8 (2026-10-25)
+- [Maintenance] Document that a transaction timeout also raises `:fenced`, indistinguishable from a takeover, and when removing `:fenced` from `non_reloadable_errors` is safe.
+- [Fix] Stop `Producer#close` from raising `Errno::EPIPE` on JRuby when its close signal races the poller closing the pipe.
+- [Fix] Abort a failed transaction only once: raise an abortable commit error as is instead of a follow-up `Operation not valid in state Ready` error, and do not abort after a failed `begin_transaction`.
+
 ## 2.10.7 (2026-10-19)
 - [Change] Require `karafka-rdkafka` `>=` `0.31.0` so delivery reports carry the message persistence status, latency and broker id.
+- [Maintenance] Stop the transaction purge instrumentation spec from flaking when a slow runner lets librdkafka deliver the message before the abort, which leaves nothing to purge.
 
 ## 2.10.6 (2026-10-12)
 - **[EOL]** Drop Ruby 3.2 support; require Ruby 3.3+.
@@ -48,7 +54,7 @@
 ## 2.10.0 (2026-05-07)
 - [Fix] Clean up the native client, instrumentation callbacks and poller registration when `init_transactions` fails during producer construction, so failed attempts no longer leak threads and file descriptors.
 - **[Breaking]** Skip librdkafka statistics when nothing subscribes to `statistics.emitted` before the rdkafka client is built, to save allocations. To use statistics, subscribe a listener BEFORE the first producer use.
-- **[Breaking]** Raise `WaterDrop::Errors::StatisticsNotEnabledError` when attempting to subscribe to `statistics.emitted` on a monitor where librdkafka statistics have been disabled at client build time, replacing the previous "silent nothing" failure mode.
+- **[Breaking]** Raise `WaterDrop::Errors::StatisticsNotEnabledError` when subscribing to `statistics.emitted` on a monitor whose librdkafka statistics were disabled at client build time, instead of silently doing nothing.
 - [Feature] Add tombstone API (`#tombstone_sync`, `#tombstone_async`, `#tombstone_many_sync`, `#tombstone_many_async`) for producing tombstone records (nil-payload messages) with required key and partition validation. Works with variants.
 - [Fix] Add `ensure_same_process!` to `Poller#unregister` for fork safety, so a child that inherited a pre-fork producer no longer deadlocks on `producer.close`.
 
@@ -57,7 +63,7 @@
 - [Fix] Exclude test files, `.github/`, and `log/` directories from gem releases to reduce package size.
 - **[Breaking]** Switch default polling mode from `:thread` to `:fd`. If you experience any issues, you can revert to the previous behavior by setting `config.polling.mode = :thread`. The `:thread` mode will be deprecated in 2.10 and removed in 2.11.
 - **[Breaking]** Decorate only the statistics keys used by the built-in Datadog listener (`tx`, `txretries`, `txerrs`, `rxerrs`). If you use other `_d` or `_fd` keys, set a custom `config.statistics_decorator`.
-- [Feature] Add `config.statistics_decorator` setting to allow full control over the `StatisticsDecorator` instance used for statistics decoration. Users can provide a custom decorator with different `only_keys` and `excluded_keys` to match their instrumentation needs.
+- [Feature] Add a `config.statistics_decorator` setting to use a custom `StatisticsDecorator` (e.g. with different `only_keys` and `excluded_keys`).
 - [Change] Upscale default timeout values 3x closer to librdkafka defaults to prevent intermediate timeouts during node recovery (`message.timeout.ms`: 50s → 150s, `transaction.timeout.ms`: 55s → 165s, `max_wait_timeout`: 60s → 180s).
 
 **Upgrade Notes**: https://karafka.io/docs/Upgrades-WaterDrop-2.9/
@@ -65,7 +71,7 @@
 ## 2.8.16 (2026-02-25)
 - [Feature] Add FD-based polling (`config.polling.mode = :fd`): a single Ruby thread with `IO.select` instead of the default thread mode, with 39-54% higher throughput, lower memory usage and fewer threads.
 - [Feature] Add `config.polling.fd.max_time` setting (default: 100ms) to control maximum polling time per producer per cycle. This enables per-producer priority differentiation.
-- [Feature] Add `Producer#queue_size` (alias: `#queue_length`) to return the count of messages pending in the librdkafka queue. This counts messages that have been dispatched to librdkafka but not yet transmitted to the Kafka broker.
+- [Feature] Add `Producer#queue_size` (alias: `#queue_length`) to return the count of messages dispatched to librdkafka but not yet sent to the Kafka broker.
 - [Enhancement] Add `poll_nb` and `poll_drain` methods to rdkafka Producer for efficient non-blocking polling without GVL release overhead.
 - [Enhancement] Add `poller.producer_registered` and `poller.producer_unregistered` instrumentation events for FD mode.
 - [Change] Update to use `max_wait_timeout_ms` parameter in rdkafka `AbstractHandle#wait` for consistency with librdkafka conventions.
@@ -108,10 +114,10 @@
 
 ## 2.8.10 (2025-09-25)
 - [Enhancement] Add `#close` alias for `WaterDrop::ConnectionPool#shutdown` to align with producer API for consistent interface across both individual producers and connection pools.
-- [Enhancement] Add `WaterDrop.monitor` method as the preferred alias for `WaterDrop.instrumentation` to align with per-producer API naming convention. The `instrumentation` method remains available as a deprecated alias for backward compatibility.
+- [Enhancement] Add `WaterDrop.monitor` as the preferred alias for `WaterDrop.instrumentation`, matching the per-producer API. `instrumentation` remains as a deprecated alias.
 
 ## 2.8.9 (2025-09-23)
-- [Enhancement] Add connection pool lifecycle events to global instrumentation for improved observability. Events include `connection_pool.created`, `connection_pool.setup`, `connection_pool.shutdown`, `connection_pool.reload`, and `connection_pool.reloaded`.
+- [Enhancement] Add connection pool lifecycle events to global instrumentation: `connection_pool.created`, `connection_pool.setup`, `connection_pool.shutdown`, `connection_pool.reload` and `connection_pool.reloaded`.
 - [Enhancement] Add default connection pool transactional direct API.
 
 ## 2.8.8 (2025-09-23)

data/Gemfile

@@ -6,10 +6,6 @@ gemspec
 
 gem "zeitwerk", "~> 2.8.0"
 
-group :development do
-  gem "byebug"
-end
-
 group :test do
   gem "connection_pool"
   gem "ostruct"

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    waterdrop (2.10.7)
+    waterdrop (2.10.8)
       karafka-core (>= 2.5.12, < 3.0.0)
       karafka-rdkafka (>= 0.31.0)
       zeitwerk (~> 2.3)
@@ -9,12 +9,9 @@ PATH
 GEM
   remote: https://rubygems.org/
   specs:
-    byebug (13.0.0)
-      reline (>= 0.6.0)
     connection_pool (3.0.2)
     drb (2.2.3)
     ffi (1.17.4)
-    io-console (0.9.4)
     json (3.0.2)
     karafka-core (2.6.4)
       karafka-rdkafka (>= 0.20.0)
@@ -35,8 +32,6 @@ GEM
     ostruct (0.6.3)
     prism (1.9.0)
     rake (13.4.2)
-    reline (0.7.0)
-      io-console (~> 0.5)
     ruby2_keywords (0.0.5)
     simplecov (1.3.2)
     warning (1.6.0)
@@ -46,7 +41,6 @@ PLATFORMS
   ruby
 
 DEPENDENCIES
-  byebug
   connection_pool
   minitest
   mocha
@@ -57,11 +51,9 @@ DEPENDENCIES
   zeitwerk (~> 2.8.0)
 
 CHECKSUMS
-  byebug (13.0.0) sha256=d2263efe751941ca520fa29744b71972d39cbc41839496706f5d9b22e92ae05d
   connection_pool (3.0.2) sha256=33fff5ba71a12d2aa26cb72b1db8bba2a1a01823559fb01d29eb74c286e62e0a
   drb (2.2.3) sha256=0b00d6fdb50995fe4a45dea13663493c841112e4068656854646f418fda13373
   ffi (1.17.4) sha256=bcd1642e06f0d16fc9e09ac6d49c3a7298b9789bcb58127302f934e437d60acf
-  io-console (0.9.4) sha256=a403304f32928ee85e56d457b4612ca43f2703b561a3258844992bd24085fbb1
   json (3.0.2) sha256=8e6d7e7b11384c21230430cef90b71f14849a34a1f4452796670f7c981bd19df
   karafka-core (2.6.4) sha256=e08166ca7ef12fadf0500f2a3aba63b5ba918a98533c3feaac9f43c90f159363
   karafka-rdkafka (0.31.0) sha256=7d21274122d66ee8d08b8e56a0d06084933deb2306cf2a6e4bd0d073c14e7f9f
@@ -72,11 +64,10 @@ CHECKSUMS
   ostruct (0.6.3) sha256=95a2ed4a4bd1d190784e666b47b2d3f078e4a9efda2fccf18f84ddc6538ed912
   prism (1.9.0) sha256=7b530c6a9f92c24300014919c9dcbc055bf4cdf51ec30aed099b06cd6674ef85
   rake (13.4.2) sha256=cb825b2bd5f1f8e91ca37bddb4b9aaf345551b4731da62949be002fa89283701
-  reline (0.7.0) sha256=5b012d8e55dbf9d450f12bde2cf7d15ff546ae80b3f8f3b30e570d431815583d
   ruby2_keywords (0.0.5) sha256=ffd13740c573b7301cf7a2e61fc857b2a8e3d3aff32545d6f8300d8bae10e3ef
   simplecov (1.3.2) sha256=2a22ec1759f1b8db60881a2e038758956fc69ef93ed669be0bde0678de6d0e85
   warning (1.6.0) sha256=a49cdfae19fb77d19afff2efbe45f8ab759e9cd25b4e4ce2c79dbaf46bdb6c9e
-  waterdrop (2.10.7)
+  waterdrop (2.10.8)
   zeitwerk (2.8.3) sha256=2c85125a8467ce069e20123d1e709a08955c9d29c118c25b46b7b7fafdbb92e5
 
 BUNDLED WITH

data/lib/waterdrop/config.rb

@@ -130,13 +130,18 @@ module WaterDrop
     #   WARNING: Modifying this setting can cause infinite reload loops if not properly understood.
     #   The default includes :fenced errors because:
     #   - Fencing occurs when another producer with the same transactional.id takes over
-    #   - This is an unrecoverable state - reloading won't help as the other producer is active
+    #   - Fencing also occurs when the broker aborts a transaction that exceeded
+    #     `transaction.timeout.ms`. Both cases raise the same :fenced error, so WaterDrop cannot
+    #     tell them apart
+    #   - After a takeover this is an unrecoverable state - reloading won't help as the other
+    #     producer is active
     #   - Attempting to reload on fenced errors creates: produce -> fenced -> reload -> produce ->
     #     fenced -> reload (infinite loop)
     #
-    #   Only remove :fenced from this list if you have explicit logic to handle producer fencing
-    #   in your application (e.g., coordinated transactional.id assignment, manual intervention).
-    #   In most cases, you should keep the default value.
+    #   Only remove :fenced from this list if each transactional.id is used by one producer at a
+    #   time (e.g., coordinated transactional.id assignment). Otherwise a reload fences the newer
+    #   producer and the two can keep fencing each other for as long as both run. In most cases,
+    #   you should keep the default value.
     setting :non_reloadable_errors, default: %i[fenced]
     # option [Integer] Idle disconnect timeout in milliseconds. When set to 0, idle disconnection
     #   is disabled. When set to a positive value, WaterDrop will automatically disconnect

data/lib/waterdrop/polling/queue_pipe.rb

@@ -29,8 +29,8 @@ module WaterDrop
       # Thread-safe and non-blocking; silently ignores errors
       def signal
         @writer.write_nonblock("W", exception: false)
-      rescue IOError, Errno::EBADF
-        # Pipe closed
+      rescue IOError, Errno::EBADF, Errno::EPIPE
+        # Pipe closed. EPIPE: the reader is already closed, so there is no one left to wake
       end
 
       # Drains all pending bytes from the pipe

data/lib/waterdrop/producer/transactions.rb

@@ -65,7 +65,12 @@ module WaterDrop
 
           transactional_instrument(:finished) do
             with_transactional_error_handling(:begin) do
-              transactional_instrument(:started) { client.begin_transaction }
+              transactional_instrument(:started) do
+                client.begin_transaction
+                # Set before the `started` listeners run, so an error raised by one of them still
+                # aborts the transaction that has begun
+                @transaction_open = true
+              end
             end
 
             result = nil
@@ -100,6 +105,15 @@ module WaterDrop
           # We need to handle any interrupt including critical in order not to have the transaction
           # running. This will also handle things like `IRB::Abort`
           rescue Exception => e
+            # Neither a transaction that never began nor one the error handling already aborted
+            # (abortable error) has anything left to abort: aborting again would fail with an
+            # invalid state error that replaces the original one
+            unless @transaction_open
+              transactional_reload_client_if_needed(e)
+
+              raise
+            end
+
             # This code is a bit tricky. We have an error and when it happens we try to rollback
             # the transaction. However we may end up in a state where transaction aborting itself
             # produces error. In such case we also want to handle it as fatal and reload client.
@@ -124,6 +138,8 @@ module WaterDrop
 
             raise unless e.is_a?(WaterDrop::Errors::AbortTransaction)
           ensure
+            @transaction_open = false
+
             # The first delivery handle is scoped to the transaction that produced it and is only
             # ever read by the abort path (which runs in the rescue above, before this). Clearing it
             # here - on commit, on abort and on a re-raise alike - keeps it from outliving its
@@ -273,6 +289,8 @@ module WaterDrop
           with_transactional_error_handling(:abort, allow_abortable: false) do
             transactional_instrument(:aborted) { client.abort_transaction }
           end
+
+          @transaction_open = false
         end
 
         raise

data/lib/waterdrop/producer.rb

@@ -66,6 +66,7 @@ module WaterDrop
       @idempotent_fatal_error_attempts = 0
       @transaction_fatal_error_attempts = 0
       @transaction_first_handle = nil
+      @transaction_open = false
 
       @status = Status.new
       @messages = []

data/lib/waterdrop/version.rb

@@ -2,5 +2,5 @@
 
 module WaterDrop
   # Current WaterDrop version
-  VERSION = "2.10.7"
+  VERSION = "2.10.8"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: waterdrop
 version: !ruby/object:Gem::Version
-  version: 2.10.7
+  version: 2.10.8
 platform: ruby
 authors:
 - Maciej Mensfeld