waterdrop 2.10.3 → 2.10.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ec39a827b8ea9216ef39589a8ce2c8261ce29b5ba1386ddb513c2ca393b5ade1
-  data.tar.gz: d37a7617f72d101f75c5d02f4c77627757ad5254322ecbf8bf1c30b0b502fb98
+  metadata.gz: d61c97700e2882107b4e3309176c22fc85d06df909e3f11f0779bb33cab0ec96
+  data.tar.gz: 1b4c0e3c7ac54d6b86905544396865934ee783ba730dc0ccc6b3bd6b954a7ce8
 SHA512:
-  metadata.gz: e65469f5af00b6976f8d6bcc7efecbb8bc7ce3e84418928d6364197c2b84c600730bb36cfa0ffd2333ad7a726367406aea69fa56d9ecc796b1a31a8b5cb11f94
-  data.tar.gz: 654c121690450afdbb356527cbfb142bacc0b7eddbd3dd0c69017d5564cbdeb0712647cd8b13c27180a7cc94a23cca4049d748935e0ca6ccef068f5f55c079d7
+  metadata.gz: 9d509ae8aafda829f217357e6d8447905910137a4af850cd8d4bac7a35bb3af703432339c05fa45117dd8acff2a481718f18a94dd7e7ea0ba0259a86b6ae8d42
+  data.tar.gz: 4a8c01eb9aa11c49eb810888a65a520731298ea500bdcaa2c0b78bc163ffc346cc3975f8cad06ea7e50c9c9d21357457eaf745269b35e598a9eccebaf611e7a4

data/.gitignore

@@ -19,6 +19,8 @@ db/*.sqlite3
 *.rbc
 *.sassc
 .redcar/
+.codex
+.claude
 .capistrano/
 .sass-cache
 /config/god/sidekiq.rb

data/.ruby-version

@@ -1 +1 @@
-4.0.5
+4.0.7

data/.yard-lint.yml

@@ -61,6 +61,34 @@ Documentation/BlankLineBeforeDefinition:
     SingleBlankLine: true
     OrphanedDocs: true
 
+Documentation/DuplicateNamespaceComment:
+  Description: Detects namespaces documented with a YARD comment in more than one file.
+  Enabled: true
+  Severity: error
+
+Documentation/UnderfilledLines:
+  Description: Detects documentation prose that wraps too early and wastes horizontal space.
+  Enabled: true
+  Severity: error
+  # Aligned with RuboCop's Layout/LineLength (Max: 100) so documentation prose
+  # wraps to the same width as code.
+  MaxLength: 100
+
+Documentation/LineLength:
+  Description: Detects documentation lines that exceed the maximum length.
+  Enabled: true
+  Severity: error
+  # Aligned with RuboCop's Layout/LineLength (Max: 100).
+  MaxLength: 100
+
+Documentation/TextSubstitution:
+  Description: Detects em/en-dashes in documentation and replaces them with hyphens.
+  Enabled: true
+  Severity: error
+  Substitutions:
+    "—": "-"    # em-dash (U+2014)
+    "–": "-"    # en-dash (U+2013)
+
 # Tags validators
 Tags/Order:
   Description: Enforces consistent ordering of YARD tags.
@@ -141,7 +169,7 @@ Tags/OptionTags:
 Tags/ExampleSyntax:
   Description: Validates Ruby syntax in @example tags.
   Enabled: true
-  Severity: warning
+  Severity: error
 
 Tags/RedundantParamDescription:
   Description: Detects meaningless parameter descriptions that add no value.

data/CHANGELOG.md

@@ -1,43 +1,54 @@
 # WaterDrop changelog
 
+## 2.10.5 (2026-09-23)
+- [Fix] Stop `#flush` from running middleware twice on messages re-buffered after a failed flush (regression of #474 reintroduced by the failure/requeue path), which double-encrypted/compressed/serialized payloads and duplicated headers on retry.
+- [Fix] Re-check the producer liveness under `@buffer_mutex` in `#buffer`/`#buffer_many`, so a producer closed between the liveness check and the append raises `ProducerClosedError` instead of accepting the message into a closed producer's buffer and losing it silently.
+- [Fix] Restore both buffers when `#flush` fails before anything is dispatched (a middleware step raising, for example) instead of discarding the whole batch. Only `ProduceManyError` and `MessageInvalidError` were covered before.
+
+## 2.10.4 (2026-08-26)
+- [Fix] Avoid a `FrozenError` on `Producer#close` when the producer is configured with a frozen string id (for example a frozen string literal, `config.id = "rspec"`).
+- [Fix] Emit `connection_pool.shutdown` only once when shutting down the global connection pool.
+- [Fix] Retry a produce that races a concurrent idempotent fatal-error reload instead of leaking a raw `Rdkafka::ClosedProducerError`/`Rdkafka::ClosedInnerError`.
+- [Maintenance] Stop the `#partition_count when topic does not exist` spec from flaking on slow CI runners by waiting for authoritative broker metadata before asserting the count.
+
 ## 2.10.3 (2026-07-15)
-- [Feature] Add `wait_timeout_on_transaction_abort` (default `0`, disabled) - an opt-in mitigation for [librdkafka#4849](https://github.com/confluentinc/librdkafka/issues/4849). librdkafka only marks a transaction as ongoing at the coordinator once the `AddPartitionsToTxn` **response** arrives, but it fires `EndTxn` as soon as that request has merely been **sent**. Aborting while the first produce is still in flight can therefore reach a coordinator that does not yet consider the transaction started, failing the abort with a fatal `INVALID_TXN_STATE` that poisons the client and forces a full reload. When set to a positive value, we wait for the first delivery of the transaction to be acknowledged before aborting, for at most that long (in ms) - it is a ceiling rather than a fixed delay, so the wait ends as soon as the delivery is acknowledged. A single ack proves its partition completed registration, which is enough for the coordinator to accept `EndTxn` regardless of how many partitions the transaction spans. It is off by default because it changes abort semantics: the awaited message is really written to the log (aborted, hence invisible to `read_committed` consumers) instead of being purged, so its delivery handle reports a real offset rather than a `Purged in queue` error and no `message.purged` is emitted for it. The wait is bounded and best-effort - if it expires we abort exactly as before, and the fatal remains recoverable through `reload_on_transaction_fatal_error`.
+- [Feature] Add `wait_timeout_on_transaction_abort` (default `0`, disabled) - an opt-in mitigation for [librdkafka#4849](https://github.com/confluentinc/librdkafka/issues/4849) that waits (up to the given ms) for the first delivery to be acknowledged before aborting, avoiding a fatal `INVALID_TXN_STATE`. Off by default because it changes abort semantics: the awaited message is written to the log (aborted) instead of being purged.
 
 ## 2.10.2 (2026-06-15)
-- [Feature] Expose `Producer#current_variant` as a public method. It returns the variant active for the current dispatch on the current fiber - the custom variant while inside a `#with`/`#variant`-wrapped call, otherwise the producer's default variant - so middleware and instrumentation listeners running synchronously within a dispatch can read the effective per-dispatch settings (`topic_config`, `max_wait_timeout`, `default?`). The lookup is fiber-local and dispatch-scoped: outside a variant-wrapped call (or from an asynchronous delivery callback) it returns the default variant.
-- [Enhancement] Stop allocating one interpolated string per message in `LoggerListener` batch produce handlers. The quoted topic strings were only ever counted (quoting is a 1:1 mapping), never displayed, so counting the raw topic values yields the identical number with zero string allocations - relevant for large `produce_many_*` batches with the default logger listener attached.
+- [Feature] Expose `Producer#current_variant` as a public method returning the variant active for the current dispatch on the current fiber, so middleware and instrumentation listeners can read the effective per-dispatch settings (`topic_config`, `max_wait_timeout`, `default?`).
+- [Enhancement] Stop allocating one interpolated string per message in `LoggerListener` batch produce handlers.
 - [Enhancement] Use `Array#concat` in `Producer#buffer_many` instead of appending messages one by one.
-- [Enhancement] Skip building the `message.acknowledged` instrumentation payload in the delivery callback when nothing is subscribed to that event. The notifications bus already short-circuits on empty listeners, but only after the payload hash was allocated - once per delivered message on the polling thread. Mirrors the listener guard already used by the statistics callback. Late subscribers keep working as the check happens on each emission.
-- [Enhancement] Resolve the fiber-local variant once per `#produce` call and once per `#produce_many_sync` wait phase instead of re-resolving it for every usage and for every waited delivery handle. For a 1,000-message sync batch this removes ~2,000 redundant fiber-local lookups.
-- [Enhancement] Do not allocate the fiber-local variants hash on the `Producer#current_variant` read path. Previously every fiber that produced messages got a Hash pinned to it for the fiber's lifetime (per producer use), even when variants were never used - wasteful under fiber-per-request servers (Falcon, async). The hash is now only created by variant wrapper methods that actually need to write to it.
-- [Enhancement] Cache the variant validation contract in a constant instead of instantiating a new `Contracts::Variant` on every `Producer#with` / `Producer#variant` call (mirrors the existing `Transactions::CONTRACT` pattern).
-- [Enhancement] Cache the tombstone validation contract in a constant instead of instantiating a new `Contracts::Tombstone` per tombstone message, removing per-message allocations in the `tombstone_*` APIs (mirrors the existing `Transactions::CONTRACT` pattern).
-- [Enhancement] Replace explicit `Warning[:performance]` opt-in with a dynamic approach using `Warning.categories` (available since Ruby 3.4) to automatically enable all stable opt-in warning categories in the test suite, including `:strict_unused_block` introduced in Ruby 4.0.
-- [Fix] Prevent a deadlock between a transactional single-message dispatch and `#close`. A single `produce_sync`/`produce_async` on a transactional producer incremented the operations counter (which `#close` drains while holding `@transaction_mutex`) before acquiring `@transaction_mutex` for its per-message transaction - an inverted lock order. A dispatch that had counted itself but not yet taken `@transaction_mutex` could deadlock a concurrent `#close` permanently (the close wait loop has no timeout). Transactional dispatches now take `@transaction_mutex` before the operation is counted, matching `#close`'s lock order (`@transaction_mutex` -> `@operating_mutex` -> operations counter).
-- [Fix] Prevent a deadlock (`ThreadError: deadlock; recursive locking`) when closing an idempotent producer (with `reload_on_idempotent_fatal_error` enabled) that has buffered messages whose final flush surfaces a fatal librdkafka error. `#close` performs the final flush while already holding `@operating_mutex`, and the idempotent fatal-error reload tried to re-acquire that same mutex, leaving the producer stuck in `:closing` with the native client leaked. The idempotent reload is now skipped on the closing path, and the final buffer flush is best-effort so client teardown always completes.
-- [Fix] Make concurrent idempotent fatal-error reload thread-safe. When several threads shared an idempotent producer (with `reload_on_idempotent_fatal_error` enabled), a single fatal librdkafka condition failed all their in-flight produces at once and each entered the reload path; the second reload ran `reload!` after the first had already reset `@client` to `nil`, raising `NoMethodError`. The idempotent reload now bails out if another thread already reloaded (mirroring the transactional path's `return if @status.configured?` guard). Additionally, `Status#active?` now classifies the lifecycle from a single atomic read and `Producer#ensure_active!` branches on one snapshot, so a concurrent `configured -> connected` transition during a reload can no longer make `ensure_active!` raise `StatusInvalidError` for a valid, active producer.
-- [Fix] Stop `#flush_async` / `#flush_sync` from silently dropping valid buffered messages when the dispatch fails. `#flush` removes the batch from the internal buffer before dispatching it, and a failure (a single invalid message failing validation before anything is sent, or a mid-batch inline error such as queue full) previously discarded the entire taken batch - the removed messages were never restored. A failed flush now re-buffers the messages that never reached librdkafka (the whole batch on validation failure or on a transactional rollback, the unsent remainder otherwise) so they can be retried instead of being lost.
-- [Fix] Make `Producer#close` fork-safe so the GC finalizer inherited by a forked child can no longer close the parent's client. `#client` registers an `ObjectSpace` finalizer that calls `#close`; that finalizer is inherited across `fork`, and a child that inherited a used producer, never touched it, and exited normally would run `#close` in the child - flushing and closing (with the real rdkafka client, `rd_kafka_destroy` on a fork-inherited handle, i.e. undefined behavior) a client owned by the parent. `#close` now detects when it runs in a process other than the one that built the client, drops the inherited references and finalizer, and returns without touching the native client (matching the existing fork guard on the `#client` path).
-- [Fix] Guard the internal buffer appends in `Producer#buffer` and `Producer#buffer_many` with `@buffer_mutex`. The appends mutated the shared `@messages` buffer without the lock that `flush`/`purge`/`close` hold while swapping it for a fresh array, so a concurrent swap landing between reading `@messages` and appending could drop the message into an orphaned array that is never dispatched - silently losing buffered messages in the documented "buffer in one thread, flush in another" pattern.
-- [Fix] Stop a nested same-producer variant call from clobbering the outer variant inside a variant `transaction` block. `transaction` is the only variant-wrapped method that yields user code, so a variant call nested inside it (another `variant.produce_*`, or a raw producer dispatch in the same scope) used to delete the shared `Fiber.current.waterdrop_clients` entry on return, making the rest of the block silently fall back to the default variant and dispatch with default `topic_config` (timeouts, compression, partitioner) instead of the altered one. The wrapper now saves and restores the previous entry instead of unconditionally deleting it (still deleting when there was none, so the fiber-local hash does not accumulate stale keys).
-- [Fix] Stop `ConnectionPool#shutdown` and `#reload` from silently dropping in-flight messages. Both closed every pooled producer with `close!` (force), which flushes for the max wait timeout and then purges whatever has not drained - so on a slow or unreachable broker, queued `produce_async` messages were cancelled and lost with no delivery report. They now close producers gracefully by default (`#reload` always; `#shutdown` unless called with the new `force: true`), letting messages flush instead of being purged. Pass `pool.shutdown(force: true)` to keep the old force-and-purge behavior.
-- [Fix] Close a race in the FD poller where a producer registered while the last one was being torn down could be left permanently unpolled (sync produces hang until timeout, async deliveries are never acknowledged). The poller thread decided to exit (last producer unregistered) and cleared its thread reference in two separate, unsynchronized steps, so a `register` landing in that gap saw the still-alive exiting thread, skipped starting a fresh one, and then had its producer's state closed by the exiting thread's cleanup. The thread now decides to stop and clears its reference in a single mutex section, so a racing `register` either keeps it running or starts a fresh thread; and the exit cleanup runs only on an abnormal exit, since a normal exit always leaves an empty registry and so can never close a producer registered in the gap.
+- [Enhancement] Skip building the `message.acknowledged` instrumentation payload in the delivery callback when nothing is subscribed to that event.
+- [Enhancement] Resolve the fiber-local variant once per `#produce` call and once per `#produce_many_sync` wait phase instead of re-resolving it for every usage.
+- [Enhancement] Do not allocate the fiber-local variants hash on the `Producer#current_variant` read path when variants are never used.
+- [Enhancement] Cache the variant validation contract in a constant instead of instantiating a new `Contracts::Variant` on every `Producer#with` / `Producer#variant` call.
+- [Enhancement] Cache the tombstone validation contract in a constant instead of instantiating a new `Contracts::Tombstone` per tombstone message.
+- [Enhancement] Replace explicit `Warning[:performance]` opt-in with a dynamic approach using `Warning.categories` to enable all stable opt-in warning categories in the test suite.
+- [Fix] Prevent a deadlock between a transactional single-message dispatch and `#close` caused by an inverted lock order; transactional dispatches now take `@transaction_mutex` before the operation is counted.
+- [Fix] Prevent a deadlock (`ThreadError: deadlock; recursive locking`) when closing an idempotent producer (with `reload_on_idempotent_fatal_error` enabled) whose final buffer flush surfaces a fatal librdkafka error; the idempotent reload is now skipped on the closing path.
+- [Fix] Make concurrent idempotent fatal-error reload thread-safe so a second thread's reload can no longer raise `NoMethodError` after the first reset `@client`.
+- [Fix] Stop `#flush_async` / `#flush_sync` from silently dropping valid buffered messages when the dispatch fails; unsent messages are now re-buffered so they can be retried instead of being lost.
+- [Fix] Make `Producer#close` fork-safe so the GC finalizer inherited by a forked child can no longer flush and close the parent's client.
+- [Fix] Guard the internal buffer appends in `Producer#buffer` and `Producer#buffer_many` with `@buffer_mutex` so a concurrent flush/purge/close swap can no longer drop a message into an orphaned array.
+- [Fix] Stop a nested same-producer variant call from clobbering the outer variant inside a variant `transaction` block; the wrapper now saves and restores the previous fiber-local entry instead of deleting it.
+- [Fix] Stop `ConnectionPool#shutdown` and `#reload` from silently dropping in-flight messages. They now close producers gracefully by default (`#reload` always; `#shutdown` unless called with the new `force: true`); pass `pool.shutdown(force: true)` to keep the old force-and-purge behavior.
+- [Fix] Close a race in the FD poller where a producer registered while the last one was being torn down could be left permanently unpolled; the poller now decides to stop and clears its thread reference in a single mutex section.
 
 ## 2.10.1 (2026-05-25)
 - [Fix] Prevent `Producer#close` from raising `ThreadError: can't be called from trap context` when invoked from a Ruby signal trap context (e.g. Puma's `after_stopped` DSL hook in single mode). `close` now detects this case and delegates to a background thread, joining it so the caller blocks until the producer is fully closed (#866).
 
 ## 2.10.0 (2026-05-07)
-- [Fix] Clean up native rdkafka client, global instrumentation callbacks, and poller registration when `init_transactions` fails during producer client construction. Previously, each failed attempt permanently leaked native threads, pipe file descriptors, and callback registry entries because the started `rd_kafka_t` handle was abandoned without being destroyed.
-- **[Breaking]** Skip emitting librdkafka statistics when nothing is subscribed to `statistics.emitted` at the time the underlying rdkafka client is constructed. When no listener is present at build time, `statistics.interval.ms` is forced to `0` regardless of user configuration and the statistics callback is not registered, saving substantial allocations in the hot path (no JSON parsing, no statistics hash materialization, no decoration work). To use statistics, subscribe a listener to `statistics.emitted` BEFORE the first producer use (before the underlying client is lazily initialized).
-- **[Breaking]** Raise `WaterDrop::Errors::StatisticsNotEnabledError` when attempting to subscribe to `statistics.emitted` (either via block or via a listener that responds to `on_statistics_emitted`) on a monitor where librdkafka statistics have been disabled at client build time. This replaces the "silent nothing" failure mode with an immediate, actionable error that pinpoints the timing mistake.
+- [Fix] Clean up native rdkafka client, global instrumentation callbacks, and poller registration when `init_transactions` fails during producer client construction, so failed attempts no longer leak native threads, file descriptors, and callback registry entries.
+- **[Breaking]** Skip emitting librdkafka statistics when nothing is subscribed to `statistics.emitted` at the time the underlying rdkafka client is constructed (`statistics.interval.ms` is forced to `0`), saving substantial allocations in the hot path. To use statistics, subscribe a listener to `statistics.emitted` BEFORE the first producer use.
+- **[Breaking]** Raise `WaterDrop::Errors::StatisticsNotEnabledError` when attempting to subscribe to `statistics.emitted` on a monitor where librdkafka statistics have been disabled at client build time, replacing the previous "silent nothing" failure mode.
 - [Feature] Add tombstone API (`#tombstone_sync`, `#tombstone_async`, `#tombstone_many_sync`, `#tombstone_many_async`) for producing tombstone records (nil-payload messages) with required key and partition validation. Works with variants.
-- [Fix] Add `ensure_same_process!` to `Poller#unregister` for fork safety. Without this, a child process that inherited a pre-fork producer would deadlock on `producer.close` because `unregister` waited on a latch the dead parent poller thread would never release.
+- [Fix] Add `ensure_same_process!` to `Poller#unregister` for fork safety, so a child that inherited a pre-fork producer no longer deadlocks on `producer.close`.
 
 ## 2.9.0 (2026-04-08)
 - [Fix] Use `delete` in the variant ensure block to avoid leaving stale nil entries in `Fiber.current.waterdrop_clients` and prevent memory leaks in long-running processes (#836).
 - [Fix] Exclude test files, `.github/`, and `log/` directories from gem releases to reduce package size.
 - **[Breaking]** Switch default polling mode from `:thread` to `:fd`. If you experience any issues, you can revert to the previous behavior by setting `config.polling.mode = :thread`. The `:thread` mode will be deprecated in 2.10 and removed in 2.11.
-- **[Breaking]** Statistics decorator now only decorates keys used by the built-in Datadog metrics listener (`tx`, `txretries`, `txerrs`, `rxerrs`) and skips unused subtrees (`topics`, broker window stats). This reduces decoration cost by ~425x on large clusters (10 brokers, 20 topics, 2000 partitions). Users who rely on other `_d` or `_fd` keys in custom instrumentation should provide a custom decorator via `config.statistics_decorator`.
+- **[Breaking]** Statistics decorator now only decorates keys used by the built-in Datadog metrics listener (`tx`, `txretries`, `txerrs`, `rxerrs`) and skips unused subtrees, greatly reducing decoration cost on large clusters. Users who rely on other `_d` or `_fd` keys in custom instrumentation should provide a custom decorator via `config.statistics_decorator`.
 - [Feature] Add `config.statistics_decorator` setting to allow full control over the `StatisticsDecorator` instance used for statistics decoration. Users can provide a custom decorator with different `only_keys` and `excluded_keys` to match their instrumentation needs.
 - [Change] Upscale default timeout values 3x closer to librdkafka defaults to prevent intermediate timeouts during node recovery (`message.timeout.ms`: 50s → 150s, `transaction.timeout.ms`: 55s → 165s, `max_wait_timeout`: 60s → 180s).
 

data/Gemfile.lint.lock

@@ -58,11 +58,11 @@ GEM
     unicode-display_width (3.2.0)
       unicode-emoji (~> 4.1)
     unicode-emoji (4.2.0)
-    yard (0.9.38)
-    yard-lint (1.4.0)
+    yard (0.9.45)
+    yard-lint (1.10.1)
       yard (~> 0.9)
       zeitwerk (~> 2.6)
-    zeitwerk (2.7.4)
+    zeitwerk (2.8.2)
 
 PLATFORMS
   ruby
@@ -79,6 +79,7 @@ DEPENDENCIES
 
 CHECKSUMS
   ast (2.4.3) sha256=954615157c1d6a382bc27d690d973195e79db7f55e9765ac7c481c60bdb4d383
+  bundler (4.0.17) sha256=214e21431b5665dd2f99df8a5511c6b151d7a72e8015c8b38f8b775b61cbb6c1
   json (2.18.0) sha256=b10506aee4183f5cf49e0efc48073d7b75843ce3782c68dbeb763351c08fd505
   language_server-protocol (3.17.0.5) sha256=fd1e39a51a28bf3eec959379985a72e296e9f9acfce46f6a79d31ca8760803cc
   lint_roller (1.1.0) sha256=2c0c845b632a7d172cb849cc90c1bce937a28c5c8ccccb50dfd46a485003cc87
@@ -100,9 +101,9 @@ CHECKSUMS
   standard-performance (1.9.0) sha256=49483d31be448292951d80e5e67cdcb576c2502103c7b40aec6f1b6e9c88e3f2
   unicode-display_width (3.2.0) sha256=0cdd96b5681a5949cdbc2c55e7b420facae74c4aaf9a9815eee1087cb1853c42
   unicode-emoji (4.2.0) sha256=519e69150f75652e40bf736106cfbc8f0f73aa3fb6a65afe62fefa7f80b0f80f
-  yard (0.9.38) sha256=721fb82afb10532aa49860655f6cc2eaa7130889df291b052e1e6b268283010f
-  yard-lint (1.4.0) sha256=7dd88fbb08fd77cb840bea899d58812817b36d92291b5693dd0eeb3af9f91f0f
-  zeitwerk (2.7.4) sha256=2bef90f356bdafe9a6c2bd32bcd804f83a4f9b8bc27f3600fff051eb3edcec8b
+  yard (0.9.45) sha256=52e211493f7cb8a3ebf7e104a25a1e73937a3103092545d34cb88fafebb3dc51
+  yard-lint (1.10.1) sha256=172ea1905304580bd515c093d82831ec4889e725100c49d5bf77e62101f49129
+  zeitwerk (2.8.2) sha256=7212a61311083c604184b1ea2574b9aa05cd14f855a0841c06985cabe9181d12
 
 BUNDLED WITH
-   2.6.9
+  4.0.17

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    waterdrop (2.10.3)
+    waterdrop (2.10.5)
       karafka-core (>= 2.5.12, < 3.0.0)
       karafka-rdkafka (>= 0.24.0)
       zeitwerk (~> 2.3)
@@ -14,12 +14,12 @@ GEM
     connection_pool (3.0.2)
     drb (2.2.3)
     ffi (1.17.4)
-    io-console (0.8.2)
-    json (2.20.0)
-    karafka-core (2.6.2)
+    io-console (0.9.2)
+    json (2.21.2)
+    karafka-core (2.6.3)
       karafka-rdkafka (>= 0.20.0)
       logger (>= 1.6.0)
-    karafka-rdkafka (0.27.2)
+    karafka-rdkafka (0.28.0)
       ffi (~> 1.17.1)
       json (> 2.0)
       logger
@@ -35,12 +35,12 @@ GEM
     ostruct (0.6.3)
     prism (1.9.0)
     rake (13.4.2)
-    reline (0.6.3)
+    reline (0.7.0)
       io-console (~> 0.5)
     ruby2_keywords (0.0.5)
-    simplecov (1.0.0)
+    simplecov (1.3.0)
     warning (1.6.0)
-    zeitwerk (2.8.2)
+    zeitwerk (2.8.3)
 
 PLATFORMS
   ruby
@@ -61,10 +61,10 @@ CHECKSUMS
   connection_pool (3.0.2) sha256=33fff5ba71a12d2aa26cb72b1db8bba2a1a01823559fb01d29eb74c286e62e0a
   drb (2.2.3) sha256=0b00d6fdb50995fe4a45dea13663493c841112e4068656854646f418fda13373
   ffi (1.17.4) sha256=bcd1642e06f0d16fc9e09ac6d49c3a7298b9789bcb58127302f934e437d60acf
-  io-console (0.8.2) sha256=d6e3ae7a7cc7574f4b8893b4fca2162e57a825b223a177b7afa236c5ef9814cc
-  json (2.20.0) sha256=9362bc6e55a952b056abf9167cf053358181c904cb70cd6eee0808ea830fc32b
-  karafka-core (2.6.2) sha256=c2fd7f277201b8ca97b824b364ad76bf776b8f5527bc422dafb71f7ae48d3a13
-  karafka-rdkafka (0.27.2) sha256=3ccce96306642be70bff8168e4e737fc10f2ffae20bc0ff0a43d88dbb7452d31
+  io-console (0.9.2) sha256=efa74f891dd03c0939a931dfc6e74c2813d904763d456ea9762b0525e748db08
+  json (2.21.2) sha256=1f1d3b7cf2b3ba1a69beca0bb6db13d5438b80bff3cd54cdaaa620b9b07c1c6a
+  karafka-core (2.6.3) sha256=83a783895d8f69e5fd4b7e77617f6a43c129bbb905c93e1babb9ff299470ba4a
+  karafka-rdkafka (0.28.0) sha256=fd845140c42341c5892215cac1e8adacda53b0461688f2e16379d82cb33eb006
   logger (1.7.0) sha256=196edec7cc44b66cfb40f9755ce11b392f21f7967696af15d274dde7edff0203
   mini_portile2 (2.8.9) sha256=0cd7c7f824e010c072e33f68bc02d85a00aeb6fce05bb4819c03dfd3c140c289
   minitest (6.0.6) sha256=153ea36d1d987a62942382b61075745042a2b3123b1cd48f4c3675af9cc7d6f1
@@ -72,12 +72,12 @@ CHECKSUMS
   ostruct (0.6.3) sha256=95a2ed4a4bd1d190784e666b47b2d3f078e4a9efda2fccf18f84ddc6538ed912
   prism (1.9.0) sha256=7b530c6a9f92c24300014919c9dcbc055bf4cdf51ec30aed099b06cd6674ef85
   rake (13.4.2) sha256=cb825b2bd5f1f8e91ca37bddb4b9aaf345551b4731da62949be002fa89283701
-  reline (0.6.3) sha256=1198b04973565b36ec0f11542ab3f5cfeeec34823f4e54cebde90968092b1835
+  reline (0.7.0) sha256=5b012d8e55dbf9d450f12bde2cf7d15ff546ae80b3f8f3b30e570d431815583d
   ruby2_keywords (0.0.5) sha256=ffd13740c573b7301cf7a2e61fc857b2a8e3d3aff32545d6f8300d8bae10e3ef
-  simplecov (1.0.0) sha256=bdc50b41fa4a8b3c860da4bf5d61ba4704479c51d08a8f53cbd56ae50778dca8
+  simplecov (1.3.0) sha256=d9886307863c1ead47657dcbed869bfffd82318808b09014d8f0ff77dcfe7492
   warning (1.6.0) sha256=a49cdfae19fb77d19afff2efbe45f8ab759e9cd25b4e4ce2c79dbaf46bdb6c9e
-  waterdrop (2.10.3)
-  zeitwerk (2.8.2) sha256=7212a61311083c604184b1ea2574b9aa05cd14f855a0841c06985cabe9181d12
+  waterdrop (2.10.5)
+  zeitwerk (2.8.3) sha256=2c85125a8467ce069e20123d1e709a08955c9d29c118c25b46b7b7fafdbb92e5
 
 BUNDLED WITH
   4.0.6

data/README.md

@@ -1,7 +1,7 @@
 # WaterDrop
 
-[![Build Status](https://github.com/karafka/waterdrop/workflows/ci/badge.svg)](https://github.com/karafka/waterdrop/actions?query=workflow%3Aci)
-[![Gem Version](https://badge.fury.io/rb/waterdrop.svg)](http://badge.fury.io/rb/waterdrop)
+[![Build Status](https://github.com/karafka/waterdrop/actions/workflows/ci.yml/badge.svg)](https://github.com/karafka/waterdrop/actions/workflows/ci.yml)
+[![Gem Version](https://badge.fury.io/rb/waterdrop.svg)](https://badge.fury.io/rb/waterdrop)
 [![Join the chat at https://slack.karafka.io](https://raw.githubusercontent.com/karafka/misc/master/slack.svg)](https://slack.karafka.io)
 
 WaterDrop is a standalone gem that sends messages to Kafka easily with an extra validation layer. It is a part of the [Karafka](https://github.com/karafka/karafka) ecosystem.

data/bin/verify_kafka_warnings

@@ -4,6 +4,11 @@
 # Only specified warnings are allowed, all others should trigger failure
 
 allowed_patterns=(
+   # Enabling share groups (KIP-932) requires "share" in group.coordinator.rebalance.protocols
+   # on cp-kafka 8.3.0 (Kafka 4.x). That mechanism is deprecated in favour of the share.version
+   # feature flag (Kafka 5.0), but is still functionally required to enable share groups in this
+   # version, so the broker logs this deprecation warning on every startup. Expected here.
+   "group.coordinator.rebalance.protocols. is deprecated"
    "Performing controller activation"
    "registered with feature metadata.version"
    "TOPIC_ALREADY_EXISTS"

data/docker-compose.oauth.yml

@@ -49,6 +49,9 @@ services:
       KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
       KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
       KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+      KAFKA_GROUP_COORDINATOR_REBALANCE_PROTOCOLS: 'classic,consumer,share'
+      KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
       # Required by Confluent image when SASL is enabled
       # Also allow HTTP URLs for JWKS endpoint (Kafka 3.0+ blocks non-HTTPS by default)
       KAFKA_OPTS: "-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf -Dorg.apache.kafka.sasl.oauthbearer.allowed.urls=http://keycloak:8080/realms/kafka/protocol/openid-connect/certs"

data/docker-compose.sasl.yml

@@ -21,6 +21,9 @@ services:
       KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
       KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
       KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+      KAFKA_GROUP_COORDINATOR_REBALANCE_PROTOCOLS: 'classic,consumer,share'
+      KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
       KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
       KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer
       KAFKA_OPTS: ""

data/docker-compose.yml

@@ -21,5 +21,8 @@ services:
       KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
       KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
       KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+      KAFKA_GROUP_COORDINATOR_REBALANCE_PROTOCOLS: 'classic,consumer,share'
+      KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
       KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
       KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer

data/lib/waterdrop/clients/buffered.rb

@@ -80,8 +80,7 @@ module WaterDrop
         @topics[topic]
       end
 
-      # Clears internal buffer
-      # Used in between specs so messages do not leak out
+      # Clears internal buffer. Used in between specs so messages do not leak out
       def reset
         @transaction_level = 0
         @transaction_active = false

data/lib/waterdrop/clients/rdkafka.rb

@@ -51,7 +51,7 @@ module WaterDrop
         # saves a significant number of allocations on the Ruby side (no JSON parsing, no
         # statistics hash materialization, no decorator work). Any listener subscribed after
         # the client has been built will not receive `statistics.emitted` events because
-        # librdkafka never emits them in the first place — to use statistics, subscribe a
+        # librdkafka never emits them in the first place - to use statistics, subscribe a
         # listener BEFORE the first producer use.
         #
         # When statistics end up disabled (either because the user explicitly set the interval
@@ -94,7 +94,7 @@ module WaterDrop
 
         # Registers the global callbacks (statistics, error, oauth refresh) for this producer
         # on the shared `Karafka::Core::Instrumentation` managers. The statistics callback is
-        # only registered when librdkafka is actually going to emit statistics — otherwise it
+        # only registered when librdkafka is actually going to emit statistics - otherwise it
         # would never fire and would only waste memory and a manager slot.
         #
         # @param producer [WaterDrop::Producer]

data/lib/waterdrop/config.rb

@@ -36,7 +36,7 @@ module WaterDrop
       constructor: ->(id) { id || "waterdrop-#{SecureRandom.hex(6)}" }
     )
     # option [Instance] logger that we want to use
-    # @note Due to how rdkafka works, this setting is global for all the producers
+    # Due to how rdkafka works, this setting is global for all the producers
     setting(
       :logger,
       default: false,
@@ -169,7 +169,7 @@ module WaterDrop
     # option [Class] class for usage when creating the underlying client used to dispatch messages
     setting :client_class, default: Clients::Rdkafka
     # rdkafka options
-    # @see https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
+    # See https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md
     setting :kafka, default: {}
     # Middleware chain that can be expanded with useful middleware steps
     setting(

data/lib/waterdrop/connection_pool.rb

@@ -1,6 +1,5 @@
 # frozen_string_literal: true
 
-# WaterDrop library
 module WaterDrop
   # Connection pool wrapper for WaterDrop producers using the proven connection_pool gem.
   #
@@ -119,15 +118,11 @@ module WaterDrop
       def shutdown(force: false)
         return unless @default_pool
 
-        pool = @default_pool
+        # The instrumentation event is emitted by the instance-level #shutdown we delegate to
+        # (with `pool: self`, i.e. this same global pool). We deliberately do not emit it again
+        # here - doing so fired `connection_pool.shutdown` twice for a single global shutdown.
         @default_pool.shutdown(force: force)
         @default_pool = nil
-
-        # Emit global event for pool shutdown
-        WaterDrop.instrumentation.instrument(
-          "connection_pool.shutdown",
-          pool: pool
-        )
       end
 
       # Alias for shutdown to align with producer API

data/lib/waterdrop/instrumentation/monitor.rb

@@ -2,9 +2,8 @@
 
 module WaterDrop
   module Instrumentation
-    # WaterDrop instrumentation monitor that we use to publish events
-    # By default uses our internal notifications bus but can be used with
-    # `ActiveSupport::Notifications` as well
+    # WaterDrop instrumentation monitor that we use to publish events. By default uses our internal
+    # notifications bus but can be used with `ActiveSupport::Notifications` as well
     class Monitor < ::Karafka::Core::Monitoring::Monitor
       # Event name for librdkafka statistics emissions
       STATISTICS_EVENT = "statistics.emitted"
@@ -29,8 +28,8 @@ module WaterDrop
       # Marks this monitor as no longer accepting new subscriptions to `statistics.emitted`.
       # Called by the rdkafka client builder when it decides to leave librdkafka statistics
       # disabled (because no listener was present at build time). Any subsequent attempt to
-      # subscribe to `statistics.emitted` — either via a block or via a listener object that
-      # responds to `on_statistics_emitted` — will raise
+      # subscribe to `statistics.emitted` - either via a block or via a listener object that
+      # responds to `on_statistics_emitted` - will raise
       # `WaterDrop::Errors::StatisticsNotEnabledError` instead of silently doing nothing.
       def freeze_statistics_listeners!
         @statistics_listeners_frozen = true

data/lib/waterdrop/polling/config.rb

@@ -1,8 +1,6 @@
 # frozen_string_literal: true
 
-# WaterDrop main module
 module WaterDrop
-  # Namespace for FD-based polling components
   module Polling
     # Configuration for the global FD poller singleton
     # These settings apply to all producers using FD polling mode

data/lib/waterdrop/polling/poller.rb

@@ -1,8 +1,6 @@
 # frozen_string_literal: true
 
 module WaterDrop
-  # Namespace for FD-based polling components
-  # Contains the global Poller singleton and State class for managing producer polling
   module Polling
     # Global poller singleton that manages a single polling thread for all FD-mode producers
     # This replaces librdkafka's native background polling threads with a single Ruby thread
@@ -272,7 +270,8 @@ module WaterDrop
 
       # Collects all IOs to monitor and builds a mapping from IO to State
       # Uses cached arrays when possible to avoid allocations in the hot path
-      # @return [Array<Array<IO>, Hash{IO => State}, Array<State>>] tuple of ios, io-to-state map, states
+      # @return [Array<Array<IO>, Hash{IO => State}, Array<State>>] tuple of ios, io-to-state map,
+      #   states
       def collect_readable_ios
         # Fast path: return cached result if not dirty (no mutex needed)
         # Safe because @cached_result is frozen and assigned atomically
@@ -345,9 +344,8 @@ module WaterDrop
         end
       end
 
-      # Polls all registered producers
-      # Called when IO.select times out to ensure periodic polling happens
-      # This ensures OAuth token refresh and statistics callbacks fire for all producers
+      # Polls all registered producers. Called when IO.select times out to ensure periodic polling
+      # happens. This ensures OAuth token refresh and statistics callbacks fire for all producers
       def poll_all_producers
         @cached_states.each { |state| poll_producer(state) }
       end

data/lib/waterdrop/polling/state.rb

@@ -26,7 +26,8 @@ module WaterDrop
       # @param client [Rdkafka::Producer] the rdkafka producer client
       # @param monitor [Object] the producer's monitor for error reporting
       # @param max_poll_time [Integer] max time in ms to poll per cycle
-      # @param periodic_poll_interval [Integer] max time in ms before this producer needs periodic poll
+      # @param periodic_poll_interval [Integer] max time in ms before this producer needs periodic
+      #   poll
       # @raise [StandardError] if queue pipe setup fails (FD mode requires this to work)
       def initialize(producer_id, client, monitor, max_poll_time, periodic_poll_interval)
         @producer_id = producer_id
@@ -106,9 +107,8 @@ module WaterDrop
         @last_stale_result = (now - @last_poll_time) >= @periodic_poll_interval
       end
 
-      # Signals the poller to remove this producer
-      # Called from any thread when the producer is being closed
-      # Sets closing flag BEFORE signaling to ensure poller sees it
+      # Signals the poller to remove this producer. Called from any thread when the producer is
+      # being closed. Sets closing flag BEFORE signaling to ensure poller sees it
       def signal_close
         @closing = true
         @queue_pipe.signal
@@ -133,9 +133,8 @@ module WaterDrop
         @close_latch.release!
       end
 
-      # Waits for this state to be closed
-      # Used by unregister to ensure synchronous cleanup before returning
-      # This matches the threaded polling behavior which drains without timeout
+      # Waits for this state to be closed. Used by unregister to ensure synchronous cleanup before
+      # returning. This matches the threaded polling behavior which drains without timeout
       def wait_for_close
         @close_latch.wait
       end

data/lib/waterdrop/polling.rb

@@ -1,6 +1,5 @@
 # frozen_string_literal: true
 
-# WaterDrop main module
 module WaterDrop
   # Namespace for FD-based polling components
   # Contains the global Poller singleton and State class for managing producer polling

data/lib/waterdrop/producer/async.rb

@@ -44,15 +44,18 @@ module WaterDrop
       #
       # @param messages [Array<Hash>] array with messages that comply with the
       #   {Contracts::Message} contract
+      # @param run_middleware [Boolean] should the middleware chain be run on the messages. Set to
+      #   false only when the messages have already been through middleware (for example when
+      #   re-dispatching a re-buffered flush batch) so it is not applied twice.
       #
       # @return [Array<Rdkafka::Producer::DeliveryHandle>] deliveries handles
       #
       # @raise [Rdkafka::RdkafkaError] When adding the messages to rdkafka's queue failed
       # @raise [Errors::MessageInvalidError] When any of the provided messages details are invalid
       #   and the message could not be sent to Kafka
-      def produce_many_async(messages)
+      def produce_many_async(messages, run_middleware: true)
         dispatched = []
-        messages = middleware.run_many(messages)
+        messages = middleware.run_many(messages) if run_middleware
         messages.each { |message| validate_message!(message) }
 
         @monitor.instrument(

data/lib/waterdrop/producer/buffer.rb

@@ -15,12 +15,22 @@ module WaterDrop
         # The append runs under @buffer_mutex because flush/purge/close swap @messages for a fresh
         # array under the same lock. Without it, a concurrent swap between reading @messages and
         # appending would land the message in the orphaned old array and silently lose it.
+        #
+        # The liveness check is repeated under the same lock because `close` marks the producer
+        # `:closing` before its final flush acquires @buffer_mutex. Checking only above the lock
+        # would let a close complete in between, stranding the message in a closed producer's
+        # buffer while this call returned success.
         @monitor.instrument(
           "message.buffered",
           producer_id: id,
           message: message,
           buffer: @messages
-        ) { @buffer_mutex.synchronize { @messages << message } }
+        ) do
+          @buffer_mutex.synchronize do
+            ensure_active!
+            @messages << message
+          end
+        end
       end
 
       # Adds given messages into the internal producer buffer without flushing them to Kafka
@@ -34,14 +44,20 @@ module WaterDrop
 
         # The concat runs under @buffer_mutex for the same reason as #buffer: flush/purge/close swap
         # @messages under the lock, so an unguarded concat could append into an array that has just
-        # been captured for dispatch (or discarded), silently losing the messages.
+        # been captured for dispatch (or discarded), silently losing the messages. The liveness
+        # check is repeated under the lock for the same reason as in #buffer: a close that
+        # completes between the check above and the lock would otherwise strand the batch.
         @monitor.instrument(
           "messages.buffered",
           producer_id: id,
           messages: messages,
           buffer: @messages
         ) do
-          @buffer_mutex.synchronize { @messages.concat(messages) }
+          @buffer_mutex.synchronize do
+            ensure_active!
+            @messages.concat(messages)
+          end
+
           messages
         end
       end
@@ -53,7 +69,7 @@ module WaterDrop
         @monitor.instrument(
           "buffer.flushed_async",
           producer_id: id,
-          messages: @messages
+          messages: messages
         ) { flush(false) }
       end
 
@@ -64,7 +80,7 @@ module WaterDrop
         @monitor.instrument(
           "buffer.flushed_sync",
           producer_id: id,
-          messages: @messages
+          messages: messages
         ) { flush(true) }
       end
 
@@ -78,17 +94,47 @@ module WaterDrop
       # @note We use this method underneath to provide a different instrumentation for sync and
       #   async flushing within the public API
       def flush(sync)
-        data_for_dispatch = nil
+        fresh = nil
+        requeued = nil
+        data_for_dispatch = []
 
         @buffer_mutex.synchronize do
-          data_for_dispatch = @messages
+          requeued = @requeued
+          @requeued = []
+          fresh = @messages
           @messages = []
         end
 
+        # Middleware runs exactly once per message: on the fresh messages here, never again on the
+        # already-transformed ones coming back from a failed flush, which a second pass would
+        # corrupt.
+        #
+        # A middleware step raising here (a serializer, encryptor or schema lookup failing) would
+        # otherwise destroy both buffers, since they were emptied above and nothing has been
+        # dispatched yet to recover them from. Messages are run one by one so that on failure the
+        # ones whose chain completed are kept as returned and are not transformed again, while the
+        # failing one and those after it stay on the middleware path. The failing one is not
+        # requeued because that would dispatch it without the rest of its chain.
+        processed = []
+
+        begin
+          fresh.each { |message| processed << middleware.run(message) }
+        rescue
+          restore_undispatched(fresh.drop(processed.size), requeued.concat(processed))
+
+          raise
+        end
+
+        data_for_dispatch = requeued.concat(processed)
+
         # Do nothing if nothing to flush
         return data_for_dispatch if data_for_dispatch.empty?
 
-        sync ? produce_many_sync(data_for_dispatch) : produce_many_async(data_for_dispatch)
+        if sync
+          produce_many_sync(data_for_dispatch, run_middleware: false)
+        else
+          produce_many_async(data_for_dispatch, run_middleware: false)
+        end
       rescue Errors::ProduceManyError => e
         # A dispatch failed partway through the batch. Re-buffer the messages that never reached
         # librdkafka so a partial failure does not silently drop valid buffered messages. For a
@@ -106,15 +152,27 @@ module WaterDrop
         raise
       end
 
-      # Puts not-yet-dispatched messages back at the front of the buffer (preserving their original
-      # order relative to each other and to anything buffered concurrently), so a failed flush does
-      # not lose them.
+      # Puts both sets back after a failure that happened before anything reached librdkafka. They
+      # go to different buffers because middleware has run on one of them and not the other, which
+      # is what keeps it applied exactly once per message.
+      #
+      # @param fresh [Array<Hash>] messages whose middleware chain has not completed
+      # @param requeued [Array<Hash>] already middleware-processed messages
+      def restore_undispatched(fresh, requeued)
+        @buffer_mutex.synchronize do
+          @messages.unshift(*fresh)
+          @requeued.unshift(*requeued)
+        end
+      end
+
+      # Puts not-yet-dispatched messages back at the front of the retry buffer (preserving their
+      # original order relative to each other), so a failed flush does not lose them.
       #
-      # @param messages [Array<Hash>] messages to restore to the buffer
+      # @param messages [Array<Hash>] already middleware-processed messages to restore
       def requeue_unflushed(messages)
         return if messages.empty?
 
-        @buffer_mutex.synchronize { @messages.unshift(*messages) }
+        @buffer_mutex.synchronize { @requeued.unshift(*messages) }
       end
     end
   end

data/lib/waterdrop/producer/sync.rb

@@ -48,6 +48,9 @@ module WaterDrop
       #
       # @param messages [Array<Hash>] array with messages that comply with the
       #   {Contracts::Message} contract
+      # @param run_middleware [Boolean] should the middleware chain be run on the messages. Set to
+      #   false only when the messages have already been through middleware (for example when
+      #   re-dispatching a re-buffered flush batch) so it is not applied twice.
       #
       # @return [Array<Rdkafka::Producer::DeliveryHandle>] delivery handles for messages that were
       #   sent (can be used to verify offset, partition, etc via `#create_result`)
@@ -63,8 +66,8 @@ module WaterDrop
       #   errors), returning handles guarantees the same type in the `dispatched` collection,
       #   allowing uniform error handling. Each handle is in its final state after this method
       #   returns, so you can call `handle.create_result` to obtain the delivery report if needed.
-      def produce_many_sync(messages)
-        messages = middleware.run_many(messages)
+      def produce_many_sync(messages, run_middleware: true)
+        messages = middleware.run_many(messages) if run_middleware
         messages.each { |message| validate_message!(message) }
 
         dispatched = []

data/lib/waterdrop/producer/tombstone.rb

@@ -16,7 +16,7 @@ module WaterDrop
       # Produces a tombstone message to Kafka and waits for it to be delivered
       #
       # @param message [Hash] hash with at least `:topic`, `:key`, and `:partition` keys.
-      #   `:payload` is not accepted — it will be silently removed if present.
+      #   `:payload` is not accepted - it will be silently removed if present.
       #
       # @return [Rdkafka::Producer::DeliveryReport] delivery report
       #
@@ -28,7 +28,7 @@ module WaterDrop
       # Produces a tombstone message to Kafka and does not wait for results
       #
       # @param message [Hash] hash with at least `:topic`, `:key`, and `:partition` keys.
-      #   `:payload` is not accepted — it will be silently removed if present.
+      #   `:payload` is not accepted - it will be silently removed if present.
       #
       # @return [Rdkafka::Producer::DeliveryHandle] delivery handle
       #

data/lib/waterdrop/producer/transactions.rb

@@ -282,10 +282,10 @@ module WaterDrop
       #
       # librdkafka only marks a transaction as ongoing at the coordinator once the
       # `AddPartitionsToTxn` **response** comes back, but it fires `EndTxn` as soon as that request
-      # has merely been **sent** (it gates on `txn_req_cnt`, bumped on send). Aborting with the first
-      # produce still in flight can therefore reach a coordinator that does not yet consider the
-      # transaction started, which fails the abort with a fatal `INVALID_TXN_STATE`.
-      # See https://github.com/confluentinc/librdkafka/issues/4849
+      # has merely been **sent** (it gates on `txn_req_cnt`, bumped on send). Aborting with the
+      # first produce still in flight can therefore reach a coordinator that does not yet consider
+      # the transaction started, which fails the abort with a fatal `INVALID_TXN_STATE`. See
+      # https://github.com/confluentinc/librdkafka/issues/4849
       #
       # A delivered message proves its partition completed registration, and that alone puts the
       # transaction in an `ongoing` state at the coordinator - so a single acknowledged delivery is

data/lib/waterdrop/producer/variant.rb

@@ -122,7 +122,7 @@ module WaterDrop
           topic_config: topic_config,
           # We pass this to validation, to make sure no-one alters the `acks` value when operating
           # in the transactional mode as it causes librdkafka to crash ruby
-          # @see https://github.com/confluentinc/librdkafka/issues/4710
+          # See https://github.com/confluentinc/librdkafka/issues/4710
           transactional: @producer.transactional?,
           # We pass this for a similar reason as above
           idempotent: @producer.idempotent?

data/lib/waterdrop/producer.rb

@@ -39,8 +39,6 @@ module WaterDrop
     attr_reader :id
     # @return [Status] producer status object
     attr_reader :status
-    # @return [Array] internal messages buffer
-    attr_reader :messages
     # @return [Object] monitor we want to use
     attr_reader :monitor
     # @return [Object] dry-configurable config object
@@ -71,6 +69,9 @@ module WaterDrop
 
       @status = Status.new
       @messages = []
+      # Messages re-buffered after a failed flush. Kept apart from @messages because they have
+      # already passed through middleware and must not be transformed again on the next flush.
+      @requeued = []
 
       # Instrument producer creation for global listeners
       class_monitor.instrument(
@@ -93,7 +94,9 @@ module WaterDrop
         .setup(...)
         .config
 
-      @id = @config.id
+      # dup so a frozen id (e.g. a frozen string literal) isn't used as the finalizer target, since
+      # ObjectSpace.(un)define_finalizer mutates the object and would raise FrozenError on #close
+      @id = @config.id.to_s.dup
       @monitor = @config.monitor
       @contract = Contracts::Message.new(max_payload_size: @config.max_payload_size)
       @default_variant = Variant.new(self, default: true)
@@ -175,6 +178,12 @@ module WaterDrop
       @client
     end
 
+    # @return [Array] internal messages buffer awaiting a flush. Includes both freshly buffered
+    #   messages and any messages re-buffered after a failed flush that are waiting to be retried.
+    def messages
+      @requeued + @messages
+    end
+
     # Returns the number of messages in the librdkafka producer queue.
     #
     # This count includes:
@@ -230,6 +239,7 @@ module WaterDrop
       @monitor.instrument("buffer.purged", producer_id: id) do
         @buffer_mutex.synchronize do
           @messages = []
+          @requeued = []
         end
 
         # We should not purge if there is no client initialized
@@ -319,6 +329,7 @@ module WaterDrop
             return false unless @client
             return false unless @status.connected?
             return false unless @messages.empty?
+            return false unless @requeued.empty?
             return false unless @operations_in_progress.value.zero?
 
             @status.disconnecting!
@@ -352,6 +363,7 @@ module WaterDrop
       return false unless @client
       return false unless @status.connected?
       return false unless @messages.empty?
+      return false unless @requeued.empty?
       return false if @transaction_mutex.locked?
       return false if @operating_mutex.locked?
 
@@ -501,7 +513,7 @@ module WaterDrop
       # Try to get buffer info safely
       if @buffer_mutex.try_lock
         begin
-          parts << "buffer_size=#{@messages.size}"
+          parts << "buffer_size=#{@messages.size + @requeued.size}"
         ensure
           @buffer_mutex.unlock
         end
@@ -590,12 +602,13 @@ module WaterDrop
     #
     # For a transactional producer we wrap the whole dispatch (including the operations-counter
     # bookkeeping) in `transaction`, so `@transaction_mutex` is acquired BEFORE
-    # `@operations_in_progress` is incremented. This makes `#produce` acquire locks in the same order
-    # as `#close` (`@transaction_mutex` -> `@operating_mutex` -> operations counter) and removes a
-    # lock-order inversion: without it, a dispatch that had already counted itself could block forever
-    # on `@transaction_mutex` held by a concurrent `#close` that was itself waiting for the operations
-    # counter to drain. When we already own the transaction lock (inside an explicit transaction block
-    # or the closing flush) the order is already correct, so we dispatch directly.
+    # `@operations_in_progress` is incremented. This makes `#produce` acquire locks in the same
+    # order as `#close` (`@transaction_mutex` -> `@operating_mutex` -> operations counter) and
+    # removes a lock-order inversion: without it, a dispatch that had already counted itself could
+    # block forever on `@transaction_mutex` held by a concurrent `#close` that was itself waiting
+    # for the operations counter to drain. When we already own the transaction lock (inside an
+    # explicit transaction block or the closing flush) the order is already correct, so we dispatch
+    # directly.
     #
     # @param message [Hash] message we want to send
     # @param label [String] short name of the public dispatch method (e.g. `"produce_sync"`) that
@@ -658,6 +671,29 @@ module WaterDrop
       @idempotent_fatal_error_attempts = 0
 
       result
+    rescue Rdkafka::ClosedProducerError, Rdkafka::ClosedInnerError
+      # A concurrent idempotent fatal-error reload closed the underlying client while this produce
+      # was already in flight. Unlike `#close`/`#disconnect`, which drain `@operations_in_progress`
+      # before closing the client, the idempotent reload swaps `@client` out from under sibling
+      # threads that have already passed the `@operating_mutex` gate and are inside `client.produce`.
+      # Racing with `@client.close`, such a thread sees either a producer already flagged closed
+      # (`ClosedProducerError`) or a nil inner librdkafka handle (`ClosedInnerError`) - which one
+      # depends purely on how far `close` has progressed.
+      #
+      # This is a benign, recoverable transient rather than a produce failure: the client has just
+      # been (or is being) rebuilt, so we retry the dispatch against the fresh client instead of
+      # surfacing a raw "closed producer" error to the caller - which would defeat the whole point
+      # of the transparent reload. Only the idempotent reload path closes the client with produces
+      # in flight, so we scope the retry to that configuration; anywhere else a closed client is a
+      # genuine error and must propagate. The retried pass re-runs `ensure_active!`, which raises
+      # `ProducerClosedError` once the producer is genuinely closing/closed, so this cannot spin
+      # forever.
+      raise unless config.reload_on_idempotent_fatal_error
+      raise if transactional?
+
+      @operations_in_progress.decrement
+
+      retry
     rescue SUPPORTED_FLOW_ERRORS.first => e
       # Check if this is a fatal error on an idempotent producer and we should reload.
       #

data/lib/waterdrop/version.rb

@@ -1,7 +1,6 @@
 # frozen_string_literal: true
 
-# WaterDrop library
 module WaterDrop
   # Current WaterDrop version
-  VERSION = "2.10.3"
+  VERSION = "2.10.5"
 end

data/lib/waterdrop.rb

@@ -10,7 +10,6 @@ require "singleton"
 require "karafka-core"
 require "pathname"
 
-# WaterDrop library
 module WaterDrop
   class << self
     # @return [String] root path of this gem

data/package-lock.json

@@ -286,9 +286,9 @@
       }
     },
     "node_modules/smol-toml": {
-      "version": "1.7.0",
-      "resolved": "https://registry.npmjs.org/smol-toml/-/smol-toml-1.7.0.tgz",
-      "integrity": "sha512-aqVvWoyO21L23mb+drl4RmMXbf6N7FdHjAhTRA9ZBL7apWBgfWC16KjrASI+1p9GAroljyMHj6fK67i0UiTNvQ==",
+      "version": "1.8.0",
+      "resolved": "https://registry.npmjs.org/smol-toml/-/smol-toml-1.8.0.tgz",
+      "integrity": "sha512-kCZr2V3ch9i00x8zXRhjUNVcjG9ijES5dDudkXvUVCT5QlJNQWElSJdZqyPemffHoLNUYwOcou0Fy+ojN0uHSQ==",
       "dev": true,
       "license": "BSD-3-Clause",
       "engines": {

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: waterdrop
 version: !ruby/object:Gem::Version
-  version: 2.10.3
+  version: 2.10.5
 platform: ruby
 authors:
 - Maciej Mensfeld
@@ -160,7 +160,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.10
+rubygems_version: 4.0.20
 specification_version: 4
 summary: Kafka messaging made easy!
 test_files: []