karafka-rdkafka 0.30.2 → 0.31.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7b68458fe1759b2645cce196bd5a602c236fe36853e428940641301272d5ecaf
-  data.tar.gz: 6248362a62a402e6004286abc9a7c1f7f29baa9ac375983fa1fc0a2d64159fc3
+  metadata.gz: 3cc16aee9b7a231693c92bb5abb3643a3c2d9ab47e557e46f095762306e0bb9f
+  data.tar.gz: dce5a57fab075eea90085314d5d84fcbc7c67e3d745e5ef465c5671be7302087
 SHA512:
-  metadata.gz: c69f725864465b708ac3159db0df5a8597e8cf6894f6108ed5ca34654bb4e905fc6dc1e79bd6689bbd2ff4378f4e7bb2f1edfb1bda4416e944912154fda608de
-  data.tar.gz: e0193cf81b907d704b97a27c1d0998f14b83ffb12361b8f0557e87b4345ae58c29caa2d2544192e8524a9feb719b3d707bb42b24cf2d4079ace9d481075277db
+  metadata.gz: 2b2923a1fab3ea8f16837a19128676c5690db0cf591fee5ca90f52a53b3dcd1b1f82ae59458f33a2ef2e40644b4760b53d82bc08e6bb3ece660a15764aba69ec
+  data.tar.gz: 94e99229a623f5e29100b28c8ccb6b34832b484ea11665fb93fc3b73cccc3b66515fc3638678e79216379eea1b9f383cf2e394098283ea87c24fe8f26e1a3ad8

data/CHANGELOG.md

@@ -1,33 +1,49 @@
 # Rdkafka Changelog
 
+## 0.31.0 (2026-10-12)
+- [Feature] Add `Admin#describe_cluster` to get the cluster id, controller and broker nodes (with racks) without a full metadata request. Ported from rdkafka-ruby (#1015).
+- [Feature] Add `Admin#describe_topics` to get topic ids, partition leaders and replicas, and per-topic errors. Both methods can also return the operations the client is authorized to perform. Ported from rdkafka-ruby (#1015).
+- [Feature] Add `Admin#describe_consumer_groups` to read the state, type, coordinator, authorized operations and members (with their current and target partition assignments) of consumer groups. Ported from rdkafka-ruby (#1016).
+- [Feature] Add `status`, `latency` and `broker_id` to `Producer::DeliveryReport`, with `#persisted?`, `#possibly_persisted?` and `#not_persisted?`, so a failed delivery tells whether a retry may duplicate the message. Ported from rdkafka-ruby (#1019).
+- [Enhancement] Bump librdkafka to `2.16.0` for OpenSSL and libcurl security fixes. Admin requests in flight on a decommissioned broker now fail with a retryable transport error. Ported from rdkafka-ruby (#1020).
+- [Enhancement] Bump the bundled OpenSSL to `3.5.9` (LTS) for its security fixes. Ported from rdkafka-ruby (#1007).
+- [Enhancement] Read each pointer field once in `Message#initialize`, saving 3 allocations per consumed message. Ported from rdkafka-ruby (#1026).
+- [Change] Drop Ruby 3.2 support; require Ruby 3.3+.
+- [Change] Build the precompiled `aarch64-linux-musl` gem on Alpine `3.24` (was `3.23`), matching `x86_64-linux-musl`. Ported from rdkafka-ruby (#1012).
+- [Change] Build the precompiled `macos_arm64` library on macOS 26 with the minimum macOS pinned to 14.0, instead of inheriting it from the build host. Ported from rdkafka-ruby (#1013).
+- [Maintenance] Split `lib/rdkafka/bindings.rb` into per-area files under `lib/rdkafka/bindings/`. `Rdkafka::Bindings` is unchanged. Ported from rdkafka-ruby (#1022).
+- [Maintenance] Stabilize the flaky partitions count cache "multiple topics with different TTLs" spec by driving its clock instead of sleeping. Ported from rdkafka-ruby (#1028).
+- [Fix] Sort merged share-group async acknowledgements so the broker does not reject them with `invalid_request` (librdkafka patch).
+- [Fix] Release share records acquired by an in-flight fetch on unsubscribe instead of keeping them locked (librdkafka patch). `ShareConsumer#unsubscribe` now closes the share sessions like `#close` does and waits for that (up to `fetch.wait.max.ms` with a fetch in flight).
+
 ## 0.30.2 (2026-10-01)
 - [Feature] Add `ShareConsumer#events_poll` (and `#events_poll_nb`) to service the statistics, error, log and OAuthBearer callbacks without acquiring records.
 
 ## 0.30.1 (2026-09-30)
-- [Fix] Derive `ShareConsumer#name` from the native handle at creation (mirroring `Consumer#name`) instead of leaving it `nil` until an OAuthBearer callback happened to set it. A `nil` name meant downstreams that route the global statistics and error callbacks by client name (e.g. Karafka) silently dropped every share-consumer statistic and background error.
+- [Fix] Set `ShareConsumer#name` from the native handle at creation, so name-based callback routing (e.g. Karafka) no longer drops share-consumer statistics and errors.
 
 ## 0.30.0 (2026-09-25)
-- [Fix] Register share consumers in `Rdkafka::Clients` and destroy the native handle when `Config#share_consumer` fails part-way. Share consumers own the native handle directly rather than a `NativeKafka` wrapper, so they bypass `build_native_client` and were missed by the `at_exit` shutdown hook, leaving librdkafka to be unloaded with a live share handle.
-- [Feature] Add preview support for KIP-932 share groups ("Queues for Kafka") via `Rdkafka::ShareConsumer`, created with `Config#share_consumer`. Members consume the same partitions cooperatively with per-record acknowledgements (`:accept`, `:release`, `:reject`) instead of committed offsets. Like the regular `Consumer`, `ShareConsumer` is a thin binding that exposes the librdkafka share primitives (`#subscribe`/`#unsubscribe`/`#subscription`, batch `#poll`, `#acknowledge`, `#commit_sync`, `#commit_async`, `#acknowledgement_commit_callback=`, `#close`) but drives none of them itself - there is no built-in poll loop, acknowledgement strategy or offset/lifecycle orchestration, leaving that to a higher layer such as Karafka. The handle is single-threaded by design, servicing log, statistics and error callbacks from within `#poll`. Requires a broker with share groups enabled (Apache Kafka 4.2.0+); librdkafka marks the feature as preview and not production-ready.
-- [Enhancement] `RdkafkaError.build_from_c` now uses the human readable string librdkafka attaches to the `rd_kafka_error_t` instead of always using `rd_kafka_err2str`, improving diagnostics for every error-pointer path. It also destroys a code-0 error object instead of leaking it on the early return.
-- [Enhancement] Extract the OAuthBearer token buffer/extension plumbing in `Helpers::OAuth` around two private native-call hooks so the share consumer overrides only the hooks instead of duplicating both public methods.
-- [Fix] Make `ShareConsumer#close` safe against in-flight calls from other threads: closing now waits for operations in progress (mirroring `NativeKafka`), preventing a NULL handle from reaching librdkafka or a double destroy when two threads race `close`.
-- [Fix] Keep the `ShareConsumer` native handle and its GC finalizer when `close`/`destroy` report an error. librdkafka leaves the instance intact in that case, so dropping the references beforehand leaked the client permanently; `close` is now retriable after such an error.
-- [Fix] Pin the acknowledgement commit callback `FFI::Function` in the state holder shared with the `ShareConsumer` GC finalizer, preventing a use-after-free when an unclosed consumer is garbage collected and the finalizer-driven destroy flushes pending acknowledgements through it.
-- [Fix] Make `ShareConsumer` fork-aware, mirroring `NativeKafka`. `rd_kafka_share_consumer_new` is backed by the same broker/main threads as any librdkafka client, which do not survive `fork`; the hand-rolled share lifecycle previously had no `@creator_pid` guard, so a process that forked after creating a share consumer would segfault (or deadlock) when the child's GC finalizer or `close` ran `rd_kafka_share_destroy` on the inherited handle. It now records the creating pid and reports `closed?` in any other process, so the finalizer and `close` skip the native teardown for an inherited handle while `with_native` rejects any use of it.
-- [Fix] Scope `ShareConsumer#each`'s `ClosedConsumerError` rescue to the `poll` call only. It previously wrapped the `yield`, so a `ClosedConsumerError` raised from inside the caller's block was swallowed and silently ended iteration; the block's errors now propagate, and only the consumer actually closing ends the loop. Its docs also note that, like `#poll`, `#each` can yield an `RdkafkaError` for a record that fails to build.
-- [Fix] Raise `ArgumentError` for `ShareConsumer#subscribe` without topics. librdkafka treats an empty share subscribe as an unsubscribe and returns no error (unlike the regular consumer), silently dropping the subscription.
-- [Fix] Capture the librdkafka client name onto `ShareConsumer#name` from the OAuthBearer token refresh callback. The share handle has no native name accessor, which made the documented name-based routing of the global oauth callback impossible for share consumers.
+- [Fix] Register share consumers in `Rdkafka::Clients` and destroy the native handle when `Config#share_consumer` fails part-way, so a share consumer is no longer missed by the `at_exit` shutdown hook.
+- [Feature] Add preview support for KIP-932 share groups ("Queues for Kafka") via `Rdkafka::ShareConsumer` (`Config#share_consumer`), with per-record acknowledgements (`:accept`, `:release`, `:reject`). Requires Apache Kafka 4.2.0+ with share groups enabled. Not production-ready.
+- [Enhancement] `RdkafkaError.build_from_c` now uses the human-readable string librdkafka attaches to the `rd_kafka_error_t` instead of `rd_kafka_err2str`, improving diagnostics for every error-pointer path.
+- [Enhancement] Refactor the `Helpers::OAuth` token plumbing around two private hooks so the share consumer overrides only those instead of duplicating both public methods.
+- [Fix] Make `ShareConsumer#close` thread-safe: it now waits for in-flight operations (mirroring `NativeKafka`), preventing a double destroy when two threads race `close`.
+- [Fix] Keep the `ShareConsumer` native handle and its GC finalizer when `close`/`destroy` report an error, so the client is no longer leaked and `close` stays retriable.
+- [Fix] Pin the acknowledgement commit callback `FFI::Function` for the lifetime of the `ShareConsumer`, preventing a use-after-free when an unclosed consumer is garbage collected.
+- [Fix] Make `ShareConsumer` fork-aware (mirroring `NativeKafka`): it reports `closed?` in a forked child, which no longer segfaults on `close` or finalization.
+- [Fix] Rescue `ClosedConsumerError` in `ShareConsumer#each` only around `poll`, so a `ClosedConsumerError` raised inside the block propagates instead of silently ending iteration. Like `#poll`, `#each` can also yield an `RdkafkaError` for a record that fails to build.
+- [Fix] Raise `ArgumentError` for `ShareConsumer#subscribe` without topics, which librdkafka otherwise treats as an unsubscribe without error, silently dropping the subscription.
+- [Fix] Capture the librdkafka client name onto `ShareConsumer#name` from the OAuthBearer token refresh callback, since the share handle has no native name accessor for the documented name-based oauth callback routing.
 - [Fix] Update `ext/build_common.sh` (precompiled builds) to librdkafka `2.15.0` and its tarball checksum; it still pinned `2.14.1`, whose tarball is no longer vendored.
-- [Fix] Stop the `share_consumer_multi_member` integration spec from failing on redelivery. KIP-932 share groups are documented as at-least-once, so a record whose acquisition lock expires before it is acknowledged may be redelivered with an incremented `delivery_count`; the spec now tracks distinct payloads and fails only when a record is never delivered.
-- [Fix] Apply the same at-least-once tolerance to the `share_consumer_explicit_ack` integration spec, which still asserted exactly one redelivered record. It now requires only the released record to be redelivered (with `delivery_count >= 2`), forbids only the rejected record, and tolerates an accepted record reappearing when its acquisition lock expired before the commit flushed the accept.
-- [Fix] Add the `nofile` ulimit to `docker-compose-ssl.yml`, mirroring `docker-compose.yml`. The SSL CI broker runs the same topic-heavy suite (now enlarged by the share consumer specs) and could otherwise exhaust the default file-descriptor limit under the many short-lived topics.
-- [Fix] Enable KIP-932 share groups on the macOS CI broker, whose KRaft `server.properties` never opted into the share rebalance protocol, so the one broker-backed share consumer spec received zero records on every macOS runner. `CONFLUENT_VERSION` is bumped to `8.3.0` and the download path is derived from it so a Renovate bump cannot desync them.
-- [Enhancement] Add two share consumer integration specs: `share_consumer_implicit_ack_spec` (polling alone accepts the previous batch, and accepted records are not redelivered to a second member) and `share_consumer_multi_topic_spec` (one consumer subscribed to several topics receives every topic's records, attributed to the right topic). Both track distinct payloads so an at-least-once redelivery cannot end the consume loop early.
-- [Fix] Stop the `statistics_unassigned_producer` integration spec from flaking on a metadata-propagation race: the freshly built producer could send before fetching the new topic's metadata, so the produce is now retried on `unknown_topic_or_part` and `leader_not_available`.
-- [Fix] Also stabilize the `statistics_unassigned_producer` integration spec against a delivery wait that outlives its budget. With 1000 partitions librdkafka keeps the message queued while leaders are still being elected, so the handle stays pending and the wait raises `WaitTimeoutError` - a `RuntimeError` the spec's existing code-based rescue could never catch. The produce now uses the default 60s handle wait (matching the sibling `statistics_unassigned_*` specs, instead of a self-imposed 15s) and retries on that timeout too.
-- [Fix] Stabilize the `consumer_memberid_clusterid_leak` integration spec against RSS measurement noise on Ruby 4.0. It now compacts the heap before sampling (matching the `to_native_tpl` leak spec) and the ceiling is 3 MB, still well under the ~5.7 MB a genuine per-call leak would cost.
-- [Note] Share consumers emit the regular consumer statistics JSON (including `cgrp`; there is no share-specific section in librdkafka 2.15.0) through the usual `statistics_callback`, serviced from within `#poll`. The share fetch path bypasses the per-partition stats bookkeeping entirely, so the `topics` section carries no partition entries and the `statistics.unassigned.include` filter needs no share-consumer special-casing; per-partition share metrics are not exposed by librdkafka 2.15.0 at all.
+- [Fix] Stop the `share_consumer_multi_member` integration spec from failing on an at-least-once redelivery; it now tracks distinct payloads and fails only when a record is never delivered.
+- [Fix] Apply the same at-least-once tolerance to the `share_consumer_explicit_ack` integration spec: require only the released record to be redelivered, forbid the rejected one, and tolerate an accepted record reappearing.
+- [Fix] Add the `nofile` ulimit to `docker-compose-ssl.yml` (mirroring `docker-compose.yml`) so the SSL CI broker does not exhaust file descriptors under the topic-heavy share-consumer suite.
+- [Fix] Enable KIP-932 share groups on the macOS CI broker, whose KRaft config never opted into the share rebalance protocol, so its share-consumer spec no longer receives zero records. `CONFLUENT_VERSION` is bumped to `8.3.0`.
+- [Enhancement] Add `share_consumer_implicit_ack` and `share_consumer_multi_topic` integration specs.
+- [Fix] Stop the `statistics_unassigned_producer` integration spec from flaking on a metadata-propagation race by retrying the produce on `unknown_topic_or_part` and `leader_not_available`.
+- [Fix] Also stabilize the `statistics_unassigned_producer` integration spec against a delivery wait that outlives its budget: it now uses the default 60s handle wait and retries on `WaitTimeoutError`.
+- [Fix] Stabilize the `consumer_memberid_clusterid_leak` integration spec against RSS measurement noise on Ruby 4.0 by compacting the heap before sampling and raising the ceiling to 3 MB.
+- [Note] Share consumers emit the regular consumer statistics through `statistics_callback`. librdkafka 2.15.0 has no share-specific metrics, so the `topics` section has no partition entries.
 - [Feature] Add `Admin#alter_consumer_group_offsets` and `Admin#delete_consumer_group_offsets` to set or clear a consumer group's committed offsets from the admin client. Ported from rdkafka-ruby (#983).
 - [Enhancement] Bump librdkafka to `2.15.1` for OpenSSL and libcurl security fixes. It changes how IPv6 addresses are formatted and validated against broker certificates.
 - [Enhancement] Bump the bundled zlib to `1.3.2`.
@@ -37,8 +53,8 @@
 - [Maintenance] Drop the unused `dist/openssl-3.0.16.tar.gz` build cache.
 - [Fix] Honor the `isolation_level:` argument to `Admin#list_offsets`, which was silently ignored. Ported from rdkafka-ruby (#983).
 - [Fix] Return correct results for every item of a multi-item topic, partition, group or ACL admin request, not just the first. Ported from rdkafka-ruby (#987).
-- [Maintenance] Fix a use-after-free in the multi-item admin integration test that read result-name pointers after librdkafka had destroyed the background event, which made it flaky on newer glibc (e.g. Debian trixie). Ported from rdkafka-ruby (#991).
-- [Maintenance] Stabilize consumer specs on slow CI runners: wait longer for a partition assignment, raise the `TestTopics.create` admin timeout (a timeout there also tripped the leaked-handle guard), give the non-blocking `poll_nb` message check a generous retry budget instead of a fixed ~2s, and let the long-running consumption spec drain its backlog instead of stopping at a fixed 60s. The assignment wait is tunable via `RDKAFKA_TEST_ASSIGNMENT_TIMEOUT`. Ported from rdkafka-ruby (#990).
+- [Maintenance] Fix a use-after-free in the multi-item admin integration test that made it flaky on newer glibc. Ported from rdkafka-ruby (#991).
+- [Maintenance] Stabilize consumer specs on slow CI runners. The assignment wait is tunable via `RDKAFKA_TEST_ASSIGNMENT_TIMEOUT`. Ported from rdkafka-ruby (#990).
 
 ## 0.29.0 (2026-09-14)
 - [Enhancement] Bump librdkafka to `2.15.0` (staying on `2.15.0` rather than `2.15.1` so users hitting a regression in `2.15.1` have a stable fallback).
@@ -126,8 +142,8 @@
 - [Fix] Prevent cascading test failures in admin specs when a single handle leaks into the registry (from upstream).
 
 ## 0.24.0 (2026-02-25)
-- **[Feature]** Add `Producer#queue_size` (and `#queue_length` alias) to report the number of messages waiting in the librdkafka output queue. Useful for monitoring producer backpressure, implementing custom flow control, debugging message delivery issues, and graceful shutdown logic.
-- **[Feature]** Add fiber scheduler API for integration with Ruby fiber schedulers (Falcon, Async) and custom event loops (from upstream). Expose `enable_queue_io_events` and `enable_background_queue_io_events` methods on `Consumer`, `Producer`, and `Admin`.
+- **[Feature]** Add `Producer#queue_size` (and the `#queue_length` alias) to report the number of messages waiting in the librdkafka output queue.
+- **[Feature]** Add `enable_queue_io_events` and `enable_background_queue_io_events` to `Consumer`, `Producer` and `Admin` for fiber schedulers (Falcon, Async) and custom event loops (from upstream).
 - **[Deprecation]** `AbstractHandle#wait` parameter `max_wait_timeout` (seconds) is deprecated in favor of `max_wait_timeout_ms` (milliseconds). The old parameter still works with backwards compatibility but will be removed in v1.0.0.
 - **[Deprecation]** `PartitionsCountCache` constructor parameter `ttl` (seconds) is deprecated in favor of `ttl_ms` (milliseconds). The old parameter still works with backwards compatibility but will be removed in v1.0.0.
 - [Enhancement] Add Ruby 4.0 support.
@@ -154,7 +170,7 @@
 - [Enhancement] Add `RdkafkaError.build_fatal` class method for centralized fatal error construction.
 - [Enhancement] Add comprehensive tests for fatal error handling including unit tests and integration tests.
 - [Enhancement] Add `RD_KAFKA_PARTITION_UA` constant for unassigned partition (-1).
-- [Enhancement] Replace magic numbers with named constants: use `RD_KAFKA_RESP_ERR_NO_ERROR` instead of `0` for error code checks (18 instances) and `RD_KAFKA_PARTITION_UA` instead of `-1` for partition values (9 instances) across the codebase for better code clarity and maintainability.
+- [Enhancement] Replace magic numbers with the `RD_KAFKA_RESP_ERR_NO_ERROR` and `RD_KAFKA_PARTITION_UA` constants.
 - [Enhancement] Add `Rdkafka::Testing` module for testing fatal error scenarios on both producers and consumers.
 - [Deprecated] `RdkafkaError.validate_fatal!` - use `validate!` with `client_ptr` parameter instead.
 
@@ -168,7 +184,7 @@
 
 ## 0.22.1 (2025-10-09)
 - [Enhancement] Optimize header processing to eliminate double hash lookups and method checking overhead.
-- [Enhancement] Optimize producer header processing with early returns and efficient array operations (69% faster for nil headers, 41% faster for empty headers, 12-32% faster when headers are present, with larger improvements for complex header scenarios).
+- [Enhancement] Speed up producer header processing (69% faster for nil headers, 41% for empty headers, 12-32% when headers are present).
 
 ## 0.22.0 (2025-09-26)
 - **[EOL]** Drop support for Ruby 3.1 to move forward with the fiber scheduler work.

data/README.md

@@ -63,6 +63,7 @@ Contributions should generally be made to the upstream [rdkafka-ruby repository]
 
 | karafka-rdkafka | librdkafka | patches |
 |-|-|-|
+| 0.31.x (2026-10-12) | 2.16.0 (2026-10-06)  | yes |
 | 0.30.x (2026-09-25) | 2.15.1 (2026-09-09)  | yes |
 | 0.29.x (2026-09-14) | 2.15.0 (2026-06-30)  | yes |
 | 0.28.x (2026-07-12) | 2.14.2 (2026-06-03)  | yes |

data/dist/patches/rdkafka_share_async_ack_sort.patch

@@ -0,0 +1,18 @@
+# This patch is released under the 2-clause BSD license, same as librdkafka
+#
+# Upstream: https://github.com/confluentinc/librdkafka/issues/5620
+# Drop this patch once librdkafka sorts merged async share acknowledgements itself.
+#
+--- librdkafka_2.15.1/src/rdkafka_share_acknowledgement.c
++++ librdkafka_2.15.1/src/rdkafka_share_acknowledgement.c
+@@ -656,6 +656,10 @@
+                                     &existing->entries,
+                                     rd_kafka_share_ack_batch_entry_copy(entry));
+                         }
++                        /* Batches merged from separate commits may interleave,
++                         * while the broker requires ascending ack batches. */
++                        rd_list_sort(&existing->entries,
++                                     rd_kafka_share_ack_entries_sort_cmp_ptr);
+                         rd_kafka_share_ack_batches_destroy(batch);
+                 } else {
+                         rd_list_add(leader_rkb->rkb_share_async_ack_details,