karafka-rdkafka 0.30.2-aarch64-linux-gnu → 0.32.0-aarch64-linux-gnu

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 01e95b081fe08958692a07b9433a1d60bd3ebfcc59ea9027c11b039c81311577
-  data.tar.gz: 8b608359879e1b050c30c76efde27a97e98cd7c2969458635073d48b653e6548
+  metadata.gz: d1807a66cb860ccf33c53a7afe9759238ab8c8ebe04d4195e482b1f4c2ab525a
+  data.tar.gz: f3c36b50aea64fa52f8481fc5e087ce35aa098e3788d7837dbf51140256a9ce2
 SHA512:
-  metadata.gz: 995b45f38dbe9342bf2ddd9ba5150d7279bb6674c2d9bf20b2635dab7e451518dafe38d477b3dcf02f5b6da883a6a43a2535768ed10c6945f84328f29920d8ca
-  data.tar.gz: 956e1d30ebec721b4504b8e14679a2a122836f50bdfb89795c4dcb5c7a47bd797e281a241c4f1343018e9c2bf406543a0d05e988d38c176a62286e29808bd784
+  metadata.gz: 6d429c8e67c82dae33f618dd21f6e9f5b54f17981277c14e0f2e52bbf0ec327df1f8b741e6088da9970d16aaba9aae7547cb1dc9e945013f901645d667c07598
+  data.tar.gz: 1408a545d6feb84b61cf5192a5ec3d29ce119df5ea623b083ee27f5eba4bb2fe83344ed5fb4de9e398528079864da30f9335965b63885b07dd55cacaac85978f

data/CHANGELOG.md

@@ -1,65 +1,129 @@
 # Rdkafka Changelog
 
+## 0.32.0 (2026-11-04)
+- **[Feature]** Add `Config#consumer_offset_commit_listener`: its `#on_offsets_committed(error, list)` gets the result of every offset commit when the consumer polls or closes, so an async commit that the broker rejects reaches Ruby instead of only a `COMMITFAIL` log line. Ported from rdkafka-ruby (#1061).
+- **[Feature]** Let the offset commit listener define `#on_offsets_commit_failed(error, list)`, called only for failed commits (not on success or `no_offset`). A listener with only this method does not convert the commit offsets of the other commits.
+- **[Feature]** Add `Admin#list_consumer_group_offsets` to read the committed offsets of any consumer group without joining it, with a `require_stable_offsets` option.
+- **[Feature]** Add `validate_only:`, `operation_timeout_ms:` and `request_timeout_ms:` keyword options to `Admin#create_topic` and `#create_partitions`, and the two timeouts to `#delete_topic`. A `create_topic` topic config passed without braces keeps working. Ported from rdkafka-ruby (#1092).
+- [Enhancement] Reduce object shape variations of the admin report classes to one shape per class.
+- [Enhancement] Keep `partition_key` produces working during a cluster outage once the topic partition count is known, instead of raising. Ported from rdkafka-ruby (#1039).
+- [Enhancement] Add `Rdkafka::Config.partition_count_cache_ttl_ms=` to set how long `partition_key` produces reuse a cached partition count (default 30 seconds).
+- [Enhancement] Make `Producer#produce` use about 13% less CPU per call. Ported from rdkafka-ruby (#1064).
+- [Enhancement] Allocate fewer objects per delivered message: delivery callback reports reuse the topic String given to `produce`. Ported from rdkafka-ruby (#1066).
+- [Change] Build the precompiled `x86_64-linux-gnu` and `aarch64-linux-gnu` libraries in an Ubuntu 22.04 container on Ubuntu 24.04 runners, keeping the glibc 2.34 floor after GitHub retires its Ubuntu 22.04 runners. Ported from rdkafka-ruby (#1029).
+- [Change] Depend on `mini_portile2` and `rake` only in the source gem, as only it compiles librdkafka. Apps that got `rake` only through this gem must add it to their Gemfile.
+- [Change] Remove the unused `Rdkafka::Defaults::NATIVE_KAFKA_SYNCHRONIZE_SLEEP_INTERVAL_MS`: `close` and `ShareConsumer#close` now wait for operations in progress without polling. Ported from rdkafka-ruby (#1055).
+- [Maintenance] Fix a use-after-free in the `client_shutdown_dlclose` integration test that made it crash intermittently after its own unmap.
+- [Maintenance] Stabilize the flaky `Consumer#seek` and `#seek_by` "works when a partition is paused" specs on loaded CI runners.
+- [Maintenance] Stop the `client_shutdown_dlclose` integration test from failing with `Errno::ESRCH` when a librdkafka thread exits while its name is read.
+- [Maintenance] Stabilize the flaky `Consumer` "long running consumption" spec, and stop its producer when it fails so its statistics do not break later specs.
+- [Maintenance] Drop `pry` from the development bundle and stop requiring it in the spec helper.
+- [Maintenance] Stabilize the flaky `admin_describe_topics` integration test, which could describe its new topic before the topic was in the broker's metadata.
+- [Maintenance] Name positional String topic parameters `topic_name` (`Consumer#query_watermark_offsets`, `#seek_by`, `Producer#partition_count`, `TopicPartitionList#add_topic` and others). Callers are not affected. Ported from rdkafka-ruby (#1052).
+- [Maintenance] Name the positional String topic parameters of `Producer::PartitionsCountCache` `topic_name`. Callers are not affected.
+- [Maintenance] Cover with integration tests that coordinator errors on group offset requests do not crash the admin client or a consumer without `group.id`. Ported from rdkafka-ruby (#1044).
+- [Maintenance] Skip the bindings signature parity spec on JRuby, whose FFI does not expose function signatures. Ported from rdkafka-ruby (#1046).
+- [Maintenance] Match the FFI pointer expectations in the consumer specs by address, so they pass on JRuby. Ported from rdkafka-ruby (#1047).
+- [Maintenance] Run the `client_shutdown_dlclose` integration test on JRuby on Linux. Ported from rdkafka-ruby (#1049).
+- [Maintenance] Check the native call balance instead of RSS in the leak integration specs on JRuby. Ported from rdkafka-ruby (#1050).
+- [Maintenance] Give the `Consumer#lag` spec a 30 s list offsets wait.
+- [Maintenance] Give the share group offset reset in the `ShareConsumer` specs a 30 s wait and always close its admin, so a timeout no longer fails the stats specs that run after it.
+- [Maintenance] Define `Consumer.finalizer` and `Consumer.report_close_error` in a `class << self` block. Callers are not affected. Ported from rdkafka-ruby (#1068).
+- [Maintenance] Wait 30 s for the `before(:suite)` topic creates. Ported from rdkafka-ruby (#1065).
+- [Maintenance] Run the concurrent first produce checks on JRuby, now that `close` no longer hangs there after concurrent produces. Ported from rdkafka-ruby (#1083).
+- [Maintenance] Create the topics of the producer `timestamp` specs with `retention.ms: -1`, so broker retention no longer deletes their 2017-timestamped messages before the specs read them back. Ported from rdkafka-ruby (#1077).
+- [Maintenance] Time out the SSL spec step of the Ubuntu aarch64 complementary CI workflow after 25 minutes, so a hanging example fails the step and its log is kept. Ported from rdkafka-ruby (#1086).
+- [Maintenance] Fix the YARD notes of `events_poll_nb_each` and `Consumer#poll_nb_each`: they do not hold a lock while they loop, so other operations run concurrently and only `close` waits. Ported from rdkafka-ruby (#1089).
+- [Maintenance] Stop the fake SSL servers in integration specs without `Thread#kill`. Ported from rdkafka-ruby (#1059).
+- [Fix] Stop `Consumer#store_offset` and other calls that pass partition metadata from raising `NoMethodError` on JRuby. Ported from rdkafka-ruby (#1045).
+- [Fix] Pass the error buffer size to librdkafka as `size_t` when setting config options and creating clients.
+- [Fix] Raise `Rdkafka::LibraryLoadError` with the glibc upgrade hint instead of `NameError` when the precompiled librdkafka needs a newer glibc. Ported from rdkafka-ruby (#1031).
+- [Fix] Report a failed consumer close (e.g. of a fatally-errored consumer) to `Rdkafka::Config.error_callback` instead of dropping it. `Consumer#close` still does not raise.
+- [Fix] Stop the statistics, error and OAuth token refresh callbacks from raising `NoMethodError` when another thread clears the global callback while they run. On JRuby this aborted the process from the polling thread. Ported from rdkafka-ruby (#1058).
+- [Fix] Stop `ShareConsumer#close` from hanging forever when it runs while the first partition assignment waits for topic metadata, by patching the bundled librdkafka (confluentinc/librdkafka#5658).
+- [Fix] Pass the `#oauthbearer_set_token` size arguments to librdkafka as `size_t`, so the token set error is no longer truncated at random on JRuby.
+- [Fix] Pass the fatal error buffer size (used by `Producer#fatal_error` and the error callback) and the config resource array count to librdkafka as `size_t`.
+- [Fix] Stop the producer from retrying a message without backoff after about 25 retries, by patching the bundled librdkafka (confluentinc/librdkafka#5646).
+- [Fix] Stop the rebalance callback from raising `LocalJumpError` when called for a client whose opaque is already gone.
+- [Fix] Create one topic handle per topic when many threads produce to a new topic at once. On JRuby this race leaked topic handles and could crash with a segfault; `produce` now raises `TopicHandleCreationError` instead of passing a missing handle to librdkafka. Ported from rdkafka-ruby (#1056).
+- [Fix] Count a client's (and a share consumer's) operations in progress under one lock, so on JRuby `close` no longer hangs or destroys the client while another thread still uses it. Ported from rdkafka-ruby (#1055).
+
+## 0.31.0 (2026-10-12)
+- **[EOL]** Drop Ruby 3.2 support; require Ruby 3.3+.
+- [Feature] Add `Admin#describe_cluster` to get the cluster id, controller and broker nodes (with racks) without a full metadata request. Ported from rdkafka-ruby (#1015).
+- [Feature] Add `Admin#describe_topics` to get topic ids, partition leaders, replicas and per-topic errors. Both methods can also return the client's authorized operations. Ported from rdkafka-ruby (#1015).
+- [Feature] Add `Admin#describe_consumer_groups` to read consumer groups' state, type, coordinator, authorized operations and members with their current and target assignments. Ported from rdkafka-ruby (#1016).
+- [Feature] Add `status`, `latency` and `broker_id` to `Producer::DeliveryReport`, with `#persisted?`, `#possibly_persisted?` and `#not_persisted?`, so a failed delivery tells whether a retry may duplicate the message. Ported from rdkafka-ruby (#1019).
+- [Enhancement] Bump librdkafka to `2.16.0` for OpenSSL and libcurl security fixes. Admin requests in flight on a decommissioned broker now fail with a retryable transport error. Ported from rdkafka-ruby (#1020).
+- [Enhancement] Bump the bundled OpenSSL to `3.5.9` (LTS) for its security fixes. Ported from rdkafka-ruby (#1007).
+- [Enhancement] Read each pointer field once in `Message#initialize`, saving 3 allocations per consumed message. Ported from rdkafka-ruby (#1026).
+- [Change] Build the precompiled `aarch64-linux-musl` gem on Alpine `3.24` (was `3.23`), matching `x86_64-linux-musl`. Ported from rdkafka-ruby (#1012).
+- [Change] Build the precompiled `macos_arm64` library on macOS 26 with the minimum macOS pinned to 14.0, instead of inheriting it from the build host. Ported from rdkafka-ruby (#1013).
+- [Maintenance] Split `lib/rdkafka/bindings.rb` into per-area files under `lib/rdkafka/bindings/`. `Rdkafka::Bindings` is unchanged. Ported from rdkafka-ruby (#1022).
+- [Maintenance] Stabilize a flaky partitions count cache spec. Ported from rdkafka-ruby (#1028).
+- [Fix] Sort merged share-group async acknowledgements so the broker does not reject them with `invalid_request` (librdkafka patch).
+- [Fix] Release share records acquired by an in-flight fetch on unsubscribe instead of keeping them locked (librdkafka patch). `ShareConsumer#unsubscribe` now closes the share sessions like `#close` and can wait up to `fetch.wait.max.ms`.
+
 ## 0.30.2 (2026-10-01)
 - [Feature] Add `ShareConsumer#events_poll` (and `#events_poll_nb`) to service the statistics, error, log and OAuthBearer callbacks without acquiring records.
 
 ## 0.30.1 (2026-09-30)
-- [Fix] Derive `ShareConsumer#name` from the native handle at creation (mirroring `Consumer#name`) instead of leaving it `nil` until an OAuthBearer callback happened to set it. A `nil` name meant downstreams that route the global statistics and error callbacks by client name (e.g. Karafka) silently dropped every share-consumer statistic and background error.
+- [Fix] Set `ShareConsumer#name` from the native handle at creation, so name-based callback routing (e.g. Karafka) no longer drops share-consumer statistics and errors.
 
 ## 0.30.0 (2026-09-25)
-- [Fix] Register share consumers in `Rdkafka::Clients` and destroy the native handle when `Config#share_consumer` fails part-way. Share consumers own the native handle directly rather than a `NativeKafka` wrapper, so they bypass `build_native_client` and were missed by the `at_exit` shutdown hook, leaving librdkafka to be unloaded with a live share handle.
-- [Feature] Add preview support for KIP-932 share groups ("Queues for Kafka") via `Rdkafka::ShareConsumer`, created with `Config#share_consumer`. Members consume the same partitions cooperatively with per-record acknowledgements (`:accept`, `:release`, `:reject`) instead of committed offsets. Like the regular `Consumer`, `ShareConsumer` is a thin binding that exposes the librdkafka share primitives (`#subscribe`/`#unsubscribe`/`#subscription`, batch `#poll`, `#acknowledge`, `#commit_sync`, `#commit_async`, `#acknowledgement_commit_callback=`, `#close`) but drives none of them itself - there is no built-in poll loop, acknowledgement strategy or offset/lifecycle orchestration, leaving that to a higher layer such as Karafka. The handle is single-threaded by design, servicing log, statistics and error callbacks from within `#poll`. Requires a broker with share groups enabled (Apache Kafka 4.2.0+); librdkafka marks the feature as preview and not production-ready.
-- [Enhancement] `RdkafkaError.build_from_c` now uses the human readable string librdkafka attaches to the `rd_kafka_error_t` instead of always using `rd_kafka_err2str`, improving diagnostics for every error-pointer path. It also destroys a code-0 error object instead of leaking it on the early return.
-- [Enhancement] Extract the OAuthBearer token buffer/extension plumbing in `Helpers::OAuth` around two private native-call hooks so the share consumer overrides only the hooks instead of duplicating both public methods.
-- [Fix] Make `ShareConsumer#close` safe against in-flight calls from other threads: closing now waits for operations in progress (mirroring `NativeKafka`), preventing a NULL handle from reaching librdkafka or a double destroy when two threads race `close`.
-- [Fix] Keep the `ShareConsumer` native handle and its GC finalizer when `close`/`destroy` report an error. librdkafka leaves the instance intact in that case, so dropping the references beforehand leaked the client permanently; `close` is now retriable after such an error.
-- [Fix] Pin the acknowledgement commit callback `FFI::Function` in the state holder shared with the `ShareConsumer` GC finalizer, preventing a use-after-free when an unclosed consumer is garbage collected and the finalizer-driven destroy flushes pending acknowledgements through it.
-- [Fix] Make `ShareConsumer` fork-aware, mirroring `NativeKafka`. `rd_kafka_share_consumer_new` is backed by the same broker/main threads as any librdkafka client, which do not survive `fork`; the hand-rolled share lifecycle previously had no `@creator_pid` guard, so a process that forked after creating a share consumer would segfault (or deadlock) when the child's GC finalizer or `close` ran `rd_kafka_share_destroy` on the inherited handle. It now records the creating pid and reports `closed?` in any other process, so the finalizer and `close` skip the native teardown for an inherited handle while `with_native` rejects any use of it.
-- [Fix] Scope `ShareConsumer#each`'s `ClosedConsumerError` rescue to the `poll` call only. It previously wrapped the `yield`, so a `ClosedConsumerError` raised from inside the caller's block was swallowed and silently ended iteration; the block's errors now propagate, and only the consumer actually closing ends the loop. Its docs also note that, like `#poll`, `#each` can yield an `RdkafkaError` for a record that fails to build.
-- [Fix] Raise `ArgumentError` for `ShareConsumer#subscribe` without topics. librdkafka treats an empty share subscribe as an unsubscribe and returns no error (unlike the regular consumer), silently dropping the subscription.
-- [Fix] Capture the librdkafka client name onto `ShareConsumer#name` from the OAuthBearer token refresh callback. The share handle has no native name accessor, which made the documented name-based routing of the global oauth callback impossible for share consumers.
-- [Fix] Update `ext/build_common.sh` (precompiled builds) to librdkafka `2.15.0` and its tarball checksum; it still pinned `2.14.1`, whose tarball is no longer vendored.
-- [Fix] Stop the `share_consumer_multi_member` integration spec from failing on redelivery. KIP-932 share groups are documented as at-least-once, so a record whose acquisition lock expires before it is acknowledged may be redelivered with an incremented `delivery_count`; the spec now tracks distinct payloads and fails only when a record is never delivered.
-- [Fix] Apply the same at-least-once tolerance to the `share_consumer_explicit_ack` integration spec, which still asserted exactly one redelivered record. It now requires only the released record to be redelivered (with `delivery_count >= 2`), forbids only the rejected record, and tolerates an accepted record reappearing when its acquisition lock expired before the commit flushed the accept.
-- [Fix] Add the `nofile` ulimit to `docker-compose-ssl.yml`, mirroring `docker-compose.yml`. The SSL CI broker runs the same topic-heavy suite (now enlarged by the share consumer specs) and could otherwise exhaust the default file-descriptor limit under the many short-lived topics.
-- [Fix] Enable KIP-932 share groups on the macOS CI broker, whose KRaft `server.properties` never opted into the share rebalance protocol, so the one broker-backed share consumer spec received zero records on every macOS runner. `CONFLUENT_VERSION` is bumped to `8.3.0` and the download path is derived from it so a Renovate bump cannot desync them.
-- [Enhancement] Add two share consumer integration specs: `share_consumer_implicit_ack_spec` (polling alone accepts the previous batch, and accepted records are not redelivered to a second member) and `share_consumer_multi_topic_spec` (one consumer subscribed to several topics receives every topic's records, attributed to the right topic). Both track distinct payloads so an at-least-once redelivery cannot end the consume loop early.
-- [Fix] Stop the `statistics_unassigned_producer` integration spec from flaking on a metadata-propagation race: the freshly built producer could send before fetching the new topic's metadata, so the produce is now retried on `unknown_topic_or_part` and `leader_not_available`.
-- [Fix] Also stabilize the `statistics_unassigned_producer` integration spec against a delivery wait that outlives its budget. With 1000 partitions librdkafka keeps the message queued while leaders are still being elected, so the handle stays pending and the wait raises `WaitTimeoutError` - a `RuntimeError` the spec's existing code-based rescue could never catch. The produce now uses the default 60s handle wait (matching the sibling `statistics_unassigned_*` specs, instead of a self-imposed 15s) and retries on that timeout too.
-- [Fix] Stabilize the `consumer_memberid_clusterid_leak` integration spec against RSS measurement noise on Ruby 4.0. It now compacts the heap before sampling (matching the `to_native_tpl` leak spec) and the ceiling is 3 MB, still well under the ~5.7 MB a genuine per-call leak would cost.
-- [Note] Share consumers emit the regular consumer statistics JSON (including `cgrp`; there is no share-specific section in librdkafka 2.15.0) through the usual `statistics_callback`, serviced from within `#poll`. The share fetch path bypasses the per-partition stats bookkeeping entirely, so the `topics` section carries no partition entries and the `statistics.unassigned.include` filter needs no share-consumer special-casing; per-partition share metrics are not exposed by librdkafka 2.15.0 at all.
+- [Feature] Add preview support for KIP-932 share groups ("Queues for Kafka") via `Rdkafka::ShareConsumer` (`Config#share_consumer`), with per-record acknowledgements (`:accept`, `:release`, `:reject`). Requires Apache Kafka 4.2.0+ with share groups enabled. Not production-ready.
 - [Feature] Add `Admin#alter_consumer_group_offsets` and `Admin#delete_consumer_group_offsets` to set or clear a consumer group's committed offsets from the admin client. Ported from rdkafka-ruby (#983).
+- [Enhancement] Use the error string librdkafka attaches to each error in `RdkafkaError.build_from_c`, for clearer error messages.
+- [Enhancement] Refactor the `Helpers::OAuth` token handling so the share consumer reuses it instead of duplicating it.
 - [Enhancement] Bump librdkafka to `2.15.1` for OpenSSL and libcurl security fixes. It changes how IPv6 addresses are formatted and validated against broker certificates.
 - [Enhancement] Bump the bundled zlib to `1.3.2`.
 - [Enhancement] Bump the bundled MIT Kerberos (krb5) to `1.22.2`. Ported from rdkafka-ruby (#979).
 - [Enhancement] Add `Rdkafka::Config.partitioner_key_uses_bytesize` to hash partition keys by byte length, matching other Kafka clients for multibyte keys. Defaults to `false`. Ported from rdkafka-ruby (#985).
+- [Maintenance] Stop the `share_consumer_multi_member` integration spec from failing on an at-least-once redelivery.
+- [Maintenance] Apply the same at-least-once tolerance to the `share_consumer_explicit_ack` integration spec.
+- [Maintenance] Add the `nofile` ulimit to the SSL CI broker so the share-consumer suite does not exhaust file descriptors.
+- [Maintenance] Enable KIP-932 share groups on the macOS CI broker and bump its `CONFLUENT_VERSION` to `8.3.0`.
+- [Maintenance] Add `share_consumer_implicit_ack` and `share_consumer_multi_topic` integration specs.
+- [Maintenance] Stop the `statistics_unassigned_producer` integration spec from flaking on a metadata-propagation race.
+- [Maintenance] Also stabilize the `statistics_unassigned_producer` integration spec against delivery wait timeouts.
+- [Maintenance] Stabilize the `consumer_memberid_clusterid_leak` integration spec against RSS measurement noise on Ruby 4.0.
 - [Maintenance] Derive the zlib `CHECKSUMS` entry from `ZLIB_VERSION`.
 - [Maintenance] Drop the unused `dist/openssl-3.0.16.tar.gz` build cache.
+- [Maintenance] Fix a use-after-free in the multi-item admin integration test that made it flaky on newer glibc. Ported from rdkafka-ruby (#991).
+- [Maintenance] Stabilize consumer specs on slow CI runners. The assignment wait is tunable via `RDKAFKA_TEST_ASSIGNMENT_TIMEOUT`. Ported from rdkafka-ruby (#990).
+- [Fix] Register share consumers in `Rdkafka::Clients` and destroy the native handle when `Config#share_consumer` fails part-way, so a share consumer is no longer missed by the `at_exit` shutdown hook.
+- [Fix] Make `ShareConsumer#close` thread-safe: it now waits for in-flight operations (mirroring `NativeKafka`), preventing a double destroy when two threads race `close`.
+- [Fix] Keep the `ShareConsumer` native handle and its GC finalizer when `close`/`destroy` report an error, so the client is no longer leaked and `close` stays retriable.
+- [Fix] Prevent a use-after-free when an unclosed `ShareConsumer` is garbage collected.
+- [Fix] Make `ShareConsumer` fork-aware (mirroring `NativeKafka`): it reports `closed?` in a forked child, which no longer segfaults on `close` or finalization.
+- [Fix] Rescue `ClosedConsumerError` in `ShareConsumer#each` only around `poll`, so a `ClosedConsumerError` raised inside the block propagates instead of silently ending iteration. Like `#poll`, `#each` can also yield an `RdkafkaError` for a record that fails to build.
+- [Fix] Raise `ArgumentError` for `ShareConsumer#subscribe` without topics, which librdkafka otherwise treats as an unsubscribe without error, silently dropping the subscription.
+- [Fix] Set `ShareConsumer#name` from the OAuthBearer token refresh callback, so name-based OAuth callback routing works for share consumers.
+- [Fix] Update `ext/build_common.sh` (precompiled builds) to librdkafka `2.15.0` and its tarball checksum; it still pinned `2.14.1`, whose tarball is no longer vendored.
 - [Fix] Honor the `isolation_level:` argument to `Admin#list_offsets`, which was silently ignored. Ported from rdkafka-ruby (#983).
 - [Fix] Return correct results for every item of a multi-item topic, partition, group or ACL admin request, not just the first. Ported from rdkafka-ruby (#987).
-- [Maintenance] Fix a use-after-free in the multi-item admin integration test that read result-name pointers after librdkafka had destroyed the background event, which made it flaky on newer glibc (e.g. Debian trixie). Ported from rdkafka-ruby (#991).
-- [Maintenance] Stabilize consumer specs on slow CI runners: wait longer for a partition assignment, raise the `TestTopics.create` admin timeout (a timeout there also tripped the leaked-handle guard), give the non-blocking `poll_nb` message check a generous retry budget instead of a fixed ~2s, and let the long-running consumption spec drain its backlog instead of stopping at a fixed 60s. The assignment wait is tunable via `RDKAFKA_TEST_ASSIGNMENT_TIMEOUT`. Ported from rdkafka-ruby (#990).
+- [Note] Share consumers emit the regular consumer statistics through `statistics_callback`. librdkafka 2.15.0 has no share-specific metrics, so the `topics` section has no partition entries.
 
 ## 0.29.0 (2026-09-14)
 - [Enhancement] Bump librdkafka to `2.15.0` (staying on `2.15.0` rather than `2.15.1` so users hitting a regression in `2.15.1` have a stable fallback).
 
-## 0.28.2 (2026-09-11)
+## 0.28.2 (2026-09-12)
+- [Maintenance] Bump the bundled OpenSSL used by the precompiled builds to `3.5.8` (LTS). Ported from rdkafka-ruby (#972, Scott Francis).
 - [Fix] Close live clients before Ruby shutdown finalization, preventing a possible segfault on exit. Ported from rdkafka-ruby (#964, Alex Selesse).
 - [Fix] Destroy the native handle when client construction fails partway, so it is no longer orphaned. Ported from rdkafka-ruby (#964, Alex Selesse).
 - [Fix] Fix a double-free in `Consumer#poll_batch`/`#poll_batch_nb` that could abort the process when building a message raised a non-`RdkafkaError`. Ported from rdkafka-ruby (#973).
 - [Fix] Free native resources when `Admin#describe_configs`/`#incremental_alter_configs` raise while building their request (e.g. a non-String resource name). Ported from rdkafka-ruby (#973).
 - [Fix] Raise `ConfigError` instead of segfaulting when `Admin#describe_configs`/`#incremental_alter_configs` get an empty resource name or a negative resource type. Ported from rdkafka-ruby (#973).
 - [Fix] Stop the topic-partition metadata string from being freed while librdkafka still uses it. Ported from rdkafka-ruby (#969, Randy Stauner).
-- [Maintenance] Bump the bundled OpenSSL used by the precompiled builds to `3.5.8` (LTS). Ported from rdkafka-ruby (#972, Scott Francis).
 
-## v0.28.1 (2026-09-04)
+## 0.28.1 (2026-09-04)
 - [Enhancement] Add `Admin#delete_records` to delete messages in a partition up to a given offset, or all current data with `:end`. Ported from rdkafka-ruby (#956).
 - [Enhancement] Add `Admin#list_consumer_groups` for a cluster-wide listing of consumer groups. Ported from rdkafka-ruby (#955).
 - [Fix] Make `NativeKafka#close` fork-aware so a forked child no longer segfaults on exit. Handles record their creator pid and skip the native teardown in any other process.
 - [Fix] Stabilize the flaky partitions count cache statistics spec.
 - [Fix] Stabilize the `to_native_tpl` leak integration spec against RSS measurement noise.
 
-## v0.28.0 (2026-07-12)
+## 0.28.0 (2026-07-12)
 - [Enhancement] Bump librdkafka to `2.14.2` (maintenance release with bundled dependency CVE fixes and a fix for duplicate groups in `ListConsumerGroups`).
 - [Enhancement] Add `Consumer#metadata` and `Producer#metadata`, mirroring `Admin#metadata`, to fetch cluster metadata without a separate admin connection.
 - [Enhancement] Name the failing topic and partition in `RdkafkaError`s raised for per-partition `list_offsets` errors.
@@ -95,10 +159,10 @@
 ## 0.27.2 (2026-05-21)
 - [Enhancement] `poll_batch` and `poll_batch_nb` now return error events inline as `RdkafkaError` objects rather than raising on the first error. The return type is `Array<Message, RdkafkaError>` and callers are responsible for handling errors in the result.
 
-## 0.27.1 (2026-05-14)
+## 0.27.1 (2026-05-15)
 - [Fix] `poll_nb`, `poll_nb_each`, `poll_batch` and `poll_batch_nb` now raise `RdkafkaError` with `details` populated (`{topic:, partition:, offset:}`) when a message contains an error, consistent with `poll`.
 
-## 0.27.0 (2026-05-08)
+## 0.27.0 (2026-05-10)
 - [Feature] Add `Consumer#poll_batch(timeout_ms, max_items:)` and `Consumer#poll_batch_nb(timeout_ms, max_items:)` for batch message polling via `rd_kafka_consume_batch_queue` (from upstream).
 - [Enhancement] Bump librdkafka to `2.14.1`.
 - [Fix] Fix resource leak in `Admin#describe_configs` and `Admin#incremental_alter_configs` where `admin_options_ptr` and `queue_ptr` were not destroyed in the ensure block (from upstream).
@@ -126,8 +190,8 @@
 - [Fix] Prevent cascading test failures in admin specs when a single handle leaks into the registry (from upstream).
 
 ## 0.24.0 (2026-02-25)
-- **[Feature]** Add `Producer#queue_size` (and `#queue_length` alias) to report the number of messages waiting in the librdkafka output queue. Useful for monitoring producer backpressure, implementing custom flow control, debugging message delivery issues, and graceful shutdown logic.
-- **[Feature]** Add fiber scheduler API for integration with Ruby fiber schedulers (Falcon, Async) and custom event loops (from upstream). Expose `enable_queue_io_events` and `enable_background_queue_io_events` methods on `Consumer`, `Producer`, and `Admin`.
+- **[Feature]** Add `Producer#queue_size` (and the `#queue_length` alias) to report the number of messages waiting in the librdkafka output queue.
+- **[Feature]** Add `enable_queue_io_events` and `enable_background_queue_io_events` to `Consumer`, `Producer` and `Admin` for fiber schedulers (Falcon, Async) and custom event loops (from upstream).
 - **[Deprecation]** `AbstractHandle#wait` parameter `max_wait_timeout` (seconds) is deprecated in favor of `max_wait_timeout_ms` (milliseconds). The old parameter still works with backwards compatibility but will be removed in v1.0.0.
 - **[Deprecation]** `PartitionsCountCache` constructor parameter `ttl` (seconds) is deprecated in favor of `ttl_ms` (milliseconds). The old parameter still works with backwards compatibility but will be removed in v1.0.0.
 - [Enhancement] Add Ruby 4.0 support.
@@ -154,7 +218,7 @@
 - [Enhancement] Add `RdkafkaError.build_fatal` class method for centralized fatal error construction.
 - [Enhancement] Add comprehensive tests for fatal error handling including unit tests and integration tests.
 - [Enhancement] Add `RD_KAFKA_PARTITION_UA` constant for unassigned partition (-1).
-- [Enhancement] Replace magic numbers with named constants: use `RD_KAFKA_RESP_ERR_NO_ERROR` instead of `0` for error code checks (18 instances) and `RD_KAFKA_PARTITION_UA` instead of `-1` for partition values (9 instances) across the codebase for better code clarity and maintainability.
+- [Enhancement] Replace magic numbers with the `RD_KAFKA_RESP_ERR_NO_ERROR` and `RD_KAFKA_PARTITION_UA` constants.
 - [Enhancement] Add `Rdkafka::Testing` module for testing fatal error scenarios on both producers and consumers.
 - [Deprecated] `RdkafkaError.validate_fatal!` - use `validate!` with `client_ptr` parameter instead.
 
@@ -163,12 +227,12 @@
 - [Enhancement] Force lock FFI to 1.17.1 or higher to include critical bug fixes around GCC, write barriers, and thread restarts for forks.
 - [Fix] Fix for Core dump when providing extensions to oauthbearer_set_token (dssjoblom)
 
-## 0.22.2 (2025-10-09)
+## 0.22.2 (2025-10-10)
 - [Fix] Fix Github Action Ruby reference preventing non-compiled releases.
 
 ## 0.22.1 (2025-10-09)
 - [Enhancement] Optimize header processing to eliminate double hash lookups and method checking overhead.
-- [Enhancement] Optimize producer header processing with early returns and efficient array operations (69% faster for nil headers, 41% faster for empty headers, 12-32% faster when headers are present, with larger improvements for complex header scenarios).
+- [Enhancement] Speed up producer header processing (69% faster for nil headers, 41% for empty headers, 12-32% when headers are present).
 
 ## 0.22.0 (2025-09-26)
 - **[EOL]** Drop support for Ruby 3.1 to move forward with the fiber scheduler work.
@@ -185,7 +249,7 @@
 - [Enhancement] Introduce an integration suite layer for non RSpec specs execution.
 - [Fix] Add `json` gem as a dependency (was missing but used).
 
-## 0.20.1 (2025-07-17)
+## 0.20.1 (2025-07-18)
 - [Enhancement] Drastically increase number of platforms in the integration suite
 - [Fix] Support Ubuntu `22.04` and older Alpine precompiled versions
 - [Fix] FFI::DynamicLibrary.load_library': Could not open library

data/README.md

@@ -63,10 +63,12 @@ Contributions should generally be made to the upstream [rdkafka-ruby repository]
 
 | karafka-rdkafka | librdkafka | patches |
 |-|-|-|
+| 0.32.x (2026-11-04) | 2.16.0 (2026-10-06)  | yes |
+| 0.31.x (2026-10-12) | 2.16.0 (2026-10-06)  | yes |
 | 0.30.x (2026-09-25) | 2.15.1 (2026-09-09)  | yes |
 | 0.29.x (2026-09-14) | 2.15.0 (2026-06-30)  | yes |
 | 0.28.x (2026-07-12) | 2.14.2 (2026-06-03)  | yes |
-| 0.27.x (2026-05-08) | 2.14.1 (2026-04-15)  | yes |
+| 0.27.x (2026-05-10) | 2.14.1 (2026-04-15)  | yes |
 | 0.26.x (2026-04-11) | 2.14.0 (2026-04-01)  | yes |
 | 0.25.x (2026-04-02) | 2.13.2 (2026-03-02)  | yes |
 | 0.24.x (2026-02-25) | 2.13.0 (2026-01-05)  | yes |

data/docker-compose-ssl.yml

@@ -25,6 +25,7 @@ services:
       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL
       KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:9092,SSL://127.0.0.1:9093
       KAFKA_BROKER_ID: 1
+      KAFKA_BROKER_RACK: rack-1
       KAFKA_CONTROLLER_QUORUM_VOTERS: 1@127.0.0.1:9094
       ALLOW_PLAINTEXT_LISTENER: 'yes'
       KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'

data/docker-compose.yml

@@ -23,6 +23,7 @@ services:
       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
       KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:9092
       KAFKA_BROKER_ID: 1
+      KAFKA_BROKER_RACK: rack-1
       KAFKA_CONTROLLER_QUORUM_VOTERS: 1@127.0.0.1:9093
       ALLOW_PLAINTEXT_LISTENER: 'yes'
       KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'

data/karafka-rdkafka.gemspec

@@ -6,13 +6,13 @@ Gem::Specification.new do |gem|
   gem.authors = ["Thijs Cadier", "Maciej Mensfeld"]
   gem.email = ["contact@karafka.io"]
   gem.description = "Modern Kafka client library for Ruby based on librdkafka"
-  gem.summary = "The rdkafka gem is a modern Kafka client library for Ruby based on librdkafka. It wraps the production-ready C client using the ffi gem and targets Kafka 1.0+ and Ruby 2.7+."
+  gem.summary = "The rdkafka gem is a modern Kafka client library for Ruby based on librdkafka. It wraps the production-ready C client using the ffi gem and targets Kafka 1.0+ and Ruby 3.3+."
   gem.license = "MIT"
 
   gem.name = "karafka-rdkafka"
   gem.require_paths = ["lib"]
   gem.version = Rdkafka::VERSION
-  gem.required_ruby_version = ">= 3.2"
+  gem.required_ruby_version = ">= 3.3"
 
   files = `git ls-files`.split($\)
   files = files.reject do |file|
@@ -65,13 +65,14 @@ Gem::Specification.new do |gem|
     end
 
     gem.extensions = %w[ext/Rakefile]
+
+    gem.add_dependency "mini_portile2", "~> 2.6"
+    gem.add_dependency "rake", "> 12"
   end
 
   gem.add_dependency "ffi", "~> 1.17.1"
   gem.add_dependency "json", "> 2.0"
   gem.add_dependency "logger"
-  gem.add_dependency "mini_portile2", "~> 2.6"
-  gem.add_dependency "rake", "> 12"
 
   gem.metadata = {
     "funding_uri" => "https://karafka.io/#become-pro",

data/lib/rdkafka/admin/acl_binding_result.rb

@@ -18,30 +18,18 @@ module Rdkafka
         rd_kafka_error_pointer = Rdkafka::Bindings.rd_kafka_AclBinding_error(matching_acl)
         @result_error = Rdkafka::Bindings.rd_kafka_error_code(rd_kafka_error_pointer)
         error_string = Rdkafka::Bindings.rd_kafka_error_string(rd_kafka_error_pointer)
-
-        if error_string != FFI::Pointer::NULL
-          @error_string = error_string.read_string
-        end
+        @error_string = error_string.null? ? nil : error_string.read_string
 
         @matching_acl_resource_type = Rdkafka::Bindings.rd_kafka_AclBinding_restype(matching_acl)
         matching_acl_resource_name = Rdkafka::Bindings.rd_kafka_AclBinding_name(matching_acl)
-
-        if matching_acl_resource_name != FFI::Pointer::NULL
-          @matching_acl_resource_name = matching_acl_resource_name.read_string
-        end
+        @matching_acl_resource_name = matching_acl_resource_name.null? ? nil : matching_acl_resource_name.read_string
 
         @matching_acl_resource_pattern_type = Rdkafka::Bindings.rd_kafka_AclBinding_resource_pattern_type(matching_acl)
         matching_acl_principal = Rdkafka::Bindings.rd_kafka_AclBinding_principal(matching_acl)
-
-        if matching_acl_principal != FFI::Pointer::NULL
-          @matching_acl_principal = matching_acl_principal.read_string
-        end
+        @matching_acl_principal = matching_acl_principal.null? ? nil : matching_acl_principal.read_string
 
         matching_acl_host = Rdkafka::Bindings.rd_kafka_AclBinding_host(matching_acl)
-
-        if matching_acl_host != FFI::Pointer::NULL
-          @matching_acl_host = matching_acl_host.read_string
-        end
+        @matching_acl_host = matching_acl_host.null? ? nil : matching_acl_host.read_string
 
         @matching_acl_operation = Rdkafka::Bindings.rd_kafka_AclBinding_operation(matching_acl)
         @matching_acl_permission_type = Rdkafka::Bindings.rd_kafka_AclBinding_permission_type(matching_acl)

data/lib/rdkafka/admin/alter_consumer_group_offsets_report.rb

@@ -20,8 +20,8 @@ module Rdkafka
       # @param result_name [FFI::Pointer] pointer to the group name
       # @param partitions_ptr [FFI::Pointer] pointer to the per partition result list
       def initialize(error_string, result_name, partitions_ptr = FFI::Pointer::NULL)
-        @error_string = error_string.read_string if error_string != FFI::Pointer::NULL
-        @group_name = result_name.read_string if result_name != FFI::Pointer::NULL
+        @error_string = error_string.null? ? nil : error_string.read_string
+        @group_name = result_name.null? ? nil : result_name.read_string
         @partitions = extract_partitions(partitions_ptr)
       end
 

data/lib/rdkafka/admin/create_acl_report.rb

@@ -17,9 +17,7 @@ module Rdkafka
       # @param rdkafka_response_string [FFI::Pointer] pointer to response string
       def initialize(rdkafka_response:, rdkafka_response_string:)
         @rdkafka_response = rdkafka_response
-        if rdkafka_response_string != FFI::Pointer::NULL
-          @rdkafka_response_string = rdkafka_response_string.read_string
-        end
+        @rdkafka_response_string = rdkafka_response_string.null? ? nil : rdkafka_response_string.read_string
       end
     end
   end

data/lib/rdkafka/admin/create_topic_report.rb

@@ -15,12 +15,8 @@ module Rdkafka
       # @param error_string [FFI::Pointer] pointer to error string
       # @param result_name [FFI::Pointer] pointer to topic name
       def initialize(error_string, result_name)
-        if error_string != FFI::Pointer::NULL
-          @error_string = error_string.read_string
-        end
-        if result_name != FFI::Pointer::NULL
-          @result_name = result_name.read_string
-        end
+        @error_string = error_string.null? ? nil : error_string.read_string
+        @result_name = result_name.null? ? nil : result_name.read_string
       end
     end
   end

data/lib/rdkafka/admin/delete_consumer_group_offsets_report.rb

@@ -20,8 +20,8 @@ module Rdkafka
       # @param result_name [FFI::Pointer] pointer to the group name
       # @param partitions_ptr [FFI::Pointer] pointer to the per partition result list
       def initialize(error_string, result_name, partitions_ptr = FFI::Pointer::NULL)
-        @error_string = error_string.read_string if error_string != FFI::Pointer::NULL
-        @group_name = result_name.read_string if result_name != FFI::Pointer::NULL
+        @error_string = error_string.null? ? nil : error_string.read_string
+        @group_name = result_name.null? ? nil : result_name.read_string
         @partitions = extract_partitions(partitions_ptr)
       end
 

data/lib/rdkafka/admin/delete_groups_report.rb

@@ -15,12 +15,8 @@ module Rdkafka
       # @param error_string [FFI::Pointer] pointer to error string
       # @param result_name [FFI::Pointer] pointer to group name
       def initialize(error_string, result_name)
-        if error_string != FFI::Pointer::NULL
-          @error_string = error_string.read_string
-        end
-        if result_name != FFI::Pointer::NULL
-          @result_name = result_name.read_string
-        end
+        @error_string = error_string.null? ? nil : error_string.read_string
+        @result_name = result_name.null? ? nil : result_name.read_string
       end
     end
   end

data/lib/rdkafka/admin/delete_topic_report.rb

@@ -15,12 +15,8 @@ module Rdkafka
       # @param error_string [FFI::Pointer] pointer to error string
       # @param result_name [FFI::Pointer] pointer to topic name
       def initialize(error_string, result_name)
-        if error_string != FFI::Pointer::NULL
-          @error_string = error_string.read_string
-        end
-        if result_name != FFI::Pointer::NULL
-          @result_name = result_name.read_string
-        end
+        @error_string = error_string.null? ? nil : error_string.read_string
+        @result_name = result_name.null? ? nil : result_name.read_string
       end
     end
   end

data/lib/rdkafka/admin/describe_cluster_handle.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Handle for describe cluster operation
+    class DescribeClusterHandle < AbstractHandle
+      layout :pending, :bool,
+        :response, :int
+
+      # @return [String] the name of the operation
+      def operation_name
+        "describe cluster"
+      end
+
+      # @return [DescribeClusterReport] report prepared by the background event callback, with
+      #   the cluster id, controller and broker nodes.
+      def create_result
+        prepared_result
+      end
+
+      # Raises an error if the operation failed
+      # @raise [RdkafkaError]
+      def raise_error
+        raise RdkafkaError.new(
+          self[:response],
+          broker_message: broker_message
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/describe_cluster_report.rb

@@ -0,0 +1,51 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Report for describe cluster operation result
+    class DescribeClusterReport
+      include DescribeResultParsing
+
+      # @return [String, nil] the cluster id
+      attr_reader :cluster_id
+
+      # The current controller node, as a hash with `:id` [Integer], `:host` [String],
+      # `:port` [Integer] and `:rack` [String, nil]
+      # @return [Hash, nil]
+      attr_reader :controller
+
+      # Broker nodes of the cluster, each in the same shape as {#controller}
+      # @return [Array<Hash>]
+      attr_reader :nodes
+
+      # Operations the client is authorized to perform on the cluster, as
+      # `Bindings::RD_KAFKA_ACL_OPERATION_*` codes. `nil` unless requested with
+      # `include_authorized_operations: true`.
+      # @return [Array<Integer>, nil]
+      attr_reader :authorized_operations
+
+      # @param result_ptr [FFI::Pointer] pointer to the `rd_kafka_DescribeCluster_result_t`
+      def initialize(result_ptr)
+        @nodes = []
+        @cluster_id = nil
+        @controller = nil
+        @authorized_operations = nil
+
+        return if result_ptr.null?
+
+        cluster_id_ptr = Bindings.rd_kafka_DescribeCluster_result_cluster_id(result_ptr)
+        @cluster_id = cluster_id_ptr.null? ? nil : cluster_id_ptr.read_string
+        @controller = extract_node(Bindings.rd_kafka_DescribeCluster_result_controller(result_ptr))
+
+        count_ptr = FFI::MemoryPointer.new(:size_t)
+        @nodes = extract_nodes(Bindings.rd_kafka_DescribeCluster_result_nodes(result_ptr, count_ptr), count_ptr)
+
+        count_ptr = FFI::MemoryPointer.new(:size_t)
+        @authorized_operations = extract_authorized_operations(
+          Bindings.rd_kafka_DescribeCluster_result_authorized_operations(result_ptr, count_ptr),
+          count_ptr
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/describe_consumer_groups_handle.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Handle for describe consumer groups operation
+    class DescribeConsumerGroupsHandle < AbstractHandle
+      layout :pending, :bool,
+        :response, :int
+
+      # @return [String] the name of the operation
+      def operation_name
+        "describe consumer groups"
+      end
+
+      # @return [DescribeConsumerGroupsReport] report prepared by the background event callback,
+      #   with the described consumer groups.
+      def create_result
+        prepared_result
+      end
+
+      # Raises an error if the operation failed
+      # @raise [RdkafkaError]
+      def raise_error
+        raise RdkafkaError.new(
+          self[:response],
+          broker_message: broker_message
+        )
+      end
+    end
+  end
+end