karafka-rdkafka 0.28.0-aarch64-linux-gnu → 0.28.1-aarch64-linux-gnu

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ed54b417d876a61e01603aebf12e29ec1b56a3f43cbae330d19aad1222f86d58
-  data.tar.gz: a4925b7300fc077003ad5a588964902507a77152cc057e48cbf49beee5966522
+  metadata.gz: cb6dd6580bf5c14ddceaa2ccd38deaad34cf4e6e81e994c82df73e62fb831f77
+  data.tar.gz: c26f74ea68b3c14ace182f7f8b2da1184cceb59d6063b044a7efb3a84a022eef
 SHA512:
-  metadata.gz: 5fe6ad75e784238e3781fe85eaff6bf36fdc2f4b7f5128c167a730d48607d17c10312ce3f5b9cac88ab47d0e841e72a3536cd5739bfa6b7a4c7de23934cec59d
-  data.tar.gz: 78df953bd672fbf55a97ffa3372b1ccd3e2b8cd19d748d18f159ce5f92298101d40896e3e93721841f8d6e5c0e41f96c71f6503bd80df90760024cdfa853bc2e
+  metadata.gz: 6b954711968c6acaa5beffa2995a33262bef8e2297b18eb3d9e828dd0be43818bfa5be73a7a20bfe13a8d19ee08ec4436e38e7b63a69948f3b85b7d06843d5db
+  data.tar.gz: f2003f4f608143b78dd9caab59a30d2fd96b5ed1a280ea9d1388cb4d5ff386d26655f2b0b3e15ec16b450b7df23118cfae1f1f99f229139e19243cd6bb7f3d5c

data/CHANGELOG.md

@@ -1,43 +1,50 @@
 # Rdkafka Changelog
 
-## v0.28.0
+## v0.28.1 (2026-09-04)
+- [Enhancement] Add `Admin#delete_records`, wrapping librdkafka's `DeleteRecords` admin API. Deletes all messages in a partition up to (but not including) a given offset - accepts an integer offset or `:end` to clear all current data. Ported from rdkafka-ruby (#956).
+- [Enhancement] Add `Admin#list_consumer_groups`, wrapping librdkafka's `ListConsumerGroups` admin API. Returns a cluster-wide listing of consumer groups (`group_id`, `is_simple_consumer_group`, state) plus any per-broker errors. Ported from rdkafka-ruby (#955).
+- [Fix] Make `NativeKafka#close` fork-aware so a forked child no longer segfaults on exit. Handles record their creator pid and skip the native teardown in any other process.
+- [Fix] Stabilize the flaky partitions count cache statistics spec.
+- [Fix] Stabilize the `to_native_tpl` leak integration spec against RSS measurement noise.
+
+## v0.28.0 (2026-07-12)
 - [Enhancement] Bump librdkafka to `2.14.2`. Maintenance release: fixes duplicate groups in `ListConsumerGroups` when multiple brokers return the same group, a data race in timers, and bumps bundled OpenSSL/libcurl/zstd/zlib/cJSON dependencies (several CVEs).
 - [Enhancement] Add `Consumer#metadata` and `Producer#metadata`, mirroring `Admin#metadata`, so cluster/topic metadata can be fetched from an existing consumer or producer handle without opening a dedicated admin connection.
 - [Enhancement] Name the failing partition and topic in the `RdkafkaError` raised for per-partition `list_offsets` errors (previously a bare error code), preserving the per-partition context the pre-batching `Consumer#lag` watermark errors carried.
-- [Enhancement] Add `Consumer#list_offsets`, mirroring `Admin#list_offsets`, so batched offset queries (one `ListOffsets` request carrying all requested partitions, fanned out to the partition leaders by librdkafka) can be issued on an existing consumer handle without opening a dedicated admin connection, and rebuild `Consumer#lag` on top of it: lag is now computed from a single batched end-offsets query instead of one blocking `query_watermark_offsets` broker roundtrip per partition. To receive the results, consumer clients now register the librdkafka background event callback, so librdkafka spawns its internally-managed background thread for consumers as well. `Consumer#lag` keeps its exact pre-batching semantics: it forwards the consumer's configured `isolation.level` to the batched query (librdkafka resolves the per-partition watermark query with that level, so end offsets stay LSO-based for the default `read_committed`), surfaces a timeout of the batched query as an `RdkafkaError` (`timed_out`), and now also raises `ClosedConsumerError` when called on a closed consumer.
-- [Enhancement] Extract the admin background-event result handlers into one class per operation under `lib/rdkafka/callbacks/` (`CreateTopicHandler`, `DescribeConfigsHandler`, etc., all subclasses of `Callbacks::BaseHandler`). `Callbacks::BackgroundEventCallback` is now a thin dispatcher that maps the event type to its handler and destroys the event. Purely internal reorganization (`Rdkafka::Callbacks` is private) with no behavior or API change.
-- [Enhancement] Expose `replicas` and `isrs` (in-sync replica broker ids) on each partition in topic metadata (`Metadata#topics` partition hashes). The base struct `#to_h` skips FFI pointer members, so these two arrays were dropped entirely and the partition replica assignment was unavailable to callers (e.g. for planning replication-factor changes). `PartitionMetadata#to_h` now dereferences both pointers into arrays of broker ids.
-- [Enhancement] Reuse per-thread scratch pointers in `Consumer::Headers.from_native` instead of allocating them for every consumed message. Previously each message paid one native pointer allocation just to check for headers and three more when headers were present; now the scratch pointers are allocated once per thread/fiber and reused, removing all per-message native scratch allocations from the consumer hot path.
-- [Enhancement] Remove the unused `DeliveryHandle` `:topic_name` struct field and the per-message allocation that populated it. The delivery callback copied the topic name into a native `FFI::MemoryPointer` on every delivered message, retained for the lifetime of the handle, yet nothing ever read it: the topic is already available via `DeliveryHandle#topic` (a Ruby attribute set during `produce`) and `DeliveryReport#topic_name`, both of which work exactly as before.
-- [Fix] Stop `poll_batch`/`poll_batch_nb` from discarding a whole batch when one message fails to build. If building a `Consumer::Message` raised (e.g. a header-read `RdkafkaError`), the exception propagated out of the batch loop and the `ensure` destroyed every remaining message, so all the messages already built in that batch were lost even though their offsets had been stored - permanent silent loss of up to `max_items` messages. A per-message build error is now surfaced inline as an `RdkafkaError` in the returned array (the same way native error events already are), so the rest of the batch is preserved and the failure is visible to the caller. (Single-message `poll`/`poll_nb` still raise as before.)
-- [Fix] Add the missing `closed_consumer_check` to `Consumer#position`. Every sibling offset method guards against a closed consumer, but `position` did not: with an explicit `list` it raised `ClosedInnerError` instead of `ClosedConsumerError`, and with no `list` it raised `ClosedConsumerError` naming `assignment` rather than `position`. It now raises a consistent `ClosedConsumerError` for `position`.
-- [Fix] Stop leaking the native `rd_kafka_topic_conf_t` in `Producer#set_topic_config` when a per-topic config value is rejected. The conf is built with `rd_kafka_topic_conf_new` and only handed to `rd_kafka_topic_new` afterwards; if `rd_kafka_topic_conf_set` returned a non-`:config_ok` result the method raised before `rd_kafka_topic_new` was ever called, so librdkafka never took ownership of the conf and it leaked (repeatable per produce with an invalid `topic_config:`). The conf is now destroyed before raising. The `rd_kafka_topic_new` path is unchanged: librdkafka frees the conf there on both success and failure, so it is never destroyed by us.
-- [Fix] Raise instead of silently dropping a rejected `incremental_alter_configs` entry. `rd_kafka_ConfigResource_add_incremental_config` returns an `rd_kafka_error_t` for an invalid op_type, an empty/nil name, or a nil value on a non-delete op; the result was ignored, so the entry was dropped, the alter request still reported success, and the error object leaked. The error is now surfaced as an `RdkafkaError` (raised before the request is sent) and the native error object is freed.
-- [Fix] Let `PartitionsCountCache` adopt a lower partition count once the cached entry has expired. The cache prioritizes higher counts (partition counts only grow during normal operation), but it did so unconditionally: after the TTL expired it fetched the true lower count, discarded it, and re-armed the TTL on the stale higher count - permanently. A topic recreated with fewer partitions then made `produce` (with a partition key) fail with `unknown_partition` for the dropped partitions until process restart. A lower value is now adopted on the first refresh after expiry, while still being ignored within the TTL window so a transient or racy lower read cannot clobber a correct higher count.
-- [Fix] Make the `Consumer` GC finalizer close the consumer and destroy its consumer queue, not just the native client. The finalizer used the generic `NativeKafka` one, which went straight to `rd_kafka_destroy` and skipped `rd_kafka_consumer_close` and `rd_kafka_queue_destroy`. A consumer that used `poll_batch` (which takes a consumer-queue reference) and was then garbage-collected without an explicit `close` left that reference dangling, which could make `rd_kafka_destroy` block inside the finalizer (process hang at GC/shutdown) or leak the handle. The consumer now installs its own finalizer that mirrors `#close` (the queue pointer is shared with the finalizer via a holder so it never captures the consumer and prevents collection).
-- [Fix] Stop admin operations from leaking when their arguments are rejected. `describe_configs` and `incremental_alter_configs` allocated the background queue and AdminOptions and registered their handle before parsing input, so a missing key raised a `KeyError` out of the building loop that orphaned the handle in the process-global registry forever (and leaked the queue, AdminOptions and any `ConfigResource`s already built); `list_offsets` likewise allocated its native topic-partition list before validating the offset specs. All three now parse and validate input up front, so a bad argument raises with nothing allocated.
-- [Fix] Destroy the native topic-partition list in `TopicPartitionList#to_native_tpl` when population fails partway. The list is allocated with `rd_kafka_topic_partition_list_new` and only handed back (for the caller to destroy) on success; if building it raised - e.g. a non-string metadata value or an offset FFI cannot coerce to int64 - the half-built list leaked, since destruction is fully manual (the previous doc comment claiming GC handled it was wrong). It is now destroyed before the error propagates.
-- [Fix] Allocate the admin result-count out-parameter as `:size_t` instead of `:int32`. Every librdkafka `rd_kafka_*_result_*(result, size_t *cntp)` accessor writes a full native `size_t` (8 bytes on 64-bit), but the count pointer was a 4-byte `FFI::MemoryPointer.new(:int32)` across the create/delete topic, create partitions, create/delete ACL, describe/incremental-alter configs and config-synonyms paths - a 4-byte heap overflow on every admin result parse (benign on little-endian, where the low word still reads the correct count, but undefined behavior). Now uses `:size_t`, matching the already-correct list-offsets and `get_err_descs` paths.
-- [Fix] Destroy admin API background events after processing. librdkafka requires the application to destroy each background event, but `rd_kafka_event_destroy` was never called (nor even bound), so every admin operation leaked its entire result event with all result arrays and strings. Reports are now built inside the callback (copying event-owned memory into Ruby objects) before the event is destroyed, and `DescribeConfigsReport`/`IncrementalAlterConfigsReport` no longer destroy the event-owned ConfigResource array, which also fixes a double free on repeated `wait` calls on the same handle. As part of this, the internal FFI struct fields on admin operation handles (e.g. `handle[:error_string]`, `handle[:result_name]`, `handle[:config_entries]`, `handle[:response_string]`, `handle[:matching_acls]`) were removed; they were never part of the public API (use `handle.wait` and the returned report objects, whose interfaces are unchanged).
-- [Fix] Resolve admin operation handles from the event error when an admin operation fails at the operation level (e.g. brokers unreachable, or the client closed with the request in flight). librdkafka delivers such failures as a result event with the error set and an empty results array, but the create topic, delete topic, create partitions, delete groups, create ACL and delete ACL handlers indexed `results[0]` unconditionally. That raised inside the background event callback, so the handle was never unlocked and `wait` blocked until its own timeout and raised `WaitTimeoutError`, discarding the real error. These handlers now check the event error first and resolve the handle with the actual error code (the describe configs, incremental alter configs, describe ACL and list offsets handlers already did).
-- [Fix] Stop leaking the native `rd_kafka_conf_t` when client creation fails. `native_config` raised `ConfigError` mid-build (e.g. on an invalid option) without destroying the conf, and `native_kafka` raised `ClientCreationError` on a null `rd_kafka_new` without destroying it either. Both paths now call `rd_kafka_conf_destroy` before re-raising (librdkafka keeps app ownership of the conf on `rd_kafka_new` failure, so this is safe). Multi-KB leak per failed creation, relevant for supervisors retrying client creation on transient SASL/SSL misconfig.
-- [Fix] Stop `Metadata` from leaking the native metadata struct (and a topic reference) on every retried fetch. `retry` restarts the `begin` block without running its `ensure`, so each retried attempt reassigned the pointers and only the last attempt's `rd_kafka_metadata` struct was ever destroyed; up to `METADATA_MAX_RETRIES` whole-cluster structs leaked per call (the `leader_not_available` case is routine during topic creation/leader election). Each attempt now frees its own native resources, and a failed fetch that never allocated a struct no longer calls `rd_kafka_metadata_destroy` on a NULL pointer.
-- [Fix] Free the librdkafka-allocated string in `Consumer#cluster_id` and `Consumer#member_id` (previously copied via a `:string` binding but never freed) and fix the `rd_kafka_clusterid` arity to pass `timeout_ms`. `Consumer#cluster_id` now accepts a `timeout_ms` (default `Defaults::CONSUMER_CLUSTER_ID_TIMEOUT_MS`).
-- [Fix] Guard the message delivery callback so a raising user `delivery_callback` can no longer skip the handle unlock or crash the producer. `DeliveryCallback` invoked the user callback and only then unlocked the handle, with no rescue; if the callback raised, the handle stayed pending (so `wait` blocked until its timeout and raised `WaitTimeoutError` for a message that was actually delivered) and the exception unwound out of the FFI callback on librdkafka's polling thread (`abort_on_exception = true`), taking down the whole process. The user callback is now wrapped so exceptions are logged and swallowed (matching the rebalance callback) and the handle is always unlocked in an `ensure`.
-- [Fix] Stop `Producer#produce` from orphaning the delivery handle in the process-global registry when it fails after registering it. The handle was only removed on a non-zero `rd_kafka_producev` return, so any exception between registration and that check (a concurrent `close` making `with_inner` raise `ClosedInnerError`, or a header value whose `#to_s` raises) leaked the handle forever - it survives producer close and accumulates in apps that recreate/close producers. `produce` now removes the handle on any such failure before re-raising.
-- [Fix] Attach `rd_kafka_query_watermark_offsets` with `blocking: true` so it releases the GVL during its broker round-trip. It was the only synchronous network call bound without the flag, so `Consumer#query_watermark_offsets` (and `Consumer#lag`, which calls it once per partition) froze every other Ruby thread in the process - including producer polling threads - for up to `timeout_ms`. Matches the neighboring `rd_kafka_offsets_for_times` binding.
-- [Fix] Stabilize the flaky `Consumer#lag` "calculates the consumer lag" spec on overloaded CI. The manual `consumer.commit` could raise `no_offset` when the default 5s background auto-commit had already committed the stored offsets, or when the auto offset store had not yet caught up with `poll`. The spec now raises `auto.commit.interval.ms` to 60s (matching the existing `#seek`/pause specs) and lets the offset store settle before committing. Backported from rdkafka-ruby (#912).
-- [Fix] Fix the NULL background-queue cleanup branches in `Admin#delete_group`, `Admin#delete_acl` and `Admin#describe_acl`, which referenced undefined local variables (`delete_topic_ptr`/`new_acl_ptr`). When `rd_kafka_queue_get_background` returned NULL, those branches raised a `NameError` instead of the intended `ConfigError` and leaked the already-allocated native request object (the `rd_kafka_DeleteGroup_t` / ACL binding filter), and `delete_group` additionally called the wrong destructor. They now destroy the correct object and raise `ConfigError`.
-- [Fix] Forward `broker_message` and `instance_name` through `RdkafkaError.build`. On the `rd_kafka_error_t` pointer path `build` called `build_from_c` without passing either, so a caller-supplied broker message was discarded (falling back to `rd_kafka_err2str`) and the instance name was lost; the `Bindings::Message` path also dropped `instance_name`. Both are now forwarded, and `build_from_c` accepts `instance_name`.
-- [Fix] Synchronize `AbstractHandle::REGISTRY` mutations. The handle registry is a plain Hash mutated from producing/consuming threads and the background polling thread (which removes handles from FFI callbacks). That is effectively safe on MRI under the GVL but not on JRuby, where a lost write could leave a handle unregistered (never unlocked, so `wait` times out for a delivered message) or never removed (permanent leak). `register`/`remove` now guard the Hash with a mutex.
-- [Fix] Stop the `Metadata` retry loop from clobbering the request timeout and blocking for minutes. Each retry overwrote the per-request timeout with the exponential backoff value, so the first retries ran with a far-too-short request timeout (e.g. 200ms vs the 2,000ms default, near-guaranteeing another timeout) while later ones inflated it to ~100s; the cumulative sleeps alone reached ~204s. The request timeout is now left unchanged across retries, the per-attempt backoff is capped at `Defaults::METADATA_RETRY_BACKOFF_MAX_MS` (1,000ms), and the loop is bounded by a wall-clock budget (`Defaults::METADATA_RETRY_BUDGET_MS`, 5,000ms) after a floor of `Defaults::METADATA_MIN_ATTEMPTS` (3) tries. This keeps a synchronous metadata fetch from blocking for minutes while still giving a slow broker a few attempts: fast-recovering clusters retry quickly within the budget, an unresponsive one fails in ~5s (a bit over if its requests each consume the full timeout), and at least 3 attempts always happen.
-- [Fix] Cache the partition count for a missing topic so `produce` with a `partition_key` to a not-yet-created topic no longer runs a blocking metadata query on every message. The `unknown_topic_or_part` result was resolved to `RD_KAFKA_PARTITION_UA` outside the partition-count cache, so nothing was stored and each call performed a fresh synchronous metadata RPC (the promised negative cache did not exist). The miss is now resolved inside the cache block, so it is cached like any other value and reused for the cache TTL.
+- [Enhancement] Add `Consumer#list_offsets`, mirroring `Admin#list_offsets`, so offsets can be queried on an existing consumer handle without a dedicated admin connection, and rebuild `Consumer#lag` on top of it as one batched query instead of a roundtrip per partition.
+- [Enhancement] Extract the admin background-event result handlers into one class per operation under `lib/rdkafka/callbacks/`. Internal reorganization with no API or behavior change.
+- [Enhancement] Expose `replicas` and `isrs` (in-sync replica broker ids) on each partition in topic metadata; both were previously dropped from the `Metadata#topics` partition hashes.
+- [Enhancement] Reuse per-thread scratch pointers in `Consumer::Headers.from_native`, removing the per-message native allocations from the consumer hot path.
+- [Enhancement] Remove the unused `DeliveryHandle` `:topic_name` struct field and its per-message allocation. Use `DeliveryHandle#topic` or `DeliveryReport#topic_name`, both unchanged.
+- [Fix] Stop `poll_batch`/`poll_batch_nb` from discarding a whole batch when one message fails to build. The failure is now surfaced inline as an `RdkafkaError` in the returned array, preserving the rest of the batch.
+- [Fix] Add the missing `closed_consumer_check` to `Consumer#position`, so it raises a consistent `ClosedConsumerError` like every sibling offset method.
+- [Fix] Stop leaking the native `rd_kafka_topic_conf_t` in `Producer#set_topic_config` when a per-topic config value is rejected.
+- [Fix] Raise instead of silently dropping a rejected `incremental_alter_configs` entry, which previously left the alter request reporting success.
+- [Fix] Let `PartitionsCountCache` adopt a lower partition count once the cached entry has expired, so a topic recreated with fewer partitions no longer fails `produce` until process restart.
+- [Fix] Make the `Consumer` GC finalizer close the consumer and destroy its consumer queue, not just the native client, preventing a hang or leak when a consumer is collected without an explicit `close`.
+- [Fix] Stop `describe_configs`, `incremental_alter_configs` and `list_offsets` from leaking handles and native resources when their arguments are rejected; all three now validate input before allocating.
+- [Fix] Destroy the native topic-partition list in `TopicPartitionList#to_native_tpl` when population fails partway, which previously leaked the half-built list.
+- [Fix] Allocate the admin result-count out-parameter as `:size_t` instead of `:int32`, fixing a 4-byte overflow on every admin result parse.
+- [Fix] Destroy admin API background events after processing; they were never destroyed, so every admin operation leaked its whole result event. The internal FFI struct fields on admin handles were removed as part of this - use `handle.wait` and the returned report objects, whose interfaces are unchanged.
+- [Fix] Resolve admin operation handles from the event error when an operation fails at the operation level (e.g. brokers unreachable), instead of blocking until `wait` timed out and discarding the real error.
+- [Fix] Stop leaking the native `rd_kafka_conf_t` when client creation fails, a multi-KB leak per failed attempt for supervisors retrying on transient SASL/SSL misconfiguration.
+- [Fix] Stop `Metadata` from leaking the native metadata struct on every retried fetch; each attempt now frees its own native resources.
+- [Fix] Free the librdkafka-allocated string in `Consumer#cluster_id` and `Consumer#member_id`, and fix the `rd_kafka_clusterid` arity. `Consumer#cluster_id` now accepts a `timeout_ms`.
+- [Fix] Guard the message delivery callback so a raising user `delivery_callback` can no longer leave the handle locked or crash the producer; exceptions are now logged and swallowed.
+- [Fix] Stop `Producer#produce` from orphaning the delivery handle in the process-global registry when it fails after registering it.
+- [Fix] Attach `rd_kafka_query_watermark_offsets` with `blocking: true` so it releases the GVL; it previously froze every other Ruby thread for up to `timeout_ms`.
+- [Fix] Stabilize the flaky `Consumer#lag` spec on overloaded CI. Backported from rdkafka-ruby (#912).
+- [Fix] Fix the NULL background-queue cleanup branches in `Admin#delete_group`, `Admin#delete_acl` and `Admin#describe_acl`, which raised a `NameError` and leaked the native request object instead of raising `ConfigError`.
+- [Fix] Forward `broker_message` and `instance_name` through `RdkafkaError.build`; both were previously discarded on the `rd_kafka_error_t` pointer path.
+- [Fix] Synchronize `AbstractHandle::REGISTRY` mutations with a mutex, which could otherwise lose a write on JRuby and leave a handle unregistered or leaked.
+- [Fix] Stop the `Metadata` retry loop from clobbering the request timeout and blocking for minutes. The request timeout is left unchanged across retries, the backoff is capped, and the loop is bounded by a ~5s wall-clock budget after a floor of 3 attempts.
+- [Fix] Cache the partition count for a missing topic, so `produce` with a `partition_key` to a not-yet-created topic no longer runs a blocking metadata query on every message.
 
 ## 0.27.2 (2026-05-21)
 - [Enhancement] `poll_batch` and `poll_batch_nb` now return error events inline as `RdkafkaError` objects rather than raising on the first error. The return type is `Array<Message, RdkafkaError>` and callers are responsible for handling errors in the result.
 
 ## 0.27.1 (2026-05-14)
-- [Fix] `poll_nb`, `poll_nb_each`, `poll_batch`, and `poll_batch_nb` now raise `RdkafkaError` with `details` populated (`{topic:, partition:, offset:}`) when a message contains an error (e.g. `:partition_eof`). Previously these methods raised via `RdkafkaError.new(code)`, discarding the native message struct context. They now use `RdkafkaError.validate!(native_message, client_ptr: inner)`, consistent with `poll`.
+- [Fix] `poll_nb`, `poll_nb_each`, `poll_batch` and `poll_batch_nb` now raise `RdkafkaError` with `details` populated (`{topic:, partition:, offset:}`) when a message contains an error, consistent with `poll`.
 
 ## 0.27.0 (2026-05-08)
 - [Feature] Add `Consumer#poll_batch(timeout_ms, max_items:)` and `Consumer#poll_batch_nb(timeout_ms, max_items:)` for batch message polling via `rd_kafka_consume_batch_queue` (from upstream).

data/README.md

@@ -1,6 +1,6 @@
 # Karafka-Rdkafka
 
-[![Build Status](https://github.com/karafka/karafka-rdkafka/actions/workflows/ci_linux_ubuntu_x86_64_gnu.yml/badge.svg)](https://github.com/karafka/karafka-rdkafka/actions/workflows/ci_linux_x86_64_gnu.yml)
+[![Build Status](https://github.com/karafka/karafka-rdkafka/actions/workflows/ci_linux_ubuntu_x86_64_gnu.yml/badge.svg)](https://github.com/karafka/karafka-rdkafka/actions/workflows/ci_linux_ubuntu_x86_64_gnu.yml)
 [![Gem Version](https://badge.fury.io/rb/karafka-rdkafka.svg)](https://badge.fury.io/rb/karafka-rdkafka)
 [![Join the chat at https://slack.karafka.io](https://raw.githubusercontent.com/karafka/misc/master/slack.svg)](https://slack.karafka.io)
 
@@ -61,14 +61,14 @@ Contributions should generally be made to the upstream [rdkafka-ruby repository]
 
 ## Versions
 
-| rdkafka-ruby | librdkafka | patches |
+| karafka-rdkafka | librdkafka | patches |
 |-|-|-|
 | 0.28.x (2026-07-12) | 2.14.2 (2026-06-03)  | yes |
 | 0.27.x (2026-05-08) | 2.14.1 (2026-04-15)  | yes |
 | 0.26.x (2026-04-11) | 2.14.0 (2026-04-01)  | yes |
 | 0.25.x (2026-04-02) | 2.13.2 (2026-03-02)  | yes |
 | 0.24.x (2026-02-25) | 2.13.0 (2026-01-05)  | yes |
-| 0.23.x (2025-11-01) | 2.12.1 (2025-10-16)  | yes |
+| 0.23.x (2025-11-01) | 2.12.1 (2025-10-21)  | yes |
 | 0.22.x (2025-09-26) | 2.11.1 (2025-08-18)  | yes |
 | 0.21.x (2025-08-18) | 2.11.0 (2025-07-03)  | yes |
 | 0.20.x (2025-07-17) | 2.8.0  (2025-01-07)  | yes |

data/lib/rdkafka/admin/create_acl_report.rb

@@ -4,7 +4,8 @@ module Rdkafka
   class Admin
     # Report for create ACL operation result
     class CreateAclReport
-      # Upon successful creation of Acl RD_KAFKA_RESP_ERR_NO_ERROR - 0 is returned as rdkafka_response
+      # Upon successful creation of Acl RD_KAFKA_RESP_ERR_NO_ERROR - 0 is returned as
+      # rdkafka_response
       # @return [Integer]
       attr_reader :rdkafka_response
 

data/lib/rdkafka/admin/delete_records_handle.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Handle for delete records operation
+    class DeleteRecordsHandle < AbstractHandle
+      layout :pending, :bool,
+        :response, :int
+
+      # @return [String] the name of the operation
+      def operation_name
+        "delete records"
+      end
+
+      # @return [DeleteRecordsReport] report prepared by the background event callback, with the
+      #   post-deletion low-watermark offsets (or per-partition errors)
+      def create_result
+        prepared_result
+      end
+
+      # Raises an error if the operation failed
+      # @raise [RdkafkaError]
+      def raise_error
+        raise RdkafkaError.new(
+          self[:response],
+          broker_message: broker_message
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/delete_records_report.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Report for delete records operation result
+    class DeleteRecordsReport
+      # Per-partition results. Each partition's `offset` is the post-deletion low-watermark (the
+      # smallest available offset of all live replicas) and its `err` carries the per-partition
+      # error code, if deletion failed for that partition.
+      # @return [Rdkafka::Consumer::TopicPartitionList]
+      attr_reader :offsets
+
+      # @param result_ptr [FFI::Pointer] pointer to the `rd_kafka_DeleteRecords_result_t`
+      def initialize(result_ptr)
+        @offsets = Rdkafka::Consumer::TopicPartitionList.new
+
+        return if result_ptr.null?
+
+        native_tpl = Bindings.rd_kafka_DeleteRecords_result_offsets(result_ptr)
+
+        @offsets = Rdkafka::Consumer::TopicPartitionList.from_native_tpl(native_tpl)
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/describe_acl_report.rb

@@ -4,7 +4,8 @@ module Rdkafka
   class Admin
     # Report for describe ACL operation result
     class DescribeAclReport
-      # acls that exists in the cluster for the resource_type, resource_name and pattern_type filters provided in the request.
+      # acls that exists in the cluster for the resource_type, resource_name and pattern_type
+      # filters provided in the request.
       # @return [Rdkafka::Bindings::AclBindingResult] array of matching acls.
       attr_reader :acls
 

data/lib/rdkafka/admin/list_consumer_groups_handle.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Handle for list consumer groups operation
+    class ListConsumerGroupsHandle < AbstractHandle
+      layout :pending, :bool,
+        :response, :int
+
+      # @return [String] the name of the operation
+      def operation_name
+        "list consumer groups"
+      end
+
+      # @return [ListConsumerGroupsReport] report prepared by the background event callback, with
+      #   the listed consumer groups.
+      def create_result
+        prepared_result
+      end
+
+      # Raises an error if the operation failed
+      # @raise [RdkafkaError]
+      def raise_error
+        raise RdkafkaError.new(
+          self[:response],
+          broker_message: broker_message
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/list_consumer_groups_report.rb

@@ -0,0 +1,83 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Report for list consumer groups operation result
+    class ListConsumerGroupsReport
+      # Consumer groups listed cluster-wide. Each entry is a hash with:
+      #   - `:group_id` [String] the consumer group id
+      #   - `:is_simple_consumer_group` [Boolean] `true` for a "simple" consumer group - one that
+      #     assigns partitions manually (via `assign`) and uses Kafka only for offset storage,
+      #     rather than joining the group-management protocol and letting Kafka assign partitions
+      #     and rebalance automatically (`subscribe`). Simple groups have no members from the
+      #     broker's point of view, so they never rebalance.
+      #   - `:state` [Integer] the group state as a `Bindings::RD_KAFKA_CONSUMER_GROUP_STATE_*`
+      #     code, one of: `RD_KAFKA_CONSUMER_GROUP_STATE_UNKNOWN`,
+      #     `RD_KAFKA_CONSUMER_GROUP_STATE_PREPARING_REBALANCE`,
+      #     `RD_KAFKA_CONSUMER_GROUP_STATE_COMPLETING_REBALANCE`,
+      #     `RD_KAFKA_CONSUMER_GROUP_STATE_STABLE`, `RD_KAFKA_CONSUMER_GROUP_STATE_DEAD`,
+      #     `RD_KAFKA_CONSUMER_GROUP_STATE_EMPTY`
+      #   - `:state_name` [String] human-readable name of that state (e.g. `"Stable"`, `"Empty"`)
+      # @return [Array<Hash>]
+      attr_reader :groups
+
+      # Per-broker errors reported alongside the (partial) group listing. `ListConsumerGroups`
+      # fans out to every broker and returns valid groups and errors separately, so a broker
+      # being unreachable does not discard the groups the reachable brokers returned.
+      # @return [Array<RdkafkaError>]
+      attr_reader :errors
+
+      # @param result_ptr [FFI::Pointer] pointer to the `rd_kafka_ListConsumerGroups_result_t`
+      def initialize(result_ptr)
+        @groups = []
+        @errors = []
+
+        return if result_ptr.null?
+
+        extract_groups(result_ptr)
+        extract_errors(result_ptr)
+      end
+
+      private
+
+      # @param result_ptr [FFI::Pointer] pointer to the result
+      def extract_groups(result_ptr)
+        count_ptr = FFI::MemoryPointer.new(:size_t)
+        array_ptr = Bindings.rd_kafka_ListConsumerGroups_result_valid(result_ptr, count_ptr)
+
+        return if array_ptr.null?
+
+        array_ptr.read_array_of_pointer(count_ptr.read(:size_t)).each do |listing_ptr|
+          state = Bindings.rd_kafka_ConsumerGroupListing_state(listing_ptr)
+          group_id_ptr = Bindings.rd_kafka_ConsumerGroupListing_group_id(listing_ptr)
+          state_name_ptr = Bindings.rd_kafka_consumer_group_state_name(state)
+
+          @groups << {
+            group_id: group_id_ptr.null? ? nil : group_id_ptr.read_string,
+            is_simple_consumer_group:
+              Bindings.rd_kafka_ConsumerGroupListing_is_simple_consumer_group(listing_ptr) != 0,
+            state: state,
+            state_name: state_name_ptr.null? ? nil : state_name_ptr.read_string
+          }
+        end
+      end
+
+      # @param result_ptr [FFI::Pointer] pointer to the result
+      def extract_errors(result_ptr)
+        count_ptr = FFI::MemoryPointer.new(:size_t)
+        array_ptr = Bindings.rd_kafka_ListConsumerGroups_result_errors(result_ptr, count_ptr)
+
+        return if array_ptr.null?
+
+        array_ptr.read_array_of_pointer(count_ptr.read(:size_t)).each do |error_ptr|
+          string_ptr = Bindings.rd_kafka_error_string(error_ptr)
+
+          @errors << RdkafkaError.new(
+            Bindings.rd_kafka_error_code(error_ptr),
+            broker_message: string_ptr.null? ? nil : string_ptr.read_string
+          )
+        end
+      end
+    end
+  end
+end

data/lib/rdkafka/admin.rb

@@ -108,10 +108,6 @@ module Rdkafka
     # @return [nil]
     # @raise [Rdkafka::ClosedAdminError] if called on a closed admin client
     #
-    # @note This method holds the inner lock until the queue is empty or `:stop` is returned.
-    #   Other admin operations will wait until this method returns.
-    # @note This method is thread-safe as it uses @native_kafka.with_inner synchronization
-    #
     # @example Drain all pending events
     #   admin.events_poll_nb_each { |_count| }
     #
@@ -120,6 +116,9 @@ module Rdkafka
     #   admin.events_poll_nb_each do |_count|
     #     :stop if monotonic_now >= deadline
     #   end
+    # @note This method holds the inner lock until the queue is empty or `:stop` is returned.
+    #   Other admin operations will wait until this method returns.
+    # @note This method is thread-safe as it uses @native_kafka.with_inner synchronization
     def events_poll_nb_each
       closed_admin_check(__method__)
 
@@ -283,6 +282,172 @@ module Rdkafka
       delete_groups_handle
     end
 
+    # Deletes all messages in the given partitions up to (but not including) the given offset.
+    #
+    # The programmatic equivalent of `kafka-delete-records.sh`. Useful for GDPR/right-to-erasure
+    # compliance, clearing out poison messages, or retention cleanup outside of the configured
+    # time/size-based retention policy.
+    #
+    # @param topic_partition_offsets [Hash{String => Array<Hash>}] hash mapping topic names to
+    #   arrays of partition delete specifications. Each specification is a hash with:
+    #   - `:partition` [Integer] partition number
+    #   - `:offset` [Symbol, Integer] delete all messages before this offset (exclusive) - an
+    #     integer offset, or `:end` to delete all data currently in the partition
+    #
+    # @return [DeleteRecordsHandle] handle that can be used to wait for the result
+    # @raise [RdkafkaError] when deleting records fails
+    #
+    # @example Delete all messages before offset 100 in one partition, and all data in another
+    #   report = admin.delete_records(
+    #     "my_topic" => [
+    #       { partition: 0, offset: 100 },
+    #       { partition: 1, offset: :end }
+    #     ]
+    #   ).wait(max_wait_timeout_ms: 15_000)
+    #
+    #   report.offsets.to_h
+    #   # => { "my_topic" => [#<Partition @partition=0, @offset=100, @err=0>, ...] }
+    def delete_records(topic_partition_offsets)
+      closed_admin_check(__method__)
+
+      parsed = topic_partition_offsets.flat_map do |topic, partitions|
+        partitions.map do |spec|
+          offset = spec.fetch(:offset)
+
+          native_offset = case offset
+          when :end then Rdkafka::Bindings::RD_KAFKA_OFFSET_END
+          when Integer then offset
+          else
+            raise ArgumentError, "Unknown offset specification: #{offset.inspect}"
+          end
+
+          [topic, spec.fetch(:partition), native_offset]
+        end
+      end
+
+      tpl = Rdkafka::Bindings.rd_kafka_topic_partition_list_new(parsed.size)
+
+      parsed.each do |topic, partition, native_offset|
+        Rdkafka::Bindings.rd_kafka_topic_partition_list_add(tpl, topic, partition)
+        Rdkafka::Bindings.rd_kafka_topic_partition_list_set_offset(tpl, topic, partition, native_offset)
+      end
+
+      # rd_kafka_DeleteRecords_new copies the tpl it is given, so it does not need to (and must
+      # not) outlive this call - it is destroyed below, independently of the wrapping
+      # DeleteRecords_t object.
+      delete_records_ptr = Rdkafka::Bindings.rd_kafka_DeleteRecords_new(tpl)
+      Rdkafka::Bindings.rd_kafka_topic_partition_list_destroy(tpl)
+
+      pointer_array = [delete_records_ptr]
+      records_array_ptr = FFI::MemoryPointer.new(:pointer)
+      records_array_ptr.write_array_of_pointer(pointer_array)
+
+      # Get a pointer to the queue that our request will be enqueued on
+      queue_ptr = @native_kafka.with_inner do |inner|
+        Rdkafka::Bindings.rd_kafka_queue_get_background(inner)
+      end
+      if queue_ptr.null?
+        Rdkafka::Bindings.rd_kafka_DeleteRecords_destroy(delete_records_ptr)
+        raise Rdkafka::Config::ConfigError.new("rd_kafka_queue_get_background was NULL")
+      end
+
+      # Create and register the handle we will return to the caller
+      handle = DeleteRecordsHandle.new
+      handle[:pending] = true
+      handle[:response] = Rdkafka::Bindings::RD_KAFKA_PARTITION_UA
+      DeleteRecordsHandle.register(handle)
+
+      admin_options_ptr = @native_kafka.with_inner do |inner|
+        Rdkafka::Bindings.rd_kafka_AdminOptions_new(inner, Rdkafka::Bindings::RD_KAFKA_ADMIN_OP_DELETERECORDS)
+      end
+      Rdkafka::Bindings.rd_kafka_AdminOptions_set_opaque(admin_options_ptr, handle.to_ptr)
+
+      begin
+        @native_kafka.with_inner do |inner|
+          Rdkafka::Bindings.rd_kafka_DeleteRecords(
+            inner,
+            records_array_ptr,
+            1,
+            admin_options_ptr,
+            queue_ptr
+          )
+        end
+      rescue Exception
+        DeleteRecordsHandle.remove(handle.to_ptr.address)
+        raise
+      ensure
+        Rdkafka::Bindings.rd_kafka_AdminOptions_destroy(admin_options_ptr)
+        Rdkafka::Bindings.rd_kafka_queue_destroy(queue_ptr)
+        Rdkafka::Bindings.rd_kafka_DeleteRecords_destroy(delete_records_ptr)
+      end
+
+      handle
+    end
+
+    # Lists consumer groups cluster-wide.
+    #
+    # librdkafka issues a single `ListConsumerGroups` request that is fanned out to every broker
+    # internally, so the result covers all consumer groups in the cluster, not only those
+    # coordinated by the connected broker.
+    #
+    # @return [ListConsumerGroupsHandle] handle that can be used to wait for the result
+    # @raise [RdkafkaError] when listing the consumer groups fails
+    #
+    # @example List every consumer group in the cluster and print its name and attributes
+    #   report = admin.list_consumer_groups.wait(max_wait_timeout_ms: 15_000)
+    #
+    #   report.groups.each do |group|
+    #     puts "#{group[:group_id]} - #{group[:state_name]} " \
+    #          "(simple: #{group[:is_simple_consumer_group]})"
+    #   end
+    #
+    #   # Any brokers that could not be reached are reported separately
+    #   report.errors.each { |error| warn "partial listing error: #{error.message}" }
+    def list_consumer_groups
+      closed_admin_check(__method__)
+
+      # Get a pointer to the queue that our request will be enqueued on
+      queue_ptr = @native_kafka.with_inner do |inner|
+        Rdkafka::Bindings.rd_kafka_queue_get_background(inner)
+      end
+
+      if queue_ptr.null?
+        raise Rdkafka::Config::ConfigError.new("rd_kafka_queue_get_background was NULL")
+      end
+
+      # Create and register the handle we will return to the caller
+      handle = ListConsumerGroupsHandle.new
+      handle[:pending] = true
+      handle[:response] = Rdkafka::Bindings::RD_KAFKA_PARTITION_UA
+      ListConsumerGroupsHandle.register(handle)
+
+      admin_options_ptr = @native_kafka.with_inner do |inner|
+        Rdkafka::Bindings.rd_kafka_AdminOptions_new(
+          inner,
+          Rdkafka::Bindings::RD_KAFKA_ADMIN_OP_LISTCONSUMERGROUPS
+        )
+      end
+      Rdkafka::Bindings.rd_kafka_AdminOptions_set_opaque(admin_options_ptr, handle.to_ptr)
+
+      begin
+        @native_kafka.with_inner do |inner|
+          Rdkafka::Bindings.rd_kafka_ListConsumerGroups(
+            inner,
+            admin_options_ptr,
+            queue_ptr
+          )
+        end
+      rescue Exception
+        ListConsumerGroupsHandle.remove(handle.to_ptr.address)
+        raise
+      ensure
+        Rdkafka::Bindings.rd_kafka_AdminOptions_destroy(admin_options_ptr)
+        Rdkafka::Bindings.rd_kafka_queue_destroy(queue_ptr)
+      end
+
+      handle
+    end
+
     # Deletes the named topic
     #
     # @param topic_name [String] name of the topic to delete
@@ -345,7 +510,8 @@ module Rdkafka
     #
     # @param topic_name [String] name of the topic
     # @param partition_count [Integer] how many partitions we want to end up with for given topic
-    # @return [CreatePartitionsHandle] Create partitions handle that can be used to wait for the result
+    # @return [CreatePartitionsHandle] Create partitions handle that can be used to wait for the
+    #   result
     # @raise [ConfigError] When the partition count or replication factor are out of valid range
     # @raise [RdkafkaError] When the topic name is invalid or the topic already exists
     # @raise [RdkafkaError] When the topic configuration is invalid
@@ -434,7 +600,8 @@ module Rdkafka
     # @param permission_type [Integer] rd_kafka_AclPermissionType_t value:
     #   - RD_KAFKA_ACL_PERMISSION_TYPE_DENY  = 2
     #   - RD_KAFKA_ACL_PERMISSION_TYPE_ALLOW = 3
-    # @return [CreateAclHandle] Create acl handle that can be used to wait for the result of creating the acl
+    # @return [CreateAclHandle] Create acl handle that can be used to wait for the result of
+    #   creating the acl
     # @raise [RdkafkaError]
     def create_acl(resource_type:, resource_name:, resource_pattern_type:, principal:, host:, operation:, permission_type:)
       closed_admin_check(__method__)
@@ -535,7 +702,8 @@ module Rdkafka
     # @param permission_type [Integer] rd_kafka_AclPermissionType_t value:
     #   - RD_KAFKA_ACL_PERMISSION_TYPE_DENY  = 2
     #   - RD_KAFKA_ACL_PERMISSION_TYPE_ALLOW = 3
-    # @return [DeleteAclHandle] Delete acl handle that can be used to wait for the result of deleting the acl
+    # @return [DeleteAclHandle] Delete acl handle that can be used to wait for the result of
+    #   deleting the acl
     # @raise [RdkafkaError]
     def delete_acl(resource_type:, resource_name:, resource_pattern_type:, principal:, host:, operation:, permission_type:)
       closed_admin_check(__method__)
@@ -638,7 +806,8 @@ module Rdkafka
     # @param permission_type [Integer] rd_kafka_AclPermissionType_t value:
     #   - RD_KAFKA_ACL_PERMISSION_TYPE_DENY  = 2
     #   - RD_KAFKA_ACL_PERMISSION_TYPE_ALLOW = 3
-    # @return [DescribeAclHandle] Describe acl handle that can be used to wait for the result of fetching acls
+    # @return [DescribeAclHandle] Describe acl handle that can be used to wait for the result of
+    #   fetching acls
     # @raise [RdkafkaError]
     def describe_acl(resource_type:, resource_name:, resource_pattern_type:, principal:, host:, operation:, permission_type:)
       closed_admin_check(__method__)