karafka-rdkafka 0.30.1-aarch64-linux-gnu → 0.31.0-aarch64-linux-gnu

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7948198b70e6be9ea2ceae0392b50b1babc7875c4986028e2e244f2391391be2
-  data.tar.gz: ad3a6555db52fad384f1ebc1a7b45a8b0e058d85396c4e4c775091be73dae645
+  metadata.gz: 31a31d86d0d0ee5888adc297e5bfc9338f18b453b4834095fd4159f6562458ad
+  data.tar.gz: 1da9da2a1268fb3a9256475dd5cb7e8cccaf8dbaaa061436ae4a1174ba1f3f44
 SHA512:
-  metadata.gz: 5b53dfba1d8765777243906de1ef4522c8466c14e54316b418a74a1feb70218ed347f089a52a10359474d5bca63b911aeb88ffd7018d82b29fe8879e9de69ab9
-  data.tar.gz: f261e03bfa7eaf8f7bbcf85ac1563214f066759b1d6aa805e9997aa17eaa141349ea9cad3b13b88d401bad8baf68f934c45bb10c56f4c99fc95a16fbc9cd90fa
+  metadata.gz: 70d99fbf5bf5d225f51b461266c8aac8a4f21e7d5ea860ffd17aa94429ec44bfb52e84e7d770e37b9297218e38147610ecdac24181d93a8c6910c4e791e8a52c
+  data.tar.gz: 52271992a6802a31a8032264d9a7b61b8f890cdfc479c22402ea9c59c22193f77e5f637318a1701dea71e9c442d03162c1401b3383331d3702b4676f7f978a01

data/CHANGELOG.md

@@ -1,30 +1,49 @@
 # Rdkafka Changelog
 
+## 0.31.0 (2026-10-12)
+- [Feature] Add `Admin#describe_cluster` to get the cluster id, controller and broker nodes (with racks) without a full metadata request. Ported from rdkafka-ruby (#1015).
+- [Feature] Add `Admin#describe_topics` to get topic ids, partition leaders and replicas, and per-topic errors. Both methods can also return the operations the client is authorized to perform. Ported from rdkafka-ruby (#1015).
+- [Feature] Add `Admin#describe_consumer_groups` to read the state, type, coordinator, authorized operations and members (with their current and target partition assignments) of consumer groups. Ported from rdkafka-ruby (#1016).
+- [Feature] Add `status`, `latency` and `broker_id` to `Producer::DeliveryReport`, with `#persisted?`, `#possibly_persisted?` and `#not_persisted?`, so a failed delivery tells whether a retry may duplicate the message. Ported from rdkafka-ruby (#1019).
+- [Enhancement] Bump librdkafka to `2.16.0` for OpenSSL and libcurl security fixes. Admin requests in flight on a decommissioned broker now fail with a retryable transport error. Ported from rdkafka-ruby (#1020).
+- [Enhancement] Bump the bundled OpenSSL to `3.5.9` (LTS) for its security fixes. Ported from rdkafka-ruby (#1007).
+- [Enhancement] Read each pointer field once in `Message#initialize`, saving 3 allocations per consumed message. Ported from rdkafka-ruby (#1026).
+- [Change] Drop Ruby 3.2 support; require Ruby 3.3+.
+- [Change] Build the precompiled `aarch64-linux-musl` gem on Alpine `3.24` (was `3.23`), matching `x86_64-linux-musl`. Ported from rdkafka-ruby (#1012).
+- [Change] Build the precompiled `macos_arm64` library on macOS 26 with the minimum macOS pinned to 14.0, instead of inheriting it from the build host. Ported from rdkafka-ruby (#1013).
+- [Maintenance] Split `lib/rdkafka/bindings.rb` into per-area files under `lib/rdkafka/bindings/`. `Rdkafka::Bindings` is unchanged. Ported from rdkafka-ruby (#1022).
+- [Maintenance] Stabilize the flaky partitions count cache "multiple topics with different TTLs" spec by driving its clock instead of sleeping. Ported from rdkafka-ruby (#1028).
+- [Fix] Sort merged share-group async acknowledgements so the broker does not reject them with `invalid_request` (librdkafka patch).
+- [Fix] Release share records acquired by an in-flight fetch on unsubscribe instead of keeping them locked (librdkafka patch). `ShareConsumer#unsubscribe` now closes the share sessions like `#close` does and waits for that (up to `fetch.wait.max.ms` with a fetch in flight).
+
+## 0.30.2 (2026-10-01)
+- [Feature] Add `ShareConsumer#events_poll` (and `#events_poll_nb`) to service the statistics, error, log and OAuthBearer callbacks without acquiring records.
+
 ## 0.30.1 (2026-09-30)
-- [Fix] Derive `ShareConsumer#name` from the native handle at creation (mirroring `Consumer#name`) instead of leaving it `nil` until an OAuthBearer callback happened to set it. A `nil` name meant downstreams that route the global statistics and error callbacks by client name (e.g. Karafka) silently dropped every share-consumer statistic and background error.
+- [Fix] Set `ShareConsumer#name` from the native handle at creation, so name-based callback routing (e.g. Karafka) no longer drops share-consumer statistics and errors.
 
 ## 0.30.0 (2026-09-25)
-- [Fix] Register share consumers in `Rdkafka::Clients` and destroy the native handle when `Config#share_consumer` fails part-way. Share consumers own the native handle directly rather than a `NativeKafka` wrapper, so they bypass `build_native_client` and were missed by the `at_exit` shutdown hook, leaving librdkafka to be unloaded with a live share handle.
-- [Feature] Add preview support for KIP-932 share groups ("Queues for Kafka") via `Rdkafka::ShareConsumer`, created with `Config#share_consumer`. Members consume the same partitions cooperatively with per-record acknowledgements (`:accept`, `:release`, `:reject`) instead of committed offsets. Like the regular `Consumer`, `ShareConsumer` is a thin binding that exposes the librdkafka share primitives (`#subscribe`/`#unsubscribe`/`#subscription`, batch `#poll`, `#acknowledge`, `#commit_sync`, `#commit_async`, `#acknowledgement_commit_callback=`, `#close`) but drives none of them itself - there is no built-in poll loop, acknowledgement strategy or offset/lifecycle orchestration, leaving that to a higher layer such as Karafka. The handle is single-threaded by design, servicing log, statistics and error callbacks from within `#poll`. Requires a broker with share groups enabled (Apache Kafka 4.2.0+); librdkafka marks the feature as preview and not production-ready.
-- [Enhancement] `RdkafkaError.build_from_c` now uses the human readable string librdkafka attaches to the `rd_kafka_error_t` instead of always using `rd_kafka_err2str`, improving diagnostics for every error-pointer path. It also destroys a code-0 error object instead of leaking it on the early return.
-- [Enhancement] Extract the OAuthBearer token buffer/extension plumbing in `Helpers::OAuth` around two private native-call hooks so the share consumer overrides only the hooks instead of duplicating both public methods.
-- [Fix] Make `ShareConsumer#close` safe against in-flight calls from other threads: closing now waits for operations in progress (mirroring `NativeKafka`), preventing a NULL handle from reaching librdkafka or a double destroy when two threads race `close`.
-- [Fix] Keep the `ShareConsumer` native handle and its GC finalizer when `close`/`destroy` report an error. librdkafka leaves the instance intact in that case, so dropping the references beforehand leaked the client permanently; `close` is now retriable after such an error.
-- [Fix] Pin the acknowledgement commit callback `FFI::Function` in the state holder shared with the `ShareConsumer` GC finalizer, preventing a use-after-free when an unclosed consumer is garbage collected and the finalizer-driven destroy flushes pending acknowledgements through it.
-- [Fix] Make `ShareConsumer` fork-aware, mirroring `NativeKafka`. `rd_kafka_share_consumer_new` is backed by the same broker/main threads as any librdkafka client, which do not survive `fork`; the hand-rolled share lifecycle previously had no `@creator_pid` guard, so a process that forked after creating a share consumer would segfault (or deadlock) when the child's GC finalizer or `close` ran `rd_kafka_share_destroy` on the inherited handle. It now records the creating pid and reports `closed?` in any other process, so the finalizer and `close` skip the native teardown for an inherited handle while `with_native` rejects any use of it.
-- [Fix] Scope `ShareConsumer#each`'s `ClosedConsumerError` rescue to the `poll` call only. It previously wrapped the `yield`, so a `ClosedConsumerError` raised from inside the caller's block was swallowed and silently ended iteration; the block's errors now propagate, and only the consumer actually closing ends the loop. Its docs also note that, like `#poll`, `#each` can yield an `RdkafkaError` for a record that fails to build.
-- [Fix] Raise `ArgumentError` for `ShareConsumer#subscribe` without topics. librdkafka treats an empty share subscribe as an unsubscribe and returns no error (unlike the regular consumer), silently dropping the subscription.
-- [Fix] Capture the librdkafka client name onto `ShareConsumer#name` from the OAuthBearer token refresh callback. The share handle has no native name accessor, which made the documented name-based routing of the global oauth callback impossible for share consumers.
+- [Fix] Register share consumers in `Rdkafka::Clients` and destroy the native handle when `Config#share_consumer` fails part-way, so a share consumer is no longer missed by the `at_exit` shutdown hook.
+- [Feature] Add preview support for KIP-932 share groups ("Queues for Kafka") via `Rdkafka::ShareConsumer` (`Config#share_consumer`), with per-record acknowledgements (`:accept`, `:release`, `:reject`). Requires Apache Kafka 4.2.0+ with share groups enabled. Not production-ready.
+- [Enhancement] `RdkafkaError.build_from_c` now uses the human-readable string librdkafka attaches to the `rd_kafka_error_t` instead of `rd_kafka_err2str`, improving diagnostics for every error-pointer path.
+- [Enhancement] Refactor the `Helpers::OAuth` token plumbing around two private hooks so the share consumer overrides only those instead of duplicating both public methods.
+- [Fix] Make `ShareConsumer#close` thread-safe: it now waits for in-flight operations (mirroring `NativeKafka`), preventing a double destroy when two threads race `close`.
+- [Fix] Keep the `ShareConsumer` native handle and its GC finalizer when `close`/`destroy` report an error, so the client is no longer leaked and `close` stays retriable.
+- [Fix] Pin the acknowledgement commit callback `FFI::Function` for the lifetime of the `ShareConsumer`, preventing a use-after-free when an unclosed consumer is garbage collected.
+- [Fix] Make `ShareConsumer` fork-aware (mirroring `NativeKafka`): it reports `closed?` in a forked child, which no longer segfaults on `close` or finalization.
+- [Fix] Rescue `ClosedConsumerError` in `ShareConsumer#each` only around `poll`, so a `ClosedConsumerError` raised inside the block propagates instead of silently ending iteration. Like `#poll`, `#each` can also yield an `RdkafkaError` for a record that fails to build.
+- [Fix] Raise `ArgumentError` for `ShareConsumer#subscribe` without topics, which librdkafka otherwise treats as an unsubscribe without error, silently dropping the subscription.
+- [Fix] Capture the librdkafka client name onto `ShareConsumer#name` from the OAuthBearer token refresh callback, since the share handle has no native name accessor for the documented name-based oauth callback routing.
 - [Fix] Update `ext/build_common.sh` (precompiled builds) to librdkafka `2.15.0` and its tarball checksum; it still pinned `2.14.1`, whose tarball is no longer vendored.
-- [Fix] Stop the `share_consumer_multi_member` integration spec from failing on redelivery. KIP-932 share groups are documented as at-least-once, so a record whose acquisition lock expires before it is acknowledged may be redelivered with an incremented `delivery_count`; the spec now tracks distinct payloads and fails only when a record is never delivered.
-- [Fix] Apply the same at-least-once tolerance to the `share_consumer_explicit_ack` integration spec, which still asserted exactly one redelivered record. It now requires only the released record to be redelivered (with `delivery_count >= 2`), forbids only the rejected record, and tolerates an accepted record reappearing when its acquisition lock expired before the commit flushed the accept.
-- [Fix] Add the `nofile` ulimit to `docker-compose-ssl.yml`, mirroring `docker-compose.yml`. The SSL CI broker runs the same topic-heavy suite (now enlarged by the share consumer specs) and could otherwise exhaust the default file-descriptor limit under the many short-lived topics.
-- [Fix] Enable KIP-932 share groups on the macOS CI broker, whose KRaft `server.properties` never opted into the share rebalance protocol, so the one broker-backed share consumer spec received zero records on every macOS runner. `CONFLUENT_VERSION` is bumped to `8.3.0` and the download path is derived from it so a Renovate bump cannot desync them.
-- [Enhancement] Add two share consumer integration specs: `share_consumer_implicit_ack_spec` (polling alone accepts the previous batch, and accepted records are not redelivered to a second member) and `share_consumer_multi_topic_spec` (one consumer subscribed to several topics receives every topic's records, attributed to the right topic). Both track distinct payloads so an at-least-once redelivery cannot end the consume loop early.
-- [Fix] Stop the `statistics_unassigned_producer` integration spec from flaking on a metadata-propagation race: the freshly built producer could send before fetching the new topic's metadata, so the produce is now retried on `unknown_topic_or_part` and `leader_not_available`.
-- [Fix] Also stabilize the `statistics_unassigned_producer` integration spec against a delivery wait that outlives its budget. With 1000 partitions librdkafka keeps the message queued while leaders are still being elected, so the handle stays pending and the wait raises `WaitTimeoutError` - a `RuntimeError` the spec's existing code-based rescue could never catch. The produce now uses the default 60s handle wait (matching the sibling `statistics_unassigned_*` specs, instead of a self-imposed 15s) and retries on that timeout too.
-- [Fix] Stabilize the `consumer_memberid_clusterid_leak` integration spec against RSS measurement noise on Ruby 4.0. It now compacts the heap before sampling (matching the `to_native_tpl` leak spec) and the ceiling is 3 MB, still well under the ~5.7 MB a genuine per-call leak would cost.
-- [Note] Share consumers emit the regular consumer statistics JSON (including `cgrp`; there is no share-specific section in librdkafka 2.15.0) through the usual `statistics_callback`, serviced from within `#poll`. The share fetch path bypasses the per-partition stats bookkeeping entirely, so the `topics` section carries no partition entries and the `statistics.unassigned.include` filter needs no share-consumer special-casing; per-partition share metrics are not exposed by librdkafka 2.15.0 at all.
+- [Fix] Stop the `share_consumer_multi_member` integration spec from failing on an at-least-once redelivery; it now tracks distinct payloads and fails only when a record is never delivered.
+- [Fix] Apply the same at-least-once tolerance to the `share_consumer_explicit_ack` integration spec: require only the released record to be redelivered, forbid the rejected one, and tolerate an accepted record reappearing.
+- [Fix] Add the `nofile` ulimit to `docker-compose-ssl.yml` (mirroring `docker-compose.yml`) so the SSL CI broker does not exhaust file descriptors under the topic-heavy share-consumer suite.
+- [Fix] Enable KIP-932 share groups on the macOS CI broker, whose KRaft config never opted into the share rebalance protocol, so its share-consumer spec no longer receives zero records. `CONFLUENT_VERSION` is bumped to `8.3.0`.
+- [Enhancement] Add `share_consumer_implicit_ack` and `share_consumer_multi_topic` integration specs.
+- [Fix] Stop the `statistics_unassigned_producer` integration spec from flaking on a metadata-propagation race by retrying the produce on `unknown_topic_or_part` and `leader_not_available`.
+- [Fix] Also stabilize the `statistics_unassigned_producer` integration spec against a delivery wait that outlives its budget: it now uses the default 60s handle wait and retries on `WaitTimeoutError`.
+- [Fix] Stabilize the `consumer_memberid_clusterid_leak` integration spec against RSS measurement noise on Ruby 4.0 by compacting the heap before sampling and raising the ceiling to 3 MB.
+- [Note] Share consumers emit the regular consumer statistics through `statistics_callback`. librdkafka 2.15.0 has no share-specific metrics, so the `topics` section has no partition entries.
 - [Feature] Add `Admin#alter_consumer_group_offsets` and `Admin#delete_consumer_group_offsets` to set or clear a consumer group's committed offsets from the admin client. Ported from rdkafka-ruby (#983).
 - [Enhancement] Bump librdkafka to `2.15.1` for OpenSSL and libcurl security fixes. It changes how IPv6 addresses are formatted and validated against broker certificates.
 - [Enhancement] Bump the bundled zlib to `1.3.2`.
@@ -34,8 +53,8 @@
 - [Maintenance] Drop the unused `dist/openssl-3.0.16.tar.gz` build cache.
 - [Fix] Honor the `isolation_level:` argument to `Admin#list_offsets`, which was silently ignored. Ported from rdkafka-ruby (#983).
 - [Fix] Return correct results for every item of a multi-item topic, partition, group or ACL admin request, not just the first. Ported from rdkafka-ruby (#987).
-- [Maintenance] Fix a use-after-free in the multi-item admin integration test that read result-name pointers after librdkafka had destroyed the background event, which made it flaky on newer glibc (e.g. Debian trixie). Ported from rdkafka-ruby (#991).
-- [Maintenance] Stabilize consumer specs on slow CI runners: wait longer for a partition assignment, raise the `TestTopics.create` admin timeout (a timeout there also tripped the leaked-handle guard), give the non-blocking `poll_nb` message check a generous retry budget instead of a fixed ~2s, and let the long-running consumption spec drain its backlog instead of stopping at a fixed 60s. The assignment wait is tunable via `RDKAFKA_TEST_ASSIGNMENT_TIMEOUT`. Ported from rdkafka-ruby (#990).
+- [Maintenance] Fix a use-after-free in the multi-item admin integration test that made it flaky on newer glibc. Ported from rdkafka-ruby (#991).
+- [Maintenance] Stabilize consumer specs on slow CI runners. The assignment wait is tunable via `RDKAFKA_TEST_ASSIGNMENT_TIMEOUT`. Ported from rdkafka-ruby (#990).
 
 ## 0.29.0 (2026-09-14)
 - [Enhancement] Bump librdkafka to `2.15.0` (staying on `2.15.0` rather than `2.15.1` so users hitting a regression in `2.15.1` have a stable fallback).
@@ -123,8 +142,8 @@
 - [Fix] Prevent cascading test failures in admin specs when a single handle leaks into the registry (from upstream).
 
 ## 0.24.0 (2026-02-25)
-- **[Feature]** Add `Producer#queue_size` (and `#queue_length` alias) to report the number of messages waiting in the librdkafka output queue. Useful for monitoring producer backpressure, implementing custom flow control, debugging message delivery issues, and graceful shutdown logic.
-- **[Feature]** Add fiber scheduler API for integration with Ruby fiber schedulers (Falcon, Async) and custom event loops (from upstream). Expose `enable_queue_io_events` and `enable_background_queue_io_events` methods on `Consumer`, `Producer`, and `Admin`.
+- **[Feature]** Add `Producer#queue_size` (and the `#queue_length` alias) to report the number of messages waiting in the librdkafka output queue.
+- **[Feature]** Add `enable_queue_io_events` and `enable_background_queue_io_events` to `Consumer`, `Producer` and `Admin` for fiber schedulers (Falcon, Async) and custom event loops (from upstream).
 - **[Deprecation]** `AbstractHandle#wait` parameter `max_wait_timeout` (seconds) is deprecated in favor of `max_wait_timeout_ms` (milliseconds). The old parameter still works with backwards compatibility but will be removed in v1.0.0.
 - **[Deprecation]** `PartitionsCountCache` constructor parameter `ttl` (seconds) is deprecated in favor of `ttl_ms` (milliseconds). The old parameter still works with backwards compatibility but will be removed in v1.0.0.
 - [Enhancement] Add Ruby 4.0 support.
@@ -151,7 +170,7 @@
 - [Enhancement] Add `RdkafkaError.build_fatal` class method for centralized fatal error construction.
 - [Enhancement] Add comprehensive tests for fatal error handling including unit tests and integration tests.
 - [Enhancement] Add `RD_KAFKA_PARTITION_UA` constant for unassigned partition (-1).
-- [Enhancement] Replace magic numbers with named constants: use `RD_KAFKA_RESP_ERR_NO_ERROR` instead of `0` for error code checks (18 instances) and `RD_KAFKA_PARTITION_UA` instead of `-1` for partition values (9 instances) across the codebase for better code clarity and maintainability.
+- [Enhancement] Replace magic numbers with the `RD_KAFKA_RESP_ERR_NO_ERROR` and `RD_KAFKA_PARTITION_UA` constants.
 - [Enhancement] Add `Rdkafka::Testing` module for testing fatal error scenarios on both producers and consumers.
 - [Deprecated] `RdkafkaError.validate_fatal!` - use `validate!` with `client_ptr` parameter instead.
 
@@ -165,7 +184,7 @@
 
 ## 0.22.1 (2025-10-09)
 - [Enhancement] Optimize header processing to eliminate double hash lookups and method checking overhead.
-- [Enhancement] Optimize producer header processing with early returns and efficient array operations (69% faster for nil headers, 41% faster for empty headers, 12-32% faster when headers are present, with larger improvements for complex header scenarios).
+- [Enhancement] Speed up producer header processing (69% faster for nil headers, 41% for empty headers, 12-32% when headers are present).
 
 ## 0.22.0 (2025-09-26)
 - **[EOL]** Drop support for Ruby 3.1 to move forward with the fiber scheduler work.

data/README.md

@@ -63,6 +63,7 @@ Contributions should generally be made to the upstream [rdkafka-ruby repository]
 
 | karafka-rdkafka | librdkafka | patches |
 |-|-|-|
+| 0.31.x (2026-10-12) | 2.16.0 (2026-10-06)  | yes |
 | 0.30.x (2026-09-25) | 2.15.1 (2026-09-09)  | yes |
 | 0.29.x (2026-09-14) | 2.15.0 (2026-06-30)  | yes |
 | 0.28.x (2026-07-12) | 2.14.2 (2026-06-03)  | yes |

data/docker-compose-ssl.yml

@@ -25,6 +25,7 @@ services:
       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL
       KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:9092,SSL://127.0.0.1:9093
       KAFKA_BROKER_ID: 1
+      KAFKA_BROKER_RACK: rack-1
       KAFKA_CONTROLLER_QUORUM_VOTERS: 1@127.0.0.1:9094
       ALLOW_PLAINTEXT_LISTENER: 'yes'
       KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'

data/docker-compose.yml

@@ -23,6 +23,7 @@ services:
       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
       KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:9092
       KAFKA_BROKER_ID: 1
+      KAFKA_BROKER_RACK: rack-1
       KAFKA_CONTROLLER_QUORUM_VOTERS: 1@127.0.0.1:9093
       ALLOW_PLAINTEXT_LISTENER: 'yes'
       KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'

data/karafka-rdkafka.gemspec

@@ -12,7 +12,7 @@ Gem::Specification.new do |gem|
   gem.name = "karafka-rdkafka"
   gem.require_paths = ["lib"]
   gem.version = Rdkafka::VERSION
-  gem.required_ruby_version = ">= 3.2"
+  gem.required_ruby_version = ">= 3.3"
 
   files = `git ls-files`.split($\)
   files = files.reject do |file|

data/lib/rdkafka/admin/describe_cluster_handle.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Handle for describe cluster operation
+    class DescribeClusterHandle < AbstractHandle
+      layout :pending, :bool,
+        :response, :int
+
+      # @return [String] the name of the operation
+      def operation_name
+        "describe cluster"
+      end
+
+      # @return [DescribeClusterReport] report prepared by the background event callback, with
+      #   the cluster id, controller and broker nodes.
+      def create_result
+        prepared_result
+      end
+
+      # Raises an error if the operation failed
+      # @raise [RdkafkaError]
+      def raise_error
+        raise RdkafkaError.new(
+          self[:response],
+          broker_message: broker_message
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/describe_cluster_report.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Report for describe cluster operation result
+    class DescribeClusterReport
+      include DescribeResultParsing
+
+      # @return [String, nil] the cluster id
+      attr_reader :cluster_id
+
+      # The current controller node, as a hash with `:id` [Integer], `:host` [String],
+      # `:port` [Integer] and `:rack` [String, nil]
+      # @return [Hash, nil]
+      attr_reader :controller
+
+      # Broker nodes of the cluster, each in the same shape as {#controller}
+      # @return [Array<Hash>]
+      attr_reader :nodes
+
+      # Operations the client is authorized to perform on the cluster, as
+      # `Bindings::RD_KAFKA_ACL_OPERATION_*` codes. `nil` unless requested with
+      # `include_authorized_operations: true`.
+      # @return [Array<Integer>, nil]
+      attr_reader :authorized_operations
+
+      # @param result_ptr [FFI::Pointer] pointer to the `rd_kafka_DescribeCluster_result_t`
+      def initialize(result_ptr)
+        @nodes = []
+
+        return if result_ptr.null?
+
+        cluster_id_ptr = Bindings.rd_kafka_DescribeCluster_result_cluster_id(result_ptr)
+        @cluster_id = cluster_id_ptr.null? ? nil : cluster_id_ptr.read_string
+        @controller = extract_node(Bindings.rd_kafka_DescribeCluster_result_controller(result_ptr))
+
+        count_ptr = FFI::MemoryPointer.new(:size_t)
+        @nodes = extract_nodes(Bindings.rd_kafka_DescribeCluster_result_nodes(result_ptr, count_ptr), count_ptr)
+
+        count_ptr = FFI::MemoryPointer.new(:size_t)
+        @authorized_operations = extract_authorized_operations(
+          Bindings.rd_kafka_DescribeCluster_result_authorized_operations(result_ptr, count_ptr),
+          count_ptr
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/describe_consumer_groups_handle.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Handle for describe consumer groups operation
+    class DescribeConsumerGroupsHandle < AbstractHandle
+      layout :pending, :bool,
+        :response, :int
+
+      # @return [String] the name of the operation
+      def operation_name
+        "describe consumer groups"
+      end
+
+      # @return [DescribeConsumerGroupsReport] report prepared by the background event callback,
+      #   with the described consumer groups.
+      def create_result
+        prepared_result
+      end
+
+      # Raises an error if the operation failed
+      # @raise [RdkafkaError]
+      def raise_error
+        raise RdkafkaError.new(
+          self[:response],
+          broker_message: broker_message
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/describe_consumer_groups_report.rb

@@ -0,0 +1,151 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Report for describe consumer groups operation result
+    class DescribeConsumerGroupsReport
+      include DescribeResultParsing
+
+      # Described consumer groups, one per requested group id. Each entry is a hash with:
+      #   - `:group_id` [String] the consumer group id
+      #   - `:error` [RdkafkaError, nil] error describing this group, or `nil` on success. The
+      #     other fields are not meaningful when it is set.
+      #   - `:is_simple_consumer_group` [Boolean] `true` for a group that assigns partitions
+      #     manually and uses Kafka only for offset storage
+      #   - `:state` [Integer] the group state as a `Bindings::RD_KAFKA_CONSUMER_GROUP_STATE_*`
+      #     code. A group id the cluster does not know is reported as
+      #     `RD_KAFKA_CONSUMER_GROUP_STATE_DEAD` with no members.
+      #   - `:state_name` [String] human-readable name of that state (e.g. `"Stable"`, `"Empty"`)
+      #   - `:type` [Integer] the group protocol type as a
+      #     `Bindings::RD_KAFKA_CONSUMER_GROUP_TYPE_*` code, one of:
+      #     `RD_KAFKA_CONSUMER_GROUP_TYPE_CLASSIC`,
+      #     `RD_KAFKA_CONSUMER_GROUP_TYPE_CONSUMER` (KIP-848),
+      #     `RD_KAFKA_CONSUMER_GROUP_TYPE_UNKNOWN`
+      #   - `:type_name` [String] human-readable name of that type (e.g. `"Classic"`, `"Consumer"`)
+      #   - `:partition_assignor` [String, nil] partition assignor in use (e.g. `"range"`)
+      #   - `:coordinator` [Hash, nil] the group coordinator broker as
+      #     `{ id:, host:, port:, rack: }`
+      #   - `:authorized_operations` [Array<Integer>, nil] ACL operations the client may perform on
+      #     the group, as `Bindings::RD_KAFKA_ACL_OPERATION_*` codes. `nil` unless requested with
+      #     `include_authorized_operations: true`.
+      #   - `:members` [Array<Hash>] current group members, each with:
+      #     - `:member_id` [String] the member (consumer) id
+      #     - `:client_id` [String] the member's `client.id`
+      #     - `:group_instance_id` [String, nil] the static membership instance id, if any
+      #     - `:host` [String] the host the member connects from
+      #     - `:assignment` [Hash{String => Array<Integer>}] assigned partitions, by topic
+      #     - `:target_assignment` [Hash{String => Array<Integer>}, nil] partitions the member is
+      #       moving to during a rebalance, by topic. Only `consumer` (KIP-848) groups have it;
+      #       `nil` for `classic` groups.
+      # @return [Array<Hash>]
+      attr_reader :groups
+
+      # @param result_ptr [FFI::Pointer] pointer to the `rd_kafka_DescribeConsumerGroups_result_t`
+      def initialize(result_ptr)
+        @groups = []
+
+        return if result_ptr.null?
+
+        count_ptr = FFI::MemoryPointer.new(:size_t)
+        array_ptr = Bindings.rd_kafka_DescribeConsumerGroups_result_groups(result_ptr, count_ptr)
+
+        return if array_ptr.null?
+
+        array_ptr.read_array_of_pointer(count_ptr.read(:size_t)).each do |group_ptr|
+          @groups << build_group(group_ptr)
+        end
+      end
+
+      private
+
+      # @param group_ptr [FFI::Pointer] pointer to the `rd_kafka_ConsumerGroupDescription_t`
+      # @return [Hash]
+      def build_group(group_ptr)
+        state = Bindings.rd_kafka_ConsumerGroupDescription_state(group_ptr)
+        type = Bindings.rd_kafka_ConsumerGroupDescription_type(group_ptr)
+
+        {
+          group_id: read_string(Bindings.rd_kafka_ConsumerGroupDescription_group_id(group_ptr)),
+          error: build_error(Bindings.rd_kafka_ConsumerGroupDescription_error(group_ptr)),
+          is_simple_consumer_group:
+            Bindings.rd_kafka_ConsumerGroupDescription_is_simple_consumer_group(group_ptr) != 0,
+          state: state,
+          state_name: read_string(Bindings.rd_kafka_consumer_group_state_name(state)),
+          type: type,
+          type_name: read_string(Bindings.rd_kafka_consumer_group_type_name(type)),
+          partition_assignor:
+            read_string(Bindings.rd_kafka_ConsumerGroupDescription_partition_assignor(group_ptr)),
+          coordinator: extract_node(Bindings.rd_kafka_ConsumerGroupDescription_coordinator(group_ptr)),
+          authorized_operations: build_authorized_operations(group_ptr),
+          members: Array.new(Bindings.rd_kafka_ConsumerGroupDescription_member_count(group_ptr)) do |index|
+            build_member(Bindings.rd_kafka_ConsumerGroupDescription_member(group_ptr, index))
+          end
+        }
+      end
+
+      # @param member_ptr [FFI::Pointer] pointer to the `rd_kafka_MemberDescription_t`
+      # @return [Hash]
+      def build_member(member_ptr)
+        {
+          member_id: read_string(Bindings.rd_kafka_MemberDescription_consumer_id(member_ptr)),
+          client_id: read_string(Bindings.rd_kafka_MemberDescription_client_id(member_ptr)),
+          group_instance_id: read_string(Bindings.rd_kafka_MemberDescription_group_instance_id(member_ptr)),
+          host: read_string(Bindings.rd_kafka_MemberDescription_host(member_ptr)),
+          assignment: build_assignment(Bindings.rd_kafka_MemberDescription_assignment(member_ptr)),
+          target_assignment: build_target_assignment(member_ptr)
+        }
+      end
+
+      # @param assignment_ptr [FFI::Pointer] pointer to the `rd_kafka_MemberAssignment_t`
+      # @return [Hash{String => Array<Integer>}]
+      def build_assignment(assignment_ptr)
+        return {} if assignment_ptr.null?
+
+        tpl_ptr = Bindings.rd_kafka_MemberAssignment_partitions(assignment_ptr)
+
+        return {} if tpl_ptr.null?
+
+        Consumer::TopicPartitionList
+          .from_native_tpl(tpl_ptr)
+          .to_h
+          .transform_values { |partitions| (partitions || []).map(&:partition) }
+      end
+
+      # @param member_ptr [FFI::Pointer] pointer to the `rd_kafka_MemberDescription_t`
+      # @return [Hash{String => Array<Integer>}, nil]
+      def build_target_assignment(member_ptr)
+        assignment_ptr = Bindings.rd_kafka_MemberDescription_target_assignment(member_ptr)
+
+        assignment_ptr.null? ? nil : build_assignment(assignment_ptr)
+      end
+
+      # @param group_ptr [FFI::Pointer] pointer to the `rd_kafka_ConsumerGroupDescription_t`
+      # @return [Array<Integer>, nil]
+      def build_authorized_operations(group_ptr)
+        count_ptr = FFI::MemoryPointer.new(:size_t)
+
+        extract_authorized_operations(
+          Bindings.rd_kafka_ConsumerGroupDescription_authorized_operations(group_ptr, count_ptr),
+          count_ptr
+        )
+      end
+
+      # @param error_ptr [FFI::Pointer] pointer to the `rd_kafka_error_t`
+      # @return [RdkafkaError, nil]
+      def build_error(error_ptr)
+        return nil if error_ptr.null?
+
+        RdkafkaError.new(
+          Bindings.rd_kafka_error_code(error_ptr),
+          broker_message: read_string(Bindings.rd_kafka_error_string(error_ptr))
+        )
+      end
+
+      # @param ptr [FFI::Pointer] pointer to a C string
+      # @return [String, nil]
+      def read_string(ptr)
+        ptr.null? ? nil : ptr.read_string
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/describe_result_parsing.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Helpers shared by the describe-topics and describe-cluster reports to copy nodes and
+    # authorized operations out of event-owned memory.
+    # @private
+    module DescribeResultParsing
+      private
+
+      # @param node_ptr [FFI::Pointer] pointer to a `rd_kafka_Node_t`
+      # @return [Hash, nil] node with `:id`, `:host`, `:port` and `:rack` or nil when NULL
+      def extract_node(node_ptr)
+        return nil if node_ptr.null?
+
+        host_ptr = Bindings.rd_kafka_Node_host(node_ptr)
+        rack_ptr = Bindings.rd_kafka_Node_rack(node_ptr)
+
+        {
+          id: Bindings.rd_kafka_Node_id(node_ptr),
+          host: host_ptr.null? ? nil : host_ptr.read_string,
+          port: Bindings.rd_kafka_Node_port(node_ptr),
+          rack: rack_ptr.null? ? nil : rack_ptr.read_string
+        }
+      end
+
+      # @param array_ptr [FFI::Pointer] pointer to a `rd_kafka_Node_t*` array
+      # @param count_ptr [FFI::MemoryPointer] size_t holding the array length
+      # @return [Array<Hash>] nodes as returned by {#extract_node}
+      def extract_nodes(array_ptr, count_ptr)
+        return [] if array_ptr.null?
+
+        array_ptr.read_array_of_pointer(count_ptr.read(:size_t)).map { |node_ptr| extract_node(node_ptr) }
+      end
+
+      # @param array_ptr [FFI::Pointer] pointer to a `rd_kafka_AclOperation_t` array
+      # @param count_ptr [FFI::MemoryPointer] size_t holding the array length
+      # @return [Array<Integer>, nil] `Bindings::RD_KAFKA_ACL_OPERATION_*` codes or nil when the
+      #   authorized operations were not requested
+      def extract_authorized_operations(array_ptr, count_ptr)
+        return nil if array_ptr.null?
+
+        array_ptr.read_array_of_int(count_ptr.read(:size_t))
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/describe_topics_handle.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Handle for describe topics operation
+    class DescribeTopicsHandle < AbstractHandle
+      layout :pending, :bool,
+        :response, :int
+
+      # @return [String] the name of the operation
+      def operation_name
+        "describe topics"
+      end
+
+      # @return [DescribeTopicsReport] report prepared by the background event callback, with
+      #   the described topics.
+      def create_result
+        prepared_result
+      end
+
+      # Raises an error if the operation failed
+      # @raise [RdkafkaError]
+      def raise_error
+        raise RdkafkaError.new(
+          self[:response],
+          broker_message: broker_message
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/describe_topics_report.rb

@@ -0,0 +1,118 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Report for describe topics operation result
+    class DescribeTopicsReport
+      include DescribeResultParsing
+
+      # Described topics, one per requested topic name. Each entry is a hash with:
+      #   - `:name` [String] the topic name
+      #   - `:topic_id` [String, nil] the topic id (KIP-516), base64 encoded, or `nil` when the
+      #     topic could not be described
+      #   - `:is_internal` [Boolean] whether the topic is internal (e.g. `__consumer_offsets`)
+      #   - `:partitions` [Array<Hash>] each with `:partition` [Integer], `:leader` [Hash, nil] and
+      #     `:replicas` / `:isr` [Array<Hash>]; nodes have `:id`, `:host`, `:port` and `:rack`
+      #   - `:authorized_operations` [Array<Integer>, nil] `Bindings::RD_KAFKA_ACL_OPERATION_*`
+      #     codes, `nil` unless requested with `include_authorized_operations: true`
+      #   - `:error` [RdkafkaError, nil] per-topic error (e.g. `unknown_topic_or_part`), `nil`
+      #     on success
+      # @return [Array<Hash>]
+      attr_reader :topics
+
+      # @param result_ptr [FFI::Pointer] pointer to the `rd_kafka_DescribeTopics_result_t`
+      def initialize(result_ptr)
+        @topics = []
+
+        return if result_ptr.null?
+
+        count_ptr = FFI::MemoryPointer.new(:size_t)
+        array_ptr = Bindings.rd_kafka_DescribeTopics_result_topics(result_ptr, count_ptr)
+
+        return if array_ptr.null?
+
+        array_ptr.read_array_of_pointer(count_ptr.read(:size_t)).each do |topic_ptr|
+          @topics << extract_topic(topic_ptr)
+        end
+      end
+
+      private
+
+      # @param topic_ptr [FFI::Pointer] pointer to a `rd_kafka_TopicDescription_t`
+      # @return [Hash]
+      def extract_topic(topic_ptr)
+        name_ptr = Bindings.rd_kafka_TopicDescription_name(topic_ptr)
+        operations_count_ptr = FFI::MemoryPointer.new(:size_t)
+
+        {
+          name: name_ptr.null? ? nil : name_ptr.read_string,
+          topic_id: extract_topic_id(topic_ptr),
+          is_internal: Bindings.rd_kafka_TopicDescription_is_internal(topic_ptr) != 0,
+          partitions: extract_partitions(topic_ptr),
+          authorized_operations: extract_authorized_operations(
+            Bindings.rd_kafka_TopicDescription_authorized_operations(topic_ptr, operations_count_ptr),
+            operations_count_ptr
+          ),
+          error: extract_error(topic_ptr)
+        }
+      end
+
+      # librdkafka reports the zero UUID for a topic it could not describe
+      # @param topic_ptr [FFI::Pointer] pointer to a `rd_kafka_TopicDescription_t`
+      # @return [String, nil]
+      def extract_topic_id(topic_ptr)
+        uuid_ptr = Bindings.rd_kafka_TopicDescription_topic_id(topic_ptr)
+
+        return nil if uuid_ptr.null?
+        return nil if Bindings.rd_kafka_Uuid_most_significant_bits(uuid_ptr).zero? &&
+          Bindings.rd_kafka_Uuid_least_significant_bits(uuid_ptr).zero?
+
+        string_ptr = Bindings.rd_kafka_Uuid_base64str(uuid_ptr)
+        string_ptr.null? ? nil : string_ptr.read_string
+      end
+
+      # @param topic_ptr [FFI::Pointer] pointer to a `rd_kafka_TopicDescription_t`
+      # @return [Array<Hash>]
+      def extract_partitions(topic_ptr)
+        count_ptr = FFI::MemoryPointer.new(:size_t)
+        array_ptr = Bindings.rd_kafka_TopicDescription_partitions(topic_ptr, count_ptr)
+
+        return [] if array_ptr.null?
+
+        array_ptr.read_array_of_pointer(count_ptr.read(:size_t)).map do |partition_ptr|
+          replicas_count_ptr = FFI::MemoryPointer.new(:size_t)
+          isr_count_ptr = FFI::MemoryPointer.new(:size_t)
+
+          {
+            partition: Bindings.rd_kafka_TopicPartitionInfo_partition(partition_ptr),
+            leader: extract_node(Bindings.rd_kafka_TopicPartitionInfo_leader(partition_ptr)),
+            replicas: extract_nodes(
+              Bindings.rd_kafka_TopicPartitionInfo_replicas(partition_ptr, replicas_count_ptr),
+              replicas_count_ptr
+            ),
+            isr: extract_nodes(
+              Bindings.rd_kafka_TopicPartitionInfo_isr(partition_ptr, isr_count_ptr),
+              isr_count_ptr
+            )
+          }
+        end
+      end
+
+      # The error is owned by the result, so it is copied and not destroyed here.
+      # @param topic_ptr [FFI::Pointer] pointer to a `rd_kafka_TopicDescription_t`
+      # @return [RdkafkaError, nil]
+      def extract_error(topic_ptr)
+        error_ptr = Bindings.rd_kafka_TopicDescription_error(topic_ptr)
+
+        return nil if error_ptr.null?
+
+        code = Bindings.rd_kafka_error_code(error_ptr)
+
+        return nil if code == Bindings::RD_KAFKA_RESP_ERR_NO_ERROR
+
+        string_ptr = Bindings.rd_kafka_error_string(error_ptr)
+        RdkafkaError.new(code, broker_message: string_ptr.null? ? nil : string_ptr.read_string)
+      end
+    end
+  end
+end