karafka-rdkafka 0.29.0 → 0.30.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5c14ac2b8e2474a4823811f9a61cf7da21345d41f2a9fa08931b9549c0a67798
-  data.tar.gz: 31196228e2d00eb211c529c6daade3b378c3b707bc39aea8dc93bad5902b1fff
+  metadata.gz: 290a1e79997166534339567e0f46bf560191141a22c754bdd67c656ea68594f4
+  data.tar.gz: 972ca0af039c92a2b68d6767214ede8c06d7da0eaffde72f98c79bfcef0b8943
 SHA512:
-  metadata.gz: 4aca7ee63fdbc488e15c712c80ba94fd4b8a7ef7e82cd5a0923cb0dfe9170a742bd0c9bb98d0c31a431f0d39900c466012c09497514eb5a9a05d5545ddd6b0c8
-  data.tar.gz: ad04a75550357443f46710f5884f56b95cccd6d9ac635e50768dcef33146944bb84e2a7f7fabf5870a48d3442b0f51c2d11f3eadda4251bd2880a0d602fe5477
+  metadata.gz: 33ba78845a20d938ad79d61c859b08303dfe28ffdbf6a0fa5e99184805fa1d842153c0d676ba6fa25640241b71b2188a65d8ac58208e78526a5e2e3eedc433ad
+  data.tar.gz: 92f7639fca0c51e06844f705285df809a8680315d44531953d6d9b0049aff2964a7633933c31522b8fd75f4bb1606cb13f6a1a5c1d097e8a3f7011b5f3e04083

data/CHANGELOG.md

@@ -1,44 +1,78 @@
 # Rdkafka Changelog
 
-## 0.29.0 (2026-09-13)
-- [Enhancement] Bump librdkafka to `2.15.0` (staying on `2.15.0` rather than `2.15.1` so users hitting a regression in `2.15.1` have a stable fallback). The karafka-only statistics filter patch was rebased onto `2.15.0` (upstream grew the config property index from `64 * 35` to `64 * 36`; the patch keeps extending it to `64 * 40`). Ported from rdkafka-ruby (#970).
+## 0.30.0 (2026-09-25)
+- [Fix] Register share consumers in `Rdkafka::Clients` and destroy the native handle when `Config#share_consumer` fails part-way. Share consumers own the native handle directly rather than a `NativeKafka` wrapper, so they bypass `build_native_client` and were missed by the `at_exit` shutdown hook, leaving librdkafka to be unloaded with a live share handle.
+- [Feature] Add preview support for KIP-932 share groups ("Queues for Kafka") via `Rdkafka::ShareConsumer`, created with `Config#share_consumer`. Members consume the same partitions cooperatively with per-record acknowledgements (`:accept`, `:release`, `:reject`) instead of committed offsets. Like the regular `Consumer`, `ShareConsumer` is a thin binding that exposes the librdkafka share primitives (`#subscribe`/`#unsubscribe`/`#subscription`, batch `#poll`, `#acknowledge`, `#commit_sync`, `#commit_async`, `#acknowledgement_commit_callback=`, `#close`) but drives none of them itself - there is no built-in poll loop, acknowledgement strategy or offset/lifecycle orchestration, leaving that to a higher layer such as Karafka. The handle is single-threaded by design, servicing log, statistics and error callbacks from within `#poll`. Requires a broker with share groups enabled (Apache Kafka 4.2.0+); librdkafka marks the feature as preview and not production-ready.
+- [Enhancement] `RdkafkaError.build_from_c` now uses the human readable string librdkafka attaches to the `rd_kafka_error_t` instead of always using `rd_kafka_err2str`, improving diagnostics for every error-pointer path. It also destroys a code-0 error object instead of leaking it on the early return.
+- [Enhancement] Extract the OAuthBearer token buffer/extension plumbing in `Helpers::OAuth` around two private native-call hooks so the share consumer overrides only the hooks instead of duplicating both public methods.
+- [Fix] Make `ShareConsumer#close` safe against in-flight calls from other threads: closing now waits for operations in progress (mirroring `NativeKafka`), preventing a NULL handle from reaching librdkafka or a double destroy when two threads race `close`.
+- [Fix] Keep the `ShareConsumer` native handle and its GC finalizer when `close`/`destroy` report an error. librdkafka leaves the instance intact in that case, so dropping the references beforehand leaked the client permanently; `close` is now retriable after such an error.
+- [Fix] Pin the acknowledgement commit callback `FFI::Function` in the state holder shared with the `ShareConsumer` GC finalizer, preventing a use-after-free when an unclosed consumer is garbage collected and the finalizer-driven destroy flushes pending acknowledgements through it.
+- [Fix] Make `ShareConsumer` fork-aware, mirroring `NativeKafka`. `rd_kafka_share_consumer_new` is backed by the same broker/main threads as any librdkafka client, which do not survive `fork`; the hand-rolled share lifecycle previously had no `@creator_pid` guard, so a process that forked after creating a share consumer would segfault (or deadlock) when the child's GC finalizer or `close` ran `rd_kafka_share_destroy` on the inherited handle. It now records the creating pid and reports `closed?` in any other process, so the finalizer and `close` skip the native teardown for an inherited handle while `with_native` rejects any use of it.
+- [Fix] Scope `ShareConsumer#each`'s `ClosedConsumerError` rescue to the `poll` call only. It previously wrapped the `yield`, so a `ClosedConsumerError` raised from inside the caller's block was swallowed and silently ended iteration; the block's errors now propagate, and only the consumer actually closing ends the loop. Its docs also note that, like `#poll`, `#each` can yield an `RdkafkaError` for a record that fails to build.
+- [Fix] Raise `ArgumentError` for `ShareConsumer#subscribe` without topics. librdkafka treats an empty share subscribe as an unsubscribe and returns no error (unlike the regular consumer), silently dropping the subscription.
+- [Fix] Capture the librdkafka client name onto `ShareConsumer#name` from the OAuthBearer token refresh callback. The share handle has no native name accessor, which made the documented name-based routing of the global oauth callback impossible for share consumers.
+- [Fix] Update `ext/build_common.sh` (precompiled builds) to librdkafka `2.15.0` and its tarball checksum; it still pinned `2.14.1`, whose tarball is no longer vendored.
+- [Fix] Stop the `share_consumer_multi_member` integration spec from failing on redelivery. KIP-932 share groups are documented as at-least-once, so a record whose acquisition lock expires before it is acknowledged may be redelivered with an incremented `delivery_count`; the spec now tracks distinct payloads and fails only when a record is never delivered.
+- [Fix] Apply the same at-least-once tolerance to the `share_consumer_explicit_ack` integration spec, which still asserted exactly one redelivered record. It now requires only the released record to be redelivered (with `delivery_count >= 2`), forbids only the rejected record, and tolerates an accepted record reappearing when its acquisition lock expired before the commit flushed the accept.
+- [Fix] Add the `nofile` ulimit to `docker-compose-ssl.yml`, mirroring `docker-compose.yml`. The SSL CI broker runs the same topic-heavy suite (now enlarged by the share consumer specs) and could otherwise exhaust the default file-descriptor limit under the many short-lived topics.
+- [Fix] Enable KIP-932 share groups on the macOS CI broker, whose KRaft `server.properties` never opted into the share rebalance protocol, so the one broker-backed share consumer spec received zero records on every macOS runner. `CONFLUENT_VERSION` is bumped to `8.3.0` and the download path is derived from it so a Renovate bump cannot desync them.
+- [Enhancement] Add two share consumer integration specs: `share_consumer_implicit_ack_spec` (polling alone accepts the previous batch, and accepted records are not redelivered to a second member) and `share_consumer_multi_topic_spec` (one consumer subscribed to several topics receives every topic's records, attributed to the right topic). Both track distinct payloads so an at-least-once redelivery cannot end the consume loop early.
+- [Fix] Stop the `statistics_unassigned_producer` integration spec from flaking on a metadata-propagation race: the freshly built producer could send before fetching the new topic's metadata, so the produce is now retried on `unknown_topic_or_part` and `leader_not_available`.
+- [Fix] Also stabilize the `statistics_unassigned_producer` integration spec against a delivery wait that outlives its budget. With 1000 partitions librdkafka keeps the message queued while leaders are still being elected, so the handle stays pending and the wait raises `WaitTimeoutError` - a `RuntimeError` the spec's existing code-based rescue could never catch. The produce now uses the default 60s handle wait (matching the sibling `statistics_unassigned_*` specs, instead of a self-imposed 15s) and retries on that timeout too.
+- [Fix] Stabilize the `consumer_memberid_clusterid_leak` integration spec against RSS measurement noise on Ruby 4.0. It now compacts the heap before sampling (matching the `to_native_tpl` leak spec) and the ceiling is 3 MB, still well under the ~5.7 MB a genuine per-call leak would cost.
+- [Note] Share consumers emit the regular consumer statistics JSON (including `cgrp`; there is no share-specific section in librdkafka 2.15.0) through the usual `statistics_callback`, serviced from within `#poll`. The share fetch path bypasses the per-partition stats bookkeeping entirely, so the `topics` section carries no partition entries and the `statistics.unassigned.include` filter needs no share-consumer special-casing; per-partition share metrics are not exposed by librdkafka 2.15.0 at all.
+- [Feature] Add `Admin#alter_consumer_group_offsets` and `Admin#delete_consumer_group_offsets` to set or clear a consumer group's committed offsets from the admin client. Ported from rdkafka-ruby (#983).
+- [Enhancement] Bump librdkafka to `2.15.1` for OpenSSL and libcurl security fixes. It changes how IPv6 addresses are formatted and validated against broker certificates.
+- [Enhancement] Bump the bundled zlib to `1.3.2`.
+- [Enhancement] Bump the bundled MIT Kerberos (krb5) to `1.22.2`. Ported from rdkafka-ruby (#979).
+- [Enhancement] Add `Rdkafka::Config.partitioner_key_uses_bytesize` to hash partition keys by byte length, matching other Kafka clients for multibyte keys. Defaults to `false`. Ported from rdkafka-ruby (#985).
+- [Maintenance] Derive the zlib `CHECKSUMS` entry from `ZLIB_VERSION`.
+- [Maintenance] Drop the unused `dist/openssl-3.0.16.tar.gz` build cache.
+- [Fix] Honor the `isolation_level:` argument to `Admin#list_offsets`, which was silently ignored. Ported from rdkafka-ruby (#983).
+- [Fix] Return correct results for every item of a multi-item topic, partition, group or ACL admin request, not just the first. Ported from rdkafka-ruby (#987).
+- [Maintenance] Fix a use-after-free in the multi-item admin integration test that read result-name pointers after librdkafka had destroyed the background event, which made it flaky on newer glibc (e.g. Debian trixie). Ported from rdkafka-ruby (#991).
+- [Maintenance] Stabilize consumer specs on slow CI runners: wait longer for a partition assignment, raise the `TestTopics.create` admin timeout (a timeout there also tripped the leaked-handle guard), give the non-blocking `poll_nb` message check a generous retry budget instead of a fixed ~2s, and let the long-running consumption spec drain its backlog instead of stopping at a fixed 60s. The assignment wait is tunable via `RDKAFKA_TEST_ASSIGNMENT_TIMEOUT`. Ported from rdkafka-ruby (#990).
+
+## 0.29.0 (2026-09-14)
+- [Enhancement] Bump librdkafka to `2.15.0` (staying on `2.15.0` rather than `2.15.1` so users hitting a regression in `2.15.1` have a stable fallback).
 
 ## 0.28.2 (2026-09-11)
-- [Fix] Close live clients from an `at_exit` hook before Ruby's shutdown finalization, so librdkafka is no longer `dlclose`d while its native threads are still running (which could segfault on exit). Ported from rdkafka-ruby (#964, Alex Selesse).
-- [Fix] Make client construction exception-safe and destroy the native handle when setup fails after `rd_kafka_new`, so a later error no longer orphans the native client. Ported from rdkafka-ruby (#964, Alex Selesse).
-- [Fix] Destroy each polled message exactly once in `Consumer#poll_batch`/`#poll_batch_nb` when building a message raises a non-`RdkafkaError`, fixing a double-free that could abort the process and closing the matching leak window. Ported from rdkafka-ruby (#973).
-- [Fix] Free the background queue, `AdminOptions` and any already-built `ConfigResource`s (and remove the handle) when `Admin#describe_configs`/`#incremental_alter_configs` raise while building native resources (e.g. a non-String resource name). Ported from rdkafka-ruby (#973).
-- [Fix] Raise a clear `ConfigError` instead of segfaulting when `Admin#describe_configs`/`#incremental_alter_configs` are given an empty resource name or a negative resource type (`rd_kafka_ConfigResource_new` returns NULL), or when `rd_kafka_AdminOptions_new` returns NULL. Ported from rdkafka-ruby (#973).
-- [Fix] Allocate the topic-partition metadata string with a process-global malloc instead of an autorelease-off `FFI::MemoryPointer`, so the buffer handed to librdkafka is not freed under it. Ported from rdkafka-ruby (#969, Randy Stauner).
+- [Fix] Close live clients before Ruby shutdown finalization, preventing a possible segfault on exit. Ported from rdkafka-ruby (#964, Alex Selesse).
+- [Fix] Destroy the native handle when client construction fails partway, so it is no longer orphaned. Ported from rdkafka-ruby (#964, Alex Selesse).
+- [Fix] Fix a double-free in `Consumer#poll_batch`/`#poll_batch_nb` that could abort the process when building a message raised a non-`RdkafkaError`. Ported from rdkafka-ruby (#973).
+- [Fix] Free native resources when `Admin#describe_configs`/`#incremental_alter_configs` raise while building their request (e.g. a non-String resource name). Ported from rdkafka-ruby (#973).
+- [Fix] Raise `ConfigError` instead of segfaulting when `Admin#describe_configs`/`#incremental_alter_configs` get an empty resource name or a negative resource type. Ported from rdkafka-ruby (#973).
+- [Fix] Stop the topic-partition metadata string from being freed while librdkafka still uses it. Ported from rdkafka-ruby (#969, Randy Stauner).
 - [Maintenance] Bump the bundled OpenSSL used by the precompiled builds to `3.5.8` (LTS). Ported from rdkafka-ruby (#972, Scott Francis).
 
 ## v0.28.1 (2026-09-04)
-- [Enhancement] Add `Admin#delete_records`, wrapping librdkafka's `DeleteRecords` admin API. Deletes all messages in a partition up to (but not including) a given offset - accepts an integer offset or `:end` to clear all current data. Ported from rdkafka-ruby (#956).
-- [Enhancement] Add `Admin#list_consumer_groups`, wrapping librdkafka's `ListConsumerGroups` admin API. Returns a cluster-wide listing of consumer groups (`group_id`, `is_simple_consumer_group`, state) plus any per-broker errors. Ported from rdkafka-ruby (#955).
+- [Enhancement] Add `Admin#delete_records` to delete messages in a partition up to a given offset, or all current data with `:end`. Ported from rdkafka-ruby (#956).
+- [Enhancement] Add `Admin#list_consumer_groups` for a cluster-wide listing of consumer groups. Ported from rdkafka-ruby (#955).
 - [Fix] Make `NativeKafka#close` fork-aware so a forked child no longer segfaults on exit. Handles record their creator pid and skip the native teardown in any other process.
 - [Fix] Stabilize the flaky partitions count cache statistics spec.
 - [Fix] Stabilize the `to_native_tpl` leak integration spec against RSS measurement noise.
 
 ## v0.28.0 (2026-07-12)
-- [Enhancement] Bump librdkafka to `2.14.2`. Maintenance release: fixes duplicate groups in `ListConsumerGroups` when multiple brokers return the same group, a data race in timers, and bumps bundled OpenSSL/libcurl/zstd/zlib/cJSON dependencies (several CVEs).
-- [Enhancement] Add `Consumer#metadata` and `Producer#metadata`, mirroring `Admin#metadata`, so cluster/topic metadata can be fetched from an existing consumer or producer handle without opening a dedicated admin connection.
-- [Enhancement] Name the failing partition and topic in the `RdkafkaError` raised for per-partition `list_offsets` errors (previously a bare error code), preserving the per-partition context the pre-batching `Consumer#lag` watermark errors carried.
-- [Enhancement] Add `Consumer#list_offsets`, mirroring `Admin#list_offsets`, so offsets can be queried on an existing consumer handle without a dedicated admin connection, and rebuild `Consumer#lag` on top of it as one batched query instead of a roundtrip per partition.
+- [Enhancement] Bump librdkafka to `2.14.2` (maintenance release with bundled dependency CVE fixes and a fix for duplicate groups in `ListConsumerGroups`).
+- [Enhancement] Add `Consumer#metadata` and `Producer#metadata`, mirroring `Admin#metadata`, to fetch cluster metadata without a separate admin connection.
+- [Enhancement] Name the failing topic and partition in `RdkafkaError`s raised for per-partition `list_offsets` errors.
+- [Enhancement] Add `Consumer#list_offsets`, mirroring `Admin#list_offsets`, and compute `Consumer#lag` with one batched query instead of a roundtrip per partition.
 - [Enhancement] Extract the admin background-event result handlers into one class per operation under `lib/rdkafka/callbacks/`. Internal reorganization with no API or behavior change.
 - [Enhancement] Expose `replicas` and `isrs` (in-sync replica broker ids) on each partition in topic metadata; both were previously dropped from the `Metadata#topics` partition hashes.
 - [Enhancement] Reuse per-thread scratch pointers in `Consumer::Headers.from_native`, removing the per-message native allocations from the consumer hot path.
 - [Enhancement] Remove the unused `DeliveryHandle` `:topic_name` struct field and its per-message allocation. Use `DeliveryHandle#topic` or `DeliveryReport#topic_name`, both unchanged.
-- [Fix] Stop `poll_batch`/`poll_batch_nb` from discarding a whole batch when one message fails to build. The failure is now surfaced inline as an `RdkafkaError` in the returned array, preserving the rest of the batch.
+- [Fix] Stop `poll_batch`/`poll_batch_nb` from discarding a whole batch when one message fails to build. The failure is returned inline as an `RdkafkaError`.
 - [Fix] Add the missing `closed_consumer_check` to `Consumer#position`, so it raises a consistent `ClosedConsumerError` like every sibling offset method.
 - [Fix] Stop leaking the native `rd_kafka_topic_conf_t` in `Producer#set_topic_config` when a per-topic config value is rejected.
 - [Fix] Raise instead of silently dropping a rejected `incremental_alter_configs` entry, which previously left the alter request reporting success.
 - [Fix] Let `PartitionsCountCache` adopt a lower partition count once the cached entry has expired, so a topic recreated with fewer partitions no longer fails `produce` until process restart.
-- [Fix] Make the `Consumer` GC finalizer close the consumer and destroy its consumer queue, not just the native client, preventing a hang or leak when a consumer is collected without an explicit `close`.
-- [Fix] Stop `describe_configs`, `incremental_alter_configs` and `list_offsets` from leaking handles and native resources when their arguments are rejected; all three now validate input before allocating.
+- [Fix] Fully close a consumer collected without an explicit `close`, preventing a hang or leak.
+- [Fix] Stop `describe_configs`, `incremental_alter_configs` and `list_offsets` from leaking native resources when their arguments are rejected.
 - [Fix] Destroy the native topic-partition list in `TopicPartitionList#to_native_tpl` when population fails partway, which previously leaked the half-built list.
 - [Fix] Allocate the admin result-count out-parameter as `:size_t` instead of `:int32`, fixing a 4-byte overflow on every admin result parse.
-- [Fix] Destroy admin API background events after processing; they were never destroyed, so every admin operation leaked its whole result event. The internal FFI struct fields on admin handles were removed as part of this - use `handle.wait` and the returned report objects, whose interfaces are unchanged.
-- [Fix] Resolve admin operation handles from the event error when an operation fails at the operation level (e.g. brokers unreachable), instead of blocking until `wait` timed out and discarding the real error.
+- [Fix] Stop every admin operation from leaking its result event. The internal FFI struct fields on admin handles are removed; use `handle.wait` and the returned report objects, which are unchanged.
+- [Fix] Return the real error from an admin operation that fails at the operation level (e.g. brokers unreachable) instead of blocking until `wait` times out.
 - [Fix] Stop leaking the native `rd_kafka_conf_t` when client creation fails, a multi-KB leak per failed attempt for supervisors retrying on transient SASL/SSL misconfiguration.
 - [Fix] Stop `Metadata` from leaking the native metadata struct on every retried fetch; each attempt now frees its own native resources.
 - [Fix] Free the librdkafka-allocated string in `Consumer#cluster_id` and `Consumer#member_id`, and fix the `rd_kafka_clusterid` arity. `Consumer#cluster_id` now accepts a `timeout_ms`.
@@ -46,10 +80,10 @@
 - [Fix] Stop `Producer#produce` from orphaning the delivery handle in the process-global registry when it fails after registering it.
 - [Fix] Attach `rd_kafka_query_watermark_offsets` with `blocking: true` so it releases the GVL; it previously froze every other Ruby thread for up to `timeout_ms`.
 - [Fix] Stabilize the flaky `Consumer#lag` spec on overloaded CI. Backported from rdkafka-ruby (#912).
-- [Fix] Fix the NULL background-queue cleanup branches in `Admin#delete_group`, `Admin#delete_acl` and `Admin#describe_acl`, which raised a `NameError` and leaked the native request object instead of raising `ConfigError`.
+- [Fix] Raise `ConfigError` instead of `NameError` in `Admin#delete_group`, `#delete_acl` and `#describe_acl` when the background queue is unavailable.
 - [Fix] Forward `broker_message` and `instance_name` through `RdkafkaError.build`; both were previously discarded on the `rd_kafka_error_t` pointer path.
 - [Fix] Synchronize `AbstractHandle::REGISTRY` mutations with a mutex, which could otherwise lose a write on JRuby and leave a handle unregistered or leaked.
-- [Fix] Stop the `Metadata` retry loop from clobbering the request timeout and blocking for minutes. The request timeout is left unchanged across retries, the backoff is capped, and the loop is bounded by a ~5s wall-clock budget after a floor of 3 attempts.
+- [Fix] Bound the `Metadata` retry loop to about 5 seconds, so a synchronous metadata fetch can no longer block for minutes.
 - [Fix] Cache the partition count for a missing topic, so `produce` with a `partition_key` to a not-yet-created topic no longer runs a blocking metadata query on every message.
 
 ## 0.27.2 (2026-05-21)

data/README.md

@@ -63,7 +63,8 @@ Contributions should generally be made to the upstream [rdkafka-ruby repository]
 
 | karafka-rdkafka | librdkafka | patches |
 |-|-|-|
-| 0.29.x (2026-09-13) | 2.15.0 (2026-06-30)  | yes |
+| 0.30.x (2026-09-25) | 2.15.1 (2026-09-09)  | yes |
+| 0.29.x (2026-09-14) | 2.15.0 (2026-06-30)  | yes |
 | 0.28.x (2026-07-12) | 2.14.2 (2026-06-03)  | yes |
 | 0.27.x (2026-05-08) | 2.14.1 (2026-04-15)  | yes |
 | 0.26.x (2026-04-11) | 2.14.0 (2026-04-01)  | yes |

data/bin/verify_kafka_warnings

@@ -14,6 +14,8 @@ allowed_patterns=(
   "closing connection"
   "sent a heartbeat request but received error REQUEST_TIMED_OUT"
   "Topic '__consumer_offsets' already exists"
+  "is deprecated. Share groups are controlled by the 'share.version' feature"
+  "is deprecated and will be removed in Kafka 5.0. The following protocol(s) are currently disabled"
 )
 
 # Get all warnings

data/docker-compose-ssl.yml

@@ -7,6 +7,14 @@ services:
       - 9093:9093
     volumes:
       - ./ssl:/etc/kafka/secrets
+
+    # The suite creates hundreds of short-lived topics; each keeps several segment/index file
+    # descriptors open on the broker
+    ulimits:
+      nofile:
+        soft: 262144
+        hard: 262144
+
     environment:
       CLUSTER_ID: kafka-docker-cluster-1
       KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
@@ -25,6 +33,11 @@ services:
       KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
       KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer
 
+      # KIP-932 share groups (queues for Kafka) used by the share consumer specs
+      KAFKA_GROUP_COORDINATOR_REBALANCE_PROTOCOLS: classic,consumer,share
+      KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1
+
       # SSL Configuration
       KAFKA_SSL_KEYSTORE_FILENAME: kafka.server.keystore.jks
       KAFKA_SSL_KEYSTORE_CREDENTIALS: kafka_keystore_creds

data/docker-compose.yml

@@ -6,6 +6,13 @@ services:
     ports:
       - 9092:9092
 
+    # The suite creates hundreds of short-lived topics; each keeps several segment/index file
+    # descriptors open on the broker
+    ulimits:
+      nofile:
+        soft: 262144
+        hard: 262144
+
     environment:
       CLUSTER_ID: kafka-docker-cluster-1
       KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
@@ -23,3 +30,7 @@ services:
       KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
       KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
       KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer
+      # KIP-932 share groups (queues for Kafka) used by the share consumer integration specs
+      KAFKA_GROUP_COORDINATOR_REBALANCE_PROTOCOLS: classic,consumer,share
+      KAFKA_SHARE_COORDINATOR_STATE_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_SHARE_COORDINATOR_STATE_TOPIC_MIN_ISR: 1

data/lib/rdkafka/admin/alter_consumer_group_offsets_handle.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Handle for alter consumer group offsets operation
+    class AlterConsumerGroupOffsetsHandle < AbstractHandle
+      layout :pending, :bool,
+        :response, :int
+
+      # @return [String] the name of the operation
+      def operation_name
+        "alter consumer group offsets"
+      end
+
+      # Creates the result report
+      # @return [AlterConsumerGroupOffsetsReport]
+      def create_result
+        prepared_result
+      end
+
+      # Raises an error if the operation failed
+      # @raise [RdkafkaError]
+      def raise_error
+        raise RdkafkaError.new(
+          self[:response],
+          broker_message: broker_message
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/alter_consumer_group_offsets_report.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Report for alter consumer group offsets operation result
+    class AlterConsumerGroupOffsetsReport
+      # @return [String, nil] the group the operation ran against
+      attr_reader :group_name
+
+      # @return [String, nil] group level error message, if any
+      attr_reader :error_string
+
+      # Per partition outcomes, each with `:topic`, `:partition`, `:offset` and `:error`
+      # (an `RdkafkaError` or nil). Partitions are reported individually because Kafka can
+      # fail some and accept others within a single request.
+      # @return [Array<Hash>]
+      attr_reader :partitions
+
+      # @param error_string [FFI::Pointer] pointer to the group level error string
+      # @param result_name [FFI::Pointer] pointer to the group name
+      # @param partitions_ptr [FFI::Pointer] pointer to the per partition result list
+      def initialize(error_string, result_name, partitions_ptr = FFI::Pointer::NULL)
+        @error_string = error_string.read_string if error_string != FFI::Pointer::NULL
+        @group_name = result_name.read_string if result_name != FFI::Pointer::NULL
+        @partitions = extract_partitions(partitions_ptr)
+      end
+
+      private
+
+      # @param partitions_ptr [FFI::Pointer] pointer to a native topic partition list
+      # @return [Array<Hash>]
+      def extract_partitions(partitions_ptr)
+        return [] if partitions_ptr.nil? || partitions_ptr.null?
+
+        native = Bindings::TopicPartitionList.new(partitions_ptr)
+
+        Array.new(native[:cnt]) do |i|
+          tp = Bindings::TopicPartition.new(native[:elems] + (i * Bindings::TopicPartition.size))
+
+          {
+            topic: tp[:topic],
+            partition: tp[:partition],
+            offset: tp[:offset],
+            error: tp[:err].zero? ? nil : RdkafkaError.new(tp[:err])
+          }
+        end
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/delete_consumer_group_offsets_handle.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Handle for delete consumer group offsets operation
+    class DeleteConsumerGroupOffsetsHandle < AbstractHandle
+      layout :pending, :bool,
+        :response, :int
+
+      # @return [String] the name of the operation
+      def operation_name
+        "delete consumer group offsets"
+      end
+
+      # Creates the result report
+      # @return [DeleteConsumerGroupOffsetsReport]
+      def create_result
+        prepared_result
+      end
+
+      # Raises an error if the operation failed
+      # @raise [RdkafkaError]
+      def raise_error
+        raise RdkafkaError.new(
+          self[:response],
+          broker_message: broker_message
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/delete_consumer_group_offsets_report.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Report for delete consumer group offsets operation result
+    class DeleteConsumerGroupOffsetsReport
+      # @return [String, nil] the group the operation ran against
+      attr_reader :group_name
+
+      # @return [String, nil] group level error message, if any
+      attr_reader :error_string
+
+      # Per partition outcomes, each with `:topic`, `:partition`, `:offset` and `:error`
+      # (an `RdkafkaError` or nil). Partitions are reported individually because Kafka can
+      # fail some and accept others within a single request.
+      # @return [Array<Hash>]
+      attr_reader :partitions
+
+      # @param error_string [FFI::Pointer] pointer to the group level error string
+      # @param result_name [FFI::Pointer] pointer to the group name
+      # @param partitions_ptr [FFI::Pointer] pointer to the per partition result list
+      def initialize(error_string, result_name, partitions_ptr = FFI::Pointer::NULL)
+        @error_string = error_string.read_string if error_string != FFI::Pointer::NULL
+        @group_name = result_name.read_string if result_name != FFI::Pointer::NULL
+        @partitions = extract_partitions(partitions_ptr)
+      end
+
+      private
+
+      # @param partitions_ptr [FFI::Pointer] pointer to a native topic partition list
+      # @return [Array<Hash>]
+      def extract_partitions(partitions_ptr)
+        return [] if partitions_ptr.nil? || partitions_ptr.null?
+
+        native = Bindings::TopicPartitionList.new(partitions_ptr)
+
+        Array.new(native[:cnt]) do |i|
+          tp = Bindings::TopicPartition.new(native[:elems] + (i * Bindings::TopicPartition.size))
+
+          {
+            topic: tp[:topic],
+            partition: tp[:partition],
+            offset: tp[:offset],
+            error: tp[:err].zero? ? nil : RdkafkaError.new(tp[:err])
+          }
+        end
+      end
+    end
+  end
+end

data/lib/rdkafka/admin.rb

@@ -224,6 +224,78 @@ module Rdkafka
       create_topic_handle
     end
 
+    # Sets the committed offsets of a consumer group from outside the group.
+    #
+    # Kafka requires the group to be empty (no active members) for this to be accepted; when it
+    # is not, the broker rejects the request and the handle raises with `group_not_empty` rather
+    # than reporting success.
+    #
+    # @param group_id [String] the group whose offsets we want to set
+    # @param topic_partition_list [TopicPartitionList] partitions with the offsets to commit
+    # @return [AlterConsumerGroupOffsetsHandle] handle that can be used to wait for the result
+    # @raise [RdkafkaError] when altering the offsets fails
+    def alter_consumer_group_offsets(group_id, topic_partition_list)
+      closed_admin_check(__method__)
+
+      native_tpl = topic_partition_list.to_native_tpl
+
+      begin
+        alter_ptr = Rdkafka::Bindings.rd_kafka_AlterConsumerGroupOffsets_new(
+          FFI::MemoryPointer.from_string(group_id),
+          native_tpl
+        )
+      rescue Exception
+        Rdkafka::Bindings.rd_kafka_topic_partition_list_destroy(native_tpl)
+        raise
+      end
+
+      # librdkafka copies the list into the request object, so ours is no longer needed
+      Rdkafka::Bindings.rd_kafka_topic_partition_list_destroy(native_tpl)
+
+      enqueue_group_offsets_op(
+        request_ptr: alter_ptr,
+        handle: AlterConsumerGroupOffsetsHandle.new,
+        admin_op: Rdkafka::Bindings::RD_KAFKA_ADMIN_OP_ALTERCONSUMERGROUPOFFSETS,
+        destroy: Rdkafka::Bindings.method(:rd_kafka_AlterConsumerGroupOffsets_destroy),
+        enqueue: Rdkafka::Bindings.method(:rd_kafka_AlterConsumerGroupOffsets)
+      )
+    end
+
+    # Deletes the committed offsets of specific partitions of a consumer group, leaving the
+    # group and its remaining offsets in place.
+    #
+    # This is the partition scoped counterpart of {#delete_group}, which removes the whole group.
+    #
+    # @param group_id [String] the group whose offsets we want to delete
+    # @param topic_partition_list [TopicPartitionList] partitions whose offsets to delete
+    # @return [DeleteConsumerGroupOffsetsHandle] handle that can be used to wait for the result
+    # @raise [RdkafkaError] when deleting the offsets fails
+    def delete_consumer_group_offsets(group_id, topic_partition_list)
+      closed_admin_check(__method__)
+
+      native_tpl = topic_partition_list.to_native_tpl
+
+      begin
+        delete_ptr = Rdkafka::Bindings.rd_kafka_DeleteConsumerGroupOffsets_new(
+          FFI::MemoryPointer.from_string(group_id),
+          native_tpl
+        )
+      rescue Exception
+        Rdkafka::Bindings.rd_kafka_topic_partition_list_destroy(native_tpl)
+        raise
+      end
+
+      Rdkafka::Bindings.rd_kafka_topic_partition_list_destroy(native_tpl)
+
+      enqueue_group_offsets_op(
+        request_ptr: delete_ptr,
+        handle: DeleteConsumerGroupOffsetsHandle.new,
+        admin_op: Rdkafka::Bindings::RD_KAFKA_ADMIN_OP_DELETECONSUMERGROUPOFFSETS,
+        destroy: Rdkafka::Bindings.method(:rd_kafka_DeleteConsumerGroupOffsets_destroy),
+        enqueue: Rdkafka::Bindings.method(:rd_kafka_DeleteConsumerGroupOffsets)
+      )
+    end
+
     # Deletes a consumer group
     #
     # @param group_id [String] the group id to delete
@@ -1134,6 +1206,54 @@ module Rdkafka
 
     private
 
+    # Shared enqueue path for the two consumer group offset admin operations. They differ only
+    # in their C entry points, so keeping one body avoids two near identical copies of the
+    # queue/handle/options dance and its cleanup.
+    #
+    # @param request_ptr [FFI::Pointer] the op specific request object
+    # @param handle [AbstractHandle] the handle to register and return
+    # @param admin_op [Integer] the RD_KAFKA_ADMIN_OP_* constant for this operation
+    # @param destroy [Method] bound function used to free the request object
+    # @param enqueue [Method] bound function used to enqueue the request
+    # @return [AbstractHandle] the registered handle
+    def enqueue_group_offsets_op(request_ptr:, handle:, admin_op:, destroy:, enqueue:)
+      array_ptr = FFI::MemoryPointer.new(:pointer)
+      array_ptr.write_array_of_pointer([request_ptr])
+
+      queue_ptr = @native_kafka.with_inner do |inner|
+        Rdkafka::Bindings.rd_kafka_queue_get_background(inner)
+      end
+
+      if queue_ptr.null?
+        destroy.call(request_ptr)
+        raise Rdkafka::Config::ConfigError.new("rd_kafka_queue_get_background was NULL")
+      end
+
+      handle[:pending] = true
+      handle[:response] = Rdkafka::Bindings::RD_KAFKA_PARTITION_UA
+      handle.class.register(handle)
+
+      admin_options_ptr = @native_kafka.with_inner do |inner|
+        Rdkafka::Bindings.rd_kafka_AdminOptions_new(inner, admin_op)
+      end
+      Rdkafka::Bindings.rd_kafka_AdminOptions_set_opaque(admin_options_ptr, handle.to_ptr)
+
+      begin
+        @native_kafka.with_inner do |inner|
+          enqueue.call(inner, array_ptr, 1, admin_options_ptr, queue_ptr)
+        end
+      rescue Exception
+        handle.class.remove(handle.to_ptr.address)
+        raise
+      ensure
+        Rdkafka::Bindings.rd_kafka_AdminOptions_destroy(admin_options_ptr)
+        Rdkafka::Bindings.rd_kafka_queue_destroy(queue_ptr)
+        destroy.call(request_ptr)
+      end
+
+      handle
+    end
+
     # Checks if the admin is closed and raises an error if so
     # @param method [Symbol] name of the calling method for error context
     # @raise [ClosedAdminError] when the admin is closed