rdkafka 0.29.1 → 0.30.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9df14d82195673f4747cc9a5d20eeb705a547295b1fa25f6554a5f9fac415ff0
-  data.tar.gz: d98858ef1d0c6694acd2de1726e0727a17aae1694adf2d75ee9890bdf8608d57
+  metadata.gz: 9a5ff1b8b0535c2a9781c7a629907b034e372fd69707895be94867aacdf70c69
+  data.tar.gz: 8594c653516b17b414ff6c1e02709d1f00e776caa497980c699cb2ac8ea131d0
 SHA512:
-  metadata.gz: 82715ad307088ce3559789b9445c968fb1dcbfa2e87ba70b8e90c786ff92066c37497b8592153e8512ed297706a8499d38e2c04754129390665478a3174bcf0e
-  data.tar.gz: 00fd0221b7afde5cf110437d3b83f7672b820b3ebc596d2ad7126c248f31bb89b35d29ced75c24da0a17af2828a2fc4f7feb4397cdcfe6d04bf78882f2ee02b6
+  metadata.gz: 88d5515df49496899ecbb474ed379b2405b4aacbf8a7cd2c10b9356276bdeb862091a450b87713dca495d4101e585717b459a96b2a1bdffb55ef80caf948165e
+  data.tar.gz: 17743522771d227ad2f729a87a6c6976322908221cbebb74b5af7a46d7218cf7e2775bda7b031ace347ceccad1f20d7035c486b23cc6158c1d5ed29e2ecee56e

data/CHANGELOG.md

@@ -1,5 +1,26 @@
 # Rdkafka Changelog
 
+## 0.30.0 (2026-09-25)
+- [Feature] Add `Admin#alter_consumer_group_offsets` and `Admin#delete_consumer_group_offsets` to set or clear a consumer group's committed offsets from the admin client.
+- [Enhancement] Bump the bundled zlib to `1.3.2`.
+- [Enhancement] Bump librdkafka to `2.15.0`.
+- [Enhancement] Bump the bundled MIT Kerberos (krb5) to `1.22.2`.
+- [Enhancement] Add `Rdkafka::Config.partitioner_key_uses_bytesize` to hash partition keys by byte length, matching other Kafka clients for multibyte keys. Defaults to `false`.
+- [Maintenance] Derive the zlib `CHECKSUMS` entry from `ZLIB_VERSION`.
+- [Maintenance] Drop the unused `dist/openssl-3.0.16.tar.gz` build cache.
+- [Fix] Honor the `isolation_level:` argument to `Admin#list_offsets`, which was silently ignored.
+- [Fix] Return correct results for every item of a multi-item topic, partition, group or ACL admin request, not just the first.
+- [Maintenance] Fix a use-after-free in the multi-item admin integration test that read result-name pointers after librdkafka had destroyed the background event, which made it flaky on newer glibc (e.g. Debian trixie).
+- [Maintenance] Stabilize consumer specs on slow CI runners: wait longer for a partition assignment, raise the `TestTopics.create` admin timeout (a timeout there also tripped the leaked-handle guard), and let the long-running consumption spec drain its backlog instead of stopping at a fixed 60s.
+- [Maintenance] Raise the `TestTopics.create` admin timeout to 30s so a loaded CI runner no longer hits `create topic timed out` (which also tripped the leaked-handle guard).
+
+## 0.29.2 (2026-09-11)
+- [Fix] Close live clients from an `at_exit` hook before Ruby's shutdown finalization, so librdkafka is no longer `dlclose`d while its native threads are still running (which could segfault on exit).
+- [Fix] Make client construction exception-safe and destroy the native handle when setup fails after `rd_kafka_new`, so a later error no longer orphans the native client.
+- [Fix] Fix a double-free in `Consumer#poll_batch`/`#poll_batch_nb` that could abort the process when building a message raised a non-`RdkafkaError`.
+- [Fix] Free native resources when `Admin#describe_configs`/`#incremental_alter_configs` raise while building their request (e.g. a non-String resource name).
+- [Fix] Raise `ConfigError` instead of segfaulting when `Admin#describe_configs`/`#incremental_alter_configs` get an empty resource name or a negative resource type.
+
 ## 0.29.1 (2026-09-04)
 - [Enhancement] Add `Admin#delete_records` to delete all messages in a partition up to a given offset.
 - [Enhancement] Add `Admin#list_consumer_groups` to list the cluster's consumer groups.
@@ -8,11 +29,12 @@
 - [Fix] Make `NativeKafka#close` fork-aware so a forked child no longer segfaults on exit.
 - [Fix] Stabilize the `to_native_tpl` leak integration spec.
 - [Fix] Stabilize the flaky partitions count cache statistics spec.
+- [Fix] Use global allocator in topic_partition_list for pointer freed by librdkafka.
 
 ## 0.29.0 (2026-07-10)
 - [Enhancement] Bump librdkafka to `2.14.2`
-- [Enhancement] Add `Consumer#metadata` and `Producer#metadata`, mirroring `Admin#metadata`, so cluster/topic metadata can be fetched from an existing consumer or producer handle without opening a dedicated admin connection.
-- [Enhancement] Support offset-commit metadata: `Consumer#store_offset(message, metadata)` stores an optional metadata string alongside the offset, `Consumer::Partition#metadata` exposes it, and `TopicPartitionList` marshals it to and from the native list (read back via `Consumer#committed`).
+- [Enhancement] Add `Consumer#metadata` and `Producer#metadata`, mirroring `Admin#metadata`, to fetch cluster metadata without a separate admin connection.
+- [Enhancement] Support offset-commit metadata via `Consumer#store_offset(message, metadata)`, readable through `Consumer::Partition#metadata` and `Consumer#committed`.
 - [Enhancement] Expose `replicas` and `isrs` (in-sync replica broker ids) on each partition in topic metadata; both were previously dropped from the `Metadata#topics` partition hashes.
 - [Enhancement] Extract the admin background-event result handlers into one class per operation under `lib/rdkafka/callbacks/`. Internal reorganization with no API or behavior change.
 - [Enhancement] Reuse per-thread scratch pointers in `Consumer::Headers.from_native`, removing the per-message native allocations from the consumer hot path.
@@ -20,32 +42,32 @@
 - [Fix] Cache the partition count for a missing topic, so `produce` with a `partition_key` to a missing topic no longer re-runs a blocking metadata query on every message.
 - [Fix] Let `PartitionsCountCache` adopt a lower partition count once the cached entry has expired, so a topic recreated with fewer partitions no longer fails `produce` until process restart.
 - [Fix] Surface per-message build errors inline in `Consumer#poll_batch`/`#poll_batch_nb` instead of discarding the batch, preserving the messages that built successfully.
-- [Fix] Bound the `Metadata` retry loop by a ~5s wall-clock budget with a floor of 3 attempts, cap the backoff at 1s, and stop overwriting the request timeout with the backoff value, which could block a synchronous fetch for minutes.
+- [Fix] Bound the `Metadata` retry loop to about 5 seconds, so a synchronous metadata fetch can no longer block for minutes.
 - [Fix] Synchronize `AbstractHandle::REGISTRY` mutations with a mutex, which could otherwise lose a write on JRuby and leave a handle unregistered or leaked.
-- [Fix] Fix the NULL background-queue cleanup branches in `Admin#delete_group`, `Admin#delete_acl` and `Admin#describe_acl`, which raised a `NameError` and leaked the native request object instead of raising `ConfigError`.
+- [Fix] Raise `ConfigError` instead of `NameError` in `Admin#delete_group`, `#delete_acl` and `#describe_acl` when the background queue is unavailable.
 - [Fix] Actually populate `ConfigBindingResult#synonyms`. The synonym loop built the synonym binding objects and discarded them, so `#synonyms` was always an empty array; they are now appended.
 - [Fix] Stop leaking the native `rd_kafka_topic_conf_t` in `Producer#set_topic_config` when a per-topic config value is rejected.
-- [Fix] Add the missing `closed_consumer_check` to `Consumer#position`, the only public consumer method without one; calling it on a closed consumer now raises a consistent `ClosedConsumerError` instead of failing deeper in the call.
+- [Fix] Raise `ClosedConsumerError` from `Consumer#position` on a closed consumer, like the other consumer methods.
 - [Fix] Guard the OAuthBearer token refresh callback against a NULL client pointer before calling `rd_kafka_name` on it.
-- [Fix] Make the `Consumer` GC finalizer close the consumer and destroy its consumer queue, not just the native client, preventing a hang or leak when a consumer is collected without an explicit `close`.
+- [Fix] Fully close a consumer collected without an explicit `close`, preventing a hang or leak.
 - [Fix] Free the librdkafka-allocated string in `Consumer#cluster_id` and `Consumer#member_id`, and fix the `rd_kafka_clusterid` arity. `Consumer#cluster_id` now accepts a `timeout_ms`.
 - [Fix] Attach `rd_kafka_query_watermark_offsets` with `blocking: true` so it releases the GVL; it previously froze every other Ruby thread for up to `timeout_ms`.
 - [Fix] Stop leaking the native `rd_kafka_conf_t` when client creation fails, a multi-KB leak per failed attempt for supervisors retrying on transient SASL/SSL misconfiguration.
 - [Fix] Stop `Metadata` from leaking the native metadata struct on every retried fetch; each attempt now frees its own native resources.
 - [Fix] Stop `Producer#produce` from orphaning the delivery handle in the process-global registry when it fails after registering it.
 - [Fix] Guard the message delivery callback so a raising user `delivery_callback` can no longer leave the handle locked or crash the producer; exceptions are now logged and swallowed.
-- [Fix] Stop `describe_configs`, `incremental_alter_configs` and `list_offsets` from leaking handles and native resources when their arguments are rejected; all three now validate input before allocating.
-- [Fix] Destroy the native topic-partition list in `TopicPartitionList#to_native_tpl` when population fails partway (e.g. an invalid partition). The caller-owned list was leaked on any exception mid-build; it is now destroyed before the error propagates.
+- [Fix] Stop `describe_configs`, `incremental_alter_configs` and `list_offsets` from leaking native resources when their arguments are rejected.
+- [Fix] Stop `TopicPartitionList#to_native_tpl` from leaking the native list when building it fails partway.
 - [Fix] Raise instead of silently dropping a rejected `incremental_alter_configs` entry, which previously left the alter request reporting success.
 - [Fix] Allocate the admin result-count out-parameter as `:size_t` instead of `:int32`, fixing a 4-byte overflow on every admin result parse.
 - [Fix] Stabilize the flaky `Consumer#lag` "calculates the consumer lag" spec on overloaded CI.
-- [Fix] Resolve admin operation handles from the event error when an operation fails at the operation level (e.g. brokers unreachable), instead of blocking until `wait` timed out and discarding the real error.
-- [Fix] Destroy admin API background events after processing; they were never destroyed, so every admin operation leaked its whole result event. The internal FFI struct fields on admin handles were removed as part of this - use `handle.wait` and the returned report objects, whose interfaces are unchanged.
+- [Fix] Return the real error from an admin operation that fails at the operation level (e.g. brokers unreachable) instead of blocking until `wait` times out.
+- [Fix] Stop every admin operation from leaking its result event. The internal FFI struct fields on admin handles are removed; use `handle.wait` and the returned report objects, which are unchanged.
 
 ## 0.28.0 (2026-06-03)
 - [Enhancement] Bump librdkafka to `2.14.1`
 - [Enhancement] Replace version-gated `Warning[:performance]` with dynamic `Warning.categories` opt-in in spec helper so new Ruby warning categories are automatically enabled without manual updates.
-- [Enhancement] `poll_batch` and `poll_batch_nb` now return error events inline as `RdkafkaError` objects rather than raising on the first error. The return type is `Array<Message, RdkafkaError>` and callers are responsible for handling errors in the result.
+- [Enhancement] `poll_batch` and `poll_batch_nb` now return errors inline as `RdkafkaError` objects instead of raising on the first one. The return type is `Array<Message, RdkafkaError>`, so callers must handle errors in the result.
 
 ## 0.27.0 (2026-05-07)
 - [Feature] Add `Consumer#poll_batch(timeout_ms, max_items:)` and `Consumer#poll_batch_nb(timeout_ms, max_items:)` for batch message polling via `rd_kafka_consume_batch_queue`.

data/README.md

@@ -163,6 +163,7 @@ bundle exec rake produce_messages
 
 | rdkafka-ruby | librdkafka | patches |
 |-|-|-|
+| 0.30.x (2026-09-25) | 2.15.0 (2026-06-30) | yes |
 | 0.29.x (2026-07-10) | 2.14.2 (2026-06-03) | yes |
 | 0.28.x (2026-06-03) | 2.14.1 (2026-04-15) | yes |
 | 0.27.x (2026-05-07) | 2.14.0 (2026-04-01) | yes |

data/lib/rdkafka/admin/alter_consumer_group_offsets_handle.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Handle for alter consumer group offsets operation
+    class AlterConsumerGroupOffsetsHandle < AbstractHandle
+      layout :pending, :bool,
+        :response, :int
+
+      # @return [String] the name of the operation
+      def operation_name
+        "alter consumer group offsets"
+      end
+
+      # Creates the result report
+      # @return [AlterConsumerGroupOffsetsReport]
+      def create_result
+        prepared_result
+      end
+
+      # Raises an error if the operation failed
+      # @raise [RdkafkaError]
+      def raise_error
+        raise RdkafkaError.new(
+          self[:response],
+          broker_message: broker_message
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/alter_consumer_group_offsets_report.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Report for alter consumer group offsets operation result
+    class AlterConsumerGroupOffsetsReport
+      # @return [String, nil] the group the operation ran against
+      attr_reader :group_name
+
+      # @return [String, nil] group level error message, if any
+      attr_reader :error_string
+
+      # Per partition outcomes, each with `:topic`, `:partition`, `:offset` and `:error`
+      # (an `RdkafkaError` or nil). Partitions are reported individually because Kafka can
+      # fail some and accept others within a single request.
+      # @return [Array<Hash>]
+      attr_reader :partitions
+
+      # @param error_string [FFI::Pointer] pointer to the group level error string
+      # @param result_name [FFI::Pointer] pointer to the group name
+      # @param partitions_ptr [FFI::Pointer] pointer to the per partition result list
+      def initialize(error_string, result_name, partitions_ptr = FFI::Pointer::NULL)
+        @error_string = error_string.read_string if error_string != FFI::Pointer::NULL
+        @group_name = result_name.read_string if result_name != FFI::Pointer::NULL
+        @partitions = extract_partitions(partitions_ptr)
+      end
+
+      private
+
+      # @param partitions_ptr [FFI::Pointer] pointer to a native topic partition list
+      # @return [Array<Hash>]
+      def extract_partitions(partitions_ptr)
+        return [] if partitions_ptr.nil? || partitions_ptr.null?
+
+        native = Bindings::TopicPartitionList.new(partitions_ptr)
+
+        Array.new(native[:cnt]) do |i|
+          tp = Bindings::TopicPartition.new(native[:elems] + (i * Bindings::TopicPartition.size))
+
+          {
+            topic: tp[:topic],
+            partition: tp[:partition],
+            offset: tp[:offset],
+            error: tp[:err].zero? ? nil : RdkafkaError.new(tp[:err])
+          }
+        end
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/delete_consumer_group_offsets_handle.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Handle for delete consumer group offsets operation
+    class DeleteConsumerGroupOffsetsHandle < AbstractHandle
+      layout :pending, :bool,
+        :response, :int
+
+      # @return [String] the name of the operation
+      def operation_name
+        "delete consumer group offsets"
+      end
+
+      # Creates the result report
+      # @return [DeleteConsumerGroupOffsetsReport]
+      def create_result
+        prepared_result
+      end
+
+      # Raises an error if the operation failed
+      # @raise [RdkafkaError]
+      def raise_error
+        raise RdkafkaError.new(
+          self[:response],
+          broker_message: broker_message
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/delete_consumer_group_offsets_report.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Report for delete consumer group offsets operation result
+    class DeleteConsumerGroupOffsetsReport
+      # @return [String, nil] the group the operation ran against
+      attr_reader :group_name
+
+      # @return [String, nil] group level error message, if any
+      attr_reader :error_string
+
+      # Per partition outcomes, each with `:topic`, `:partition`, `:offset` and `:error`
+      # (an `RdkafkaError` or nil). Partitions are reported individually because Kafka can
+      # fail some and accept others within a single request.
+      # @return [Array<Hash>]
+      attr_reader :partitions
+
+      # @param error_string [FFI::Pointer] pointer to the group level error string
+      # @param result_name [FFI::Pointer] pointer to the group name
+      # @param partitions_ptr [FFI::Pointer] pointer to the per partition result list
+      def initialize(error_string, result_name, partitions_ptr = FFI::Pointer::NULL)
+        @error_string = error_string.read_string if error_string != FFI::Pointer::NULL
+        @group_name = result_name.read_string if result_name != FFI::Pointer::NULL
+        @partitions = extract_partitions(partitions_ptr)
+      end
+
+      private
+
+      # @param partitions_ptr [FFI::Pointer] pointer to a native topic partition list
+      # @return [Array<Hash>]
+      def extract_partitions(partitions_ptr)
+        return [] if partitions_ptr.nil? || partitions_ptr.null?
+
+        native = Bindings::TopicPartitionList.new(partitions_ptr)
+
+        Array.new(native[:cnt]) do |i|
+          tp = Bindings::TopicPartition.new(native[:elems] + (i * Bindings::TopicPartition.size))
+
+          {
+            topic: tp[:topic],
+            partition: tp[:partition],
+            offset: tp[:offset],
+            error: tp[:err].zero? ? nil : RdkafkaError.new(tp[:err])
+          }
+        end
+      end
+    end
+  end
+end