rdkafka 0.29.2 → 0.30.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f2acf73116281ea22d96af51b56fecc615cb32616d25834a5a990fae1213a1ba
-  data.tar.gz: 04c06f93561b22719261fab54fc0191f1a6edba01926596124f430adcc11d7d5
+  metadata.gz: 9a5ff1b8b0535c2a9781c7a629907b034e372fd69707895be94867aacdf70c69
+  data.tar.gz: 8594c653516b17b414ff6c1e02709d1f00e776caa497980c699cb2ac8ea131d0
 SHA512:
-  metadata.gz: 7ea74276532d8cd48ee913a46166497ace6426bd8859eb41d72f5407bc5a1eb4273bbe2d82a793618e3420a956b95a86919e416c46e0ac657d8b8056167dcdc0
-  data.tar.gz: 20e3c8ac1bb1113673f5af90e977dc3e59e6e6b9f9b0b550e1f6b1a9796bd19b1915d8ba8e345e5cfb0e574d61e8fe354788e0f3445ee0333a8634c7f81e4d97
+  metadata.gz: 88d5515df49496899ecbb474ed379b2405b4aacbf8a7cd2c10b9356276bdeb862091a450b87713dca495d4101e585717b459a96b2a1bdffb55ef80caf948165e
+  data.tar.gz: 17743522771d227ad2f729a87a6c6976322908221cbebb74b5af7a46d7218cf7e2775bda7b031ace347ceccad1f20d7035c486b23cc6158c1d5ed29e2ecee56e

data/CHANGELOG.md

@@ -1,11 +1,25 @@
 # Rdkafka Changelog
 
+## 0.30.0 (2026-09-25)
+- [Feature] Add `Admin#alter_consumer_group_offsets` and `Admin#delete_consumer_group_offsets` to set or clear a consumer group's committed offsets from the admin client.
+- [Enhancement] Bump the bundled zlib to `1.3.2`.
+- [Enhancement] Bump librdkafka to `2.15.0`.
+- [Enhancement] Bump the bundled MIT Kerberos (krb5) to `1.22.2`.
+- [Enhancement] Add `Rdkafka::Config.partitioner_key_uses_bytesize` to hash partition keys by byte length, matching other Kafka clients for multibyte keys. Defaults to `false`.
+- [Maintenance] Derive the zlib `CHECKSUMS` entry from `ZLIB_VERSION`.
+- [Maintenance] Drop the unused `dist/openssl-3.0.16.tar.gz` build cache.
+- [Fix] Honor the `isolation_level:` argument to `Admin#list_offsets`, which was silently ignored.
+- [Fix] Return correct results for every item of a multi-item topic, partition, group or ACL admin request, not just the first.
+- [Maintenance] Fix a use-after-free in the multi-item admin integration test that read result-name pointers after librdkafka had destroyed the background event, which made it flaky on newer glibc (e.g. Debian trixie).
+- [Maintenance] Stabilize consumer specs on slow CI runners: wait longer for a partition assignment, raise the `TestTopics.create` admin timeout (a timeout there also tripped the leaked-handle guard), and let the long-running consumption spec drain its backlog instead of stopping at a fixed 60s.
+- [Maintenance] Raise the `TestTopics.create` admin timeout to 30s so a loaded CI runner no longer hits `create topic timed out` (which also tripped the leaked-handle guard).
+
 ## 0.29.2 (2026-09-11)
 - [Fix] Close live clients from an `at_exit` hook before Ruby's shutdown finalization, so librdkafka is no longer `dlclose`d while its native threads are still running (which could segfault on exit).
 - [Fix] Make client construction exception-safe and destroy the native handle when setup fails after `rd_kafka_new`, so a later error no longer orphans the native client.
-- [Fix] Destroy each polled message exactly once in `Consumer#poll_batch`/`#poll_batch_nb` when building a message raises a non-`RdkafkaError`, fixing a double-free that could abort the process and closing the matching leak window.
-- [Fix] Free the background queue, `AdminOptions` and any already-built `ConfigResource`s (and remove the handle) when `Admin#describe_configs`/`#incremental_alter_configs` raise while building native resources (e.g. a non-String resource name).
-- [Fix] Raise a clear `ConfigError` instead of segfaulting when `Admin#describe_configs`/`#incremental_alter_configs` are given an empty resource name or a negative resource type (`rd_kafka_ConfigResource_new` returns NULL), or when `rd_kafka_AdminOptions_new` returns NULL.
+- [Fix] Fix a double-free in `Consumer#poll_batch`/`#poll_batch_nb` that could abort the process when building a message raised a non-`RdkafkaError`.
+- [Fix] Free native resources when `Admin#describe_configs`/`#incremental_alter_configs` raise while building their request (e.g. a non-String resource name).
+- [Fix] Raise `ConfigError` instead of segfaulting when `Admin#describe_configs`/`#incremental_alter_configs` get an empty resource name or a negative resource type.
 
 ## 0.29.1 (2026-09-04)
 - [Enhancement] Add `Admin#delete_records` to delete all messages in a partition up to a given offset.
@@ -19,8 +33,8 @@
 
 ## 0.29.0 (2026-07-10)
 - [Enhancement] Bump librdkafka to `2.14.2`
-- [Enhancement] Add `Consumer#metadata` and `Producer#metadata`, mirroring `Admin#metadata`, so cluster/topic metadata can be fetched from an existing consumer or producer handle without opening a dedicated admin connection.
-- [Enhancement] Support offset-commit metadata: `Consumer#store_offset(message, metadata)` stores an optional metadata string alongside the offset, `Consumer::Partition#metadata` exposes it, and `TopicPartitionList` marshals it to and from the native list (read back via `Consumer#committed`).
+- [Enhancement] Add `Consumer#metadata` and `Producer#metadata`, mirroring `Admin#metadata`, to fetch cluster metadata without a separate admin connection.
+- [Enhancement] Support offset-commit metadata via `Consumer#store_offset(message, metadata)`, readable through `Consumer::Partition#metadata` and `Consumer#committed`.
 - [Enhancement] Expose `replicas` and `isrs` (in-sync replica broker ids) on each partition in topic metadata; both were previously dropped from the `Metadata#topics` partition hashes.
 - [Enhancement] Extract the admin background-event result handlers into one class per operation under `lib/rdkafka/callbacks/`. Internal reorganization with no API or behavior change.
 - [Enhancement] Reuse per-thread scratch pointers in `Consumer::Headers.from_native`, removing the per-message native allocations from the consumer hot path.
@@ -28,32 +42,32 @@
 - [Fix] Cache the partition count for a missing topic, so `produce` with a `partition_key` to a missing topic no longer re-runs a blocking metadata query on every message.
 - [Fix] Let `PartitionsCountCache` adopt a lower partition count once the cached entry has expired, so a topic recreated with fewer partitions no longer fails `produce` until process restart.
 - [Fix] Surface per-message build errors inline in `Consumer#poll_batch`/`#poll_batch_nb` instead of discarding the batch, preserving the messages that built successfully.
-- [Fix] Bound the `Metadata` retry loop by a ~5s wall-clock budget with a floor of 3 attempts, cap the backoff at 1s, and stop overwriting the request timeout with the backoff value, which could block a synchronous fetch for minutes.
+- [Fix] Bound the `Metadata` retry loop to about 5 seconds, so a synchronous metadata fetch can no longer block for minutes.
 - [Fix] Synchronize `AbstractHandle::REGISTRY` mutations with a mutex, which could otherwise lose a write on JRuby and leave a handle unregistered or leaked.
-- [Fix] Fix the NULL background-queue cleanup branches in `Admin#delete_group`, `Admin#delete_acl` and `Admin#describe_acl`, which raised a `NameError` and leaked the native request object instead of raising `ConfigError`.
+- [Fix] Raise `ConfigError` instead of `NameError` in `Admin#delete_group`, `#delete_acl` and `#describe_acl` when the background queue is unavailable.
 - [Fix] Actually populate `ConfigBindingResult#synonyms`. The synonym loop built the synonym binding objects and discarded them, so `#synonyms` was always an empty array; they are now appended.
 - [Fix] Stop leaking the native `rd_kafka_topic_conf_t` in `Producer#set_topic_config` when a per-topic config value is rejected.
-- [Fix] Add the missing `closed_consumer_check` to `Consumer#position`, the only public consumer method without one; calling it on a closed consumer now raises a consistent `ClosedConsumerError` instead of failing deeper in the call.
+- [Fix] Raise `ClosedConsumerError` from `Consumer#position` on a closed consumer, like the other consumer methods.
 - [Fix] Guard the OAuthBearer token refresh callback against a NULL client pointer before calling `rd_kafka_name` on it.
-- [Fix] Make the `Consumer` GC finalizer close the consumer and destroy its consumer queue, not just the native client, preventing a hang or leak when a consumer is collected without an explicit `close`.
+- [Fix] Fully close a consumer collected without an explicit `close`, preventing a hang or leak.
 - [Fix] Free the librdkafka-allocated string in `Consumer#cluster_id` and `Consumer#member_id`, and fix the `rd_kafka_clusterid` arity. `Consumer#cluster_id` now accepts a `timeout_ms`.
 - [Fix] Attach `rd_kafka_query_watermark_offsets` with `blocking: true` so it releases the GVL; it previously froze every other Ruby thread for up to `timeout_ms`.
 - [Fix] Stop leaking the native `rd_kafka_conf_t` when client creation fails, a multi-KB leak per failed attempt for supervisors retrying on transient SASL/SSL misconfiguration.
 - [Fix] Stop `Metadata` from leaking the native metadata struct on every retried fetch; each attempt now frees its own native resources.
 - [Fix] Stop `Producer#produce` from orphaning the delivery handle in the process-global registry when it fails after registering it.
 - [Fix] Guard the message delivery callback so a raising user `delivery_callback` can no longer leave the handle locked or crash the producer; exceptions are now logged and swallowed.
-- [Fix] Stop `describe_configs`, `incremental_alter_configs` and `list_offsets` from leaking handles and native resources when their arguments are rejected; all three now validate input before allocating.
-- [Fix] Destroy the native topic-partition list in `TopicPartitionList#to_native_tpl` when population fails partway (e.g. an invalid partition). The caller-owned list was leaked on any exception mid-build; it is now destroyed before the error propagates.
+- [Fix] Stop `describe_configs`, `incremental_alter_configs` and `list_offsets` from leaking native resources when their arguments are rejected.
+- [Fix] Stop `TopicPartitionList#to_native_tpl` from leaking the native list when building it fails partway.
 - [Fix] Raise instead of silently dropping a rejected `incremental_alter_configs` entry, which previously left the alter request reporting success.
 - [Fix] Allocate the admin result-count out-parameter as `:size_t` instead of `:int32`, fixing a 4-byte overflow on every admin result parse.
 - [Fix] Stabilize the flaky `Consumer#lag` "calculates the consumer lag" spec on overloaded CI.
-- [Fix] Resolve admin operation handles from the event error when an operation fails at the operation level (e.g. brokers unreachable), instead of blocking until `wait` timed out and discarding the real error.
-- [Fix] Destroy admin API background events after processing; they were never destroyed, so every admin operation leaked its whole result event. The internal FFI struct fields on admin handles were removed as part of this - use `handle.wait` and the returned report objects, whose interfaces are unchanged.
+- [Fix] Return the real error from an admin operation that fails at the operation level (e.g. brokers unreachable) instead of blocking until `wait` times out.
+- [Fix] Stop every admin operation from leaking its result event. The internal FFI struct fields on admin handles are removed; use `handle.wait` and the returned report objects, which are unchanged.
 
 ## 0.28.0 (2026-06-03)
 - [Enhancement] Bump librdkafka to `2.14.1`
 - [Enhancement] Replace version-gated `Warning[:performance]` with dynamic `Warning.categories` opt-in in spec helper so new Ruby warning categories are automatically enabled without manual updates.
-- [Enhancement] `poll_batch` and `poll_batch_nb` now return error events inline as `RdkafkaError` objects rather than raising on the first error. The return type is `Array<Message, RdkafkaError>` and callers are responsible for handling errors in the result.
+- [Enhancement] `poll_batch` and `poll_batch_nb` now return errors inline as `RdkafkaError` objects instead of raising on the first one. The return type is `Array<Message, RdkafkaError>`, so callers must handle errors in the result.
 
 ## 0.27.0 (2026-05-07)
 - [Feature] Add `Consumer#poll_batch(timeout_ms, max_items:)` and `Consumer#poll_batch_nb(timeout_ms, max_items:)` for batch message polling via `rd_kafka_consume_batch_queue`.

data/README.md

@@ -163,6 +163,7 @@ bundle exec rake produce_messages
 
 | rdkafka-ruby | librdkafka | patches |
 |-|-|-|
+| 0.30.x (2026-09-25) | 2.15.0 (2026-06-30) | yes |
 | 0.29.x (2026-07-10) | 2.14.2 (2026-06-03) | yes |
 | 0.28.x (2026-06-03) | 2.14.1 (2026-04-15) | yes |
 | 0.27.x (2026-05-07) | 2.14.0 (2026-04-01) | yes |

data/lib/rdkafka/admin/alter_consumer_group_offsets_handle.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Handle for alter consumer group offsets operation
+    class AlterConsumerGroupOffsetsHandle < AbstractHandle
+      layout :pending, :bool,
+        :response, :int
+
+      # @return [String] the name of the operation
+      def operation_name
+        "alter consumer group offsets"
+      end
+
+      # Creates the result report
+      # @return [AlterConsumerGroupOffsetsReport]
+      def create_result
+        prepared_result
+      end
+
+      # Raises an error if the operation failed
+      # @raise [RdkafkaError]
+      def raise_error
+        raise RdkafkaError.new(
+          self[:response],
+          broker_message: broker_message
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/alter_consumer_group_offsets_report.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Report for alter consumer group offsets operation result
+    class AlterConsumerGroupOffsetsReport
+      # @return [String, nil] the group the operation ran against
+      attr_reader :group_name
+
+      # @return [String, nil] group level error message, if any
+      attr_reader :error_string
+
+      # Per partition outcomes, each with `:topic`, `:partition`, `:offset` and `:error`
+      # (an `RdkafkaError` or nil). Partitions are reported individually because Kafka can
+      # fail some and accept others within a single request.
+      # @return [Array<Hash>]
+      attr_reader :partitions
+
+      # @param error_string [FFI::Pointer] pointer to the group level error string
+      # @param result_name [FFI::Pointer] pointer to the group name
+      # @param partitions_ptr [FFI::Pointer] pointer to the per partition result list
+      def initialize(error_string, result_name, partitions_ptr = FFI::Pointer::NULL)
+        @error_string = error_string.read_string if error_string != FFI::Pointer::NULL
+        @group_name = result_name.read_string if result_name != FFI::Pointer::NULL
+        @partitions = extract_partitions(partitions_ptr)
+      end
+
+      private
+
+      # @param partitions_ptr [FFI::Pointer] pointer to a native topic partition list
+      # @return [Array<Hash>]
+      def extract_partitions(partitions_ptr)
+        return [] if partitions_ptr.nil? || partitions_ptr.null?
+
+        native = Bindings::TopicPartitionList.new(partitions_ptr)
+
+        Array.new(native[:cnt]) do |i|
+          tp = Bindings::TopicPartition.new(native[:elems] + (i * Bindings::TopicPartition.size))
+
+          {
+            topic: tp[:topic],
+            partition: tp[:partition],
+            offset: tp[:offset],
+            error: tp[:err].zero? ? nil : RdkafkaError.new(tp[:err])
+          }
+        end
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/delete_consumer_group_offsets_handle.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Handle for delete consumer group offsets operation
+    class DeleteConsumerGroupOffsetsHandle < AbstractHandle
+      layout :pending, :bool,
+        :response, :int
+
+      # @return [String] the name of the operation
+      def operation_name
+        "delete consumer group offsets"
+      end
+
+      # Creates the result report
+      # @return [DeleteConsumerGroupOffsetsReport]
+      def create_result
+        prepared_result
+      end
+
+      # Raises an error if the operation failed
+      # @raise [RdkafkaError]
+      def raise_error
+        raise RdkafkaError.new(
+          self[:response],
+          broker_message: broker_message
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/delete_consumer_group_offsets_report.rb

@@ -0,0 +1,50 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    # Report for delete consumer group offsets operation result
+    class DeleteConsumerGroupOffsetsReport
+      # @return [String, nil] the group the operation ran against
+      attr_reader :group_name
+
+      # @return [String, nil] group level error message, if any
+      attr_reader :error_string
+
+      # Per partition outcomes, each with `:topic`, `:partition`, `:offset` and `:error`
+      # (an `RdkafkaError` or nil). Partitions are reported individually because Kafka can
+      # fail some and accept others within a single request.
+      # @return [Array<Hash>]
+      attr_reader :partitions
+
+      # @param error_string [FFI::Pointer] pointer to the group level error string
+      # @param result_name [FFI::Pointer] pointer to the group name
+      # @param partitions_ptr [FFI::Pointer] pointer to the per partition result list
+      def initialize(error_string, result_name, partitions_ptr = FFI::Pointer::NULL)
+        @error_string = error_string.read_string if error_string != FFI::Pointer::NULL
+        @group_name = result_name.read_string if result_name != FFI::Pointer::NULL
+        @partitions = extract_partitions(partitions_ptr)
+      end
+
+      private
+
+      # @param partitions_ptr [FFI::Pointer] pointer to a native topic partition list
+      # @return [Array<Hash>]
+      def extract_partitions(partitions_ptr)
+        return [] if partitions_ptr.nil? || partitions_ptr.null?
+
+        native = Bindings::TopicPartitionList.new(partitions_ptr)
+
+        Array.new(native[:cnt]) do |i|
+          tp = Bindings::TopicPartition.new(native[:elems] + (i * Bindings::TopicPartition.size))
+
+          {
+            topic: tp[:topic],
+            partition: tp[:partition],
+            offset: tp[:offset],
+            error: tp[:err].zero? ? nil : RdkafkaError.new(tp[:err])
+          }
+        end
+      end
+    end
+  end
+end

data/lib/rdkafka/admin.rb

@@ -224,6 +224,78 @@ module Rdkafka
       create_topic_handle
     end
 
+    # Sets the committed offsets of a consumer group from outside the group.
+    #
+    # Kafka requires the group to be empty (no active members) for this to be accepted; when it
+    # is not, the broker rejects the request and the handle raises with `group_not_empty` rather
+    # than reporting success.
+    #
+    # @param group_id [String] the group whose offsets we want to set
+    # @param topic_partition_list [TopicPartitionList] partitions with the offsets to commit
+    # @return [AlterConsumerGroupOffsetsHandle] handle that can be used to wait for the result
+    # @raise [RdkafkaError] when altering the offsets fails
+    def alter_consumer_group_offsets(group_id, topic_partition_list)
+      closed_admin_check(__method__)
+
+      native_tpl = topic_partition_list.to_native_tpl
+
+      begin
+        alter_ptr = Rdkafka::Bindings.rd_kafka_AlterConsumerGroupOffsets_new(
+          FFI::MemoryPointer.from_string(group_id),
+          native_tpl
+        )
+      rescue Exception
+        Rdkafka::Bindings.rd_kafka_topic_partition_list_destroy(native_tpl)
+        raise
+      end
+
+      # librdkafka copies the list into the request object, so ours is no longer needed
+      Rdkafka::Bindings.rd_kafka_topic_partition_list_destroy(native_tpl)
+
+      enqueue_group_offsets_op(
+        request_ptr: alter_ptr,
+        handle: AlterConsumerGroupOffsetsHandle.new,
+        admin_op: Rdkafka::Bindings::RD_KAFKA_ADMIN_OP_ALTERCONSUMERGROUPOFFSETS,
+        destroy: Rdkafka::Bindings.method(:rd_kafka_AlterConsumerGroupOffsets_destroy),
+        enqueue: Rdkafka::Bindings.method(:rd_kafka_AlterConsumerGroupOffsets)
+      )
+    end
+
+    # Deletes the committed offsets of specific partitions of a consumer group, leaving the
+    # group and its remaining offsets in place.
+    #
+    # This is the partition scoped counterpart of {#delete_group}, which removes the whole group.
+    #
+    # @param group_id [String] the group whose offsets we want to delete
+    # @param topic_partition_list [TopicPartitionList] partitions whose offsets to delete
+    # @return [DeleteConsumerGroupOffsetsHandle] handle that can be used to wait for the result
+    # @raise [RdkafkaError] when deleting the offsets fails
+    def delete_consumer_group_offsets(group_id, topic_partition_list)
+      closed_admin_check(__method__)
+
+      native_tpl = topic_partition_list.to_native_tpl
+
+      begin
+        delete_ptr = Rdkafka::Bindings.rd_kafka_DeleteConsumerGroupOffsets_new(
+          FFI::MemoryPointer.from_string(group_id),
+          native_tpl
+        )
+      rescue Exception
+        Rdkafka::Bindings.rd_kafka_topic_partition_list_destroy(native_tpl)
+        raise
+      end
+
+      Rdkafka::Bindings.rd_kafka_topic_partition_list_destroy(native_tpl)
+
+      enqueue_group_offsets_op(
+        request_ptr: delete_ptr,
+        handle: DeleteConsumerGroupOffsetsHandle.new,
+        admin_op: Rdkafka::Bindings::RD_KAFKA_ADMIN_OP_DELETECONSUMERGROUPOFFSETS,
+        destroy: Rdkafka::Bindings.method(:rd_kafka_DeleteConsumerGroupOffsets_destroy),
+        enqueue: Rdkafka::Bindings.method(:rd_kafka_DeleteConsumerGroupOffsets)
+      )
+    end
+
     # Deletes a consumer group
     #
     # @param group_id [String] the group id to delete
@@ -1139,6 +1211,54 @@ module Rdkafka
 
     private
 
+    # Shared enqueue path for the two consumer group offset admin operations. They differ only
+    # in their C entry points, so keeping one body avoids two near identical copies of the
+    # queue/handle/options dance and its cleanup.
+    #
+    # @param request_ptr [FFI::Pointer] the op specific request object
+    # @param handle [AbstractHandle] the handle to register and return
+    # @param admin_op [Integer] the RD_KAFKA_ADMIN_OP_* constant for this operation
+    # @param destroy [Method] bound function used to free the request object
+    # @param enqueue [Method] bound function used to enqueue the request
+    # @return [AbstractHandle] the registered handle
+    def enqueue_group_offsets_op(request_ptr:, handle:, admin_op:, destroy:, enqueue:)
+      array_ptr = FFI::MemoryPointer.new(:pointer)
+      array_ptr.write_array_of_pointer([request_ptr])
+
+      queue_ptr = @native_kafka.with_inner do |inner|
+        Rdkafka::Bindings.rd_kafka_queue_get_background(inner)
+      end
+
+      if queue_ptr.null?
+        destroy.call(request_ptr)
+        raise Rdkafka::Config::ConfigError.new("rd_kafka_queue_get_background was NULL")
+      end
+
+      handle[:pending] = true
+      handle[:response] = Rdkafka::Bindings::RD_KAFKA_PARTITION_UA
+      handle.class.register(handle)
+
+      admin_options_ptr = @native_kafka.with_inner do |inner|
+        Rdkafka::Bindings.rd_kafka_AdminOptions_new(inner, admin_op)
+      end
+      Rdkafka::Bindings.rd_kafka_AdminOptions_set_opaque(admin_options_ptr, handle.to_ptr)
+
+      begin
+        @native_kafka.with_inner do |inner|
+          enqueue.call(inner, array_ptr, 1, admin_options_ptr, queue_ptr)
+        end
+      rescue Exception
+        handle.class.remove(handle.to_ptr.address)
+        raise
+      ensure
+        Rdkafka::Bindings.rd_kafka_AdminOptions_destroy(admin_options_ptr)
+        Rdkafka::Bindings.rd_kafka_queue_destroy(queue_ptr)
+        destroy.call(request_ptr)
+      end
+
+      handle
+    end
+
     # Checks if the admin is closed and raises an error if so
     # @param method [Symbol] name of the calling method for error context
     # @raise [ClosedAdminError] when the admin is closed

data/lib/rdkafka/bindings.rb

@@ -192,7 +192,7 @@ module Rdkafka
     RD_KAFKA_ALTER_CONFIG_OP_TYPE_SUBTRACT = 3
 
     # List Offsets
-    RD_KAFKA_ADMIN_OP_LISTOFFSETS = 20
+    RD_KAFKA_ADMIN_OP_LISTOFFSETS = 21
     RD_KAFKA_EVENT_LISTOFFSETS_RESULT = 0x400000
 
     # rd_kafka_IsolationLevel_t
@@ -503,9 +503,31 @@ module Rdkafka
         raise Rdkafka::Config::ConfigError.new("Unknown partitioner: #{partitioner}")
       end
 
-      public_send(method_name, topic_ptr, str_ptr, str.size, partition_count, nil, nil)
+      public_send(method_name, topic_ptr, str_ptr, partition_key_length(str), partition_count, nil, nil)
     end
 
+    # Partition key length as the character count (legacy default).
+    #
+    # @param str [String] the partition key string
+    # @return [Integer]
+    def self.partition_key_size(str)
+      str.size
+    end
+
+    # Partition key length as the byte count, which is what librdkafka hashes (UTF-8 bytes copied
+    # into the key pointer) and what other Kafka clients use.
+    #
+    # @param str [String] the partition key string
+    # @return [Integer]
+    def self.partition_key_bytesize(str)
+      str.bytesize
+    end
+
+    # Length of the partition key passed to librdkafka. Aliased to one of the methods above by
+    # `Rdkafka::Config.partitioner_key_uses_bytesize=`, so the per-message path does not check the
+    # setting on every call.
+    singleton_class.alias_method :partition_key_length, :partition_key_size
+
     # Create Topics
     RD_KAFKA_ADMIN_OP_CREATETOPICS = 1   # rd_kafka_admin_op_t
     RD_KAFKA_EVENT_CREATETOPICS_RESULT = 100 # rd_kafka_event_type_t
@@ -547,6 +569,26 @@ module Rdkafka
     attach_function :rd_kafka_event_DeleteGroups_result, [:pointer], :pointer, blocking: true # rd_kafka_event_t* => rd_kafka_DeleteGroups_result_t*
     attach_function :rd_kafka_DeleteGroups_result_groups, [:pointer, :pointer], :pointer, blocking: true # rd_kafka_DeleteGroups_result_t*, size_t* => rd_kafka_group_result_t**
 
+    # Alter Consumer Group Offsets
+    RD_KAFKA_ADMIN_OP_ALTERCONSUMERGROUPOFFSETS = 15   # rd_kafka_admin_op_t
+    RD_KAFKA_EVENT_ALTERCONSUMERGROUPOFFSETS_RESULT = 0x10000 # rd_kafka_event_type_t
+
+    attach_function :rd_kafka_AlterConsumerGroupOffsets, [:pointer, :pointer, :size_t, :pointer, :pointer], :void, blocking: true
+    attach_function :rd_kafka_AlterConsumerGroupOffsets_new, [:pointer, :pointer], :pointer, blocking: true # const char*, const rd_kafka_topic_partition_list_t* => rd_kafka_AlterConsumerGroupOffsets_t*
+    attach_function :rd_kafka_AlterConsumerGroupOffsets_destroy, [:pointer], :void, blocking: true
+    attach_function :rd_kafka_event_AlterConsumerGroupOffsets_result, [:pointer], :pointer, blocking: true # rd_kafka_event_t* => rd_kafka_AlterConsumerGroupOffsets_result_t*
+    attach_function :rd_kafka_AlterConsumerGroupOffsets_result_groups, [:pointer, :pointer], :pointer, blocking: true # rd_kafka_AlterConsumerGroupOffsets_result_t*, size_t* => rd_kafka_group_result_t**
+
+    # Delete Consumer Group Offsets
+    RD_KAFKA_ADMIN_OP_DELETECONSUMERGROUPOFFSETS = 8   # rd_kafka_admin_op_t
+    RD_KAFKA_EVENT_DELETECONSUMERGROUPOFFSETS_RESULT = 107 # rd_kafka_event_type_t
+
+    attach_function :rd_kafka_DeleteConsumerGroupOffsets, [:pointer, :pointer, :size_t, :pointer, :pointer], :void, blocking: true
+    attach_function :rd_kafka_DeleteConsumerGroupOffsets_new, [:pointer, :pointer], :pointer, blocking: true # const char*, const rd_kafka_topic_partition_list_t* => rd_kafka_DeleteConsumerGroupOffsets_t*
+    attach_function :rd_kafka_DeleteConsumerGroupOffsets_destroy, [:pointer], :void, blocking: true
+    attach_function :rd_kafka_event_DeleteConsumerGroupOffsets_result, [:pointer], :pointer, blocking: true # rd_kafka_event_t* => rd_kafka_DeleteConsumerGroupOffsets_result_t*
+    attach_function :rd_kafka_DeleteConsumerGroupOffsets_result_groups, [:pointer, :pointer], :pointer, blocking: true # rd_kafka_DeleteConsumerGroupOffsets_result_t*, size_t* => rd_kafka_group_result_t**
+
     # Delete Records
     RD_KAFKA_ADMIN_OP_DELETERECORDS = 6   # rd_kafka_admin_op_t
     RD_KAFKA_EVENT_DELETERECORDS_RESULT = 105 # rd_kafka_event_type_t
@@ -696,5 +738,6 @@ module Rdkafka
 
     attach_function :rd_kafka_group_result_error, [:pointer], NativeError.by_ref # rd_kafka_group_result_t* => rd_kafka_error_t*
     attach_function :rd_kafka_group_result_name, [:pointer], :pointer
+    attach_function :rd_kafka_group_result_partitions, [:pointer], :pointer # rd_kafka_group_result_t* => rd_kafka_topic_partition_list_t*
   end
 end

data/lib/rdkafka/callbacks/alter_consumer_group_offsets_handler.rb

@@ -0,0 +1,65 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  module Callbacks
+    # Handles `RD_KAFKA_EVENT_ALTERCONSUMERGROUPOFFSETS_RESULT` events
+    # @private
+    class AlterConsumerGroupOffsetsHandler < BaseHandler
+      class << self
+        # Resolves the alter consumer group offsets handle from its result event
+        # @param event_ptr [FFI::Pointer] pointer to the event
+        # @return [void]
+        def call(event_ptr)
+          result = Rdkafka::Bindings.rd_kafka_event_AlterConsumerGroupOffsets_result(event_ptr)
+
+          pointer_to_size_t = FFI::MemoryPointer.new(:size_t)
+          group_result_array = Rdkafka::Bindings.rd_kafka_AlterConsumerGroupOffsets_result_groups(result, pointer_to_size_t)
+          group_results = GroupResult.create_group_results_from_array(pointer_to_size_t.read_int, group_result_array)
+          handle_ptr = Rdkafka::Bindings.rd_kafka_event_opaque(event_ptr)
+
+          if (handle = Rdkafka::Admin::AlterConsumerGroupOffsetsHandle.remove(handle_ptr.address))
+            unless resolve_operation_error(event_ptr, handle)
+              group_result = group_results[0]
+
+              # librdkafka only supports one group per invocation, so a missing group result
+              # means the broker returned nothing to act on rather than a partial success.
+              if group_result.nil?
+                handle[:response] = Rdkafka::Bindings::RD_KAFKA_RESP_ERR_NO_ERROR
+                handle.result = Rdkafka::Admin::AlterConsumerGroupOffsetsReport.new(
+                  FFI::Pointer::NULL,
+                  FFI::Pointer::NULL
+                )
+              else
+                report = Rdkafka::Admin::AlterConsumerGroupOffsetsReport.new(
+                  group_result.error_string,
+                  group_result.result_name,
+                  Rdkafka::Bindings.rd_kafka_group_result_partitions(group_result_array.read_pointer)
+                )
+                handle.result = report
+
+                # Kafka reports a rejected offset change per partition, leaving the group level
+                # error unset - a group with live members comes back with the group error NULL
+                # and `unknown_member_id` on each partition. Resolving on the group error alone
+                # would hand back a report that looks like success, so any failed partition has
+                # to fail the handle too.
+                failed = report.partitions.find { |partition| partition[:error] }
+
+                if !group_result.result_error.zero?
+                  handle[:response] = group_result.result_error
+                  handle.broker_message = report.error_string
+                elsif failed
+                  handle[:response] = failed[:error].rdkafka_response
+                  handle.broker_message = "Failed for partition #{failed[:partition]} of '#{failed[:topic]}'"
+                else
+                  handle[:response] = Rdkafka::Bindings::RD_KAFKA_RESP_ERR_NO_ERROR
+                end
+              end
+
+              handle.unlock
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/rdkafka/callbacks/delete_consumer_group_offsets_handler.rb

@@ -0,0 +1,65 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  module Callbacks
+    # Handles `RD_KAFKA_EVENT_DELETECONSUMERGROUPOFFSETS_RESULT` events
+    # @private
+    class DeleteConsumerGroupOffsetsHandler < BaseHandler
+      class << self
+        # Resolves the delete consumer group offsets handle from its result event
+        # @param event_ptr [FFI::Pointer] pointer to the event
+        # @return [void]
+        def call(event_ptr)
+          result = Rdkafka::Bindings.rd_kafka_event_DeleteConsumerGroupOffsets_result(event_ptr)
+
+          pointer_to_size_t = FFI::MemoryPointer.new(:size_t)
+          group_result_array = Rdkafka::Bindings.rd_kafka_DeleteConsumerGroupOffsets_result_groups(result, pointer_to_size_t)
+          group_results = GroupResult.create_group_results_from_array(pointer_to_size_t.read_int, group_result_array)
+          handle_ptr = Rdkafka::Bindings.rd_kafka_event_opaque(event_ptr)
+
+          if (handle = Rdkafka::Admin::DeleteConsumerGroupOffsetsHandle.remove(handle_ptr.address))
+            unless resolve_operation_error(event_ptr, handle)
+              group_result = group_results[0]
+
+              # librdkafka only supports one group per invocation, so a missing group result
+              # means the broker returned nothing to act on rather than a partial success.
+              if group_result.nil?
+                handle[:response] = Rdkafka::Bindings::RD_KAFKA_RESP_ERR_NO_ERROR
+                handle.result = Rdkafka::Admin::DeleteConsumerGroupOffsetsReport.new(
+                  FFI::Pointer::NULL,
+                  FFI::Pointer::NULL
+                )
+              else
+                report = Rdkafka::Admin::DeleteConsumerGroupOffsetsReport.new(
+                  group_result.error_string,
+                  group_result.result_name,
+                  Rdkafka::Bindings.rd_kafka_group_result_partitions(group_result_array.read_pointer)
+                )
+                handle.result = report
+
+                # Kafka reports a rejected offset change per partition, leaving the group level
+                # error unset - a group with live members comes back with the group error NULL
+                # and `unknown_member_id` on each partition. Resolving on the group error alone
+                # would hand back a report that looks like success, so any failed partition has
+                # to fail the handle too.
+                failed = report.partitions.find { |partition| partition[:error] }
+
+                if !group_result.result_error.zero?
+                  handle[:response] = group_result.result_error
+                  handle.broker_message = report.error_string
+                elsif failed
+                  handle[:response] = failed[:error].rdkafka_response
+                  handle.broker_message = "Failed for partition #{failed[:partition]} of '#{failed[:topic]}'"
+                else
+                  handle[:response] = Rdkafka::Bindings::RD_KAFKA_RESP_ERR_NO_ERROR
+                end
+              end
+
+              handle.unlock
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/rdkafka/callbacks.rb

@@ -21,10 +21,9 @@ module Rdkafka
       # @param array_pointer [FFI::Pointer] pointer to the results array
       # @return [Array<TopicResult>] array of topic results
       def self.create_topic_results_from_array(count, array_pointer)
-        (1..count).map do |index|
-          result_pointer = (array_pointer + (index - 1)).read_pointer
-          new(result_pointer)
-        end
+        return [] if count.zero?
+
+        array_pointer.read_array_of_pointer(count).map { |result_pointer| new(result_pointer) }
       end
     end
 
@@ -53,10 +52,9 @@ module Rdkafka
       # @param array_pointer [FFI::Pointer] pointer to the results array
       # @return [Array<GroupResult>] array of group results
       def self.create_group_results_from_array(count, array_pointer)
-        (1..count).map do |index|
-          result_pointer = (array_pointer + (index - 1)).read_pointer
-          new(result_pointer)
-        end
+        return [] if count.zero?
+
+        array_pointer.read_array_of_pointer(count).map { |result_pointer| new(result_pointer) }
       end
     end
 
@@ -77,10 +75,9 @@ module Rdkafka
       # @param array_pointer [FFI::Pointer] pointer to the results array
       # @return [Array<CreateAclResult>] array of ACL results
       def self.create_acl_results_from_array(count, array_pointer)
-        (1..count).map do |index|
-          result_pointer = (array_pointer + (index - 1)).read_pointer
-          new(result_pointer)
-        end
+        return [] if count.zero?
+
+        array_pointer.read_array_of_pointer(count).map { |result_pointer| new(result_pointer) }
       end
     end
 
@@ -108,10 +105,9 @@ module Rdkafka
       # @param array_pointer [FFI::Pointer] pointer to the results array
       # @return [Array<DeleteAclResult>] array of delete ACL results
       def self.delete_acl_results_from_array(count, array_pointer)
-        (1..count).map do |index|
-          result_pointer = (array_pointer + (index - 1)).read_pointer
-          new(result_pointer)
-        end
+        return [] if count.zero?
+
+        array_pointer.read_array_of_pointer(count).map { |result_pointer| new(result_pointer) }
       end
     end
 
@@ -213,6 +209,8 @@ module Rdkafka
         when Rdkafka::Bindings::RD_KAFKA_EVENT_DELETETOPICS_RESULT then DeleteTopicHandler
         when Rdkafka::Bindings::RD_KAFKA_ADMIN_OP_CREATEPARTITIONS_RESULT then CreatePartitionsHandler
         when Rdkafka::Bindings::RD_KAFKA_EVENT_DELETEGROUPS_RESULT then DeleteGroupsHandler
+        when Rdkafka::Bindings::RD_KAFKA_EVENT_ALTERCONSUMERGROUPOFFSETS_RESULT then AlterConsumerGroupOffsetsHandler
+        when Rdkafka::Bindings::RD_KAFKA_EVENT_DELETECONSUMERGROUPOFFSETS_RESULT then DeleteConsumerGroupOffsetsHandler
         when Rdkafka::Bindings::RD_KAFKA_EVENT_DELETERECORDS_RESULT then DeleteRecordsHandler
         when Rdkafka::Bindings::RD_KAFKA_EVENT_CREATEACLS_RESULT then CreateAclHandler
         when Rdkafka::Bindings::RD_KAFKA_EVENT_DELETEACLS_RESULT then DeleteAclHandler

data/lib/rdkafka/config.rb

@@ -22,6 +22,12 @@ module Rdkafka
     @@log_mutex = Mutex.new
     # @!visibility private
     @@oauthbearer_token_refresh_callback = nil
+    # @!visibility private
+    # Whether the partition key length passed to the partitioner is the byte length (`bytesize`)
+    # rather than the character length (`size`). `false` (character length) is the legacy default
+    # to avoid changing partition assignment for existing multibyte/binary keys; set to `true` to
+    # opt into the correct, other-client-consistent byte-length hashing.
+    @@partitioner_key_uses_bytesize = false
 
     # Returns the current logger, by default this is a logger to stdout.
     #
@@ -122,6 +128,34 @@ module Rdkafka
       @@oauthbearer_token_refresh_callback
     end
 
+    # Whether the partitioner hashes the partition key by its byte length instead of its character
+    # length. Defaults to `false` (character length) to preserve existing partition assignment for
+    # multibyte/binary keys. Passing the byte length is the correct, other-client-consistent
+    # behavior; enable it (and be aware it can move such keys to a different partition) with
+    # `Rdkafka::Config.partitioner_key_uses_bytesize = true`. This default is expected to change in
+    # a future major version.
+    #
+    # @return [Boolean]
+    def self.partitioner_key_uses_bytesize
+      @@partitioner_key_uses_bytesize
+    end
+
+    # Sets whether the partitioner hashes the partition key by its byte length rather than its
+    # character length. The key length method is resolved here once rather than on every produced
+    # message, and each call re-resolves it, so changing the setting at runtime takes effect
+    # immediately.
+    #
+    # @param value [Boolean] true to hash by byte length, false (default) for character length
+    # @return [void]
+    def self.partitioner_key_uses_bytesize=(value)
+      @@partitioner_key_uses_bytesize = value
+
+      Rdkafka::Bindings.singleton_class.alias_method(
+        :partition_key_length,
+        value ? :partition_key_bytesize : :partition_key_size
+      )
+    end
+
     # @private
     def self.opaques
       @@opaques

data/lib/rdkafka/version.rb

@@ -2,9 +2,9 @@
 
 module Rdkafka
   # Current rdkafka-ruby gem version
-  VERSION = "0.29.2"
+  VERSION = "0.30.0"
   # Target librdkafka version to be used
-  LIBRDKAFKA_VERSION = "2.14.2"
+  LIBRDKAFKA_VERSION = "2.15.0"
   # SHA256 hash of the librdkafka source tarball for verification
-  LIBRDKAFKA_SOURCE_SHA256 = "d7eec9c31c817fa44402f679c252dfbf97e4c338a849a25c3579a31fd127beb8"
+  LIBRDKAFKA_SOURCE_SHA256 = "259015220cdca708afe838b5aa79ebf1a5fb710fb4179cf918d390aed85d5dbc"
 end

data/lib/rdkafka.rb

@@ -13,6 +13,10 @@ require "rdkafka/helpers/list_offsets"
 require "rdkafka/defaults"
 require "rdkafka/abstract_handle"
 require "rdkafka/admin"
+require "rdkafka/admin/alter_consumer_group_offsets_handle"
+require "rdkafka/admin/alter_consumer_group_offsets_report"
+require "rdkafka/admin/delete_consumer_group_offsets_handle"
+require "rdkafka/admin/delete_consumer_group_offsets_report"
 require "rdkafka/admin/create_topic_handle"
 require "rdkafka/admin/create_topic_report"
 require "rdkafka/admin/delete_groups_handle"
@@ -46,6 +50,8 @@ require "rdkafka/callbacks/base_handler"
 require "rdkafka/callbacks/create_topic_handler"
 require "rdkafka/callbacks/delete_topic_handler"
 require "rdkafka/callbacks/create_partitions_handler"
+require "rdkafka/callbacks/alter_consumer_group_offsets_handler"
+require "rdkafka/callbacks/delete_consumer_group_offsets_handler"
 require "rdkafka/callbacks/delete_groups_handler"
 require "rdkafka/callbacks/delete_records_handler"
 require "rdkafka/callbacks/create_acl_handler"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rdkafka
 version: !ruby/object:Gem::Version
-  version: 0.29.2
+  version: 0.30.0
 platform: ruby
 authors:
 - Thijs Cadier
@@ -93,7 +93,7 @@ files:
 - README.md
 - Rakefile
 - bin/verify_kafka_warnings
-- dist/librdkafka-2.14.2.tar.gz
+- dist/librdkafka-2.15.0.tar.gz
 - dist/patches/rdkafka_global_init.patch
 - docker-compose-ssl.yml
 - docker-compose.yml
@@ -102,6 +102,8 @@ files:
 - lib/rdkafka/abstract_handle.rb
 - lib/rdkafka/admin.rb
 - lib/rdkafka/admin/acl_binding_result.rb
+- lib/rdkafka/admin/alter_consumer_group_offsets_handle.rb
+- lib/rdkafka/admin/alter_consumer_group_offsets_report.rb
 - lib/rdkafka/admin/config_binding_result.rb
 - lib/rdkafka/admin/config_resource_binding_result.rb
 - lib/rdkafka/admin/create_acl_handle.rb
@@ -112,6 +114,8 @@ files:
 - lib/rdkafka/admin/create_topic_report.rb
 - lib/rdkafka/admin/delete_acl_handle.rb
 - lib/rdkafka/admin/delete_acl_report.rb
+- lib/rdkafka/admin/delete_consumer_group_offsets_handle.rb
+- lib/rdkafka/admin/delete_consumer_group_offsets_report.rb
 - lib/rdkafka/admin/delete_groups_handle.rb
 - lib/rdkafka/admin/delete_groups_report.rb
 - lib/rdkafka/admin/delete_records_handle.rb
@@ -130,11 +134,13 @@ files:
 - lib/rdkafka/admin/list_offsets_report.rb
 - lib/rdkafka/bindings.rb
 - lib/rdkafka/callbacks.rb
+- lib/rdkafka/callbacks/alter_consumer_group_offsets_handler.rb
 - lib/rdkafka/callbacks/base_handler.rb
 - lib/rdkafka/callbacks/create_acl_handler.rb
 - lib/rdkafka/callbacks/create_partitions_handler.rb
 - lib/rdkafka/callbacks/create_topic_handler.rb
 - lib/rdkafka/callbacks/delete_acl_handler.rb
+- lib/rdkafka/callbacks/delete_consumer_group_offsets_handler.rb
 - lib/rdkafka/callbacks/delete_groups_handler.rb
 - lib/rdkafka/callbacks/delete_records_handler.rb
 - lib/rdkafka/callbacks/delete_topic_handler.rb
@@ -192,7 +198,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.16
+rubygems_version: 4.0.20
 specification_version: 4
 summary: The rdkafka gem is a modern Kafka client library for Ruby based on librdkafka.
   It wraps the production-ready C client using the ffi gem and targets Kafka 1.0+