rdkafka 0.22.2 → 0.29.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6bdfd8912cec56ebe8303d0ad14f3ab7ab2e5e0565e9418a6265a0e48172bc31
-  data.tar.gz: 646f8a174c1019a0417a571f5894c745701c5d670a0585beb40200a2ffe08193
+  metadata.gz: ca57ade30bf93bb0e222709e64ff09c365ea29bf93b8f040d59157a39ed9ca79
+  data.tar.gz: 0c255e20b62b018075182ae474237765fa2e8d78f88047ac19ff92c239373fc7
 SHA512:
-  metadata.gz: 9d0711b10d02a1c6792fb21206d7729c958fadc186da5b6d8382d69b0d98183cddf23c123ea1602dd890f39603cd44341b864726c4a737966148fffb7e5e5dbd
-  data.tar.gz: deeacf4c205c164d2bd2d786f4a512f62caf8aeda6d6fcc11c24d2d665f4b7bd0a7f44447dfa1ebffc1fe54e7a9cdf36c7bbd4ae71c01e167b46b36a298d7482
+  metadata.gz: 48ee067dc2d35d877601198b436450fce3ff8884ca10d129d00af822e2cf823310d6636be55f48bb82658a42b8221437a6b7025a5c402719cabd091f48d1b092
+  data.tar.gz: ed3695c92af1b5b8d08a38e797fceca8ec5e081ab0b88741f788f40b945df4707d58579f93aa6acbe70d3cb9002653bb65d4f5daa6d960f310ca5df91ab85fc2

data/CHANGELOG.md

@@ -1,9 +1,107 @@
 # Rdkafka Changelog
 
+## 0.29.0 (2026-07-10)
+- [Enhancement] Bump librdkafka to `2.14.2`
+- [Enhancement] Add `Consumer#metadata` and `Producer#metadata`, mirroring `Admin#metadata`, so cluster/topic metadata can be fetched from an existing consumer or producer handle without opening a dedicated admin connection.
+- [Enhancement] Support offset-commit metadata: `Consumer#store_offset(message, metadata)` stores an optional metadata string alongside the offset, `Consumer::Partition#metadata` exposes it, and `TopicPartitionList` marshals it to and from the native list (read back via `Consumer#committed`).
+- [Fix] Cache the partition count for a missing topic. `Producer#partition_count` rescued `unknown_topic_or_part` outside the cache block, so nothing was cached and every `produce(partition_key:)` to a missing topic re-ran a blocking metadata query. The rescue now runs inside the cache block so `RD_KAFKA_PARTITION_UA` is cached like any other count; other errors are still re-raised.
+- [Fix] Let `PartitionsCountCache` adopt a lower partition count once the cached entry has expired. The cache prioritizes higher counts (partition counts only grow during normal operation), but it did so unconditionally: after the TTL expired it fetched the true lower count, discarded it, and re-armed the TTL on the stale higher count - permanently. A topic recreated with fewer partitions then made `produce` (with a partition key) fail with `unknown_partition` for the dropped partitions until process restart. A lower value is now adopted on the first refresh after expiry, while still being ignored within the TTL window so a transient or racy lower read cannot clobber a correct higher count.
+- [Fix] Surface per-message build errors inline in `Consumer#poll_batch`/`#poll_batch_nb` instead of discarding the batch. A message that failed to build (e.g. a header read error) raised out of the loop, losing the already-built messages - whose offsets may have been auto-stored - while error-flagged messages were already returned inline. Build failures are now returned as inline `RdkafkaError` entries alongside the successfully built messages.
+- [Fix] Bound the `Metadata` retry loop by a wall-clock budget (`Defaults::METADATA_RETRY_BUDGET_MS`, 5s) with a minimum-attempts floor (`Defaults::METADATA_MIN_ATTEMPTS`, 3), cap the exponential backoff at `Defaults::METADATA_RETRY_BACKOFF_MAX_MS` (1s), and stop overwriting the request timeout with the backoff value. Each retry used to shrink the first retries' request timeout below the configured value (near-guaranteeing another timeout) and inflate later ones to ~100s, with cumulative sleeps alone reaching ~204s - a synchronous metadata fetch could block the caller for minutes.
+- [Fix] Synchronize `AbstractHandle::REGISTRY` mutations. The handle registry is a plain Hash mutated from producing/consuming threads and the background polling thread (which removes handles from FFI callbacks). That is effectively safe on MRI under the GVL but not on JRuby, where a lost write could leave a handle unregistered (never unlocked, so `wait` times out for a delivered message) or never removed (permanent leak). `register`/`remove` now guard the Hash with a mutex.
+- [Fix] Fix the NULL background-queue cleanup branches in `Admin#delete_group`, `Admin#delete_acl` and `Admin#describe_acl`, which referenced undefined local variables. When `rd_kafka_queue_get_background` returned NULL, those branches raised a `NameError` instead of the intended `ConfigError` and leaked the already-allocated native request object, and `delete_group` additionally called the wrong destructor. They now destroy the correct object and raise `ConfigError`.
+- [Fix] Actually populate `ConfigBindingResult#synonyms`. The synonym loop built the synonym binding objects and discarded them, so `#synonyms` was always an empty array; they are now appended.
+- [Fix] Stop leaking the native `rd_kafka_topic_conf_t` in `Producer#set_topic_config` when a per-topic config value is rejected. The conf is built with `rd_kafka_topic_conf_new` and only handed to `rd_kafka_topic_new` afterwards; if `rd_kafka_topic_conf_set` returned a non-`:config_ok` result the method raised before `rd_kafka_topic_new` was ever called, so librdkafka never took ownership of the conf and it leaked. The conf is now destroyed before raising.
+- [Fix] Add the missing `closed_consumer_check` to `Consumer#position`, the only public consumer method without one; calling it on a closed consumer now raises a consistent `ClosedConsumerError` instead of failing deeper in the call.
+- [Fix] Guard the OAuthBearer token refresh callback against a NULL client pointer before calling `rd_kafka_name` on it.
+- [Fix] Make the `Consumer` GC finalizer close the consumer and destroy its consumer queue, not just the native client. The finalizer used the generic `NativeKafka` one, which went straight to `rd_kafka_destroy` and skipped `rd_kafka_consumer_close` and `rd_kafka_queue_destroy`. A consumer that used `poll_batch` (which takes a consumer-queue reference) and was then garbage-collected without an explicit `close` left that reference dangling, which could make `rd_kafka_destroy` block inside the finalizer (process hang at GC/shutdown) or leak the handle. The consumer now installs its own finalizer that mirrors `#close` (the queue pointer is shared with the finalizer via a holder so it never captures the consumer and prevents collection).
+- [Fix] Free the librdkafka-allocated string in `Consumer#cluster_id` and `Consumer#member_id` (previously copied via a `:string` binding but never freed) and fix the `rd_kafka_clusterid` arity to pass `timeout_ms`. `Consumer#cluster_id` now accepts a `timeout_ms` (default `Defaults::CONSUMER_CLUSTER_ID_TIMEOUT_MS`).
+- [Fix] Attach `rd_kafka_query_watermark_offsets` with `blocking: true` so it releases the GVL during its broker round-trip. It was the only synchronous network call bound without the flag, so `Consumer#query_watermark_offsets` (and `Consumer#lag`, which calls it once per partition) froze every other Ruby thread in the process - including producer polling threads - for up to `timeout_ms`. Matches the neighboring `rd_kafka_offsets_for_times` binding.
+- [Fix] Stop leaking the native `rd_kafka_conf_t` when client creation fails. `native_config` raised `ConfigError` mid-build (e.g. on an invalid option) without destroying the conf, and `native_kafka` raised `ClientCreationError` on a null `rd_kafka_new` without destroying it either. Both paths now call `rd_kafka_conf_destroy` before re-raising (librdkafka keeps app ownership of the conf on `rd_kafka_new` failure, so this is safe). Multi-KB leak per failed creation, relevant for supervisors retrying client creation on transient SASL/SSL misconfig.
+- [Fix] Stop `Metadata` from leaking the native metadata struct (and a topic reference) on every retried fetch. `retry` restarts the `begin` block without running its `ensure`, so each retried attempt reassigned the pointers and only the last attempt's `rd_kafka_metadata` struct was ever destroyed; up to `METADATA_MAX_RETRIES` whole-cluster structs leaked per call (the `leader_not_available` case is routine during topic creation/leader election). Each attempt now frees its own native resources, and a failed fetch that never allocated a struct no longer calls `rd_kafka_metadata_destroy` on a NULL pointer.
+- [Fix] Stop `Producer#produce` from orphaning the delivery handle in the process-global registry when it fails after registering it. The handle was only removed on a non-zero `rd_kafka_producev` return, so any exception between registration and that check (a concurrent `close` making `with_inner` raise `ClosedInnerError`, or a header value whose `#to_s` raises) leaked the handle forever - it survives producer close and accumulates in apps that recreate/close producers. `produce` now removes the handle on any such failure before re-raising.
+- [Fix] Guard the message delivery callback so a raising user `delivery_callback` can no longer skip the handle unlock or crash the producer. `DeliveryCallback` invoked the user callback and only then unlocked the handle, with no rescue; if the callback raised, the handle stayed pending (so `wait` blocked until its timeout and raised `WaitTimeoutError` for a message that was actually delivered) and the exception unwound out of the FFI callback on librdkafka's polling thread (`abort_on_exception = true`), taking down the whole process. The user callback is now wrapped so exceptions are logged and swallowed (matching the rebalance callback) and the handle is always unlocked in an `ensure`.
+- [Fix] Stop admin operations (`describe_configs`, `incremental_alter_configs`, `list_offsets`) from leaking native resources when their arguments are rejected. Each parsed user input (`.fetch`/offset validation) only after allocating the background queue, `AdminOptions`, the registered handle and per-resource native objects, so a `KeyError`/`ArgumentError` leaked all of them. Input is now fully parsed and validated before any native allocation.
+- [Fix] Destroy the native topic-partition list in `TopicPartitionList#to_native_tpl` when population fails partway (e.g. an invalid partition). The caller-owned list was leaked on any exception mid-build; it is now destroyed before the error propagates.
+- [Fix] Raise instead of silently dropping a rejected `incremental_alter_configs` entry. `rd_kafka_ConfigResource_add_incremental_config` returns an `rd_kafka_error_t` for an invalid op_type, an empty/nil name, or a nil value on a non-delete op; the result was ignored, so the entry was dropped, the alter request still reported success, and the error object leaked. The error is now surfaced as an `RdkafkaError` (raised before the request is sent) and the native error object is freed.
+- [Fix] Allocate the admin result-count out-parameter as `:size_t` instead of `:int32`. Every librdkafka `rd_kafka_*_result_*(result, size_t *cntp)` accessor writes a full native `size_t` (8 bytes on 64-bit), but the count pointer was a 4-byte `FFI::MemoryPointer.new(:int32)` across the create/delete topic, create partitions, create/delete ACL, describe/incremental-alter configs and config-synonyms paths - a 4-byte heap overflow on every admin result parse (benign on little-endian, where the low word still reads the correct count, but undefined behavior). Now uses `:size_t`.
+- [Fix] Stabilize the flaky `Consumer#lag` "calculates the consumer lag" spec on overloaded CI (e.g. macOS). The manual `consumer.commit` could raise `no_offset` when the default 5s background auto-commit had already committed the stored offsets, or when the auto offset store had not yet caught up with `poll`. The spec now raises `auto.commit.interval.ms` to 60s (matching the existing `#seek`/pause specs) and lets the offset store settle before committing.
+- [Enhancement] Expose `replicas` and `isrs` (in-sync replica broker ids) on each partition in topic metadata (`Metadata#topics` partition hashes). The base struct `#to_h` skips FFI pointer members, so these two arrays were dropped entirely and the partition replica assignment was unavailable to callers (e.g. for planning replication-factor changes). `PartitionMetadata#to_h` now dereferences both pointers into arrays of broker ids.
+- [Fix] Resolve admin operation handles from the event error when an admin operation fails at the operation level (e.g. brokers unreachable, or the client closed with the request in flight). librdkafka delivers such failures as a result event with the error set and an empty results array, but the create topic, delete topic, create partitions, delete groups, create ACL and delete ACL handlers indexed `results[0]` unconditionally. That raised inside the background event callback, so the handle was never unlocked and `wait` blocked until its own timeout and raised `WaitTimeoutError`, discarding the real error. These handlers now check the event error first and resolve the handle with the actual error code (the describe configs, incremental alter configs, describe ACL and list offsets handlers already did).
+- [Enhancement] Extract the admin background-event result handlers into one class per operation under `lib/rdkafka/callbacks/` (`CreateTopicHandler`, `DescribeConfigsHandler`, etc., all subclasses of `Callbacks::BaseHandler`). `Callbacks::BackgroundEventCallback` is now a thin dispatcher that maps the event type to its handler and destroys the event. Purely internal reorganization (`Rdkafka::Callbacks` is private) with no behavior or API change.
+- [Fix] Destroy admin API background events after processing. librdkafka requires the application to destroy each background event, but `rd_kafka_event_destroy` was never called (nor even bound), so every admin operation leaked its entire result event with all result arrays and strings. Reports are now built inside the callback (copying event-owned memory into Ruby objects) before the event is destroyed, and `DescribeConfigsReport`/`IncrementalAlterConfigsReport` no longer destroy the event-owned ConfigResource array, which also fixes a double free on repeated `wait` calls on the same handle. As part of this, the internal FFI struct fields on admin operation handles (e.g. `handle[:error_string]`, `handle[:result_name]`, `handle[:config_entries]`, `handle[:response_string]`, `handle[:matching_acls]`) were removed; they were never part of the public API (use `handle.wait` and the returned report objects, whose interfaces are unchanged).
+- [Enhancement] Reuse per-thread scratch pointers in `Consumer::Headers.from_native` instead of allocating them for every consumed message. Previously each message paid one native pointer allocation just to check for headers and three more when headers were present; now the scratch pointers are allocated once per thread/fiber and reused, removing all per-message native scratch allocations from the consumer hot path.
+- [Enhancement] Remove the unused `DeliveryHandle` `:topic_name` struct field and the per-message allocation that populated it. The delivery callback copied the topic name into a native `FFI::MemoryPointer` on every delivered message, retained for the lifetime of the handle, yet nothing ever read it: the topic is already available via `DeliveryHandle#topic` (a Ruby attribute set during `produce`) and `DeliveryReport#topic_name`, both of which work exactly as before.
+
+## 0.28.0 (2026-06-03)
+- [Enhancement] Bump librdkafka to `2.14.1`
+- [Enhancement] Replace version-gated `Warning[:performance]` with dynamic `Warning.categories` opt-in in spec helper so new Ruby warning categories are automatically enabled without manual updates.
+- [Enhancement] `poll_batch` and `poll_batch_nb` now return error events inline as `RdkafkaError` objects rather than raising on the first error. The return type is `Array<Message, RdkafkaError>` and callers are responsible for handling errors in the result.
+
+## 0.27.0 (2026-05-07)
+- [Feature] Add `Consumer#poll_batch(timeout_ms, max_items:)` and `Consumer#poll_batch_nb(timeout_ms, max_items:)` for batch message polling via `rd_kafka_consume_batch_queue`.
+- [Feature] Add `Config#describe_properties` to dump all librdkafka configuration properties (including defaults and hidden properties) as a Hash via `rd_kafka_conf_dump`.
+- [Enhancement] Bump librdkafka to `2.14.0`
+- [Fix] Fix resource leak in `Admin#describe_configs` and `Admin#incremental_alter_configs` where `admin_options_ptr` and `queue_ptr` were not destroyed in the ensure block.
+- [Fix] Fix leaked queue reference in `Config#native_kafka` where `rd_kafka_queue_get_main` return value was not destroyed after passing to `rd_kafka_set_log_queue`.
+- [Fix] Fix native topic partition list leak in `Consumer#position` where `tpl` was never destroyed.
+
+## 0.26.0 (2026-04-02)
+- [Enhancement] Bump librdkafka to `2.13.2`
+- [Enhancement] Embed a per-file SPEC_HASH in test topic and consumer group names for tracing Kafka warnings back to specific spec files.
+- [Fix] Register `ObjectSpace.define_finalizer` in `Rdkafka::Consumer` to prevent segfaults when a consumer is GC'd without being explicitly closed (jturkel).
+- [Fix] Remove dead `#finalizer` instance methods from `Consumer` and `Admin` that could never work as GC finalizers.
+- [Fix] Prevent cascading test failures in admin specs when a single handle leaks into the registry.
+- [Feature] Extend `Rdkafka::RdkafkaError` with `instance_name` attribute containing the `rd_kafka_name` for tying errors back to specific native Kafka instances (#181).
+
+## 0.25.1 (2026-02-26)
+- [Feature] Support `rd_kafka_ListOffsets` admin API for querying partition offsets by specification (earliest, latest, max_timestamp, or by timestamp) without requiring a consumer group (#613).
+- [Enhancement] Use native ARM64 runners instead of QEMU emulation for Alpine musl aarch64 builds, improving build performance and reliability.
+- [Enhancement] Enable parallel compilation (`make -j$(nproc)`) for ARM64 Alpine musl builds.
+- [Enhancement] Add file descriptor API for fiber scheduler integration. Expose `enable_queue_io_events` and `enable_background_queue_io_events` on `Consumer`, `Producer`, and `Admin` to enable non-blocking monitoring with select/poll/epoll for integration with Ruby fiber schedulers (Falcon, Async) and custom event loops.
+- [Enhancement] Add non-blocking poll methods (`poll_nb`, `events_poll_nb`) on `Consumer` that skip GVL release for efficient fiber scheduler integration when using `poll(0)`.
+- [Enhancement] Add `events_poll_nb_each` method on `Producer`, `Consumer`, and `Admin` for polling events in a single GVL/mutex session. Yields count after each iteration, caller returns `:stop` to break.
+- [Enhancement] Add `poll_nb_each` method on `Consumer` for non-blocking message polling with proper resource cleanup, yielding each message and supporting early termination via `:stop` return value.
+
+## 0.25.0 (2026-01-20)
+- **[Deprecation]** `AbstractHandle#wait` parameter `max_wait_timeout:` (seconds) is deprecated in favor of `max_wait_timeout_ms:` (milliseconds). The old parameter still works but will be removed in v1.0.0.
+- **[Deprecation]** `PartitionsCountCache` constructor parameter `ttl` (seconds) is deprecated in favor of `ttl_ms:` (milliseconds). The old parameter still works but will be removed in v1.0.0.
+- [Enhancement] Extract all timeout defaults to `Rdkafka::Defaults` module for discoverability and per-call overrides (#310). All time-related values are now in milliseconds for consistency.
+- [Enhancement] Add `timeout_ms` parameter to `Consumer#each` for configurable poll timeout.
+- [Enhancement] Extract non-time configuration values (`METADATA_MAX_RETRIES`, `PARTITIONS_COUNT_CACHE_TTL_MS`) to `Rdkafka::Defaults` module.
+- [Enhancement] Bump librdkafka to `2.12.1`
+- [Enhancement] Add descriptive error messages for glibc compatibility issues with instructions for resolution (#654)
+- [Enhancement] Replace magic numbers with named constants throughout codebase for improved readability and maintainability
+
+## 0.24.2 (2025-10-31)
+- [Enhancement] Force lock FFI to 1.17.1 or higher to include critical bug fixes around GCC, write barriers, and thread restarts for forks.
+- [Fix] Fix for Core dump when providing extensions to oauthbearer_set_token (dssjoblom)
+
+## 0.24.1 (2025-10-10)
+- [Fix] Fix Github Action Ruby reference preventing non-compiled releases.
+
+## 0.24.0 (2025-10-10)
+- [Enhancement] Bump librdkafka to `2.11.1`
+
+## 0.23.1 (2025-09-25)
+- [Enhancement] Improve sigstore attestation for precompiled releases.
+- [Fix] Fix incorrectly set default SSL certs dir.
+- [Fix] Disable OpenSSL Heartbeats during compilation.
+
+## 0.23.0 (2025-09-04)
+- **[EOL]** Drop support for Ruby 3.1 to move forward with the fiber scheduler work.
+- [Enhancement] Bump librdkafka to `2.11.0`
+- [Enhancement] Support explicit Debian testing due to lib issues.
+- [Enhancement] Support ARM64 Gnu precompilation.
+- [Enhancement] Improve what symbols are exposed outside of the precompiled extensions.
+- [Enhancement] Introduce an integration suite layer for non RSpec specs execution.
+- [Fix] Add `json` gem as a dependency (was missing but used).
+
 ## 0.22.2 (2025-07-21)
 - [Enhancement] Drastically increase number of platforms in the integration suite
 - [Fix] Support Ubuntu `22.04` and older Alpine precompiled versions
 - [Fix] FFI::DynamicLibrary.load_library': Could not open library
+- [Change] Add new CI action to trigger auto-doc refresh.
 
 ## 0.22.1 (2025-07-17)
 - [Fix] Fix `Rakefile` being available in the precompiled versions causing build failures.
@@ -38,7 +136,7 @@
 - [Patch] Retire no longer needed cooperative-sticky patch.
 
 ## 0.19.0 (2024-10-01)
-- **[Breaking]** Drop Ruby 3.0 support
+- **[EOL]** Drop Ruby 3.0 support
 - [Enhancement] Update `librdkafka` to `2.5.3`
 - [Enhancement] Use default oauth callback if none is passed (bachmanity1)
 - [Fix] Fix incorrectly behaving CI on failures. 
@@ -62,7 +160,7 @@
 - [Fix] Switch to local release of librdkafka to mitigate its unavailability.
 
 ## 0.16.0 (2024-06-13)
-- **[Breaking]** Retire support for Ruby 2.7.
+- **[EOL]** Retire support for Ruby 2.7.
 - **[Breaking]** Messages without headers returned by `#poll` contain frozen empty hash.
 - **[Breaking]** `HashWithSymbolKeysTreatedLikeStrings` has been removed so headers are regular hashes with string keys.
 - **[Feature]** Support incremental config describe + alter API.
@@ -123,7 +221,7 @@
 - [Fix] Reference to Opaque is not released when Admin, Consumer or Producer is closed (mensfeld)
 - [Fix] Trigger `#poll` on native kafka creation to handle oauthbearer cb (mensfeld)
 - [Fix] `#flush` does not handle the timeouts errors by making it return `true` if all flushed or `false` if failed. We do **not** raise an exception here to keep it backwards compatible (mensfeld)
-- [Change] Remove support for Ruby 2.6 due to it being EOL and WeakMap incompatibilities (mensfeld)
+- **[EOL]** Remove support for Ruby 2.6 due to it being EOL and WeakMap incompatibilities (mensfeld)
 - [Change] Update Kafka Docker with Confluent KRaft (mensfeld)
 - [Change] Update librdkafka repo reference from edenhill to confluentinc (mensfeld)
 

data/README.md

@@ -1,6 +1,6 @@
 # Rdkafka
 
-[![Build Status](https://github.com/karafka/rdkafka-ruby/actions/workflows/ci_linux_x86_64_gnu.yml/badge.svg)](https://github.com/karafka/rdkafka-ruby/actions/workflows/ci_linux_x86_64_gnu.yml)
+[![Build Status](https://github.com/karafka/rdkafka-ruby/actions/workflows/ci_linux_ubuntu_x86_64_gnu.yml/badge.svg)](https://github.com/karafka/rdkafka-ruby/actions/workflows/ci_linux_x86_64_gnu.yml)
 [![Gem Version](https://badge.fury.io/rb/rdkafka.svg)](https://badge.fury.io/rb/rdkafka)
 [![Join the chat at https://slack.karafka.io](https://raw.githubusercontent.com/karafka/misc/master/slack.svg)](https://slack.karafka.io)
 
@@ -163,16 +163,23 @@ bundle exec rake produce_messages
 
 | rdkafka-ruby | librdkafka | patches |
 |-|-|-|
-| 0.22.x (2025-07-17) | 2.8.0 (2025-01-07) | yes |
-| 0.21.x (2025-02-13) | 2.8.0 (2025-01-07) | yes |
-| 0.20.0 (2025-01-07) | 2.6.1 (2024-11-18) | yes |
-| 0.19.0 (2024-10-01) | 2.5.3 (2024-09-02) | yes |
-| 0.18.0 (2024-09-02) | 2.5.0 (2024-06-10) | yes |
-| 0.17.0 (2024-08-03) | 2.4.0 (2024-05-07) | no  |
-| 0.16.0 (2024-06-13) | 2.3.0 (2023-10-25) | no  |
-| 0.15.0 (2023-12-03) | 2.3.0 (2023-10-25) | no  |
-| 0.14.0 (2023-11-21) | 2.2.0 (2023-07-12) | no  |
-| 0.13.0 (2023-07-24) | 2.0.2 (2023-01-20) | no  |
-| 0.12.0 (2022-06-17) | 1.9.0 (2022-06-16) | no  |
-| 0.11.0 (2021-11-17) | 1.8.2 (2021-10-18) | no  |
-| 0.10.0 (2021-09-07) | 1.5.0 (2020-07-20) | no  |
+| 0.29.x (2026-07-10) | 2.14.2 (2026-06-03) | yes |
+| 0.28.x (2026-06-03) | 2.14.1 (2026-04-15) | yes |
+| 0.27.x (2026-05-07) | 2.14.0 (2026-04-01) | yes |
+| 0.26.x (2026-03-30) | 2.13.2 (2026-03-02) | yes |
+| 0.25.x (2026-01-21) | 2.12.1 (2025-10-21) | yes |
+| 0.24.x (2025-10-10) | 2.11.1 (2025-08-18) | yes |
+| 0.23.x (2025-09-04) | 2.11.0 (2025-07-03) | yes |
+| 0.22.x (2025-07-17) | 2.8.0 (2025-01-07)  | yes |
+| 0.21.x (2025-02-13) | 2.8.0 (2025-01-07)  | yes |
+| 0.20.0 (2025-01-07) | 2.6.1 (2024-11-18)  | yes |
+| 0.19.0 (2024-10-01) | 2.5.3 (2024-09-02)  | yes |
+| 0.18.0 (2024-09-02) | 2.5.0 (2024-06-10)  | yes |
+| 0.17.0 (2024-08-03) | 2.4.0 (2024-05-07)  | no  |
+| 0.16.0 (2024-06-13) | 2.3.0 (2023-10-25)  | no  |
+| 0.15.0 (2023-12-03) | 2.3.0 (2023-10-25)  | no  |
+| 0.14.0 (2023-11-21) | 2.2.0 (2023-07-12)  | no  |
+| 0.13.0 (2023-07-24) | 2.0.2 (2023-01-20)  | no  |
+| 0.12.0 (2022-06-17) | 1.9.0 (2022-06-16)  | no  |
+| 0.11.0 (2021-11-17) | 1.8.2 (2021-10-18)  | no  |
+| 0.10.0 (2021-09-07) | 1.5.0 (2020-07-20)  | no  |

data/Rakefile

@@ -1,11 +1,11 @@
 # frozen_string_literal: true
 
-require 'bundler/gem_tasks'
+require "bundler/gem_tasks"
 require "./lib/rdkafka"
 
-desc 'Generate some message traffic'
+desc "Generate some message traffic"
 task :produce_messages do
-  config = {:"bootstrap.servers" => "localhost:9092"}
+  config = { "bootstrap.servers": "localhost:9092" }
   if ENV["DEBUG"]
     config[:debug] = "broker,topic,msg"
   end
@@ -15,24 +15,24 @@ task :produce_messages do
   100.times do |i|
     puts "Producing message #{i}"
     delivery_handles << producer.produce(
-        topic:   "rake_test_topic",
-        payload: "Payload #{i} from Rake",
-        key:     "Key #{i} from Rake"
+      topic: "rake_test_topic",
+      payload: "Payload #{i} from Rake",
+      key: "Key #{i} from Rake"
     )
   end
-  puts 'Waiting for delivery'
+  puts "Waiting for delivery"
   delivery_handles.each(&:wait)
-  puts 'Done'
+  puts "Done"
 end
 
-desc 'Consume some messages'
+desc "Consume some messages"
 task :consume_messages do
   config = {
-    :"bootstrap.servers" => "localhost:9092",
-    :"group.id" => "rake_test",
-    :"enable.partition.eof" => false,
-    :"auto.offset.reset" => "earliest",
-    :"statistics.interval.ms" => 10_000
+    "bootstrap.servers": "localhost:9092",
+    "group.id": "rake_test",
+    "enable.partition.eof": false,
+    "auto.offset.reset": "earliest",
+    "statistics.interval.ms": 10_000
   }
   if ENV["DEBUG"]
     config[:debug] = "cgrp,topic,fetch"
@@ -40,7 +40,7 @@ task :consume_messages do
   Rdkafka::Config.statistics_callback = lambda do |stats|
     puts stats
   end
-  consumer = Rdkafka::Config.new(config).consumer
+  Rdkafka::Config.new(config).consumer
   consumer = Rdkafka::Config.new(config).consumer
   consumer.subscribe("rake_test_topic")
   consumer.each do |message|
@@ -48,14 +48,14 @@ task :consume_messages do
   end
 end
 
-desc 'Hammer down'
+desc "Hammer down"
 task :load_test do
   puts "Starting load test"
 
   config = Rdkafka::Config.new(
-    :"bootstrap.servers" => "localhost:9092",
-    :"group.id" => "load-test",
-    :"enable.partition.eof" => false
+    "bootstrap.servers": "localhost:9092",
+    "group.id": "load-test",
+    "enable.partition.eof": false
   )
 
   # Create a producer in a thread
@@ -65,9 +65,9 @@ task :load_test do
       handles = []
       1000.times do |i|
         handles.push(producer.produce(
-          topic:   "load_test_topic",
+          topic: "load_test_topic",
           payload: "Payload #{i}",
-          key:     "Key #{i}"
+          key: "Key #{i}"
         ))
       end
       handles.each(&:wait)

data/bin/verify_kafka_warnings

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# Checks Kafka logs for unsupported warning patterns
+# Only specified warnings are allowed, all others should trigger failure
+
+allowed_patterns=(
+  "Performing controller activation"
+  "registered with feature metadata.version"
+  "Replayed TopicRecord for"
+  "Replayed PartitionRecord for"
+  "Previous leader None and previous leader epoch"
+  "Creating new"
+  "Unloaded transaction metadata"
+  "closing connection"
+  "sent a heartbeat request but received error REQUEST_TIMED_OUT"
+  "Topic '__consumer_offsets' already exists"
+)
+
+# Get all warnings
+warnings=$(docker logs --since=0 kafka | grep "] WARN ")
+exit_code=0
+
+while IFS= read -r line; do
+  allowed=0
+  for pattern in "${allowed_patterns[@]}"; do
+    if echo "$line" | grep -q "$pattern"; then
+      allowed=1
+      break
+    fi
+  done
+
+  if [ $allowed -eq 0 ]; then
+    echo "Unexpected warning found:"
+    echo "$line"
+    exit_code=1
+  fi
+done <<< "$warnings"
+
+exit $exit_code

data/docker-compose-ssl.yml

@@ -0,0 +1,35 @@
+services:
+  kafka:
+    container_name: kafka
+    image: confluentinc/cp-kafka:8.3.0
+    ports:
+      - 9092:9092 # Support PLAINTEXT so we can run one docker setup for SSL and PLAINTEXT
+      - 9093:9093
+    volumes:
+      - ./ssl:/etc/kafka/secrets
+    environment:
+      CLUSTER_ID: kafka-docker-cluster-1
+      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
+      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_PROCESS_ROLES: broker,controller
+      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
+      KAFKA_LISTENERS: PLAINTEXT://:9092,SSL://:9093,CONTROLLER://:9094
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL
+      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:9092,SSL://127.0.0.1:9093
+      KAFKA_BROKER_ID: 1
+      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@127.0.0.1:9094
+      ALLOW_PLAINTEXT_LISTENER: 'yes'
+      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
+      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
+      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
+      KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer
+
+      # SSL Configuration
+      KAFKA_SSL_KEYSTORE_FILENAME: kafka.server.keystore.jks
+      KAFKA_SSL_KEYSTORE_CREDENTIALS: kafka_keystore_creds
+      KAFKA_SSL_KEY_CREDENTIALS: kafka_ssl_key_creds
+      KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.server.truststore.jks
+      KAFKA_SSL_TRUSTSTORE_CREDENTIALS: kafka_truststore_creds
+      KAFKA_SSL_CLIENT_AUTH: none
+      KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""

data/docker-compose.yml

@@ -1,7 +1,7 @@
 services:
   kafka:
     container_name: kafka
-    image: confluentinc/cp-kafka:8.0.0
+    image: confluentinc/cp-kafka:8.3.0
 
     ports:
       - 9092:9092
@@ -18,7 +18,7 @@ services:
       KAFKA_BROKER_ID: 1
       KAFKA_CONTROLLER_QUORUM_VOTERS: 1@127.0.0.1:9093
       ALLOW_PLAINTEXT_LISTENER: 'yes'
-      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
+      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
       KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
       KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
       KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"

data/ext/Rakefile

@@ -1,53 +1,53 @@
 # frozen_string_literal: true
 
-require File.expand_path('../../lib/rdkafka/version', __FILE__)
+require File.expand_path("../../lib/rdkafka/version", __FILE__)
 require "digest"
 require "fileutils"
 require "open-uri"
 
-task :default => :clean do
+task default: :clean do
   # For nix users, nix can't locate the file paths because the packages it's requiring aren't managed by the system but are
   # managed by nix itself, so using the normal file paths doesn't work for nix users.
   #
   # Mini_portile causes an issue because it's dependencies are downloaded on the fly and therefore don't exist/aren't
   # accessible in the nix environment
-  if ENV.fetch('RDKAFKA_EXT_PATH', '').empty?
+  if ENV.fetch("RDKAFKA_EXT_PATH", "").empty?
     # Download and compile librdkafka if RDKAFKA_EXT_PATH is not set
     require "mini_portile2"
     recipe = MiniPortile.new("librdkafka", Rdkafka::LIBRDKAFKA_VERSION)
 
     # Use default homebrew openssl if we're on mac and the directory exists, is not using nix-prepared libraries
     # and each of flags is not already set
-    if recipe.host&.include?("darwin") && system("which brew &> /dev/null") && Dir.exist?("#{homebrew_prefix = %x(brew --prefix openssl).strip}") && !ENV.key?("NIX_LDFLAGS")
+    if recipe.host&.include?("darwin") && system("which brew &> /dev/null") && Dir.exist?((homebrew_prefix = `brew --prefix openssl`.strip).to_s) && !ENV.key?("NIX_LDFLAGS")
       ENV["CPPFLAGS"] = "-I#{homebrew_prefix}/include" unless ENV.key?("CPPFLAGS")
       ENV["LDFLAGS"] = "-L#{homebrew_prefix}/lib" unless ENV.key?("LDFLAGS")
     end
 
-    releases = File.expand_path(File.join(File.dirname(__FILE__), '../dist'))
+    releases = File.expand_path(File.join(File.dirname(__FILE__), "../dist"))
 
     recipe.files << {
-      :url => "file://#{releases}/librdkafka-#{Rdkafka::LIBRDKAFKA_VERSION}.tar.gz",
-      :sha256 => Rdkafka::LIBRDKAFKA_SOURCE_SHA256
+      url: "file://#{releases}/librdkafka-#{Rdkafka::LIBRDKAFKA_VERSION}.tar.gz",
+      sha256: Rdkafka::LIBRDKAFKA_SOURCE_SHA256
     }
     recipe.configure_options = ["--host=#{recipe.host}"]
 
-    recipe.patch_files = Dir[File.join(releases, 'patches', "*.patch")].sort
+    recipe.patch_files = Dir[File.join(releases, "patches", "*.patch")].sort
 
     # Disable using libc regex engine in favor of the embedded one
     # The default regex engine of librdkafka does not always work exactly as most of the users
     # would expect, hence this flag allows for changing it to the other one
-    if ENV.key?('RDKAFKA_DISABLE_REGEX_EXT')
-      recipe.configure_options << '--disable-regex-ext'
+    if ENV.key?("RDKAFKA_DISABLE_REGEX_EXT")
+      recipe.configure_options << "--disable-regex-ext"
     end
 
     recipe.cook
     # Move dynamic library we're interested in
-    if recipe.host.include?('darwin')
-      from_extension = '1.dylib'
-      to_extension   = 'dylib'
+    if recipe.host.include?("darwin")
+      from_extension = "1.dylib"
+      to_extension = "dylib"
     else
-      from_extension = 'so.1'
-      to_extension = 'so'
+      from_extension = "so.1"
+      to_extension = "so"
     end
     lib_path = File.join(File.dirname(__FILE__), "ports/#{recipe.host}/librdkafka/#{Rdkafka::LIBRDKAFKA_VERSION}/lib/librdkafka.#{from_extension}")
     FileUtils.mv(lib_path, File.join(File.dirname(__FILE__), "librdkafka.#{to_extension}"))
@@ -56,12 +56,12 @@ task :default => :clean do
     FileUtils.rm_rf File.join(File.dirname(__FILE__), "ports")
   else
     # Otherwise, copy existing libraries to ./ext
-    if ENV['RDKAFKA_EXT_PATH'].nil? || ENV['RDKAFKA_EXT_PATH'].empty?
+    if ENV["RDKAFKA_EXT_PATH"].nil? || ENV["RDKAFKA_EXT_PATH"].empty?
       raise "RDKAFKA_EXT_PATH must be set in your nix config when running under nix"
     end
     files = [
-      File.join(ENV['RDKAFKA_EXT_PATH'], 'lib', 'librdkafka.dylib'),
-      File.join(ENV['RDKAFKA_EXT_PATH'], 'lib', 'librdkafka.so')
+      File.join(ENV["RDKAFKA_EXT_PATH"], "lib", "librdkafka.dylib"),
+      File.join(ENV["RDKAFKA_EXT_PATH"], "lib", "librdkafka.so")
     ]
     files.each { |ext| FileUtils.cp(ext, File.dirname(__FILE__)) if File.exist?(ext) }
   end
@@ -76,28 +76,28 @@ end
 
 namespace :dist do
   task :dir do
-    ENV["RDKAFKA_DIST_PATH"] ||= File.expand_path(File.join(File.dirname(__FILE__), '..', 'dist'))
+    ENV["RDKAFKA_DIST_PATH"] ||= File.expand_path(File.join(File.dirname(__FILE__), "..", "dist"))
   end
 
-  task :file => "dist:dir" do
+  task file: "dist:dir" do
     ENV["RDKAFKA_DIST_FILE"] ||= File.join(ENV["RDKAFKA_DIST_PATH"], "librdkafka_#{Rdkafka::LIBRDKAFKA_VERSION}.tar.gz")
   end
 
-  task :clean => "dist:file" do
-    Dir.glob(File.join("#{ENV['RDKAFKA_DIST_PATH']}", "*")).each do |filename|
+  task clean: "dist:file" do
+    Dir.glob(File.join(ENV["RDKAFKA_DIST_PATH"].to_s, "*")).each do |filename|
       next if filename.include? ENV["RDKAFKA_DIST_FILE"]
 
       FileUtils.rm_rf filename
     end
   end
 
-  task :download => "dist:file" do
+  task download: "dist:file" do
     version = Rdkafka::LIBRDKAFKA_VERSION
     librdkafka_download = "https://codeload.github.com/confluentinc/librdkafka/tar.gz/v#{version}"
 
     URI.open(librdkafka_download) do |file|
       filename = ENV["RDKAFKA_DIST_FILE"]
-      data     = file.read
+      data = file.read
 
       if Digest::SHA256.hexdigest(data) != Rdkafka::LIBRDKAFKA_SOURCE_SHA256
         raise "SHA256 does not match downloaded file"
@@ -107,7 +107,7 @@ namespace :dist do
     end
   end
 
-  task :update => %w[dist:download dist:clean]
+  task update: %w[dist:download dist:clean]
 end
 
 namespace :build do
@@ -118,8 +118,8 @@ namespace :build do
 
     recipe = MiniPortile.new("librdkafka", version)
     recipe.files << "https://github.com/confluentinc/librdkafka/archive/#{ref}.tar.gz"
-    recipe.configure_options = ["--host=#{recipe.host}","--enable-static", "--enable-zstd"]
-    recipe.patch_files = Dir[File.join(releases, 'patches', "*.patch")].sort
+    recipe.configure_options = ["--host=#{recipe.host}", "--enable-static", "--enable-zstd"]
+    recipe.patch_files = Dir[File.join(releases, "patches", "*.patch")].sort
     recipe.cook
 
     ext = recipe.host.include?("darwin") ? "dylib" : "so"