rdkafka 0.22.2 → 0.27.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6bdfd8912cec56ebe8303d0ad14f3ab7ab2e5e0565e9418a6265a0e48172bc31
-  data.tar.gz: 646f8a174c1019a0417a571f5894c745701c5d670a0585beb40200a2ffe08193
+  metadata.gz: d818b1dce30cffe7b01f9961f05016b70bd99c823f487b3a2c075d335ab31dab
+  data.tar.gz: 5bfd08ae7392e5d13f7d06b4dc366515aacbd995d139b19a82dcc74b4324eaab
 SHA512:
-  metadata.gz: 9d0711b10d02a1c6792fb21206d7729c958fadc186da5b6d8382d69b0d98183cddf23c123ea1602dd890f39603cd44341b864726c4a737966148fffb7e5e5dbd
-  data.tar.gz: deeacf4c205c164d2bd2d786f4a512f62caf8aeda6d6fcc11c24d2d665f4b7bd0a7f44447dfa1ebffc1fe54e7a9cdf36c7bbd4ae71c01e167b46b36a298d7482
+  metadata.gz: b022eff9faf7d80689e17f832fc1017c395c3463100884fcd3c2fb3610a546cf5bf62362b6f2766ef3de13cf2c4ba1b0d6322dcd1908a38ab2db25a8530a3fda
+  data.tar.gz: f489a965325b6c7ab4faa6430b4fdc4912d9223e0cdf623df865e54f4c2baeae8e3d09f167f478d94b77b6f73ddad506b5b7d4f3098dc79c1a3dc58f85ebd941

data/CHANGELOG.md

@@ -1,9 +1,69 @@
 # Rdkafka Changelog
 
+## 0.27.0 (2026-05-07)
+- [Feature] Add `Consumer#poll_batch(timeout_ms, max_items:)` and `Consumer#poll_batch_nb(timeout_ms, max_items:)` for batch message polling via `rd_kafka_consume_batch_queue`.
+- [Feature] Add `Config#describe_properties` to dump all librdkafka configuration properties (including defaults and hidden properties) as a Hash via `rd_kafka_conf_dump`.
+- [Enhancement] Bump librdkafka to `2.14.0`
+- [Fix] Fix resource leak in `Admin#describe_configs` and `Admin#incremental_alter_configs` where `admin_options_ptr` and `queue_ptr` were not destroyed in the ensure block.
+- [Fix] Fix leaked queue reference in `Config#native_kafka` where `rd_kafka_queue_get_main` return value was not destroyed after passing to `rd_kafka_set_log_queue`.
+- [Fix] Fix native topic partition list leak in `Consumer#position` where `tpl` was never destroyed.
+
+## 0.26.0 (2026-04-02)
+- [Enhancement] Bump librdkafka to `2.13.2`
+- [Enhancement] Embed a per-file SPEC_HASH in test topic and consumer group names for tracing Kafka warnings back to specific spec files.
+- [Fix] Register `ObjectSpace.define_finalizer` in `Rdkafka::Consumer` to prevent segfaults when a consumer is GC'd without being explicitly closed (jturkel).
+- [Fix] Remove dead `#finalizer` instance methods from `Consumer` and `Admin` that could never work as GC finalizers.
+- [Fix] Prevent cascading test failures in admin specs when a single handle leaks into the registry.
+- [Feature] Extend `Rdkafka::RdkafkaError` with `instance_name` attribute containing the `rd_kafka_name` for tying errors back to specific native Kafka instances (#181).
+
+## 0.25.1 (2026-02-26)
+- [Feature] Support `rd_kafka_ListOffsets` admin API for querying partition offsets by specification (earliest, latest, max_timestamp, or by timestamp) without requiring a consumer group (#613).
+- [Enhancement] Use native ARM64 runners instead of QEMU emulation for Alpine musl aarch64 builds, improving build performance and reliability.
+- [Enhancement] Enable parallel compilation (`make -j$(nproc)`) for ARM64 Alpine musl builds.
+- [Enhancement] Add file descriptor API for fiber scheduler integration. Expose `enable_queue_io_events` and `enable_background_queue_io_events` on `Consumer`, `Producer`, and `Admin` to enable non-blocking monitoring with select/poll/epoll for integration with Ruby fiber schedulers (Falcon, Async) and custom event loops.
+- [Enhancement] Add non-blocking poll methods (`poll_nb`, `events_poll_nb`) on `Consumer` that skip GVL release for efficient fiber scheduler integration when using `poll(0)`.
+- [Enhancement] Add `events_poll_nb_each` method on `Producer`, `Consumer`, and `Admin` for polling events in a single GVL/mutex session. Yields count after each iteration, caller returns `:stop` to break.
+- [Enhancement] Add `poll_nb_each` method on `Consumer` for non-blocking message polling with proper resource cleanup, yielding each message and supporting early termination via `:stop` return value.
+
+## 0.25.0 (2026-01-20)
+- **[Deprecation]** `AbstractHandle#wait` parameter `max_wait_timeout:` (seconds) is deprecated in favor of `max_wait_timeout_ms:` (milliseconds). The old parameter still works but will be removed in v1.0.0.
+- **[Deprecation]** `PartitionsCountCache` constructor parameter `ttl` (seconds) is deprecated in favor of `ttl_ms:` (milliseconds). The old parameter still works but will be removed in v1.0.0.
+- [Enhancement] Extract all timeout defaults to `Rdkafka::Defaults` module for discoverability and per-call overrides (#310). All time-related values are now in milliseconds for consistency.
+- [Enhancement] Add `timeout_ms` parameter to `Consumer#each` for configurable poll timeout.
+- [Enhancement] Extract non-time configuration values (`METADATA_MAX_RETRIES`, `PARTITIONS_COUNT_CACHE_TTL_MS`) to `Rdkafka::Defaults` module.
+- [Enhancement] Bump librdkafka to `2.12.1`
+- [Enhancement] Add descriptive error messages for glibc compatibility issues with instructions for resolution (#654)
+- [Enhancement] Replace magic numbers with named constants throughout codebase for improved readability and maintainability
+
+## 0.24.2 (2025-10-31)
+- [Enhancement] Force lock FFI to 1.17.1 or higher to include critical bug fixes around GCC, write barriers, and thread restarts for forks.
+- [Fix] Fix for Core dump when providing extensions to oauthbearer_set_token (dssjoblom)
+
+## 0.24.1 (2025-10-10)
+- [Fix] Fix Github Action Ruby reference preventing non-compiled releases.
+
+## 0.24.0 (2025-10-10)
+- [Enhancement] Bump librdkafka to `2.11.1`
+
+## 0.23.1 (2025-09-25)
+- [Enhancement] Improve sigstore attestation for precompiled releases.
+- [Fix] Fix incorrectly set default SSL certs dir.
+- [Fix] Disable OpenSSL Heartbeats during compilation.
+
+## 0.23.0 (2025-09-04)
+- **[EOL]** Drop support for Ruby 3.1 to move forward with the fiber scheduler work.
+- [Enhancement] Bump librdkafka to `2.11.0`
+- [Enhancement] Support explicit Debian testing due to lib issues.
+- [Enhancement] Support ARM64 Gnu precompilation.
+- [Enhancement] Improve what symbols are exposed outside of the precompiled extensions.
+- [Enhancement] Introduce an integration suite layer for non RSpec specs execution.
+- [Fix] Add `json` gem as a dependency (was missing but used).
+
 ## 0.22.2 (2025-07-21)
 - [Enhancement] Drastically increase number of platforms in the integration suite
 - [Fix] Support Ubuntu `22.04` and older Alpine precompiled versions
 - [Fix] FFI::DynamicLibrary.load_library': Could not open library
+- [Change] Add new CI action to trigger auto-doc refresh.
 
 ## 0.22.1 (2025-07-17)
 - [Fix] Fix `Rakefile` being available in the precompiled versions causing build failures.
@@ -38,7 +98,7 @@
 - [Patch] Retire no longer needed cooperative-sticky patch.
 
 ## 0.19.0 (2024-10-01)
-- **[Breaking]** Drop Ruby 3.0 support
+- **[EOL]** Drop Ruby 3.0 support
 - [Enhancement] Update `librdkafka` to `2.5.3`
 - [Enhancement] Use default oauth callback if none is passed (bachmanity1)
 - [Fix] Fix incorrectly behaving CI on failures. 
@@ -62,7 +122,7 @@
 - [Fix] Switch to local release of librdkafka to mitigate its unavailability.
 
 ## 0.16.0 (2024-06-13)
-- **[Breaking]** Retire support for Ruby 2.7.
+- **[EOL]** Retire support for Ruby 2.7.
 - **[Breaking]** Messages without headers returned by `#poll` contain frozen empty hash.
 - **[Breaking]** `HashWithSymbolKeysTreatedLikeStrings` has been removed so headers are regular hashes with string keys.
 - **[Feature]** Support incremental config describe + alter API.
@@ -123,7 +183,7 @@
 - [Fix] Reference to Opaque is not released when Admin, Consumer or Producer is closed (mensfeld)
 - [Fix] Trigger `#poll` on native kafka creation to handle oauthbearer cb (mensfeld)
 - [Fix] `#flush` does not handle the timeouts errors by making it return `true` if all flushed or `false` if failed. We do **not** raise an exception here to keep it backwards compatible (mensfeld)
-- [Change] Remove support for Ruby 2.6 due to it being EOL and WeakMap incompatibilities (mensfeld)
+- **[EOL]** Remove support for Ruby 2.6 due to it being EOL and WeakMap incompatibilities (mensfeld)
 - [Change] Update Kafka Docker with Confluent KRaft (mensfeld)
 - [Change] Update librdkafka repo reference from edenhill to confluentinc (mensfeld)
 

data/Gemfile

@@ -3,3 +3,11 @@
 source "https://rubygems.org"
 
 gemspec
+
+group :development do
+  gem "ostruct"
+  gem "pry"
+  gem "rspec"
+  gem "simplecov"
+  gem "warning"
+end

data/Gemfile.lint

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+source "https://rubygems.org"
+
+# Documentation linting
+gem "yard-lint"
+
+# Code style (StandardRB via RuboCop)
+gem "standard"
+gem "standard-performance"
+gem "rubocop-performance"
+gem "rubocop-rspec"
+gem "standard-rspec"
+gem "rubocop-thread_safety"

data/Gemfile.lint.lock

@@ -0,0 +1,123 @@
+GEM
+  remote: https://rubygems.org/
+  specs:
+    ast (2.4.3)
+    json (2.18.0)
+    language_server-protocol (3.17.0.5)
+    lint_roller (1.1.0)
+    parallel (1.27.0)
+    parser (3.3.10.1)
+      ast (~> 2.4.1)
+      racc
+    prism (1.8.0)
+    racc (1.8.1)
+    rainbow (3.1.1)
+    regexp_parser (2.11.3)
+    rubocop (1.82.1)
+      json (~> 2.3)
+      language_server-protocol (~> 3.17.0.2)
+      lint_roller (~> 1.1.0)
+      parallel (~> 1.10)
+      parser (>= 3.3.0.2)
+      rainbow (>= 2.2.2, < 4.0)
+      regexp_parser (>= 2.9.3, < 3.0)
+      rubocop-ast (>= 1.48.0, < 2.0)
+      ruby-progressbar (~> 1.7)
+      unicode-display_width (>= 2.4.0, < 4.0)
+    rubocop-ast (1.49.0)
+      parser (>= 3.3.7.2)
+      prism (~> 1.7)
+    rubocop-capybara (2.22.1)
+      lint_roller (~> 1.1)
+      rubocop (~> 1.72, >= 1.72.1)
+    rubocop-factory_bot (2.28.0)
+      lint_roller (~> 1.1)
+      rubocop (~> 1.72, >= 1.72.1)
+    rubocop-performance (1.26.1)
+      lint_roller (~> 1.1)
+      rubocop (>= 1.75.0, < 2.0)
+      rubocop-ast (>= 1.47.1, < 2.0)
+    rubocop-rspec (3.9.0)
+      lint_roller (~> 1.1)
+      rubocop (~> 1.81)
+    rubocop-rspec_rails (2.32.0)
+      lint_roller (~> 1.1)
+      rubocop (~> 1.72, >= 1.72.1)
+      rubocop-rspec (~> 3.5)
+    rubocop-thread_safety (0.7.3)
+      lint_roller (~> 1.1)
+      rubocop (~> 1.72, >= 1.72.1)
+      rubocop-ast (>= 1.44.0, < 2.0)
+    ruby-progressbar (1.13.0)
+    standard (1.53.0)
+      language_server-protocol (~> 3.17.0.2)
+      lint_roller (~> 1.0)
+      rubocop (~> 1.82.0)
+      standard-custom (~> 1.0.0)
+      standard-performance (~> 1.8)
+    standard-custom (1.0.2)
+      lint_roller (~> 1.0)
+      rubocop (~> 1.50)
+    standard-performance (1.9.0)
+      lint_roller (~> 1.1)
+      rubocop-performance (~> 1.26.0)
+    standard-rspec (0.3.1)
+      lint_roller (>= 1.0)
+      rubocop-capybara (~> 2.22)
+      rubocop-factory_bot (~> 2.27)
+      rubocop-rspec (~> 3.5)
+      rubocop-rspec_rails (~> 2.31)
+    unicode-display_width (3.2.0)
+      unicode-emoji (~> 4.1)
+    unicode-emoji (4.2.0)
+    yard (0.9.38)
+    yard-lint (1.4.0)
+      yard (~> 0.9)
+      zeitwerk (~> 2.6)
+    zeitwerk (2.7.4)
+
+PLATFORMS
+  ruby
+  x86_64-linux
+
+DEPENDENCIES
+  rubocop-performance
+  rubocop-rspec
+  rubocop-thread_safety
+  standard
+  standard-performance
+  standard-rspec
+  yard-lint
+
+CHECKSUMS
+  ast (2.4.3) sha256=954615157c1d6a382bc27d690d973195e79db7f55e9765ac7c481c60bdb4d383
+  json (2.18.0) sha256=b10506aee4183f5cf49e0efc48073d7b75843ce3782c68dbeb763351c08fd505
+  language_server-protocol (3.17.0.5) sha256=fd1e39a51a28bf3eec959379985a72e296e9f9acfce46f6a79d31ca8760803cc
+  lint_roller (1.1.0) sha256=2c0c845b632a7d172cb849cc90c1bce937a28c5c8ccccb50dfd46a485003cc87
+  parallel (1.27.0) sha256=4ac151e1806b755fb4e2dc2332cbf0e54f2e24ba821ff2d3dcf86bf6dc4ae130
+  parser (3.3.10.1) sha256=06f6a725d2cd91e5e7f2b7c32ba143631e1f7c8ae2fb918fc4cebec187e6a688
+  prism (1.8.0) sha256=84453a16ef5530ea62c5f03ec16b52a459575ad4e7b9c2b360fd8ce2c39c1254
+  racc (1.8.1) sha256=4a7f6929691dbec8b5209a0b373bc2614882b55fc5d2e447a21aaa691303d62f
+  rainbow (3.1.1) sha256=039491aa3a89f42efa1d6dec2fc4e62ede96eb6acd95e52f1ad581182b79bc6a
+  regexp_parser (2.11.3) sha256=ca13f381a173b7a93450e53459075c9b76a10433caadcb2f1180f2c741fc55a4
+  rubocop (1.82.1) sha256=09f1a6a654a960eda767aebea33e47603080f8e9c9a3f019bf9b94c9cab5e273
+  rubocop-ast (1.49.0) sha256=49c3676d3123a0923d333e20c6c2dbaaae2d2287b475273fddee0c61da9f71fd
+  rubocop-capybara (2.22.1) sha256=ced88caef23efea53f46e098ff352f8fc1068c649606ca75cb74650970f51c0c
+  rubocop-factory_bot (2.28.0) sha256=4b17fc02124444173317e131759d195b0d762844a71a29fe8139c1105d92f0cb
+  rubocop-performance (1.26.1) sha256=cd19b936ff196df85829d264b522fd4f98b6c89ad271fa52744a8c11b8f71834
+  rubocop-rspec (3.9.0) sha256=8fa70a3619408237d789aeecfb9beef40576acc855173e60939d63332fdb55e2
+  rubocop-rspec_rails (2.32.0) sha256=4a0d641c72f6ebb957534f539d9d0a62c47abd8ce0d0aeee1ef4701e892a9100
+  rubocop-thread_safety (0.7.3) sha256=067cdd52fbf5deffc18995437e45b5194236eaff4f71de3375a1f6052e48f431
+  ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33
+  standard (1.53.0) sha256=f3c9493385db7079d0abce6f7582f553122156997b81258cd361d3480eeacf9c
+  standard-custom (1.0.2) sha256=424adc84179a074f1a2a309bb9cf7cd6bfdb2b6541f20c6bf9436c0ba22a652b
+  standard-performance (1.9.0) sha256=49483d31be448292951d80e5e67cdcb576c2502103c7b40aec6f1b6e9c88e3f2
+  standard-rspec (0.3.1) sha256=67bc957281cacf24f0d88235ca1bf28a8995265b1a60eb519cd0451858b56a22
+  unicode-display_width (3.2.0) sha256=0cdd96b5681a5949cdbc2c55e7b420facae74c4aaf9a9815eee1087cb1853c42
+  unicode-emoji (4.2.0) sha256=519e69150f75652e40bf736106cfbc8f0f73aa3fb6a65afe62fefa7f80b0f80f
+  yard (0.9.38) sha256=721fb82afb10532aa49860655f6cc2eaa7130889df291b052e1e6b268283010f
+  yard-lint (1.4.0) sha256=7dd88fbb08fd77cb840bea899d58812817b36d92291b5693dd0eeb3af9f91f0f
+  zeitwerk (2.7.4) sha256=2bef90f356bdafe9a6c2bd32bcd804f83a4f9b8bc27f3600fff051eb3edcec8b
+
+BUNDLED WITH
+  4.0.3

data/README.md

@@ -1,6 +1,6 @@
 # Rdkafka
 
-[![Build Status](https://github.com/karafka/rdkafka-ruby/actions/workflows/ci_linux_x86_64_gnu.yml/badge.svg)](https://github.com/karafka/rdkafka-ruby/actions/workflows/ci_linux_x86_64_gnu.yml)
+[![Build Status](https://github.com/karafka/rdkafka-ruby/actions/workflows/ci_linux_ubuntu_x86_64_gnu.yml/badge.svg)](https://github.com/karafka/rdkafka-ruby/actions/workflows/ci_linux_x86_64_gnu.yml)
 [![Gem Version](https://badge.fury.io/rb/rdkafka.svg)](https://badge.fury.io/rb/rdkafka)
 [![Join the chat at https://slack.karafka.io](https://raw.githubusercontent.com/karafka/misc/master/slack.svg)](https://slack.karafka.io)
 
@@ -163,16 +163,21 @@ bundle exec rake produce_messages
 
 | rdkafka-ruby | librdkafka | patches |
 |-|-|-|
-| 0.22.x (2025-07-17) | 2.8.0 (2025-01-07) | yes |
-| 0.21.x (2025-02-13) | 2.8.0 (2025-01-07) | yes |
-| 0.20.0 (2025-01-07) | 2.6.1 (2024-11-18) | yes |
-| 0.19.0 (2024-10-01) | 2.5.3 (2024-09-02) | yes |
-| 0.18.0 (2024-09-02) | 2.5.0 (2024-06-10) | yes |
-| 0.17.0 (2024-08-03) | 2.4.0 (2024-05-07) | no  |
-| 0.16.0 (2024-06-13) | 2.3.0 (2023-10-25) | no  |
-| 0.15.0 (2023-12-03) | 2.3.0 (2023-10-25) | no  |
-| 0.14.0 (2023-11-21) | 2.2.0 (2023-07-12) | no  |
-| 0.13.0 (2023-07-24) | 2.0.2 (2023-01-20) | no  |
-| 0.12.0 (2022-06-17) | 1.9.0 (2022-06-16) | no  |
-| 0.11.0 (2021-11-17) | 1.8.2 (2021-10-18) | no  |
-| 0.10.0 (2021-09-07) | 1.5.0 (2020-07-20) | no  |
+| 0.27.x (2026-05-07) | 2.14.0 (2026-04-01) | yes |
+| 0.26.x (2026-03-30) | 2.13.2 (2026-03-02) | yes |
+| 0.25.x (2026-01-21) | 2.12.1 (2025-10-21) | yes |
+| 0.24.x (2025-10-10) | 2.11.1 (2025-08-18) | yes |
+| 0.23.x (2025-09-04) | 2.11.0 (2025-07-03) | yes |
+| 0.22.x (2025-07-17) | 2.8.0 (2025-01-07)  | yes |
+| 0.21.x (2025-02-13) | 2.8.0 (2025-01-07)  | yes |
+| 0.20.0 (2025-01-07) | 2.6.1 (2024-11-18)  | yes |
+| 0.19.0 (2024-10-01) | 2.5.3 (2024-09-02)  | yes |
+| 0.18.0 (2024-09-02) | 2.5.0 (2024-06-10)  | yes |
+| 0.17.0 (2024-08-03) | 2.4.0 (2024-05-07)  | no  |
+| 0.16.0 (2024-06-13) | 2.3.0 (2023-10-25)  | no  |
+| 0.15.0 (2023-12-03) | 2.3.0 (2023-10-25)  | no  |
+| 0.14.0 (2023-11-21) | 2.2.0 (2023-07-12)  | no  |
+| 0.13.0 (2023-07-24) | 2.0.2 (2023-01-20)  | no  |
+| 0.12.0 (2022-06-17) | 1.9.0 (2022-06-16)  | no  |
+| 0.11.0 (2021-11-17) | 1.8.2 (2021-10-18)  | no  |
+| 0.10.0 (2021-09-07) | 1.5.0 (2020-07-20)  | no  |

data/Rakefile

@@ -1,11 +1,11 @@
 # frozen_string_literal: true
 
-require 'bundler/gem_tasks'
+require "bundler/gem_tasks"
 require "./lib/rdkafka"
 
-desc 'Generate some message traffic'
+desc "Generate some message traffic"
 task :produce_messages do
-  config = {:"bootstrap.servers" => "localhost:9092"}
+  config = { "bootstrap.servers": "localhost:9092" }
   if ENV["DEBUG"]
     config[:debug] = "broker,topic,msg"
   end
@@ -15,24 +15,24 @@ task :produce_messages do
   100.times do |i|
     puts "Producing message #{i}"
     delivery_handles << producer.produce(
-        topic:   "rake_test_topic",
-        payload: "Payload #{i} from Rake",
-        key:     "Key #{i} from Rake"
+      topic: "rake_test_topic",
+      payload: "Payload #{i} from Rake",
+      key: "Key #{i} from Rake"
     )
   end
-  puts 'Waiting for delivery'
+  puts "Waiting for delivery"
   delivery_handles.each(&:wait)
-  puts 'Done'
+  puts "Done"
 end
 
-desc 'Consume some messages'
+desc "Consume some messages"
 task :consume_messages do
   config = {
-    :"bootstrap.servers" => "localhost:9092",
-    :"group.id" => "rake_test",
-    :"enable.partition.eof" => false,
-    :"auto.offset.reset" => "earliest",
-    :"statistics.interval.ms" => 10_000
+    "bootstrap.servers": "localhost:9092",
+    "group.id": "rake_test",
+    "enable.partition.eof": false,
+    "auto.offset.reset": "earliest",
+    "statistics.interval.ms": 10_000
   }
   if ENV["DEBUG"]
     config[:debug] = "cgrp,topic,fetch"
@@ -40,7 +40,7 @@ task :consume_messages do
   Rdkafka::Config.statistics_callback = lambda do |stats|
     puts stats
   end
-  consumer = Rdkafka::Config.new(config).consumer
+  Rdkafka::Config.new(config).consumer
   consumer = Rdkafka::Config.new(config).consumer
   consumer.subscribe("rake_test_topic")
   consumer.each do |message|
@@ -48,14 +48,14 @@ task :consume_messages do
   end
 end
 
-desc 'Hammer down'
+desc "Hammer down"
 task :load_test do
   puts "Starting load test"
 
   config = Rdkafka::Config.new(
-    :"bootstrap.servers" => "localhost:9092",
-    :"group.id" => "load-test",
-    :"enable.partition.eof" => false
+    "bootstrap.servers": "localhost:9092",
+    "group.id": "load-test",
+    "enable.partition.eof": false
   )
 
   # Create a producer in a thread
@@ -65,9 +65,9 @@ task :load_test do
       handles = []
       1000.times do |i|
         handles.push(producer.produce(
-          topic:   "load_test_topic",
+          topic: "load_test_topic",
           payload: "Payload #{i}",
-          key:     "Key #{i}"
+          key: "Key #{i}"
         ))
       end
       handles.each(&:wait)

data/bin/verify_kafka_warnings

@@ -0,0 +1,39 @@
+#!/bin/bash
+
+# Checks Kafka logs for unsupported warning patterns
+# Only specified warnings are allowed, all others should trigger failure
+
+allowed_patterns=(
+  "Performing controller activation"
+  "registered with feature metadata.version"
+  "Replayed TopicRecord for"
+  "Replayed PartitionRecord for"
+  "Previous leader None and previous leader epoch"
+  "Creating new"
+  "Unloaded transaction metadata"
+  "closing connection"
+  "sent a heartbeat request but received error REQUEST_TIMED_OUT"
+  "Topic '__consumer_offsets' already exists"
+)
+
+# Get all warnings
+warnings=$(docker logs --since=0 kafka | grep "] WARN ")
+exit_code=0
+
+while IFS= read -r line; do
+  allowed=0
+  for pattern in "${allowed_patterns[@]}"; do
+    if echo "$line" | grep -q "$pattern"; then
+      allowed=1
+      break
+    fi
+  done
+
+  if [ $allowed -eq 0 ]; then
+    echo "Unexpected warning found:"
+    echo "$line"
+    exit_code=1
+  fi
+done <<< "$warnings"
+
+exit $exit_code

data/docker-compose-ssl.yml

@@ -0,0 +1,35 @@
+services:
+  kafka:
+    container_name: kafka
+    image: confluentinc/cp-kafka:8.2.0
+    ports:
+      - 9092:9092 # Support PLAINTEXT so we can run one docker setup for SSL and PLAINTEXT
+      - 9093:9093
+    volumes:
+      - ./ssl:/etc/kafka/secrets
+    environment:
+      CLUSTER_ID: kafka-docker-cluster-1
+      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
+      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_PROCESS_ROLES: broker,controller
+      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
+      KAFKA_LISTENERS: PLAINTEXT://:9092,SSL://:9093,CONTROLLER://:9094
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL
+      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:9092,SSL://127.0.0.1:9093
+      KAFKA_BROKER_ID: 1
+      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@127.0.0.1:9094
+      ALLOW_PLAINTEXT_LISTENER: 'yes'
+      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
+      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
+      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
+      KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer
+
+      # SSL Configuration
+      KAFKA_SSL_KEYSTORE_FILENAME: kafka.server.keystore.jks
+      KAFKA_SSL_KEYSTORE_CREDENTIALS: kafka_keystore_creds
+      KAFKA_SSL_KEY_CREDENTIALS: kafka_ssl_key_creds
+      KAFKA_SSL_TRUSTSTORE_FILENAME: kafka.server.truststore.jks
+      KAFKA_SSL_TRUSTSTORE_CREDENTIALS: kafka_truststore_creds
+      KAFKA_SSL_CLIENT_AUTH: none
+      KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: ""

data/docker-compose.yml

@@ -1,7 +1,7 @@
 services:
   kafka:
     container_name: kafka
-    image: confluentinc/cp-kafka:8.0.0
+    image: confluentinc/cp-kafka:8.2.0
 
     ports:
       - 9092:9092
@@ -18,7 +18,7 @@ services:
       KAFKA_BROKER_ID: 1
       KAFKA_CONTROLLER_QUORUM_VOTERS: 1@127.0.0.1:9093
       ALLOW_PLAINTEXT_LISTENER: 'yes'
-      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
+      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
       KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
       KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
       KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"

data/ext/Rakefile

@@ -1,53 +1,53 @@
 # frozen_string_literal: true
 
-require File.expand_path('../../lib/rdkafka/version', __FILE__)
+require File.expand_path("../../lib/rdkafka/version", __FILE__)
 require "digest"
 require "fileutils"
 require "open-uri"
 
-task :default => :clean do
+task default: :clean do
   # For nix users, nix can't locate the file paths because the packages it's requiring aren't managed by the system but are
   # managed by nix itself, so using the normal file paths doesn't work for nix users.
   #
   # Mini_portile causes an issue because it's dependencies are downloaded on the fly and therefore don't exist/aren't
   # accessible in the nix environment
-  if ENV.fetch('RDKAFKA_EXT_PATH', '').empty?
+  if ENV.fetch("RDKAFKA_EXT_PATH", "").empty?
     # Download and compile librdkafka if RDKAFKA_EXT_PATH is not set
     require "mini_portile2"
     recipe = MiniPortile.new("librdkafka", Rdkafka::LIBRDKAFKA_VERSION)
 
     # Use default homebrew openssl if we're on mac and the directory exists, is not using nix-prepared libraries
     # and each of flags is not already set
-    if recipe.host&.include?("darwin") && system("which brew &> /dev/null") && Dir.exist?("#{homebrew_prefix = %x(brew --prefix openssl).strip}") && !ENV.key?("NIX_LDFLAGS")
+    if recipe.host&.include?("darwin") && system("which brew &> /dev/null") && Dir.exist?((homebrew_prefix = `brew --prefix openssl`.strip).to_s) && !ENV.key?("NIX_LDFLAGS")
       ENV["CPPFLAGS"] = "-I#{homebrew_prefix}/include" unless ENV.key?("CPPFLAGS")
       ENV["LDFLAGS"] = "-L#{homebrew_prefix}/lib" unless ENV.key?("LDFLAGS")
     end
 
-    releases = File.expand_path(File.join(File.dirname(__FILE__), '../dist'))
+    releases = File.expand_path(File.join(File.dirname(__FILE__), "../dist"))
 
     recipe.files << {
-      :url => "file://#{releases}/librdkafka-#{Rdkafka::LIBRDKAFKA_VERSION}.tar.gz",
-      :sha256 => Rdkafka::LIBRDKAFKA_SOURCE_SHA256
+      url: "file://#{releases}/librdkafka-#{Rdkafka::LIBRDKAFKA_VERSION}.tar.gz",
+      sha256: Rdkafka::LIBRDKAFKA_SOURCE_SHA256
     }
     recipe.configure_options = ["--host=#{recipe.host}"]
 
-    recipe.patch_files = Dir[File.join(releases, 'patches', "*.patch")].sort
+    recipe.patch_files = Dir[File.join(releases, "patches", "*.patch")].sort
 
     # Disable using libc regex engine in favor of the embedded one
     # The default regex engine of librdkafka does not always work exactly as most of the users
     # would expect, hence this flag allows for changing it to the other one
-    if ENV.key?('RDKAFKA_DISABLE_REGEX_EXT')
-      recipe.configure_options << '--disable-regex-ext'
+    if ENV.key?("RDKAFKA_DISABLE_REGEX_EXT")
+      recipe.configure_options << "--disable-regex-ext"
     end
 
     recipe.cook
     # Move dynamic library we're interested in
-    if recipe.host.include?('darwin')
-      from_extension = '1.dylib'
-      to_extension   = 'dylib'
+    if recipe.host.include?("darwin")
+      from_extension = "1.dylib"
+      to_extension = "dylib"
     else
-      from_extension = 'so.1'
-      to_extension = 'so'
+      from_extension = "so.1"
+      to_extension = "so"
     end
     lib_path = File.join(File.dirname(__FILE__), "ports/#{recipe.host}/librdkafka/#{Rdkafka::LIBRDKAFKA_VERSION}/lib/librdkafka.#{from_extension}")
     FileUtils.mv(lib_path, File.join(File.dirname(__FILE__), "librdkafka.#{to_extension}"))
@@ -56,12 +56,12 @@ task :default => :clean do
     FileUtils.rm_rf File.join(File.dirname(__FILE__), "ports")
   else
     # Otherwise, copy existing libraries to ./ext
-    if ENV['RDKAFKA_EXT_PATH'].nil? || ENV['RDKAFKA_EXT_PATH'].empty?
+    if ENV["RDKAFKA_EXT_PATH"].nil? || ENV["RDKAFKA_EXT_PATH"].empty?
       raise "RDKAFKA_EXT_PATH must be set in your nix config when running under nix"
     end
     files = [
-      File.join(ENV['RDKAFKA_EXT_PATH'], 'lib', 'librdkafka.dylib'),
-      File.join(ENV['RDKAFKA_EXT_PATH'], 'lib', 'librdkafka.so')
+      File.join(ENV["RDKAFKA_EXT_PATH"], "lib", "librdkafka.dylib"),
+      File.join(ENV["RDKAFKA_EXT_PATH"], "lib", "librdkafka.so")
     ]
     files.each { |ext| FileUtils.cp(ext, File.dirname(__FILE__)) if File.exist?(ext) }
   end
@@ -76,28 +76,28 @@ end
 
 namespace :dist do
   task :dir do
-    ENV["RDKAFKA_DIST_PATH"] ||= File.expand_path(File.join(File.dirname(__FILE__), '..', 'dist'))
+    ENV["RDKAFKA_DIST_PATH"] ||= File.expand_path(File.join(File.dirname(__FILE__), "..", "dist"))
   end
 
-  task :file => "dist:dir" do
+  task file: "dist:dir" do
     ENV["RDKAFKA_DIST_FILE"] ||= File.join(ENV["RDKAFKA_DIST_PATH"], "librdkafka_#{Rdkafka::LIBRDKAFKA_VERSION}.tar.gz")
   end
 
-  task :clean => "dist:file" do
-    Dir.glob(File.join("#{ENV['RDKAFKA_DIST_PATH']}", "*")).each do |filename|
+  task clean: "dist:file" do
+    Dir.glob(File.join(ENV["RDKAFKA_DIST_PATH"].to_s, "*")).each do |filename|
       next if filename.include? ENV["RDKAFKA_DIST_FILE"]
 
       FileUtils.rm_rf filename
     end
   end
 
-  task :download => "dist:file" do
+  task download: "dist:file" do
     version = Rdkafka::LIBRDKAFKA_VERSION
     librdkafka_download = "https://codeload.github.com/confluentinc/librdkafka/tar.gz/v#{version}"
 
     URI.open(librdkafka_download) do |file|
       filename = ENV["RDKAFKA_DIST_FILE"]
-      data     = file.read
+      data = file.read
 
       if Digest::SHA256.hexdigest(data) != Rdkafka::LIBRDKAFKA_SOURCE_SHA256
         raise "SHA256 does not match downloaded file"
@@ -107,7 +107,7 @@ namespace :dist do
     end
   end
 
-  task :update => %w[dist:download dist:clean]
+  task update: %w[dist:download dist:clean]
 end
 
 namespace :build do
@@ -118,8 +118,8 @@ namespace :build do
 
     recipe = MiniPortile.new("librdkafka", version)
     recipe.files << "https://github.com/confluentinc/librdkafka/archive/#{ref}.tar.gz"
-    recipe.configure_options = ["--host=#{recipe.host}","--enable-static", "--enable-zstd"]
-    recipe.patch_files = Dir[File.join(releases, 'patches', "*.patch")].sort
+    recipe.configure_options = ["--host=#{recipe.host}", "--enable-static", "--enable-zstd"]
+    recipe.patch_files = Dir[File.join(releases, "patches", "*.patch")].sort
     recipe.cook
 
     ext = recipe.host.include?("darwin") ? "dylib" : "so"