karafka-rdkafka 0.13.8 → 0.13.10

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4082c381a9d131273cc005b61b06f0e4c73b27044213d730fa5c7faeec606e07
-  data.tar.gz: 06754bdba16fc3feaf648670e766cdbbc60342b1a43902ef08e42a5acfd4b2ac
+  metadata.gz: a78b060310c1f99afc9955f17d3a2fe92d5252cc7aec870c3fa34b20e543385b
+  data.tar.gz: 5714200f1cc64e0b1b7f4f5118f5a10010055ceb1232986fee4caae952199c44
 SHA512:
-  metadata.gz: 89c0b5078f97c31d0e209ebbf13279a2d079aef35f7b4c2a4bf92cac17a69ccfed1769f84d1d06ca734c9a95174b0b5fa4fa9b6933f63e30fe71c8493bda4ed1
-  data.tar.gz: 490a160689be7c4e261b2b8ff1ffdbcb5aec53cfe1dd0acabd0da33eff708cb7a1d09afe3c3a6752fc9a6345c005f1f0919604035a97240c5f9ed120870e802c
+  metadata.gz: 33338b74897b69cc42f370c57803b048bb6055617509518fba2accd705bb674b09bc311f43116669b135d8e6d5263746c40371d7935bba50e934d1254f50f4a8
+  data.tar.gz: 9b77936f7fa8a53474e04f79ed47e9bedba8524311da0d4bc501261508b6d5e6bf4c02a2202a9b3d41126fe98ce0d03dad5e663a9f73777cf48c4c2644ef4295

data/.gitignore

@@ -1,3 +1,6 @@
+# Ignore bundler config.
+/.bundle
+
 Gemfile.lock
 ext/ports
 ext/tmp
@@ -6,3 +9,4 @@ ext/librdkafka.*
 .yardoc
 doc
 coverage
+vendor

data/.rspec

@@ -1 +1,2 @@
+--require spec_helper
 --format documentation

data/.ruby-gemset

@@ -0,0 +1 @@
+rdkafka-ruby

data/.ruby-version

@@ -0,0 +1 @@
+3.2.2

data/CHANGELOG.md

@@ -1,33 +1,42 @@
-# 0.13.8 (2023-10-31)
+# Rdkafka Changelog
+
+## 0.13.10 (2024-07-10)
+- [Fix] Switch to local release of librdkafka to mitigate its unavailability.
+
+## 0.13.9 (2023-11-07)
+- [Enhancement] Expose alternative way of managing consumer events via a separate queue.
+- [Enhancement] Allow for setting `statistics_callback` as nil to reset predefined settings configured by a different gem.
+
+## 0.13.8 (2023-10-31)
 - [Enhancement] Get consumer position (thijsc & mensfeld)
 
-# 0.13.7 (2023-10-31)
+## 0.13.7 (2023-10-31)
 - [Change] Drop support for Ruby 2.6 due to incompatibilities in usage of `ObjectSpace::WeakMap`
 - [Fix] Fix dangling Opaque references.
 
-# 0.13.6 (2023-10-17)
+## 0.13.6 (2023-10-17)
 * **[Feature]** Support transactions API in the producer
 * [Enhancement] Add `raise_response_error` flag to the `Rdkafka::AbstractHandle`.
 * [Enhancement] Provide `#purge` to remove any outstanding requests from the producer.
 * [Enhancement] Fix `#flush` does not handle the timeouts errors by making it return true if all flushed or false if failed. We do **not** raise an exception here to keep it backwards compatible.
 
-# 0.13.5
+## 0.13.5
 * Fix DeliveryReport `create_result#error` being nil despite an error being associated with it
 
-# 0.13.4
+## 0.13.4
 * Always call initial poll on librdkafka to make sure oauth bearer cb is handled pre-operations.
 
-# 0.13.3
+## 0.13.3
 * Bump librdkafka to 2.2.0
 
-# 0.13.2
+## 0.13.2
 * Ensure operations counter decrement is fully thread-safe
 * Bump librdkafka to 2.1.1
 
-# 0.13.1
+## 0.13.1
 * Add offsets_for_times method on consumer (timflapper)
 
-# 0.13.0
+## 0.13.0 (2023-07-24)
 * Support cooperative sticky partition assignment in the rebalance callback (methodmissing)
 * Support both string and symbol header keys (ColinDKelley)
 * Handle tombstone messages properly (kgalieva)
@@ -48,32 +57,32 @@
 * Retry metadta fetches on certain errors with a backoff (mensfeld)
 * Do not lock access to underlying native kafka client and rely on Karafka granular locking (mensfeld)
 
-# 0.12.3
+## 0.12.3
 - Include backtrace in non-raised binded errors.
 - Include topic name in the delivery reports
 
-# 0.12.2
+## 0.12.2
 * Increase the metadata default timeout from 250ms to 2 seconds. This should allow for working with remote clusters.
 
-# 0.12.1
+## 0.12.1
 * Bumps librdkafka to 2.0.2 (lmaia)
 * Add support for adding more partitions via Admin API
 
-# 0.12.0
+## 0.12.0 (2022-06-17)
 * Bumps librdkafka to 1.9.0
 * Fix crash on empty partition key (mensfeld)
 * Pass the delivery handle to the callback (gvisokinskas)
 
-# 0.11.0
+## 0.11.0 (2021-11-17)
 * Upgrade librdkafka to 1.8.2
 * Bump supported minimum Ruby version to 2.6
 * Better homebrew path detection
 
-# 0.10.0
+## 0.10.0 (2021-09-07)
 * Upgrade librdkafka to 1.5.0
 * Add error callback config
 
-# 0.9.0
+## 0.9.0 (2021-06-23)
 * Fixes for Ruby 3.0
 * Allow any callable object for callbacks (gremerritt)
 * Reduce memory allocations in Rdkafka::Producer#produce (jturkel)
@@ -81,13 +90,13 @@
 * Allow passing in topic configuration on create_topic (dezka)
 * Add each_batch method to consumer (mgrosso)
 
-# 0.8.1
+## 0.8.1 (2020-12-07)
 * Fix topic_flag behaviour and add tests for Metadata (geoff2k)
 * Add topic admin interface (geoff2k)
 * Raise an exception if @native_kafka is nil (geoff2k)
 * Option to use zstd compression (jasonmartens)
 
-# 0.8.0
+## 0.8.0 (2020-06-02)
 * Upgrade librdkafka to 1.4.0
 * Integrate librdkafka metadata API and add partition_key (by Adithya-copart)
 * Ruby 2.7 compatibility fix (by Geoff Thé)A
@@ -95,22 +104,22 @@
 * Don't override CPPFLAGS and LDFLAGS if already set on Mac (by Hiroshi Hatake)
 * Allow use of Rake 13.x and up (by Tomasz Pajor)
 
-# 0.7.0
+## 0.7.0 (2019-09-21)
 * Bump librdkafka to 1.2.0 (by rob-as)
 * Allow customizing the wait time for delivery report availability (by mensfeld)
 
-# 0.6.0
+## 0.6.0 (2019-07-23)
 * Bump librdkafka to 1.1.0 (by Chris Gaffney)
 * Implement seek (by breunigs)
 
-# 0.5.0
+## 0.5.0 (2019-04-11)
 * Bump librdkafka to 1.0.0 (by breunigs)
 * Add cluster and member information (by dmexe)
 * Support message headers for consumer & producer (by dmexe)
 * Add consumer rebalance listener (by dmexe)
 * Implement pause/resume partitions (by dmexe)
 
-# 0.4.2
+## 0.4.2 (2019-01-12)
 * Delivery callback for producer
 * Document list param of commit method
 * Use default Homebrew openssl location if present
@@ -119,10 +128,10 @@
 * Add support for storing message offsets
 * Add missing runtime dependency to rake
 
-# 0.4.1
+## 0.4.1 (2018-10-19)
 * Bump librdkafka to 0.11.6
 
-# 0.4.0
+## 0.4.0 (2018-09-24)
 * Improvements in librdkafka archive download
 * Add global statistics callback
 * Use Time for timestamps, potentially breaking change if you
@@ -134,34 +143,34 @@
 * Support committing a topic partition list
 * Add consumer assignment method
 
-# 0.3.5
+## 0.3.5 (2018-01-17)
 * Fix crash when not waiting for delivery handles
 * Run specs on Ruby 2.5
 
-# 0.3.4
+## 0.3.4 (2017-12-05)
 * Bump librdkafka to 0.11.3
 
-# 0.3.3
+## 0.3.3 (2017-10-27)
 * Fix bug that prevent display of `RdkafkaError` message
 
-# 0.3.2
+## 0.3.2 (2017-10-25)
 * `add_topic` now supports using a partition count
 * Add way to make errors clearer with an extra message
 * Show topics in subscribe error message
 * Show partition and topic in query watermark offsets error message
 
-# 0.3.1
+## 0.3.1 (2017-10-23)
 * Bump librdkafka to 0.11.1
 * Officially support ranges in `add_topic` for topic partition list.
 * Add consumer lag calculator
 
-# 0.3.0
+## 0.3.0 (2017-10-17)
 * Move both add topic methods to one `add_topic` in `TopicPartitionList`
 * Add committed offsets to consumer
 * Add query watermark offset to consumer
 
-# 0.2.0
+## 0.2.0 (2017-10-13)
 * Some refactoring and add inline documentation
 
-# 0.1.x
+## 0.1.x (2017-09-10)
 * Initial working version including producing and consuming

data/{LICENSE → MIT-LICENSE}

@@ -1,6 +1,7 @@
 The MIT License (MIT)
 
-Copyright (c) 2017 Thijs Cadier
+Copyright (c) 2017-2023 Thijs Cadier
+              2023, Maciej Mensfeld
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -10,16 +10,16 @@
 ---
 
 The `rdkafka` gem is a modern Kafka client library for Ruby based on
-[librdkafka](https://github.com/edenhill/librdkafka/).
+[librdkafka](https://github.com/confluentinc/librdkafka/).
 It wraps the production-ready C client using the [ffi](https://github.com/ffi/ffi)
-gem and targets Kafka 1.0+ and Ruby versions that are under security or
-active maintenance. We remove Ruby version from our CI builds if they 
+gem and targets Kafka 1.0+ and Ruby versions under security or
+active maintenance. We remove a Ruby version from our CI builds when they 
 become EOL.
 
 `rdkafka` was written because of the need for a reliable Ruby client for Kafka that supports modern Kafka at [AppSignal](https://appsignal.com). AppSignal runs it in production on very high-traffic systems.
 
 The most important pieces of a Kafka client are implemented. We're
-working towards feature completeness, you can track that here:
+working towards feature completeness. You can track that here:
 https://github.com/appsignal/rdkafka-ruby/milestone/1
 
 ## Table of content
@@ -38,7 +38,7 @@ https://github.com/appsignal/rdkafka-ruby/milestone/1
 ## Installation
 
 This gem downloads and compiles librdkafka when it is installed. If you
-have any problems installing the gem please open an issue.
+If you have any problems installing the gem, please open an issue.
 
 ## Usage
 
@@ -64,9 +64,9 @@ end
 
 ### Producing messages
 
-Produce a number of messages, put the delivery handles in an array and
+Produce a number of messages, put the delivery handles in an array, and
 wait for them before exiting. This way the messages will be batched and
-sent to Kafka in an efficient way.
+efficiently sent to Kafka.
 
 ```ruby
 config = {:"bootstrap.servers" => "localhost:9092"}
@@ -91,7 +91,7 @@ released until it `#close` is explicitly called, so be sure to call
 
 ## Higher level libraries
 
-Currently, there are two actively developed frameworks based on rdkafka-ruby, that provide higher level API that can be used to work with Kafka messages and one library for publishing messages.
+Currently, there are two actively developed frameworks based on rdkafka-ruby, that provide higher-level API that can be used to work with Kafka messages and one library for publishing messages.
 
 ### Message processing frameworks
 
@@ -104,7 +104,7 @@ Currently, there are two actively developed frameworks based on rdkafka-ruby, th
 
 ## Development
 
-A Docker Compose file is included to run Kafka and Zookeeper. To run
+A Docker Compose file is included to run Kafka. To run
 that:
 
 ```
@@ -122,7 +122,7 @@ DEBUG_PRODUCER=true bundle exec rspec
 DEBUG_CONSUMER=true bundle exec rspec
 ```
 
-After running the tests you can bring the cluster down to start with a
+After running the tests, you can bring the cluster down to start with a
 clean slate:
 
 ```
@@ -131,7 +131,7 @@ docker-compose down
 
 ## Example
 
-To see everything working run these in separate tabs:
+To see everything working, run these in separate tabs:
 
 ```
 bundle exec rake consume_messages

data/ext/README.md

@@ -5,7 +5,7 @@ this gem is installed.
 
 To update the `librdkafka` version follow the following steps:
 
-* Go to https://github.com/edenhill/librdkafka/releases to get the new
+* Go to https://github.com/confluentinc/librdkafka/releases to get the new
   version number and asset checksum for `tar.gz`.
 * Change the version in `lib/rdkafka/version.rb`
 * Change the `sha256` in `lib/rdkafka/version.rb`

data/ext/Rakefile

@@ -1,40 +1,67 @@
 # frozen_string_literal: true
 
 require File.expand_path('../../lib/rdkafka/version', __FILE__)
-require "mini_portile2"
 require "fileutils"
 require "open-uri"
 
 task :default => :clean do
-  # Download and compile librdkafka
-  recipe = MiniPortile.new("librdkafka", Rdkafka::LIBRDKAFKA_VERSION)
+  # For nix users, nix can't locate the file paths because the packages it's requiring aren't managed by the system but are
+  # managed by nix itself, so using the normal file paths doesn't work for nix users.
+  #
+  # Mini_portile causes an issue because it's dependencies are downloaded on the fly and therefore don't exist/aren't
+  # accessible in the nix environment
+  if ENV.fetch('RDKAFKA_EXT_PATH', '').empty?
+    # Download and compile librdkafka if RDKAFKA_EXT_PATH is not set
+    require "mini_portile2"
+    recipe = MiniPortile.new("librdkafka", Rdkafka::LIBRDKAFKA_VERSION)
 
-  # Use default homebrew openssl if we're on mac and the directory exists
-  # and each of flags is not empty
-  if recipe.host&.include?("darwin") && system("which brew &> /dev/null") && Dir.exist?("#{homebrew_prefix = %x(brew --prefix openssl).strip}")
-    ENV["CPPFLAGS"] = "-I#{homebrew_prefix}/include" unless ENV["CPPFLAGS"]
-    ENV["LDFLAGS"] = "-L#{homebrew_prefix}/lib" unless ENV["LDFLAGS"]
-  end
+    # Use default homebrew openssl if we're on mac and the directory exists
+    # and each of flags is not empty
+    if recipe.host&.include?("darwin") && system("which brew &> /dev/null") && Dir.exist?("#{homebrew_prefix = %x(brew --prefix openssl).strip}")
+      ENV["CPPFLAGS"] = "-I#{homebrew_prefix}/include" unless ENV["CPPFLAGS"]
+      ENV["LDFLAGS"] = "-L#{homebrew_prefix}/lib" unless ENV["LDFLAGS"]
+    end
+
+    releases = File.expand_path(File.join(File.dirname(__FILE__), '../dist'))
+
+    recipe.files << {
+      :url => "file://#{releases}/librdkafka_#{Rdkafka::LIBRDKAFKA_VERSION}.tar.gz",
+      :sha256 => Rdkafka::LIBRDKAFKA_SOURCE_SHA256
+    }
+    recipe.configure_options = ["--host=#{recipe.host}"]
+
+    # Disable using libc regex engine in favor of the embedded one
+    # The default regex engine of librdkafka does not always work exactly as most of the users
+    # would expect, hence this flag allows for changing it to the other one
+    if ENV.key?('RDKAFKA_DISABLE_REGEX_EXT')
+      recipe.configure_options << '--disable-regex-ext'
+    end
 
-  recipe.files << {
-    :url => "https://codeload.github.com/edenhill/librdkafka/tar.gz/v#{Rdkafka::LIBRDKAFKA_VERSION}",
-    :sha256 => Rdkafka::LIBRDKAFKA_SOURCE_SHA256
-  }
-  recipe.configure_options = ["--host=#{recipe.host}"]
-  recipe.cook
-  # Move dynamic library we're interested in
-  if recipe.host.include?('darwin')
-    from_extension = '1.dylib'
-    to_extension   = 'dylib'
+    recipe.cook
+    # Move dynamic library we're interested in
+    if recipe.host.include?('darwin')
+      from_extension = '1.dylib'
+      to_extension   = 'dylib'
+    else
+      from_extension = 'so.1'
+      to_extension = 'so'
+    end
+    lib_path = File.join(File.dirname(__FILE__), "ports/#{recipe.host}/librdkafka/#{Rdkafka::LIBRDKAFKA_VERSION}/lib/librdkafka.#{from_extension}")
+    FileUtils.mv(lib_path, File.join(File.dirname(__FILE__), "librdkafka.#{to_extension}"))
+    # Cleanup files created by miniportile we don't need in the gem
+    FileUtils.rm_rf File.join(File.dirname(__FILE__), "tmp")
+    FileUtils.rm_rf File.join(File.dirname(__FILE__), "ports")
   else
-    from_extension = 'so.1'
-    to_extension = 'so'
+    # Otherwise, copy existing libraries to ./ext
+    if ENV['RDKAFKA_EXT_PATH'].nil? || ENV['RDKAFKA_EXT_PATH'].empty?
+      raise "RDKAFKA_EXT_PATH must be set in your nix config when running under nix"
+    end
+    files = [
+      File.join(ENV['RDKAFKA_EXT_PATH'], 'lib', 'librdkafka.dylib'),
+      File.join(ENV['RDKAFKA_EXT_PATH'], 'lib', 'librdkafka.so')
+    ]
+    files.each { |ext| FileUtils.cp(ext, File.dirname(__FILE__)) if File.exist?(ext) }
   end
-  lib_path = File.join(File.dirname(__FILE__), "ports/#{recipe.host}/librdkafka/#{Rdkafka::LIBRDKAFKA_VERSION}/lib/librdkafka.#{from_extension}")
-  FileUtils.mv(lib_path, File.join(File.dirname(__FILE__), "librdkafka.#{to_extension}"))
-  # Cleanup files created by miniportile we don't need in the gem
-  FileUtils.rm_rf File.join(File.dirname(__FILE__), "tmp")
-  FileUtils.rm_rf File.join(File.dirname(__FILE__), "ports")
 end
 
 task :clean do

data/lib/rdkafka/abstract_handle.rb

@@ -1,28 +1,37 @@
 # frozen_string_literal: true
 
-require "ffi"
-
 module Rdkafka
+  # This class serves as an abstract base class to represent handles within the Rdkafka module.
+  # As a subclass of `FFI::Struct`, this class provides a blueprint for other specific handle
+  # classes to inherit from, ensuring they adhere to a particular structure and behavior.
+  #
+  # Subclasses must define their own layout, and the layout must start with:
+  #
+  # layout :pending, :bool,
+  #        :response, :int
   class AbstractHandle < FFI::Struct
-    # Subclasses must define their own layout, and the layout must start with:
-    #
-    # layout :pending, :bool,
-    #        :response, :int
+    include Helpers::Time
 
+    # Registry for registering all the handles.
     REGISTRY = {}
 
-    CURRENT_TIME = -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) }.freeze
-
-    private_constant :CURRENT_TIME
+    class << self
+      # Adds handle to the register
+      #
+      # @param handle [AbstractHandle] any handle we want to register
+      def register(handle)
+        address = handle.to_ptr.address
+        REGISTRY[address] = handle
+      end
 
-    def self.register(handle)
-      address = handle.to_ptr.address
-      REGISTRY[address] = handle
+      # Removes handle from the register based on the handle address
+      #
+      # @param address [Integer] address of the registered handle we want to remove
+      def remove(address)
+        REGISTRY.delete(address)
+      end
     end
 
-    def self.remove(address)
-      REGISTRY.delete(address)
-    end
 
     # Whether the handle is still pending.
     #
@@ -32,27 +41,31 @@ module Rdkafka
     end
 
     # Wait for the operation to complete or raise an error if this takes longer than the timeout.
-    # If there is a timeout this does not mean the operation failed, rdkafka might still be working on the operation.
-    # In this case it is possible to call wait again.
+    # If there is a timeout this does not mean the operation failed, rdkafka might still be working
+    # on the operation. In this case it is possible to call wait again.
     #
-    # @param max_wait_timeout [Numeric, nil] Amount of time to wait before timing out. If this is nil it does not time out.
-    # @param wait_timeout [Numeric] Amount of time we should wait before we recheck if the operation has completed
+    # @param max_wait_timeout [Numeric, nil] Amount of time to wait before timing out.
+    #   If this is nil it does not time out.
+    # @param wait_timeout [Numeric] Amount of time we should wait before we recheck if the
+    #   operation has completed
     # @param raise_response_error [Boolean] should we raise error when waiting finishes
     #
+    # @return [Object] Operation-specific result
+    #
     # @raise [RdkafkaError] When the operation failed
     # @raise [WaitTimeoutError] When the timeout has been reached and the handle is still pending
-    #
-    # @return [Object] Operation-specific result
     def wait(max_wait_timeout: 60, wait_timeout: 0.1, raise_response_error: true)
       timeout = if max_wait_timeout
-                  CURRENT_TIME.call + max_wait_timeout
+                  monotonic_now + max_wait_timeout
                 else
                   nil
                 end
       loop do
         if pending?
-          if timeout && timeout <= CURRENT_TIME.call
-            raise WaitTimeoutError.new("Waiting for #{operation_name} timed out after #{max_wait_timeout} seconds")
+          if timeout && timeout <= monotonic_now
+            raise WaitTimeoutError.new(
+              "Waiting for #{operation_name} timed out after #{max_wait_timeout} seconds"
+            )
           end
           sleep wait_timeout
         elsif self[:response] != 0 && raise_response_error

data/lib/rdkafka/admin.rb

@@ -1,7 +1,5 @@
 # frozen_string_literal: true
 
-require "objspace"
-
 module Rdkafka
   class Admin
     # @private
@@ -30,11 +28,12 @@ module Rdkafka
 
     # Create a topic with the given partition count and replication factor
     #
+    # @return [CreateTopicHandle] Create topic handle that can be used to wait for the result of
+    #   creating the topic
+    #
     # @raise [ConfigError] When the partition count or replication factor are out of valid range
     # @raise [RdkafkaError] When the topic name is invalid or the topic already exists
     # @raise [RdkafkaError] When the topic configuration is invalid
-    #
-    # @return [CreateTopicHandle] Create topic handle that can be used to wait for the result of creating the topic
     def create_topic(topic_name, partition_count, replication_factor, topic_config={})
       closed_admin_check(__method__)
 
@@ -107,11 +106,11 @@ module Rdkafka
       create_topic_handle
     end
 
-    # Delete the named topic
+    # Deletes the named topic
     #
+    # @return [DeleteTopicHandle] Delete topic handle that can be used to wait for the result of
+    #   deleting the topic
     # @raise [RdkafkaError] When the topic name is invalid or the topic does not exist
-    #
-    # @return [DeleteTopicHandle] Delete topic handle that can be used to wait for the result of deleting the topic
     def delete_topic(topic_name)
       closed_admin_check(__method__)
 

data/lib/rdkafka/bindings.rb

@@ -1,9 +1,5 @@
 # frozen_string_literal: true
 
-require "ffi"
-require "json"
-require "logger"
-
 module Rdkafka
   # @private
   #