karafka-rdkafka 0.13.8 → 0.14.0.beta1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4082c381a9d131273cc005b61b06f0e4c73b27044213d730fa5c7faeec606e07
-  data.tar.gz: 06754bdba16fc3feaf648670e766cdbbc60342b1a43902ef08e42a5acfd4b2ac
+  metadata.gz: 5e180bea275e4b1e2b0f43ba8ec7215b0b96cb926b708ee38a6a4cb73116be53
+  data.tar.gz: 6558fb60a50c96bd11054400e5effb89f42c9b2a35ea23f02cf3b56f96c7142a
 SHA512:
-  metadata.gz: 89c0b5078f97c31d0e209ebbf13279a2d079aef35f7b4c2a4bf92cac17a69ccfed1769f84d1d06ca734c9a95174b0b5fa4fa9b6933f63e30fe71c8493bda4ed1
-  data.tar.gz: 490a160689be7c4e261b2b8ff1ffdbcb5aec53cfe1dd0acabd0da33eff708cb7a1d09afe3c3a6752fc9a6345c005f1f0919604035a97240c5f9ed120870e802c
+  metadata.gz: 7449b5221257909fbf806e5a78446566659ffb71ed40ee2ced5ef51624a2ad9b8c710edf4a34dd024c96e03cd698698cd235c1b58cf3e9d407e051e95be80fa9
+  data.tar.gz: 3ad4a979ce55647485d75f2a45da3dd444ced6231b555557b3d888ef6641365b4fb675a7e29f019a2df7ff960cc2561d0141291d8a344be5bd87b413ee64a6e3

data/.gitignore

@@ -1,3 +1,6 @@
+# Ignore bundler config.
+/.bundle
+
 Gemfile.lock
 ext/ports
 ext/tmp
@@ -6,3 +9,4 @@ ext/librdkafka.*
 .yardoc
 doc
 coverage
+vendor

data/.rspec

@@ -1 +1,2 @@
+--require spec_helper
 --format documentation

data/.ruby-gemset

@@ -0,0 +1 @@
+rdkafka-ruby

data/.ruby-version

@@ -0,0 +1 @@
+3.2.2

data/CHANGELOG.md

@@ -1,33 +1,43 @@
-# 0.13.8 (2023-10-31)
+# Rdkafka Changelog
+
+## 0.14.0 (Unreleased)
+- [Enhancement] Bump librdkafka to 2.3.0
+- [Enhancement] Increase the `#lag` and `#query_watermark_offsets` default timeouts from 100ms to 1000ms. This will compensate for network glitches and remote clusters operations.
+
+## 0.13.9 (2023-11-07)
+- [Enhancement] Expose alternative way of managing consumer events via a separate queue.
+- [Enhancement] Allow for setting `statistics_callback` as nil to reset predefined settings configured by a different gem.
+
+## 0.13.8 (2023-10-31)
 - [Enhancement] Get consumer position (thijsc & mensfeld)
 
-# 0.13.7 (2023-10-31)
+## 0.13.7 (2023-10-31)
 - [Change] Drop support for Ruby 2.6 due to incompatibilities in usage of `ObjectSpace::WeakMap`
 - [Fix] Fix dangling Opaque references.
 
-# 0.13.6 (2023-10-17)
+## 0.13.6 (2023-10-17)
 * **[Feature]** Support transactions API in the producer
 * [Enhancement] Add `raise_response_error` flag to the `Rdkafka::AbstractHandle`.
 * [Enhancement] Provide `#purge` to remove any outstanding requests from the producer.
 * [Enhancement] Fix `#flush` does not handle the timeouts errors by making it return true if all flushed or false if failed. We do **not** raise an exception here to keep it backwards compatible.
 
-# 0.13.5
+## 0.13.5
 * Fix DeliveryReport `create_result#error` being nil despite an error being associated with it
 
-# 0.13.4
+## 0.13.4
 * Always call initial poll on librdkafka to make sure oauth bearer cb is handled pre-operations.
 
-# 0.13.3
+## 0.13.3
 * Bump librdkafka to 2.2.0
 
-# 0.13.2
+## 0.13.2
 * Ensure operations counter decrement is fully thread-safe
 * Bump librdkafka to 2.1.1
 
-# 0.13.1
+## 0.13.1
 * Add offsets_for_times method on consumer (timflapper)
 
-# 0.13.0
+## 0.13.0 (2023-07-24)
 * Support cooperative sticky partition assignment in the rebalance callback (methodmissing)
 * Support both string and symbol header keys (ColinDKelley)
 * Handle tombstone messages properly (kgalieva)
@@ -48,32 +58,32 @@
 * Retry metadta fetches on certain errors with a backoff (mensfeld)
 * Do not lock access to underlying native kafka client and rely on Karafka granular locking (mensfeld)
 
-# 0.12.3
+## 0.12.3
 - Include backtrace in non-raised binded errors.
 - Include topic name in the delivery reports
 
-# 0.12.2
+## 0.12.2
 * Increase the metadata default timeout from 250ms to 2 seconds. This should allow for working with remote clusters.
 
-# 0.12.1
+## 0.12.1
 * Bumps librdkafka to 2.0.2 (lmaia)
 * Add support for adding more partitions via Admin API
 
-# 0.12.0
+## 0.12.0 (2022-06-17)
 * Bumps librdkafka to 1.9.0
 * Fix crash on empty partition key (mensfeld)
 * Pass the delivery handle to the callback (gvisokinskas)
 
-# 0.11.0
+## 0.11.0 (2021-11-17)
 * Upgrade librdkafka to 1.8.2
 * Bump supported minimum Ruby version to 2.6
 * Better homebrew path detection
 
-# 0.10.0
+## 0.10.0 (2021-09-07)
 * Upgrade librdkafka to 1.5.0
 * Add error callback config
 
-# 0.9.0
+## 0.9.0 (2021-06-23)
 * Fixes for Ruby 3.0
 * Allow any callable object for callbacks (gremerritt)
 * Reduce memory allocations in Rdkafka::Producer#produce (jturkel)
@@ -81,13 +91,13 @@
 * Allow passing in topic configuration on create_topic (dezka)
 * Add each_batch method to consumer (mgrosso)
 
-# 0.8.1
+## 0.8.1 (2020-12-07)
 * Fix topic_flag behaviour and add tests for Metadata (geoff2k)
 * Add topic admin interface (geoff2k)
 * Raise an exception if @native_kafka is nil (geoff2k)
 * Option to use zstd compression (jasonmartens)
 
-# 0.8.0
+## 0.8.0 (2020-06-02)
 * Upgrade librdkafka to 1.4.0
 * Integrate librdkafka metadata API and add partition_key (by Adithya-copart)
 * Ruby 2.7 compatibility fix (by Geoff Thé)A
@@ -95,22 +105,22 @@
 * Don't override CPPFLAGS and LDFLAGS if already set on Mac (by Hiroshi Hatake)
 * Allow use of Rake 13.x and up (by Tomasz Pajor)
 
-# 0.7.0
+## 0.7.0 (2019-09-21)
 * Bump librdkafka to 1.2.0 (by rob-as)
 * Allow customizing the wait time for delivery report availability (by mensfeld)
 
-# 0.6.0
+## 0.6.0 (2019-07-23)
 * Bump librdkafka to 1.1.0 (by Chris Gaffney)
 * Implement seek (by breunigs)
 
-# 0.5.0
+## 0.5.0 (2019-04-11)
 * Bump librdkafka to 1.0.0 (by breunigs)
 * Add cluster and member information (by dmexe)
 * Support message headers for consumer & producer (by dmexe)
 * Add consumer rebalance listener (by dmexe)
 * Implement pause/resume partitions (by dmexe)
 
-# 0.4.2
+## 0.4.2 (2019-01-12)
 * Delivery callback for producer
 * Document list param of commit method
 * Use default Homebrew openssl location if present
@@ -119,10 +129,10 @@
 * Add support for storing message offsets
 * Add missing runtime dependency to rake
 
-# 0.4.1
+## 0.4.1 (2018-10-19)
 * Bump librdkafka to 0.11.6
 
-# 0.4.0
+## 0.4.0 (2018-09-24)
 * Improvements in librdkafka archive download
 * Add global statistics callback
 * Use Time for timestamps, potentially breaking change if you
@@ -134,34 +144,34 @@
 * Support committing a topic partition list
 * Add consumer assignment method
 
-# 0.3.5
+## 0.3.5 (2018-01-17)
 * Fix crash when not waiting for delivery handles
 * Run specs on Ruby 2.5
 
-# 0.3.4
+## 0.3.4 (2017-12-05)
 * Bump librdkafka to 0.11.3
 
-# 0.3.3
+## 0.3.3 (2017-10-27)
 * Fix bug that prevent display of `RdkafkaError` message
 
-# 0.3.2
+## 0.3.2 (2017-10-25)
 * `add_topic` now supports using a partition count
 * Add way to make errors clearer with an extra message
 * Show topics in subscribe error message
 * Show partition and topic in query watermark offsets error message
 
-# 0.3.1
+## 0.3.1 (2017-10-23)
 * Bump librdkafka to 0.11.1
 * Officially support ranges in `add_topic` for topic partition list.
 * Add consumer lag calculator
 
-# 0.3.0
+## 0.3.0 (2017-10-17)
 * Move both add topic methods to one `add_topic` in `TopicPartitionList`
 * Add committed offsets to consumer
 * Add query watermark offset to consumer
 
-# 0.2.0
+## 0.2.0 (2017-10-13)
 * Some refactoring and add inline documentation
 
-# 0.1.x
+## 0.1.x (2017-09-10)
 * Initial working version including producing and consuming

data/{LICENSE → MIT-LICENSE}

@@ -1,6 +1,7 @@
 The MIT License (MIT)
 
-Copyright (c) 2017 Thijs Cadier
+Copyright (c) 2017-2023 Thijs Cadier
+              2023, Maciej Mensfeld
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -10,16 +10,16 @@
 ---
 
 The `rdkafka` gem is a modern Kafka client library for Ruby based on
-[librdkafka](https://github.com/edenhill/librdkafka/).
+[librdkafka](https://github.com/confluentinc/librdkafka/).
 It wraps the production-ready C client using the [ffi](https://github.com/ffi/ffi)
-gem and targets Kafka 1.0+ and Ruby versions that are under security or
-active maintenance. We remove Ruby version from our CI builds if they 
+gem and targets Kafka 1.0+ and Ruby versions under security or
+active maintenance. We remove a Ruby version from our CI builds when they 
 become EOL.
 
 `rdkafka` was written because of the need for a reliable Ruby client for Kafka that supports modern Kafka at [AppSignal](https://appsignal.com). AppSignal runs it in production on very high-traffic systems.
 
 The most important pieces of a Kafka client are implemented. We're
-working towards feature completeness, you can track that here:
+working towards feature completeness. You can track that here:
 https://github.com/appsignal/rdkafka-ruby/milestone/1
 
 ## Table of content
@@ -38,7 +38,7 @@ https://github.com/appsignal/rdkafka-ruby/milestone/1
 ## Installation
 
 This gem downloads and compiles librdkafka when it is installed. If you
-have any problems installing the gem please open an issue.
+If you have any problems installing the gem, please open an issue.
 
 ## Usage
 
@@ -64,9 +64,9 @@ end
 
 ### Producing messages
 
-Produce a number of messages, put the delivery handles in an array and
+Produce a number of messages, put the delivery handles in an array, and
 wait for them before exiting. This way the messages will be batched and
-sent to Kafka in an efficient way.
+efficiently sent to Kafka.
 
 ```ruby
 config = {:"bootstrap.servers" => "localhost:9092"}
@@ -91,7 +91,7 @@ released until it `#close` is explicitly called, so be sure to call
 
 ## Higher level libraries
 
-Currently, there are two actively developed frameworks based on rdkafka-ruby, that provide higher level API that can be used to work with Kafka messages and one library for publishing messages.
+Currently, there are two actively developed frameworks based on rdkafka-ruby, that provide higher-level API that can be used to work with Kafka messages and one library for publishing messages.
 
 ### Message processing frameworks
 
@@ -104,7 +104,7 @@ Currently, there are two actively developed frameworks based on rdkafka-ruby, th
 
 ## Development
 
-A Docker Compose file is included to run Kafka and Zookeeper. To run
+A Docker Compose file is included to run Kafka. To run
 that:
 
 ```
@@ -122,7 +122,7 @@ DEBUG_PRODUCER=true bundle exec rspec
 DEBUG_CONSUMER=true bundle exec rspec
 ```
 
-After running the tests you can bring the cluster down to start with a
+After running the tests, you can bring the cluster down to start with a
 clean slate:
 
 ```
@@ -131,7 +131,7 @@ docker-compose down
 
 ## Example
 
-To see everything working run these in separate tabs:
+To see everything working, run these in separate tabs:
 
 ```
 bundle exec rake consume_messages

data/docker-compose.yml

@@ -3,7 +3,7 @@ version: '2'
 services:
   kafka:
     container_name: kafka
-    image: confluentinc/cp-kafka:7.5.1
+    image: confluentinc/cp-kafka:7.5.2
 
     ports:
       - 9092:9092

data/ext/README.md

@@ -5,7 +5,7 @@ this gem is installed.
 
 To update the `librdkafka` version follow the following steps:
 
-* Go to https://github.com/edenhill/librdkafka/releases to get the new
+* Go to https://github.com/confluentinc/librdkafka/releases to get the new
   version number and asset checksum for `tar.gz`.
 * Change the version in `lib/rdkafka/version.rb`
 * Change the `sha256` in `lib/rdkafka/version.rb`

data/ext/Rakefile

@@ -17,7 +17,7 @@ task :default => :clean do
   end
 
   recipe.files << {
-    :url => "https://codeload.github.com/edenhill/librdkafka/tar.gz/v#{Rdkafka::LIBRDKAFKA_VERSION}",
+    :url => "https://codeload.github.com/confluentinc/librdkafka/tar.gz/v#{Rdkafka::LIBRDKAFKA_VERSION}",
     :sha256 => Rdkafka::LIBRDKAFKA_SOURCE_SHA256
   }
   recipe.configure_options = ["--host=#{recipe.host}"]

data/lib/rdkafka/abstract_handle.rb

@@ -1,28 +1,37 @@
 # frozen_string_literal: true
 
-require "ffi"
-
 module Rdkafka
+  # This class serves as an abstract base class to represent handles within the Rdkafka module.
+  # As a subclass of `FFI::Struct`, this class provides a blueprint for other specific handle
+  # classes to inherit from, ensuring they adhere to a particular structure and behavior.
+  #
+  # Subclasses must define their own layout, and the layout must start with:
+  #
+  # layout :pending, :bool,
+  #        :response, :int
   class AbstractHandle < FFI::Struct
-    # Subclasses must define their own layout, and the layout must start with:
-    #
-    # layout :pending, :bool,
-    #        :response, :int
+    include Helpers::Time
 
+    # Registry for registering all the handles.
     REGISTRY = {}
 
-    CURRENT_TIME = -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) }.freeze
-
-    private_constant :CURRENT_TIME
+    class << self
+      # Adds handle to the register
+      #
+      # @param handle [AbstractHandle] any handle we want to register
+      def register(handle)
+        address = handle.to_ptr.address
+        REGISTRY[address] = handle
+      end
 
-    def self.register(handle)
-      address = handle.to_ptr.address
-      REGISTRY[address] = handle
+      # Removes handle from the register based on the handle address
+      #
+      # @param address [Integer] address of the registered handle we want to remove
+      def remove(address)
+        REGISTRY.delete(address)
+      end
     end
 
-    def self.remove(address)
-      REGISTRY.delete(address)
-    end
 
     # Whether the handle is still pending.
     #
@@ -32,27 +41,31 @@ module Rdkafka
     end
 
     # Wait for the operation to complete or raise an error if this takes longer than the timeout.
-    # If there is a timeout this does not mean the operation failed, rdkafka might still be working on the operation.
-    # In this case it is possible to call wait again.
+    # If there is a timeout this does not mean the operation failed, rdkafka might still be working
+    # on the operation. In this case it is possible to call wait again.
     #
-    # @param max_wait_timeout [Numeric, nil] Amount of time to wait before timing out. If this is nil it does not time out.
-    # @param wait_timeout [Numeric] Amount of time we should wait before we recheck if the operation has completed
+    # @param max_wait_timeout [Numeric, nil] Amount of time to wait before timing out.
+    #   If this is nil it does not time out.
+    # @param wait_timeout [Numeric] Amount of time we should wait before we recheck if the
+    #   operation has completed
     # @param raise_response_error [Boolean] should we raise error when waiting finishes
     #
+    # @return [Object] Operation-specific result
+    #
     # @raise [RdkafkaError] When the operation failed
     # @raise [WaitTimeoutError] When the timeout has been reached and the handle is still pending
-    #
-    # @return [Object] Operation-specific result
     def wait(max_wait_timeout: 60, wait_timeout: 0.1, raise_response_error: true)
       timeout = if max_wait_timeout
-                  CURRENT_TIME.call + max_wait_timeout
+                  monotonic_now + max_wait_timeout
                 else
                   nil
                 end
       loop do
         if pending?
-          if timeout && timeout <= CURRENT_TIME.call
-            raise WaitTimeoutError.new("Waiting for #{operation_name} timed out after #{max_wait_timeout} seconds")
+          if timeout && timeout <= monotonic_now
+            raise WaitTimeoutError.new(
+              "Waiting for #{operation_name} timed out after #{max_wait_timeout} seconds"
+            )
           end
           sleep wait_timeout
         elsif self[:response] != 0 && raise_response_error

data/lib/rdkafka/admin.rb

@@ -1,7 +1,5 @@
 # frozen_string_literal: true
 
-require "objspace"
-
 module Rdkafka
   class Admin
     # @private
@@ -30,11 +28,12 @@ module Rdkafka
 
     # Create a topic with the given partition count and replication factor
     #
+    # @return [CreateTopicHandle] Create topic handle that can be used to wait for the result of
+    #   creating the topic
+    #
     # @raise [ConfigError] When the partition count or replication factor are out of valid range
     # @raise [RdkafkaError] When the topic name is invalid or the topic already exists
     # @raise [RdkafkaError] When the topic configuration is invalid
-    #
-    # @return [CreateTopicHandle] Create topic handle that can be used to wait for the result of creating the topic
     def create_topic(topic_name, partition_count, replication_factor, topic_config={})
       closed_admin_check(__method__)
 
@@ -107,11 +106,11 @@ module Rdkafka
       create_topic_handle
     end
 
-    # Delete the named topic
+    # Deletes the named topic
     #
+    # @return [DeleteTopicHandle] Delete topic handle that can be used to wait for the result of
+    #   deleting the topic
     # @raise [RdkafkaError] When the topic name is invalid or the topic does not exist
-    #
-    # @return [DeleteTopicHandle] Delete topic handle that can be used to wait for the result of deleting the topic
     def delete_topic(topic_name)
       closed_admin_check(__method__)
 

data/lib/rdkafka/bindings.rb

@@ -1,9 +1,5 @@
 # frozen_string_literal: true
 
-require "ffi"
-require "json"
-require "logger"
-
 module Rdkafka
   # @private
   #

data/lib/rdkafka/config.rb

@@ -1,11 +1,9 @@
 # frozen_string_literal: true
 
-require "logger"
-
 module Rdkafka
   # Configuration for a Kafka consumer or producer. You can create an instance and use
   # the consumer and producer methods to create a client. Documentation of the available
-  # configuration options is available on https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md.
+  # configuration options is available on https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md.
   class Config
     # @private
     @@logger = Logger.new(STDOUT)
@@ -53,13 +51,13 @@ module Rdkafka
 
     # Set a callback that will be called every time the underlying client emits statistics.
     # You can configure if and how often this happens using `statistics.interval.ms`.
-    # The callback is called with a hash that's documented here: https://github.com/edenhill/librdkafka/blob/master/STATISTICS.md
+    # The callback is called with a hash that's documented here: https://github.com/confluentinc/librdkafka/blob/master/STATISTICS.md
     #
     # @param callback [Proc, #call] The callback
     #
     # @return [nil]
     def self.statistics_callback=(callback)
-      raise TypeError.new("Callback has to be callable") unless callback.respond_to?(:call)
+      raise TypeError.new("Callback has to be callable") unless callback.respond_to?(:call) || callback == nil
       @@statistics_callback = callback
     end
 
@@ -114,6 +112,7 @@ module Rdkafka
     def initialize(config_hash = {})
       @config_hash = DEFAULT_CONFIG.merge(config_hash)
       @consumer_rebalance_listener = nil
+      @consumer_poll_set = true
     end
 
     # Set a config option.
@@ -142,12 +141,28 @@ module Rdkafka
       @consumer_rebalance_listener = listener
     end
 
-    # Create a consumer with this configuration.
+    # Should we use a single queue for the underlying consumer and events.
     #
-    # @raise [ConfigError] When the configuration contains invalid options
-    # @raise [ClientCreationError] When the native client cannot be created
+    # This is an advanced API that allows for more granular control of the polling process.
+    # When this value is set to `false` (`true` by defualt), there will be two queues that need to
+    # be polled:
+    #   - main librdkafka queue for events
+    #   - consumer queue with messages and rebalances
+    #
+    # It is recommended to use the defaults and only set it to `false` in advance multi-threaded
+    # and complex cases where granular events handling control is needed.
+    #
+    # @param poll_set [Boolean]
+    def consumer_poll_set=(poll_set)
+      @consumer_poll_set = poll_set
+    end
+
+    # Creates a consumer with this configuration.
     #
     # @return [Consumer] The created consumer
+    #
+    # @raise [ConfigError] When the configuration contains invalid options
+    # @raise [ClientCreationError] When the native client cannot be created
     def consumer
       opaque = Opaque.new
       config = native_config(opaque)
@@ -160,8 +175,8 @@ module Rdkafka
       # Create native client
       kafka = native_kafka(config, :rd_kafka_consumer)
 
-      # Redirect the main queue to the consumer
-      Rdkafka::Bindings.rd_kafka_poll_set_consumer(kafka)
+      # Redirect the main queue to the consumer queue
+      Rdkafka::Bindings.rd_kafka_poll_set_consumer(kafka) if @consumer_poll_set
 
       # Return consumer with Kafka client
       Rdkafka::Consumer.new(
@@ -175,10 +190,10 @@ module Rdkafka
 
     # Create a producer with this configuration.
     #
+    # @return [Producer] The created producer
+    #
     # @raise [ConfigError] When the configuration contains invalid options
     # @raise [ClientCreationError] When the native client cannot be created
-    #
-    # @return [Producer] The created producer
     def producer
       # Create opaque
       opaque = Opaque.new
@@ -200,12 +215,12 @@ module Rdkafka
       end
     end
 
-    # Create an admin instance with this configuration.
+    # Creates an admin instance with this configuration.
+    #
+    # @return [Admin] The created admin instance
     #
     # @raise [ConfigError] When the configuration contains invalid options
     # @raise [ClientCreationError] When the native client cannot be created
-    #
-    # @return [Admin] The created admin instance
     def admin
       opaque = Opaque.new
       config = native_config(opaque)

data/lib/rdkafka/consumer/headers.rb

@@ -18,13 +18,11 @@ module Rdkafka
 
       # Reads a librdkafka native message's headers and returns them as a Ruby Hash
       #
-      # @param [librdkakfa message] native_message
+      # @private
       #
+      # @param [librdkakfa message] native_message
       # @return [Hash<String, String>] headers Hash for the native_message
-      #
       # @raise [Rdkafka::RdkafkaError] when fail to read headers
-      #
-      # @private
       def self.from_native(native_message)
         headers_ptrptr = FFI::MemoryPointer.new(:pointer)
         err = Rdkafka::Bindings.rd_kafka_message_headers(native_message, headers_ptrptr)