rdkafka 0.14.1 → 0.15.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4155932560c141d7110501ec00ac51d49311130335538deb684c670bcc0e4905
-  data.tar.gz: 9ab67e523adcbf68fd03b755a2384d790f46591c140fd1b545bf974be5cb5c3d
+  metadata.gz: d906b2e71dae5b5f45459e915c48dc8cb88e0d51ebb90ded80cef3c8e5531b77
+  data.tar.gz: 8f0df2688bbc3b264de22b5943b18462ad41898781cc12e6e534804409133ce0
 SHA512:
-  metadata.gz: 26a68921d4fc49dbfbcbc36b23d19c62f3de04f0ba5c8da488d9cc1bc1de2800ade788dcaa8bff381e83e7f9f5fbb3726aeb799f8a0ad703cec9fde3b0ed7e64
-  data.tar.gz: 266634c85162b6a0f46137e4f2080ac8eae470a46a10b13ee555c5184932ccb218e0dbbd91a250df2e0e7517f70723832002a4266bade55a75040c39fb3a971f
+  metadata.gz: c35d392b326f4d47077f419bced92b929436be548651afc9364f5ada2eda51883ad75feeeb30183369aa6b15db3ac4630410f408eae449d1b0cc5a007cf011fc
+  data.tar.gz: 1487bb54713e6330ce55fd95f656dffd2edc34bcd8bc151d94faf2d6f043b8183276e3407d5f45c6152e99463c8323ccb83c15fff7007ebb96c4b369533002d5

data/.github/FUNDING.yml

@@ -0,0 +1 @@
+custom: ['https://karafka.io/#become-pro']

data/CHANGELOG.md

@@ -1,7 +1,15 @@
 # Rdkafka Changelog
 
-## 0.14.1 (2024-07-10)
-- [Fix] Switch to local release of librdkafka to mitigate its unavailability.
+## 0.15.0 (2023-12-03)
+- **[Feature]** Add `Admin#metadata` (mensfeld)
+- **[Feature]** Add `Admin#create_partitions` (mensfeld)
+- **[Feature]** Add `Admin#delete_group` utility (piotaixr)
+- **[Feature]** Add Create and Delete ACL Feature To Admin Functions (vgnanasekaran)
+- **[Feature]** Support `#assignment_lost?` on a consumer to check for involuntary assignment revocation (mensfeld)
+- [Enhancement] Expose alternative way of managing consumer events via a separate queue (mensfeld) 
+- [Enhancement] **Bump** librdkafka to 2.3.0 (mensfeld)
+- [Enhancement] Increase the `#lag` and `#query_watermark_offsets` default timeouts from 100ms to 1000ms. This will compensate for network glitches and remote clusters operations (mensfeld)
+- [Change] Use `SecureRandom.uuid` instead of `random` for test consumer groups (mensfeld)
 
 ## 0.14.0 (2023-11-21)
 - [Enhancement] Add `raise_response_error` flag to the `Rdkafka::AbstractHandle`.

data/README.md

@@ -18,22 +18,31 @@ become EOL.
 
 `rdkafka` was written because of the need for a reliable Ruby client for Kafka that supports modern Kafka at [AppSignal](https://appsignal.com). AppSignal runs it in production on very high-traffic systems.
 
-The most important pieces of a Kafka client are implemented. We're
-working towards feature completeness. You can track that here:
-https://github.com/appsignal/rdkafka-ruby/milestone/1
+The most important pieces of a Kafka client are implemented, and we aim to provide all relevant consumer, producer, and admin APIs.
 
 ## Table of content
 
+- [Project Scope](#project-scope)
 - [Installation](#installation)
 - [Usage](#usage)
-  * [Consuming messages](#consuming-messages)
-  * [Producing messages](#producing-messages)
-- [Higher level libraries](#higher-level-libraries)
-  * [Message processing frameworks](#message-processing-frameworks)
-  * [Message publishing libraries](#message-publishing-libraries)
+  * [Consuming Messages](#consuming-messages)
+  * [Producing Messages](#producing-messages)
+- [Higher Level Libraries](#higher-level-libraries)
+  * [Message Processing Frameworks](#message-processing-frameworks)
+  * [Message Publishing Libraries](#message-publishing-libraries)
 - [Development](#development)
 - [Example](#example)
 
+## Project Scope
+
+While rdkafka-ruby aims to simplify the use of librdkafka in Ruby applications, it's important to understand the limitations of this library:
+
+- **No Complex Producers/Consumers**: This library does not intend to offer complex producers or consumers. The aim is to stick closely to the functionalities provided by librdkafka itself.
+
+- **Focus on librdkafka Capabilities**: Features that can be achieved directly in Ruby, without specific needs from librdkafka, are outside the scope of this library.
+
+- **Existing High-Level Functionalities**: Certain high-level functionalities like producer metadata cache and simple consumer are already part of the library. Although they fall slightly outside the primary goal, they will remain part of the contract, given their existing usage.
+
 
 ## Installation
 
@@ -42,9 +51,9 @@ If you have any problems installing the gem, please open an issue.
 
 ## Usage
 
-See the [documentation](https://www.rubydoc.info/github/appsignal/rdkafka-ruby) for full details on how to use this gem. Two quick examples:
+See the [documentation](https://karafka.io/docs/code/rdkafka-ruby/) for full details on how to use this gem. Two quick examples:
 
-### Consuming messages
+### Consuming Messages
 
 Subscribe to a topic and get messages. Kafka will automatically spread
 the available partitions over consumers with the same group id.
@@ -62,7 +71,7 @@ consumer.each do |message|
 end
 ```
 
-### Producing messages
+### Producing Messages
 
 Produce a number of messages, put the delivery handles in an array, and
 wait for them before exiting. This way the messages will be batched and
@@ -89,41 +98,42 @@ Note that creating a producer consumes some resources that will not be
 released until it `#close` is explicitly called, so be sure to call
 `Config#producer` only as necessary.
 
-## Higher level libraries
+## Higher Level Libraries
 
 Currently, there are two actively developed frameworks based on rdkafka-ruby, that provide higher-level API that can be used to work with Kafka messages and one library for publishing messages.
 
-### Message processing frameworks
+### Message Processing Frameworks
 
 * [Karafka](https://github.com/karafka/karafka) - Ruby and Rails efficient Kafka processing framework.
 * [Racecar](https://github.com/zendesk/racecar) - A simple framework for Kafka consumers in Ruby 
 
-### Message publishing libraries
+### Message Publishing Libraries
 
 * [WaterDrop](https://github.com/karafka/waterdrop) – Standalone Karafka library for producing Kafka messages.
 
 ## Development
 
-A Docker Compose file is included to run Kafka. To run
-that:
+Contributors are encouraged to focus on enhancements that align with the core goal of the library. We appreciate contributions but will likely not accept pull requests for features that:
+
+- Implement functionalities that can achieved using standard Ruby capabilities without changes to the underlying rdkafka-ruby bindings.
+- Deviate significantly from the primary aim of providing librdkafka bindings with Ruby-friendly interfaces.
+
+A Docker Compose file is included to run Kafka. To run that:
 
 ```
 docker-compose up
 ```
 
-Run `bundle` and `cd ext && bundle exec rake && cd ..` to download and
-compile `librdkafka`.
+Run `bundle` and `cd ext && bundle exec rake && cd ..` to download and compile `librdkafka`.
 
-You can then run `bundle exec rspec` to run the tests. To see rdkafka
-debug output:
+You can then run `bundle exec rspec` to run the tests. To see rdkafka debug output:
 
 ```
 DEBUG_PRODUCER=true bundle exec rspec
 DEBUG_CONSUMER=true bundle exec rspec
 ```
 
-After running the tests, you can bring the cluster down to start with a
-clean slate:
+After running the tests, you can bring the cluster down to start with a clean slate:
 
 ```
 docker-compose down

data/docker-compose.yml

@@ -23,3 +23,5 @@ services:
       KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
       KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
       KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
+      KAFKA_AUTHORIZER_CLASS_NAME: org.apache.kafka.metadata.authorizer.StandardAuthorizer

data/ext/Rakefile

@@ -1,67 +1,40 @@
 # frozen_string_literal: true
 
 require File.expand_path('../../lib/rdkafka/version', __FILE__)
+require "mini_portile2"
 require "fileutils"
 require "open-uri"
 
 task :default => :clean do
-  # For nix users, nix can't locate the file paths because the packages it's requiring aren't managed by the system but are
-  # managed by nix itself, so using the normal file paths doesn't work for nix users.
-  #
-  # Mini_portile causes an issue because it's dependencies are downloaded on the fly and therefore don't exist/aren't
-  # accessible in the nix environment
-  if ENV.fetch('RDKAFKA_EXT_PATH', '').empty?
-    # Download and compile librdkafka if RDKAFKA_EXT_PATH is not set
-    require "mini_portile2"
-    recipe = MiniPortile.new("librdkafka", Rdkafka::LIBRDKAFKA_VERSION)
+  # Download and compile librdkafka
+  recipe = MiniPortile.new("librdkafka", Rdkafka::LIBRDKAFKA_VERSION)
 
-    # Use default homebrew openssl if we're on mac and the directory exists
-    # and each of flags is not empty
-    if recipe.host&.include?("darwin") && system("which brew &> /dev/null") && Dir.exist?("#{homebrew_prefix = %x(brew --prefix openssl).strip}")
-      ENV["CPPFLAGS"] = "-I#{homebrew_prefix}/include" unless ENV["CPPFLAGS"]
-      ENV["LDFLAGS"] = "-L#{homebrew_prefix}/lib" unless ENV["LDFLAGS"]
-    end
-
-    releases = File.expand_path(File.join(File.dirname(__FILE__), '../dist'))
-
-    recipe.files << {
-      :url => "file://#{releases}/librdkafka_#{Rdkafka::LIBRDKAFKA_VERSION}.tar.gz",
-      :sha256 => Rdkafka::LIBRDKAFKA_SOURCE_SHA256
-    }
-    recipe.configure_options = ["--host=#{recipe.host}"]
-
-    # Disable using libc regex engine in favor of the embedded one
-    # The default regex engine of librdkafka does not always work exactly as most of the users
-    # would expect, hence this flag allows for changing it to the other one
-    if ENV.key?('RDKAFKA_DISABLE_REGEX_EXT')
-      recipe.configure_options << '--disable-regex-ext'
-    end
+  # Use default homebrew openssl if we're on mac and the directory exists
+  # and each of flags is not empty
+  if recipe.host&.include?("darwin") && system("which brew &> /dev/null") && Dir.exist?("#{homebrew_prefix = %x(brew --prefix openssl).strip}")
+    ENV["CPPFLAGS"] = "-I#{homebrew_prefix}/include" unless ENV["CPPFLAGS"]
+    ENV["LDFLAGS"] = "-L#{homebrew_prefix}/lib" unless ENV["LDFLAGS"]
+  end
 
-    recipe.cook
-    # Move dynamic library we're interested in
-    if recipe.host.include?('darwin')
-      from_extension = '1.dylib'
-      to_extension   = 'dylib'
-    else
-      from_extension = 'so.1'
-      to_extension = 'so'
-    end
-    lib_path = File.join(File.dirname(__FILE__), "ports/#{recipe.host}/librdkafka/#{Rdkafka::LIBRDKAFKA_VERSION}/lib/librdkafka.#{from_extension}")
-    FileUtils.mv(lib_path, File.join(File.dirname(__FILE__), "librdkafka.#{to_extension}"))
-    # Cleanup files created by miniportile we don't need in the gem
-    FileUtils.rm_rf File.join(File.dirname(__FILE__), "tmp")
-    FileUtils.rm_rf File.join(File.dirname(__FILE__), "ports")
+  recipe.files << {
+    :url => "https://codeload.github.com/confluentinc/librdkafka/tar.gz/v#{Rdkafka::LIBRDKAFKA_VERSION}",
+    :sha256 => Rdkafka::LIBRDKAFKA_SOURCE_SHA256
+  }
+  recipe.configure_options = ["--host=#{recipe.host}"]
+  recipe.cook
+  # Move dynamic library we're interested in
+  if recipe.host.include?('darwin')
+    from_extension = '1.dylib'
+    to_extension   = 'dylib'
   else
-    # Otherwise, copy existing libraries to ./ext
-    if ENV['RDKAFKA_EXT_PATH'].nil? || ENV['RDKAFKA_EXT_PATH'].empty?
-      raise "RDKAFKA_EXT_PATH must be set in your nix config when running under nix"
-    end
-    files = [
-      File.join(ENV['RDKAFKA_EXT_PATH'], 'lib', 'librdkafka.dylib'),
-      File.join(ENV['RDKAFKA_EXT_PATH'], 'lib', 'librdkafka.so')
-    ]
-    files.each { |ext| FileUtils.cp(ext, File.dirname(__FILE__)) if File.exist?(ext) }
+    from_extension = 'so.1'
+    to_extension = 'so'
   end
+  lib_path = File.join(File.dirname(__FILE__), "ports/#{recipe.host}/librdkafka/#{Rdkafka::LIBRDKAFKA_VERSION}/lib/librdkafka.#{from_extension}")
+  FileUtils.mv(lib_path, File.join(File.dirname(__FILE__), "librdkafka.#{to_extension}"))
+  # Cleanup files created by miniportile we don't need in the gem
+  FileUtils.rm_rf File.join(File.dirname(__FILE__), "tmp")
+  FileUtils.rm_rf File.join(File.dirname(__FILE__), "ports")
 end
 
 task :clean do

data/lib/rdkafka/admin/acl_binding_result.rb

@@ -0,0 +1,37 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+
+    # Extracts attributes of rd_kafka_AclBinding_t
+    #
+    class AclBindingResult
+      attr_reader :result_error, :error_string, :matching_acl_resource_type, :matching_acl_resource_name, :matching_acl_pattern_type, :matching_acl_principal, :matching_acl_host, :matching_acl_operation, :matching_acl_permission_type
+
+      def initialize(matching_acl)
+          rd_kafka_error_pointer        = Rdkafka::Bindings.rd_kafka_AclBinding_error(matching_acl)
+          @result_error                 = Rdkafka::Bindings.rd_kafka_error_code(rd_kafka_error_pointer)
+          error_string                 = Rdkafka::Bindings.rd_kafka_error_string(rd_kafka_error_pointer)
+          if error_string != FFI::Pointer::NULL
+            @error_string = error_string.read_string
+          end
+          @matching_acl_resource_type   = Rdkafka::Bindings.rd_kafka_AclBinding_restype(matching_acl)
+          matching_acl_resource_name    = Rdkafka::Bindings.rd_kafka_AclBinding_name(matching_acl)
+          if matching_acl_resource_name != FFI::Pointer::NULL
+            @matching_acl_resource_name = matching_acl_resource_name.read_string
+          end
+          @matching_acl_pattern_type    = Rdkafka::Bindings.rd_kafka_AclBinding_resource_pattern_type(matching_acl)
+          matching_acl_principal        = Rdkafka::Bindings.rd_kafka_AclBinding_principal(matching_acl)
+          if matching_acl_principal != FFI::Pointer::NULL
+            @matching_acl_principal = matching_acl_principal.read_string
+          end
+          matching_acl_host             = Rdkafka::Bindings.rd_kafka_AclBinding_host(matching_acl)
+          if matching_acl_host != FFI::Pointer::NULL
+            @matching_acl_host = matching_acl_host.read_string
+          end
+          @matching_acl_operation       = Rdkafka::Bindings.rd_kafka_AclBinding_operation(matching_acl)
+          @matching_acl_permission_type = Rdkafka::Bindings.rd_kafka_AclBinding_permission_type(matching_acl)
+        end
+      end
+  end
+end

data/lib/rdkafka/admin/create_acl_handle.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    class CreateAclHandle < AbstractHandle
+      layout :pending, :bool,
+             :response, :int,
+             :response_string, :pointer
+
+      # @return [String] the name of the operation
+      def operation_name
+        "create acl"
+      end
+
+      # @return [CreateAclReport] instance with rdkafka_response value as 0 and rdkafka_response_string value as empty string if the acl creation was successful
+      def create_result
+        CreateAclReport.new(rdkafka_response: self[:response], rdkafka_response_string: self[:response_string])
+      end
+
+      def raise_error
+        raise RdkafkaError.new(
+            self[:response],
+            broker_message: self[:response_string].read_string
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/create_acl_report.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    class CreateAclReport
+
+      # Upon successful creation of Acl RD_KAFKA_RESP_ERR_NO_ERROR - 0 is returned as rdkafka_response
+      # @return [Integer]
+      attr_reader :rdkafka_response
+
+
+      # Upon successful creation of Acl empty string will be returned as rdkafka_response_string
+      # @return [String]
+      attr_reader :rdkafka_response_string
+
+      def initialize(rdkafka_response:, rdkafka_response_string:)
+        @rdkafka_response = rdkafka_response
+        if rdkafka_response_string != FFI::Pointer::NULL
+          @rdkafka_response_string = rdkafka_response_string.read_string
+        end
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/create_partitions_handle.rb

@@ -0,0 +1,27 @@
+module Rdkafka
+  class Admin
+    class CreatePartitionsHandle < AbstractHandle
+      layout :pending, :bool,
+             :response, :int,
+             :error_string, :pointer,
+             :result_name, :pointer
+
+      # @return [String] the name of the operation
+      def operation_name
+        "create partitions"
+      end
+
+      # @return [Boolean] whether the create topic was successful
+      def create_result
+        CreatePartitionsReport.new(self[:error_string], self[:result_name])
+      end
+
+      def raise_error
+        raise RdkafkaError.new(
+            self[:response],
+            broker_message: CreateTopicReport.new(self[:error_string], self[:result_name]).error_string
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/create_partitions_report.rb

@@ -0,0 +1,6 @@
+module Rdkafka
+  class Admin
+    class CreatePartitionsReport < CreateTopicReport
+    end
+  end
+end

data/lib/rdkafka/admin/delete_acl_handle.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    class DeleteAclHandle < AbstractHandle
+      layout :pending, :bool,
+             :response, :int,
+             :response_string, :pointer,
+             :matching_acls, :pointer,
+             :matching_acls_count, :int
+
+      # @return [String] the name of the operation
+      def operation_name
+        "delete acl"
+      end
+
+      # @return [DeleteAclReport] instance with an array of matching_acls
+      def create_result
+        DeleteAclReport.new(matching_acls: self[:matching_acls], matching_acls_count: self[:matching_acls_count])
+      end
+
+      def raise_error
+        raise RdkafkaError.new(
+            self[:response],
+            broker_message: self[:response_string].read_string
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/delete_acl_report.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    class DeleteAclReport
+
+      # deleted acls
+      # @return [Rdkafka::Bindings::AclBindingResult]
+      attr_reader :deleted_acls
+
+      def initialize(matching_acls:, matching_acls_count:)
+        @deleted_acls=[]
+        if matching_acls != FFI::Pointer::NULL
+          acl_binding_result_pointers  = matching_acls.read_array_of_pointer(matching_acls_count)
+          (1..matching_acls_count).map do |matching_acl_index|
+            acl_binding_result = AclBindingResult.new(acl_binding_result_pointers[matching_acl_index - 1])
+            @deleted_acls << acl_binding_result
+          end
+        end
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/delete_groups_handle.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    class DeleteGroupsHandle < AbstractHandle
+      layout :pending, :bool, # TODO: ???
+             :response, :int,
+             :error_string, :pointer,
+             :result_name, :pointer
+
+      # @return [String] the name of the operation
+      def operation_name
+        "delete groups"
+      end
+
+      def create_result
+        DeleteGroupsReport.new(self[:error_string], self[:result_name])
+      end
+
+      def raise_error
+        raise RdkafkaError.new(
+            self[:response],
+            broker_message: create_result.error_string
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/delete_groups_report.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    class DeleteGroupsReport
+      # Any error message generated from the DeleteTopic
+      # @return [String]
+      attr_reader :error_string
+
+      # The name of the topic deleted
+      # @return [String]
+      attr_reader :result_name
+
+      def initialize(error_string, result_name)
+        if error_string != FFI::Pointer::NULL
+          @error_string = error_string.read_string
+        end
+        if result_name != FFI::Pointer::NULL
+          @result_name = @result_name = result_name.read_string
+        end
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/describe_acl_handle.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    class DescribeAclHandle < AbstractHandle
+      layout :pending, :bool,
+             :response, :int,
+             :response_string, :pointer,
+             :acls, :pointer,
+             :acls_count, :int
+
+      # @return [String] the name of the operation.
+      def operation_name
+        "describe acl"
+      end
+
+      # @return [DescribeAclReport] instance with an array of acls that matches the request filters.
+      def create_result
+        DescribeAclReport.new(acls: self[:acls], acls_count: self[:acls_count])
+      end
+
+      def raise_error
+        raise RdkafkaError.new(
+            self[:response],
+            broker_message: self[:response_string].read_string
+        )
+      end
+    end
+  end
+end

data/lib/rdkafka/admin/describe_acl_report.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+module Rdkafka
+  class Admin
+    class DescribeAclReport
+
+      # acls that exists in the cluster for the resource_type, resource_name and pattern_type filters provided in the request.
+      # @return [Rdkafka::Bindings::AclBindingResult] array of matching acls.
+      attr_reader :acls
+
+      def initialize(acls:, acls_count:)
+        @acls=[]
+        if acls != FFI::Pointer::NULL
+          acl_binding_result_pointers  = acls.read_array_of_pointer(acls_count)
+          (1..acls_count).map do |acl_index|
+            acl_binding_result = AclBindingResult.new(acl_binding_result_pointers[acl_index - 1])
+            @acls << acl_binding_result
+          end
+        end
+      end
+    end
+  end
+end