ruby-kafka 0.5.0 → 0.5.1.beta1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 85fb6cfbef6aefd7ddfd6c414075e73ad9fd4ef1
-  data.tar.gz: e23aff1782ebc6871e8fab5d24ee88475d557436
+  metadata.gz: 872944b4c0fb6b670fac704b827cb2ffa713e8ae
+  data.tar.gz: 17507f698aa6eb93007164d72cb24d2f79d9f2d8
 SHA512:
-  metadata.gz: 1a9c938f3554e6156f8d1b2315f26fad6b10c09fb70ed5b4228a48a5357833df43c00756873c569439fbbca112121ab857048f26c068ac53eac24a0686d2a576
-  data.tar.gz: 75211978d2f94eeb4b03c39aad903604d6ab1fdda0b30e4f582d02adb3b7293886e6e381259ad15a392ae6616e4e39aa74591168f9f161bb5b3b7efbbf5aab1d
+  metadata.gz: a5ac176d6dfa6db9ac431cf6a3c7bb1eda4183ef2a0f19864d0904ebb3024861b92ef98083fdf06512ab85c5545c2b93db10e83d8539858817953e9fc2689a4b
+  data.tar.gz: d72d3db50e98dd12840494c0691943d2f332dca639caf0cda65e750ce3109bd03ad194f720129a887100242f19ef52d2a710dd290fa02ec43eafa3f8fa025c2e

data/.circleci/config.yml

@@ -0,0 +1,33 @@
+version: 2
+jobs:
+  build:
+    docker:
+      - image: circleci/ruby:2.4.1-node
+        environment:
+          LOG_LEVEL: DEBUG
+      - image: wurstmeister/zookeeper
+      - image: wurstmeister/kafka:0.10.2.1
+        environment:
+          KAFKA_ADVERTISED_HOST_NAME: localhost
+          KAFKA_ADVERTISED_PORT: 9092
+          KAFKA_PORT: 9092
+          KAFKA_ZOOKEEPER_CONNECT: localhost:2181
+      - image: wurstmeister/kafka:0.10.2.1
+        environment:
+          KAFKA_ADVERTISED_HOST_NAME: localhost
+          KAFKA_ADVERTISED_PORT: 9093
+          KAFKA_PORT: 9093
+          KAFKA_ZOOKEEPER_CONNECT: localhost:2181
+      - image: wurstmeister/kafka:0.10.2.1
+        environment:
+          KAFKA_ADVERTISED_HOST_NAME: localhost
+          KAFKA_ADVERTISED_PORT: 9094
+          KAFKA_PORT: 9094
+          KAFKA_ZOOKEEPER_CONNECT: localhost:2181
+
+    steps:
+      - checkout
+      - run: bundle install --path vendor/bundle
+      - run: bundle exec rspec
+      - run: bundle exec rspec --profile --tag functional spec/functional
+      - run: bundle exec rubocop

data/CHANGELOG.md

@@ -4,6 +4,14 @@ Changes and additions to the library will be listed here.
 
 ## Unreleased
 
+Requires Kafka 0.10.1+ due to usage of a few new APIs.
+
+- Fix bug when using compression (#458).
+- Update the v3 of the Fetch API, allowing a per-request `max_bytes` setting (#468).
+- Make `#deliver_message` more resilient using retries and backoff.
+- Add support for SASL SCRAM authentication (#465).
+- Refactor and simplify SASL code.
+
 ## v0.5.0
 
 - Drops support for Kafka 0.9 in favor of Kafka 0.10 (#381)!

data/README.md

@@ -46,6 +46,8 @@ Although parts of this library work with Kafka 0.8 – specifically, the Produce
 6. [Support and Discussion](#support-and-discussion)
 7. [Roadmap](#roadmap)
 8. [Higher level libraries](#higher-level-libraries)
+    1. [Message processing frameworks](#message-processing-framework)
+    2. [Message publishing libraries](#message-publishing-libraries)
 
 ## Installation
 
@@ -91,7 +93,7 @@ Or install it yourself as:
 
 This library is targeting Kafka 0.9 with the v0.4.x series and Kafka 0.10 with the v0.5.x series. There's limited support for Kafka 0.8, and things should work with Kafka 0.11, although there may be performance issues due to changes in the protocol.
 
-- **Kafka 0.8:** Full support for the Producer API, but no support for consumer groups. Simple message fetching works.
+- **Kafka 0.8:** Full support for the Producer API in ruby-kafka v0.4.x, but no support for consumer groups. Simple message fetching works.
 - **Kafka 0.9:** Full support for the Producer and Consumer API in ruby-kafka v0.4.x.
 - **Kafka 0.10:** Full support for the Producer and Consumer API in ruby-kafka v0.5.x.
 - **Kafka 0.11:** Everything that works with Kafka 0.10 should still work, but so far no features specific to Kafka 0.11 have been added.
@@ -899,8 +901,9 @@ Typically, Kafka certificates come in the JKS format, which isn't supported by r
 
 #### Authentication using SASL
 
-Kafka has support for using SASL to authenticate clients. Currently GSSAPI and PLAIN mechanisms are supported by ruby-kafka.
+Kafka has support for using SASL to authenticate clients. Currently GSSAPI, SCRAM and PLAIN mechanisms are supported by ruby-kafka.
 
+##### GSSAPI
 In order to authenticate using GSSAPI, set your principal and optionally your keytab when initializing the Kafka client:
 
 ```ruby
@@ -911,6 +914,7 @@ kafka = Kafka.new(
 )
 ```
 
+##### PLAIN
 In order to authenticate using PLAIN, you must set your username and password when initializing the Kafka client:
 
 ```ruby
@@ -924,6 +928,18 @@ kafka = Kafka.new(
 
 **NOTE**: It is __highly__ recommended that you use SSL for encryption when using SASL_PLAIN
 
+##### SCRAM
+Since 0.11 kafka supports [SCRAM](https://kafka.apache.org/documentation.html#security_sasl_scram). 
+
+```ruby
+kafka = Kafka.new(
+  sasl_scram_username: 'username',
+  sasl_scram_password: 'password',
+  sasl_scram_mechanism: 'sha256',
+  # ...
+)
+```
+
 ## Design
 
 The library has been designed as a layered system, with each layer having a clear responsibility:
@@ -978,24 +994,30 @@ Version 0.4 will be the last minor release with support for the Kafka 0.9 protoc
 
 ### v0.4
 
-Current stable release with support for the Kafka 0.9 protocol.
+Last stable release with support for the Kafka 0.9 protocol. Bug and security fixes will be released in patch updates.
 
 ### v0.5
 
-Next stable release, with support for the Kafka 0.10 protocol and eventually newer protocol versions.
+Latest stable release, with native support for the Kafka 0.10 protocol and eventually newer protocol versions. Kafka 0.9 is no longer supported by this release series.
 
 ## Higher level libraries
 
-Currently, there are three actively developed frameworks based on ruby-kafka, that provide higher level API that can be used to work with Kafka messages:
+Currently, there are three actively developed frameworks based on ruby-kafka, that provide higher level API that can be used to work with Kafka messages and two libraries for publishing messages.
 
-* [Racecar](https://github.com/zendesk/racecar) - A simple framework that integrates with Ruby on Rails to provide a seamless way to write, test, configure, and run Kafka consumers. It comes with sensible defaults and conventions.
+### Message processing frameworks
 
-* [DeliveryBoy](https://github.com/zendesk/delivery_boy) – A library that integrates with Ruby on Rails, making it easy to publish Kafka messages from any Rails application.
+* [Racecar](https://github.com/zendesk/racecar) - A simple framework that integrates with Ruby on Rails to provide a seamless way to write, test, configure, and run Kafka consumers. It comes with sensible defaults and conventions.
 
 * [Karafka](https://github.com/karafka/karafka) - Framework used to simplify Apache Kafka based Ruby and Rails applications development. Karafka provides higher abstraction layers, including Capistrano, Docker and Heroku support.
 
 * [Phobos](https://github.com/klarna/phobos) - Micro framework and library for applications dealing with Apache Kafka. It wraps common behaviors needed by consumers and producers in an easy and convenient API.
 
+### Message publishing libraries
+
+* [DeliveryBoy](https://github.com/zendesk/delivery_boy) – A library that integrates with Ruby on Rails, making it easy to publish Kafka messages from any Rails application.
+
+* [WaterDrop](https://github.com/karafka/waterdrop) – A library for Ruby and Ruby on Rails applications, to easy publish Kafka messages in both sync and async way.
+
 ## Why Create A New Library?
 
 There are a few existing Kafka clients in Ruby:

data/docker-compose.yml

@@ -0,0 +1,39 @@
+version: '2'
+services:
+  zookeeper:
+    image: wurstmeister/zookeeper
+    ports:
+      - "2181:2181"
+  kafka1:
+    image: wurstmeister/kafka:0.10.2.1
+    ports:
+      - "9092:9092"
+    environment:
+      KAFKA_BROKER_ID: 1
+      KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
+      KAFKA_ADVERTISED_PORT: 9092
+      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+  kafka2:
+    image: wurstmeister/kafka:0.10.2.1
+    ports:
+      - "9093:9092"
+    environment:
+      KAFKA_BROKER_ID: 2
+      KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
+      KAFKA_ADVERTISED_PORT: 9093
+      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+  kafka3:
+    image: wurstmeister/kafka:0.10.2.1
+    ports:
+      - "9094:9092"
+    environment:
+      KAFKA_BROKER_ID: 3
+      KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
+      KAFKA_ADVERTISED_PORT: 9094
+      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock

data/lib/kafka.rb

@@ -21,6 +21,14 @@ module Kafka
   class NoPartitionsToFetchFrom < Error
   end
 
+  # A message in a partition is larger than the maximum we've asked for.
+  class MessageTooLargeToRead < Error
+  end
+
+  # A connection has been unused for too long, we assume the server has killed it.
+  class IdleConnection < Error
+  end
+
   # Subclasses of this exception class map to an error code described in the
   # Kafka protocol specification.
   #
@@ -225,6 +233,12 @@ module Kafka
   class FetchError < Error
   end
 
+  class SaslScramError < Error
+  end
+
+  class FailedScramAuthentication < SaslScramError
+  end
+
   # Initializes a new Kafka client.
   #
   # @see Client#initialize

data/lib/kafka/broker.rb

@@ -4,24 +4,27 @@ require "kafka/protocol"
 
 module Kafka
   class Broker
-    def initialize(connection:, node_id: nil, logger:)
-      @connection = connection
+    def initialize(connection_builder:, host:, port:, node_id: nil, logger:)
+      @connection_builder = connection_builder
+      @connection = nil
+      @host = host
+      @port = port
       @node_id = node_id
       @logger = logger
     end
 
     def address_match?(host, port)
-      @connection.address_match?(host, port)
+      host == @host && port == @port
     end
 
     # @return [String]
     def to_s
-      "#{@connection} (node_id=#{@node_id.inspect})"
+      "#{connection} (node_id=#{@node_id.inspect})"
     end
 
     # @return [nil]
     def disconnect
-      @connection.close
+      connection.close
     end
 
     # Fetches cluster metadata from the broker.
@@ -31,7 +34,7 @@ module Kafka
     def fetch_metadata(**options)
       request = Protocol::TopicMetadataRequest.new(**options)
 
-      @connection.send_request(request)
+      send_request(request)
     end
 
     # Fetches messages from a specified topic and partition.
@@ -41,7 +44,7 @@ module Kafka
     def fetch_messages(**options)
       request = Protocol::FetchRequest.new(**options)
 
-      @connection.send_request(request)
+      send_request(request)
     end
 
     # Lists the offset of the specified topics and partitions.
@@ -51,7 +54,7 @@ module Kafka
     def list_offsets(**options)
       request = Protocol::ListOffsetRequest.new(**options)
 
-      @connection.send_request(request)
+      send_request(request)
     end
 
     # Produces a set of messages to the broker.
@@ -61,55 +64,81 @@ module Kafka
     def produce(**options)
       request = Protocol::ProduceRequest.new(**options)
 
-      @connection.send_request(request)
+      send_request(request)
     end
 
     def fetch_offsets(**options)
       request = Protocol::OffsetFetchRequest.new(**options)
 
-      @connection.send_request(request)
+      send_request(request)
     end
 
     def commit_offsets(**options)
       request = Protocol::OffsetCommitRequest.new(**options)
 
-      @connection.send_request(request)
+      send_request(request)
     end
 
     def join_group(**options)
       request = Protocol::JoinGroupRequest.new(**options)
 
-      @connection.send_request(request)
+      send_request(request)
     end
 
     def sync_group(**options)
       request = Protocol::SyncGroupRequest.new(**options)
 
-      @connection.send_request(request)
+      send_request(request)
     end
 
     def leave_group(**options)
       request = Protocol::LeaveGroupRequest.new(**options)
 
-      @connection.send_request(request)
+      send_request(request)
     end
 
     def find_group_coordinator(**options)
       request = Protocol::GroupCoordinatorRequest.new(**options)
 
-      @connection.send_request(request)
+      send_request(request)
     end
 
     def heartbeat(**options)
       request = Protocol::HeartbeatRequest.new(**options)
 
-      @connection.send_request(request)
+      send_request(request)
     end
 
-    def sasl_handshake(**options)
-      request = Protocol::SaslHandshakeRequest(**options)
+    def create_topics(**options)
+      request = Protocol::CreateTopicsRequest.new(**options)
 
-      @connection.send_request(request)
+      send_request(request)
+    end
+
+    def api_versions
+      request = Protocol::ApiVersionsRequest.new
+
+      send_request(request)
+    end
+
+    private
+
+    def send_request(request)
+      connection.send_request(request)
+    rescue IdleConnection
+      @logger.warn "Connection has been unused for too long, re-connecting..."
+      @connection.close rescue nil
+      @connection = nil
+      retry
+    rescue ConnectionError
+      @connection.close rescue nil
+      @connection = nil
+
+      raise
+    end
+
+    def connection
+      @connection ||= @connection_builder.build_connection(@host, @port)
     end
   end
 end

data/lib/kafka/broker_pool.rb

@@ -17,7 +17,9 @@ module Kafka
       end
 
       broker = Broker.new(
-        connection: @connection_builder.build_connection(host, port),
+        connection_builder: @connection_builder,
+        host: host,
+        port: port,
         node_id: node_id,
         logger: @logger,
       )

data/lib/kafka/client.rb

@@ -49,11 +49,18 @@ module Kafka
     #
     # @param sasl_gssapi_keytab [String, nil] a KRB5 keytab filepath
     #
+    # @param sasl_scram_username [String, nil] SCRAM username
+    #
+    # @param sasl_scram_password [String, nil] SCRAM password
+    #
+    # @param sasl_scram_mechanism [String, nil] Scram mechanism, either "sha256" or "sha512"
+    #
     # @return [Client]
     def initialize(seed_brokers:, client_id: "ruby-kafka", logger: nil, connect_timeout: nil, socket_timeout: nil,
                    ssl_ca_cert_file_path: nil, ssl_ca_cert: nil, ssl_client_cert: nil, ssl_client_cert_key: nil,
                    sasl_gssapi_principal: nil, sasl_gssapi_keytab: nil,
-                   sasl_plain_authzid: '', sasl_plain_username: nil, sasl_plain_password: nil)
+                   sasl_plain_authzid: '', sasl_plain_username: nil, sasl_plain_password: nil,
+                   sasl_scram_username: nil, sasl_scram_password: nil, sasl_scram_mechanism: nil)
       @logger = logger || Logger.new(nil)
       @instrumenter = Instrumenter.new(client_id: client_id)
       @seed_brokers = normalize_seed_brokers(seed_brokers)
@@ -66,6 +73,9 @@ module Kafka
         sasl_plain_authzid: sasl_plain_authzid,
         sasl_plain_username: sasl_plain_username,
         sasl_plain_password: sasl_plain_password,
+        sasl_scram_username: sasl_scram_username,
+        sasl_scram_password: sasl_scram_password,
+        sasl_scram_mechanism: sasl_scram_mechanism,
         logger: @logger
       )
 
@@ -96,8 +106,10 @@ module Kafka
     #   chosen at random.
     # @param partition_key [String] a value used to deterministically choose a
     #   partition to write to.
+    # @param retries [Integer] the number of times to retry the delivery before giving
+    #   up.
     # @return [nil]
-    def deliver_message(value, key: nil, topic:, partition: nil, partition_key: nil)
+    def deliver_message(value, key: nil, topic:, partition: nil, partition_key: nil, retries: 1)
       create_time = Time.now
 
       message = PendingMessage.new(
@@ -140,10 +152,27 @@ module Kafka
         instrumenter: @instrumenter,
       )
 
-      operation.execute
+      attempt = 1
+
+      begin
+        operation.execute
+
+        unless buffer.empty?
+          raise DeliveryFailed.new(nil, [message])
+        end
+      rescue Kafka::Error => e
+        @cluster.mark_as_stale!
 
-      unless buffer.empty?
-        raise DeliveryFailed.new(nil, [message])
+        if attempt >= (retries + 1)
+          raise
+        else
+          attempt += 1
+          @logger.warn "Error while delivering message, #{e.class}: #{e.message}; retrying after 1s..."
+
+          sleep 1
+
+          retry
+        end
       end
     end
 
@@ -345,17 +374,32 @@ module Kafka
     #   expect messages to be larger than this.
     #
     # @return [Array<Kafka::FetchedMessage>] the messages returned from the broker.
-    def fetch_messages(topic:, partition:, offset: :latest, max_wait_time: 5, min_bytes: 1, max_bytes: 1048576)
+    def fetch_messages(topic:, partition:, offset: :latest, max_wait_time: 5, min_bytes: 1, max_bytes: 1048576, retries: 1)
       operation = FetchOperation.new(
         cluster: @cluster,
         logger: @logger,
         min_bytes: min_bytes,
+        max_bytes: max_bytes,
         max_wait_time: max_wait_time,
       )
 
       operation.fetch_from_partition(topic, partition, offset: offset, max_bytes: max_bytes)
 
-      operation.execute.flat_map {|batch| batch.messages }
+      attempt = 1
+
+      begin
+        operation.execute.flat_map {|batch| batch.messages }
+      rescue Kafka::Error => e
+        @cluster.mark_as_stale!
+
+        if attempt >= (retries + 1)
+          raise
+        else
+          attempt += 1
+          @logger.warn "Error while fetching messages, #{e.class}: #{e.message}; retrying..."
+          retry
+        end
+      end
     end
 
     # Enumerate all messages in a topic.
@@ -407,6 +451,10 @@ module Kafka
       end
     end
 
+    def create_topic(name, **options)
+      @cluster.create_topic(name, **options)
+    end
+
     # Lists all topics in the cluster.
     #
     # @return [Array<String>] the list of topic names.
@@ -415,6 +463,12 @@ module Kafka
       @cluster.topics
     end
 
+    def has_topic?(topic)
+      @cluster.clear_target_topics
+      @cluster.add_target_topics([topic])
+      @cluster.topics.include?(topic)
+    end
+
     # Counts the number of partitions in a topic.
     #
     # @param topic [String]
@@ -455,6 +509,10 @@ module Kafka
       }.to_h
     end
 
+    def apis
+      @cluster.apis
+    end
+
     # Closes all connections to the Kafka brokers and frees up used resources.
     #
     # @return [nil]