ruby-kafka 0.4.0 → 0.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 8e5c2b78f0d4883922641e90ac150a738959c00e
-  data.tar.gz: 0166be6c6bef06356abdf4f771dd633043e996ba
+  metadata.gz: 9845964409d385d935b55e4ded95f210721fe7ce
+  data.tar.gz: d7bebf3d0222a80c0fe95ebec4d83b992abe306d
 SHA512:
-  metadata.gz: 1207e355018718c240fabd89ba2286949dc156f5ce9734f9cf48635154ee9d20587d4401e4ca8c5f31edb4fda3b1e049766ad0a93030b5df99b8762e160f5970
-  data.tar.gz: 3c7b974bf3347c841d9bd85118dc8bf66dc892f6c48df7fc32599949bca6b1bf71d301559ce28709eac108dd92acced6a9f15557226d12d0c3311ac4b3c1115e
+  metadata.gz: 5656f9723720414c954a7087385d52d9618322d31e70f9600bd4b8f34bb76432a31fb0bc88bc6baf5506fe518c47a7a5c9cb0134c91f224acdaa6ef76bbd8fce
+  data.tar.gz: 042d3a1bf5f8847e9b50a93e2bc8b151e78c4fac463e999e32e9d0acd7f1d5498ebb301d166a8a255adcc2efd15f7bd53de6397348a94822f2462c626c49741f

data/CHANGELOG.md

@@ -4,6 +4,9 @@ Changes and additions to the library will be listed here.
 
 ## Unreleased
 
+- Allow seeking the consumer position (#386).
+- Reopen idle connections after 5 minutes (#399).
+
 ## v0.4.0
 
 - Support SASL authentication (#334 and #370)

data/Gemfile.lock

@@ -1,8 +1,7 @@
 PATH
   remote: .
   specs:
-    ruby-kafka (0.3.18.beta2)
-      gssapi (>= 1.2.0)
+    ruby-kafka (0.4.0)
 
 GEM
   remote: https://rubygems.org/
@@ -74,6 +73,7 @@ DEPENDENCIES
   docker-api
   dogstatsd-ruby (>= 2.0.0)
   dotenv
+  gssapi (>= 1.2.0)
   pry
   rake (~> 10.0)
   rspec

data/README.md

@@ -42,7 +42,9 @@ Although parts of this library work with Kafka 0.8 – specifically, the Produce
     2. [Asynchronous Producer Design](#asynchronous-producer-design)
     3. [Consumer Design](#consumer-design)
 5. [Development](#development)
-6. [Roadmap](#roadmap)
+6. [Support and Discussion](#support-and-discussion)
+7. [Roadmap](#roadmap)
+8. [Higher level libraries](#higher-level-libraries)
 
 ## Installation
 
@@ -68,25 +70,30 @@ Or install it yourself as:
     <th>Kafka 0.8</th>
     <th>Kafka 0.9</th>
     <th>Kafka 0.10</th>
+    <th>Kafka 0.11</th>
   </tr>
   <tr>
     <th>Producer API</th>
     <td>Full support</td>
-    <td>Full support</td>
+    <td>Full support in v0.4.x</td>
+    <td>Full support in v0.5.x</td>
     <td>Limited support</td>
   </tr>
   <tr>
     <th>Consumer API</th>
     <td>Unsupported</td>
-    <td>Full support</td>
+    <td>Full support in v0.4.x</td>
+    <td>Full support in v0.5.x</td>
     <td>Limited support</td>
   </tr>
 </table>
 
-This library is targeting Kafka 0.9, although there is limited support for versions 0.8 and 0.10:
+This library is targeting Kafka 0.9 with the v0.4.x series and Kafka 0.10 with the v0.5.x series. There's limited support for Kafka 0.8, and things should work with Kafka 0.11, although there may be performance issues due to changes in the protocol.
 
 - **Kafka 0.8:** Full support for the Producer API, but no support for consumer groups. Simple message fetching works.
-- **Kafka 0.10:** Full support for both the Producer and Consumer APIs, but the addition of message timestamps is not supported. However, ruby-kafka should be completely compatible with Kafka 0.10 brokers.
+- **Kafka 0.9:** Full support for the Producer and Consumer API in ruby-kafka v0.4.x.
+- **Kafka 0.10:** Full support for the Producer and Consumer API in ruby-kafka v0.5.x.
+- **Kafka 0.11:** Everything that works with Kafka 0.10 should still work, but so far no features specific to Kafka 0.11 have been added.
 
 This library requires Ruby 2.1 or higher.
 
@@ -452,7 +459,9 @@ end
 
 ### Consuming Messages from Kafka
 
-Consuming messages from a Kafka topic is simple:
+**Note:** If you're just looking to get started with Kafka consumers, you might be interested in visiting the [Higher level libraries](#higher-level-libraries) section that lists ruby-kafka based frameworks. Read on, if you're interested in either rolling your own executable consumers or if you want to learn more about how consumers work in Kafka.
+
+Consuming messages from a Kafka topic with ruby-kafka is simple:
 
 ```ruby
 require "kafka"
@@ -489,6 +498,10 @@ consumer = kafka.consumer(group_id: "my-consumer")
 # repeatedly.
 consumer.subscribe("greetings")
 
+# Stop the consumer when the SIGTERM signal is sent to the process.
+# It's better to shut down gracefully than to kill the process.
+trap("TERM") { consumer.stop }
+
 # This will loop indefinitely, yielding each message in turn.
 consumer.each_message do |message|
   puts message.topic, message.partition
@@ -591,8 +604,8 @@ In order to shut down a running consumer process cleanly, call `#stop` on it. A
 consumer = kafka.consumer(...)
 
 # The consumer can be stopped from the command line by executing
-# `kill -s QUIT <process-id>`.
-trap("QUIT") { consumer.stop }
+# `kill -s TERM <process-id>`.
+trap("TERM") { consumer.stop }
 
 consumer.each_message do |message|
   ...
@@ -741,7 +754,7 @@ You can quite easily build monitoring on top of the provided [instrumentation ho
 The Statsd reporter is automatically enabled when the `kafka/statsd` library is required. You can optionally change the configuration.
 
 ```ruby
-require "kafka/statds"
+require "kafka/statsd"
 
 # Default is "ruby_kafka".
 Kafka::Statsd.namespace = "custom-namespace"
@@ -897,19 +910,37 @@ After checking out the repo, run `bin/setup` to install dependencies. Then, run
 
 **Note:** the specs require a working [Docker](https://www.docker.com/) instance, but should work out of the box if you have Docker installed. Please create an issue if that's not the case.
 
+If you would like to contribute to ruby-kafka, please [join our Slack team](https://ruby-kafka-slack.herokuapp.com/) and ask how best to do it.
+
 [![Circle CI](https://circleci.com/gh/zendesk/ruby-kafka.svg?style=shield)](https://circleci.com/gh/zendesk/ruby-kafka/tree/master)
 
+## Support and Discussion
+
+If you've discovered a bug, please file a [Github issue](https://github.com/zendesk/ruby-kafka/issues/new), and make sure to include all the relevant information, including the version of ruby-kafka and Kafka that you're using.
+
+If you have other questions, or would like to discuss best practises, how to contribute to the project, or any other ruby-kafka related topic, [join our Slack team](https://ruby-kafka-slack.herokuapp.com/)!
+
 ## Roadmap
 
-The current stable release is v0.3. This release is running in production at Zendesk, but it's still not recommended that you use it when data loss is unacceptable. It will take a little while until all edge cases have been uncovered and handled.
+Version 0.4 will be the last minor release with support for the Kafka 0.9 protocol. It is recommended that you pin your dependency on ruby-kafka to `~> 0.4.0` in order to receive bugfixes and security updates. New features will only target version 0.5 and up, which will be incompatible with the Kafka 0.9 protocol.
 
 ### v0.4
 
-Beta release of the Consumer API, allowing balanced Consumer Groups coordinating access to partitions. Kafka 0.9 only.
+Current stable release with support for the Kafka 0.9 protocol.
+
+### v0.5
+
+Next stable release, with support for the Kafka 0.10 protocol and eventually newer protocol versions.
+
+## Higher level libraries
+
+Currently, there are three actively developed frameworks based on ruby-kafka, that provide higher level API that can be used to work with Kafka messages:
+
+* [Racecar](https://github.com/zendesk/racecar) - A simple framework that integrates with Ruby on Rails to provide a seamless way to write, test, configure, and run Kafka consumers. It comes with sensible defaults and conventions.
 
-### v1.0
+* [Karafka](https://github.com/karafka/karafka) - Framework used to simplify Apache Kafka based Ruby and Rails applications development. Karafka provides higher abstraction layers, including Capistrano, Docker and Heroku support.
 
-API freeze. All new changes will be backwards compatible.
+* [Phobos](https://github.com/klarna/phobos) - Micro framework and library for applications dealing with Apache Kafka. It wraps common behaviors needed by consumers and producers in an easy and convenient API.
 
 ## Why Create A New Library?
 

data/lib/kafka.rb

@@ -114,7 +114,7 @@ module Kafka
   class ReplicaNotAvailable < ProtocolError
   end
 
-  #
+  # 25
   class UnknownMemberId < ProtocolError
   end
 
@@ -163,37 +163,40 @@ module Kafka
   end
 
   # 36
-  class TopicAlreadyExists < Error
+  class TopicAlreadyExists < ProtocolError
   end
 
   # 37
-  class InvalidPartitions < Error
+  class InvalidPartitions < ProtocolError
   end
 
   # 38
-  class InvalidReplicationFactor < Error
+  class InvalidReplicationFactor < ProtocolError
   end
 
   # 39
-  class InvalidReplicaAssignment < Error
+  class InvalidReplicaAssignment < ProtocolError
   end
 
   # 40
-  class InvalidConfig < Error
+  class InvalidConfig < ProtocolError
   end
 
   # 41
-  class NotController < Error
+  class NotController < ProtocolError
   end
 
   # 42
-  class InvalidRequest < Error
+  class InvalidRequest < ProtocolError
   end
 
   # Raised when there's a network connection error.
   class ConnectionError < Error
   end
 
+  class NoSuchBroker < Error
+  end
+
   # Raised when a producer buffer has reached its maximum size.
   class BufferOverflow < Error
   end

data/lib/kafka/client.rb

@@ -100,7 +100,6 @@ module Kafka
         partition,
         partition_key,
         create_time,
-        key.to_s.bytesize + value.to_s.bytesize
       )
 
       if partition.nil?

data/lib/kafka/connection.rb

@@ -28,6 +28,9 @@ module Kafka
     SOCKET_TIMEOUT = 10
     CONNECT_TIMEOUT = 10
 
+    # Time after which an idle connection will be reopened.
+    IDLE_TIMEOUT = 60 * 5
+
     attr_reader :encoder
     attr_reader :decoder
 
@@ -88,13 +91,18 @@ module Kafka
 
       @instrumenter.instrument("request.connection", notification) do
         open unless open?
+        reopen if idle?
 
         @correlation_id += 1
 
         write_request(request, notification)
 
         response_class = request.response_class
-        wait_for_response(response_class, notification) unless response_class.nil?
+        response = wait_for_response(response_class, notification) unless response_class.nil?
+
+        @last_request = Time.now
+
+        response
       end
     rescue Errno::EPIPE, Errno::ECONNRESET, Errno::ETIMEDOUT, EOFError => e
       close
@@ -118,6 +126,8 @@ module Kafka
 
       # Correlation id is initialized to zero and bumped for each request.
       @correlation_id = 0
+
+      @last_request = nil
     rescue Errno::ETIMEDOUT => e
       @logger.error "Timed out while trying to connect to #{self}: #{e}"
       raise ConnectionError, e
@@ -126,6 +136,15 @@ module Kafka
       raise ConnectionError, e
     end
 
+    def reopen
+      close
+      open
+    end
+
+    def idle?
+      @last_request && @last_request < Time.now - IDLE_TIMEOUT
+    end
+
     # Writes a request over the connection.
     #
     # @param request [#encode] the request that should be encoded and written.

data/lib/kafka/consumer.rb

@@ -4,9 +4,6 @@ require "kafka/fetch_operation"
 
 module Kafka
 
-  # @note This code is still alpha level. Don't use this for anything important.
-  #   The API may also change without warning.
-  #
   # A client that consumes messages from a Kafka cluster in coordination with
   # other clients.
   #
@@ -90,6 +87,11 @@ module Kafka
       nil
     end
 
+    # Stop the consumer.
+    #
+    # The consumer will finish any in-progress work and shut down.
+    #
+    # @return [nil]
     def stop
       @running = false
     end
@@ -280,6 +282,21 @@ module Kafka
       end
     end
 
+    # Move the consumer's position in a topic partition to the specified offset.
+    #
+    # Note that this has to be done prior to calling {#each_message} or {#each_batch}
+    # and only has an effect if the consumer is assigned the partition. Typically,
+    # you will want to do this in every consumer group member in order to make sure
+    # that the member that's assigned the partition knows where to start.
+    #
+    # @param topic [String] 
+    # @param partition [Integer] 
+    # @param offset [Integer] 
+    # @return [nil]
+    def seek(topic, partition, offset)
+      @offset_manager.seek_to(topic, partition, offset)
+    end
+
     def commit_offsets
       @offset_manager.commit_offsets
     end

data/lib/kafka/consumer_group.rb

@@ -46,6 +46,7 @@ module Kafka
       retry
     rescue ConnectionError
       @logger.error "Connection error while trying to join group `#{@group_id}`; retrying..."
+      sleep 1
       @coordinator = nil
       retry
     end

data/lib/kafka/datadog.rb

@@ -141,6 +141,7 @@ module Kafka
         buffer_size = event.payload.fetch(:buffer_size)
         max_buffer_size = event.payload.fetch(:max_buffer_size)
         buffer_fill_ratio = buffer_size.to_f / max_buffer_size.to_f
+        buffer_fill_percentage = buffer_fill_ratio * 100.0
 
         tags = {
           client: client,
@@ -156,6 +157,7 @@ module Kafka
 
         # This gets us the avg/max buffer fill ratio per producer.
         histogram("producer.buffer.fill_ratio", buffer_fill_ratio, tags: tags)
+        histogram("producer.buffer.fill_percentage", buffer_fill_percentage, tags: tags)
       end
 
       def buffer_overflow(event)

data/lib/kafka/offset_manager.rb

@@ -36,8 +36,12 @@ module Kafka
     end
 
     def seek_to_default(topic, partition)
+      seek_to(topic, partition, -1)
+    end
+
+    def seek_to(topic, partition, offset)
       @processed_offsets[topic] ||= {}
-      @processed_offsets[topic][partition] = -1
+      @processed_offsets[topic][partition] = offset
     end
 
     def next_offset_for(topic, partition)

data/lib/kafka/pending_message.rb

@@ -1,11 +1,15 @@
 module Kafka
-  PendingMessage = Struct.new(
-    "PendingMessage",
-    :value,
-    :key,
-    :topic,
-    :partition,
-    :partition_key,
-    :create_time,
-    :bytesize)
+  class PendingMessage
+    attr_reader :value, :key, :topic, :partition, :partition_key, :create_time, :bytesize
+
+    def initialize(value, key, topic, partition, partition_key, create_time)
+      @value = value
+      @key = key
+      @topic = topic
+      @partition = partition
+      @partition_key = partition_key
+      @create_time = create_time
+      @bytesize = key.to_s.bytesize + value.to_s.bytesize
+    end
+  end
 end

data/lib/kafka/producer.rb

@@ -185,13 +185,12 @@ module Kafka
       create_time = Time.now
 
       message = PendingMessage.new(
-        value,
-        key,
-        topic,
-        partition,
-        partition_key,
+        value && value.to_s,
+        key && key.to_s,
+        topic.to_s,
+        partition && Integer(partition),
+        partition_key && partition_key.to_s,
         create_time,
-        key.to_s.bytesize + value.to_s.bytesize
       )
 
       if buffer_size >= @max_buffer_size

data/lib/kafka/protocol.rb

@@ -1,4 +1,13 @@
 module Kafka
+
+  # The protocol layer of the library.
+  #
+  # The Kafka protocol (https://kafka.apache.org/protocol) defines a set of API
+  # requests, each with a well-known numeric API key, as well as a set of error
+  # codes with specific meanings.
+  #
+  # This module, and the classes contained in it, implement the client side of
+  # the protocol.
   module Protocol
     # The replica id of non-brokers is always -1.
     REPLICA_ID = -1
@@ -16,6 +25,7 @@ module Kafka
     SYNC_GROUP_API = 14
     SASL_HANDSHAKE_API = 17
 
+    # A mapping from numeric API keys to symbolic API names.
     APIS = {
       PRODUCE_API => :produce,
       FETCH_API => :fetch,
@@ -31,6 +41,7 @@ module Kafka
       SASL_HANDSHAKE_API => :sasl_handshake,
     }
 
+    # A mapping from numeric error codes to exception classes.
     ERRORS = {
       -1 => UnknownError,
        1 => OffsetOutOfRange,
@@ -72,6 +83,12 @@ module Kafka
       42 => InvalidRequest
     }
 
+    # Handles an error code by either doing nothing (if there was no error) or
+    # by raising an appropriate exception.
+    #
+    # @param error_code Integer
+    # @raise [ProtocolError]
+    # @return [nil]
     def self.handle_error(error_code)
       if error_code == 0
         # No errors, yay!
@@ -82,6 +99,10 @@ module Kafka
       end
     end
 
+    # Returns the symbolic name for an API key.
+    #
+    # @param api_key Integer
+    # @return [Symbol]
     def self.api_name(api_key)
       APIS.fetch(api_key, :unknown)
     end

data/lib/kafka/protocol/metadata_response.rb

@@ -123,7 +123,7 @@ module Kafka
       def find_broker(node_id)
         broker = @brokers.find {|broker| broker.node_id == node_id }
 
-        raise Kafka::Error, "No broker with id #{node_id}" if broker.nil?
+        raise Kafka::NoSuchBroker, "No broker with id #{node_id}" if broker.nil?
 
         broker
       end

data/lib/kafka/sasl_gssapi_authenticator.rb

@@ -1,5 +1,3 @@
-require 'gssapi'
-
 module Kafka
   class SaslGssapiAuthenticator
     GSSAPI_IDENT = "GSSAPI"
@@ -11,6 +9,7 @@ module Kafka
       @principal = sasl_gssapi_principal
       @keytab = sasl_gssapi_keytab
 
+      load_gssapi
       initialize_gssapi_context
     end
 
@@ -59,6 +58,15 @@ module Kafka
       @decoder.bytes
     end
 
+    def load_gssapi
+      begin
+        require "gssapi"
+      rescue LoadError
+        @logger.error "In order to use GSSAPI authentication you need to install the `gssapi` gem."
+        raise
+      end
+    end
+
     def initialize_gssapi_context
       @logger.debug "GSSAPI: Initializing context with #{@connection.to_s}, principal #{@principal}"
 

data/lib/kafka/statsd.rb

@@ -10,7 +10,7 @@ require "active_support/subscriber"
 module Kafka
   # Reports operational metrics to a Statsd agent.
   #
-  #     require "kafka/statds"
+  #     require "kafka/statsd"
   #
   #     # Default is "ruby_kafka".
   #     Kafka::Statsd.namespace = "custom-namespace"
@@ -122,6 +122,7 @@ module Kafka
         buffer_size = event.payload.fetch(:buffer_size)
         max_buffer_size = event.payload.fetch(:max_buffer_size)
         buffer_fill_ratio = buffer_size.to_f / max_buffer_size.to_f
+        buffer_fill_percentage = buffer_fill_ratio * 100.0
 
         # This gets us the write rate.
         increment("producer.#{client}.#{topic}.produce.messages")
@@ -133,6 +134,7 @@ module Kafka
 
         # This gets us the avg/max buffer fill ratio per producer.
         timing("producer.#{client}.buffer.fill_ratio", buffer_fill_ratio)
+        timing("producer.#{client}.buffer.fill_percentage", buffer_fill_percentage)
       end
 
       def buffer_overflow(event)

data/lib/kafka/version.rb

@@ -1,3 +1,3 @@
 module Kafka
-  VERSION = "0.4.0"
+  VERSION = "0.4.1"
 end

data/ruby-kafka.gemspec

@@ -25,8 +25,6 @@ Gem::Specification.new do |spec|
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
 
-  spec.add_runtime_dependency 'gssapi', '>=1.2.0'
-
   spec.add_development_dependency "bundler", ">= 1.9.5"
   spec.add_development_dependency "rake", "~> 10.0"
   spec.add_development_dependency "rspec"
@@ -42,4 +40,5 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "statsd-ruby"
   spec.add_development_dependency "ruby-prof"
   spec.add_development_dependency "timecop"
+  spec.add_development_dependency "gssapi", '>=1.2.0'
 end

metadata

@@ -1,29 +1,15 @@
 --- !ruby/object:Gem::Specification
 name: ruby-kafka
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.4.1
 platform: ruby
 authors:
 - Daniel Schierbeck
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2017-07-19 00:00:00.000000000 Z
+date: 2017-08-25 00:00:00.000000000 Z
 dependencies:
-- !ruby/object:Gem::Dependency
-  name: gssapi
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: 1.2.0
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: 1.2.0
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
@@ -234,6 +220,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: gssapi
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.2.0
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.2.0
 description: A client library for the Kafka distributed commit log.
 email:
 - daniel.schierbeck@gmail.com