ruby-kafka 0.3.16 → 0.3.17

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 02443c4134a1dd98934b6bd2f73b2edc09fe4cf5
-  data.tar.gz: b03995318665f91f070ecb00a14aa483e4c4444e
+  metadata.gz: ae14c68f2b224bf04659de3a17203e7065d5efd7
+  data.tar.gz: d33dd2594e8159eb23d3a8589b789f2b66c5993b
 SHA512:
-  metadata.gz: 2c69cb9baa6eed992cdd7dc66b14547df32059144748b0c12fe269ec126bbd4ed62b18300aacb955f62b17235e26e605cfea53425adf278a57edcb99d4e7c272
-  data.tar.gz: d2b7dd37bc7eb3ac6666808172d289334d926d9cb17b01690c19e1f67174af4f054c003fcc204fb9f711441d4207a4693e2d54419af873930306ae9f2fe8d1a0
+  metadata.gz: 9f2d374b0237fad7d3678f18ee903640a40bc735f995b3138286928398d5d6b595bc3ce395a86d60810934617cdc2c1e207d305c5b245323c9281ffe4bc33bc5
+  data.tar.gz: 0403e6ba029d666660395fc666dc4003195e9aa947b2c44936415394b12abd14946830d4b369ac5587f5d71858f40c6ff0630628c51602603e98f340e065ef36

data/CHANGELOG.md

@@ -4,11 +4,19 @@ Changes and additions to the library will be listed here.
 
 ## Unreleased
 
+## v0.3.17
+
+- Re-commit previously committed offsets periodically with an interval of half
+  the offset retention time, starting with the first commit (#318).
+- Expose offset retention time in the Consumer API (#316).
+- Don't get blocked when there's temporarily no leader for a topic (#336).
+
 ## v0.3.16
 
 - Fix SSL socket timeout (#283).
 - Update to the latest Datadog gem (#296).
 - Automatically detect private key type (#297).
+- Only fetch messages for subscribed topics (#309).
 
 ## v0.3.15
 

data/README.md

@@ -9,35 +9,35 @@ Although parts of this library work with Kafka 0.8 – specifically, the Produce
 1. [Installation](#installation)
 2. [Compatibility](#compatibility)
 3. [Usage](#usage)
-  1. [Setting up the Kafka Client](#setting-up-the-kafka-client)
-  2. [Producing Messages to Kafka](#producing-messages-to-kafka)
-    1. [Efficiently Producing Messages](#efficiently-producing-messages)
-    1. [Asynchronously Producing Messages](#asynchronously-producing-messages)
-    2. [Serialization](#serialization)
-    3. [Partitioning](#partitioning)
-    4. [Buffering and Error Handling](#buffering-and-error-handling)
-    5. [Message Durability](#message-durability)
-    6. [Message Delivery Guarantees](#message-delivery-guarantees)
-    7. [Compression](#compression)
-    8. [Producing Messages from a Rails Application](#producing-messages-from-a-rails-application)
-  3. [Consuming Messages from Kafka](#consuming-messages-from-kafka)
-    1. [Consumer Groups](#consumer-groups)
-    2. [Consumer Checkpointing](#consumer-checkpointing)
-    3. [Topic Subscriptions](#topic-subscriptions)
-    4. [Shutting Down a Consumer](#shutting-down-a-consumer)
-    5. [Consuming Messages in Batches](#consuming-messages-in-batches)
-    6. [Balancing Throughput and Latency](#balancing-throughput-and-latency)
-  4. [Thread Safety](#thread-safety)
-  5. [Logging](#logging)
-  6. [Instrumentation](#instrumentation)
-  7. [Monitoring](#monitoring)
-    1. [Reporting Metrics to Datadog](#reporting-metrics-to-datadog)
-  8. [Understanding Timeouts](#understanding-timeouts)
-  9. [Encryption and Authentication using SSL](#encryption-and-authentication-using-ssl)
+    1. [Setting up the Kafka Client](#setting-up-the-kafka-client)
+    2. [Producing Messages to Kafka](#producing-messages-to-kafka)
+        1. [Efficiently Producing Messages](#efficiently-producing-messages)
+        1. [Asynchronously Producing Messages](#asynchronously-producing-messages)
+        2. [Serialization](#serialization)
+        3. [Partitioning](#partitioning)
+        4. [Buffering and Error Handling](#buffering-and-error-handling)
+        5. [Message Durability](#message-durability)
+        6. [Message Delivery Guarantees](#message-delivery-guarantees)
+        7. [Compression](#compression)
+        8. [Producing Messages from a Rails Application](#producing-messages-from-a-rails-application)
+    3. [Consuming Messages from Kafka](#consuming-messages-from-kafka)
+        1. [Consumer Groups](#consumer-groups)
+        2. [Consumer Checkpointing](#consumer-checkpointing)
+        3. [Topic Subscriptions](#topic-subscriptions)
+        4. [Shutting Down a Consumer](#shutting-down-a-consumer)
+        5. [Consuming Messages in Batches](#consuming-messages-in-batches)
+        6. [Balancing Throughput and Latency](#balancing-throughput-and-latency)
+    4. [Thread Safety](#thread-safety)
+    5. [Logging](#logging)
+    6. [Instrumentation](#instrumentation)
+    7. [Monitoring](#monitoring)
+        1. [Reporting Metrics to Datadog](#reporting-metrics-to-datadog)
+    8. [Understanding Timeouts](#understanding-timeouts)
+    9. [Encryption and Authentication using SSL](#encryption-and-authentication-using-ssl)
 4. [Design](#design)
-  1. [Producer Design](#producer-design)
-  2. [Asynchronous Producer Design](#asynchronous-producer-design)
-  3. [Consumer Design](#consumer-design)
+    1. [Producer Design](#producer-design)
+    2. [Asynchronous Producer Design](#asynchronous-producer-design)
+    3. [Consumer Design](#consumer-design)
 5. [Development](#development)
 6. [Roadmap](#roadmap)
 
@@ -166,7 +166,7 @@ Read the docs for [Kafka::Producer](http://www.rubydoc.info/gems/ruby-kafka/Kafk
 
 #### Asynchronously Producing Messages
 
-A normal producer will block while `#deliver_messages` is sending messages to Kafka, possible for tens of seconds or even minutes at a time, depending on your timeout and retry settings. Furthermore, you have to call `#deliver_messages` manually, with a frequency that balances batch size with message delay.
+A normal producer will block while `#deliver_messages` is sending messages to Kafka, possibly for tens of seconds or even minutes at a time, depending on your timeout and retry settings. Furthermore, you have to call `#deliver_messages` manually, with a frequency that balances batch size with message delay.
 
 In order to avoid blocking during message deliveries you can use the _asynchronous producer_ API. It is mostly similar to the synchronous API, with calls to `#produce` and `#deliver_messages`. The main difference is that rather than blocking, these calls will return immediately. The actual work will be done in a background thread, with the messages and operations being sent from the caller over a thread safe queue.
 
@@ -505,6 +505,10 @@ By default, offsets are committed every 10 seconds. You can increase the frequen
 
 In addition to the time based trigger it's possible to trigger checkpointing in response to _n_ messages having been processed, known as the _offset commit threshold_. This puts a bound on the number of messages that can be double-processed before the problem is detected. Setting this to 1 will cause an offset commit to take place every time a message has been processed. By default this trigger is disabled.
 
+Stale offsets are periodically purged by the broker. The broker setting `offsets.retention.minutes` controls the retention window for committed offsets, and defaults to 1 day. The length of the retention window, known as _offset retention time_, can be changed for the consumer.
+
+Previously committed offsets are re-committed, to reset the retention window, at the first commit and periodically at an interval of half the _offset retention time_.
+
 ```ruby
 consumer = kafka.consumer(
   group_id: "some-group",
@@ -514,6 +518,9 @@ consumer = kafka.consumer(
 
   # Commit offsets when 100 messages have been processed.
   offset_commit_threshold: 100,
+
+  # Increase the length of time that committed offsets are kept.
+  offset_retention_time: 7 * 60 * 60
 )
 ```
 
@@ -668,7 +675,7 @@ end
 
 It is highly recommended that you monitor your Kafka client applications in production. Typical problems you'll see are:
 
-* high network errors rates, which may impact performance and time-to-delivery;
+* high network error rates, which may impact performance and time-to-delivery;
 * producer buffer growth, which may indicate that producers are unable to deliver messages at the rate they're being produced;
 * consumer processing errors, indicating exceptions are being raised in the processing code;
 * frequent consumer rebalances, which may indicate unstable network conditions or consumer configurations.

data/lib/kafka/client.rb

@@ -31,8 +31,8 @@ module Kafka
     # @param socket_timeout [Integer, nil] the timeout setting for socket
     #   connections. See {BrokerPool#initialize}.
     #
-    # @param ssl_ca_cert [String, nil] a PEM encoded CA cert to use with an
-    #   SSL connection.
+    # @param ssl_ca_cert [String, Array<String>, nil] a PEM encoded CA cert, or an Array of
+    #   PEM encoded CA certs, to use with an SSL connection.
     #
     # @param ssl_client_cert [String, nil] a PEM encoded client cert to use with an
     #   SSL connection. Must be used in combination with ssl_client_cert_key.
@@ -216,19 +216,25 @@ module Kafka
     #   not triggered by message processing.
     # @param heartbeat_interval [Integer] the interval between heartbeats; must be less
     #   than the session window.
+    # @param offset_retention_time [Integer] the time period that committed
+    #   offsets will be retained, in seconds. Defaults to the broker setting.
     # @return [Consumer]
-    def consumer(group_id:, session_timeout: 30, offset_commit_interval: 10, offset_commit_threshold: 0, heartbeat_interval: 10)
+    def consumer(group_id:, session_timeout: 30, offset_commit_interval: 10, offset_commit_threshold: 0, heartbeat_interval: 10, offset_retention_time: nil)
       cluster = initialize_cluster
 
       instrumenter = DecoratingInstrumenter.new(@instrumenter, {
         group_id: group_id,
       })
 
+      # The Kafka protocol expects the retention time to be in ms.
+      retention_time = (offset_retention_time && offset_retention_time * 1_000) || -1
+
       group = ConsumerGroup.new(
         cluster: cluster,
         logger: @logger,
         group_id: group_id,
         session_timeout: session_timeout,
+        retention_time: retention_time
       )
 
       offset_manager = OffsetManager.new(
@@ -237,6 +243,7 @@ module Kafka
         logger: @logger,
         commit_interval: offset_commit_interval,
         commit_threshold: offset_commit_threshold,
+        offset_retention_time: offset_retention_time
       )
 
       heartbeat = Heartbeat.new(
@@ -447,7 +454,9 @@ module Kafka
 
       if ca_cert
         store = OpenSSL::X509::Store.new
-        store.add_cert(OpenSSL::X509::Certificate.new(ca_cert))
+        Array(ca_cert).each do |cert|
+          store.add_cert(OpenSSL::X509::Certificate.new(cert))
+        end
         ssl_context.cert_store = store
       end
 

data/lib/kafka/consumer.rb

@@ -286,6 +286,7 @@ module Kafka
           @cluster.mark_as_stale!
         rescue LeaderNotAvailable => e
           @logger.error "Leader not available; waiting 1s before retrying"
+          @cluster.mark_as_stale!
           sleep 1
         end
       end

data/lib/kafka/consumer_group.rb

@@ -5,7 +5,7 @@ module Kafka
   class ConsumerGroup
     attr_reader :assigned_partitions, :generation_id
 
-    def initialize(cluster:, logger:, group_id:, session_timeout:)
+    def initialize(cluster:, logger:, group_id:, session_timeout:, retention_time:)
       @cluster = cluster
       @logger = logger
       @group_id = group_id
@@ -16,6 +16,7 @@ module Kafka
       @topics = Set.new
       @assigned_partitions = {}
       @assignment_strategy = RoundRobinAssignmentStrategy.new(cluster: @cluster)
+      @retention_time = retention_time
     end
 
     def subscribe(topic)
@@ -68,6 +69,7 @@ module Kafka
         member_id: @member_id,
         generation_id: @generation_id,
         offsets: offsets,
+        retention_time: @retention_time
       )
 
       response.topics.each do |topic, partitions|

data/lib/kafka/fetched_batch.rb

@@ -45,7 +45,7 @@ module Kafka
       if empty?
         0
       else
-        highwater_mark_offset - last_offset
+        (highwater_mark_offset - 1) - last_offset
       end
     end
   end

data/lib/kafka/message_buffer.rb

@@ -56,7 +56,7 @@ module Kafka
       return unless @buffer.key?(topic) && @buffer[topic].key?(partition)
 
       @size -= @buffer[topic][partition].count
-      @bytesize -= @buffer[topic][partition].map(&:bytesize).reduce(:+)
+      @bytesize -= @buffer[topic][partition].map(&:bytesize).reduce(0, :+)
 
       @buffer[topic].delete(partition)
       @buffer.delete(topic) if @buffer[topic].empty?

data/lib/kafka/offset_manager.rb

@@ -1,6 +1,10 @@
 module Kafka
   class OffsetManager
-    def initialize(cluster:, group:, logger:, commit_interval:, commit_threshold:)
+
+    # The default broker setting for offsets.retention.minutes is 1440.
+    DEFAULT_RETENTION_TIME = 1440 * 60
+
+    def initialize(cluster:, group:, logger:, commit_interval:, commit_threshold:, offset_retention_time:)
       @cluster = cluster
       @group = group
       @logger = logger
@@ -13,6 +17,8 @@ module Kafka
       @committed_offsets = nil
       @resolved_offsets = {}
       @last_commit = Time.now
+      @last_recommit = nil
+      @recommit_interval = (offset_retention_time || DEFAULT_RETENTION_TIME) / 2
     end
 
     def set_default_offset(topic, default_offset)
@@ -49,17 +55,15 @@ module Kafka
       end
     end
 
-    def commit_offsets
-      unless @processed_offsets.empty?
-        pretty_offsets = @processed_offsets.flat_map {|topic, partitions|
-          partitions.map {|partition, offset| "#{topic}/#{partition}:#{offset}" }
-        }.join(", ")
-
-        @logger.info "Committing offsets: #{pretty_offsets}"
+    def commit_offsets(recommit = false)
+      offsets = offsets_to_commit(recommit)
+      unless offsets.empty?
+        @logger.info "Committing offsets#{recommit ? ' with recommit' : ''}: #{prettify_offsets(offsets)}"
 
-        @group.commit_offsets(@processed_offsets)
+        @group.commit_offsets(offsets)
 
         @last_commit = Time.now
+        @last_recommit = Time.now if recommit
 
         @uncommitted_offsets = 0
         @committed_offsets = nil
@@ -67,8 +71,9 @@ module Kafka
     end
 
     def commit_offsets_if_necessary
-      if commit_timeout_reached? || commit_threshold_reached?
-        commit_offsets
+      recommit = recommit_timeout_reached?
+      if recommit || commit_timeout_reached? || commit_threshold_reached?
+        commit_offsets(recommit)
       end
     end
 
@@ -107,13 +112,44 @@ module Kafka
       @cluster.resolve_offsets(topic, partitions, default_offset)
     end
 
+    def seconds_since(time)
+      Time.now - time
+    end
+
     def seconds_since_last_commit
-      Time.now - @last_commit
+      seconds_since(@last_commit)
     end
 
-    def committed_offset_for(topic, partition)
+    def committed_offsets
       @committed_offsets ||= @group.fetch_offsets
-      @committed_offsets.offset_for(topic, partition)
+    end
+
+    def committed_offset_for(topic, partition)
+      committed_offsets.offset_for(topic, partition)
+    end
+
+    def offsets_to_commit(recommit = false)
+      if recommit
+        offsets_to_recommit.merge!(@processed_offsets) do |_topic, committed, processed|
+          committed.merge!(processed)
+        end
+      else
+        @processed_offsets
+      end
+    end
+
+    def offsets_to_recommit
+      committed_offsets.topics.each_with_object({}) do |(topic, partition_info), offsets|
+        topic_offsets = partition_info.keys.each_with_object({}) do |partition, partition_map|
+          offset = committed_offsets.offset_for(topic, partition)
+          partition_map[partition] = offset unless offset == -1
+        end
+        offsets[topic] = topic_offsets unless topic_offsets.empty?
+      end
+    end
+
+    def recommit_timeout_reached?
+      @last_recommit.nil? || seconds_since(@last_recommit) >= @recommit_interval
     end
 
     def commit_timeout_reached?
@@ -123,5 +159,11 @@ module Kafka
     def commit_threshold_reached?
       @commit_threshold != 0 && @uncommitted_offsets >= @commit_threshold
     end
+
+    def prettify_offsets(offsets)
+      offsets.flat_map do |topic, partitions|
+        partitions.map { |partition, offset| "#{topic}/#{partition}:#{offset}" }
+      end.join(', ')
+    end
   end
 end

data/lib/kafka/version.rb

@@ -1,3 +1,3 @@
 module Kafka
-  VERSION = "0.3.16"
+  VERSION = "0.3.17"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: ruby-kafka
 version: !ruby/object:Gem::Version
-  version: 0.3.16
+  version: 0.3.17
 platform: ruby
 authors:
 - Daniel Schierbeck
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2017-01-20 00:00:00.000000000 Z
+date: 2017-04-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler