ruby-kafka 0.3.15.beta3 → 0.3.15

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 99f22b235423d8050945416fa7b423e568ece95e
-  data.tar.gz: c9706320fa69bdcb0bfe20378dd606dbd9d9213c
+  metadata.gz: 3e781eb534c5c2ee94905fe5665270df1d688689
+  data.tar.gz: a95297b844741019599e14bae2ca7afa34304538
 SHA512:
-  metadata.gz: 1095866be7a6181392ba636f22a171b867ce7c04e340172f18b0f72e6aa3692a47f5dfd98f7360794e75e3cd334260e53c87c7b4bab29b31a5178eb80bfbcf89
-  data.tar.gz: 0f5ed394bb0a57e4b854d254d503a8af10e2189702dfd7f5af7ed208b5232232738ba409044edfad4934805a854a7d08b822c350f1e3f52a18d2b3e9048b04f7
+  metadata.gz: f3d32a672b934ccaa48e49b2af92220d2bc7d435a642e43b1161f0a57e419f495c4a569c738cc85a24c0c719da608284164f9f83e96ba2a46c06e13e436d2700
+  data.tar.gz: 5283ffa3ba2f5968acf24ae73eb5b300ad38dd78907a8524e22d5d45e4ba20ac9caaa9efd6458cf7fddca3b897c1f6ff7344efd7bf55536b178c9c427e669854

data/CHANGELOG.md

@@ -4,12 +4,9 @@ Changes and additions to the library will be listed here.
 
 ## Unreleased
 
-## v0.3.15.beta3
+## v0.3.15
 
 - Allow setting a timeout on a partition pause (#272).
-
-## v0.3.15.beta1
-
 - Allow pausing consumption of a partition (#268).
 
 ## v0.3.14

data/README.md

@@ -32,11 +32,14 @@ Although parts of this library work with Kafka 0.8 – specifically, the Produce
   4. [Thread Safety](#thread-safety)
   5. [Logging](#logging)
   6. [Instrumentation](#instrumentation)
-  7. [Understanding Timeouts](#understanding-timeouts)
-  8. [Encryption and Authentication using SSL](#encryption-and-authentication-using-ssl)
+  7. [Monitoring](#monitoring)
+    1. [Reporting Metrics to Datadog](#reporting-metrics-to-datadog)
+  8. [Understanding Timeouts](#understanding-timeouts)
+  9. [Encryption and Authentication using SSL](#encryption-and-authentication-using-ssl)
 4. [Design](#design)
   1. [Producer Design](#producer-design)
   2. [Asynchronous Producer Design](#asynchronous-producer-design)
+  3. [Consumer Design](#consumer-design)
 5. [Development](#development)
 6. [Roadmap](#roadmap)
 
@@ -140,9 +143,9 @@ kafka.deliver_message("Hello, World!", key: "hello", topic: "greetings")
 
 #### Efficiently Producing Messages
 
-While `#deliver_message` works fine for infrequent writes, there are a number of downside:
+While `#deliver_message` works fine for infrequent writes, there are a number of downsides:
 
-* Kafka is optimized for transmitting _batches_ of messages rather than individual messages, so there's a significant overhead and performance penalty in using the single-message API.
+* Kafka is optimized for transmitting messages in _batches_ rather than individually, so there's a significant overhead and performance penalty in using the single-message API.
 * The message delivery can fail in a number of different ways, but this simplistic API does not provide automatic retries.
 * The message is not buffered, so if there is an error, it is lost.
 
@@ -662,6 +665,37 @@ end
   * `request_size` is the number of bytes in the request.
   * `response_size` is the number of bytes in the response.
 
+
+### Monitoring
+
+It is highly recommended that you monitor your Kafka client applications in production. Typical problems you'll see are:
+
+* high network errors rates, which may impact performance and time-to-delivery;
+* producer buffer growth, which may indicate that producers are unable to deliver messages at the rate they're being produced;
+* consumer processing errors, indicating exceptions are being raised in the processing code;
+* frequent consumer rebalances, which may indicate unstable network conditions or consumer configurations.
+
+You can quite easily build monitoring on top of the provided [instrumentation hooks](#instrumentation). In order to further help with monitoring, a prebuilt [Datadog](https://www.datadoghq.com/) reporter is included with ruby-kafka.
+
+
+#### Reporting Metrics to Datadog
+
+The Datadog reporter is automatically enabled when the `kafka/datadog` library is required. You can optionally change the configuration.
+
+```ruby
+# This enables the reporter:
+require "kafka/datadog"
+
+# Default is "ruby_kafka".
+Kafka::Datadog.namespace = "custom-namespace"
+
+# Default is "127.0.0.1".
+Kafka::Datadog.host = "statsd.something.com"
+
+# Default is 8125.
+Kafka::Datadog.port = 1234
+```
+
 ### Understanding Timeouts
 
 It's important to understand how timeouts work if you have a latency sensitive application. This library allows configuring timeouts on different levels:
@@ -744,6 +778,10 @@ Instead of writing directly into the pending message list, [`Kafka::AsyncProduce
 
 Rather than triggering message deliveries directly, users of the async producer will typically set up _automatic triggers_, such as a timer.
 
+### Consumer Design
+
+The Consumer API is designed for flexibility and stability. The first is accomplished by not dictating any high-level object model, instead opting for a simple loop-based approach. The second is accomplished by handling group membership, heartbeats, and checkpointing automatically. Messages are marked as processed as soon as they've been successfully yielded to the user-supplied processing block, minimizing the cost of processing errors.
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
@@ -764,7 +802,7 @@ Beta release of the Consumer API, allowing balanced Consumer Groups coordinating
 
 API freeze. All new changes will be backwards compatible.
 
-## Why a new library?
+## Why Create A New Library?
 
 There are a few existing Kafka clients in Ruby:
 

data/lib/kafka/consumer.rb

@@ -143,6 +143,7 @@ module Kafka
         elsif Time.now < timeout
           true
         else
+          @logger.info "Automatically resuming partition #{topic}/#{partition}, pause timeout expired"
           resume(topic, partition)
           false
         end
@@ -280,7 +281,7 @@ module Kafka
           yield
         rescue HeartbeatError, OffsetCommitError
           join_group
-        rescue FetchError
+        rescue FetchError, NotLeaderForPartition, UnknownTopicOrPartition
           @cluster.mark_as_stale!
         rescue LeaderNotAvailable => e
           @logger.error "Leader not available; waiting 1s before retrying"

data/lib/kafka/consumer_group.rb

@@ -89,6 +89,11 @@ module Kafka
     rescue ConnectionError, UnknownMemberId, RebalanceInProgress, IllegalGeneration => e
       @logger.error "Error sending heartbeat: #{e}"
       raise HeartbeatError, e
+    rescue NotCoordinatorForGroup
+      @logger.error "Failed to find coordinator for group `#{@group_id}`; retrying..."
+      sleep 1
+      @coordinator = nil
+      retry
     end
 
     private

data/lib/kafka/offset_manager.rb

@@ -22,7 +22,10 @@ module Kafka
     def mark_as_processed(topic, partition, offset)
       @uncommitted_offsets += 1
       @processed_offsets[topic] ||= {}
-      @processed_offsets[topic][partition] = offset
+
+      # The committed offset should always be the offset of the next message that the
+      # application will read, thus adding one to the last message processed
+      @processed_offsets[topic][partition] = offset + 1
       @logger.debug "Marking #{topic}/#{partition}:#{offset} as committed"
     end
 
@@ -41,8 +44,8 @@ module Kafka
       if offset < 0
         resolve_offset(topic, partition)
       else
-        # The next offset is the last offset plus one.
-        offset + 1
+        # The next offset is the last offset.
+        offset
       end
     end
 

data/lib/kafka/version.rb

@@ -1,3 +1,3 @@
 module Kafka
-  VERSION = "0.3.15.beta3"
+  VERSION = "0.3.15"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: ruby-kafka
 version: !ruby/object:Gem::Version
-  version: 0.3.15.beta3
+  version: 0.3.15
 platform: ruby
 authors:
 - Daniel Schierbeck
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-08-29 00:00:00.000000000 Z
+date: 2016-09-12 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -317,9 +317,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: 2.1.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">"
+  - - ">="
     - !ruby/object:Gem::Version
-      version: 1.3.1
+      version: '0'
 requirements: []
 rubyforge_project: 
 rubygems_version: 2.4.5.1