ruby-kafka 0.3.7 → 0.3.8

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f399830029bb1bf2e51d01d01c1ade3015b216c2
-  data.tar.gz: c527a2ddd56a1e4d9e7399727b1dcc4d9a56efa4
+  metadata.gz: ca28feeee3d3fd46dab6145eb79a67b842d7f6ec
+  data.tar.gz: 720984362d2a5c4492a35022749d458618010736
 SHA512:
-  metadata.gz: 7f8f5ce7b095ba1dee302e8ffc0a7a23208e66ef6cdcd9f8c39cba753808c7ec741014c593deb41b936219d39c2841998f34c3c3e05bf498bd052721f2eaff78
-  data.tar.gz: 46987298c959729ea40693c07cf3a1e71c2c43dda70141d64870393ef67fc95ec4d8c8fd1443ddf581edb2b022faf6398deaaeb91440ed3cc4871581cf71b391
+  metadata.gz: c95bc7e7befbf91326d87a3a21af6bed1ba138b2243849f071084b1ec351eae728e0d03da20581f20a47a1b11a1b8842495aaf5ac55b8d6530a8c17c6e219a45
+  data.tar.gz: 56b6062281cec53d293a3e4e2e295b423dc8fbc9e2affef51f67c5c5f6697daf064f07bf6a6b34d3cd7b94161dffcfe050cdfc3e967cbe58a317ba041cfc84ef

data/CHANGELOG.md

@@ -4,6 +4,12 @@ Changes and additions to the library will be listed here.
 
 ## Unreleased
 
+## v0.3.8
+
+- Keep separate connection pools for consumers and producers initialized from
+  the same client.
+- Handle connection errors automatically in the async producer.
+
 ## v0.3.7
 
 - Default to port 9092 if no port is provided for a seed broker.

data/README.md

@@ -21,10 +21,11 @@ Although parts of this library work with Kafka 0.8 – specifically, the Produce
   2. [Consuming Messages from Kafka](#consuming-messages-from-kafka)
     1. [Consumer Checkpointing](#consumer-checkpointing)
     2. [Consuming Messages in Batches](#consuming-messages-in-batches)
-  3. [Logging](#logging)
-  4. [Instrumentation](#instrumentation)
-  5. [Understanding Timeouts](#understanding-timeouts)
-  6. [Encryption and Authentication using SSL](#encryption-and-authentication-using-ssl)
+  3. [Thread Safety](#thread-safety)
+  4. [Logging](#logging)
+  5. [Instrumentation](#instrumentation)
+  6. [Understanding Timeouts](#understanding-timeouts)
+  7. [Encryption and Authentication using SSL](#encryption-and-authentication-using-ssl)
 3. [Development](#development)
 4. [Roadmap](#roadmap)
 
@@ -259,6 +260,25 @@ That is, once `#deliver_messages` returns we can be sure that Kafka has received
 - If you configure the producer to not require acknowledgements from the Kafka brokers by setting `required_acks` to zero there is no guarantee that the messsage will ever make it to a Kafka broker.
 - If you use the asynchronous producer there's no guarantee that messages will have been delivered after `#deliver_messages` returns. A way of blocking until a message has been delivered with the asynchronous producer may be implemented in the future.
 
+It's possible to improve your chances of success when calling `#deliver_messages`, at the price of a longer max latency:
+
+```ruby
+producer = kafka.producer(
+  # The number of retries when attempting to deliver messages. The default is
+  # 2, so 3 attempts in total, but you can configure a higher or lower number:
+  max_retries: 5,
+
+  # The number of seconds to wait between retries. In order to handle longer
+  # periods of Kafka being unavailable, increase this number. The default is
+  # 1 second.
+  retry_backoff: 5,
+)
+```
+
+Note that these values affect the max latency of the operation; see [Understanding Timeouts](#understanding-timeouts) for an explanation of the various timeouts and latencies.
+
+If you use the asynchronous producer you typically don't have to worry too much about this, as retries will be done in the background.
+
 #### Compression
 
 Depending on what kind of data you produce, enabling compression may yield improved bandwidth and space usage. Compression in Kafka is done on entire messages sets rather than on individual messages. This improves the compression rate and generally means that compressions works better the larger your buffers get, since the message sets will be larger by the time they're compressed.
@@ -424,6 +444,12 @@ end
 
 One important thing to note is that the client commits the offset of the batch's messages only after the _entire_ batch has been processed.
 
+### Thread Safety
+
+You typically don't want to share a Kafka client between threads, since the network communication is not synchronized. Furthermore, you should avoid using threads in a consumer unless you're very careful about waiting for all work to complete before returning from the `#each_message` or `#each_batch` block. This is because _checkpointing_ assumes that returning from the block means that the messages that have been yielded have been successfully processed.
+
+You should also avoid sharing a synchronous producer between threads, as the internal buffers are not thread safe. However, the _asynchronous_ producer should be safe to use in a multi-threaded environment.
+
 ### Logging
 
 It's a very good idea to configure the Kafka client with a logger. All important operations and errors are logged. When instantiating your client, simply pass in a valid logger:

data/circle.yml

@@ -1,8 +1,9 @@
 machine:
+  pre:
+    - curl -sSL https://s3.amazonaws.com/circle-downloads/install-circleci-docker.sh | bash -s -- 1.10.0
   services:
     - docker
   environment:
-    LOG_TO_STDERR: true
     LOG_LEVEL: DEBUG
 
 dependencies:
@@ -14,6 +15,7 @@ dependencies:
 
 test:
   override:
-    - bundle exec rspec
+    - bundle exec rspec -r rspec_junit_formatter --format RspecJunitFormatter -o $CIRCLE_TEST_REPORTS/rspec/unit.xml
+    - bundle exec rspec -r rspec_junit_formatter --format RspecJunitFormatter -o $CIRCLE_TEST_REPORTS/rspec/functional.xml --tag functional
   post:
     - cp *.log $CIRCLE_ARTIFACTS/ || true

data/lib/kafka/async_producer.rb

@@ -211,8 +211,8 @@ module Kafka
 
       def deliver_messages
         @producer.deliver_messages
-      rescue DeliveryFailed
-        # Delivery failed.
+      rescue DeliveryFailed, ConnectionError
+        # Failed to deliver messages -- nothing to do but try again later.
       end
 
       def threshold_reached?

data/lib/kafka/client.rb

@@ -43,10 +43,11 @@ module Kafka
     def initialize(seed_brokers:, client_id: "ruby-kafka", logger: nil, connect_timeout: nil, socket_timeout: nil, ssl_ca_cert: nil, ssl_client_cert: nil, ssl_client_cert_key: nil)
       @logger = logger || Logger.new(nil)
       @instrumenter = Instrumenter.new(client_id: client_id)
+      @seed_brokers = normalize_seed_brokers(seed_brokers)
 
       ssl_context = build_ssl_context(ssl_ca_cert, ssl_client_cert, ssl_client_cert_key)
 
-      connection_builder = ConnectionBuilder.new(
+      @connection_builder = ConnectionBuilder.new(
         client_id: client_id,
         connect_timeout: connect_timeout,
         socket_timeout: socket_timeout,
@@ -55,16 +56,7 @@ module Kafka
         instrumenter: @instrumenter,
       )
 
-      broker_pool = BrokerPool.new(
-        connection_builder: connection_builder,
-        logger: @logger,
-      )
-
-      @cluster = Cluster.new(
-        seed_brokers: normalize_seed_brokers(seed_brokers),
-        broker_pool: broker_pool,
-        logger: @logger,
-      )
+      @cluster = initialize_cluster
     end
 
     # Initializes a new Kafka producer.
@@ -105,7 +97,7 @@ module Kafka
       )
 
       Producer.new(
-        cluster: @cluster,
+        cluster: initialize_cluster,
         logger: @logger,
         instrumenter: @instrumenter,
         compressor: compressor,
@@ -158,8 +150,10 @@ module Kafka
     #   than the session window.
     # @return [Consumer]
     def consumer(group_id:, session_timeout: 30, offset_commit_interval: 10, offset_commit_threshold: 0, heartbeat_interval: 10)
+      cluster = initialize_cluster
+
       group = ConsumerGroup.new(
-        cluster: @cluster,
+        cluster: cluster,
         logger: @logger,
         group_id: group_id,
         session_timeout: session_timeout,
@@ -178,7 +172,7 @@ module Kafka
       )
 
       Consumer.new(
-        cluster: @cluster,
+        cluster: cluster,
         logger: @logger,
         instrumenter: @instrumenter,
         group: group,
@@ -286,6 +280,19 @@ module Kafka
 
     private
 
+    def initialize_cluster
+      broker_pool = BrokerPool.new(
+        connection_builder: @connection_builder,
+        logger: @logger,
+      )
+
+      Cluster.new(
+        seed_brokers: @seed_brokers,
+        broker_pool: broker_pool,
+        logger: @logger,
+      )
+    end
+
     def build_ssl_context(ca_cert, client_cert, client_cert_key)
       return nil unless ca_cert || client_cert || client_cert_key
 

data/lib/kafka/message_buffer.rb

@@ -27,7 +27,7 @@ module Kafka
       buffer_for(topic, partition).concat(messages)
 
       @size += messages.count
-      @bytesize += messages.map(&:bytesize).reduce(:+)
+      @bytesize += messages.map(&:bytesize).reduce(0, :+)
     end
 
     def to_h
@@ -53,6 +53,8 @@ module Kafka
     #
     # @return [nil]
     def clear_messages(topic:, partition:)
+      return unless @buffer.key?(topic) && @buffer[topic].key?(partition)
+
       @size -= @buffer[topic][partition].count
       @bytesize -= @buffer[topic][partition].map(&:bytesize).reduce(:+)
 
@@ -70,6 +72,7 @@ module Kafka
     def clear
       @buffer = {}
       @size = 0
+      @bytesize = 0
     end
 
     private

data/lib/kafka/version.rb

@@ -1,3 +1,3 @@
 module Kafka
-  VERSION = "0.3.7"
+  VERSION = "0.3.8"
 end

data/ruby-kafka.gemspec

@@ -37,4 +37,5 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "activesupport", ">= 4.2.0", "< 5.1"
   spec.add_development_dependency "snappy"
   spec.add_development_dependency "colored"
+  spec.add_development_dependency "rspec_junit_formatter", "0.2.2"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: ruby-kafka
 version: !ruby/object:Gem::Version
-  version: 0.3.7
+  version: 0.3.8
 platform: ruby
 authors:
 - Daniel Schierbeck
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-05-10 00:00:00.000000000 Z
+date: 2016-05-23 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -156,6 +156,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec_junit_formatter
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 0.2.2
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '='
+      - !ruby/object:Gem::Version
+        version: 0.2.2
 description: |-
   A client library for the Kafka distributed commit log.