ruby-kafka 0.1.4 → 0.1.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 19434a542527d7e7ccc4b2c5d768f42a6f6e6837
-  data.tar.gz: 23392b9a3567d5ef80acf3a31ed6f7d97e4c0648
+  metadata.gz: add61b5ca7d42ab4a606bf411d78d9fb8baa0224
+  data.tar.gz: c8704b6429bed2e7e1159c5fc32e03db9476717f
 SHA512:
-  metadata.gz: 6acc0c5c6b58a6cd6e44573d26d86e1c1eb6d1222cb37f81d820514cc8bb0cfd9d8821e05aff8e46b0172247cc4803dcffbffe32f7cde32274cdea4f1fd3941e
-  data.tar.gz: f4d57945d640265d13c75efaa91c3f3ba182b5d33f0c33adadbea8d6f41a9b0d50d215a6841e3dc2f9431585a05193879e8185044ad4c35b015ac3ee8e05c71c
+  metadata.gz: 1669663bd73bee8b0ac73d0aefe93b8219a996f1c482ad6e7867c0b46ce5d3f91857acb21e418880d7754b67d687ff785596383b31eff7793f1211af13c943d8
+  data.tar.gz: f1de6fec8420801ca562d99925da270e61b955495a486117cc1659ccafe3c431fe8872000ebb4346af69ea8306ff4cceb237d047a957f1bebce3d8e48a87ef78

data/README.md

@@ -26,57 +26,122 @@ Or install it yourself as:
 
 Please see the [documentation site](http://www.rubydoc.info/gems/ruby-kafka) for detailed documentation on the latest release.
 
-An example of a fairly simple Kafka producer:
+### Producing Messages to Kafka
+
+A client must be initialized with at least one Kafka broker. Each client keeps a separate pool of broker connections. Don't use the same client from more than one thread.
 
 ```ruby
 require "kafka"
 
-# A client must be initialized with at least one Kafka broker. Each client keeps
-# a separate pool of broker connections. Don't use the same client from more than
-# one thread.
-kafka = Kafka.new(
-  seed_brokers: ["kafka1:9092", "kafka2:9092"],
-  client_id: "my-app",
-  logger: Logger.new($stderr),
-)
-
-# A producer buffers messages and sends them to the broker that is the leader of
-# the partition a given message is being produced to.
+kafka = Kafka.new(seed_brokers: ["kafka1:9092", "kafka2:9092"])
+```
+
+A producer buffers messages and sends them to the broker that is the leader of the partition a given message is assigned to.
+
+```ruby
 producer = kafka.get_producer
+```
+
+`produce` will buffer the message in the producer but will _not_ actually send it to the Kafka cluster.
 
-# `produce` will buffer the message in the producer.
+```ruby
 producer.produce("hello1", topic: "test-messages")
+```
+
+It's possible to specify a message key.
 
-# It's possible to specify a message key:
+```ruby
 producer.produce("hello2", key: "x", topic: "test-messages")
+```
+
+If you need to control which partition a message should be assigned to, you can pass in the `partition` parameter.
 
-# If you need to control which partition a message should be written to, you
-# can pass in the `partition` parameter:
+```ruby
 producer.produce("hello3", topic: "test-messages", partition: 1)
+```
 
-# If you don't know exactly how many partitions are in the topic, or you'd
-# rather have some level of indirection, you can pass in `partition_key`.
-# Two messages with the same partition key will always be written to the
-# same partition.
+If you don't know exactly how many partitions are in the topic, or you'd rather have some level of indirection, you can pass in `partition_key`. Two messages with the same partition key will always be assigned to the same partition.
+
+```ruby
 producer.produce("hello4", topic: "test-messages", partition_key: "yo")
+```
+
+`deliver_messages` will send the buffered messages to the cluster. Since messages may be destined for different partitions, this could involve writing to more than one Kafka broker. Note that a failure to send all buffered messages after the configured number of retries will result in `Kafka::FailedToSendMessages` being raised. This can be rescued and ignored; the messages will be kept in the buffer until the next attempt.
 
-# `send_messages` will send the buffered messages to the cluster. Since messages
-# may be destined for different partitions, this could involve writing to more
-# than one Kafka broker.
-producer.send_messages
+```ruby
+producer.deliver_messages
 ```
 
 Read the docs for [Kafka::Producer](http://www.rubydoc.info/gems/ruby-kafka/Kafka/Producer) for more details.
 
+### Partitioning
+
+Kafka topics are partitioned, with messages being assigned to a partition by the client. This allows a great deal of flexibility for the users. This section describes several strategies for partitioning and how they impact performance, data locality, etc.
+
+
+#### Load Balanced Partitioning
+
+When optimizing for efficiency, we either distribute messages as evenly as possible to all partitions, or make sure each producer always writes to a single partition. The former ensures an even load for downstream consumers; the latter ensures the highest producer performance, since message batching is done per partition.
+
+If no explicit partition is specified, the producer will look to the partition key or the message key for a value that can be used to deterministically assign the message to a partition. If there is a big number of different keys, the resulting distribution will be pretty even. If no keys are passed, the producer will randomly assign a partition. Random partitioning can be achieved even if you use message keys by passing a random partition key, e.g. `partition_key: rand(100)`.
+
+If you wish to have the producer write all messages to a single partition, simply generate a random value and re-use that as the partition key:
+
+```ruby
+partition_key = rand(100)
+
+producer.produce(msg1, topic: "messages", partition_key: partition_key)
+producer.produce(msg2, topic: "messages", partition_key: partition_key)
+
+# ...
+```
+
+You can also base the partition key on some property of the producer, for example the host name.
+
+#### Semantic Partitioning
+
+By assigning messages to a partition based on some property of the message, e.g. making sure all events tracked in a user session are assigned to the same partition, downstream consumers can make simplifying assumptions about data locality. In this example, a consumer can keep process local state pertaining to a user session knowing that all events for the session will be read from a single partition. This is also called _semantic partitioning_, since the partition assignment is part of the application behavior.
+
+Typically it's sufficient to simply pass a partition key in order to guarantee that a set of messages will be assigned to the same partition, e.g.
+
+```ruby
+# All messages with the same `session_id` will be assigned to the same partition.
+producer.produce(event, topic: "user-events", partition_key: session_id)
+```
+
+However, sometimes it's necessary to select a specific partition. When doing this, make sure that you don't pick a partition number outside the range of partitions for the topic:
+
+```ruby
+partitions = kafka.partitions_for("events")
+
+# Make sure that we don't exceed the partition count!
+partition = some_number % partitions
+
+producer.produce(event, topic: "events", partition: partition)
+```
+
+#### Compatibility with Other Clients
+
+There's no standardized way to assign messages to partitions across different Kafka client implementations. If you have a heterogeneous set of clients producing messages to the same topics it may be important to ensure a consistent partitioning scheme. This library doesn't try to implement all schemes, so you'll have to figure out which scheme the other client is using and replicate it. An example:
+
+```ruby
+partitions = kafka.partitions_for("events")
+
+# Insert your custom partitioning scheme here:
+partition = PartitioningScheme.assign(partitions, event)
+
+producer.produce(event, topic: "events", partition: partition)
+```
+
 ### Buffering and Error Handling
 
 The producer is designed for resilience in the face of temporary network errors, Kafka broker failovers, and other issues that prevent the client from writing messages to the destination topics. It does this by employing local, in-memory buffers. Only when messages are acknowledged by a Kafka broker will they be removed from the buffer.
 
-Typically, you'd configure the producer to retry failed attempts at sending messages, but sometimes all retries are exhausted. In that case, `Kafka::FailedToSendMessages` is raised from `Kafka::Producer#send_messages`. If you wish to have your application be resilient to this happening (e.g. if you're logging to Kafka from a web application) you can rescue this exception. The failed messages are still retained in the buffer, so a subsequent call to `#send_messages` will still attempt to send them.
+Typically, you'd configure the producer to retry failed attempts at sending messages, but sometimes all retries are exhausted. In that case, `Kafka::FailedToSendMessages` is raised from `Kafka::Producer#deliver_messages`. If you wish to have your application be resilient to this happening (e.g. if you're logging to Kafka from a web application) you can rescue this exception. The failed messages are still retained in the buffer, so a subsequent call to `#deliver_messages` will still attempt to send them.
 
 Note that there's a maximum buffer size; pass in a different value for `max_buffer_size` when calling `#get_producer` in order to configure this.
 
-A final note on buffers: local buffers give resilience against broker and network failures, and allow higher throughput due to message batching, but they also trade off consistency guarantees for higher availibility and resilience. If your local process dies while messages are buffered, those messages will be lost. If you require high levels of consistency, you should call `#send_messages` immediately after `#produce`.
+A final note on buffers: local buffers give resilience against broker and network failures, and allow higher throughput due to message batching, but they also trade off consistency guarantees for higher availibility and resilience. If your local process dies while messages are buffered, those messages will be lost. If you require high levels of consistency, you should call `#deliver_messages` immediately after `#produce`.
 
 ### Understanding Timeouts
 
@@ -92,7 +157,7 @@ It's important to understand how timeouts work if you have a latency sensitive a
 * `ack_timeout` is a timeout executed by a broker when the client is sending messages to it. It defines the number of seconds the broker should wait for replicas to acknowledge the write before responding to the client with an error. As such, it relates to the `required_acks` setting. It should be set lower than `socket_timeout`.
 * `retry_backoff` configures the number of seconds to wait after a failed attempt to send messages to a Kafka broker before retrying. The `max_retries` setting defines the maximum number of retries to attempt, and so the total duration could be up to `max_retries * retry_backoff` seconds. The timeout can be arbitrarily long, and shouldn't be too short: if a broker goes down its partitions will be handed off to another broker, and that can take tens of seconds.
 
-When sending many messages, it's likely that the client needs to send some messages to each broker in the cluster. Given `n` brokers in the cluster, the total wait time when calling `Kafka::Producer#send_messages` can be up to
+When sending many messages, it's likely that the client needs to send some messages to each broker in the cluster. Given `n` brokers in the cluster, the total wait time when calling `Kafka::Producer#deliver_messages` can be up to
 
     n * (connect_timeout + socket_timeout + retry_backoff) * max_retries
 

data/examples/simple-producer.rb

@@ -30,11 +30,11 @@ begin
     producer.produce(line, topic: topic)
 
     # Send messages for every 10 lines.
-    producer.send_messages if index % 10 == 0
+    producer.deliver_messages if index % 10 == 0
   end
 ensure
   # Make sure to send any remaining messages.
-  producer.send_messages
+  producer.deliver_messages
 
   producer.shutdown
 end

data/lib/kafka/broker.rb

@@ -31,9 +31,8 @@ module Kafka
     # @return [Kafka::Protocol::MetadataResponse]
     def fetch_metadata(**options)
       request = Protocol::TopicMetadataRequest.new(**options)
-      response_class = Protocol::MetadataResponse
 
-      @connection.send_request(request, response_class)
+      @connection.send_request(request)
     end
 
     # Fetches messages from a specified topic and partition.
@@ -42,9 +41,8 @@ module Kafka
     # @return [Kafka::Protocol::FetchResponse]
     def fetch_messages(**options)
       request = Protocol::FetchRequest.new(**options)
-      response_class = Protocol::FetchResponse
 
-      @connection.send_request(request, response_class)
+      @connection.send_request(request)
     end
 
     # Lists the offset of the specified topics and partitions.
@@ -53,9 +51,8 @@ module Kafka
     # @return [Kafka::Protocol::ListOffsetResponse]
     def list_offsets(**options)
       request = Protocol::ListOffsetRequest.new(**options)
-      response_class = Protocol::ListOffsetResponse
 
-      @connection.send_request(request, response_class)
+      @connection.send_request(request)
     end
 
     # Produces a set of messages to the broker.
@@ -64,9 +61,8 @@ module Kafka
     # @return [Kafka::Protocol::ProduceResponse]
     def produce(**options)
       request = Protocol::ProduceRequest.new(**options)
-      response_class = request.requires_acks? ? Protocol::ProduceResponse : nil
 
-      @connection.send_request(request, response_class)
+      @connection.send_request(request)
     end
   end
 end

data/lib/kafka/client.rb

@@ -132,6 +132,14 @@ module Kafka
       @cluster.topics
     end
 
+    # Counts the number of partitions in a topic.
+    #
+    # @param topic [String]
+    # @return [Integer] the number of partitions in the topic.
+    def partitions_for(topic)
+      @cluster.partitions_for(topic).count
+    end
+
     def close
       @cluster.disconnect
     end

data/lib/kafka/cluster.rb

@@ -67,6 +67,7 @@ module Kafka
     end
 
     def partitions_for(topic)
+      add_target_topics([topic])
       cluster_info.partitions_for(topic)
     end
 

data/lib/kafka/connection.rb

@@ -15,7 +15,7 @@ module Kafka
   #
   # ## Instrumentation
   #
-  # Connections emit a `request.kafka` notification on each request. The following
+  # Connections emit a `request.connection.kafka` notification on each request. The following
   # keys will be found in the payload:
   #
   # * `:api` — the name of the API being invoked.
@@ -68,23 +68,26 @@ module Kafka
 
     # Sends a request over the connection.
     #
-    # @param request [#encode] the request that should be encoded and written.
-    # @param response_class [#decode] an object that can decode the response.
+    # @param request [#encode, #response_class] the request that should be
+    #   encoded and written.
     #
-    # @return [Object] the response that was decoded by `response_class`.
-    def send_request(request, response_class)
-      Instrumentation.instrument("request.kafka") do |notification|
+    # @return [Object] the response.
+    def send_request(request)
+      # Default notification payload.
+      notification = {
+        api: Protocol.api_name(request.api_key),
+        request_size: 0,
+        response_size: 0,
+      }
+
+      Instrumentation.instrument("request.connection.kafka", notification) do
         open unless open?
 
         @correlation_id += 1
 
-        # Look up the API name.
-        notification[:api] = Protocol.api_name(request.api_key)
-
-        # We may not read a response, in which case the size is zero.
-        notification[:response_size] = 0
-
         write_request(request, notification)
+
+        response_class = request.response_class
         wait_for_response(response_class, notification) unless response_class.nil?
       end
     rescue Errno::EPIPE, Errno::ECONNRESET, Errno::ETIMEDOUT, EOFError => e

data/lib/kafka/produce_operation.rb

@@ -5,7 +5,7 @@ module Kafka
   #
   # ## Instrumentation
   #
-  # When executing the operation, an `append_message_set.kafka` notification will be
+  # When executing the operation, an `ack_messages.producer.kafka` notification will be
   # emitted for each message set that was successfully appended to a topic partition.
   # The following keys will be found in the payload:
   #
@@ -14,10 +14,13 @@ module Kafka
   # * `:offset` — the offset of the first message in the message set.
   # * `:message_count` — the number of messages that were appended.
   #
-  # If there was an error appending the message set, the key `:exception` will be set
-  # in the payload. In that case, the message set will most likely not have been
-  # appended and will possibly be retried later. Check this key before reporting the
-  # operation as successful.
+  # In addition to these notifications, a `send_messages.producer.kafka` notification will
+  # be emitted after the operation completes, regardless of whether it succeeds. This
+  # notification will have the following keys:
+  #
+  # * `message_count` – the total number of messages that the operation tried to
+  #   send. Note that not all messages may get delivered.
+  # * `sent_message_count` – the number of messages that were successfully sent.
   #
   class ProduceOperation
     def initialize(cluster:, buffer:, required_acks:, ack_timeout:, logger:)
@@ -29,6 +32,22 @@ module Kafka
     end
 
     def execute
+      Instrumentation.instrument("send_messages.producer.kafka") do |notification|
+        message_count = @buffer.size
+
+        notification[:message_count] = message_count
+
+        begin
+          send_buffered_messages
+        ensure
+          notification[:sent_message_count] = message_count - @buffer.size
+        end
+      end
+    end
+
+    private
+
+    def send_buffered_messages
       messages_for_broker = {}
 
       @buffer.each do |topic, partition, messages|
@@ -68,8 +87,6 @@ module Kafka
       end
     end
 
-    private
-
     def handle_response(response)
       response.each_partition do |topic_info, partition_info|
         topic = topic_info.topic
@@ -78,16 +95,14 @@ module Kafka
         message_count = @buffer.message_count_for_partition(topic: topic, partition: partition)
 
         begin
-          payload = {
+          Protocol.handle_error(partition_info.error_code)
+
+          Instrumentation.instrument("ack_messages.producer.kafka", {
             topic: topic,
             partition: partition,
             offset: offset,
             message_count: message_count,
-          }
-
-          Instrumentation.instrument("append_message_set.kafka", payload) do
-            Protocol.handle_error(partition_info.error_code)
-          end
+          })
         rescue Kafka::CorruptMessage
           @logger.error "Corrupt message when writing to #{topic}/#{partition}"
         rescue Kafka::UnknownTopicOrPartition

data/lib/kafka/producer.rb

@@ -23,10 +23,10 @@ module Kafka
   #
   # ## Buffering
   #
-  # The producer buffers pending messages until {#send_messages} is called. Note that there is
+  # The producer buffers pending messages until {#deliver_messages} is called. Note that there is
   # a maximum buffer size (default is 1,000 messages) and writing messages after the
   # buffer has reached this size will result in a BufferOverflow exception. Make sure
-  # to periodically call {#send_messages} or set `max_buffer_size` to an appropriate value.
+  # to periodically call {#deliver_messages} or set `max_buffer_size` to an appropriate value.
   #
   # Buffering messages and sending them in batches greatly improves performance, so
   # try to avoid sending messages after every write. The tradeoff between throughput and
@@ -46,6 +46,17 @@ module Kafka
   # not, we do another round of requests, this time with just the remaining messages.
   # We do this for as long as `max_retries` permits.
   #
+  # ## Instrumentation
+  #
+  # After {#deliver_messages} completes, the notification
+  # `deliver_messages.producer.kafka` will be emitted.
+  #
+  # * `message_count` – the total number of messages that the producer tried to
+  #   deliver. Note that not all messages may get delivered.
+  # * `delivered_message_count` – the number of messages that were successfully
+  #   delivered.
+  # * `attempts` – the number of attempts made to deliver the messages.
+  #
   # ## Example
   #
   # This is an example of an application which reads lines from stdin and writes them
@@ -73,11 +84,11 @@ module Kafka
   #         producer.produce(line, topic: topic)
   #
   #         # Send messages for every 10 lines.
-  #         producer.send_messages if index % 10 == 0
+  #         producer.deliver_messages if index % 10 == 0
   #       end
   #     ensure
   #       # Make sure to send any remaining messages.
-  #       producer.send_messages
+  #       producer.deliver_messages
   #
   #       producer.shutdown
   #     end
@@ -123,7 +134,7 @@ module Kafka
     end
 
     # Produces a message to the specified topic. Note that messages are buffered in
-    # the producer until {#send_messages} is called.
+    # the producer until {#deliver_messages} is called.
     #
     # ## Partitioning
     #
@@ -176,7 +187,41 @@ module Kafka
     #
     # @raise [FailedToSendMessages] if not all messages could be successfully sent.
     # @return [nil]
-    def send_messages
+    def deliver_messages
+      # There's no need to do anything if the buffer is empty.
+      return if buffer_size == 0
+
+      Instrumentation.instrument("deliver_messages.producer.kafka") do |notification|
+        message_count = buffer_size
+
+        notification[:message_count] = message_count
+        notification[:attempts] = 0
+
+        begin
+          deliver_messages_with_retries(notification)
+        ensure
+          notification[:delivered_message_count] = message_count - buffer_size
+        end
+      end
+    end
+
+    # Returns the number of messages currently held in the buffer.
+    #
+    # @return [Integer] buffer size.
+    def buffer_size
+      @pending_messages.size + @buffer.size
+    end
+
+    # Closes all connections to the brokers.
+    #
+    # @return [nil]
+    def shutdown
+      @cluster.disconnect
+    end
+
+    private
+
+    def deliver_messages_with_retries(notification)
       attempt = 0
 
       # Make sure we get metadata for this topic.
@@ -194,6 +239,8 @@ module Kafka
       loop do
         attempt += 1
 
+        notification[:attempts] = attempt
+
         @cluster.refresh_metadata_if_necessary!
 
         assign_partitions!
@@ -225,22 +272,6 @@ module Kafka
       end
     end
 
-    # Returns the number of messages currently held in the buffer.
-    #
-    # @return [Integer] buffer size.
-    def buffer_size
-      @pending_messages.size + @buffer.size
-    end
-
-    # Closes all connections to the brokers.
-    #
-    # @return [nil]
-    def shutdown
-      @cluster.disconnect
-    end
-
-    private
-
     def assign_partitions!
       until @pending_messages.empty?
         # We want to keep the message in the first-stage buffer in case there's an error.

data/lib/kafka/protocol/fetch_request.rb

@@ -30,6 +30,10 @@ module Kafka
         1
       end
 
+      def response_class
+        Protocol::FetchResponse
+      end
+
       def encode(encoder)
         encoder.write_int32(@replica_id)
         encoder.write_int32(@max_wait_time)

data/lib/kafka/protocol/list_offset_request.rb

@@ -23,6 +23,10 @@ module Kafka
         2
       end
 
+      def response_class
+        Protocol::ListOffsetResponse
+      end
+
       def encode(encoder)
         encoder.write_int32(@replica_id)
 

data/lib/kafka/protocol/produce_request.rb

@@ -40,6 +40,10 @@ module Kafka
         0
       end
 
+      def response_class
+        requires_acks? ? Protocol::ProduceResponse : nil
+      end
+
       # Whether this request requires any acknowledgements at all. If no acknowledgements
       # are required, the server will not send back a response at all.
       #

data/lib/kafka/protocol/topic_metadata_request.rb

@@ -13,6 +13,10 @@ module Kafka
         3
       end
 
+      def response_class
+        Protocol::MetadataResponse
+      end
+
       def encode(encoder)
         encoder.write_array(@topics) {|topic| encoder.write_string(topic) }
       end

data/lib/kafka/version.rb

@@ -1,3 +1,3 @@
 module Kafka
-  VERSION = "0.1.4"
+  VERSION = "0.1.5"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: ruby-kafka
 version: !ruby/object:Gem::Version
-  version: 0.1.4
+  version: 0.1.5
 platform: ruby
 authors:
 - Daniel Schierbeck
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-02-15 00:00:00.000000000 Z
+date: 2016-02-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler