ruby-kafka 0.1.3 → 0.1.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: a2754b8ff5d028b2883c5657b91d1ab13881fa27
-  data.tar.gz: 71ddafeb0cabe9f03ec326037b6391070372befa
+  metadata.gz: 19434a542527d7e7ccc4b2c5d768f42a6f6e6837
+  data.tar.gz: 23392b9a3567d5ef80acf3a31ed6f7d97e4c0648
 SHA512:
-  metadata.gz: 09421c5b62a1c90e6be2ca25f2b1823d92384e7176d48252c787b8c4de73feac02f7362ac36f98e9da8b1a44968591302e4ee9020f4af5ef02baa8f852f89b7d
-  data.tar.gz: 361c720fd7bc8af0e162c463c0f24a31a5d92b942e23e6b1eeb17bb1843516ffb0b7a3778b43ee411ff492ea69274ba2e7624ab5d99631089f3da333a85fa5ed
+  metadata.gz: 6acc0c5c6b58a6cd6e44573d26d86e1c1eb6d1222cb37f81d820514cc8bb0cfd9d8821e05aff8e46b0172247cc4803dcffbffe32f7cde32274cdea4f1fd3941e
+  data.tar.gz: f4d57945d640265d13c75efaa91c3f3ba182b5d33f0c33adadbea8d6f41a9b0d50d215a6841e3dc2f9431585a05193879e8185044ad4c35b015ac3ee8e05c71c

data/README.md

@@ -2,7 +2,7 @@
 
 [![Circle CI](https://circleci.com/gh/zendesk/ruby-kafka.svg?style=shield)](https://circleci.com/gh/zendesk/ruby-kafka/tree/master)
 
-A Ruby client library for the Kafka distributed log system. The focus of this library will be operational simplicity, with good logging and metrics that can make debugging issues easier.
+A Ruby client library for [Apache Kafka](http://kafka.apache.org/), a distributed log and message bus. The focus of this library will be operational simplicity, with good logging and metrics that can make debugging issues easier.
 
 Currently, only the Producer API has been implemented, but a fully-fledged Consumer implementation compatible with Kafka 0.9 is on the roadmap.
 
@@ -24,6 +24,10 @@ Or install it yourself as:
 
 ## Usage
 
+Please see the [documentation site](http://www.rubydoc.info/gems/ruby-kafka) for detailed documentation on the latest release.
+
+An example of a fairly simple Kafka producer:
+
 ```ruby
 require "kafka"
 
@@ -64,6 +68,16 @@ producer.send_messages
 
 Read the docs for [Kafka::Producer](http://www.rubydoc.info/gems/ruby-kafka/Kafka/Producer) for more details.
 
+### Buffering and Error Handling
+
+The producer is designed for resilience in the face of temporary network errors, Kafka broker failovers, and other issues that prevent the client from writing messages to the destination topics. It does this by employing local, in-memory buffers. Only when messages are acknowledged by a Kafka broker will they be removed from the buffer.
+
+Typically, you'd configure the producer to retry failed attempts at sending messages, but sometimes all retries are exhausted. In that case, `Kafka::FailedToSendMessages` is raised from `Kafka::Producer#send_messages`. If you wish to have your application be resilient to this happening (e.g. if you're logging to Kafka from a web application) you can rescue this exception. The failed messages are still retained in the buffer, so a subsequent call to `#send_messages` will still attempt to send them.
+
+Note that there's a maximum buffer size; pass in a different value for `max_buffer_size` when calling `#get_producer` in order to configure this.
+
+A final note on buffers: local buffers give resilience against broker and network failures, and allow higher throughput due to message batching, but they also trade off consistency guarantees for higher availibility and resilience. If your local process dies while messages are buffered, those messages will be lost. If you require high levels of consistency, you should call `#send_messages` immediately after `#produce`.
+
 ### Understanding Timeouts
 
 It's important to understand how timeouts work if you have a latency sensitive application. This library allows configuring timeouts on different levels:
@@ -94,7 +108,7 @@ After checking out the repo, run `bin/setup` to install dependencies. Then, run
 
 The current stable release is v0.1. This release is running in production at Zendesk, but it's still not recommended that you use it when data loss is unacceptable. It will take a little while until all edge cases have been uncovered and handled.
 
-The API may still be changed in v0.2. 
+The API may still be changed in v0.2.
 
 ### v0.2: Stable Producer API
 
@@ -114,7 +128,7 @@ There are a few existing Kafka clients in Ruby:
 * [Hermann](https://github.com/reiseburo/hermann) wraps the C library [librdkafka](https://github.com/edenhill/librdkafka) and seems to be very efficient, but its API and mode of operation is too intrusive for our needs.
 * [jruby-kafka](https://github.com/joekiller/jruby-kafka) is a great option if you're running on JRuby.
 
-We needed a robust client that could be used from our existing Ruby apps, allowed our Ops to monitor operation, and provided flexible error handling. There didn't exist such a client, hence this project. 
+We needed a robust client that could be used from our existing Ruby apps, allowed our Ops to monitor operation, and provided flexible error handling. There didn't exist such a client, hence this project.
 
 ## Contributing
 

data/Rakefile

@@ -3,4 +3,4 @@ require "rspec/core/rake_task"
 
 RSpec::Core::RakeTask.new(:spec)
 
-task :default => :spec
+task default: :spec

data/examples/simple-consumer.rb

@@ -0,0 +1,48 @@
+# Consumes lines from a Kafka partition and writes them to STDOUT.
+#
+# You need to define the environment variable KAFKA_BROKERS for this
+# to work, e.g.
+#
+#     export KAFKA_BROKERS=localhost:9092
+#
+
+$LOAD_PATH.unshift(File.expand_path("../../lib", __FILE__))
+
+require "kafka"
+
+# We don't want log output to clutter the console. Replace `StringIO.new`
+# with e.g. `$stderr` if you want to see what's happening under the hood.
+logger = Logger.new(StringIO.new)
+
+brokers = ENV.fetch("KAFKA_BROKERS").split(",")
+
+# Make sure to create this topic in your Kafka cluster or configure the
+# cluster to auto-create topics.
+topic = "text"
+
+kafka = Kafka.new(
+  seed_brokers: brokers,
+  client_id: "simple-consumer",
+  socket_timeout: 20,
+  logger: logger,
+)
+
+begin
+  offset = :latest
+  partition = 0
+
+  loop do
+    messages = kafka.fetch_messages(
+      topic: topic,
+      partition: partition,
+      offset: offset
+    )
+
+    messages.each do |message|
+      puts message.value
+      offset = message.offset + 1
+    end
+  end
+ensure
+  kafka.close
+end

data/examples/simple-producer.rb

@@ -1,4 +1,10 @@
 # Reads lines from STDIN, writing them to Kafka.
+#
+# You need to define the environment variable KAFKA_BROKERS for this
+# to work, e.g.
+#
+#     export KAFKA_BROKERS=localhost:9092
+#
 
 $LOAD_PATH.unshift(File.expand_path("../../lib", __FILE__))
 
@@ -9,7 +15,7 @@ brokers = ENV.fetch("KAFKA_BROKERS").split(",")
 
 # Make sure to create this topic in your Kafka cluster or configure the
 # cluster to auto-create topics.
-topic = "random-messages"
+topic = "text"
 
 kafka = Kafka.new(
   seed_brokers: brokers,

data/lib/kafka.rb

@@ -8,7 +8,7 @@ module Kafka
   # Kafka protocol specification.
   #
   # See https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
-  class ProtocolError < StandardError
+  class ProtocolError < Error
   end
 
   # This indicates that a message contents does not match its CRC.

data/lib/kafka/broker.rb

@@ -15,14 +15,20 @@ module Kafka
       @logger = logger
     end
 
+    # @return [String]
     def to_s
       "#{@connection} (node_id=#{@node_id.inspect})"
     end
 
+    # @return [nil]
     def disconnect
       @connection.close
     end
 
+    # Fetches cluster metadata from the broker.
+    #
+    # @param (see Kafka::Protocol::TopicMetadataRequest#initialize)
+    # @return [Kafka::Protocol::MetadataResponse]
     def fetch_metadata(**options)
       request = Protocol::TopicMetadataRequest.new(**options)
       response_class = Protocol::MetadataResponse
@@ -30,6 +36,32 @@ module Kafka
       @connection.send_request(request, response_class)
     end
 
+    # Fetches messages from a specified topic and partition.
+    #
+    # @param (see Kafka::Protocol::FetchRequest#initialize)
+    # @return [Kafka::Protocol::FetchResponse]
+    def fetch_messages(**options)
+      request = Protocol::FetchRequest.new(**options)
+      response_class = Protocol::FetchResponse
+
+      @connection.send_request(request, response_class)
+    end
+
+    # Lists the offset of the specified topics and partitions.
+    #
+    # @param (see Kafka::Protocol::ListOffsetRequest#initialize)
+    # @return [Kafka::Protocol::ListOffsetResponse]
+    def list_offsets(**options)
+      request = Protocol::ListOffsetRequest.new(**options)
+      response_class = Protocol::ListOffsetResponse
+
+      @connection.send_request(request, response_class)
+    end
+
+    # Produces a set of messages to the broker.
+    #
+    # @param (see Kafka::Protocol::ProduceRequest#initialize)
+    # @return [Kafka::Protocol::ProduceResponse]
     def produce(**options)
       request = Protocol::ProduceRequest.new(**options)
       response_class = request.requires_acks? ? Protocol::ProduceResponse : nil

data/lib/kafka/broker_pool.rb

@@ -1,124 +1,38 @@
 require "kafka/broker"
 
 module Kafka
-
-  # A broker pool represents the set of brokers in a cluster. It needs to be initialized
-  # with a non-empty list of seed brokers. The first seed broker that the pool can connect
-  # to will be asked for the cluster metadata, which allows the pool to map topic
-  # partitions to the current leader for those partitions.
   class BrokerPool
-
-    # Initializes a broker pool with a set of seed brokers.
-    #
-    # The pool will try to fetch cluster metadata from one of the brokers.
-    #
-    # @param seed_brokers [Array<String>]
-    # @param client_id [String]
-    # @param logger [Logger]
-    # @param connect_timeout [Integer, nil] see {Connection#initialize}.
-    # @param socket_timeout [Integer, nil] see {Connection#initialize}.
-    def initialize(seed_brokers:, client_id:, logger:, connect_timeout: nil, socket_timeout: nil)
+    def initialize(client_id:, connect_timeout: nil, socket_timeout: nil, logger:)
       @client_id = client_id
-      @logger = logger
       @connect_timeout = connect_timeout
       @socket_timeout = socket_timeout
+      @logger = logger
       @brokers = {}
-      @seed_brokers = seed_brokers
-      @cluster_info = nil
     end
 
-    def mark_as_stale!
-      @cluster_info = nil
-    end
+    def connect(host, port, node_id: nil)
+      return @brokers.fetch(node_id) if @brokers.key?(node_id)
 
-    # Finds the broker acting as the leader of the given topic and partition.
-    #
-    # @param topic [String]
-    # @param partition [Integer]
-    # @return [Broker] the broker that's currently leader.
-    def get_leader(topic, partition)
-      get_broker(get_leader_id(topic, partition))
-    end
+      broker = Broker.connect(
+        host: host,
+        port: port,
+        node_id: node_id,
+        client_id: @client_id,
+        connect_timeout: @connect_timeout,
+        socket_timeout: @socket_timeout,
+        logger: @logger,
+      )
 
-    def partitions_for(topic)
-      cluster_info.partitions_for(topic)
-    end
+      @brokers[node_id] = broker unless node_id.nil?
 
-    def topics
-      cluster_info.topics.map(&:topic_name)
+      broker
     end
 
-    def shutdown
+    def close
       @brokers.each do |id, broker|
         @logger.info "Disconnecting broker #{id}"
         broker.disconnect
       end
     end
-
-    private
-
-    def get_leader_id(topic, partition)
-      cluster_info.find_leader_id(topic, partition)
-    end
-
-    def get_broker(broker_id)
-      @brokers[broker_id] ||= connect_to_broker(broker_id)
-    end
-
-    def cluster_info
-      @cluster_info ||= fetch_cluster_info
-    end
-
-    # Fetches the cluster metadata.
-    #
-    # This is used to update the partition leadership information, among other things.
-    # The methods will go through each node listed in `seed_brokers`, connecting to the
-    # first one that is available. This node will be queried for the cluster metadata.
-    #
-    # @raise [ConnectionError] if none of the nodes in `seed_brokers` are available.
-    # @return [Protocol::MetadataResponse] the cluster metadata.
-    def fetch_cluster_info
-      @seed_brokers.each do |node|
-        @logger.info "Trying to initialize broker pool from node #{node}"
-
-        begin
-          host, port = node.split(":", 2)
-
-          broker = Broker.connect(
-            host: host,
-            port: port.to_i,
-            client_id: @client_id,
-            socket_timeout: @socket_timeout,
-            logger: @logger,
-          )
-
-          cluster_info = broker.fetch_metadata
-
-          @logger.info "Initialized broker pool with brokers: #{cluster_info.brokers.inspect}"
-
-          return cluster_info
-        rescue Error => e
-          @logger.error "Failed to fetch metadata from #{node}: #{e}"
-        ensure
-          broker.disconnect unless broker.nil?
-        end
-      end
-
-      raise ConnectionError, "Could not connect to any of the seed brokers: #{@seed_brokers.inspect}"
-    end
-
-    def connect_to_broker(broker_id)
-      broker_info = cluster_info.find_broker(broker_id)
-
-      Broker.connect(
-        host: broker_info.host,
-        port: broker_info.port,
-        node_id: broker_info.node_id,
-        client_id: @client_id,
-        connect_timeout: @connect_timeout,
-        socket_timeout: @socket_timeout,
-        logger: @logger,
-      )
-    end
   end
 end

data/lib/kafka/client.rb

@@ -1,9 +1,12 @@
-require "kafka/broker_pool"
+require "kafka/cluster"
 require "kafka/producer"
+require "kafka/fetched_message"
+require "kafka/fetch_operation"
 
 module Kafka
   class Client
     DEFAULT_CLIENT_ID = "ruby-kafka"
+    DEFAULT_LOGGER = Logger.new("/dev/null")
 
     # Initializes a new Kafka client.
     #
@@ -21,15 +24,20 @@ module Kafka
     #   connections. See {BrokerPool#initialize}.
     #
     # @return [Client]
-    def initialize(seed_brokers:, client_id: DEFAULT_CLIENT_ID, logger:, connect_timeout: nil, socket_timeout: nil)
+    def initialize(seed_brokers:, client_id: DEFAULT_CLIENT_ID, logger: DEFAULT_LOGGER, connect_timeout: nil, socket_timeout: nil)
       @logger = logger
 
-      @broker_pool = BrokerPool.new(
-        seed_brokers: seed_brokers,
+      broker_pool = BrokerPool.new(
         client_id: client_id,
-        logger: logger,
         connect_timeout: connect_timeout,
         socket_timeout: socket_timeout,
+        logger: logger,
+      )
+
+      @cluster = Cluster.new(
+        seed_brokers: seed_brokers,
+        broker_pool: broker_pool,
+        logger: logger,
       )
     end
 
@@ -38,20 +46,94 @@ module Kafka
     # `options` are passed to {Producer#initialize}.
     #
     # @see Producer#initialize
-    # @return [Producer] the Kafka producer.
+    # @return [Kafka::Producer] the Kafka producer.
     def get_producer(**options)
-      Producer.new(broker_pool: @broker_pool, logger: @logger, **options)
+      Producer.new(cluster: @cluster, logger: @logger, **options)
+    end
+
+    # Fetches a batch of messages from a single partition. Note that it's possible
+    # to get back empty batches.
+    #
+    # The starting point for the fetch can be configured with the `:offset` argument.
+    # If you pass a number, the fetch will start at that offset. However, there are
+    # two special Symbol values that can be passed instead:
+    #
+    # * `:earliest` — the first offset in the partition.
+    # * `:latest` — the next offset that will be written to, effectively making the
+    #   call block until there is a new message in the partition.
+    #
+    # The Kafka protocol specifies the numeric values of these two options: -2 and -1,
+    # respectively. You can also pass in these numbers directly.
+    #
+    # ## Example
+    #
+    # When enumerating the messages in a partition, you typically fetch batches
+    # sequentially.
+    #
+    #     offset = :earliest
+    #
+    #     loop do
+    #       messages = kafka.fetch_messages(
+    #         topic: "my-topic",
+    #         partition: 42,
+    #         offset: offset,
+    #       )
+    #
+    #       messages.each do |message|
+    #         puts message.offset, message.key, message.value
+    #
+    #         # Set the next offset that should be read to be the subsequent
+    #         # offset.
+    #         offset = message.offset + 1
+    #       end
+    #     end
+    #
+    # See a working example in `examples/simple-consumer.rb`.
+    #
+    # @note This API is still alpha level. Don't try to use it in production.
+    #
+    # @param topic [String] the topic that messages should be fetched from.
+    #
+    # @param partition [Integer] the partition that messages should be fetched from.
+    #
+    # @param offset [Integer, Symbol] the offset to start reading from. Default is
+    #   the latest offset.
+    #
+    # @param max_wait_time [Integer] the maximum amount of time to wait before
+    #   the server responds, in seconds.
+    #
+    # @param min_bytes [Integer] the minimum number of bytes to wait for. If set to
+    #   zero, the broker will respond immediately, but the response may be empty.
+    #   The default is 1 byte, which means that the broker will respond as soon as
+    #   a message is written to the partition.
+    #
+    # @param max_bytes [Integer] the maximum number of bytes to include in the
+    #   response message set. Default is 1 MB. You need to set this higher if you
+    #   expect messages to be larger than this.
+    #
+    # @return [Array<Kafka::FetchedMessage>] the messages returned from the broker.
+    def fetch_messages(topic:, partition:, offset: :latest, max_wait_time: 5, min_bytes: 1, max_bytes: 1048576)
+      operation = FetchOperation.new(
+        cluster: @cluster,
+        logger: @logger,
+        min_bytes: min_bytes,
+        max_wait_time: max_wait_time,
+      )
+
+      operation.fetch_from_partition(topic, partition, offset: offset, max_bytes: max_bytes)
+
+      operation.execute
     end
 
     # Lists all topics in the cluster.
     #
     # @return [Array<String>] the list of topic names.
     def topics
-      @broker_pool.topics
+      @cluster.topics
     end
 
     def close
-      @broker_pool.shutdown
+      @cluster.disconnect
     end
   end
 end