ruby-kafka 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e58cc6609bed1f99291323a15a90e1a8433370a9
-  data.tar.gz: eca9c8f1efc71812b4d4d7580f0917a7921864f1
+  metadata.gz: 87132b87fc32443be48994590b059a88b6cc5fea
+  data.tar.gz: c32e7ed87e9dcc7c2ba6f3128db0aadb123d1962
 SHA512:
-  metadata.gz: 7be411f4c72bd6de0154bbee286df4a0d67052af54f371fc643248cc6f6a8cb4760a2f4c18c47921a6031722be8c280502617fb27aa38423cf6add20667db992
-  data.tar.gz: cb721b17f832fc9c4485e6db40902e5ebdd2fbbcf62d24b27879ffa697455f38eb2ebc05355b37a413264a2452169052a35c3f0910312752a328f35d787371a2
+  metadata.gz: 84d5def3fa8963f928d2e59239e9c2678cfbe2ba904ad67bc13080a16b861bb5296394c442c14853144d065a4b86f205a8ba378e28302f67f8b80c5eaf0b7100
+  data.tar.gz: b39147eb0fd72f1753af38831871a00f597d0191a70f3f47734f87a2212b7d1500c0b1e712c36755b7ab09b27d66a26ea020f6a2fc413ea9ed9a0e2e15449341

data/CHANGELOG.md

@@ -4,6 +4,13 @@ Changes and additions to the library will be listed here.
 
 ## Unreleased
 
+## v0.3.0
+
+- Add support for encryption and authentication with SSL (Tom Crayford).
+- Allow configuring consumer offset commit policies.
+- Instrument consumer message processing.
+- Fixed an issue causing exceptions when no logger was specified.
+
 ## v0.2.0
 
 - Add instrumentation of message compression.

data/README.md

@@ -1,11 +1,29 @@
 # ruby-kafka
 
-[![Circle CI](https://circleci.com/gh/zendesk/ruby-kafka.svg?style=shield)](https://circleci.com/gh/zendesk/ruby-kafka/tree/master)
-
 A Ruby client library for [Apache Kafka](http://kafka.apache.org/), a distributed log and message bus. The focus of this library will be operational simplicity, with good logging and metrics that can make debugging issues easier.
 
 The Producer API is currently beta level and used in production. There's an alpha level Consumer Group API that has not yet been used in production and that may change without warning. Feel free to try it out but don't expect it to be stable or correct quite yet.
 
+Although parts of this library work with Kafka 0.8 – specifically, the Producer API – it's being tested and developed against Kafka 0.9. The Consumer API will be 0.9 only.
+
+#### Table of Contents
+
+1. [Installation](#installation)
+2. [Usage](#usage)
+  1. [Producing Messages to Kafka](#producing-messages-to-kafka)
+    1. [Asynchronously Producing Messages](#asynchronously-producing-messages)
+    2. [Serialization](#serialization)
+    3. [Partitioning](#partitioning)
+    4. [Buffering and Error Handling](#buffering-and-error-handling)
+    5. [Message Delivery Guarantees](#message-delivery-guarantees)
+    6. [Compression](#compression)
+  2. [Consuming Messages from Kafka](#consuming-messages-from-kafka)
+3. [Logging](#logging)
+4. [Understanding Timeouts](#understanding-timeouts)
+5. [Encryption and Authentication using SSL](#encryption-and-authentication-using-ssl)
+6. [Development](#development)
+7. [Roadmap](#roadmap)
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -74,7 +92,7 @@ producer.deliver_messages
 
 Read the docs for [Kafka::Producer](http://www.rubydoc.info/gems/ruby-kafka/Kafka/Producer) for more details.
 
-### Asynchronously Producing Messages
+#### Asynchronously Producing Messages
 
 A normal producer will block while `#deliver_messages` is sending messages to Kafka, possible for tens of seconds or even minutes at a time, depending on your timeout and retry settings. Furthermore, you have to call `#deliver_messages` manually, with a frequency that balances batch size with message delay.
 
@@ -119,7 +137,7 @@ producer.produce("hello", topic: "greetings")
 
 **Note:** if the calling thread produces messages faster than the producer can write them to Kafka, you'll eventually run into problems. The internal queue used for sending messages from the calling thread to the background worker has a size limit; once this limit is reached, a call to `#produce` will raise `Kafka::BufferOverflow`.
 
-### Serialization
+#### Serialization
 
 This library is agnostic to which serialization format you prefer. Both the value and key of a message is treated as a binary string of data. This makes it easier to use whatever serialization format you want, since you don't have to do anything special to make it work with ruby-kafka. Here's an example of encoding data with JSON:
 
@@ -139,12 +157,12 @@ data = JSON.dump(event)
 producer.produce(data, topic: "events")
 ```
 
-### Partitioning
+#### Partitioning
 
 Kafka topics are partitioned, with messages being assigned to a partition by the client. This allows a great deal of flexibility for the users. This section describes several strategies for partitioning and how they impact performance, data locality, etc.
 
 
-#### Load Balanced Partitioning
+##### Load Balanced Partitioning
 
 When optimizing for efficiency, we either distribute messages as evenly as possible to all partitions, or make sure each producer always writes to a single partition. The former ensures an even load for downstream consumers; the latter ensures the highest producer performance, since message batching is done per partition.
 
@@ -163,7 +181,7 @@ producer.produce(msg2, topic: "messages", partition_key: partition_key)
 
 You can also base the partition key on some property of the producer, for example the host name.
 
-#### Semantic Partitioning
+##### Semantic Partitioning
 
 By assigning messages to a partition based on some property of the message, e.g. making sure all events tracked in a user session are assigned to the same partition, downstream consumers can make simplifying assumptions about data locality. In this example, a consumer can keep process local state pertaining to a user session knowing that all events for the session will be read from a single partition. This is also called _semantic partitioning_, since the partition assignment is part of the application behavior.
 
@@ -185,7 +203,7 @@ partition = some_number % partitions
 producer.produce(event, topic: "events", partition: partition)
 ```
 
-#### Compatibility with Other Clients
+##### Compatibility with Other Clients
 
 There's no standardized way to assign messages to partitions across different Kafka client implementations. If you have a heterogeneous set of clients producing messages to the same topics it may be important to ensure a consistent partitioning scheme. This library doesn't try to implement all schemes, so you'll have to figure out which scheme the other client is using and replicate it. An example:
 
@@ -198,7 +216,7 @@ partition = PartitioningScheme.assign(partitions, event)
 producer.produce(event, topic: "events", partition: partition)
 ```
 
-### Buffering and Error Handling
+#### Buffering and Error Handling
 
 The producer is designed for resilience in the face of temporary network errors, Kafka broker failovers, and other issues that prevent the client from writing messages to the destination topics. It does this by employing local, in-memory buffers. Only when messages are acknowledged by a Kafka broker will they be removed from the buffer.
 
@@ -208,6 +226,108 @@ Note that there's a maximum buffer size; pass in a different value for `max_buff
 
 A final note on buffers: local buffers give resilience against broker and network failures, and allow higher throughput due to message batching, but they also trade off consistency guarantees for higher availibility and resilience. If your local process dies while messages are buffered, those messages will be lost. If you require high levels of consistency, you should call `#deliver_messages` immediately after `#produce`.
 
+#### Message Delivery Guarantees
+
+There are basically two different and incompatible guarantees that can be made in a message delivery system such as Kafka:
+
+1. _at-most-once_ delivery guarantees that a message is at most delivered to the recipient _once_. This is useful only if delivering the message twice carries some risk and should be avoided. Implicit is the fact that there's no guarantee that the message will be delivered at all.
+2. _at-least-once_ delivery guarantees that a message is delivered, but it may be delivered more than once. If the final recipient de-duplicates messages, e.g. by checking a unique message id, then it's even possible to implement _exactly-once_ delivery.
+
+Of these two options, ruby-kafka implements the second one: when in doubt about whether a message has been delivered, a producer will try to deliver it again.
+
+The guarantee is made only for the synchronous producer and boils down to this:
+
+```ruby
+producer = kafka.producer
+
+producer.produce("hello", topic: "greetings")
+
+# If this line fails with Kafka::DeliveryFailed we *may* have succeeded in deliverying
+# the message to Kafka but won't know for sure.
+producer.deliver_messages
+
+# If we get to this line we can be sure that the message has been delivered to Kafka!
+```
+
+That is, once `#deliver_messages` returns we can be sure that Kafka has received the message. Note that there are some big caveats here:
+
+- Depending on how your cluster and topic is configured the message could still be lost by Kafka.
+- If you configure the producer to not require acknowledgements from the Kafka brokers by setting `required_acks` to zero there is no guarantee that the messsage will ever make it to a Kafka broker.
+- If you use the asynchronous producer there's no guarantee that messages will have been delivered after `#deliver_messages` returns. A way of blocking until a message has been delivered with the asynchronous producer may be implemented in the future.
+
+#### Compression
+
+Depending on what kind of data you produce, enabling compression may yield improved bandwidth and space usage. Compression in Kafka is done on entire messages sets rather than on individual messages. This improves the compression rate and generally means that compressions works better the larger your buffers get, since the message sets will be larger by the time they're compressed.
+
+Since many workloads have variations in throughput and distribution across partitions, it's possible to configure a threshold for when to enable compression by setting `compression_threshold`. Only if the defined number of messages are buffered for a partition will the messages be compressed.
+
+Compression is enabled by passing the `compression_codec` parameter to `#producer` with the name of one of the algorithms allowed by Kafka:
+
+* `:snappy` for [Snappy](http://google.github.io/snappy/) compression.
+* `:gzip` for [gzip](https://en.wikipedia.org/wiki/Gzip) compression.
+
+By default, all message sets will be compressed if you specify a compression codec. To increase the compression threshold, set `compression_threshold` to an integer value higher than one.
+
+```ruby
+producer = kafka.producer(
+  compression_codec: :snappy,
+  compression_threshold: 10,
+)
+```
+
+### Consuming Messages from Kafka
+
+**Warning:** The Consumer API is still alpha level and will likely change. The consumer code should not be considered stable, as it hasn't been exhaustively tested in production environments yet.
+
+The simplest way to consume messages from a Kafka topic is using the `#fetch_messages` API:
+
+```ruby
+require "kafka"
+
+kafka = Kafka.new(seed_brokers: ["kafka1:9092", "kafka2:9092"])
+
+messages = kafka.fetch_messages(topic: "greetings", partition: 42)
+
+messages.each do |message|
+  puts message.offset, message.key, message.value
+end
+```
+
+While this is great for extremely simple use cases, there are a number of downsides:
+
+- You can only fetch from a single topic and partition at a time.
+- If you want to have multiple processes consume from the same topic, there's no way of coordinating which processes should fetch from which partitions.
+- If a process dies, there's no way to have another process resume fetching from the point in the partition that the original process had reached.
+
+The Consumer API solves all of these issues, and more. It uses the Consumer Groups feature released in Kafka 0.9 to allow multiple consumer processes to coordinate access to a topic, assigning each partition to a single consumer. When a consumer fails, the partitions that were assigned to it are re-assigned to other members of the group.
+
+Using the API is simple:
+
+```ruby
+require "kafka"
+
+kafka = Kafka.new(seed_brokers: ["kafka1:9092", "kafka2:9092"])
+
+# Consumers with the same group id will form a Consumer Group together.
+consumer = kafka.consumer(group_id: "my-consumer")
+
+consumer.subscribe("greetings")
+
+begin
+  # This will loop indefinitely, yielding each message in turn.
+  consumer.each_message do |message|
+    puts message.topic, message.partition
+    puts message.offset, message.key, message.value
+  end
+ensure
+  # Always make sure to shut down the consumer properly.
+  consumer.shutdown
+end
+```
+
+Each consumer process will be assigned one or more partitions from each topic that the group subscribes to. In order to handle more messages, simply start more processes.
+
+
 ### Logging
 
 It's a very good idea to configure the Kafka client with a logger. All important operations and errors are logged. When instantiating your client, simply pass in a valid logger:
@@ -239,33 +359,65 @@ When sending many messages, it's likely that the client needs to send some messa
 
 Make sure your application can survive being blocked for so long.
 
+### Encryption and Authentication using SSL
+
+By default, communication between Kafka clients and brokers is unencrypted and unauthenticated. Kafka 0.9 added optional support for [encryption and client authentication and authorization](http://kafka.apache.org/documentation.html#security_ssl). There are two layers of security made possible by this:
+
+#### Encryption of Communication
+
+By enabling SSL encryption you can have some confidence that messages can be sent to Kafka over an untrusted network without being intercepted.
+
+In this case you just need to pass a valid CA certificate as a string when configuring your `Kafka` client:
+
+```ruby
+kafka = Kafka.new(
+  ssl_ca_cert: File.read('my_ca_cert.pem'),
+  # ...
+)
+```
+
+Without passing the CA certificate to the client it would be impossible to protect against [man-in-the-middle attacks](https://en.wikipedia.org/wiki/Man-in-the-middle_attack).
+
+#### Client Authentication
+
+In order to authenticate the client to the cluster, you need to pass in a certificate and key created for the client and trusted by the brokers.
+
+```ruby
+kafka = Kafka.new(
+  ssl_ca_cert: File.read('my_ca_cert.pem'),
+  ssl_client_cert: File.read('my_client_cert.pem'),
+  ssl_client_cert_key: File.read('my_client_cert_key.pem'),
+  # ...
+)
+```
+
+Once client authentication is set up, it is possible to configure the Kafka cluster to [authorize client requests](http://kafka.apache.org/documentation.html#security_authz).
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
 **Note:** the specs require a working [Docker](https://www.docker.com/) instance, but should work out of the box if you have Docker installed. Please create an issue if that's not the case.
 
-## Roadmap
-
-The current stable release is v0.1. This release is running in production at Zendesk, but it's still not recommended that you use it when data loss is unacceptable. It will take a little while until all edge cases have been uncovered and handled.
+[![Circle CI](https://circleci.com/gh/zendesk/ruby-kafka.svg?style=shield)](https://circleci.com/gh/zendesk/ruby-kafka/tree/master)
 
-The API may still be changed in v0.2.
+## Roadmap
 
-### v0.2: Stable Producer API
+The current stable release is v0.2. This release is running in production at Zendesk, but it's still not recommended that you use it when data loss is unacceptable. It will take a little while until all edge cases have been uncovered and handled.
 
-Target date: end of February.
+### v0.3
 
-The API should now have stabilized and the library should be battle tested enough to deploy for critical use cases.
+Beta release of the Consumer API, allowing balanced Consumer Groups coordinating access to partitions. Kafka 0.9 only.
 
-### v1.0: Consumer API
+### v1.0
 
-The Consumer API defined by Kafka 0.9 will be implemented.
+API freeze. All new changes will be backwards compatible.
 
 ## Why a new library?
 
 There are a few existing Kafka clients in Ruby:
 
-* [Poseidon](https://github.com/bpot/poseidon) seems to work for Kafka 0.8, but the project has is unmaintained and has known issues.
+* [Poseidon](https://github.com/bpot/poseidon) seems to work for Kafka 0.8, but the project is unmaintained and has known issues.
 * [Hermann](https://github.com/reiseburo/hermann) wraps the C library [librdkafka](https://github.com/edenhill/librdkafka) and seems to be very efficient, but its API and mode of operation is too intrusive for our needs.
 * [jruby-kafka](https://github.com/joekiller/jruby-kafka) is a great option if you're running on JRuby.
 

data/examples/firehose-producer.rb

@@ -0,0 +1,49 @@
+$LOAD_PATH.unshift(File.expand_path("../../lib", __FILE__))
+
+require "kafka"
+
+KAFKA_CLIENT_CERT = ENV.fetch("KAFKA_CLIENT_CERT")
+KAFKA_CLIENT_CERT_KEY = ENV.fetch("KAFKA_CLIENT_CERT_KEY")
+KAFKA_SERVER_CERT = ENV.fetch("KAFKA_SERVER_CERT")
+KAFKA_URL = ENV.fetch("KAFKA_URL")
+KAFKA_BROKERS = KAFKA_URL.gsub("kafka+ssl://", "").split(",")
+KAFKA_TOPIC = "test-messages"
+
+NUM_THREADS = 20
+
+threads = NUM_THREADS.times.map do
+  Thread.new do
+    logger = Logger.new($stderr)
+    logger.level = Logger::INFO
+
+    kafka = Kafka.new(
+      seed_brokers: KAFKA_BROKERS,
+      logger: logger,
+      ssl_client_cert: KAFKA_CLIENT_CERT,
+      ssl_client_cert_key: KAFKA_CLIENT_CERT_KEY,
+      ssl_ca_cert: KAFKA_SERVER_CERT,
+    )
+
+    producer = kafka.async_producer(
+      delivery_interval: 1,
+      max_queue_size: 5_000,
+      max_buffer_size: 10_000,
+    )
+
+    begin
+      loop do
+        producer.produce(rand.to_s, key: rand.to_s, topic: KAFKA_TOPIC)
+      end
+    rescue Kafka::BufferOverflow
+      logger.error "Buffer overflow, backing off for 1s"
+      sleep 1
+      retry
+    ensure
+      producer.shutdown
+    end
+  end
+end
+
+threads.each {|t| t.abort_on_exception = true }
+
+threads.map(&:join)

data/examples/ssl-producer.rb

@@ -0,0 +1,42 @@
+# Reads lines from STDIN, writing them to Kafka.
+
+$LOAD_PATH.unshift(File.expand_path("../../lib", __FILE__))
+
+require "kafka"
+
+logger = Logger.new($stderr)
+brokers = ENV.fetch("KAFKA_BROKERS").split(",")
+
+# Make sure to create this topic in your Kafka cluster or configure the
+# cluster to auto-create topics.
+topic = "page-visits"
+
+ssl_context = OpenSSL::SSL::SSLContext.new
+ssl_context.set_params(
+  cert: OpenSSL::X509::Certificate.new(ENV.fetch("KAFKA_CLIENT_CERT")),
+  key: OpenSSL::PKey::RSA.new(ENV.fetch("KAFKA_CLIENT_CERT_KEY")),
+)
+
+kafka = Kafka.new(
+  seed_brokers: brokers,
+  client_id: "ssl-producer",
+  logger: logger,
+  ssl: true,
+  ssl_context: ssl_context,
+)
+
+producer = kafka.producer
+
+begin
+  $stdin.each_with_index do |line, index|
+    producer.produce(line, topic: topic)
+
+    # Send messages for every 10 lines.
+    producer.deliver_messages if index % 10 == 0
+  end
+ensure
+  # Make sure to send any remaining messages.
+  producer.deliver_messages
+
+  producer.shutdown
+end

data/lib/kafka/broker_pool.rb

@@ -2,12 +2,13 @@ require "kafka/broker"
 
 module Kafka
   class BrokerPool
-    def initialize(client_id:, connect_timeout: nil, socket_timeout: nil, logger:)
+    def initialize(client_id:, connect_timeout: nil, socket_timeout: nil, logger:, ssl_context: nil)
       @client_id = client_id
       @connect_timeout = connect_timeout
       @socket_timeout = socket_timeout
       @logger = logger
       @brokers = {}
+      @ssl_context = ssl_context
     end
 
     def connect(host, port, node_id: nil)
@@ -21,6 +22,7 @@ module Kafka
         connect_timeout: @connect_timeout,
         socket_timeout: @socket_timeout,
         logger: @logger,
+        ssl_context: @ssl_context,
       )
 
       @brokers[node_id] = broker unless node_id.nil?

data/lib/kafka/client.rb

@@ -1,3 +1,5 @@
+require "openssl"
+
 require "kafka/cluster"
 require "kafka/producer"
 require "kafka/consumer"
@@ -23,32 +25,83 @@ module Kafka
     # @param socket_timeout [Integer, nil] the timeout setting for socket
     #   connections. See {BrokerPool#initialize}.
     #
+    # @param ssl_ca_cert [String, nil] a PEM encoded CA cert to use with an
+    #   SSL connection.
+    #
+    # @param ssl_client_cert [String, nil] a PEM encoded client cert to use with an
+    #   SSL connection. Must be used in combination with ssl_client_cert_key.
+    #
+    # @param ssl_client_cert_key [String, nil] a PEM encoded client cert key to use with an
+    #   SSL connection. Must be used in combination with ssl_client_cert.
+    #
     # @return [Client]
-    def initialize(seed_brokers:, client_id: "ruby-kafka", logger: nil, connect_timeout: nil, socket_timeout: nil)
-      @logger = logger || Logger.new("/dev/null")
+    def initialize(seed_brokers:, client_id: "ruby-kafka", logger: nil, connect_timeout: nil, socket_timeout: nil, ssl_ca_cert: nil, ssl_client_cert: nil, ssl_client_cert_key: nil)
+      @logger = logger || Logger.new(nil)
+
+      ssl_context = build_ssl_context(ssl_ca_cert, ssl_client_cert, ssl_client_cert_key)
 
       broker_pool = BrokerPool.new(
         client_id: client_id,
         connect_timeout: connect_timeout,
         socket_timeout: socket_timeout,
-        logger: logger,
+        logger: @logger,
+        ssl_context: ssl_context,
       )
 
       @cluster = Cluster.new(
         seed_brokers: seed_brokers,
         broker_pool: broker_pool,
-        logger: logger,
+        logger: @logger,
       )
     end
 
-    # Builds a new producer.
+    # Initializes a new Kafka producer.
+    #
+    # @param ack_timeout [Integer] The number of seconds a broker can wait for
+    #   replicas to acknowledge a write before responding with a timeout.
+    #
+    # @param required_acks [Integer] The number of replicas that must acknowledge
+    #   a write.
     #
-    # `options` are passed to {Producer#initialize}.
+    # @param max_retries [Integer] the number of retries that should be attempted
+    #   before giving up sending messages to the cluster. Does not include the
+    #   original attempt.
+    #
+    # @param retry_backoff [Integer] the number of seconds to wait between retries.
+    #
+    # @param max_buffer_size [Integer] the number of messages allowed in the buffer
+    #   before new writes will raise {BufferOverflow} exceptions.
+    #
+    # @param max_buffer_bytesize [Integer] the maximum size of the buffer in bytes.
+    #   attempting to produce messages when the buffer reaches this size will
+    #   result in {BufferOverflow} being raised.
+    #
+    # @param compression_codec [Symbol, nil] the name of the compression codec to
+    #   use, or nil if no compression should be performed. Valid codecs: `:snappy`
+    #   and `:gzip`.
+    #
+    # @param compression_threshold [Integer] the number of messages that needs to
+    #   be in a message set before it should be compressed. Note that message sets
+    #   are per-partition rather than per-topic or per-producer.
     #
-    # @see Producer#initialize
     # @return [Kafka::Producer] the Kafka producer.
-    def producer(**options)
-      Producer.new(cluster: @cluster, logger: @logger, **options)
+    def producer(compression_codec: nil, compression_threshold: 1, ack_timeout: 5, required_acks: 1, max_retries: 2, retry_backoff: 1, max_buffer_size: 1000, max_buffer_bytesize: 10_000_000)
+      compressor = Compressor.new(
+        codec_name: compression_codec,
+        threshold: compression_threshold,
+      )
+
+      Producer.new(
+        cluster: @cluster,
+        logger: @logger,
+        compressor: compressor,
+        ack_timeout: ack_timeout,
+        required_acks: required_acks,
+        max_retries: max_retries,
+        retry_backoff: retry_backoff,
+        max_buffer_size: max_buffer_size,
+        max_buffer_bytesize: max_buffer_bytesize,
+      )
     end
 
     # Creates a new AsyncProducer instance.
@@ -76,17 +129,38 @@ module Kafka
       )
     end
 
-    # Creates a new Consumer instance.
-    #
-    # `options` are passed to {Consumer#initialize}.
-    #
-    # @see Consumer
+    # Creates a new Kafka consumer.
+    #
+    # @param group_id [String] the id of the group that the consumer should join.
+    # @param session_timeout [Integer] the number of seconds after which, if a client
+    #   hasn't contacted the Kafka cluster, it will be kicked out of the group.
+    # @param offset_commit_interval [Integer] the interval between offset commits,
+    #   in seconds.
+    # @param offset_commit_threshold [Integer] the number of messages that can be
+    #   processed before their offsets are committed. If zero, offset commits are
+    #   not triggered by message processing.
     # @return [Consumer]
-    def consumer(**options)
+    def consumer(group_id:, session_timeout: 30, offset_commit_interval: 10, offset_commit_threshold: 0)
+      group = ConsumerGroup.new(
+        cluster: @cluster,
+        logger: @logger,
+        group_id: group_id,
+        session_timeout: session_timeout,
+      )
+
+      offset_manager = OffsetManager.new(
+        group: group,
+        logger: @logger,
+        commit_interval: offset_commit_interval,
+        commit_threshold: offset_commit_threshold,
+      )
+
       Consumer.new(
         cluster: @cluster,
         logger: @logger,
-        **options,
+        group: group,
+        offset_manager: offset_manager,
+        session_timeout: session_timeout,
       )
     end
 
@@ -185,5 +259,32 @@ module Kafka
     def close
       @cluster.disconnect
     end
+
+    private
+
+    def build_ssl_context(ca_cert, client_cert, client_cert_key)
+      return nil unless ca_cert || client_cert || client_cert_key
+
+      ssl_context = OpenSSL::SSL::SSLContext.new
+
+      if client_cert && client_cert_key
+        ssl_context.set_params(
+          cert: OpenSSL::X509::Certificate.new(client_cert),
+          key: OpenSSL::PKey::RSA.new(client_cert_key)
+        )
+      elsif client_cert && !client_cert_key
+        raise ArgumentError, "Kafka client initialized with `ssl_client_cert` but no `ssl_client_cert_key`. Please provide both."
+      elsif !client_cert && client_cert_key
+        raise ArgumentError, "Kafka client initialized with `ssl_client_cert_key`, but no `ssl_client_cert`. Please provide both."
+      end
+
+      if ca_cert
+        store = OpenSSL::X509::Store.new
+        store.add_cert(OpenSSL::X509::Certificate.new(ca_cert))
+        ssl_context.cert_store = store
+      end
+
+      ssl_context
+    end
   end
 end

data/lib/kafka/connection.rb

@@ -1,5 +1,6 @@
 require "stringio"
 require "kafka/socket_with_timeout"
+require "kafka/ssl_socket_with_timeout"
 require "kafka/instrumentation"
 require "kafka/protocol/request_message"
 require "kafka/protocol/encoder"
@@ -42,12 +43,13 @@ module Kafka
     #   broker. Default is 10 seconds.
     #
     # @return [Connection] a new connection.
-    def initialize(host:, port:, client_id:, logger:, connect_timeout: nil, socket_timeout: nil)
+    def initialize(host:, port:, client_id:, logger:, connect_timeout: nil, socket_timeout: nil, ssl_context: nil)
       @host, @port, @client_id = host, port, client_id
       @logger = logger
 
       @connect_timeout = connect_timeout || CONNECT_TIMEOUT
       @socket_timeout = socket_timeout || SOCKET_TIMEOUT
+      @ssl_context = ssl_context
     end
 
     def to_s
@@ -101,7 +103,11 @@ module Kafka
     def open
       @logger.debug "Opening connection to #{@host}:#{@port} with client id #{@client_id}..."
 
-      @socket = SocketWithTimeout.new(@host, @port, connect_timeout: @connect_timeout, timeout: @socket_timeout)
+      if @ssl_context
+        @socket = SSLSocketWithTimeout.new(@host, @port, connect_timeout: @connect_timeout, timeout: @socket_timeout, ssl_context: @ssl_context)
+      else
+        @socket = SocketWithTimeout.new(@host, @port, connect_timeout: @connect_timeout, timeout: @socket_timeout)
+      end
 
       @encoder = Kafka::Protocol::Encoder.new(@socket)
       @decoder = Kafka::Protocol::Decoder.new(@socket)

data/lib/kafka/consumer.rb

@@ -1,4 +1,5 @@
 require "kafka/consumer_group"
+require "kafka/offset_manager"
 require "kafka/fetch_operation"
 
 module Kafka
@@ -50,28 +51,12 @@ module Kafka
   #
   class Consumer
 
-    # Creates a new Consumer.
-    #
-    # @param cluster [Kafka::Cluster]
-    # @param logger [Logger]
-    # @param group_id [String] the id of the group that the consumer should join.
-    # @param session_timeout [Integer] the interval between consumer heartbeats,
-    #   in seconds.
-    def initialize(cluster:, logger:, group_id:, session_timeout: 30)
+    def initialize(cluster:, logger:, group:, offset_manager:, session_timeout:)
       @cluster = cluster
       @logger = logger
-      @group_id = group_id
+      @group = group
+      @offset_manager = offset_manager
       @session_timeout = session_timeout
-
-      @group = ConsumerGroup.new(
-        cluster: cluster,
-        logger: logger,
-        group_id: group_id,
-        session_timeout: @session_timeout,
-      )
-
-      @offsets = {}
-      @default_offsets = {}
     end
 
     # Subscribes the consumer to a topic.
@@ -87,7 +72,7 @@ module Kafka
     # @return [nil]
     def subscribe(topic, default_offset: :earliest)
       @group.subscribe(topic)
-      @default_offsets[topic] = default_offset
+      @offset_manager.set_default_offset(topic, default_offset)
 
       nil
     end
@@ -111,15 +96,32 @@ module Kafka
           batch = fetch_batch
 
           batch.each do |message|
-            yield message
+            Instrumentation.instrument("process_message.consumer.kafka") do |notification|
+              notification.update(
+                topic: message.topic,
+                partition: message.partition,
+                offset: message.offset,
+                key: message.key,
+                value: message.value,
+              )
+
+              yield message
+            end
+
+            @offset_manager.commit_offsets_if_necessary
 
             send_heartbeat_if_necessary
             mark_message_as_processed(message)
           end
         rescue ConnectionError => e
-          @logger.error "Connection error while fetching messages: #{e}"
-        else
-          commit_offsets unless batch.nil? || batch.empty?
+          @logger.error "Connection error while sending heartbeat; rejoining"
+          join_group
+        rescue UnknownMemberId
+          @logger.error "Kicked out of group; rejoining"
+          join_group
+        rescue RebalanceInProgress
+          @logger.error "Group is rebalancing; rejoining"
+          join_group
         end
       end
     end
@@ -137,13 +139,20 @@ module Kafka
     #
     # @return [nil]
     def shutdown
+      @offset_manager.commit_offsets
       @group.leave
+    rescue ConnectionError
     end
 
     private
 
+    def join_group
+      @offset_manager.clear_offsets
+      @group.join
+    end
+
     def fetch_batch
-      @group.join unless @group.member?
+      join_group unless @group.member?
 
       @logger.debug "Fetching a batch of messages"
 
@@ -160,15 +169,9 @@ module Kafka
         max_wait_time: 5,
       )
 
-      offset_response = @group.fetch_offsets
-
       assigned_partitions.each do |topic, partitions|
         partitions.each do |partition|
-          offset = @offsets.fetch(topic, {}).fetch(partition) {
-            offset_response.offset_for(topic, partition)
-          }
-
-          offset = @default_offsets.fetch(topic) if offset < 0
+          offset = @offset_manager.next_offset_for(topic, partition)
 
           @logger.debug "Fetching from #{topic}/#{partition} starting at offset #{offset}"
 
@@ -178,14 +181,13 @@ module Kafka
 
       messages = operation.execute
 
-      @logger.debug "Fetched #{messages.count} messages"
+      @logger.info "Fetched #{messages.count} messages"
 
       messages
-    end
+    rescue ConnectionError => e
+      @logger.error "Connection error while fetching messages: #{e}"
 
-    def commit_offsets
-      @logger.debug "Committing offsets"
-      @group.commit_offsets(@offsets)
+      return []
     end
 
     # Sends a heartbeat if it would be necessary in order to avoid getting
@@ -204,8 +206,7 @@ module Kafka
     end
 
     def mark_message_as_processed(message)
-      @offsets[message.topic] ||= {}
-      @offsets[message.topic][message.partition] = message.offset + 1
+      @offset_manager.mark_as_processed(message.topic, message.partition, message.offset)
     end
   end
 end

data/lib/kafka/consumer_group.rb

@@ -44,6 +44,7 @@ module Kafka
     def leave
       @logger.info "[#{@member_id}] Leaving group `#{@group_id}`"
       coordinator.leave_group(group_id: @group_id, member_id: @member_id)
+    rescue ConnectionError
     end
 
     def fetch_offsets
@@ -66,14 +67,6 @@ module Kafka
           Protocol.handle_error(error_code)
         end
       end
-    rescue UnknownMemberId
-      @logger.error "Kicked out of group; rejoining"
-      join
-      retry
-    rescue IllegalGeneration
-      @logger.error "Illegal generation #{@generation_id}; rejoining group"
-      join
-      retry
     end
 
     def heartbeat
@@ -86,15 +79,6 @@ module Kafka
       )
 
       Protocol.handle_error(response.error_code)
-    rescue ConnectionError => e
-      @logger.error "Connection error while sending heartbeat; rejoining"
-      join
-    rescue UnknownMemberId
-      @logger.error "Kicked out of group; rejoining"
-      join
-    rescue RebalanceInProgress
-      @logger.error "Group is rebalancing; rejoining"
-      join
     end
 
     private
@@ -130,8 +114,6 @@ module Kafka
     end
 
     def synchronize
-      @logger.info "[#{@member_id}] Synchronizing group"
-
       group_assignment = {}
 
       if group_leader?

data/lib/kafka/offset_manager.rb

@@ -0,0 +1,75 @@
+module Kafka
+  class OffsetManager
+    def initialize(group:, logger:, commit_interval:, commit_threshold:)
+      @group = group
+      @logger = logger
+      @commit_interval = commit_interval
+      @commit_threshold = commit_threshold
+
+      @uncommitted_offsets = 0
+      @processed_offsets = {}
+      @default_offsets = {}
+      @committed_offsets = nil
+      @last_commit = Time.at(0)
+    end
+
+    def set_default_offset(topic, default_offset)
+      @default_offsets[topic] = default_offset
+    end
+
+    def mark_as_processed(topic, partition, offset)
+      @uncommitted_offsets += 1
+      @processed_offsets[topic] ||= {}
+      @processed_offsets[topic][partition] = offset + 1
+    end
+
+    def next_offset_for(topic, partition)
+      offset = @processed_offsets.fetch(topic, {}).fetch(partition) {
+        committed_offset_for(topic, partition)
+      }
+
+      offset = @default_offsets.fetch(topic) if offset < 0
+
+      offset
+    end
+
+    def commit_offsets
+      unless @processed_offsets.empty?
+        @logger.info "Committing offsets for #{@uncommitted_offsets} messages"
+
+        @group.commit_offsets(@processed_offsets)
+
+        @last_commit = Time.now
+        @processed_offsets.clear
+        @uncommitted_offsets = 0
+      end
+    end
+
+    def commit_offsets_if_necessary
+      if seconds_since_last_commit >= @commit_interval || commit_threshold_reached?
+        commit_offsets
+      end
+    end
+
+    def clear_offsets
+      @uncommitted_offsets = 0
+      @processed_offsets.clear
+      @committed_offsets = nil
+    end
+
+    private
+
+    def seconds_since_last_commit
+      Time.now - @last_commit
+    end
+
+    def committed_offset_for(topic, partition)
+      @committed_offsets ||= @group.fetch_offsets
+      @committed_offsets.offset_for(topic, partition)
+    end
+
+    def commit_threshold_reached?
+      @commit_threshold != 0 && @uncommitted_offsets >= @commit_threshold
+    end
+  end
+end

data/lib/kafka/producer.rb

@@ -130,41 +130,7 @@ module Kafka
   #
   class Producer
 
-    # Initializes a new Producer.
-    #
-    # @param cluster [Cluster] the cluster client. Typically passed in for you.
-    #
-    # @param logger [Logger] the logger that should be used. Typically passed
-    #   in for you.
-    #
-    # @param ack_timeout [Integer] The number of seconds a broker can wait for
-    #   replicas to acknowledge a write before responding with a timeout.
-    #
-    # @param required_acks [Integer] The number of replicas that must acknowledge
-    #   a write.
-    #
-    # @param max_retries [Integer] the number of retries that should be attempted
-    #   before giving up sending messages to the cluster. Does not include the
-    #   original attempt.
-    #
-    # @param retry_backoff [Integer] the number of seconds to wait between retries.
-    #
-    # @param max_buffer_size [Integer] the number of messages allowed in the buffer
-    #   before new writes will raise {BufferOverflow} exceptions.
-    #
-    # @param max_buffer_bytesize [Integer] the maximum size of the buffer in bytes.
-    #   attempting to produce messages when the buffer reaches this size will
-    #   result in {BufferOverflow} being raised.
-    #
-    # @param compression_codec [Symbol, nil] the name of the compression codec to
-    #   use, or nil if no compression should be performed. Valid codecs: `:snappy`
-    #   and `:gzip`.
-    #
-    # @param compression_threshold [Integer] the number of messages that needs to
-    #   be in a message set before it should be compressed. Note that message sets
-    #   are per-partition rather than per-topic or per-producer.
-    #
-    def initialize(cluster:, logger:, compression_codec: nil, compression_threshold: 1, ack_timeout: 5, required_acks: 1, max_retries: 2, retry_backoff: 1, max_buffer_size: 1000, max_buffer_bytesize: 10_000_000)
+    def initialize(cluster:, logger:, compressor:, ack_timeout:, required_acks:, max_retries:, retry_backoff:, max_buffer_size:, max_buffer_bytesize:)
       @cluster = cluster
       @logger = logger
       @required_acks = required_acks
@@ -173,11 +139,7 @@ module Kafka
       @retry_backoff = retry_backoff
       @max_buffer_size = max_buffer_size
       @max_buffer_bytesize = max_buffer_bytesize
-
-      @compressor = Compressor.new(
-        codec_name: @compression_codec,
-        threshold: @compression_threshold,
-      )
+      @compressor = compressor
 
       # The set of topics that are produced to.
       @target_topics = Set.new

data/lib/kafka/protocol/offset_fetch_response.rb

@@ -18,11 +18,14 @@ module Kafka
       end
 
       def offset_for(topic, partition)
-        offset_info = topics.fetch(topic).fetch(partition)
+        offset_info = topics.fetch(topic).fetch(partition, nil)
 
-        Protocol.handle_error(offset_info.error_code)
-
-        offset_info.offset
+        if offset_info
+          Protocol.handle_error(offset_info.error_code)
+          offset_info.offset
+        else
+          -1
+        end
       end
 
       def self.decode(decoder)

data/lib/kafka/ssl_socket_with_timeout.rb

@@ -0,0 +1,154 @@
+require "socket"
+
+module Kafka
+
+  # Opens sockets in a non-blocking fashion, ensuring that we're not stalling
+  # for long periods of time.
+  #
+  # It's possible to set timeouts for connecting to the server, for reading data,
+  # and for writing data. Whenever a timeout is exceeded, Errno::ETIMEDOUT is
+  # raised.
+  #
+  class SSLSocketWithTimeout
+
+    # Opens a socket.
+    #
+    # @param host [String]
+    # @param port [Integer]
+    # @param connect_timeout [Integer] the connection timeout, in seconds.
+    # @param timeout [Integer] the read and write timeout, in seconds.
+    # @param ssl_context [OpenSSL::SSL::SSLContext] which SSLContext the ssl connection should use
+    # @raise [Errno::ETIMEDOUT] if the timeout is exceeded.
+    def initialize(host, port, connect_timeout: nil, timeout: nil, ssl_context:)
+      addr = Socket.getaddrinfo(host, nil)
+      sockaddr = Socket.pack_sockaddr_in(port, addr[0][3])
+
+      @timeout = timeout
+
+      @tcp_socket = Socket.new(Socket.const_get(addr[0][0]), Socket::SOCK_STREAM, 0)
+      @tcp_socket.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_NODELAY, 1)
+
+      # first initiate the TCP socket
+      begin
+        # Initiate the socket connection in the background. If it doesn't fail 
+        # immediately it will raise an IO::WaitWritable (Errno::EINPROGRESS) 
+        # indicating the connection is in progress.
+        @tcp_socket.connect_nonblock(sockaddr)
+      rescue IO::WaitWritable
+        # IO.select will block until the socket is writable or the timeout
+        # is exceeded, whichever comes first.
+        unless IO.select(nil, [@tcp_socket], nil, connect_timeout)
+          # IO.select returns nil when the socket is not ready before timeout 
+          # seconds have elapsed
+          @tcp_socket.close
+          raise Errno::ETIMEDOUT
+        end
+
+        begin
+          # Verify there is now a good connection.
+          @tcp_socket.connect_nonblock(sockaddr)
+        rescue Errno::EISCONN
+          # The socket is connected, we're good!
+        end
+      end
+
+      # once that's connected, we can start initiating the ssl socket
+      @ssl_socket = OpenSSL::SSL::SSLSocket.new(@tcp_socket, ssl_context)
+
+      begin
+        # Initiate the socket connection in the background. If it doesn't fail 
+        # immediately it will raise an IO::WaitWritable (Errno::EINPROGRESS) 
+        # indicating the connection is in progress.
+        # Unlike waiting for a tcp socket to connect, you can't time out ssl socket
+        # connections during the connect phase properly, because IO.select only partially works.
+        # Instead, you have to retry.
+        @ssl_socket.connect_nonblock
+      rescue Errno::EAGAIN, Errno::EWOULDBLOCK, IO::WaitReadable
+        IO.select([@ssl_socket])
+        retry
+      rescue IO::WaitWritable
+        IO.select(nil, [@ssl_socket])
+        retry
+      end
+    end
+
+    # Reads bytes from the socket, possible with a timeout.
+    #
+    # @param num_bytes [Integer] the number of bytes to read.
+    # @raise [Errno::ETIMEDOUT] if the timeout is exceeded.
+    # @return [String] the data that was read from the socket.
+    def read(num_bytes)
+      buffer = ''
+      until buffer.length >= num_bytes
+        begin
+          # unlike plain tcp sockets, ssl sockets don't support IO.select
+          # properly.
+          # Instead, timeouts happen on a per read basis, and we have to
+          # catch exceptions from read_nonblock, and gradually build up
+          # our read buffer.
+          buffer << @ssl_socket.read_nonblock(num_bytes - buffer.length)
+        rescue IO::WaitReadable
+          unless IO.select([@ssl_socket], nil, nil, @timeout)
+            raise Errno::ETIMEDOUT
+          end
+          retry
+        rescue IO::WaitWritable
+          unless IO.select(nil, [@ssl_socket], nil, @timeout)
+            raise Errno::ETIMEDOUT
+          end
+          retry
+        end
+      end
+      buffer
+    end
+
+    # Writes bytes to the socket, possible with a timeout.
+    #
+    # @param bytes [String] the data that should be written to the socket.
+    # @raise [Errno::ETIMEDOUT] if the timeout is exceeded.
+    # @return [Integer] the number of bytes written.
+    def write(bytes)
+      loop do
+        written = 0
+        begin
+          # unlike plain tcp sockets, ssl sockets don't support IO.select
+          # properly.
+          # Instead, timeouts happen on a per write basis, and we have to
+          # catch exceptions from write_nonblock, and gradually build up
+          # our write buffer.
+          written += @ssl_socket.write_nonblock(bytes)
+        rescue Errno::EFAULT => error
+            raise error
+        rescue OpenSSL::SSL::SSLError, Errno::EAGAIN, Errno::EWOULDBLOCK, IO::WaitWritable => error
+          if error.is_a?(OpenSSL::SSL::SSLError) && error.message == 'write would block'
+            if IO.select(nil, [@ssl_socket], nil, @timeout)
+              retry
+            else
+              raise Errno::ETIMEDOUT
+            end
+          else
+            raise error
+          end
+        end
+
+        # Fast, common case.
+        break if written == bytes.size
+
+        # This takes advantage of the fact that most ruby implementations
+        # have Copy-On-Write strings. Thusly why requesting a subrange
+        # of data, we actually don't copy data because the new string
+        # simply references a subrange of the original.
+        bytes = bytes[written, bytes.size]
+      end
+    end
+
+    def close
+      @tcp_socket.close
+      @ssl_socket.close
+    end
+
+    def set_encoding(encoding)
+      @tcp_socket.set_encoding(encoding)
+    end
+  end
+end

data/lib/kafka/version.rb

@@ -1,3 +1,3 @@
 module Kafka
-  VERSION = "0.2.0"
+  VERSION = "0.3.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: ruby-kafka
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors:
 - Daniel Schierbeck
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-03-01 00:00:00.000000000 Z
+date: 2016-03-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -163,8 +163,10 @@ files:
 - bin/console
 - bin/setup
 - circle.yml
+- examples/firehose-producer.rb
 - examples/simple-consumer.rb
 - examples/simple-producer.rb
+- examples/ssl-producer.rb
 - lib/kafka.rb
 - lib/kafka/async_producer.rb
 - lib/kafka/broker.rb
@@ -181,6 +183,7 @@ files:
 - lib/kafka/gzip_codec.rb
 - lib/kafka/instrumentation.rb
 - lib/kafka/message_buffer.rb
+- lib/kafka/offset_manager.rb
 - lib/kafka/partitioner.rb
 - lib/kafka/pending_message.rb
 - lib/kafka/pending_message_queue.rb
@@ -219,6 +222,7 @@ files:
 - lib/kafka/round_robin_assignment_strategy.rb
 - lib/kafka/snappy_codec.rb
 - lib/kafka/socket_with_timeout.rb
+- lib/kafka/ssl_socket_with_timeout.rb
 - lib/kafka/version.rb
 - lib/ruby-kafka.rb
 - ruby-kafka.gemspec