racecar 0.3.0 → 0.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: c0651e97f7403aacaf9494cb2cc2833b7b6040a5
-  data.tar.gz: d5849ff77aec0112c92d56937cb4ab30d926dc16
+  metadata.gz: 712a502336151d29108a93f1f00d2eb3b8b0b072
+  data.tar.gz: 9cd9f1fe7b345f1dc9aaf8ceba6751f7e848df98
 SHA512:
-  metadata.gz: 5781c4b863eefed5f336c5cfd5f7832ed8d6c248f5064eb7a1729222676c0ed88663dc7db1ca9f7d1adda5dc470317fdd78ea3c15246ab4d4eb23d6480f57977
-  data.tar.gz: bcb8e7d8b5d1aed13c68d7257381d0d710979dde5dbcc34ba9e7649ce53e3e11031deaa410fc9b046f5914690eec41d114ff7ee38a16ee554f96f4259b983524
+  metadata.gz: c468794299f01ab66e9727bca985cc334cbcf0c200ec89260f6e3eca55a6325823ab3741c92d0d43f119abea99f3111a534b081da6dd7ae0aaea483b7c56723f
+  data.tar.gz: aaff7d20a5da77ea2b447fa4ed554c3a1230eafca78db0fecfee03c9cd11e37d7a0f0cb6bbf566f3343bfe31c07c643c9b248f0943b3eb2829d34ba36a2bb3ec

data/README.md

@@ -2,7 +2,7 @@
 
 Introducing Racecar, your friendly and easy-to-approach Kafka consumer framework!
 
-Using [ruby-kafka](https://github.com/zendesk/ruby-kafka) directly can be a challenge: it's a flexible library with lots of knobs and options. Most users don't need that level of flexibility, though. Racecar provides a simple and intuitive way to build and configure Kafka consumers that optionally integrates seemlessly with Rails.
+Using [ruby-kafka](https://github.com/zendesk/ruby-kafka) directly can be a challenge: it's a flexible library with lots of knobs and options. Most users don't need that level of flexibility, though. Racecar provides a simple and intuitive way to build and configure Kafka consumers that optionally integrates seamlessly with Rails.
 
 ## Table of content
 
@@ -13,8 +13,9 @@ Using [ruby-kafka](https://github.com/zendesk/ruby-kafka) directly can be a chal
     3. [Configuration](#configuration)
     4. [Testing consumers](#testing-consumers)
     5. [Deploying consumers](#deploying-consumers)
-    6. [Logging](#logging)
-    7. [Operations](#operations)
+    6. [Handling errors](#handling-errors)
+    7. [Logging](#logging)
+    8. [Operations](#operations)
 3. [Development](#development)
 4. [Contributing](#contributing)
 5. [Copyright and license](#copyright-and-license)
@@ -86,14 +87,14 @@ You can optionally add an `initialize` method if you need to do any set-up work
 ```ruby
 class PushNotificationConsumer < Racecar::Consumer
   subscribes_to "notifications"
-  
+
   def initialize
     @push_service = PushService.new # pretend this exists.
   end
-  
+
   def process(message)
     data = JSON.parse(message.value)
-    
+
     @push_service.notify!(
       recipient: data.fetch("recipient"),
       notification: data.fetch("notification"),
@@ -142,6 +143,28 @@ end
 
 An important detail is that, if an exception is raised while processing a batch, the _whole batch_ is re-processed.
 
+#### Tearing down resources when stopping
+
+When a Racecar consumer shuts down, it gets the opportunity to tear down any resources held by the consumer instance. For example, it may make sense to close any open files or network connections. Doing so is simple: just implement a `#teardown` method in your consumer class and it will be called during the shutdown procedure.
+
+```ruby
+class ArchiveConsumer < Racecar::Consumer
+  subscribes_to "events"
+
+  def initialize
+    @file = File.open("archive", "a")
+  end
+
+  def process(message)
+    @file << message.value
+  end
+
+  def teardown
+    @file.close
+  end
+end
+```
+
 ### Running consumers
 
 Racecar is first and foremost an executable _consumer runner_. The `racecar` executable takes as argument the name of the consumer class that should be run. Racecar automatically loads your Rails application before starting, and you can load any other library you need by passing the `--require` flag, e.g.
@@ -176,8 +199,9 @@ The consumers will checkpoint their positions from time to time in order to be a
 
 All timeouts are defined in number of seconds.
 
+* `session_timeout` – The idle timeout after which a consumer is kicked out of the group. Consumers must send heartbeats with at least this frequency.
 * `heartbeat_interval` – How often to send a heartbeat message to Kafka.
-* `pause_timeout` – How long to pause a partition for if the consumer raises an exception while processing a message.
+* `pause_timeout` – How long to pause a partition for if the consumer raises an exception while processing a message. Default is to pause for 10 seconds. Set this to zero in order to disable automatic pausing of partitions.
 * `connect_timeout` – How long to wait when trying to connect to a Kafka broker. Default is 10 seconds.
 * `socket_timeout` – How long to wait when trying to communicate with a Kafka broker. Default is 30 seconds.
 * `max_wait_time` – How long to allow the Kafka brokers to wait before returning messages. A higher number means larger batches, at the cost of higher latency. Default is 5 seconds.
@@ -218,10 +242,10 @@ Here's an example of testing a consumer class using [RSpec](http://rspec.info/)
 # `email-addresses` topic.
 class CreateContactsConsumer < Racecar::Consumer
   subscribes_to "email-addresses"
-  
+
   def process(message)
     email = message.value
-    
+
     Contact.create!(email: email)
   end
 end
@@ -231,9 +255,9 @@ describe CreateContactsConsumer do
   it "creates a Contact for each email address in the topic" do
     message = double("message", value: "john@example.com")
     consumer = CreateContactsConsumer.new
-    
+
     consumer.process(message)
-    
+
     expect(Contact.where(email: "john@example.com")).to exist
   end
 end
@@ -256,6 +280,44 @@ If you've ever used Heroku you'll recognize the format – indeed, deploying to
 With Foreman, you can easily run these processes locally by executing `foreman run`; in production you'll want to _export_ to another process management format such as Upstart or Runit. [capistrano-foreman](https://github.com/hyperoslo/capistrano-foreman) allows you to do this with Capistrano.
 
 
+### Handling errors
+
+When processing messages from a Kafka topic, your code may encounter an error and raise an exception. The cause is typically on of two things:
+
+1. The message being processed is somehow malformed or doesn't conform with the assumptions made by the processing code.
+2. You're using some external resource such as a database or a network API that is temporarily unavailable.
+
+In the first case, you'll need to either skip the message or deploy a new version of your consumer that can correctly handle the message that caused the error. In order to skip a message, handle the relevant exception in your `#process` method:
+
+```ruby
+def process(message)
+  data = JSON.parse(message.value)
+  # ...
+rescue JSON::ParserError => e
+  puts "Failed to process message in #{message.topic}/#{message.partition} at offset #{message.offset}: #{e}"
+  # It's probably a good idea to report the exception to an exception tracker service.
+end
+```
+
+Since the exception is handled by your `#process` method and is no longer raised, Racecar will consider the message successfully processed. Tracking these errors in an exception tracker or some other monitoring system is highly recommended, as you otherwise will have little insight into how many messages are being skipped this way.
+
+If, on the other hand, the exception was cause by a temporary network or database problem, you will probably want to retry processing of the message after some time has passed. By default, if an exception is raised by the `#process` method, the consumer will pause all processing of the message's partition for some number of seconds, configured by setting the `pause_timeout` configuration variable. This allows the consumer to continue processing messages from other partitions that may not be impacted by the problem while still making sure to not drop the original message. Since messages in a single Kafka topic partition _must_ be processed in order, it's not possible to keep processing _other_ messages in that partition.
+
+In addition to retrying the processing of messages, Racecar also allows defining an _error handler_ callback that is invoked whenever an exception is raised by your `#process` method. This allows you to track and report errors to a monitoring system:
+
+```ruby
+Racecar.config.on_error do |exception, info|
+  MyErrorTracker.report(exception, {
+    topic: info[:topic],
+    partition: info[:partition],
+    offset: info[:offset],
+  })
+end
+```
+
+It is highly recommended that you set up an error handler.
+
+
 ### Logging
 
 By default, Racecar will log to `STDOUT`. If you're using Rails, your application code will use whatever logger you've configured there.

data/lib/racecar/config.rb

@@ -11,6 +11,7 @@ module Racecar
       offset_commit_interval
       offset_commit_threshold
 
+      session_timeout
       heartbeat_interval
       pause_timeout
       connect_timeout
@@ -55,8 +56,11 @@ module Racecar
       # Default is to send a heartbeat every 10 seconds.
       heartbeat_interval: 10,
 
-      # Default is to not pause partitions on processing errors.
-      pause_timeout: 0,
+      # Default is to pause partitions for 10 seconds on processing errors.
+      pause_timeout: 10,
+
+      # Default is to kick consumers out of a group after 30 seconds without activity.
+      session_timeout: 30,
 
       # Default is to allow at most 10 seconds when connecting to a broker.
       connect_timeout: 10,

data/lib/racecar/consumer.rb

@@ -22,5 +22,7 @@ module Racecar
         end
       end
     end
+
+    def teardown; end
   end
 end

data/lib/racecar/runner.rb

@@ -2,12 +2,17 @@ require "kafka"
 
 module Racecar
   class Runner
-    attr_reader :processor, :config, :logger
+    attr_reader :processor, :config, :logger, :consumer
 
     def initialize(processor, config:, logger:)
       @processor, @config, @logger = processor, config, logger
     end
 
+    def stop
+      processor.teardown
+      consumer.stop unless consumer.nil?
+    end
+
     def run
       kafka = Kafka.new(
         client_id: config.client_id,
@@ -20,17 +25,18 @@ module Racecar
         ssl_client_cert_key: config.ssl_client_cert_key,
       )
 
-      consumer = kafka.consumer(
+      @consumer = kafka.consumer(
         group_id: config.group_id,
         offset_commit_interval: config.offset_commit_interval,
         offset_commit_threshold: config.offset_commit_threshold,
+        session_timeout: config.session_timeout,
         heartbeat_interval: config.heartbeat_interval,
       )
 
       # Stop the consumer on SIGINT, SIGQUIT or SIGTERM.
-      trap("QUIT") { consumer.stop }
-      trap("INT") { consumer.stop }
-      trap("TERM") { consumer.stop }
+      trap("QUIT") { stop }
+      trap("INT") { stop }
+      trap("TERM") { stop }
 
       # Print the consumer config to STDERR on USR1.
       trap("USR1") { $stderr.puts config.inspect }

data/lib/racecar/version.rb

@@ -1,3 +1,3 @@
 module Racecar
-  VERSION = "0.3.0"
+  VERSION = "0.3.1"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: racecar
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.3.1
 platform: ruby
 authors:
 - Daniel Schierbeck
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2017-08-16 00:00:00.000000000 Z
+date: 2017-08-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: ruby-kafka