racecar 0.1.3 → 0.1.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 251134e0054c81aba394c44e94cf126005fa8129
-  data.tar.gz: 82224a191890588b7b8841e98cb7a6f8f7d6195c
+  metadata.gz: 92c4f52171961fdbf4dcf6db7ed265eec6a729dd
+  data.tar.gz: 8bb0fc336b839f91da037cc7046dc64fa699acdd
 SHA512:
-  metadata.gz: dce61cd9bc947e1a18a95c305b64396c0da9f59cdfa742b9ab2ee122f0faa5eaaec423c42490015351f115bae662c3f0c6668cf343d6ceefbc8bbf7742536859
-  data.tar.gz: 69929c9148dda6f2db851bebdd4fa16de065205aa2d36610f9747f0a5e34df94e9fc7fca48938384f82099644256c751349172afa9e355f386b51f96eaa8d905
+  metadata.gz: 180c8bc2e058b7626b1f2d3f277d176c12b75b27191222ae40f80cb893142037132b8f469c5d693482be317e554453019b09c2ff941bbf5fc88665825dc9c2b4
+  data.tar.gz: d19f91361e5d5a59914c74fa89e4d574792dbeb0e6c27cc1ddc153bd463dc23498ed8b0d9d76e63d1b525058af0eaad16cc75aab58f3926c6a825bb3fb3ca1c8

data/README.md

@@ -2,7 +2,7 @@
 
 Introducing Racecar, your friendly and easy-to-approach Kafka consumer framework!
 
-Using [ruby-kafka](https://github.com/zendesk/ruby-kafka) directly can be a challenge: it's a flexible library with lots of knobs and options. Most users don't need that level of flexibility, though.
+Using [ruby-kafka](https://github.com/zendesk/ruby-kafka) directly can be a challenge: it's a flexible library with lots of knobs and options. Most users don't need that level of flexibility, though. Racecar provides a simple and intuitive way to build and configure Kafka consumers that optionally integrates seemlessly with Rails.
 
 ## Installation
 
@@ -20,9 +20,25 @@ Or install it yourself as:
 
     $ gem install racecar
 
+Then execute (if you're in a Rails application):
+
+    $ bundle exec rails generate racecar:install
+
+This will add a config file in `config/racecar.yml`.
+
 ## Usage
 
-Add a file in e.g. `app/consumers/user_ban_consumer.rb`:
+Racecar is built for simplicity of development and operation. If you need more flexibility, it's quite straightforward to build your own Kafka consumer executables using [ruby-kafka](https://github.com/zendesk/ruby-kafka#consuming-messages-from-kafka) directly.
+
+First, a short introduction to the Kafka consumer concept as well as some basic background on Kafka.
+
+Kafka stores messages in so-called _partitions_ which are grouped into _topics_. Within a partition, each message gets a unique offset.
+
+In Kafka, _consumer groups_ are sets of processes that collaboratively process messages from one or more Kafka topics; they divide up the topic partitions amongst themselves and make sure to reassign the partitions held by any member of the group that happens to crash or otherwise becomes unavailable, thus minimizing the risk of disruption. A consumer in a group is responsible for keeping track of which messages in a partition it has processed – since messages are processed in-order within a single partition, this means keeping track of the _offset_ into the partition that has been processed. Consumers periodically _commit_ these offsets to the Kafka brokers, making sure that another consumer can resume from those positions if there is a crash.
+
+### Creating consumers
+
+A Racecar consumer is a simple Rails class that inherits from `Racecar::Consumer`:
 
 ```ruby
 class UserBanConsumer < Racecar::Consumer
@@ -37,9 +53,75 @@ class UserBanConsumer < Racecar::Consumer
 end
 ```
 
-Now run your consumer with `bundle exec racecar UserBanConsumer`.
+In order to create your own consumer, run the Rails generator `racecar:consumer`:
+
+    $ bundle exec rails generate racecar:consumer TapDance
+
+This will create a file at `app/consumers/tap_dance_consumer.rb` which you can modify to your liking. Add one or more calls to  `subscribes_to` in order to have the consumer subscribe to Kafka topics.
+
+Now run your consumer with `bundle exec racecar TapDanceConsumer`.
+
+Note: if you're not using Rails, you'll have to add the file yourself. No-one will judge you for copy-pasting it.
+
+#### Initializing consumers
+
+You can optionally add an `initialize` method if you need to do any set-up work before processing messages, e.g.
+
+```ruby
+class PushNotificationConsumer < Racecar::Consumer
+  subscribes_to "notifications"
+  
+  def initialize
+    @push_service = PushService.new # pretend this exists.
+  end
+  
+  def process(message)
+    data = JSON.parse(message.value)
+    
+    @push_service.notify!(
+      recipient: data.fetch("recipient"),
+      notification: data.fetch("notification"),
+    )
+  end
+end
+```
+
+This is useful to do any one-off work that you wouldn't want to do for each and every message.
+
+#### Setting the starting position
+
+When a consumer is started for the first time, it needs to decide where in each partition to start. By default, it will start at the _beginning_, meaning that all past messages will be processed. If you want to instead start at the _end_ of each partition, change your `subscribes_to` like this:
+
+```ruby
+subscribes_to "some-topic", start_from_beginning: false
+```
+
+Note that once the consumer has started, it will commit the offsets it has processed until and in the future will resume from those.
+
+### Running consumers
+
+Racecar is first and foremost an executable _consumer runner_. The `racecar` executable takes as argument the name of the consumer class that should be run. Racecar automatically loads your Rails application before starting, and you can load any other library you need by passing the `--require` flag, e.g.
+
+    $ bundle exec racecar --require dance_moves TapDanceConsumer
+
+### Configuration
+
+Racecar provides a flexible way to configure your consumer in a way that feels at home in a Rails application. If you haven't already, run `bundle exec rails generate racecar:install` in order to generate a config file. You'll get a separate section for each Rails environment, with the common configuration values in a shared `common` section.
+
+The possible configuration keys are:
+
+* `brokers` (_optional_) – A list of Kafka brokers in the cluster that you're consuming from. Defaults to `localhost` on port 9092, the default Kafka port.
+* `client_id` (_optional_) – A string used to identify the client in logs and metrics.
+* `group_id_prefix` (_optional_) – A prefix used when generating consumer group names. For instance, if you set the prefix to be `kevin.` and your consumer class is named `BaconConsumer`, the resulting consumer group will be named `kevin.bacon_consumer`.
+* `offset_commit_interval` (_optional_) – How often to save the consumer's position in Kafka.
+* `heartbeat_interval` (_optional_) – How often to send a heartbeat message to Kafka.
+* `pause_timeout` (_optional_) – How long to pause a partition for if the consumer raises an exception while processing a message.
+* `connect_timeout` (_optional_) – How long to wait when trying to connect to a Kafka broker.
+* `socket_timeout` (_optional_) – How long to wait when trying to communicate with a Kafka broker.
+
+Note that many of these configuration keys correspond directly with similarly named concepts in [ruby-kafka](https://github.com/zendesk/ruby-kafka) for more details on low-level operations, read that project's documentation.
 
-That's all there is to it.
+It's also possible to configure Racecar using environment variables. For any given configuration key, there should be a corresponding environment variable with the prefix `RACECAR_`, in upper case. For instance, in order to configure the client id, set `RACECAR_CLIENT_ID=some-id` in the process in which the Racecar consumer is launched. You can set `brokers` by passing a comma-separated list, e.g. `RACECAR_BROKERS=kafka1:9092,kafka2:9092,kafka3:9092`.
 
 ## Development
 

data/exe/racecar

@@ -1,58 +1,6 @@
 #!/usr/bin/env ruby
 
-require "optparse"
 require "racecar"
+require "racecar/cli"
 
-parser = OptionParser.new do |opts|
-  opts.banner = "Usage: racecar MyConsumer [options]"
-
-  opts.on("-r", "--require LIBRARY", "Require the LIBRARY before starting the consumer") do |lib|
-    require lib
-  end
-
-  opts.on_tail("--version", "Show Racecar version") do
-    require "racecar/version"
-    puts "Racecar #{Racecar::VERSION}"
-    exit
-  end
-end
-
-parser.parse!(ARGV)
-
-consumer_name = ARGV.first or raise "No consumer specified"
-config_file = "config/racecar.yml"
-
-puts "=> Starting Racecar consumer #{consumer_name}..."
-
-puts "=> Booting Rails application..."
-
-require "./config/environment"
-
-Racecar.config.load_file(config_file, Rails.env)
-
-if Racecar.config.log_to_stdout
-  # Write to STDOUT as well as to the log file.
-  console = ActiveSupport::Logger.new($stdout)
-  console.formatter = Rails.logger.formatter
-  console.level = Rails.logger.level
-  Rails.logger.extend(ActiveSupport::Logger.broadcast(console))
-end
-
-Racecar.logger = Rails.logger
-
-# Find the consumer class by name.
-consumer_class = Kernel.const_get(consumer_name)
-
-# Load config defined by the consumer class itself.
-Racecar.config.load_consumer_class(consumer_class)
-
-Racecar.config.validate!
-
-puts "=> Wrooooom!"
-puts "=> Ctrl-C to shutdown consumer"
-
-processor = consumer_class.new
-
-Racecar.run(processor)
-
-puts "=> Shut down"
+Racecar::Cli.main(ARGV)

data/lib/generators/racecar/consumer_generator.rb

@@ -0,0 +1,11 @@
+module Racecar
+  module Generators
+    class ConsumerGenerator < Rails::Generators::NamedBase
+      source_root File.expand_path("../../templates", __FILE__)
+
+      def create_consumer_file
+        template "consumer.rb.erb", "app/consumers/#{file_name}_consumer.rb"
+      end
+    end
+  end
+end

data/lib/generators/racecar/install_generator.rb

@@ -0,0 +1,15 @@
+module Racecar
+  module Generators
+    class InstallGenerator < Rails::Generators::Base
+      source_root File.expand_path("../../templates", __FILE__)
+
+      def create_config_file
+        template "racecar.yml.erb", "config/racecar.yml"
+      end
+
+      def create_consumers_directory
+        empty_directory "app/consumers"
+      end
+    end
+  end
+end

data/lib/generators/templates/consumer.rb.erb

@@ -0,0 +1,7 @@
+class <%= class_name %>Consumer < Racecar::Consumer
+  subscribes_to "some-topic"
+
+  def process(message)
+    puts "Received message: #{message.value}"
+  end
+end

data/lib/generators/templates/racecar.yml.erb

@@ -0,0 +1,20 @@
+# These config values will be shared by all environments but can be overridden.
+common: &common
+  client_id: "<%= Rails.application.class.name.split("::").first.underscore %>"
+
+development:
+  <<: *common
+  brokers:
+    - localhost:9092
+
+test:
+  <<: *common
+  brokers:
+    - localhost:9092
+
+production:
+  <<: *common
+  brokers:
+    - kafka1.myapp.com:9092
+    - kafka2.myapp.com:9092
+    - kafka3.myapp.com:9092

data/lib/racecar.rb

@@ -5,6 +5,12 @@ require "racecar/runner"
 require "racecar/config"
 
 module Racecar
+  class Error < StandardError
+  end
+
+  class ConfigError < Error
+  end
+
   def self.config
     @config ||= Config.new
   end

data/lib/racecar/cli.rb

@@ -0,0 +1,67 @@
+require "optparse"
+
+module Racecar
+  module Cli
+    def self.main(args)
+      parser = OptionParser.new do |opts|
+        opts.banner = "Usage: racecar MyConsumer [options]"
+
+        opts.on("-r", "--require LIBRARY", "Require the LIBRARY before starting the consumer") do |lib|
+          require lib
+        end
+
+        opts.on_tail("--version", "Show Racecar version") do
+          require "racecar/version"
+          puts "Racecar #{Racecar::VERSION}"
+          exit
+        end
+      end
+
+      parser.parse!(args)
+
+      consumer_name = args.first or raise Racecar::Error, "no consumer specified"
+      config_file = "config/racecar.yml"
+
+      puts "=> Starting Racecar consumer #{consumer_name}..."
+
+      begin
+        require "rails"
+
+        puts "=> Detected Rails, booting application..."
+
+        require "./config/environment"
+
+        Racecar.config.load_file(config_file, Rails.env)
+
+        if Racecar.config.log_to_stdout
+          # Write to STDOUT as well as to the log file.
+          console = ActiveSupport::Logger.new($stdout)
+          console.formatter = Rails.logger.formatter
+          console.level = Rails.logger.level
+          Rails.logger.extend(ActiveSupport::Logger.broadcast(console))
+        end
+
+        Racecar.logger = Rails.logger
+      rescue LoadError
+        # Not a Rails application.
+      end
+
+      # Find the consumer class by name.
+      consumer_class = Kernel.const_get(consumer_name)
+
+      # Load config defined by the consumer class itself.
+      Racecar.config.load_consumer_class(consumer_class)
+
+      Racecar.config.validate!
+
+      puts "=> Wrooooom!"
+      puts "=> Ctrl-C to shutdown consumer"
+
+      processor = consumer_class.new
+
+      Racecar.run(processor)
+
+      puts "=> Shut down"
+    end
+  end
+end

data/lib/racecar/config.rb

@@ -1,5 +1,6 @@
 require "erb"
 require "yaml"
+require "racecar/env_loader"
 
 module Racecar
   class Config
@@ -22,9 +23,11 @@ module Racecar
 
     REQUIRED_KEYS = %w(
       brokers
+      client_id
     )
 
     DEFAULT_CONFIG = {
+      brokers: ["localhost:9092"],
       client_id: "racecar",
       group_id_prefix: nil,
 
@@ -59,7 +62,7 @@ module Racecar
     def validate!
       REQUIRED_KEYS.each do |key|
         if send(key).nil?
-          raise "required configuration key `#{key}` not defined"
+          raise ConfigError, "required configuration key `#{key}` not defined"
         end
       end
     end
@@ -77,13 +80,17 @@ module Racecar
       load(data)
     end
 
+    def set(key, value)
+      unless ALLOWED_KEYS.include?(key.to_s)
+        raise ConfigError, "unknown configuration key `#{key}`"
+      end
+
+      instance_variable_set("@#{key}", value)
+    end
+
     def load(data)
       data.each do |key, value|
-        unless ALLOWED_KEYS.include?(key.to_s)
-          raise "unknown configuration key `#{key}`"
-        end
-
-        instance_variable_set("@#{key}", value)
+        set(key, value)
       end
     end
 
@@ -109,9 +116,19 @@ module Racecar
     private
 
     def load_env!
-      if ENV.key?("RACECAR_BROKERS")
-        @brokers = ENV["RACECAR_BROKERS"].split(",")
-      end
+      loader = EnvLoader.new(ENV, self)
+
+      loader.string_list(:brokers)
+      loader.string(:client_id)
+      loader.string(:group_id_prefix)
+      loader.string(:group_id)
+      loader.integer(:offset_commit_interval)
+      loader.integer(:offset_commit_threshold)
+      loader.integer(:heartbeat_interval)
+      loader.integer(:pause_timeout)
+      loader.integer(:connect_timeout)
+      loader.integer(:socket_timeout)
+      loader.integer(:max_wait_time)
     end
   end
 end

data/lib/racecar/env_loader.rb

@@ -0,0 +1,37 @@
+module Racecar
+  class EnvLoader
+    def initialize(env, config)
+      @env = env
+      @config = config
+    end
+
+    def string(name)
+      set(name) {|value| value }
+    end
+
+    def integer(name)
+      set(name) do |value|
+        begin
+          Integer(value)
+        rescue ArgumentError
+          raise ConfigError, "#{value.inspect} is not an integer"
+        end
+      end
+    end
+
+    def string_list(name)
+      set(name) {|value| value.split(",") }
+    end
+
+    private
+
+    def set(name)
+      key = "RACECAR_#{name.upcase}"
+
+      if @env.key?(key)
+        value = yield @env.fetch(key)
+        @config.set(name, value)
+      end
+    end
+  end
+end

data/lib/racecar/version.rb

@@ -1,3 +1,3 @@
 module Racecar
-  VERSION = "0.1.3"
+  VERSION = "0.1.4"
 end

data/racecar.gemspec

@@ -20,6 +20,8 @@ Gem::Specification.new do |spec|
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ["lib"]
 
+  spec.add_runtime_dependency "ruby-kafka", "~> 0.3"
+
   spec.add_development_dependency "bundler", "~> 1.13"
   spec.add_development_dependency "rake", "~> 10.0"
   spec.add_development_dependency "rspec", "~> 3.0"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: racecar
 version: !ruby/object:Gem::Version
-  version: 0.1.3
+  version: 0.1.4
 platform: ruby
 authors:
 - Daniel Schierbeck
@@ -9,8 +9,22 @@ authors:
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2017-06-14 00:00:00.000000000 Z
+date: 2017-06-21 00:00:00.000000000 Z
 dependencies:
+- !ruby/object:Gem::Dependency
+  name: ruby-kafka
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.3'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.3'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
@@ -64,7 +78,6 @@ extra_rdoc_files: []
 files:
 - ".gitignore"
 - ".rspec"
-- ".travis.yml"
 - Gemfile
 - LICENSE.txt
 - README.md
@@ -73,9 +86,15 @@ files:
 - bin/setup
 - examples/cat_consumer.rb
 - exe/racecar
+- lib/generators/racecar/consumer_generator.rb
+- lib/generators/racecar/install_generator.rb
+- lib/generators/templates/consumer.rb.erb
+- lib/generators/templates/racecar.yml.erb
 - lib/racecar.rb
+- lib/racecar/cli.rb
 - lib/racecar/config.rb
 - lib/racecar/consumer.rb
+- lib/racecar/env_loader.rb
 - lib/racecar/runner.rb
 - lib/racecar/version.rb
 - racecar.gemspec

data/.travis.yml

@@ -1,5 +0,0 @@
-sudo: false
-language: ruby
-rvm:
-  - 2.2.3
-before_install: gem install bundler -v 1.13.6