phobos 1.4.2 → 1.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 1a07ed8172ccb8566701408cdcd049d3bc0802b4
-  data.tar.gz: a469b26a35c71c25b1317392365b87e938944087
+  metadata.gz: 576b1c85a5d3f389caac3e09a2c1738bb2743eda
+  data.tar.gz: 5fc77dbf88b6bfc824245cb58c9a34558562d379
 SHA512:
-  metadata.gz: 13915bc70544e1d24d2eb210e9615226272051a32b4c542ed93864b38a4315fbaaa4f8f6cd00bf88e70ce94feaed4db3ec4d4d8d16b4846b8973a4c6ce457dcc
-  data.tar.gz: 5d032d5fa7e24a03611212c08201042f9570d631551a580922ada7d7c91a120e7125e616b98ad429aa04f3cf32a76d336c286ac7265187b9bbb681666f2dadab
+  metadata.gz: 74ff72c5bd479ee5e495d8da7ccc97c2bbe998c5ef7d601ce245a1b5e0a0054c5327c1cb100798dfb0832956fe91c8611f2aa66735d3af90e06b8dc8a915d713
+  data.tar.gz: 9dbf97945d5bb802fec64af350af7749273e30b9fd000e1ce435b1bfb2780b5f2dfc99d03c3a4cb3eaa1fa8e8a53b3e16ea75431c5057381f7d0993b22cb04b6

data/.ruby-version

@@ -1 +1 @@
-2.3.0
+2.4.1

data/CHANGELOG.md

@@ -4,6 +4,11 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/)
 and this project adheres to [Semantic Versioning](http://semver.org/).
 
+## 1.5.0 (2017-10-25)
+
+- [enhancement] Add `before_consume` callback to support single point of decoding a message.
+- [enhancement] Add module `Phobos::Test::Helper` for testing, to test consumers with minimal setup required
+
 ## 1.4.2 (2017-09-29)
 
 - [bugfix] Async publishing always delivers messages #33

data/README.md

@@ -29,6 +29,7 @@ With Phobos by your side, all this becomes smooth sailing.
   1. [Instrumentation](#usage-instrumentation)
 1. [Plugins](#plugins)
 1. [Development](#development)
+1. [Test](#test)
 
 ## <a name="installation"></a> Installation
 
@@ -120,7 +121,7 @@ $ phobos start -c /var/configs/my.yml -b /opt/apps/boot.rb
 
 Messages from Kafka are consumed using __handlers__. You can use Phobos __executors__ or include it in your own project [as a library](#usage-as-library), but __handlers__ will always be used. To create a handler class, simply include the module `Phobos::Handler`. This module allows Phobos to manage the life cycle of your handler.
 
-A handler must implement the method `#consume(payload, metadata)`.
+A handler is required to implement the method `#consume(payload, metadata)`.
 
 Instances of your handler will be created for every message, so keep a constructor without arguments. If `consume` raises an exception, Phobos will retry the message indefinitely, applying the back off configuration presented in the configuration file. The `metadata` hash will contain a key called `retry_count` with the current number of retries for this message. To skip a message, simply return from `#consume`.
 
@@ -162,6 +163,19 @@ class MyHandler
 end
 ```
 
+Finally, it is also possible to preprocess the message payload before consuming it using the `before_consume` hook which is invoked before `.around_consume` and `#consume`. The result of this operation will be assigned to payload, so it is important to return the modified payload. This can be very useful, for example if you want a single point of decoding Avro messages and want the payload as a hash instead of a binary.
+
+```ruby
+class MyHandler
+  include Phobos::Handler
+
+  def before_consume(payload)
+    # optionally preprocess payload
+    payload
+  end
+end
+```
+
 Take a look at the examples folder for some ideas.
 
 The hander life cycle can be illustrated as:
@@ -170,7 +184,7 @@ The hander life cycle can be illustrated as:
 
 or optionally,
 
-  `.start` -> `.around_consume` [ `#consume` ] -> `.stop`
+  `.start` -> `#before_consume` -> `.around_consume` [ `#consume` ] -> `.stop`
 
 ### <a name="usage-producing-messages-to-kafka"></a> Producing messages to Kafka
 
@@ -303,7 +317,7 @@ __producer__ provides configurations for all producers created over the applicat
 
 __consumer__ provides configurations for all consumer groups created over the application. All [options supported by  `ruby-kafka`][ruby-kafka-consumer] can be provided.
 
-__backoff__ Phobos provides automatic retries for your handlers, if an exception is raised the listener will retry following the back off configured here
+__backoff__ Phobos provides automatic retries for your handlers, if an exception is raised the listener will retry following the back off configured here. Backoff can also be configured per listener.
 
 __listeners__ is the list of listeners configured, each listener represents a consumers group
 
@@ -403,20 +417,37 @@ List of gems that enhance Phobos:
 ## <a name="development"></a> Development
 
 After checking out the repo:
-* make sure docker is installed and running
+* make sure `docker` is installed and running (for windows and mac this also includes `docker-compose`).
+* Linux: make sure `docker-compose` is installed and running.
 * run `bin/setup` to install dependencies
-* run `sh utils/start-all.sh` to start the required kafka containers in the background
-* run `rspec` to run the tests
+* run `docker-compose up` to start the required kafka containers in a window
+* run `rspec` to run the tests in another window
 
 You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
 To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
-The `utils` folder contain some shell scripts to help with the local Kafka cluster. It uses docker to start Kafka and zookeeper.
+## <a name="test"></a> Test
 
-```sh
-sh utils/start-all.sh
-sh utils/stop-all.sh
+Phobos exports a spec helper that can help you test your consumer. The Phobos lifecycle will conveniently be activated for you with minimal setup required.
+
+* `process_message(handler:, payload:, metadata:, encoding: nil)` - Invokes your handler with payload and metadata, using a dummy listener (encoding is optional).
+
+```ruby
+require 'spec_helper'
+
+describe MyConsumer do
+  let(:payload) { 'foo' }
+  let(:metadata) { 'foo' }
+
+  it 'consumes my message' do
+    expect(described_class).to receive(:around_consume).with(payload, metadata).once.and_call_original
+    expect_any_instance_of(described_class).to receive(:before_consume).with(payload).once.and_call_original
+    expect_any_instance_of(described_class).to receive(:consume).with(payload, metadata).once.and_call_original
+
+    process_message(handler: described_class, payload: payload, metadata: metadata)
+  end
+end
 ```
 
 ## Contributing

data/bin/console

@@ -1,17 +1,17 @@
 #!/usr/bin/env ruby
 
-require "bundler/setup"
-require "phobos"
+require 'irb'
+require 'bundler/setup'
+require 'phobos'
 
 # You can add fixtures and/or initialization code here to make experimenting
 # with your gem easier. You can also use a different console, if you like.
 
 # (If you use this, don't forget to add pry to your Gemfile!)
-# require "pry"
+# require 'pry'
 # Pry.start
 
 config_path = ENV['CONFIG_PATH'] || (File.exist?('config/phobos.yml') ? 'config/phobos.yml' : 'config/phobos.yml.example')
 Phobos.configure(config_path)
 
-require "irb"
 IRB.start

data/circle.yml

@@ -8,20 +8,20 @@ machine:
     CI: true
     DEFAULT_TIMEOUT: 20
   ruby:
-    version: 2.3.1
+    version: 2.4.1
 
 dependencies:
   pre:
     - docker -v
-    - docker pull ches/kafka:0.9.0.1
-    - docker pull jplock/zookeeper:3.4.6
-    - gem install bundler -v 1.9.5
+    - docker pull ches/kafka:0.10.2.1
+    - docker pull jplock/zookeeper:3.4.10
+    - gem install bundler -v 1.13.2
     - bundle install
 
 test:
   override:
-    - docker run -d -p 2003:2181 --name zookeeper jplock/zookeeper:3.4.6; sleep 5
-    - docker run -d -p 9092:9092 --name kafka -e KAFKA_BROKER_ID=0 -e KAFKA_ADVERTISED_HOST_NAME=localhost -e KAFKA_ADVERTISED_PORT=9092 -e ZOOKEEPER_CONNECTION_STRING=zookeeper:2181 --link zookeeper:zookeeper ches/kafka:0.9.0.1; sleep 5
+    - docker run -d -p 2003:2181 --name zookeeper jplock/zookeeper:3.4.10; sleep 5
+    - docker run -d -p 9092:9092 --name kafka -e KAFKA_BROKER_ID=0 -e KAFKA_ADVERTISED_HOST_NAME=localhost -e KAFKA_ADVERTISED_PORT=9092 -e ZOOKEEPER_CONNECTION_STRING=zookeeper:2181 --link zookeeper:zookeeper ches/kafka:0.10.2.1; sleep 5
     - bundle exec rspec -r rspec_junit_formatter --format RspecJunitFormatter -o $CIRCLE_TEST_REPORTS/rspec/unit.xml
   post:
     - cp log/*.log $CIRCLE_ARTIFACTS/ || true

data/config/phobos.yml.example

@@ -95,3 +95,7 @@ listeners:
     # Apply this encoding to the message payload, if blank it uses the original encoding. This property accepts values
     # defined by the ruby Encoding class (https://ruby-doc.org/core-2.3.0/Encoding.html). Ex: UTF_8, ASCII_8BIT, etc
     force_encoding:
+    # Use this if custom backoff is required for a listener
+    backoff:
+      min_ms: 500
+      max_ms: 10000

data/docker-compose.yml

@@ -0,0 +1,18 @@
+version: '2'
+services:
+  zookeeper:
+    image: jplock/zookeeper:3.4.10
+    ports:
+     - 2181:2181
+
+  kafka:
+    depends_on:
+      - zookeeper
+    image: ches/kafka:0.10.2.1
+    ports:
+      - 9092:9092
+    environment:
+      - KAFKA_BROKER_ID=0
+      - KAFKA_ADVERTISED_HOST_NAME=localhost
+      - KAFKA_ADVERTISED_PORT=9092
+      - ZOOKEEPER_CONNECTION_STRING=zookeeper:2181

data/lib/phobos.rb

@@ -9,12 +9,15 @@ require 'concurrent'
 require 'exponential_backoff'
 require 'kafka'
 require 'logging'
+require 'erb'
 
 require 'phobos/deep_struct'
 require 'phobos/version'
 require 'phobos/instrumentation'
 require 'phobos/errors'
 require 'phobos/listener'
+require 'phobos/actions/process_batch'
+require 'phobos/actions/process_message'
 require 'phobos/producer'
 require 'phobos/handler'
 require 'phobos/echo_handler'
@@ -40,9 +43,10 @@ module Phobos
       Kafka.new(config.kafka.to_hash.merge(logger: @ruby_kafka_logger))
     end
 
-    def create_exponential_backoff
-      min = Phobos.config.backoff.min_ms / 1000.0
-      max = Phobos.config.backoff.max_ms / 1000.0
+    def create_exponential_backoff(backoff_config = nil)
+      backoff_config ||= Phobos.config.backoff.to_hash
+      min = backoff_config[:min_ms] / 1000.0
+      max = backoff_config[:max_ms] / 1000.0
       ExponentialBackoff.new(min, max).tap { |backoff| backoff.randomize_factor = rand }
     end
 

data/lib/phobos/actions/process_batch.rb

@@ -0,0 +1,61 @@
+module Phobos
+  module Actions
+    class ProcessBatch
+      include Phobos::Instrumentation
+
+      def initialize(listener:, batch:, listener_metadata:)
+        @listener = listener
+        @batch = batch
+        @listener_metadata = listener_metadata
+      end
+
+      def execute
+        @batch.messages.each do |message|
+          backoff = @listener.create_exponential_backoff
+          metadata = @listener_metadata.merge(
+            key: message.key,
+            partition: message.partition,
+            offset: message.offset,
+            retry_count: 0
+          )
+
+          begin
+            instrument('listener.process_message', metadata) do |metadata|
+              time_elapsed = measure do
+                Phobos::Actions::ProcessMessage.new(
+                  listener: @listener,
+                  message: message,
+                  metadata: metadata,
+                  encoding: @listener.encoding
+                ).execute
+              end
+              metadata.merge!(time_elapsed: time_elapsed)
+            end
+          rescue => e
+            retry_count = metadata[:retry_count]
+            interval = backoff.interval_at(retry_count).round(2)
+
+            error = {
+              waiting_time: interval,
+              exception_class: e.class.name,
+              exception_message: e.message,
+              backtrace: e.backtrace
+            }
+
+            instrument('listener.retry_handler_error', error.merge(metadata)) do
+              Phobos.logger.error do
+                { message: "error processing message, waiting #{interval}s" }.merge(error).merge(metadata)
+              end
+
+              sleep interval
+              metadata.merge!(retry_count: retry_count + 1)
+            end
+
+            raise Phobos::AbortError if @listener.should_stop?
+            retry
+          end
+        end
+      end
+    end
+  end
+end

data/lib/phobos/actions/process_message.rb

@@ -0,0 +1,26 @@
+module Phobos
+  module Actions
+    class ProcessMessage
+      def initialize(listener:, message:, metadata:, encoding:)
+        @listener = listener
+        @message = message
+        @metadata = metadata
+        @encoding = encoding
+      end
+
+      def execute
+        payload = force_encoding(@message.value)
+        decoded_payload = @listener.handler_class.new.before_consume(payload)
+        @listener.handler_class.around_consume(decoded_payload, @metadata) do
+          @listener.handler_class.new.consume(decoded_payload, @metadata)
+        end
+      end
+
+      private
+
+      def force_encoding(value)
+        @encoding ? value.force_encoding(@encoding) : value
+      end
+    end
+  end
+end

data/lib/phobos/executor.rb

@@ -1,7 +1,17 @@
 module Phobos
   class Executor
     include Phobos::Instrumentation
-    LISTENER_OPTS = %i(handler group_id topic min_bytes max_wait_time force_encoding start_from_beginning max_bytes_per_partition).freeze
+    LISTENER_OPTS = %i(
+      handler
+      group_id
+      topic
+      min_bytes
+      max_wait_time
+      force_encoding
+      start_from_beginning
+      max_bytes_per_partition
+      backoff
+    ).freeze
 
     def initialize
       @threads = Concurrent::Array.new
@@ -49,7 +59,7 @@ module Phobos
 
     def run_listener(listener)
       retry_count = 0
-      backoff = Phobos.create_exponential_backoff
+      backoff = listener.create_exponential_backoff
 
       begin
         listener.start

data/lib/phobos/handler.rb

@@ -4,6 +4,10 @@ module Phobos
       base.extend(ClassMethods)
     end
 
+    def before_consume(payload)
+      payload
+    end
+
     def consume(payload, metadata)
       raise NotImplementedError
     end

data/lib/phobos/instrumentation.rb

@@ -2,12 +2,6 @@ module Phobos
   module Instrumentation
     NAMESPACE = 'phobos'
 
-    def instrument(event, extra = {})
-      ActiveSupport::Notifications.instrument("#{NAMESPACE}.#{event}", extra) do |extra|
-        yield(extra) if block_given?
-      end
-    end
-
     def self.subscribe(event)
       ActiveSupport::Notifications.subscribe("#{NAMESPACE}.#{event}") do |*args|
         yield ActiveSupport::Notifications::Event.new(*args) if block_given?
@@ -17,5 +11,17 @@ module Phobos
     def self.unsubscribe(subscriber)
       ActiveSupport::Notifications.unsubscribe(subscriber)
     end
+
+    def instrument(event, extra = {})
+      ActiveSupport::Notifications.instrument("#{NAMESPACE}.#{event}", extra) do |extra|
+        yield(extra) if block_given?
+      end
+    end
+
+    def measure
+      start = Time.now.utc
+      yield if block_given?
+      (Time.now.utc - start).round(3)
+    end
   end
 end

data/lib/phobos/listener.rb

@@ -6,12 +6,17 @@ module Phobos
     DEFAULT_MAX_BYTES_PER_PARTITION = 524288 # 512 KB
 
     attr_reader :group_id, :topic, :id
+    attr_reader :handler_class, :encoding
 
-    def initialize(handler:, group_id:, topic:, min_bytes: nil, max_wait_time: nil, force_encoding: nil, start_from_beginning: true, max_bytes_per_partition: DEFAULT_MAX_BYTES_PER_PARTITION)
+    def initialize(handler:, group_id:, topic:, min_bytes: nil,
+                   max_wait_time: nil, force_encoding: nil,
+                   start_from_beginning: true, backoff: nil,
+                   max_bytes_per_partition: DEFAULT_MAX_BYTES_PER_PARTITION)
       @id = SecureRandom.hex[0...6]
       @handler_class = handler
       @group_id = group_id
       @topic = topic
+      @backoff = backoff
       @subscribe_opts = {
         start_from_beginning: start_from_beginning,
         max_bytes_per_partition: max_bytes_per_partition
@@ -47,12 +52,18 @@ module Phobos
           }.merge(listener_metadata)
 
           instrument('listener.process_batch', batch_metadata) do |batch_metadata|
-            time_elapsed = measure { process_batch(batch) }
+            time_elapsed = measure do
+              Phobos::Actions::ProcessBatch.new(
+                listener: self,
+                batch: batch,
+                listener_metadata: listener_metadata
+              ).execute
+            end
             batch_metadata.merge!(time_elapsed: time_elapsed)
             Phobos.logger.info { Hash(message: 'Committed offset').merge(batch_metadata) }
           end
 
-          return if @signal_to_stop
+          return if should_stop?
         end
 
       # Abort is an exception to prevent the consumer from committing the offset.
@@ -62,9 +73,7 @@ module Phobos
       #
       rescue Kafka::ProcessingError, Phobos::AbortError
         instrument('listener.retry_aborted', listener_metadata) do
-          Phobos.logger.info do
-            {message: 'Retry loop aborted, listener is shutting down'}.merge(listener_metadata)
-          end
+          Phobos.logger.info({ message: 'Retry loop aborted, listener is shutting down' }.merge(listener_metadata))
         end
       end
 
@@ -80,14 +89,14 @@ module Phobos
         end
 
         @kafka_client.close
-        if @signal_to_stop
+        if should_stop?
           Phobos.logger.info { Hash(message: 'Listener stopped').merge(listener_metadata) }
         end
       end
     end
 
     def stop
-      return if @signal_to_stop
+      return if should_stop?
       instrument('listener.stopping', listener_metadata) do
         Phobos.logger.info { Hash(message: 'Listener stopping').merge(listener_metadata) }
         @consumer&.stop
@@ -95,73 +104,23 @@ module Phobos
       end
     end
 
-    private
-
-    def listener_metadata
-      { listener_id: id, group_id: group_id, topic: topic }
+    def create_exponential_backoff
+      Phobos.create_exponential_backoff(@backoff)
     end
 
-    def process_batch(batch)
-      batch.messages.each do |message|
-        backoff = Phobos.create_exponential_backoff
-        metadata = listener_metadata.merge(
-          key: message.key,
-          partition: message.partition,
-          offset: message.offset,
-          retry_count: 0
-        )
-
-        begin
-          instrument('listener.process_message', metadata) do |metadata|
-            time_elapsed = measure { process_message(message, metadata) }
-            metadata.merge!(time_elapsed: time_elapsed)
-          end
-        rescue => e
-          retry_count = metadata[:retry_count]
-          interval = backoff.interval_at(retry_count).round(2)
-
-          error = {
-            waiting_time: interval,
-            exception_class: e.class.name,
-            exception_message: e.message,
-            backtrace: e.backtrace
-          }
-
-          instrument('listener.retry_handler_error', error.merge(metadata)) do
-            Phobos.logger.error do
-              {message: "error processing message, waiting #{interval}s"}.merge(error).merge(metadata)
-            end
-
-            sleep interval
-            metadata.merge!(retry_count: retry_count + 1)
-          end
-
-          raise Phobos::AbortError if @signal_to_stop
-          retry
-        end
-      end
+    def should_stop?
+      @signal_to_stop == true
     end
 
-    def process_message(message, metadata)
-      payload = force_encoding(message.value)
-      @handler_class.around_consume(payload, metadata) do
-        @handler_class.new.consume(payload, metadata)
-      end
+    private
+
+    def listener_metadata
+      { listener_id: id, group_id: group_id, topic: topic }
     end
 
     def create_kafka_consumer
       configs = Phobos.config.consumer_hash.select { |k| KAFKA_CONSUMER_OPTS.include?(k) }
-      @kafka_client.consumer({group_id: group_id}.merge(configs))
-    end
-
-    def force_encoding(value)
-      @encoding ? value.force_encoding(@encoding) : value
-    end
-
-    def measure
-      start = Time.now.utc
-      yield if block_given?
-      (Time.now.utc - start).round(3)
+      @kafka_client.consumer({ group_id: group_id }.merge(configs))
     end
 
     def compact(hash)

data/lib/phobos/test.rb

@@ -0,0 +1 @@
+require 'phobos/test/helper'

data/lib/phobos/test/helper.rb

@@ -0,0 +1,23 @@
+module Phobos
+  module Test
+    module Helper
+      KafkaMessage = Struct.new(:value)
+
+      def process_message(handler:, payload:, metadata:, force_encoding: nil)
+        listener = Phobos::Listener.new(
+          handler: handler,
+          group_id: 'test-group',
+          topic: 'test-topic',
+          force_encoding: force_encoding
+        )
+
+        Phobos::Actions::ProcessMessage.new(
+          listener: listener,
+          message: KafkaMessage.new(payload),
+          metadata: metadata,
+          encoding: listener.encoding
+        ).execute
+      end
+    end
+  end
+end

data/lib/phobos/version.rb

@@ -1,3 +1,3 @@
 module Phobos
-  VERSION = '1.4.2'
+  VERSION = '1.5.0'
 end

data/phobos.gemspec

@@ -12,7 +12,8 @@ Gem::Specification.new do |spec|
     'Sergey Evstifeev',
     'Thiago R. Colucci',
     'Martin Svalin',
-    'Francisco Juan'
+    'Francisco Juan',
+    'Tommy Gustafsson'
   ]
   spec.email         = [
     'ornelas.tulio@gmail.com',
@@ -20,7 +21,8 @@ Gem::Specification.new do |spec|
     'sergey.evstifeev@gmail.com',
     'ticolucci@gmail.com',
     'martin@lite.nu',
-    'francisco.juan@gmail.com'
+    'francisco.juan@gmail.com',
+    'tommydgustafsson@gmail.com'
   ]
 
   spec.summary       = %q{Simplifying Kafka for ruby apps}

data/utils/create-topic.sh

@@ -1,17 +1,13 @@
 #!/bin/bash
 set -eu
 
-UTILS_DIR=$(dirname $0)
-source ${UTILS_DIR}/env.sh
-
-ZK_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' zookeeper)
 TOPIC=${TOPIC:='test'}
 PARTITIONS=${PARTITIONS:=2}
 
 echo "creating topic ${TOPIC}, partitions ${PARTITIONS}"
-docker run --rm $KAFKA_IMAGE:$KAFKA_IMAGE_VERSION kafka-topics.sh \
-  --create \
+docker-compose run --rm -e PARTITIONS=$PARTITIONS -e TOPIC=$TOPIC kafka kafka-topics.sh --create \
   --topic $TOPIC \
   --replication-factor 1 \
   --partitions $PARTITIONS \
-  --zookeeper $ZK_IP:2181
+  --zookeeper zookeeper:2181 \
+  2>/dev/null

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: phobos
 version: !ruby/object:Gem::Version
-  version: 1.4.2
+  version: 1.5.0
 platform: ruby
 authors:
 - Túlio Ornelas
@@ -10,10 +10,11 @@ authors:
 - Thiago R. Colucci
 - Martin Svalin
 - Francisco Juan
+- Tommy Gustafsson
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-09-29 00:00:00.000000000 Z
+date: 2017-10-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -235,6 +236,7 @@ email:
 - ticolucci@gmail.com
 - martin@lite.nu
 - francisco.juan@gmail.com
+- tommydgustafsson@gmail.com
 executables:
 - phobos
 extensions: []
@@ -254,10 +256,13 @@ files:
 - bin/setup
 - circle.yml
 - config/phobos.yml.example
+- docker-compose.yml
 - examples/handler_saving_events_database.rb
 - examples/handler_using_async_producer.rb
 - examples/publishing_messages_without_consumer.rb
 - lib/phobos.rb
+- lib/phobos/actions/process_batch.rb
+- lib/phobos/actions/process_message.rb
 - lib/phobos/cli.rb
 - lib/phobos/cli/runner.rb
 - lib/phobos/cli/start.rb
@@ -269,15 +274,12 @@ files:
 - lib/phobos/instrumentation.rb
 - lib/phobos/listener.rb
 - lib/phobos/producer.rb
+- lib/phobos/test.rb
+- lib/phobos/test/helper.rb
 - lib/phobos/version.rb
 - logo.png
 - phobos.gemspec
 - utils/create-topic.sh
-- utils/env.sh
-- utils/kafka.sh
-- utils/start-all.sh
-- utils/stop-all.sh
-- utils/zk.sh
 homepage: https://github.com/klarna/phobos
 licenses:
 - Apache License Version 2.0

data/utils/env.sh

@@ -1,11 +0,0 @@
-#!/bin/bash
-set -eu
-
-DOCKER_HOSTNAME='localhost'
-FORCE_PULL=${FORCE_PULL:='false'}
-APPS=(zk kafka)
-
-ZK_IMAGE=jplock/zookeeper
-ZK_IMAGE_VERSION=3.4.6
-KAFKA_IMAGE=ches/kafka
-KAFKA_IMAGE_VERSION=0.9.0.1

data/utils/kafka.sh

@@ -1,43 +0,0 @@
-#!/bin/bash
-set -eu
-
-source $(dirname $0)/env.sh
-
-start() {
-  [ $FORCE_PULL = 'true' ] && docker pull $KAFKA_IMAGE:$KAFKA_IMAGE_VERSION
-  ZK_IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' zookeeper)
-
-  docker run \
-    -d \
-    -p 9092:9092 \
-    --name kafka \
-    -e KAFKA_BROKER_ID=0 \
-    -e KAFKA_ADVERTISED_HOST_NAME=localhost \
-    -e KAFKA_ADVERTISED_PORT=9092 \
-    -e ZOOKEEPER_CONNECTION_STRING=zookeeper:2181 \
-    --link zookeeper:zookeeper \
-    $KAFKA_IMAGE:$KAFKA_IMAGE_VERSION
-
-  # The following statement waits until kafka is up and running
-  docker exec kafka bash -c "JMX_PORT=9998 ./bin/kafka-topics.sh --zookeeper $ZK_IP:2181 --list 2> /dev/null"
-  [ $? != '0' ] && echo "[kafka] failed to start"
-}
-
-stop() {
-  docker stop kafka > /dev/null 2>&1 || true
-  docker rm kafka > /dev/null 2>&1 || true
-}
-
-case "$1" in
-  start)
-    echo "[kafka] starting $KAFKA_IMAGE:$KAFKA_IMAGE_VERSION"
-    stop
-    start
-    echo "[kafka] started"
-    ;;
-  stop)
-    printf "[kafka] stopping... "
-    stop
-    echo "Done"
-    ;;
-esac

data/utils/start-all.sh

@@ -1,9 +0,0 @@
-#!/bin/bash
-set -eu
-
-UTILS_DIR=$(dirname $0)
-source ${UTILS_DIR}/env.sh
-
-for (( i=0 ; i<${#APPS[@]} ; i++ )) ; do
-  ${UTILS_DIR}/${APPS[i]}.sh start
-done

data/utils/stop-all.sh

@@ -1,9 +0,0 @@
-#!/bin/bash
-set -eu
-
-UTILS_DIR=$(dirname $0)
-source ${UTILS_DIR}/env.sh
-
-for (( i=${#APPS[@]}-1 ; i>=0 ; i-- )) ; do
-  ${UTILS_DIR}/${APPS[i]}.sh stop
-done

data/utils/zk.sh

@@ -1,36 +0,0 @@
-#!/bin/bash
-set -eu
-
-UTILS_DIR=$(dirname $0)
-source ${UTILS_DIR}/env.sh
-
-start() {
-  [ $FORCE_PULL = 'true' ] && docker pull $ZK_IMAGE:$ZK_IMAGE_VERSION
-
-  docker run \
-    -d \
-    -p 2181:2181 \
-    --name zookeeper \
-    $ZK_IMAGE:$ZK_IMAGE_VERSION
-
-  sleep 3
-}
-
-stop() {
-  docker stop zookeeper > /dev/null 2>&1 || true
-  docker rm zookeeper > /dev/null 2>&1 || true
-}
-
-case "$1" in
-  start)
-    echo "[zookeeper] starting $ZK_IMAGE:$ZK_IMAGE_VERSION"
-    stop
-    start
-    echo "[zookeeper] started"
-    ;;
-  stop)
-    printf "[zookeeper] stopping... "
-    stop
-    echo "Done"
-    ;;
-esac