phobos 1.7.2 → 1.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 9a1be0c7e093c341850150a4fc386bb858a4cb37
-  data.tar.gz: 6873ff23a925070291916c4b227a4b139bc44463
+SHA256:
+  metadata.gz: 0b37b13898c547e212449d8b598984b80ceb9796d377991b7dd73f6e6acfdeb8
+  data.tar.gz: 66d12f8461a5e3e796b47364644a948c2f1c030d227f61a6dd675baf7e368f62
 SHA512:
-  metadata.gz: '08520a3dd1a8c53057561853ff1651a710a911128a2ef75ce65bb5f82c501c537317a92d473bdc82b4a2270acbd5c0cfd418397c68db286690b7387636f254bc'
-  data.tar.gz: 88ffede29159f84ed85f9b1a9c2efb5bc5a4b39f4b04d37cb2d9c690e828d27236c00dce1ecb365ab58a5e8efd503c2e2854cbb45d0bb074e829b48d37b977ff
+  metadata.gz: 75e2ddf1d53ea28086f0442779c2b19f7b8e9d78d62c7b7f4257fdf59d37f70d145fc014da37678d3023a4395c7b84b1c6d81145187482344af493114673931d
+  data.tar.gz: da67af5cee56a7867f99f0353b847fd8d1faf23dd7f24e539fc6a90514b9e5c3c38bef04f29d03cbd4344242818127d1f69aafb4bd36d1ac11321179b75d7cb4

data/.ruby-version

@@ -1 +1 @@
-2.4.1
+2.5.1

data/.travis.yml

@@ -1,8 +1,9 @@
 sudo: required
 language: ruby
 rvm:
-  - 2.4.1
-  - 2.3.6
+  - 2.5.1
+  - 2.4.4
+  - 2.3.7
 
 services:
   - docker

data/CHANGELOG.md

@@ -6,6 +6,14 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 
 ## UNRELEASED
 
+## [1.8.0] - 2018-07-22
+### Added
+- Possibility to configure a custom logger #81
+### Changed
+- Reduce the volume of info-level log messages #78
+- Phobos Handler `around_consume` is now an instance method #82
+- Send consumer heartbeats between retry attempts #83
+
 ## [1.7.2] - 2018-05-03
 ### Added
 - Add ability to override session_timeout, heartbeat_interval, offset_retention_time, offset_commit_interval, and offset_commit_threshold per listener

data/Dockerfile

@@ -1,8 +1,8 @@
-FROM ruby:2.4.1-alpine
+FROM ruby:2.5.1-alpine
 
 RUN apk update && apk upgrade && \
     apk add --no-cache bash git openssh build-base
-RUN gem install bundler -v 1.16.0
+RUN gem install bundler -v 1.16.1
 
 WORKDIR /opt/phobos
 

data/README.md

@@ -3,6 +3,7 @@
 [![Build Status](https://travis-ci.org/klarna/phobos.svg?branch=master)](https://travis-ci.org/klarna/phobos)
 [![Maintainability](https://api.codeclimate.com/v1/badges/2d00845fc6e7e83df6e7/maintainability)](https://codeclimate.com/github/klarna/phobos/maintainability)
 [![Test Coverage](https://api.codeclimate.com/v1/badges/2d00845fc6e7e83df6e7/test_coverage)](https://codeclimate.com/github/klarna/phobos/test_coverage)
+[![Depfu](https://badges.depfu.com/badges/57da3d5ff1da449cf8739cfe30b8d2f8/count.svg)](https://depfu.com/github/klarna/phobos?project=Bundler)
 [![Chat with us on Discord](https://discordapp.com/api/guilds/379938130326847488/widget.png)](https://discord.gg/rfMUBVD)
 
 # Phobos
@@ -155,7 +156,25 @@ class MyHandler
 end
 ```
 
-It is also possible to control the execution of `#consume` with the class method `.around_consume(payload, metadata)`. This method receives the payload and metadata, and then invokes `#consume` method by means of a block; example:
+It is also possible to control the execution of `#consume` with the method `#around_consume(payload, metadata)`. This method receives the payload and metadata, and then invokes `#consume` method by means of a block; example:
+
+```ruby
+class MyHandler
+  include Phobos::Handler
+
+  def around_consume(payload, metadata)
+    Phobos.logger.info "consuming..."
+    output = yield
+    Phobos.logger.info "done, output: #{output}"
+  end
+
+  def consume(payload, metadata)
+    # consume or skip message
+  end
+end
+```
+
+Note: `around_consume` was previously defined as a class method. The current code supports both implementations, giving precendence to the class method, but future versions will no longer support `.around_consume`.
 
 ```ruby
 class MyHandler
@@ -173,13 +192,13 @@ class MyHandler
 end
 ```
 
-Finally, it is also possible to preprocess the message payload before consuming it using the `before_consume` hook which is invoked before `.around_consume` and `#consume`. The result of this operation will be assigned to payload, so it is important to return the modified payload. This can be very useful, for example if you want a single point of decoding Avro messages and want the payload as a hash instead of a binary.
+Finally, it is also possible to preprocess the message payload before consuming it using the `before_consume` hook which is invoked before `#around_consume` and `#consume`. The result of this operation will be assigned to payload, so it is important to return the modified payload. This can be very useful, for example if you want a single point of decoding Avro messages and want the payload as a hash instead of a binary.
 
 ```ruby
 class MyHandler
   include Phobos::Handler
 
-  def before_consume(payload)
+  def before_consume(payload, metadata)
     # optionally preprocess payload
     payload
   end
@@ -194,7 +213,7 @@ The hander life cycle can be illustrated as:
 
 or optionally,
 
-  `.start` -> `#before_consume` -> `.around_consume` [ `#consume` ] -> `.stop`
+  `.start` -> `#before_consume` -> `#around_consume` [ `#consume` ] -> `.stop`
 
 ### <a name="usage-producing-messages-to-kafka"></a> Producing messages to Kafka
 
@@ -319,17 +338,24 @@ The configuration file is organized in 6 sections. Take a look at the example fi
 
 The file will be parsed through ERB so ERB syntax/file extension is supported beside the YML format.
 
-__logger__ configures the logger for all Phobos components, it automatically outputs to `STDOUT` and it saves the log in the configured file
+__logger__ configures the logger for all Phobos components. It automatically 
+outputs to `STDOUT` and it saves the log in the configured file.
 
-__kafka__ provides configurations for every `Kafka::Client` created over the application. All [options supported by  `ruby-kafka`][ruby-kafka-client] can be provided.
+__kafka__ provides configurations for every `Kafka::Client` created over the application. 
+All [options supported by  `ruby-kafka`][ruby-kafka-client] can be provided.
 
-__producer__ provides configurations for all producers created over the application, the options are the same for regular and async producers. All [options supported by  `ruby-kafka`][ruby-kafka-producer] can be provided.
+__producer__ provides configurations for all producers created over the application, 
+the options are the same for regular and async producers. 
+All [options supported by  `ruby-kafka`][ruby-kafka-producer] can be provided.
 
-__consumer__ provides configurations for all consumer groups created over the application. All [options supported by  `ruby-kafka`][ruby-kafka-consumer] can be provided.
+__consumer__ provides configurations for all consumer groups created over the application. 
+All [options supported by  `ruby-kafka`][ruby-kafka-consumer] can be provided.
 
-__backoff__ Phobos provides automatic retries for your handlers, if an exception is raised the listener will retry following the back off configured here. Backoff can also be configured per listener.
+__backoff__ Phobos provides automatic retries for your handlers. If an exception 
+is raised, the listener will retry following the back off configured here. 
+Backoff can also be configured per listener.
 
-__listeners__ is the list of listeners configured, each listener represents a consumers group
+__listeners__ is the list of listeners configured. Each listener represents a consumer group.
 
 [ruby-kafka-client]: http://www.rubydoc.info/gems/ruby-kafka/Kafka%2FClient%3Ainitialize
 [ruby-kafka-consumer]: http://www.rubydoc.info/gems/ruby-kafka/Kafka%2FClient%3Aconsumer
@@ -349,6 +375,23 @@ $ phobos start -c /var/configs/my.yml -l /var/configs/additional_listeners.yml
 Note that the config file _must_ still specify a listeners section, though it
 can be empty.
 
+#### Custom configuration/logging
+
+Phobos can be configured using a hash rather than the config file directly. This
+can be useful if you want to do some pre-processing before sending the file
+to Phobos. One particularly useful aspect is the ability to provide Phobos
+with a custom logger, e.g. by reusing the Rails logger:
+
+```ruby
+Phobos.configure(
+  custom_logger: Rails.logger,
+  custom_kafka_logger: Rails.logger
+)
+```
+
+If these keys are given, they will override the `logger` keys in the Phobos
+config file.
+
 ### <a name="usage-instrumentation"></a> Instrumentation
 
 Some operations are instrumented using [Active Support Notifications](http://api.rubyonrails.org/classes/ActiveSupport/Notifications.html).
@@ -481,8 +524,8 @@ describe MyConsumer do
   let(:metadata) { Hash(foo: 'bar') }
 
   it 'consumes my message' do
-    expect(described_class).to receive(:around_consume).with(payload, metadata).once.and_call_original
-    expect_any_instance_of(described_class).to receive(:before_consume).with(payload).once.and_call_original
+    expect_any_instance_of(described_class).to receive(:around_consume).with(payload, metadata).once.and_call_original
+    expect_any_instance_of(described_class).to receive(:before_consume).with(payload, metadata).once.and_call_original
     expect_any_instance_of(described_class).to receive(:consume).with(payload, metadata).once.and_call_original
 
     process_message(handler: described_class, payload: payload, metadata: metadata)

data/docker-compose.yml

@@ -13,18 +13,16 @@ services:
       - ./coverage:/opt/phobos/coverage
 
   zookeeper:
-    image: jplock/zookeeper:3.4.10
+    image: wurstmeister/zookeeper:latest
     ports:
      - 2181:2181
 
   kafka:
     depends_on:
       - zookeeper
-    image: ches/kafka:0.10.2.1
+    image: wurstmeister/kafka:0.11.0.1
     ports:
       - 9092:9092
     environment:
-      - KAFKA_BROKER_ID=0
-      - KAFKA_ADVERTISED_HOST_NAME=localhost
-      - KAFKA_ADVERTISED_PORT=9092
-      - ZOOKEEPER_CONNECTION_STRING=zookeeper:2181
+      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
+      KAFKA_BROKER_ID: 0

data/lib/phobos.rb

@@ -55,38 +55,53 @@ module Phobos
       ExponentialBackoff.new(min, max).tap { |backoff| backoff.randomize_factor = rand }
     end
 
+    # :nodoc:
     def configure_logger
-      log_file = config.logger.file
       ruby_kafka = config.logger.ruby_kafka
-      date_pattern = '%Y-%m-%dT%H:%M:%S:%L%zZ'
-      json_layout = Logging.layouts.json(date_pattern: date_pattern)
-
-      stdout_layout = if config.logger.stdout_json == true
-        json_layout
-      else
-        Logging.layouts.pattern(date_pattern: date_pattern)
-      end
-
-      appenders = [Logging.appenders.stdout(layout: stdout_layout)]
 
       Logging.backtrace(true)
       Logging.logger.root.level = silence_log ? :fatal : config.logger.level
-
-      if log_file
-        FileUtils.mkdir_p(File.dirname(log_file))
-        appenders << Logging.appenders.file(log_file, layout: json_layout)
-      end
+      appenders = logger_appenders
 
       @ruby_kafka_logger = nil
 
-      if ruby_kafka
+      if config.custom_kafka_logger
+        @ruby_kafka_logger = config.custom_kafka_logger
+      elsif ruby_kafka
         @ruby_kafka_logger = Logging.logger['RubyKafka']
         @ruby_kafka_logger.appenders = appenders
         @ruby_kafka_logger.level = silence_log ? :fatal : ruby_kafka.level
       end
 
-      @logger = Logging.logger[self]
-      @logger.appenders = appenders
+      if config.custom_logger
+        @logger = config.custom_logger
+      else
+        @logger = Logging.logger[self]
+        @logger.appenders = appenders
+      end
+    end
+
+    def logger_appenders
+      date_pattern = '%Y-%m-%dT%H:%M:%S:%L%zZ'
+      json_layout = Logging.layouts.json(date_pattern: date_pattern)
+      log_file = config.logger.file
+      stdout_layout = if config.logger.stdout_json == true
+                        json_layout
+                      else
+                        Logging.layouts.pattern(date_pattern: date_pattern)
+                      end
+
+      appenders = [Logging.appenders.stdout(layout: stdout_layout)]
+
+      if log_file
+        FileUtils.mkdir_p(File.dirname(log_file))
+        appenders << Logging.appenders.file(log_file, layout: json_layout)
+      end
+      appenders
+    end
+
+    def deprecate(message)
+      warn "DEPRECATION WARNING: #{message} #{Kernel.caller.first}"
     end
 
     private

data/lib/phobos/actions/process_message.rb

@@ -1,6 +1,8 @@
 module Phobos
   module Actions
     class ProcessMessage
+      MAX_SLEEP_INTERVAL = 3
+
       include Phobos::Instrumentation
 
       attr_reader :metadata
@@ -38,16 +40,26 @@ module Phobos
               { message: "error processing message, waiting #{interval}s" }.merge(error).merge(@metadata)
             end
 
-            sleep interval
+            snooze(interval)
           end
 
-          raise Phobos::AbortError if @listener.should_stop?
-
           @metadata.merge!(retry_count: retry_count + 1)
           retry
         end
       end
 
+      def snooze(interval)
+        remaining_interval = interval
+
+        @listener.send_heartbeat_if_necessary
+
+        while remaining_interval.positive?
+          sleep [remaining_interval, MAX_SLEEP_INTERVAL].min
+          remaining_interval -= MAX_SLEEP_INTERVAL
+          @listener.send_heartbeat_if_necessary
+        end
+      end
+
       private
 
       def force_encoding(value)
@@ -57,10 +69,23 @@ module Phobos
       def process_message(payload)
         instrument('listener.process_message', @metadata) do
           handler = @listener.handler_class.new
-          preprocessed_payload = handler.before_consume(payload)
+          preprocessed_payload = begin
+            handler.before_consume(payload, @metadata)
+          rescue ArgumentError => e
+            Phobos.deprecate("before_consume now expects metadata as second argument, please update your consumer."\
+              " This will not be backwards compatible in the future.")
+            handler.before_consume(payload)
+          end
+          consume_block = Proc.new { handler.consume(preprocessed_payload, @metadata) }
 
-          @listener.handler_class.around_consume(preprocessed_payload, @metadata) do
-            handler.consume(preprocessed_payload, @metadata)
+          if @listener.handler_class.respond_to?(:around_consume)
+            # around_consume class method implementation
+            Phobos.deprecate("around_consume has been moved to instance method, please update your consumer."\
+              " This will not be backwards compatible in the future.")
+            @listener.handler_class.around_consume(preprocessed_payload, @metadata, &consume_block)
+          else
+            # around_consume instance method implementation
+            handler.around_consume(preprocessed_payload, @metadata, &consume_block)
           end
         end
       end

data/lib/phobos/handler.rb

@@ -4,7 +4,7 @@ module Phobos
       base.extend(ClassMethods)
     end
 
-    def before_consume(payload)
+    def before_consume(payload, metadata)
       payload
     end
 
@@ -12,16 +12,16 @@ module Phobos
       raise NotImplementedError
     end
 
+    def around_consume(payload, metadata)
+      yield
+    end
+
     module ClassMethods
       def start(kafka_client)
       end
 
       def stop
       end
-
-      def around_consume(payload, metadata)
-        yield
-      end
     end
   end
 end

data/lib/phobos/listener.rb

@@ -103,7 +103,7 @@ module Phobos
         )
 
         batch_processor.execute
-        Phobos.logger.info { Hash(message: 'Committed offset').merge(batch_processor.metadata) }
+        Phobos.logger.debug { Hash(message: 'Committed offset').merge(batch_processor.metadata) }
         return if should_stop?
       end
     end
@@ -117,7 +117,7 @@ module Phobos
         )
 
         message_processor.execute
-        Phobos.logger.info { Hash(message: 'Committed offset').merge(message_processor.metadata) }
+        Phobos.logger.debug { Hash(message: 'Committed offset').merge(message_processor.metadata) }
         return if should_stop?
       end
     end
@@ -139,6 +139,11 @@ module Phobos
       @signal_to_stop == true
     end
 
+    def send_heartbeat_if_necessary
+      raise Phobos::AbortError if should_stop?
+      @consumer&.send_heartbeat_if_necessary
+    end
+
     private
 
     def listener_metadata

data/lib/phobos/version.rb

@@ -1,3 +1,3 @@
 module Phobos
-  VERSION = '1.7.2'
+  VERSION = '1.8.0'
 end

data/phobos.gemspec

@@ -45,7 +45,7 @@ Gem::Specification.new do |spec|
   spec.required_ruby_version = '>= 2.3'
 
   spec.add_development_dependency 'bundler'
-  spec.add_development_dependency 'rake', '~> 10.0'
+  spec.add_development_dependency 'rake', '~> 12.3'
   spec.add_development_dependency 'rspec', '~> 3.0'
   spec.add_development_dependency 'pry-byebug'
   spec.add_development_dependency 'simplecov'

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: phobos
 version: !ruby/object:Gem::Version
-  version: 1.7.2
+  version: 1.8.0
 platform: ruby
 authors:
 - Túlio Ornelas
@@ -14,7 +14,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-05-03 00:00:00.000000000 Z
+date: 2018-07-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -36,14 +36,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: '12.3'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '10.0'
+        version: '12.3'
 - !ruby/object:Gem::Dependency
   name: rspec
   requirement: !ruby/object:Gem::Requirement
@@ -275,7 +275,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.6.13
+rubygems_version: 2.7.6
 signing_key: 
 specification_version: 4
 summary: Simplifying Kafka for ruby apps