phobos 1.0.0 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 97e08e50c3b2cfdb020c71c181989cb90e865316
-  data.tar.gz: 21bab9e83626f738b5903355e477b03ad5ff4d89
+  metadata.gz: 31d36810a7dbe760625a6f08d618d37780c1f584
+  data.tar.gz: ffbca77c17a57c8bc86d6497e4e4f6fccb3efdd4
 SHA512:
-  metadata.gz: 17421090032e4212c626f708e80f5dca3d3c6421757982268980d954841473ab5176fdb907ac594bc079a5595b4f405b6402362767e01e12ec6b9a45d83e3d01
-  data.tar.gz: aa1b7bbf6f07c99cde3f2246a0dc848b9e7a1cb395e42e1513e5a1e7265488cb46b81dc48ae9d049abf61ebab4854dece2b4d594ab52a2e3310fd97412a2c406
+  metadata.gz: a6f98e9ce96d3c0671db8c7c63cb833b3189f3541706258dc2e4757d2ca714dfc675973c23dd2edb344c5f888c987674a24341d98345de0dc0c6d76de4138db6
+  data.tar.gz: 7c98f56c03f8d479bb7067bde18e6ba9290184b08eadad0a680c42c6cf001db71e9c42a86357ffd9909bbf27b3364f95e2211fcd05c56eec5a88364101411aad

data/CHANGELOG.md

@@ -0,0 +1,14 @@
+# Change Log
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](http://keepachangelog.com/)
+and this project adheres to [Semantic Versioning](http://semver.org/).
+
+## 1.1.0 (2016-09-02)
+- [enhancement] - Removed Hashie as a dependency #12
+- [feature] Allow configuring consumers min_bytes & max_wait_time #15
+- [feature] Allow configuring producers max_queue_size, delivery_threshold & delivery_interval #16
+- [feature] Allow configuring force_encoding for message payload #18
+
+## 1.0.0 (2016-08-08)
+- Published on Github with full fledged consumers and producers

data/README.md

@@ -1,4 +1,7 @@
-![Circle CI](https://circleci.com/gh/klarna/phobos.svg?style=shield&circle-token=2289e0fe5bd934074597b32e7f8f0bc98ea0e3c7)
+![Phobos](https://raw.githubusercontent.com/klarna/phobos/master/logo.png)
+
+[![Circle CI](https://circleci.com/gh/klarna/phobos.svg?style=shield&circle-token=2289e0fe5bd934074597b32e7f8f0bc98ea0e3c7)](https://circleci.com/gh/klarna/phobos/tree/master)
+[![Coverage Status](https://coveralls.io/repos/github/klarna/phobos/badge.svg?branch=master)](https://coveralls.io/github/klarna/phobos?branch=master)
 
 # Phobos
 
@@ -24,6 +27,7 @@ With Phobos by your side, all this becomes smooth sailing.
   1. [As library in another app](#usage-as-library)
   1. [Configuration file](#usage-configuration-file)
   1. [Instrumentation](#usage-instrumentation)
+1. [Plugins](#plugins)
 1. [Development](#development)
 
 ## <a name="installation"></a> Installation
@@ -73,7 +77,7 @@ $ phobos init
 
 ### Consumers (listeners and handlers)
 
-In Phobos apps __listeners__ are configured against Kafka - they are our consumers. A listener requires a __handler__ (a ruby class where you should process incoming messages), a __topic__, and a __group_id__. Consumer groups are used to coordinate the listeners across machines. We write the __handlers__ and Phobos makes sure to run them for us. An example of a handler is:
+In Phobos apps __listeners__ are configured against Kafka - they are our consumers. A listener requires a __handler__ (a ruby class where you should process incoming messages), a Kafka __topic__, and a Kafka __group_id__. Consumer groups are used to coordinate the listeners across machines. We write the __handlers__ and Phobos makes sure to run them for us. An example of a handler is:
 
 ```ruby
 class MyHandler
@@ -201,9 +205,9 @@ MyProducer
   ])
 ```
 
-There are two flavors of producers: __normal__ producers and __async__ producers.
+There are two flavors of producers: __regular__ producers and __async__ producers.
 
-Normal producers will deliver the messages synchronously and disconnect, it doesn't matter if you use `publish` or `publish_list` after the messages get delivered the producer will disconnect.
+Regular producers will deliver the messages synchronously and disconnect, it doesn't matter if you use `publish` or `publish_list` after the messages get delivered the producer will disconnect.
 
 Async producers will accept your messages without blocking, use the methods `async_publish` and `async_publish_list` to use async producers.
 
@@ -289,7 +293,7 @@ __logger__ configures the logger for all Phobos components, it automatically out
 
 __kafka__ provides configurations for every `Kafka::Client` created over the application. All [options supported by  `ruby-kafka`][ruby-kafka-client] can be provided.
 
-__producer__ provides configurations for all producers created over the application, the options are the same for normal and async producers. All [options supported by  `ruby-kafka`][ruby-kafka-producer] can be provided.
+__producer__ provides configurations for all producers created over the application, the options are the same for regular and async producers. All [options supported by  `ruby-kafka`][ruby-kafka-producer] can be provided.
 
 __consumer__ provides configurations for all consumer groups created over the application. All [options supported by  `ruby-kafka`][ruby-kafka-consumer] can be provided.
 
@@ -379,6 +383,15 @@ end
     * group_id
     * topic
 
+## <a name="plugins"></a> Plugins
+
+List of gems that enhance Phobos:
+
+* [Phobos DB Checkpoint](https://github.com/klarna/phobos_db_checkpoint) is drop in replacement to Phobos::Handler, extending it with the following features:
+  * Persists your Kafka events to an active record compatible database
+  * Ensures that your handler will consume messages only once
+  * Allows your system to quickly reprocess events in case of failures
+
 ## <a name="development"></a> Development
 
 After checking out the repo:
@@ -402,6 +415,10 @@ sh utils/stop-all.sh
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/klarna/phobos.
 
+## Acknowledgements
+
+Thanks to Sebastian Norde for the awesome logo!
+
 ## License
 
 Copyright 2016 Klarna

data/config/phobos.yml.example

@@ -45,6 +45,14 @@ producer:
   # number of messages that needs to be in a message set before it should be compressed.
   # Note that message sets are per-partition rather than per-topic or per-producer
   compression_threshold: 1
+  # maximum number of messages allowed in the queue. Only used for async_producer
+  max_queue_size: 1000
+  # if greater than zero, the number of buffered messages that will automatically
+  # trigger a delivery. Only used for async_producer
+  delivery_threshold: 0
+  # if greater than zero, the number of seconds between automatic message
+  # deliveries. Only used for async_producer
+  delivery_interval: 0
 
 consumer:
   # number of seconds after which, if a client hasn't contacted the Kafka cluster,
@@ -67,12 +75,19 @@ listeners:
     topic: test
     # id of the group that the consumer should join
     group_id: test-1
+    # Number of threads created for this listener, each thread will behave as an independent consumer.
+    # They don't share any state
+    max_concurrency: 1
     # Once the consumer group has checkpointed its progress in the topic's partitions,
     # the consumers will always start from the checkpointed offsets, regardless of config
     # As such, this setting only applies when the consumer initially starts consuming from a topic
     start_from_beginning: true
     # maximum amount of data fetched from a single partition at a time
     max_bytes_per_partition: 524288 # 512 KB
-    # Number of threads created for this listener, each thread will behave as an independent consumer.
-    # They don't share any state
-    max_concurrency: 1
+    # Minimum number of bytes to read before returning messages from the server; if `max_wait_time` is reached, this is ignored.
+    min_bytes: 1
+    # Maximum duration of time to wait before returning messages from the server, in seconds
+    max_wait_time: 5
+    # Apply this encoding to the message payload, if blank it uses the original encoding. This property accepts values
+    # defined by the ruby Encoding class (https://ruby-doc.org/core-2.3.0/Encoding.html). Ex: UTF_8, ASCII_8BIT, etc
+    force_encoding:

data/examples/{publishing_messages_without_consumer.rb.rb → publishing_messages_without_consumer.rb}

@@ -1,5 +1,10 @@
-require "bundler/setup"
-require "phobos"
+#
+# This example assumes you want to create a threaded kafka generator which
+# publish a stream of kafka messages without consuming them. It also shows
+# what happens when you produce more messages than the producer can handle.
+#
+require 'bundler/setup'
+require 'phobos'
 
 TOPIC = 'test-partitions'
 
@@ -40,8 +45,8 @@ Thread.new do
 
         puts "produced #{key}, total: #{total}"
 
-      # Since this is a simplistic code we are going to generate more messages than
-      # the producer can write to Kafka, so eventually we'll get some buffer overflows
+      # Since this is very simplistic code, we are going to generate more messages than
+      # the producer can write to Kafka. Eventually we'll get some buffer overflows
       #
       rescue Kafka::BufferOverflow => e
         puts "| waiting"

data/lib/phobos/deep_struct.rb

@@ -0,0 +1,40 @@
+# Please use this with at least the same consideration as you would when using OpenStruct.
+# Right now we only use this to parse our internal configuration files. It is not meant to
+# be used on incoming data.
+module Phobos
+  class DeepStruct < OpenStruct
+    # Based on
+    # https://docs.omniref.com/ruby/2.3.0/files/lib/ostruct.rb#line=88
+    def initialize(hash=nil)
+      @table = {}
+      @hash_table = {}
+
+      if hash
+        hash.each_pair do |k, v|
+          k = k.to_sym
+          @table[k] = to_deep_struct(v)
+          @hash_table[k] = v
+        end
+      end
+    end
+
+    def to_h
+      @hash_table.dup
+    end
+    alias_method :to_hash, :to_h
+
+    private
+
+    def to_deep_struct(v)
+      case v
+      when Hash
+        self.class.new(v)
+      when Enumerable
+        v.map { |el| to_deep_struct(el) }
+      else
+        v
+      end
+    end
+    protected :to_deep_struct
+  end
+end

data/lib/phobos/executor.rb

@@ -1,16 +1,16 @@
 module Phobos
   class Executor
     include Phobos::Instrumentation
-    LISTENER_OPTS = %i(handler group_id topic start_from_beginning max_bytes_per_partition).freeze
+    LISTENER_OPTS = %i(handler group_id topic min_bytes max_wait_time force_encoding start_from_beginning max_bytes_per_partition).freeze
 
     def initialize
       @threads = Concurrent::Array.new
       @listeners = Phobos.config.listeners.flat_map do |config|
         handler_class = config.handler.constantize
-        listener_configs = config.to_hash.symbolize_keys
+        listener_configs = config.to_hash
         max_concurrency = listener_configs[:max_concurrency] || 1
         max_concurrency.times.map do
-          configs = listener_configs.select {|k| LISTENER_OPTS.include?(k)}
+          configs = listener_configs.select { |k| LISTENER_OPTS.include?(k) }
           Phobos::Listener.new(configs.merge(handler: handler_class))
         end
       end

data/lib/phobos/listener.rb

@@ -7,7 +7,7 @@ module Phobos
 
     attr_reader :group_id, :topic, :id
 
-    def initialize(handler:, group_id:, topic:, start_from_beginning: true, max_bytes_per_partition: DEFAULT_MAX_BYTES_PER_PARTITION)
+    def initialize(handler:, group_id:, topic:, min_bytes: nil, max_wait_time: nil, force_encoding: nil, start_from_beginning: true, max_bytes_per_partition: DEFAULT_MAX_BYTES_PER_PARTITION)
       @id = SecureRandom.hex[0...6]
       @handler_class = handler
       @group_id = group_id
@@ -16,6 +16,8 @@ module Phobos
         start_from_beginning: start_from_beginning,
         max_bytes_per_partition: max_bytes_per_partition
       }
+      @encoding = Encoding.const_get(force_encoding.to_sym) if force_encoding
+      @consumer_opts = compact(min_bytes: min_bytes, max_wait_time: max_wait_time)
       @kafka_client = Phobos.create_kafka_client
       @producer_enabled = @handler_class.ancestors.include?(Phobos::Producer)
     end
@@ -35,7 +37,7 @@ module Phobos
       end
 
       begin
-        @consumer.each_batch do |batch|
+        @consumer.each_batch(@consumer_opts) do |batch|
           batch_metadata = {
             batch_size: batch.messages.count,
             partition: batch.partition,
@@ -139,7 +141,7 @@ module Phobos
     end
 
     def process_message(message, metadata)
-      payload = message.value
+      payload = force_encoding(message.value)
       @handler_class.around_consume(payload, metadata) do
         @handler_class.new.consume(payload, metadata)
       end
@@ -149,5 +151,13 @@ module Phobos
       configs = Phobos.config.consumer_hash.select { |k| KAFKA_CONSUMER_OPTS.include?(k) }
       @kafka_client.consumer({group_id: group_id}.merge(configs))
     end
+
+    def force_encoding(value)
+      @encoding ? value.force_encoding(@encoding) : value
+    end
+
+    def compact(hash)
+      hash.delete_if { |_, v| v.nil? }
+    end
   end
 end

data/lib/phobos/producer.rb

@@ -49,6 +49,7 @@ module Phobos
 
       class PublicAPI
         NAMESPACE = :phobos_producer_store
+        ASYNC_PRODUCER_PARAMS = %i(max_queue_size delivery_threshold delivery_interval).freeze
 
         # This method configures the kafka client used with publish operations
         # performed by the host class
@@ -70,7 +71,7 @@ module Phobos
 
         def publish_list(messages)
           client = kafka_client || configure_kafka_client(Phobos.create_kafka_client)
-          producer = client.producer(Phobos.config.producer_hash)
+          producer = client.producer(regular_configs)
           produce_messages(producer, messages)
         ensure
           producer&.shutdown
@@ -78,7 +79,7 @@ module Phobos
 
         def create_async_producer
           client = kafka_client || configure_kafka_client(Phobos.create_kafka_client)
-          async_producer = client.async_producer(Phobos.config.producer_hash)
+          async_producer = client.async_producer(async_configs)
           producer_store[:async_producer] = async_producer
         end
 
@@ -101,6 +102,14 @@ module Phobos
           producer_store[:async_producer] = nil
         end
 
+        def regular_configs
+          Phobos.config.producer_hash.reject { |k, _| ASYNC_PRODUCER_PARAMS.include?(k)}
+        end
+
+        def async_configs
+          Phobos.config.producer_hash
+        end
+
         private
 
         def produce_messages(producer, messages)

data/lib/phobos/version.rb

@@ -1,3 +1,3 @@
 module Phobos
-  VERSION = "1.0.0"
+  VERSION = "1.1.0"
 end

data/lib/phobos.rb

@@ -1,15 +1,16 @@
-require 'yaml'
+require 'ostruct'
 require 'securerandom'
+require 'yaml'
 
+require 'active_support/core_ext/hash/keys'
+require 'active_support/core_ext/string/inflections'
+require 'active_support/notifications'
 require 'concurrent'
+require 'exponential_backoff'
 require 'kafka'
-require 'hashie'
 require 'logging'
-require 'exponential_backoff'
-require 'active_support/notifications'
-require 'active_support/core_ext/string/inflections'
-require 'active_support/core_ext/hash/keys'
 
+require 'phobos/deep_struct'
 require 'phobos/version'
 require 'phobos/instrumentation'
 require 'phobos/errors'
@@ -28,15 +29,15 @@ module Phobos
 
     def configure(yml_path)
       ENV['RAILS_ENV'] = ENV['RACK_ENV'] ||= 'development'
-      @config = Hashie::Mash.new(YAML.load_file(File.expand_path(yml_path)))
-      @config.class.send(:define_method, :producer_hash) { Phobos.config.producer&.to_hash&.symbolize_keys }
-      @config.class.send(:define_method, :consumer_hash) { Phobos.config.consumer&.to_hash&.symbolize_keys }
+      @config = DeepStruct.new(YAML.load_file(File.expand_path(yml_path)))
+      @config.class.send(:define_method, :producer_hash) { Phobos.config.producer&.to_hash }
+      @config.class.send(:define_method, :consumer_hash) { Phobos.config.consumer&.to_hash }
       configure_logger
       logger.info { Hash(message: 'Phobos configured', env: ENV['RACK_ENV']) }
     end
 
     def create_kafka_client
-      Kafka.new(config.kafka.to_hash.symbolize_keys)
+      Kafka.new(config.kafka.to_hash)
     end
 
     def create_exponential_backoff

data/phobos.gemspec

@@ -24,7 +24,7 @@ Gem::Specification.new do |spec|
   ]
 
   spec.summary       = %q{Simplifying Kafka for ruby apps}
-  spec.description   = %q{Phobos is a microframework and library for kafka based applications, it wraps commons behaviors needed by consumers/producers in an easy an convenient API. It uses ruby-kafka as it's kafka client and core component.}
+  spec.description   = %q{Phobos is a microframework and library for kafka based applications, it wraps common behaviors needed by consumers/producers in an easy an convenient API. It uses ruby-kafka as its kafka client and core component.}
   spec.homepage      = 'https://github.com/klarna/phobos'
   spec.license       = 'Apache License Version 2.0'
 
@@ -47,12 +47,13 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency 'rspec', '~> 3.0'
   spec.add_development_dependency 'pry-byebug', '~> 3.4.0'
   spec.add_development_dependency 'rspec_junit_formatter', '0.2.2'
+  spec.add_development_dependency 'simplecov', '~> 0.12.0'
+  spec.add_development_dependency 'coveralls', '~> 0.8.15'
 
-  spec.add_dependency 'ruby-kafka', '>= 0.3.13.beta4'
+  spec.add_dependency 'ruby-kafka', '>= 0.3.14'
   spec.add_dependency 'concurrent-ruby', '>= 1.0.2'
   spec.add_dependency 'concurrent-ruby-ext', '>= 1.0.2'
   spec.add_dependency 'activesupport', '>= 4.0.0'
-  spec.add_dependency 'hashie'
   spec.add_dependency 'logging'
   spec.add_dependency 'exponential-backoff'
   spec.add_dependency 'thor'

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: phobos
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.1.0
 platform: ruby
 authors:
 - Túlio Ornelas
@@ -13,7 +13,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-08-18 00:00:00.000000000 Z
+date: 2016-09-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -86,35 +86,49 @@ dependencies:
       - !ruby/object:Gem::Version
         version: 0.2.2
 - !ruby/object:Gem::Dependency
-  name: ruby-kafka
+  name: simplecov
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.3.13.beta4
-  type: :runtime
+        version: 0.12.0
+  type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.3.13.beta4
+        version: 0.12.0
 - !ruby/object:Gem::Dependency
-  name: concurrent-ruby
+  name: coveralls
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.8.15
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 0.8.15
+- !ruby/object:Gem::Dependency
+  name: ruby-kafka
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 1.0.2
+        version: 0.3.14
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 1.0.2
+        version: 0.3.14
 - !ruby/object:Gem::Dependency
-  name: concurrent-ruby-ext
+  name: concurrent-ruby
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -128,33 +142,33 @@ dependencies:
       - !ruby/object:Gem::Version
         version: 1.0.2
 - !ruby/object:Gem::Dependency
-  name: activesupport
+  name: concurrent-ruby-ext
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 4.0.0
+        version: 1.0.2
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 4.0.0
+        version: 1.0.2
 - !ruby/object:Gem::Dependency
-  name: hashie
+  name: activesupport
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 4.0.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 4.0.0
 - !ruby/object:Gem::Dependency
   name: logging
   requirement: !ruby/object:Gem::Requirement
@@ -198,8 +212,8 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '0'
 description: Phobos is a microframework and library for kafka based applications,
-  it wraps commons behaviors needed by consumers/producers in an easy an convenient
-  API. It uses ruby-kafka as it's kafka client and core component.
+  it wraps common behaviors needed by consumers/producers in an easy an convenient
+  API. It uses ruby-kafka as its kafka client and core component.
 email:
 - ornelas.tulio@gmail.com
 - mathias.klippinge@gmail.com
@@ -215,6 +229,7 @@ files:
 - ".gitignore"
 - ".rspec"
 - ".ruby-version"
+- CHANGELOG.md
 - Dockerfile
 - Gemfile
 - LICENSE.txt
@@ -227,11 +242,12 @@ files:
 - config/phobos.yml.example
 - examples/handler_saving_events_database.rb
 - examples/handler_using_async_producer.rb
-- examples/publishing_messages_without_consumer.rb.rb
+- examples/publishing_messages_without_consumer.rb
 - lib/phobos.rb
 - lib/phobos/cli.rb
 - lib/phobos/cli/runner.rb
 - lib/phobos/cli/start.rb
+- lib/phobos/deep_struct.rb
 - lib/phobos/echo_handler.rb
 - lib/phobos/errors.rb
 - lib/phobos/executor.rb
@@ -240,6 +256,7 @@ files:
 - lib/phobos/listener.rb
 - lib/phobos/producer.rb
 - lib/phobos/version.rb
+- logo.png
 - phobos.gemspec
 - utils/create-topic.sh
 - utils/env.sh