karafka 1.1.0 → 1.3.0

data/README.md

@@ -2,41 +2,56 @@
 
 [![Build Status](https://travis-ci.org/karafka/karafka.svg?branch=master)](https://travis-ci.org/karafka/karafka)
 
+## New release in progress!
+
+**Note**: Documentation presented here refers to Karafka `1.3.0`.
+
+If you are looking for the documentation for Karafka `1.2.*`, it can be found [here](https://github.com/karafka/wiki/tree/1.2).
+
+## About Karafka
+
 Framework used to simplify Apache Kafka based Ruby applications development.
 
-It allows programmers to use approach similar to standard HTTP conventions (```params``` and ```params_batch```) when working with asynchronous Kafka messages.
+Karafka allows you to capture everything that happens in your systems in large scale, providing you with a seamless and stable core for consuming and processing this data, without having to focus on things that are not your business domain.
 
 Karafka not only handles incoming messages but also provides tools for building complex data-flow applications that receive and send messages.
 
 ## How does it work
 
-Karafka provides a higher-level abstraction that allows you to focus on your business logic development, instead of focusing on implementing lower level abstraction layers. It provides developers with a set of tools that are dedicated for building multi-topic applications similarly to how Rails applications are being built.
+Karafka provides a higher-level abstraction that allows you to focus on your business logic development, instead of focusing on implementing lower level abstraction layers. It provides developers with a set of tools that are dedicated for building multi-topic applications similar to how Rails applications are being built.
 
 ### Some things you might wonder about:
 
-- You can integrate Karafka with any Ruby based application.
+- You can integrate Karafka with **any** Ruby-based application.
 - Karafka does **not** require Sidekiq or any other third party software (apart from Kafka itself).
-- Karafka works with Ruby on Rails but it is a standalone framework that can work without it.
-- Karafka has a minimal set of dependencies, so adding it won't be a huge burden for your already existing applications.
+- Karafka works with Ruby on Rails but it is a **standalone** framework that can work without it.
+- Karafka has a **minimal** set of dependencies, so adding it won't be a huge burden for your already existing applications.
+- Karafka processes can be executed for a **given subset** of consumer groups and/or topics, so you can fine tune it depending on your business logic.
 
 Karafka based applications can be easily deployed to any type of infrastructure, including those based on:
 
 * Heroku
 * Capistrano
 * Docker
+* Terraform
 
 ## Support
 
-Karafka has a [Wiki pages](https://github.com/karafka/karafka/wiki) for almost everything and a pretty decent [FAQ](https://github.com/karafka/karafka/wiki/FAQ). It covers the whole installation, setup and deployment along with other useful details on how to run Karafka.
+Karafka has a [Wiki pages](https://github.com/karafka/karafka/wiki) for almost everything and a pretty decent [FAQ](https://github.com/karafka/karafka/wiki/FAQ). It covers the whole installation, setup, and deployment along with other useful details on how to run Karafka.
 
 If you have any questions about using Karafka, feel free to join our [Gitter](https://gitter.im/karafka/karafka) chat channel.
 
 ## Getting started
 
+If you're completely new to the subject, you can start with our "Kafka on Rails" articles series, that will get you up and running with the terminology and basic ideas behind using Kafka:
+
+- [Kafka on Rails: Using Kafka with Ruby on Rails – Part 1 – Kafka basics and its advantages](https://mensfeld.pl/2017/11/kafka-on-rails-using-kafka-with-ruby-on-rails-part-1-kafka-basics-and-its-advantages/)
+- [Kafka on Rails: Using Kafka with Ruby on Rails – Part 2 – Getting started with Ruby and Kafka](https://mensfeld.pl/2018/01/kafka-on-rails-using-kafka-with-ruby-on-rails-part-2-getting-started-with-ruby-and-kafka/)
+
 If you want to get started with Kafka and Karafka as fast as possible, then the best idea is to just clone our example repository:
 
 ```bash
-git clone https://github.com/karafka/karafka-example-app ./example_app
+git clone https://github.com/karafka/example-app ./example_app
 ```
 
 then, just bundle install all the dependencies:
@@ -46,7 +61,7 @@ cd ./example_app
 bundle install
 ```
 
-and follow the instructions from the [example app Wiki](https://github.com/karafka/karafka-example-app/blob/master/README.md).
+and follow the instructions from the [example app Wiki](https://github.com/karafka/example-app/blob/master/README.md).
 
 **Note**: you need to ensure, that you have Kafka up and running and you need to configure Kafka seed_brokers in the ```karafka.rb``` file.
 
@@ -62,26 +77,17 @@ Karafka framework and Karafka team are __not__ related to Kafka streaming servic
 * [Karafka Travis CI](https://travis-ci.org/karafka/karafka)
 * [Karafka Coditsu](https://app.coditsu.io/karafka/repositories/karafka)
 
-## Note on Patches/Pull Requests
-
-Fork the project.
-Make your feature addition or bug fix.
-Add tests for it. This is important so we don't break it in a future versions unintentionally.
-Commit, do not mess with Rakefile, version, or history. (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull). Send me a pull request. Bonus points for topic branches.
-
-[![coditsu](https://coditsu.io/assets/quality_bar.svg)](https://app.coditsu.io/karafka/repositories/karafka)
+## Note on contributions
 
-Each pull request must pass our quality requirements. To check if everything is as it should be, we use [Coditsu](https://coditsu.io) that combines multiple linters and code analyzers for both code and documentation.
+First, thank you for considering contributing to Karafka! It's people like you that make the open source community such a great community!
 
-Unfortunately, it does not yet support independent forks, however you should be fine by looking at what we require.
+Each pull request must pass all the RSpec specs and meet our quality requirements.
 
-Please run:
+To check if everything is as it should be, we use [Coditsu](https://coditsu.io) that combines multiple linters and code analyzers for both code and documentation. Once you're done with your changes, submit a pull request.
 
-```bash
-bundle exec rspec
-```
+Coditsu will automatically check your work against our quality standards. You can find your commit check results on the [builds page](https://app.coditsu.io/karafka/commit_builds) of Karafka organization.
 
-to check if everything is in order. After that you can submit a pull request.
+[![coditsu](https://coditsu.io/assets/quality_bar.svg)](https://app.coditsu.io/karafka/commit_builds)
 
 ## Contributors
 

data/bin/karafka

@@ -10,7 +10,7 @@ else
   # However when it is unavailable, we still want to be able to run help command
   # and install command as they don't require configured app itself to run
   raise(
-    Karafka::Errors::MissingBootFile,
+    Karafka::Errors::MissingBootFileError,
     Karafka.boot_file
   ) unless %w[-h install].include?(ARGV[0])
 end

data/certs/mensfeld.pem

@@ -0,0 +1,25 @@
+-----BEGIN CERTIFICATE-----
+MIIEODCCAqCgAwIBAgIBATANBgkqhkiG9w0BAQsFADAjMSEwHwYDVQQDDBhtYWNp
+ZWovREM9bWVuc2ZlbGQvREM9cGwwHhcNMTkwNzMwMTQ1NDU0WhcNMjAwNzI5MTQ1
+NDU0WjAjMSEwHwYDVQQDDBhtYWNpZWovREM9bWVuc2ZlbGQvREM9cGwwggGiMA0G
+CSqGSIb3DQEBAQUAA4IBjwAwggGKAoIBgQC9fCwtaHZG2SyyNXiH8r0QbJQx/xxl
+dkvwWz9QGJO+O8rEx20FB1Ab+MVkfOscwIv5jWpmk1U9whzDPl1uFtIbgu+sk+Zb
+uQlZyK/DPN6c+/BbBL+RryTBRyvkPLoCVwm7uxc/JZ1n4AI6eF4cCZ2ieZ9QgQbU
+MQs2QPqs9hT50Ez/40GnOdadVfiDDGz+NME2C4ms0BriXwZ1tcRTfJIHe2xjIbbb
+y5qRGfsLKcgMzvLQR24olixyX1MR0s4+Wveq3QL/gBhL4veUcv+UABJA8IJR0kyB
+seHHutusiwZ1v3SjjjW1xLLrc2ARV0mgCb0WaK2T4iA3oFTGLh6Ydz8LNl31KQFv
+94nRd8IhmJxrhQ6dQ/WT9IXoa5S9lfT5lPJeINemH4/6QPABzf9W2IZlCdI9wCdB
+TBaw57MKneGAYZiKjw6OALSy2ltQUCl3RqFl3VP7n8uFy1U987Q5VIIQ3O1UUsQD
+Oe/h+r7GUU4RSPKgPlrwvW9bD/UQ+zF51v8CAwEAAaN3MHUwCQYDVR0TBAIwADAL
+BgNVHQ8EBAMCBLAwHQYDVR0OBBYEFJNIBHdfEUD7TqHqIer2YhWaWhwcMB0GA1Ud
+EQQWMBSBEm1hY2llakBtZW5zZmVsZC5wbDAdBgNVHRIEFjAUgRJtYWNpZWpAbWVu
+c2ZlbGQucGwwDQYJKoZIhvcNAQELBQADggGBAKA4eqko6BTNhlysip6rfBkVTGri
+ZXsL+kRb2hLvsQJS/kLyM21oMlu+LN0aPj3qEFR8mE/YeDD8rLAfruBRTltPNbR7
+xA5eE1gkxY5LfExUtK3b2wPqfmo7mZgfcsMwfYg/tUXw1WpBCnrhAJodpGH6SXmp
+A40qFUZst0vjiOoO+aTblIHPmMJXoZ3K42dTlNKlEiDKUWMRKSgpjjYGEYalFNWI
+hHfCz2r8L2t+dYdMZg1JGbEkq4ADGsAA8ioZIpJd7V4hI17u5TCdi7X5wh/0gN0E
+CgP+nLox3D+l2q0QuQEkayr+auFYkzTCkF+BmEk1D0Ru4mcf3F4CJvEmW4Pzbjqt
+i1tsCWPtJ4E/UUKnKaWKqGbjrjHJ0MuShYzHkodox5IOiCXIQg+1+YSzfXUV6WEK
+KJG/fhg1JV5vVDdVy6x+tv5SQ5ctU0feCsVfESi3rE3zRd+nvzE9HcZ5aXeL1UtJ
+nT5Xrioegu2w1jPyVEgyZgTZC5rvD0nNS5sFNQ==
+-----END CERTIFICATE-----

data/config/errors.yml

@@ -1,6 +1,39 @@
 en:
-  errors:
-    broker_schema?: >
-      has an invalid format.
-      Expected schema, host and port number.
-      Example: kafka://127.0.0.1:9092 or kafka+ssl://127.0.0.1:9092
+  dry_validation:
+    errors:
+      invalid_broker_schema: >
+        has an invalid format
+        Expected schema, host and port number
+        Example: kafka://127.0.0.1:9092 or kafka+ssl://127.0.0.1:9092
+      invalid_certificate: >
+        is not a valid certificate
+      invalid_certificate_from_path: >
+        is not a valid certificate
+      invalid_private_key: >
+        is not a valid private key
+      max_timeout_size_for_exponential: >
+        pause_timeout cannot be more than pause_max_timeout
+      max_wait_time_limit:
+        max_wait_time cannot be more than socket_timeout
+      topics_names_not_unique: >
+        all topic names within a single consumer group must be unique
+      ssl_client_cert_with_ssl_client_cert_key: >
+        Both ssl_client_cert and ssl_client_cert_key need to be provided
+      ssl_client_cert_key_with_ssl_client_cert: >
+        Both ssl_client_cert_key and ssl_client_cert need to be provided
+      ssl_client_cert_chain_with_ssl_client_cert: >
+        Both ssl_client_cert_chain and ssl_client_cert need to be provided
+      ssl_client_cert_chain_with_ssl_client_cert_key: >
+        Both ssl_client_cert_chain and ssl_client_cert_key need to be provided
+      ssl_client_cert_key_password_with_ssl_client_cert_key: >
+        Both ssl_client_cert_key_password and ssl_client_cert_key need to be provided
+      does_not_respond_to_token: >
+        needs to respond to a #token method
+      required_usage_count: >
+        Given topic must be used at least once
+      pid_already_exists: >
+        Pidfile already exists
+      consumer_groups_inclusion: >
+        Unknown consumer group
+      does_not_exist:
+        Given file does not exist or cannot be read

data/karafka.gemspec

@@ -1,10 +1,11 @@
 # frozen_string_literal: true
 
-lib = File.expand_path('../lib', __FILE__)
+lib = File.expand_path('lib', __dir__)
 $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 
 require 'karafka/version'
 
+# rubocop:disable Metrics/BlockLength
 Gem::Specification.new do |spec|
   spec.name        = 'karafka'
   spec.version     = ::Karafka::VERSION
@@ -16,20 +17,28 @@ Gem::Specification.new do |spec|
   spec.description = 'Framework used to simplify Apache Kafka based Ruby applications development'
   spec.license     = 'MIT'
 
-  spec.add_dependency 'activesupport', '>= 5.0'
-  spec.add_dependency 'dry-configurable', '~> 0.7'
-  spec.add_dependency 'dry-validation', '~> 0.11'
-  spec.add_dependency 'envlogic', '~> 1.0'
+  spec.add_dependency 'dry-configurable', '~> 0.8'
+  spec.add_dependency 'dry-inflector', '~> 0.1'
+  spec.add_dependency 'dry-monitor', '~> 0.3'
+  spec.add_dependency 'dry-validation', '~> 1.2'
+  spec.add_dependency 'envlogic', '~> 1.1'
+  spec.add_dependency 'irb', '~> 1.0'
   spec.add_dependency 'multi_json', '>= 1.12'
   spec.add_dependency 'rake', '>= 11.3'
-  spec.add_dependency 'require_all', '>= 1.4'
-  spec.add_dependency 'ruby-kafka', '>= 0.5'
-  spec.add_dependency 'thor', '~> 0.19'
-  spec.add_dependency 'waterdrop', '>= 1.0.1'
+  spec.add_dependency 'ruby-kafka', '>= 0.7.8'
+  spec.add_dependency 'thor', '~> 0.20'
+  spec.add_dependency 'waterdrop', '~> 1.3.0'
+  spec.add_dependency 'zeitwerk', '~> 2.1'
 
-  spec.required_ruby_version = '>= 2.3.0'
+  spec.required_ruby_version = '>= 2.4.0'
 
+  if $PROGRAM_NAME.end_with?('gem')
+    spec.signing_key = File.expand_path('~/.ssh/gem-private_key.pem')
+  end
+
+  spec.cert_chain    = %w[certs/mensfeld.pem]
   spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(spec)/}) }
   spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
   spec.require_paths = %w[lib]
 end
+# rubocop:enable Metrics/BlockLength

data/lib/karafka.rb

@@ -6,16 +6,16 @@
   kafka
   envlogic
   thor
+  forwardable
   fileutils
   multi_json
-  require_all
   dry-configurable
   dry-validation
-  active_support/callbacks
-  active_support/core_ext/hash/indifferent_access
-  active_support/descendants_tracker
-  active_support/inflector
-  karafka/loader
+  dry/events/publisher
+  dry/inflector
+  dry/monitor/notifications
+  dry/core/constants
+  zeitwerk
 ].each(&method(:require))
 
 # Karafka library
@@ -28,14 +28,14 @@ module Karafka
       @logger ||= App.config.logger
     end
 
-    # @return [::Karafka::Monitor] monitor that we want to use. Will use dummy monitor by default
+    # @return [::Karafka::Monitor] monitor that we want to use
     def monitor
       @monitor ||= App.config.monitor
     end
 
     # @return [String] root path of this gem
     def gem_root
-      Pathname.new(File.expand_path('../..', __FILE__))
+      Pathname.new(File.expand_path('..', __dir__))
     end
 
     # @return [String] Karafka app root path (user application path)
@@ -45,13 +45,13 @@ module Karafka
 
     # @return [String] path to Karafka gem root core
     def core_root
-      Pathname.new(File.expand_path('../karafka', __FILE__))
+      Pathname.new(File.expand_path('karafka', __dir__))
     end
 
     # @return [String] path to a default file that contains booting procedure etc
     # @note By default it is a file called 'karafka.rb' but it can be specified as you wish if you
     #   have Karafka that is merged into a Sinatra/Rails app and karafka.rb is taken.
-    #   It will be used for console/controllers/etc
+    #   It will be used for console/consumers/etc
     # @example Standard only-Karafka case
     #   Karafka.boot_file #=> '/home/app_path/karafka.rb'
     # @example Non standard case
@@ -63,6 +63,9 @@ module Karafka
   end
 end
 
-Karafka::Loader.load!(Karafka.core_root)
+Zeitwerk::Loader
+  .for_gem
+  .tap(&:setup)
+  .tap(&:eager_load)
+
 Kafka::Consumer.prepend(Karafka::Patches::RubyKafka)
-Dry::Configurable::Config.prepend(Karafka::Patches::DryConfigurable)

data/lib/karafka/app.rb

@@ -3,37 +3,37 @@
 module Karafka
   # App class
   class App
-    class << self
-      # Sets up the whole configuration
-      # @param [Block] block configuration block
-      def setup(&block)
-        Setup::Config.setup(&block)
-        initialize!
-      end
+    extend Setup::Dsl
 
+    class << self
       # Sets up all the internal components and bootstrap whole app
       # We need to know details about consumers in order to setup components,
       # that's why we don't setup them after std setup is done
-      # @raise [Karafka::Errors::InvalidConfiguration] raised when configuration
-      #   doesn't match with ConfigurationSchema
+      # @raise [Karafka::Errors::InvalidConfigurationError] raised when configuration
+      #   doesn't match with the config contract
       def boot!
+        initialize!
         Setup::Config.validate!
         Setup::Config.setup_components
-      end
-
-      # @return [Karafka::Config] config instance
-      def config
-        Setup::Config.config
+        initialized!
       end
 
       # @return [Karafka::Routing::Builder] consumers builder instance
       def consumer_groups
-        Routing::Builder.instance
+        config.internal.routing_builder
+      end
+
+      # Triggers reload of all cached Karafka app components, so we can use in-process
+      # in-development hot code reloading without Karafka process restart
+      def reload
+        Karafka::Persistence::Consumers.clear
+        Karafka::Persistence::Topics.clear
+        config.internal.routing_builder.reload
       end
 
       Status.instance_methods(false).each do |delegated|
         define_method(delegated) do
-          Status.instance.public_send(delegated)
+          App.config.internal.status.send(delegated)
         end
       end
 
@@ -41,10 +41,11 @@ module Karafka
       %i[
         root
         env
-        logger monitor
+        logger
+        monitor
       ].each do |delegated|
         define_method(delegated) do
-          Karafka.public_send(delegated)
+          Karafka.send(delegated)
         end
       end
     end

data/lib/karafka/attributes_map.rb

@@ -11,18 +11,18 @@ module Karafka
   module AttributesMap
     class << self
       # What settings should go where in ruby-kafka
+      # @return [Hash] hash with proper sections on what to proxy where in Ruby-Kafka
       # @note All other settings will be passed to Kafka.new method invocation.
       #   All elements in this hash are just edge cases
-      # @return [Hash] hash with proper sections on what to proxy where in Ruby-Kafka
-      def config_adapter
+      def api_adapter
         {
           consumer: %i[
             session_timeout offset_commit_interval offset_commit_threshold
-            offset_retention_time heartbeat_interval
+            offset_retention_time heartbeat_interval fetcher_max_queue_size
           ],
-          subscription: %i[start_from_beginning max_bytes_per_partition],
-          consuming: %i[min_bytes max_wait_time],
-          pausing: %i[pause_timeout],
+          subscribe: %i[start_from_beginning max_bytes_per_partition],
+          consumption: %i[min_bytes max_bytes max_wait_time],
+          pause: %i[pause_timeout pause_max_timeout pause_exponential_backoff],
           # All the options that are under kafka config namespace, but are not used
           # directly with kafka api, but from the Karafka user perspective, they are
           # still related to kafka. They should not be proxied anywhere
@@ -32,13 +32,12 @@ module Karafka
 
       # @return [Array<Symbol>] properties that can be set on a per topic level
       def topic
-        (config_adapter[:subscription] + %i[
+        (api_adapter[:subscribe] + %i[
           backend
           name
-          parser
+          deserializer
           responder
           batch_consuming
-          persistent
         ]).uniq
       end
 
@@ -48,17 +47,19 @@ module Karafka
       #   Thanks to this solution, if any new setting is available for ruby-kafka, we just need
       #   to add it to our configuration class and it will be handled automatically.
       def consumer_group
-        # @note We don't ignore the config_adapter[:ignored] values as they should be ignored
+        # @note We don't ignore the api_adapter[:ignored] values as they should be ignored
         #   only when proxying details go ruby-kafka. We use ignored fields internally in karafka
-        ignored_settings = config_adapter[:subscription]
-        defined_settings = config_adapter.values.flatten
+        ignored_settings = api_adapter[:subscribe]
+        defined_settings = api_adapter.values.flatten
         karafka_settings = %i[batch_fetching]
-        # This is a drity and bad hack of dry-configurable to get keys before setting values
+        # This is a dirty and bad hack of dry-configurable to get keys before setting values
         dynamically_proxied = Karafka::Setup::Config
                               ._settings
+                              .settings
                               .find { |s| s.name == :kafka }
                               .value
-                              .instance_variable_get('@klass').settings
+                              .names
+                              .to_a
 
         (defined_settings + dynamically_proxied).uniq + karafka_settings - ignored_settings
       end

data/lib/karafka/backends/inline.rb

@@ -9,8 +9,7 @@ module Karafka
 
       # Executes consume code immediately (without enqueuing)
       def process
-        Karafka.monitor.notice(self.class, params_batch)
-        consume
+        Karafka.monitor.instrument('backends.inline.process', caller: self) { consume }
       end
     end
   end