karafka 1.4.12 → 2.0.0.alpha1

data/README.md

@@ -4,15 +4,15 @@
 [![Gem Version](https://badge.fury.io/rb/karafka.svg)](http://badge.fury.io/rb/karafka)
 [![Join the chat at https://slack.karafka.io](https://raw.githubusercontent.com/karafka/misc/master/slack.svg)](https://slack.karafka.io)
 
-**Note**: We're finishing the new Karafka `2.0` but for now, please use `1.4`. All the documentation presented here refers to `1.4`
+**Note**: All of the documentation here refers to Karafka `2.0`. If you are looking for the documentation to Karafka `1.4` please click here (TBA).
 
 ## About Karafka
 
-Framework used to simplify Apache Kafka based Ruby applications development.
+Karafka is a framework used to simplify Apache Kafka based Ruby and Ruby on Rails applications development.
 
 ```ruby
-# Define what topics you want to consume with which consumers
-Karafka::App.consumer_groups.draw do
+# Define what topics you want to consume with which consumers in karafka.rb
+Karafka::App.routes.draw do
   topic 'system_events' do
     consumer EventsConsumer
   end
@@ -23,39 +23,14 @@ class EventsConsumer < ApplicationConsumer
   # Example that utilizes ActiveRecord#insert_all and Karafka batch processing
   def consume
     # Store all of the incoming Kafka events locally in an efficient way
-    Event.insert_all params_batch.payloads
+    Event.insert_all messages.payloads
   end
 end
 ```
 
-Karafka allows you to capture everything that happens in your systems in large scale, providing you with a seamless and stable core for consuming and processing this data, without having to focus on things that are not your business domain.
+Karafka allows you to capture everything that happens in your systems in large scale, providing you with a seamless and stable core for consuming, processing and producing data, without having to focus on things that are not your business domain.
 
-Karafka not only handles incoming messages but also provides tools for building complex data-flow applications that receive and send messages.
-
-## How does it work
-
-Karafka provides a higher-level abstraction that allows you to focus on your business logic development, instead of focusing on implementing lower level abstraction layers. It provides developers with a set of tools that are dedicated for building multi-topic applications similar to how Rails applications are being built.
-
-### Some things you might wonder about:
-
-- You can integrate Karafka with **any** Ruby-based application.
-- Karafka does **not** require Sidekiq or any other third party software (apart from Kafka itself).
-- Karafka works with Ruby on Rails but it is a **standalone** framework that can work without it.
-- Karafka has a **minimal** set of dependencies, so adding it won't be a huge burden for your already existing applications.
-- Karafka processes can be executed for a **given subset** of consumer groups and/or topics, so you can fine tune it depending on your business logic.
-
-Karafka based applications can be easily deployed to any type of infrastructure, including those based on:
-
-* Heroku
-* Capistrano
-* Docker
-* Terraform
-
-## Support
-
-Karafka has [Wiki pages](https://github.com/karafka/karafka/wiki) for almost everything and a pretty decent [FAQ](https://github.com/karafka/karafka/wiki/FAQ). It covers the whole installation, setup, and deployment along with other useful details on how to run Karafka.
-
-If you have any questions about using Karafka, feel free to join our [Gitter](https://gitter.im/karafka/karafka) chat channel.
+Karafka **uses** threads to handle many messages at the same time in the same process. It does not require Rails but will integrate tightly with any Ruby on Rails applications to make event processing dead simple.
 
 ## Getting started
 
@@ -64,30 +39,23 @@ If you're completely new to the subject, you can start with our "Kafka on Rails"
 - [Kafka on Rails: Using Kafka with Ruby on Rails – Part 1 – Kafka basics and its advantages](https://mensfeld.pl/2017/11/kafka-on-rails-using-kafka-with-ruby-on-rails-part-1-kafka-basics-and-its-advantages/)
 - [Kafka on Rails: Using Kafka with Ruby on Rails – Part 2 – Getting started with Ruby and Kafka](https://mensfeld.pl/2018/01/kafka-on-rails-using-kafka-with-ruby-on-rails-part-2-getting-started-with-ruby-and-kafka/)
 
-If you want to get started with Kafka and Karafka as fast as possible, then the best idea is to just clone our example repository:
+If you want to get started with Kafka and Karafka as fast as possible, then the best idea is to visit our [Getting started](https://github.com/karafka/karafka/wiki/Getting-started) guides and the [example apps repository](https://github.com/karafka/example-apps).
 
-```bash
-git clone https://github.com/karafka/example-app ./example_app
-```
+We also maintain many [integration specs](https://github.com/karafka/karafka/tree/master/spec/integrations) illustrating various use-cases and features of the framework.
 
-then, just bundle install all the dependencies:
+## Want to Upgrade? LGPL is not for you? Want to help?
 
-```bash
-cd ./example_app
-bundle install
-```
+I also sell Karafka Pro subscription. It includes commercial-friendly license, priority support, architecture consultations and high throughput data processing-related features (under development).
 
-and follow the instructions from the [example app Wiki](https://github.com/karafka/example-app/blob/master/README.md).
+**20%** of the income will be distributed back to other OSS projects that Karafka uses under the hood.
 
-**Note**: you need to ensure, that you have Kafka up and running and you need to configure Kafka seed_brokers in the ```karafka.rb``` file.
+Help me provide high-quality open-source software. Please see the Karafka [homepage](https://karafka.io) for more details.
 
-If you need more details and know how on how to start Karafka with a clean installation, read the [Getting started page](https://github.com/karafka/karafka/wiki/Getting-started) section of our Wiki.
+## Support
 
-## References
+Karafka has [Wiki pages](https://github.com/karafka/karafka/wiki) for almost everything and a pretty decent [FAQ](https://github.com/karafka/karafka/wiki/FAQ). It covers the whole installation, setup and deployment along with other useful details on how to run Karafka.
 
-* [Karafka framework](https://github.com/karafka/karafka)
-* [Karafka GitHub Actions](https://github.com/karafka/karafka/actions)
-* [Karafka Coditsu](https://app.coditsu.io/karafka/repositories/karafka)
+If you have any questions about using Karafka, feel free to join our [Slack](https://slack.karafka.io) channel.
 
 ## Note on contributions
 

data/bin/benchmarks

@@ -0,0 +1,85 @@
+#!/usr/bin/env ruby
+
+# Runner for running given benchmark cases
+# Some of the cases require pre-populated data and we populate this in places that need it
+# In other cases we generate this data in a background process, so the partitions data stream
+# is consistent and we don't end up consuming huge batches of a single partition.
+
+require 'open3'
+require 'pathname'
+
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..'))
+
+ROOT_PATH = Pathname.new(File.expand_path(File.join(File.dirname(__FILE__), '../')))
+
+# Load all the benchmarks
+benchmarks = Dir[ROOT_PATH.join('spec/benchmarks/**/*.rb')]
+
+# If filter is provided, apply
+benchmarks.delete_if { |name| !name.include?(ARGV[0]) } if ARGV[0]
+
+raise ArgumentError, "No benchmarks with filter: #{ARGV[0]}" if benchmarks.empty?
+
+# We may skip seeding if we are running the benchmarks multiple times, then since we do not
+# commit offsets we can skip generating more data
+if ENV['SEED']
+  require 'spec/benchmarks_helper'
+
+  # We need to setup karafka here to have producer for data seeding
+  setup_karafka
+
+  # This takes some time but needs to run only once per benchmark session
+  puts 'Seeding benchmarks data...'
+
+  producer = Karafka::App.producer
+
+  # We make our data json compatible so we can also benchmark serialization
+  elements = Array.new(100_000) { { a: :b }.to_json }
+
+  # We do not populate data of benchmarks_0_10 as we use it with life-stream data only
+  %w[
+    benchmarks_0_01
+    benchmarks_0_05
+  ].each do |topic_name|
+    partitions_count = topic_name.split('_').last.to_i
+
+    partitions_count.times do |partition|
+      puts "Seeding #{topic_name}:#{partition}"
+
+      elements.each_slice(10_000) do |data_slice|
+        data = data_slice.map do |data|
+          { topic: topic_name, payload: data, partition: partition }
+        end
+
+        producer.buffer_many(data)
+        producer.flush_sync
+      end
+    end
+  end
+end
+
+# Selects requested benchmarks and runs them one after another
+benchmarks.each do |benchmark_path|
+  puts "Running #{benchmark_path.gsub("#{ROOT_PATH}/spec/benchmarks/", '')}"
+
+  benchmark = "bundle exec ruby -r ./spec/benchmarks_helper.rb #{benchmark_path}"
+
+  Open3.popen3(benchmark) do |stdin, stdout, stderr, thread|
+    t1 = Thread.new do
+      while line = stdout.gets
+        puts(line)
+      end
+    rescue IOError
+    end
+
+    t2 = Thread.new do
+      while line = stderr.gets
+        puts(line)
+      end
+    rescue IOError
+    end
+
+    thread.join
+  end
+end

data/bin/create_token

@@ -0,0 +1,28 @@
+#!/usr/bin/env ruby
+
+require 'openssl'
+require 'base64'
+require 'json'
+require 'date'
+
+PRIVATE_KEY_LOCATION = File.join(Dir.home, '.ssh', 'karafka-pro', 'id_rsa')
+
+# Name of the entity that acquires the license
+ENTITY = ARGV[0]
+# Date till which license is valid
+EXPIRES_ON = Date.parse(ARGV[1])
+
+raise ArgumentError, 'Entity missing' if ENTITY.nil? || ENTITY.empty?
+raise ArgumentError, 'Expires on needs to be in the future' if EXPIRES_ON <= Date.today
+
+pro_token_data = {
+  entity: ENTITY,
+  expires_on: EXPIRES_ON
+}
+
+# This code uses my private key to generate a new token for Karafka Pro capabilities
+private_key = OpenSSL::PKey::RSA.new(File.read(PRIVATE_KEY_LOCATION))
+
+bin_key = private_key.private_encrypt(pro_token_data.to_json)
+
+puts Base64.encode64(bin_key)

data/bin/integrations

@@ -0,0 +1,160 @@
+#!/usr/bin/env ruby
+
+# Runner to run integration specs in parallel
+
+require 'open3'
+require 'fileutils'
+require 'pathname'
+
+ROOT_PATH = Pathname.new(File.expand_path(File.join(File.dirname(__FILE__), '../')))
+
+# Raised from the parent process if any of the integration tests fails
+IntegrationTestError = Class.new(StandardError)
+
+# How many child processes with integration specs do we want to run in parallel
+# When the value is high, there's a problem with thread allocation on Github
+CONCURRENCY = 5
+
+# Abstraction around a single test scenario execution process
+class Scenario
+  # How long a scenario can run before we kill it
+  # This is a fail-safe just in case something would hang
+  MAX_RUN_TIME = 60 * 5
+
+  # There are rare cases where Karafka may force shutdown for some of the integration cases
+  # This includes exactly those
+  EXIT_CODES = {
+    default: [0],
+    'consumption/worker_critical_error_behaviour.rb' => [0, 2].freeze,
+    'shutdown/on_hanging_jobs_and_a_shutdown.rb' => [2].freeze,
+    'shutdown/on_hanging_on_shutdown_job_and_a_shutdown.rb' => [2].freeze,
+    'shutdown/on_hanging_poll_and_shutdown.rb' => [2].freeze
+  }.freeze
+
+  private_constant :MAX_RUN_TIME, :EXIT_CODES
+
+  # Creates scenario instance and runs in the background process
+  #
+  # @param path [String] path to the scenarios file
+  def initialize(path)
+    @path = path
+    @stdin, @stdout, @stderr, @wait_thr = Open3.popen3(
+      "bundle exec ruby -r ./spec/integrations_helper.rb #{path}"
+    )
+    @started_at = current_time
+    # Last 1024 characters from stdout
+    @stdout_tail = ''
+  end
+
+  # @return [String] integration spec name
+  def name
+    @path.gsub("#{ROOT_PATH}/spec/integrations/", '')
+  end
+
+  # @return [Boolean] did this scenario finished or is it still running
+  def finished?
+    # If the thread is running too long, kill it
+    if current_time - @started_at > MAX_RUN_TIME
+      @wait_thr.kill
+      Process.kill('TERM', pid)
+    end
+
+    # We read it so it won't grow as we use our default logger that prints to both test.log and
+    # to stdout. Otherwise after reaching the buffer size, it would hang
+    buffer = ''
+    @stdout.read_nonblock(10_240, buffer, exception: false)
+    @stdout_tail << buffer
+    @stdout_tail = @stdout_tail[-10_024..-1] || @stdout_tail
+
+    !@wait_thr.alive?
+  end
+
+  # @return [Integer] pid of the process of this scenario
+  def pid
+    @wait_thr.pid
+  end
+
+  # @return [Integer] exit code of the process running given scenario
+  def exit_code
+    # There may be no exit status if we killed the thread
+    @wait_thr.value&.exitstatus || 123
+  end
+
+  # @return [Boolean] did this scenario finish successfully or not
+  def success?
+    expected_exit_codes = EXIT_CODES[name] || EXIT_CODES[:default]
+
+    expected_exit_codes.include?(exit_code)
+  end
+
+  # Prints a status report when scenario is finished and stdout if it failed
+  def report
+    result = success? ? "\e[#{32}m#{'OK'}\e[0m" : "\e[#{31}m#{'FAILED'}\e[0m"
+
+    puts "#{result} #{name}"
+
+    unless success?
+      puts "Exit code: #{exit_code}"
+      puts @stdout_tail
+      puts @stderr.read
+    end
+  end
+
+  private
+
+  # @return [Float] current machine time
+  def current_time
+    Process.clock_gettime(Process::CLOCK_MONOTONIC)
+  end
+end
+
+# Simple array to keep track of active integration processes thread running with info on which
+# test scenario is running
+active_scenarios = []
+
+# Finished runners
+finished_scenarios = []
+
+# Waits for any of the processes to be finished and tracks exit codes
+#
+# @param active_scenarios [Array] active runners
+# @param finished_scenarios [Hash] finished forks exit codes
+def wait_and_track(active_scenarios, finished_scenarios)
+  exited = active_scenarios.find(&:finished?)
+
+  if exited
+    scenario = active_scenarios.delete(exited)
+
+    scenario.report
+
+    finished_scenarios << scenario
+  else
+    Thread.pass
+  end
+end
+
+# Load all the specs
+specs = Dir[ROOT_PATH.join('spec/integrations/**/*.rb')]
+
+# If filter is provided, apply
+specs.delete_if { |name| !name.include?(ARGV[0]) } if ARGV[0]
+
+raise ArgumentError, "No integration specs with filter: #{ARGV[0]}" if specs.empty?
+
+# Randomize order
+seed = (ENV['SEED'] || rand(0..10_000)).to_i
+
+puts "Random seed: #{seed}"
+
+specs.shuffle(random: Random.new(seed)).each do |integration_test|
+  scenario = Scenario.new(integration_test)
+
+  active_scenarios << scenario
+
+  wait_and_track(active_scenarios, finished_scenarios) until active_scenarios.size < CONCURRENCY
+end
+
+wait_and_track(active_scenarios, finished_scenarios) while !active_scenarios.empty?
+
+# Fail all if any of the tests does not have expected exit code
+raise IntegrationTestError unless finished_scenarios.all?(&:success?)

data/bin/stress

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+# Runs integration specs in an endless loop
+# This allows us to ensure (after long enough time) that the integrations test suit is stable and
+# that there are no anomalies when running it for a long period of time
+
+set -e
+
+while :
+do
+  reset
+  bundle exec bin/integrations $1
+done

data/certs/karafka-pro.pem

@@ -0,0 +1,11 @@
+-----BEGIN RSA PUBLIC KEY-----
+MIIBigKCAYEApcd6ybskiNs9WUvBGVUE8GdWDehjZ9TyjSj/fDl/UcMYqY0R5YX9
+tnYxEwZZRMdVltKWxr88Qmshh1IQz6CpJVbcfYjt/158pSGPm+AUua6tkLqIvZDM
+ocFOMafmroI+BMuL+Zu5QH7HC2tkT16jclGYfMQkJjXVUQTk2UZr+94+8RlUz/CH
+Y6hPA7xPgIyPfyPCxz1VWzAwXwT++NCJQPBr5MqT84LNSEzUSlR9pFNShf3UCUT+
+8LWOvjFSNGmMMSsbo2T7/+dz9/FM02YG00EO0x04qteggwcaEYLFrigDN6/fM0ih
+BXZILnMUqC/qrfW2YFg4ZqKZJuxaALqqkPxrkBDYqoqcAloqn36jBSke6tc/2I/J
+2Afq3r53UoAbUH7h5I/L8YeaiA4MYjAuq724lHlrOmIr4D6yjYC0a1LGlPjLk869
+2nsVXNgomhVb071E6amR+rJJnfvkdZgCmEBFnqnBV5A1u4qgNsa2rVcD+gJRvb2T
+aQtjlQWKPx5xAgMBAAE=
+-----END RSA PUBLIC KEY-----

data/config/errors.yml

@@ -1,41 +1,7 @@
 en:
   dry_validation:
     errors:
-      invalid_broker_schema: >
-        has an invalid format
-        Expected schema, host and port number
-        Example: kafka://127.0.0.1:9092 or kafka+ssl://127.0.0.1:9092
-      invalid_certificate: >
-        is not a valid certificate
-      invalid_certificate_from_path: >
-        is not a valid certificate
-      invalid_private_key: >
-        is not a valid private key
-      max_timeout_size_for_exponential: >
-        pause_timeout cannot be more than pause_max_timeout
-      max_wait_time_limit:
-        max_wait_time cannot be more than socket_timeout
-      topics_names_not_unique: >
-        all topic names within a single consumer group must be unique
-      ssl_client_cert_with_ssl_client_cert_key: >
-        Both ssl_client_cert and ssl_client_cert_key need to be provided
-      ssl_client_cert_key_with_ssl_client_cert: >
-        Both ssl_client_cert_key and ssl_client_cert need to be provided
-      ssl_client_cert_chain_with_ssl_client_cert: >
-        Both ssl_client_cert_chain and ssl_client_cert need to be provided
-      ssl_client_cert_chain_with_ssl_client_cert_key: >
-        Both ssl_client_cert_chain and ssl_client_cert_key need to be provided
-      ssl_client_cert_key_password_with_ssl_client_cert_key: >
-        Both ssl_client_cert_key_password and ssl_client_cert_key need to be provided
-      does_not_respond_to_token: >
-        needs to respond to a #token method
-      required_usage_count: >
-        Given topic must be used at least once
-      pid_already_exists: >
-        Pidfile already exists
-      consumer_groups_inclusion: >
-        Unknown consumer group
-      does_not_exist:
-        Given file does not exist or cannot be read
-      does_not_respond_to_call: >
-        needs to respond to a #call method
+      max_timeout_vs_pause_max_timeout: pause_timeout must be less or equal to pause_max_timeout
+      topics_names_not_unique: all topic names within a single consumer group must be unique
+      required_usage_count: Given topic must be used at least once
+      consumer_groups_inclusion: Unknown consumer group

data/docker-compose.yml

@@ -3,15 +3,23 @@ services:
   zookeeper:
     image: wurstmeister/zookeeper
     ports:
-      - "2181:2181"
+      - '2181:2181'
   kafka:
-    image: wurstmeister/kafka:1.0.1
+    image: wurstmeister/kafka
     ports:
-      - "9092:9092"
+      - '9092:9092'
     environment:
       KAFKA_ADVERTISED_HOST_NAME: localhost
       KAFKA_ADVERTISED_PORT: 9092
       KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
       KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
+      KAFKA_CREATE_TOPICS:
+        "integrations_0_03:3:1,\
+         integrations_1_03:3:1,\
+         integrations_0_10:10:1,\
+         integrations_1_10:10:1,\
+         benchmarks_0_01:1:1,\
+         benchmarks_0_05:5:1,\
+         benchmarks_0_10:10:1"
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock

data/karafka.gemspec

@@ -5,27 +5,24 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 
 require 'karafka/version'
 
-# rubocop:disable Metrics/BlockLength
 Gem::Specification.new do |spec|
   spec.name        = 'karafka'
   spec.version     = ::Karafka::VERSION
   spec.platform    = Gem::Platform::RUBY
-  spec.authors     = ['Maciej Mensfeld', 'Pavlo Vavruk', 'Adam Gwozdowski']
-  spec.email       = %w[maciej@mensfeld.pl pavlo.vavruk@gmail.com adam99g@gmail.com]
+  spec.authors     = ['Maciej Mensfeld']
+  spec.email       = %w[maciej@mensfeld.pl]
   spec.homepage    = 'https://karafka.io'
   spec.summary     = 'Ruby based framework for working with Apache Kafka'
   spec.description = 'Framework used to simplify Apache Kafka based Ruby applications development'
-  spec.license     = 'MIT'
+  spec.license     = 'LGPL-3.0'
 
   spec.add_dependency 'dry-configurable', '~> 0.13'
-  spec.add_dependency 'dry-inflector', '~> 0.2'
   spec.add_dependency 'dry-monitor', '~> 0.5'
   spec.add_dependency 'dry-validation', '~> 1.7'
-  spec.add_dependency 'envlogic', '~> 1.1'
-  spec.add_dependency 'ruby-kafka', '>= 1.3.0'
-  spec.add_dependency 'thor', '>= 1.1'
-  spec.add_dependency 'waterdrop', '~> 1.4'
-  spec.add_dependency 'zeitwerk', '~> 2.4'
+  spec.add_dependency 'rdkafka', '>= 0.10'
+  spec.add_dependency 'thor', '>= 0.20'
+  spec.add_dependency 'waterdrop', '>= 2.1.0', '< 3.0.0'
+  spec.add_dependency 'zeitwerk', '~> 2.3'
 
   spec.required_ruby_version = '>= 2.6.0'
 
@@ -35,8 +32,11 @@ Gem::Specification.new do |spec|
 
   spec.cert_chain    = %w[certs/mensfeld.pem]
   spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(spec)/}) }
-  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.executables   = %w[karafka]
   spec.require_paths = %w[lib]
-  spec.metadata      = { 'source_code_uri' => 'https://github.com/karafka/karafka' }
+
+  spec.metadata = {
+    'source_code_uri' => 'https://github.com/karafka/karafka',
+    'rubygems_mfa_required' => 'true'
+  }
 end
-# rubocop:enable Metrics/BlockLength

data/lib/active_job/consumer.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+module ActiveJob
+  # This is the consumer for ActiveJob that eats the messages enqueued with it one after another.
+  # It marks the offset after each message, so we make sure, none of the jobs is executed twice
+  class Consumer < Karafka::BaseConsumer
+    # Executes the ActiveJob logic
+    # @note ActiveJob does not support batches, so we just run one message after another
+    def consume
+      messages.each do |message|
+        ActiveJob::Base.execute(
+          # We technically speaking could set this as deserializer and reference it from the
+          # message instead of using the `#raw_payload`. This is not done on purpose to simplify
+          # the ActiveJob setup here
+          ActiveSupport::JSON.decode(message.raw_payload)
+        )
+
+        mark_as_consumed(message)
+      end
+    end
+  end
+end

data/lib/active_job/karafka.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+require 'active_job'
+require 'active_job/queue_adapters'
+require 'active_job/consumer'
+require 'active_job/routing_extensions'
+require 'active_job/queue_adapters/karafka_adapter'
+
+module ActiveJob
+  # Namespace for usage simplification outside of Rails where Railtie will not kick in.
+  # That way a require 'active_job/karafka' should be enough to use it
+  module Karafka
+  end
+end
+
+# We extend routing builder by adding a simple wrapper for easier jobs topics defining
+::Karafka::Routing::Builder.include ActiveJob::RoutingExtensions
+::Karafka::Routing::Proxy.include ActiveJob::RoutingExtensions

data/lib/active_job/queue_adapters/karafka_adapter.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+# ActiveJob components to allow for jobs consumption with Karafka
+module ActiveJob
+  # ActiveJob queue adapters
+  module QueueAdapters
+    # Karafka adapter for enqueuing jobs
+    class KarafkaAdapter
+      # Enqueues the job by sending all the payload to a dedicated topic in Kafka that will be
+      # later on consumed by a special ActiveJob consumer
+      #
+      # @param job [Object] job that should be enqueued
+      def enqueue(job)
+        ::Karafka.producer.produce_async(
+          topic: job.queue_name,
+          payload: ActiveSupport::JSON.encode(job.serialize)
+        )
+      end
+
+      # Raises info, that Karafka backend does not support scheduling jobs
+      #
+      # @param _job [Object] job we cannot enqueue
+      # @param _timestamp [Time] time when job should run
+      def enqueue_at(_job, _timestamp)
+        raise NotImplementedError, 'This queueing backend does not support scheduling jobs.'
+      end
+    end
+  end
+end

data/lib/active_job/routing_extensions.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+module ActiveJob
+  # Routing extensions for ActiveJob
+  module RoutingExtensions
+    # This method simplifies routes definition for ActiveJob topics / queues by auto-injecting the
+    # consumer class
+    # @param name [String, Symbol] name of the topic where ActiveJobs jobs should go
+    def active_job_topic(name)
+      topic(name) do
+        consumer ActiveJob::Consumer
+      end
+    end
+  end
+end

data/lib/karafka/app.rb

@@ -6,31 +6,23 @@ module Karafka
     extend Setup::Dsl
 
     class << self
-      # Sets up all the internal components and bootstrap whole app
-      # We need to know details about consumers in order to setup components,
-      # that's why we don't setup them after std setup is done
-      # @raise [Karafka::Errors::InvalidConfigurationError] raised when configuration
-      #   doesn't match with the config contract
-      def boot!
-        initialize!
-        Setup::Config.validate!
-        Setup::Config.setup_components
-        initialized!
-      end
-
-      # @return [Karafka::Routing::Builder] consumers builder instance
+      # @return [Karafka::Routing::Builder] consumers builder instance alias
       def consumer_groups
-        config.internal.routing_builder
+        config
+          .internal
+          .routing_builder
       end
 
-      # Triggers reload of all cached Karafka app components, so we can use in-process
-      # in-development hot code reloading without Karafka process restart
-      def reload
-        Karafka::Persistence::Consumers.clear
-        Karafka::Persistence::Topics.clear
-        config.internal.routing_builder.reload
+      # @return [Array<Karafka::Routing::SubscriptionGroup>] active subscription groups
+      def subscription_groups
+        consumer_groups
+          .active
+          .flat_map(&:subscription_groups)
       end
 
+      # Just a nicer name for the consumer groups
+      alias routes consumer_groups
+
       Status.instance_methods(false).each do |delegated|
         define_method(delegated) do
           App.config.internal.status.send(delegated)
@@ -42,6 +34,7 @@ module Karafka
         root
         env
         logger
+        producer
         monitor
       ].each do |delegated|
         define_method(delegated) do