karafka 1.4.4 → 2.1.10

data/README.md

@@ -2,100 +2,98 @@
 
 [![Build Status](https://github.com/karafka/karafka/actions/workflows/ci.yml/badge.svg)](https://github.com/karafka/karafka/actions/workflows/ci.yml)
 [![Gem Version](https://badge.fury.io/rb/karafka.svg)](http://badge.fury.io/rb/karafka)
-[![Join the chat at https://gitter.im/karafka/karafka](https://badges.gitter.im/karafka/karafka.svg)](https://gitter.im/karafka/karafka)
+[![Join the chat at https://slack.karafka.io](https://raw.githubusercontent.com/karafka/misc/master/slack.svg)](https://slack.karafka.io)
 
-**Note**: We're finishing the new Karafka `2.0` but for now, please use `1.4`. All the documentation presented here refers to `1.4`
-..
-Documentation presented here refers to Karafka `1.4`.
+**Note**: Upgrade instructions for migration from Karafka `1.4` to Karafka `2.0` can be found [here](https://karafka.io/docs/Upgrades-2.0/).
 
 ## About Karafka
 
-Framework used to simplify Apache Kafka based Ruby applications development.
-
-Karafka allows you to capture everything that happens in your systems in large scale, providing you with a seamless and stable core for consuming and processing this data, without having to focus on things that are not your business domain.
-
-Karafka not only handles incoming messages but also provides tools for building complex data-flow applications that receive and send messages.
-
-## How does it work
-
-Karafka provides a higher-level abstraction that allows you to focus on your business logic development, instead of focusing on implementing lower level abstraction layers. It provides developers with a set of tools that are dedicated for building multi-topic applications similar to how Rails applications are being built.
-
-### Some things you might wonder about:
+Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework that:
+
+- Has a built-in [Web UI](https://karafka.io/docs/Web-UI-Features/) providing a convenient way to monitor and manage Karafka-based applications.
+- Supports parallel processing in [multiple threads](https://karafka.io/docs/Concurrency-and-multithreading) (also for a [single topic partition](https://karafka.io/docs/Pro-Virtual-Partitions) work)
+- [Automatically integrates](https://karafka.io/docs/Integrating-with-Ruby-on-Rails-and-other-frameworks#integrating-with-ruby-on-rails) with Ruby on Rails
+- Has [ActiveJob backend](https://karafka.io/docs/Active-Job) support (including [ordered jobs](https://karafka.io/docs/Pro-Enhanced-Active-Job#ordered-jobs))
+- Has a seamless [Dead Letter Queue](https://karafka.io/docs/Dead-Letter-Queue/) functionality built-in
+- Supports in-development [code reloading](https://karafka.io/docs/Auto-reload-of-code-changes-in-development)
+- Is powered by [librdkafka](https://github.com/edenhill/librdkafka) (the Apache Kafka C/C++ client library)
+- Has an out-of the box [StatsD/DataDog monitoring](https://karafka.io/docs/Monitoring-and-logging) with a dashboard template.
+
+```ruby
+# Define what topics you want to consume with which consumers in karafka.rb
+Karafka::App.routes.draw do
+  topic 'system_events' do
+    consumer EventsConsumer
+  end
+end
+
+# And create your consumers, within which your messages will be processed
+class EventsConsumer < ApplicationConsumer
+  # Example that utilizes ActiveRecord#insert_all and Karafka batch processing
+  def consume
+    # Store all of the incoming Kafka events locally in an efficient way
+    Event.insert_all messages.payloads
+  end
+end
+```
 
-- You can integrate Karafka with **any** Ruby-based application.
-- Karafka does **not** require Sidekiq or any other third party software (apart from Kafka itself).
-- Karafka works with Ruby on Rails but it is a **standalone** framework that can work without it.
-- Karafka has a **minimal** set of dependencies, so adding it won't be a huge burden for your already existing applications.
-- Karafka processes can be executed for a **given subset** of consumer groups and/or topics, so you can fine tune it depending on your business logic.
+Karafka **uses** threads to handle many messages simultaneously in the same process. It does not require Rails but will integrate tightly with any Ruby on Rails applications to make event processing dead simple.
 
-Karafka based applications can be easily deployed to any type of infrastructure, including those based on:
+## Getting started
 
-* Heroku
-* Capistrano
-* Docker
-* Terraform
+![karafka web ui](https://raw.githubusercontent.com/karafka/misc/master/printscreens/web-ui.png)
 
-## Support
+If you're entirely new to the subject, you can start with our "Kafka on Rails" articles series, which will get you up and running with the terminology and basic ideas behind using Kafka:
 
-Karafka has a [Wiki pages](https://github.com/karafka/karafka/wiki) for almost everything and a pretty decent [FAQ](https://github.com/karafka/karafka/wiki/FAQ). It covers the whole installation, setup, and deployment along with other useful details on how to run Karafka.
+- [Kafka on Rails: Using Kafka with Ruby on Rails – Part 1 – Kafka basics and its advantages](https://mensfeld.pl/2017/11/kafka-on-rails-using-kafka-with-ruby-on-rails-part-1-kafka-basics-and-its-advantages/)
+- [Kafka on Rails: Using Kafka with Ruby on Rails – Part 2 – Getting started with Rails and Kafka](https://mensfeld.pl/2018/01/kafka-on-rails-using-kafka-with-ruby-on-rails-part-2-getting-started-with-ruby-and-kafka/)
 
-If you have any questions about using Karafka, feel free to join our [Gitter](https://gitter.im/karafka/karafka) chat channel.
+If you want to get started with Kafka and Karafka as fast as possible, then the best idea is to visit our [Getting started](https://karafka.io/docs/Getting-Started) guides and the [example apps repository](https://github.com/karafka/example-apps).
 
-## Getting started
+We also maintain many [integration specs](https://github.com/karafka/karafka/tree/master/spec/integrations) illustrating various use-cases and features of the framework.
 
-If you're completely new to the subject, you can start with our "Kafka on Rails" articles series, that will get you up and running with the terminology and basic ideas behind using Kafka:
+### TL;DR (1 minute from setup to publishing and consuming messages)
 
-- [Kafka on Rails: Using Kafka with Ruby on Rails – Part 1 – Kafka basics and its advantages](https://mensfeld.pl/2017/11/kafka-on-rails-using-kafka-with-ruby-on-rails-part-1-kafka-basics-and-its-advantages/)
-- [Kafka on Rails: Using Kafka with Ruby on Rails – Part 2 – Getting started with Ruby and Kafka](https://mensfeld.pl/2018/01/kafka-on-rails-using-kafka-with-ruby-on-rails-part-2-getting-started-with-ruby-and-kafka/)
+**Prerequisites**: Kafka running. You can start it by following instructions from [here](https://karafka.io/docs/Setting-up-Kafka).
 
-If you want to get started with Kafka and Karafka as fast as possible, then the best idea is to just clone our example repository:
+1. Add and install Karafka:
 
 ```bash
-git clone https://github.com/karafka/example-app ./example_app
-```
+# Make sure to install Karafka 2.1
+bundle add karafka --version ">= 2.1.2"
 
-then, just bundle install all the dependencies:
-
-```bash
-cd ./example_app
-bundle install
+bundle exec karafka install
 ```
 
-and follow the instructions from the [example app Wiki](https://github.com/karafka/example-app/blob/master/README.md).
-
-**Note**: you need to ensure, that you have Kafka up and running and you need to configure Kafka seed_brokers in the ```karafka.rb``` file.
-
-If you need more details and know how on how to start Karafka with a clean installation, read the [Getting started page](https://github.com/karafka/karafka/wiki/Getting-started) section of our Wiki.
-
-## Notice
-
-Karafka framework and Karafka team are __not__ related to Kafka streaming service called CloudKarafka in any matter. We don't recommend nor discourage usage of their platform.
+2. Dispatch a message to the example topic using the Rails or Ruby console:
 
-## References
-
-* [Karafka framework](https://github.com/karafka/karafka)
-* [Karafka GitHub Actions](https://github.com/karafka/karafka/actions)
-* [Karafka Coditsu](https://app.coditsu.io/karafka/repositories/karafka)
+```ruby
+Karafka.producer.produce_sync(topic: 'example', payload: { 'ping' => 'pong' }.to_json)
+```
 
-## Note on contributions
+3. Run Karafka server and see the consumption magic happen:
 
-First, thank you for considering contributing to Karafka! It's people like you that make the open source community such a great community!
+```bash
+bundle exec karafka server
 
-Each pull request must pass all the RSpec specs and meet our quality requirements.
+[86d47f0b92f7] Polled 1 message in 1000ms
+[3732873c8a74] Consume job for ExampleConsumer on example started
+{"ping"=>"pong"}
+[3732873c8a74] Consume job for ExampleConsumer on example finished in 0ms
+```
 
-To check if everything is as it should be, we use [Coditsu](https://coditsu.io) that combines multiple linters and code analyzers for both code and documentation. Once you're done with your changes, submit a pull request.
+## Want to Upgrade? LGPL is not for you? Want to help?
 
-Coditsu will automatically check your work against our quality standards. You can find your commit check results on the [builds page](https://app.coditsu.io/karafka/commit_builds) of Karafka organization.
+I also sell Karafka Pro subscriptions. It includes a commercial-friendly license, priority support, architecture consultations, enhanced Web UI and high throughput data processing-related features (virtual partitions, long-running jobs, and more).
 
-[![coditsu](https://coditsu.io/assets/quality_bar.svg)](https://app.coditsu.io/karafka/commit_builds)
+**10%** of the income will be distributed back to other OSS projects that Karafka uses under the hood.
 
-## Contributors
+Help me provide high-quality open-source software. Please see the Karafka [homepage](https://karafka.io/#become-pro) for more details.
 
-This project exists thanks to all the people who contribute.
-<a href="https://github.com/karafka/karafka/graphs/contributors"><img src="https://opencollective.com/karafka/contributors.svg?width=890" /></a>
+## Support
 
-## Sponsors
+Karafka has [Wiki pages](https://karafka.io/docs) for almost everything and a pretty decent [FAQ](https://karafka.io/docs/FAQ). It covers the installation, setup, and deployment, along with other useful details on how to run Karafka.
 
-We are looking for sustainable sponsorship. If your company is relying on Karafka framework or simply want to see Karafka evolve faster to meet your requirements, please consider backing the project.
+If you have questions about using Karafka, feel free to join our [Slack](https://slack.karafka.io) channel.
 
-Please contact [Maciej Mensfeld](mailto:maciej@mensfeld.pl) directly for more details.
+Karafka has [priority support](https://karafka.io/docs/Pro-Support) for technical and architectural questions that is part of the Karafka Pro subscription.

data/bin/benchmarks

@@ -0,0 +1,85 @@
+#!/usr/bin/env ruby
+
+# Runner for running given benchmark cases
+# Some of the cases require pre-populated data and we populate this in places that need it
+# In other cases we generate this data in a background process, so the partitions data stream
+# is consistent and we don't end up consuming huge batches of a single partition.
+
+require 'open3'
+require 'pathname'
+
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..'))
+
+ROOT_PATH = Pathname.new(File.expand_path(File.join(File.dirname(__FILE__), '../')))
+
+# Load all the benchmarks
+benchmarks = Dir[ROOT_PATH.join('spec/benchmarks/**/*.rb')]
+
+# If filter is provided, apply
+benchmarks.delete_if { |name| !name.include?(ARGV[0]) } if ARGV[0]
+
+raise ArgumentError, "No benchmarks with filter: #{ARGV[0]}" if benchmarks.empty?
+
+# We may skip seeding if we are running the benchmarks multiple times, then since we do not
+# commit offsets we can skip generating more data
+if ENV['SEED']
+  require 'spec/benchmarks_helper'
+
+  # We need to setup karafka here to have producer for data seeding
+  setup_karafka
+
+  # This takes some time but needs to run only once per benchmark session
+  puts 'Seeding benchmarks data...'
+
+  producer = Karafka::App.producer
+
+  # We make our data json compatible so we can also benchmark serialization
+  elements = Array.new(100_000) { { a: :b }.to_json }
+
+  # We do not populate data of benchmarks_0_10 as we use it with life-stream data only
+  %w[
+    benchmarks_00_01
+    benchmarks_00_05
+  ].each do |topic_name|
+    partitions_count = topic_name.split('_').last.to_i
+
+    partitions_count.times do |partition|
+      puts "Seeding #{topic_name}:#{partition}"
+
+      elements.each_slice(10_000) do |data_slice|
+        data = data_slice.map do |data|
+          { topic: topic_name, payload: data, partition: partition }
+        end
+
+        producer.buffer_many(data)
+        producer.flush_sync
+      end
+    end
+  end
+end
+
+# Selects requested benchmarks and runs them one after another
+benchmarks.each do |benchmark_path|
+  puts "Running #{benchmark_path.gsub("#{ROOT_PATH}/spec/benchmarks/", '')}"
+
+  benchmark = "bundle exec ruby -r ./spec/benchmarks_helper.rb #{benchmark_path}"
+
+  Open3.popen3(benchmark) do |stdin, stdout, stderr, thread|
+    t1 = Thread.new do
+      while line = stdout.gets
+        puts(line)
+      end
+    rescue IOError
+    end
+
+    t2 = Thread.new do
+      while line = stderr.gets
+        puts(line)
+      end
+    rescue IOError
+    end
+
+    thread.join
+  end
+end

data/bin/create_token

@@ -0,0 +1,22 @@
+#!/usr/bin/env ruby
+
+require 'openssl'
+require 'base64'
+require 'json'
+require 'date'
+
+PRIVATE_KEY_LOCATION = File.join(Dir.home, '.ssh', 'karafka-pro', 'id_rsa')
+
+# Name of the entity that acquires the license
+ENTITY = ARGV[0]
+
+raise ArgumentError, 'Entity missing' if ENTITY.nil? || ENTITY.empty?
+
+pro_token_data = { entity: ENTITY }
+
+# This code uses my private key to generate a new token for Karafka Pro capabilities
+private_key = OpenSSL::PKey::RSA.new(File.read(PRIVATE_KEY_LOCATION))
+
+bin_key = private_key.private_encrypt(pro_token_data.to_json)
+
+puts Base64.encode64(bin_key)

data/bin/integrations

@@ -0,0 +1,297 @@
+#!/usr/bin/env ruby
+
+# Runner to run integration specs in parallel
+
+# Part of integration specs run linear without bundler.
+# If we would run bundle exec when running this code, bundler would inject its own context
+# into them, messing things up heavily
+#
+# Types of specs:
+# - regular - can run in parallel, includes all the helpers
+# - pristine - cannot run in parallel, uses custom bundler but includes helpers
+# - poro - cannot run in parallel, uses custom bundler, does not include any helpers
+raise 'This code needs to be executed WITHOUT bundle exec' if Kernel.const_defined?(:Bundler)
+
+require 'open3'
+require 'fileutils'
+require 'pathname'
+require 'tmpdir'
+require 'etc'
+
+ROOT_PATH = Pathname.new(File.expand_path(File.join(File.dirname(__FILE__), '../')))
+
+# How many child processes with integration specs do we want to run in parallel
+# When the value is high, there's a problem with thread allocation on Github CI, that is why
+# we limit it. Locally we can run a lot of those, as many of them have sleeps and do not use a lot
+# of CPU. Locally we also cannot go beyond certain limit due to how often and how many topics we
+# create in Kafka. With an overloaded system, we start getting timeouts.
+CONCURRENCY = ENV.key?('CI') ? 5 : Etc.nprocessors * 3
+
+# How may bytes do we want to keep from the stdout in the buffer for when we need to print it
+MAX_BUFFER_OUTPUT = 51_200
+
+# Abstraction around a single test scenario execution process
+class Scenario
+  # How long a scenario can run before we kill it
+  # This is a fail-safe just in case something would hang
+  MAX_RUN_TIME = 5 * 60 # 5 minutes tops
+
+  # There are rare cases where Karafka may force shutdown for some of the integration cases
+  # This includes exactly those
+  EXIT_CODES = {
+    default: [0],
+    'consumption/worker_critical_error_behaviour_spec.rb' => [0, 2].freeze,
+    'shutdown/on_hanging_jobs_and_a_shutdown_spec.rb' => [2].freeze,
+    'shutdown/on_hanging_on_shutdown_job_and_a_shutdown_spec.rb' => [2].freeze,
+    'shutdown/on_hanging_listener_and_shutdown_spec.rb' => [2].freeze
+  }.freeze
+
+  private_constant :MAX_RUN_TIME, :EXIT_CODES
+
+  # Creates scenario instance and runs in the background process
+  #
+  # @param path [String] path to the scenarios file
+  def initialize(path)
+    @path = path
+    # First 1024 characters from stdout
+    @stdout_head = ''
+    # Last 1024 characters from stdout
+    @stdout_tail = ''
+  end
+
+  # Starts running given scenario in a separate process
+  def start
+    @stdin, @stdout, @stderr, @wait_thr = Open3.popen3(init_and_build_cmd)
+    @started_at = current_time
+  end
+
+  # @return [String] integration spec name
+  def name
+    @path.gsub("#{ROOT_PATH}/spec/integrations/", '')
+  end
+
+  # @return [Symbol] type of spec
+  def type
+    scenario_dir = File.dirname(@path)
+
+    return :poro if scenario_dir.include?('_poro')
+    return :pristine if scenario_dir.include?('_pristine')
+
+    :regular
+  end
+
+  # @return [Boolean] any spec that is not a regular one should not run in parallel with others
+  def linear?
+    type != :regular
+  end
+
+  # @return [Boolean] did this scenario finished or is it still running
+  def finished?
+    # If the thread is running too long, kill it
+    if current_time - @started_at > MAX_RUN_TIME
+      begin
+        Process.kill('TERM', pid)
+      # It may finish right after we want to kill it, that's why we ignore this
+      rescue Errno::ESRCH
+      end
+    end
+
+    # We read it so it won't grow as we use our default logger that prints to both test.log and
+    # to stdout. Otherwise after reaching the buffer size, it would hang
+    buffer = ''
+    @stdout.read_nonblock(MAX_BUFFER_OUTPUT, buffer, exception: false)
+    @stdout_head = buffer if @stdout_head.empty?
+    @stdout_tail << buffer
+    @stdout_tail = @stdout_tail[-MAX_BUFFER_OUTPUT..-1] || @stdout_tail
+
+    !@wait_thr.alive?
+  end
+
+  # @return [Boolean] did this scenario finish successfully or not
+  def success?
+    expected_exit_codes = EXIT_CODES[name] || EXIT_CODES[:default]
+
+    expected_exit_codes.include?(exit_code)
+  end
+
+  # @return [Integer] pid of the process of this scenario
+  def pid
+    @wait_thr.pid
+  end
+
+  # @return [Integer] exit code of the process running given scenario
+  def exit_code
+    # There may be no exit status if we killed the thread
+    @wait_thr.value&.exitstatus || 123
+  end
+
+  # @return [String] exit status of the process
+  def exit_status
+    @wait_thr.value.to_s
+  end
+
+  # Prints a status report when scenario is finished and stdout if it failed
+  def report
+    if success?
+      print "\e[#{32}m#{'.'}\e[0m"
+    else
+      buffer = ''
+
+      @stderr.read_nonblock(MAX_BUFFER_OUTPUT, buffer, exception: false)
+
+      puts
+      puts "\e[#{31}m#{'[FAILED]'}\e[0m #{name}"
+      puts "Time taken: #{current_time - @started_at} seconds"
+      puts "Exit code: #{exit_code}"
+      puts "Exit status: #{exit_status}"
+      puts @stdout_head
+      puts '...'
+      puts @stdout_tail
+      puts buffer
+      puts
+    end
+  end
+
+  # @return [Float] number of seconds that a given spec took to run
+  def time_taken
+    @finished_at - @started_at
+  end
+
+  # Close all the files that are open, so they do not pile up
+  def close
+    @finished_at = current_time
+    @stdin.close
+    @stdout.close
+    @stderr.close
+  end
+
+  private
+
+  # Sets up a proper environment for a given spec to run and returns the run command
+  # @return [String] run command
+  def init_and_build_cmd
+    case type
+    when :poro
+      scenario_dir = File.dirname(@path)
+      # We copy the spec into a temp dir, not to pollute the spec location with logs, etc
+      temp_dir = Dir.mktmpdir
+      file_name = File.basename(@path)
+
+      FileUtils.cp_r("#{scenario_dir}/.", temp_dir)
+
+      <<~CMD
+        cd #{temp_dir} &&
+        KARAFKA_GEM_DIR=#{ROOT_PATH} \
+        BUNDLE_AUTO_INSTALL=true \
+        PRISTINE_MODE=true \
+        bundle exec ruby #{file_name}
+      CMD
+    when :pristine
+      scenario_dir = File.dirname(@path)
+      # We copy the spec into a temp dir, not to pollute the spec location with logs, etc
+      temp_dir = Dir.mktmpdir
+      file_name = File.basename(@path)
+
+      FileUtils.cp_r("#{scenario_dir}/.", temp_dir)
+
+      <<~CMD
+        cd #{temp_dir} &&
+        KARAFKA_GEM_DIR=#{ROOT_PATH} \
+        BUNDLE_AUTO_INSTALL=true \
+        PRISTINE_MODE=true \
+        bundle exec ruby -r #{ROOT_PATH}/spec/integrations_helper.rb #{file_name}
+      CMD
+    else
+      <<~CMD
+        KARAFKA_GEM_DIR=#{ROOT_PATH} \
+        bundle exec ruby -r ./spec/integrations_helper.rb #{@path}
+      CMD
+    end
+  end
+
+  # @return [Float] current machine time
+  def current_time
+    Process.clock_gettime(Process::CLOCK_MONOTONIC)
+  end
+end
+
+# Load all the specs
+specs = Dir[ROOT_PATH.join('spec/integrations/**/*_spec.rb')]
+
+# If filters is provided, apply
+# Allows to provide several filters one after another and applies all of them
+ARGV.each do |filter|
+  specs.delete_if { |name| !name.include?(filter) }
+end
+
+
+raise ArgumentError, "No integration specs with filters: #{ARGV.join(', ')}" if specs.empty?
+
+# Randomize order
+seed = (ENV['SEED'] || rand(0..10_000)).to_i
+
+puts "Random seed: #{seed}"
+
+scenarios = specs
+            .shuffle(random: Random.new(seed))
+            .map { |integration_test| Scenario.new(integration_test) }
+
+regulars = scenarios.reject(&:linear?)
+linears = scenarios - regulars
+
+active_scenarios = []
+finished_scenarios = []
+
+while finished_scenarios.size < scenarios.size
+  # If we have space to run another scenario, we add it
+  if active_scenarios.size < CONCURRENCY
+    scenario = nil
+    # We can run only one linear at the same time due to concurrency issues within bundler
+    # Since they usually take longer than others, we try to run them as fast as possible when there
+    # is a slot
+    scenario = linears.pop unless active_scenarios.any?(&:linear?)
+    scenario ||= regulars.pop
+
+    if scenario
+      scenario.start
+      active_scenarios << scenario
+    end
+  end
+
+  active_scenarios.select(&:finished?).each do |exited|
+    scenario = active_scenarios.delete(exited)
+    scenario.report
+    scenario.close
+    finished_scenarios << scenario
+  end
+
+  sleep(0.1)
+end
+
+# Report longest scenarios
+puts
+puts "\nLongest scenarios:\n\n"
+
+finished_scenarios.sort_by(&:time_taken).reverse.first(10).each do |long_scenario|
+  puts "[#{'%6.2f' % long_scenario.time_taken}] #{long_scenario.name}"
+end
+
+failed_scenarios = finished_scenarios.reject(&:success?)
+
+if failed_scenarios.empty?
+  puts
+else
+  # Report once more on the failed jobs
+  # This will only list scenarios that failed without printing their stdout here.
+  puts
+  puts "\nFailed scenarios:\n\n"
+
+  failed_scenarios.each do |scenario|
+    puts "\e[#{31}m#{'[FAILED]'}\e[0m #{scenario.name}"
+  end
+
+  puts
+
+  # Exit with 1 if not all scenarios were successful
+  exit 1
+end

data/bin/karafka

@@ -2,18 +2,10 @@
 
 require 'karafka'
 
-# If there is a boot file, we need to require it as we expect it to contain
-# Karafka app setup, routes, etc
-if File.exist?(Karafka.boot_file)
-  require Karafka.boot_file.to_s
-else
-  # However when it is unavailable, we still want to be able to run help command
-  # and install command as they don't require configured app itself to run
-  raise(
-    Karafka::Errors::MissingBootFileError,
-    Karafka.boot_file
-  ) unless %w[-h install].include?(ARGV[0])
-end
+# We set this to indicate, that the process in which we are (whatever it does) was started using
+# our bin/karafka cli
+ENV['KARAFKA_CLI'] = 'true'
 
+Karafka::Cli::Base.load
 Karafka::Cli.prepare
 Karafka::Cli.start

data/bin/rspecs

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+set -e
+
+SPECS_TYPE=regular bundle exec rspec --tag ~type:pro
+SPECS_TYPE=pro bundle exec rspec --tag type:pro

data/bin/scenario

@@ -0,0 +1,29 @@
+#!/usr/bin/env ruby
+
+# Runner for non-parallel execution of a single scenario.
+# It prints all the info stdout, etc and basically replaces itself with the scenario execution.
+# It is useful when we work with a single spec and we need all the debug info
+
+raise 'This code needs to be executed WITHOUT bundle exec' if Kernel.const_defined?(:Bundler)
+
+require 'open3'
+require 'fileutils'
+require 'pathname'
+require 'tmpdir'
+require 'etc'
+
+ROOT_PATH = Pathname.new(File.expand_path(File.join(File.dirname(__FILE__), '../')))
+
+# Load all the specs
+specs = Dir[ROOT_PATH.join('spec/integrations/**/*.rb')]
+
+# If filters is provided, apply
+# Allows to provide several filters one after another and applies all of them
+ARGV.each do |filter|
+  specs.delete_if { |name| !name.include?(filter) }
+end
+
+raise ArgumentError, "No integration specs with filters: #{ARGV.join(', ')}" if specs.empty?
+raise ArgumentError, "Many specs found with filters: #{ARGV.join(', ')}" if specs.size != 1
+
+exec("bundle exec ruby -r #{ROOT_PATH}/spec/integrations_helper.rb #{specs[0]}")

data/bin/stress_many

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+# Runs integration specs in an endless loop
+# This allows us to ensure (after long enough time) that the integrations test suit is stable and
+# that there are no anomalies when running it for a long period of time
+
+set -e
+
+while :
+do
+  clear
+  bin/integrations $1
+done

data/bin/stress_one

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+# Runs a single integration spec in an endless loop
+# This allows us to ensure (after long enough time) that the integration spec is stable and
+# that there are no anomalies when running it for a long period of time
+
+set -e
+
+while :
+do
+  clear
+  bin/scenario $1
+done