karafka 2.0.0.rc2 → 2.0.0.rc5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f4103f2661316cc4e24aab8bf9715472c04f6911b3e56bb3ecdab775a0e32139
-  data.tar.gz: 0174d1cd163ac4fd046e4d432c0a4e2a07e2d57494ada9846ec61b6c73c2e434
+  metadata.gz: 52fde84aac9ffef63ecbc1d367e3bfdc7209f6c2904f2a7142f51b1bd0613002
+  data.tar.gz: 4b4e7838e19505469ec20a654a3d7bb9c2588f351f9ea64fdb05e224b476d562
 SHA512:
-  metadata.gz: b3214a7e89730e932dbfbd3e5463ab0ce592d06589b5940f0987ded07f738d1f13557ae1d8251d8971beae7334c91bc655a8f64dd83049520d7ca889b2519e69
-  data.tar.gz: 803ae22e05ad14025dc4bda185571c586727227154e201d979052223b8270b157d7bd8a9588c0aa074c9d5bf0ef31dba6a1a696c7c392171c585945343d2fd10
+  metadata.gz: 0ebdda184eca4c7dcfe39821125ba7a8645336607300395de792a7c8808cde07f54f5382278946fab58af0e95f3a85ab05be9ed6e7b292fc7692922cb7890747
+  data.tar.gz: 416cb01ed1b6dac0681969ce72505d19855a041e7ceb29e369ed9fc80675755ad1f478a20ee2d1f2c0e925036ba6f08adda727f8ecd3f4f07f39f0be347ae4a1

data/CHANGELOG.md

@@ -1,5 +1,32 @@
 # Karafka framework changelog
 
+## 2.0.0.rc5 (2022-08-01)
+- Improve specs stability
+- Improve forceful shutdown
+- Add support for debug `TTIN` backtrace printing
+- Fix a case where logger listener would not intercept `warn` level
+- Require `rdkafka` >= `0.12`
+- Replace statistics decorator with the one from `karafka-core`
+
+## 2.0.0.rc4 (2022-07-28)
+- Remove `dry-monitor`
+- Use `karafka-core`
+- Improve forceful shutdown resources finalization
+- Cache consumer client name
+
+## 2.0.0.rc3 (2022-07-26)
+- Fix Pro partitioner hash function may not utilize all the threads (#907).
+- Improve virtual partitions messages distribution.
+- Add StatsD/DataDog optional monitoring listener + dashboard template.
+- Validate that Pro consumer is always used for Pro subscription.
+- Improve ActiveJob consumer shutdown behaviour.
+- Change default `max_wait_time` to 1 second.
+- Change default `max_messages` to 100 (#915).
+- Move logger listener polling reporting level to debug when no messages (#916).
+- Improve stability on aggressive rebalancing (multiple rebalances in a short period).
+- Improve specs stability.
+- Allow using `:key` and `:partition_key` for Enhanced Active Job partitioning.
+
 ## 2.0.0.rc2 (2022-07-19)
 - Fix `example_consumer.rb.erb` `#shutdown` and `#revoked` signatures to correct once.
 - Improve the install user experience (print status and created files).
@@ -52,8 +79,8 @@
 - Fix a case where consecutive CTRL+C (non-stop) would case an exception during forced shutdown
 - Add missing `consumer.prepared.error` into `LoggerListener`
 - Delegate partition resuming from the consumers to listeners threads.
-- Add support for Long Running Jobs (LRJ) for ActiveJob [PRO]
-- Add support for Long Running Jobs for consumers [PRO]
+- Add support for Long-Running Jobs (LRJ) for ActiveJob [PRO]
+- Add support for Long-Running Jobs for consumers [PRO]
 - Allow `active_job_topic` to accept a block for extra topic related settings
 - Remove no longer needed logger threads
 - Auto-adapt number of processes for integration specs based on the number of CPUs

data/CONTRIBUTING.md

@@ -1,4 +1,4 @@
-# Contribute
+# Contributing
 
 ## Introduction
 
@@ -8,11 +8,7 @@ We welcome any type of contribution, not only code. You can help with:
 - **QA**: file bug reports, the more details you can give the better (e.g. screenshots with the console open)
 - **Marketing**: writing blog posts, howto's, printing stickers, ...
 - **Community**: presenting the project at meetups, organizing a dedicated meetup for the local community, ...
-- **Code**: take a look at the [open issues](issues). Even if you can't write code, commenting on them, showing that you care about a given issue matters. It helps us triage them.
-
-## Your First Contribution
-
-Working on your first Pull Request? You can learn how from this *free* series, [How to Contribute to an Open Source Project on GitHub](https://egghead.io/series/how-to-contribute-to-an-open-source-project-on-github).
+- **Code**: take a look at the [open issues](https://github.com/karafka/karafka/issues). Even if you can't write code, commenting on them, showing that you care about a given issue matters. It helps us triage them.
 
 ## Submitting code
 
@@ -32,5 +28,5 @@ By sending a pull request to the pro components, you are agreeing to transfer th
 
 ## Questions
 
-If you have any questions, create an [issue](issue) (protip: do a quick search first to see if someone else didn't ask the same question before!).
-You can also reach us at hello@karafka.opencollective.com.
+If you have any questions, create an [issue](https://github.com/karafka/karafka/issues) (protip: do a quick search first to see if someone else didn't ask the same question before!).
+You can also reach us at contact@karafka.io.

data/Gemfile.lock

@@ -1,11 +1,11 @@
 PATH
   remote: .
   specs:
-    karafka (2.0.0.rc2)
-      dry-monitor (~> 0.5)
-      rdkafka (>= 0.10)
+    karafka (2.0.0.rc5)
+      karafka-core (>= 2.0.2, < 3.0.0)
+      rdkafka (>= 0.12)
       thor (>= 0.20)
-      waterdrop (>= 2.3.3, < 3.0.0)
+      waterdrop (>= 2.4.1, < 3.0.0)
       zeitwerk (~> 2.3)
 
 GEM
@@ -23,19 +23,6 @@ GEM
     concurrent-ruby (1.1.10)
     diff-lcs (1.5.0)
     docile (1.4.0)
-    dry-configurable (0.15.0)
-      concurrent-ruby (~> 1.0)
-      dry-core (~> 0.6)
-    dry-core (0.8.0)
-      concurrent-ruby (~> 1.0)
-    dry-events (0.3.0)
-      concurrent-ruby (~> 1.0)
-      dry-core (~> 0.5, >= 0.5)
-    dry-monitor (0.6.1)
-      dry-configurable (~> 0.13, >= 0.13.0)
-      dry-core (~> 0.5, >= 0.5)
-      dry-events (~> 0.2)
-      zeitwerk (~> 2.5)
     factory_bot (6.2.1)
       activesupport (>= 5.0.0)
     ffi (1.15.5)
@@ -43,6 +30,8 @@ GEM
       activesupport (>= 5.0)
     i18n (1.12.0)
       concurrent-ruby (~> 1.0)
+    karafka-core (2.0.2)
+      concurrent-ruby (>= 1.1)
     mini_portile2 (2.8.0)
     minitest (5.16.2)
     rake (13.0.6)
@@ -70,11 +59,10 @@ GEM
     simplecov-html (0.12.3)
     simplecov_json_formatter (0.1.4)
     thor (1.2.1)
-    tzinfo (2.0.4)
+    tzinfo (2.0.5)
       concurrent-ruby (~> 1.0)
-    waterdrop (2.3.3)
-      concurrent-ruby (>= 1.1)
-      dry-monitor (~> 0.5)
+    waterdrop (2.4.1)
+      karafka-core (>= 2.0.2, < 3.0.0)
       rdkafka (>= 0.10)
       zeitwerk (~> 2.3)
     zeitwerk (2.6.0)

data/LICENSE-COMM

@@ -30,7 +30,7 @@ The Open Source version of the Software (“LGPL Version”) is licensed under t
 
 4. Ownership. Notwithstanding anything to the contrary contained herein, except for the limited license rights expressly provided herein, Maciej Mensfeld and its suppliers have and will retain all rights, title and interest (including, without limitation, all patent, copyright, trademark, trade secret and other intellectual property rights) in and to the Software and all copies, modifications and derivative works thereof (including any changes which incorporate any of your ideas, feedback or suggestions). You acknowledge that you are obtaining only a limited license right to the Software, and that irrespective of any use of the words “purchase”, “sale” or like terms hereunder no ownership rights are being conveyed to you under this Agreement or otherwise.
 
-5. Fees and Payment. The Software license fees will be due and payable in full as set forth in the applicable invoice or at the time of purchase. If the Software does not function properly within two weeks of purchase, please contact us within those two weeks for a refund. You shall be responsible for all taxes, withholdings, duties and levies arising from the order (excluding taxes based on the net income of Maciej Mensfeld).
+5. Fees and Payment. The Software license fees will be due and payable in full as set forth in the applicable invoice or at the time of purchase. There are no refunds beyond the remedy refund.
 
 6. Support, Maintenance and Services. Subject to the terms and conditions of this Agreement, as set forth in your invoice, and as set forth on the Karafka Pro support page (https://github.com/karafka/karafka/wiki/Commercial-Support), support and maintenance services may be included with the purchase of your license subscription.
 

data/README.md

@@ -4,11 +4,18 @@
 [![Gem Version](https://badge.fury.io/rb/karafka.svg)](http://badge.fury.io/rb/karafka)
 [![Join the chat at https://slack.karafka.io](https://raw.githubusercontent.com/karafka/misc/master/slack.svg)](https://slack.karafka.io)
 
-**Note**: All of the documentation here refers to Karafka `2.0`. If you are looking for the documentation to Karafka `1.4` please click here (TBA).
+**Note**: All of the documentation here refers to Karafka `2.0.0.rc4` or higher. If you are looking for the documentation for Karafka `1.4`, please click [here](https://github.com/karafka/wiki/tree/1.4).
 
 ## About Karafka
 
-Karafka is a multi-threaded framework used to simplify Apache Kafka based Ruby and Ruby on Rails applications development.
+Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework that:
+
+- Supports parallel processing in [multiple threads](https://github.com/karafka/karafka/wiki/Concurrency-and-multithreading) (also for a [single topic partition](https://github.com/karafka/karafka/wiki/Pro-Virtual-Partitions) work)
+- Has [ActiveJob backend](https://github.com/karafka/karafka/wiki/Active-Job) support (including ordered jobs)
+- [Automatically integrates](https://github.com/karafka/karafka/wiki/Integrating-with-Ruby-on-Rails-and-other-frameworks#integrating-with-ruby-on-rails=) with Ruby on Rails
+- Supports in-development [code reloading](https://github.com/karafka/karafka/wiki/Auto-reload-of-code-changes-in-development)
+- Is powered by [librdkafka](https://github.com/edenhill/librdkafka) (the Apache Kafka C/C++ client library)
+- Has an out-of the box [StatsD/DataDog monitoring](https://github.com/karafka/karafka/wiki/Monitoring-and-logging) with a dashboard template.
 
 ```ruby
 # Define what topics you want to consume with which consumers in karafka.rb
@@ -28,22 +35,51 @@ class EventsConsumer < ApplicationConsumer
 end
 ```
 
-Karafka **uses** threads to handle many messages at the same time in the same process. It does not require Rails but will integrate tightly with any Ruby on Rails applications to make event processing dead simple.
+Karafka **uses** threads to handle many messages simultaneously in the same process. It does not require Rails but will integrate tightly with any Ruby on Rails applications to make event processing dead simple.
 
 ## Getting started
 
-If you're completely new to the subject, you can start with our "Kafka on Rails" articles series, that will get you up and running with the terminology and basic ideas behind using Kafka:
+If you're entirely new to the subject, you can start with our "Kafka on Rails" articles series, which will get you up and running with the terminology and basic ideas behind using Kafka:
 
 - [Kafka on Rails: Using Kafka with Ruby on Rails – Part 1 – Kafka basics and its advantages](https://mensfeld.pl/2017/11/kafka-on-rails-using-kafka-with-ruby-on-rails-part-1-kafka-basics-and-its-advantages/)
-- [Kafka on Rails: Using Kafka with Ruby on Rails – Part 2 – Getting started with Ruby and Kafka](https://mensfeld.pl/2018/01/kafka-on-rails-using-kafka-with-ruby-on-rails-part-2-getting-started-with-ruby-and-kafka/)
+- [Kafka on Rails: Using Kafka with Ruby on Rails – Part 2 – Getting started with Rails and Kafka](https://mensfeld.pl/2018/01/kafka-on-rails-using-kafka-with-ruby-on-rails-part-2-getting-started-with-ruby-and-kafka/)
 
 If you want to get started with Kafka and Karafka as fast as possible, then the best idea is to visit our [Getting started](https://github.com/karafka/karafka/wiki/Getting-started) guides and the [example apps repository](https://github.com/karafka/example-apps).
 
 We also maintain many [integration specs](https://github.com/karafka/karafka/tree/master/spec/integrations) illustrating various use-cases and features of the framework.
 
+### TL;DR (1 minute from setup to publishing and consuming messages)
+
+**Prerequisites**: Kafka running. You can start it by following instructions from [here](https://github.com/karafka/karafka/wiki/Setting-up-Kafka).
+
+1. Add and install Karafka:
+
+```bash
+bundle add karafka -v 2.0.0.rc4
+
+bundle exec karafka install
+```
+
+2. Dispatch a message to the example topic using the Rails or Ruby console:
+
+```ruby
+Karafka.producer.produce_sync(topic: 'example', payload: { 'ping' => 'pong' }.to_json)
+```
+
+3. Run Karafka server and see the consumption magic happen:
+
+```bash
+bundle exec karafka server
+
+[7616dc24-505a-417f-b87b-6bf8fc2d98c5] Polled 1 message in 1000ms
+[dcf3a8d8-0bd9-433a-8f63-b70a0cdb0732] Consume job for ExampleConsumer on example started
+{"ping"=>"pong"}
+[dcf3a8d8-0bd9-433a-8f63-b70a0cdb0732] Consume job for ExampleConsumer on example finished in 0ms
+```
+
 ## Want to Upgrade? LGPL is not for you? Want to help?
 
-I also sell Karafka Pro subscription. It includes commercial-friendly license, priority support, architecture consultations and high throughput data processing-related features (virtual partitions, long running jobs and more).
+I also sell Karafka Pro subscriptions. It includes a commercial-friendly license, priority support, architecture consultations, and high throughput data processing-related features (virtual partitions, long-running jobs, and more).
 
 **20%** of the income will be distributed back to other OSS projects that Karafka uses under the hood.
 
@@ -51,6 +87,8 @@ Help me provide high-quality open-source software. Please see the Karafka [homep
 
 ## Support
 
-Karafka has [Wiki pages](https://github.com/karafka/karafka/wiki) for almost everything and a pretty decent [FAQ](https://github.com/karafka/karafka/wiki/FAQ). It covers the whole installation, setup and deployment along with other useful details on how to run Karafka.
+Karafka has [Wiki pages](https://github.com/karafka/karafka/wiki) for almost everything and a pretty decent [FAQ](https://github.com/karafka/karafka/wiki/FAQ). It covers the installation, setup, and deployment, along with other useful details on how to run Karafka.
+
+If you have questions about using Karafka, feel free to join our [Slack](https://slack.karafka.io) channel.
 
-If you have any questions about using Karafka, feel free to join our [Slack](https://slack.karafka.io) channel.
+Karafka has priority support for technical and architectural questions that is part of the Karafka Pro subscription.

data/bin/integrations

@@ -37,7 +37,7 @@ class Scenario
     'consumption/worker_critical_error_behaviour.rb' => [0, 2].freeze,
     'shutdown/on_hanging_jobs_and_a_shutdown.rb' => [2].freeze,
     'shutdown/on_hanging_on_shutdown_job_and_a_shutdown.rb' => [2].freeze,
-    'shutdown/on_hanging_poll_and_shutdown.rb' => [2].freeze
+    'shutdown/on_hanging_listener_and_shutdown.rb' => [2].freeze
   }.freeze
 
   private_constant :MAX_RUN_TIME, :EXIT_CODES

data/config/errors.yml

@@ -46,7 +46,11 @@ en:
       missing: needs to be present
       dispatch_method_format: needs to be either :produce_async or :produce_sync
       partitioner_format: 'needs to respond to #call'
+      partition_key_type_format: 'needs to be either :key or :partition_key'
 
     test:
       missing: needs to be present
       id_format: needs to be a String
+
+    pro_consumer_group_topic:
+      consumer_format: needs to inherit from Karafka::Pro::BaseConsumer and not Karafka::Consumer

data/docker-compose.yml

@@ -36,10 +36,13 @@ services:
          integrations_17_02:2:1,\
          integrations_18_02:2:1,\
          integrations_19_02:2:1,\
+         integrations_20_02:2:1,\
+         integrations_21_02:2:1,\
          integrations_00_03:3:1,\
          integrations_01_03:3:1,\
          integrations_02_03:3:1,\
          integrations_03_03:3:1,\
+         integrations_04_03:3:1,\
          integrations_00_10:10:1,\
          integrations_01_10:10:1,\
          benchmarks_00_01:1:1,\

data/karafka.gemspec

@@ -12,17 +12,17 @@ Gem::Specification.new do |spec|
   spec.authors     = ['Maciej Mensfeld']
   spec.email       = %w[maciej@mensfeld.pl]
   spec.homepage    = 'https://karafka.io'
-  spec.summary     = 'Ruby framework for working with Apache Kafka'
+  spec.summary     = 'Efficient Kafka processing framework for Ruby and Rails'
   spec.description = 'Framework used to simplify Apache Kafka based Ruby applications development'
   spec.licenses    = ['LGPL-3.0', 'Commercial']
 
-  spec.add_dependency 'dry-monitor', '~> 0.5'
-  spec.add_dependency 'rdkafka', '>= 0.10'
+  spec.add_dependency 'karafka-core', '>= 2.0.2', '< 3.0.0'
+  spec.add_dependency 'rdkafka', '>= 0.12'
   spec.add_dependency 'thor', '>= 0.20'
-  spec.add_dependency 'waterdrop', '>= 2.3.3', '< 3.0.0'
+  spec.add_dependency 'waterdrop', '>= 2.4.1', '< 3.0.0'
   spec.add_dependency 'zeitwerk', '~> 2.3'
 
-  spec.required_ruby_version = '>= 2.6.0'
+  spec.required_ruby_version = '>= 2.7.0'
 
   if $PROGRAM_NAME.end_with?('gem')
     spec.signing_key = File.expand_path('~/.ssh/gem-private_key.pem')

data/lib/karafka/active_job/consumer.rb

@@ -9,6 +9,8 @@ module Karafka
       # @note ActiveJob does not support batches, so we just run one message after another
       def consume
         messages.each do |message|
+          break if Karafka::App.stopping?
+
           ::ActiveJob::Base.execute(
             # We technically speaking could set this as deserializer and reference it from the
             # message instead of using the `#raw_payload`. This is not done on purpose to simplify

data/lib/karafka/connection/client.rb

@@ -69,9 +69,6 @@ module Karafka
           # Put a message to the buffer if there is one
           @buffer << message if message
 
-          # Track time spent on all of the processing and polling
-          time_poll.checkpoint
-
           # Upon polling rebalance manager might have been updated.
           # If partition revocation happens, we need to remove messages from revoked partitions
           # as well as ensure we do not have duplicated due to the offset reset for partitions
@@ -82,6 +79,9 @@ module Karafka
             break
           end
 
+          # Track time spent on all of the processing and polling
+          time_poll.checkpoint
+
           # Finally once we've (potentially) removed revoked, etc, if no messages were returned
           # we can break.
           # Worth keeping in mind, that the rebalance manager might have been updated despite no
@@ -268,13 +268,18 @@ module Karafka
         true
       rescue Rdkafka::RdkafkaError => e
         return false if e.code == :assignment_lost
-        return false if e.code == :no_offset
+        return true if e.code == :no_offset
 
         raise e
       end
 
       # Commits the stored offsets in a sync way and closes the consumer.
       def close
+        # Once client is closed, we should not close it again
+        # This could only happen in case of a race-condition when forceful shutdown happens
+        # and triggers this from a different thread
+        return if @closed
+
         @mutex.synchronize do
           internal_commit_offsets(async: false)
 

data/lib/karafka/connection/listener.rb

@@ -34,6 +34,8 @@ module Karafka
         # We can do this that way because we always first schedule jobs using messages before we
         # fetch another batch.
         @messages_buffer = MessagesBuffer.new(subscription_group)
+        @mutex = Mutex.new
+        @stopped = false
       end
 
       # Runs the main listener fetch loop.
@@ -51,6 +53,25 @@ module Karafka
         fetch_loop
       end
 
+      # Stops the jobs queue, triggers shutdown on all the executors (sync), commits offsets and
+      # stops kafka client.
+      #
+      # @note This method is not private despite being part of the fetch loop because in case of
+      #   a forceful shutdown, it may be invoked from a separate thread
+      #
+      # @note We wrap it with a mutex exactly because of the above case of forceful shutdown
+      def shutdown
+        return if @stopped
+
+        @mutex.synchronize do
+          @stopped = true
+          @executors.clear
+          @coordinators.reset
+          @client.commit_offsets!
+          @client.stop
+        end
+      end
+
       private
 
       # Fetches the data and adds it to the jobs queue.
@@ -239,13 +260,6 @@ module Karafka
         @client.batch_poll until @jobs_queue.empty?(@subscription_group.id)
       end
 
-      # Stops the jobs queue, triggers shutdown on all the executors (sync), commits offsets and
-      # stops kafka client.
-      def shutdown
-        @client.commit_offsets!
-        @client.stop
-      end
-
       # We can stop client without a problem, as it will reinitialize itself when running the
       # `#fetch_loop` again. We just need to remember to also reset the runner as it is a long
       # running one, so with a new connection to Kafka, we need to initialize the state of the

data/lib/karafka/contracts/base.rb

@@ -3,7 +3,7 @@
 module Karafka
   module Contracts
     # Base contract for all Karafka contracts
-    class Base < ::WaterDrop::Contractable::Contract
+    class Base < ::Karafka::Core::Contractable::Contract
       # @param data [Hash] data for validation
       # @return [Boolean] true if all good
       # @raise [Errors::InvalidConfigurationError] invalid configuration error

data/lib/karafka/errors.rb

@@ -24,9 +24,6 @@ module Karafka
     # Raised when we try to use Karafka CLI commands (except install) without a boot file
     MissingBootFileError = Class.new(BaseError)
 
-    # Raised when want to hook up to an event that is not registered and supported
-    UnregisteredMonitorEventError = Class.new(BaseError)
-
     # Raised when we've waited enough for shutting down a non-responsive process
     ForcefulShutdownError = Class.new(BaseError)
 

data/lib/karafka/instrumentation/callbacks/statistics.rb

@@ -16,8 +16,7 @@ module Karafka
           @consumer_group_id = consumer_group_id
           @client_name = client_name
           @monitor = monitor
-          # We decorate both Karafka and WaterDrop statistics the same way
-          @statistics_decorator = ::WaterDrop::Instrumentation::Callbacks::StatisticsDecorator.new
+          @statistics_decorator = ::Karafka::Core::Monitoring::StatisticsDecorator.new
         end
 
         # Emits decorated statistics to the monitor

data/lib/karafka/instrumentation/logger_listener.rb

@@ -9,6 +9,7 @@ module Karafka
       USED_LOG_LEVELS = %i[
         debug
         info
+        warn
         error
         fatal
       ].freeze
@@ -18,7 +19,7 @@ module Karafka
       # @param event [Dry::Events::Event] event details including payload
       def on_connection_listener_fetch_loop(event)
         listener = event[:caller]
-        info "[#{listener.id}] Polling messages..."
+        debug "[#{listener.id}] Polling messages..."
       end
 
       # Logs about messages that we've received from Kafka
@@ -28,7 +29,13 @@ module Karafka
         listener = event[:caller]
         time = event[:time]
         messages_count = event[:messages_buffer].size
-        info "[#{listener.id}] Polled #{messages_count} messages in #{time}ms"
+
+        message = "[#{listener.id}] Polled #{messages_count} messages in #{time}ms"
+
+        # We don't want the "polled 0" in dev as it would spam the log
+        # Instead we publish only info when there was anything we could poll and fail over to the
+        # zero notifications when in debug mode
+        messages_count.zero? ? debug(message) : info(message)
       end
 
       # Prints info about the fact that a given job has started
@@ -54,11 +61,28 @@ module Karafka
         info "[#{job.id}] #{job_type} job for #{consumer} on #{topic} finished in #{time}ms"
       end
 
-      # Logs info about system signals that Karafka received.
+      # Logs info about system signals that Karafka received and prints backtrace for threads in
+      # case of ttin
       #
       # @param event [Dry::Events::Event] event details including payload
       def on_process_notice_signal(event)
         info "Received #{event[:signal]} system signal"
+
+        # We print backtrace only for ttin
+        return unless event[:signal] == :SIGTTIN
+
+        # Inspired by Sidekiq
+        Thread.list.each do |thread|
+          tid = (thread.object_id ^ ::Process.pid).to_s(36)
+
+          warn "Thread TID-#{tid} #{thread['label']}"
+
+          if thread.backtrace
+            warn thread.backtrace.join("\n")
+          else
+            warn '<no backtrace available>'
+          end
+        end
       end
 
       # Logs info that we're initializing Karafka app.

data/lib/karafka/instrumentation/monitor.rb

@@ -1,66 +1,21 @@
 # frozen_string_literal: true
 
 module Karafka
-  # Namespace for all the things related with Karafka instrumentation process
   module Instrumentation
-    # Monitor is used to hookup external monitoring services to monitor how Karafka works
-    # It provides a standardized API for checking incoming messages/enqueueing etc
-    # Since it is a pub-sub based on dry-monitor, you can use as many subscribers/loggers at the
-    # same time, which means that you might have for example file logging and NewRelic at the same
-    # time
-    # @note This class acts as a singleton because we are only permitted to have single monitor
-    #   per running process (just as logger)
-    class Monitor < Dry::Monitor::Notifications
-      # List of events that we support in the system and to which a monitor client can hook up
-      # @note The non-error once support timestamp benchmarking
-      # @note Depending on Karafka extensions and additional engines, this might not be the
-      #   complete list of all the events. Please use the #available_events on fully loaded
-      #   Karafka system to determine all of the events you can use.
-      BASE_EVENTS = %w[
-        app.initialized
-        app.running
-        app.stopping
-        app.stopped
-
-        consumer.consumed
-        consumer.revoked
-        consumer.shutdown
-
-        process.notice_signal
-
-        connection.listener.before_fetch_loop
-        connection.listener.fetch_loop
-        connection.listener.fetch_loop.received
-
-        worker.process
-        worker.processed
-
-        statistics.emitted
-
-        error.occurred
-      ].freeze
-
-      private_constant :BASE_EVENTS
-
-      # @return [Karafka::Instrumentation::Monitor] monitor instance for system instrumentation
-      def initialize
-        super(:karafka)
-        BASE_EVENTS.each(&method(:register_event))
-      end
-
-      # Allows us to subscribe to events with a code that will be yielded upon events
-      # @param event_name_or_listener [String, Object] name of the event we want to subscribe to
-      #   or a listener if we decide to go with object listener
-      def subscribe(event_name_or_listener)
-        return super unless event_name_or_listener.is_a?(String)
-        return super if available_events.include?(event_name_or_listener)
-
-        raise Errors::UnregisteredMonitorEventError, event_name_or_listener
-      end
-
-      # @return [Array<String>] names of available events to which we can subscribe
-      def available_events
-        __bus__.events.keys
+    # Karafka instrumentation monitor that we use to publish events
+    # By default uses our internal notifications bus but can be used with
+    # `ActiveSupport::Notifications` as well
+    class Monitor < ::Karafka::Core::Monitoring::Monitor
+      attr_reader :notifications_bus
+
+      # @param notifications_bus [Object] either our internal notifications bus or
+      #   `ActiveSupport::Notifications`
+      # @param namespace [String, nil] namespace for events or nil if no namespace
+      def initialize(
+        notifications_bus = ::Karafka::Instrumentation::Notifications.new,
+        namespace = nil
+      )
+        super(notifications_bus, namespace)
       end
     end
   end

data/lib/karafka/instrumentation/notifications.rb

@@ -0,0 +1,52 @@
+# frozen_string_literal: true
+
+module Karafka
+  # Namespace for all the things related with Karafka instrumentation process
+  module Instrumentation
+    # Monitor is used to hookup external monitoring services to monitor how Karafka works
+    # It provides a standardized API for checking incoming messages/enqueueing etc
+    # Since it is a pub-sub based on dry-monitor, you can use as many subscribers/loggers at the
+    # same time, which means that you might have for example file logging and NewRelic at the same
+    # time
+    # @note This class acts as a singleton because we are only permitted to have single monitor
+    #   per running process (just as logger)
+    class Notifications < Karafka::Core::Monitoring::Notifications
+      # List of events that we support in the system and to which a monitor client can hook up
+      # @note The non-error once support timestamp benchmarking
+      # @note Depending on Karafka extensions and additional engines, this might not be the
+      #   complete list of all the events. Please use the #available_events on fully loaded
+      #   Karafka system to determine all of the events you can use.
+      EVENTS = %w[
+        app.initialized
+        app.running
+        app.stopping
+        app.stopped
+
+        consumer.consumed
+        consumer.revoked
+        consumer.shutdown
+
+        process.notice_signal
+
+        connection.listener.before_fetch_loop
+        connection.listener.fetch_loop
+        connection.listener.fetch_loop.received
+
+        worker.process
+        worker.processed
+
+        statistics.emitted
+
+        error.occurred
+      ].freeze
+
+      private_constant :EVENTS
+
+      # @return [Karafka::Instrumentation::Monitor] monitor instance for system instrumentation
+      def initialize
+        super
+        EVENTS.each { |event| register_event(event) }
+      end
+    end
+  end
+end