karafka-web 0.5.2 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9696cc8cfbcb6124155dbac2a3c92cadca69bd1acec727418874ecdd24f49de1
-  data.tar.gz: eb60dcbf83ba9cbd8f542b2e536e2e1c5328ee3fe358b234916575202ee75bba
+  metadata.gz: 1f2269efe1b2e14f38c5265d3a8ffd1e7c1bfb1775b11382b27d2cb7a119de03
+  data.tar.gz: e2656bacf8540ea3854eb0ace8a4e5bf83d3aa7c9e5a7a06d1c36bea50083790
 SHA512:
-  metadata.gz: 133ecc3adc65e9559a317b4148813b95f21a41150a271f754e726d2617f19515e89498322b0b50e808825a9793a78472041618c584aaf3797e80259a2f296782
-  data.tar.gz: bf07811d4f0825f0c9770ae04218c2d73c24eed397296ae54fbeb5f8898a6db750952588426ad45ded5c88d9ce498b3aadf54a202852f4ec8193fd9ff4a159d3
+  metadata.gz: 665fdedafab36bb818a64c6aca5ea17355baca597878c0d5819c56c266228d1c8c14135c2dd27a8fb21d81c08d6e7958e9b89c8b21e2b8eac03a93519d8c0b77
+  data.tar.gz: 6845812a25677375a2d6931785a0ceb8e1d5ac1a454245d924e271aa8993af733b67697aecc055b65feb799c12a337c1e4969024b903b2130d2f5903c6edef3b

data/CHANGELOG.md

@@ -1,12 +1,66 @@
 # Karafka Web changelog
 
+## 0.6.0 (Unreleased)
+- **[Feature]** Introduce producers errors tracking.
+- [Improvement] Display the error origin as a badge to align with consumers view topic assignments.
+- [Improvement] Collect more job metrics for future usage.
+- [Improvement] Normalize order of job columns on multiple views.
+- [Improvement] Improve pagination by providing a "Go to first page" fast button.
+- [Improvement] Provide more explicit info in the consumers view when no consumers running.
+- [Improvement] Validate error reporting with unified error contract.
+- [Improvement] Use estimated errors count for counters presentation taken from the errors topic instead of materialization via consumers states to allow for producers errors tracking.
+- [Improvement] Introduce `schema_version` to error reports.
+- [Improvement] Do not display the dispatched error message offset in the breadcrumb and title as it was confused with the error message content.
+- [Improvement] Display `error_class` value wrapped with code tag.
+- [Improvement] Display error `type` value wrapped with label tag.
+- [Improvement] Include a blurred backtrace for non-Pro error inspection as a form of indication of this Pro feature.
+- [Fix] Fix invalid arrows style in the pagination.
+- [Fix] Fix missing empty `Process name` value in the errors index view.
+- [Fix] Fix potential empty dispatch of consumer metrics.
+- [Fix] Remove confusing part about real time resources from the "Pro feature" page.
+- [Refactor] Cleanup common components for errors extraction.
+- [Refactor] Remove not used and redundant partials.
+- [Maintenance] Require `karafka` `2.1.4` due to fixes in metrics usage for workless flows.
+
+### Upgrade notes
+
+Because of the reporting schema update, it is recommended to:
+
+- First, deploy **all** the Karafka consumer processes (`karafka server`)
+- Deploy the Web update to your web server.
+
+Please note that if you decide to use the updated Web UI with not updated consumers, you may hit a 500 error or offset related data may not be displayed correctly.
+
+#### Disabling producers instrumentation
+
+Producers error tracking **is** enabled by default. If you want to opt out of it, you need to disable the producers' instrumentation by clearing the producers' listeners:
+
+```ruby
+Karafka::Web.setup do |config|
+  # Do not instrument producers with web-ui listeners
+  config.tracking.producers.listeners = []
+end
+```
+
+#### Custom producers instrumentation
+
+By default, Karafka Web-UI instruments only `Karafka.producer`. If you use producers initialized by yourself, you need to connect the listeners to them manually. To do so, run the following code:
+
+```ruby
+::Karafka::Web.config.tracking.producers.listeners.each do |listener|
+  MY_CUSTOM_PRODUCER.monitor.subscribe(listener)
+end
+```
+
+Please make sure **not** to do it for the default `Karafka.producer` because it is instrumented out of the box.
+
 ## 0.5.2 (2023-05-22)
-- Label ActiveJob consumers jobs with `active_job` tag.
-- Label Virtual Partitions consumers with `virtual` tag.
-- Label Long Running Jobs with `long_running_job` tag.
-- Label collapsed Virtual Partition with `collapsed` tag.
-- Display consumer tags always below the consumer class name in Jobs/Consumer Jobs views.
-- Add label with the attempt count on work being retried.
+- [Improvement] Label ActiveJob consumers jobs with `active_job` tag.
+- [Improvement] Label Virtual Partitions consumers with `virtual` tag.
+- [Improvement] Label Long Running Jobs with `long_running_job` tag.
+- [Improvement] Label collapsed Virtual Partition with `collapsed` tag.
+- [Improvement] Display consumer tags always below the consumer class name in Jobs/Consumer Jobs views.
+- [Improvement] Add label with the attempt count on work being retried.
 
 ## 0.5.1 (2023-04-16)
 - [Fix] Use CSP header matching Sidekiq one to ensure styles and js loading (#55)

data/Gemfile.lock

@@ -1,11 +1,11 @@
 PATH
   remote: .
   specs:
-    karafka-web (0.5.2)
+    karafka-web (0.6.0)
       erubi (~> 1.4)
-      karafka (>= 2.0.40, < 3.0.0)
-      karafka-core (>= 2.0.12, < 3.0.0)
-      roda (~> 3.63)
+      karafka (>= 2.1.4, < 3.0.0)
+      karafka-core (>= 2.0.13, < 3.0.0)
+      roda (~> 3.68, >= 3.68)
       tilt (~> 2.0)
 
 GEM
@@ -26,15 +26,15 @@ GEM
     ffi (1.15.5)
     i18n (1.13.0)
       concurrent-ruby (~> 1.0)
-    karafka (2.0.41)
-      karafka-core (>= 2.0.12, < 3.0.0)
+    karafka (2.1.4)
+      karafka-core (>= 2.0.13, < 3.0.0)
       thor (>= 0.20)
-      waterdrop (>= 2.4.10, < 3.0.0)
+      waterdrop (>= 2.5.3, < 3.0.0)
       zeitwerk (~> 2.3)
-    karafka-core (2.0.12)
+    karafka-core (2.0.13)
       concurrent-ruby (>= 1.1)
-      karafka-rdkafka (>= 0.12.1)
-    karafka-rdkafka (0.12.1)
+      karafka-rdkafka (>= 0.12.3)
+    karafka-rdkafka (0.12.3)
       ffi (~> 1.15)
       mini_portile2 (~> 2.6)
       rake (> 12)
@@ -45,7 +45,7 @@ GEM
       rack (>= 3.0.0.beta1)
       webrick
     rake (13.0.6)
-    roda (3.67.0)
+    roda (3.68.0)
       rack
     rspec (3.12.0)
       rspec-core (~> 3.12.0)
@@ -66,12 +66,12 @@ GEM
       simplecov_json_formatter (~> 0.1)
     simplecov-html (0.12.3)
     simplecov_json_formatter (0.1.4)
-    thor (1.2.1)
+    thor (1.2.2)
     tilt (2.1.0)
     tzinfo (2.0.6)
       concurrent-ruby (~> 1.0)
-    waterdrop (2.5.2)
-      karafka-core (>= 2.0.12, < 3.0.0)
+    waterdrop (2.5.3)
+      karafka-core (>= 2.0.13, < 3.0.0)
       zeitwerk (~> 2.3)
     webrick (1.8.1)
     zeitwerk (2.6.8)

data/karafka-web.gemspec

@@ -17,9 +17,9 @@ Gem::Specification.new do |spec|
   spec.licenses    = %w[LGPL-3.0 Commercial]
 
   spec.add_dependency 'erubi', '~> 1.4'
-  spec.add_dependency 'karafka', '>= 2.0.40', '< 3.0.0'
-  spec.add_dependency 'karafka-core', '>= 2.0.12', '< 3.0.0'
-  spec.add_dependency 'roda', '~> 3.63'
+  spec.add_dependency 'karafka', '>= 2.1.4', '< 3.0.0'
+  spec.add_dependency 'karafka-core', '>= 2.0.13', '< 3.0.0'
+  spec.add_dependency 'roda', '~> 3.68', '>= 3.68'
   spec.add_dependency 'tilt', '~> 2.0'
 
   spec.add_development_dependency 'rackup', '~> 0.2'

data/lib/karafka/web/config.rb

@@ -28,16 +28,15 @@ module Karafka
 
       # Tracking and reporting related settings
       setting :tracking do
-        # Collects the metrics we will be dispatching
-        # Tracks and reports the collected metrics
-        setting :reporter, default: Tracking::Reporter.new
-
         # How often should we report data from a single process
         # You may set it to a lower value in development but in production and scale, every
         # 5 seconds should be enough
         setting :interval, default: 5_000
 
         setting :consumers do
+          # Reports the metrics collected in the sampler
+          setting :reporter, default: Tracking::Consumers::Reporter.new
+
           setting :sampler, default: Tracking::Consumers::Sampler.new
 
           setting :listeners, default: [
@@ -51,7 +50,14 @@ module Karafka
         end
 
         setting :producers do
-          setting :listeners, default: []
+          setting :reporter, default: Tracking::Producers::Reporter.new
+
+          setting :sampler, default: Tracking::Producers::Sampler.new
+
+          setting :listeners, default: [
+            Tracking::Producers::Listeners::Errors.new,
+            Tracking::Producers::Listeners::Reporter.new
+          ]
         end
       end
 

data/lib/karafka/web/installer.rb

@@ -10,7 +10,7 @@ module Karafka
       # @param replication_factor [Integer] replication factor we want to use (1 by default)
       def bootstrap!(replication_factor: 1)
         bootstrap_topics!(replication_factor)
-        bootstrap_state!
+        bootstrap_consumers_state!
       end
 
       # Removes all the Karafka topics and creates them again with the same replication factor
@@ -142,7 +142,7 @@ module Karafka
       end
 
       # Creates the initial state record with all values being empty
-      def bootstrap_state!
+      def bootstrap_consumers_state!
         ::Karafka.producer.produce_sync(
           topic: Karafka::Web.config.topics.consumers.states,
           key: Karafka::Web.config.topics.consumers.states,
@@ -151,7 +151,6 @@ module Karafka
             stats: {
               batches: 0,
               messages: 0,
-              errors: 0,
               retries: 0,
               dead: 0,
               busy: 0,

data/lib/karafka/web/tracking/consumers/contracts/consumer_group.rb

@@ -8,7 +8,7 @@ module Karafka
         module Contracts
           # Expected data for each consumer group
           # It's mostly about subscription groups details
-          class ConsumerGroup < BaseContract
+          class ConsumerGroup < Tracking::Contracts::Base
             configure
 
             required(:id) { |val| val.is_a?(String) && !val.empty? }

data/lib/karafka/web/tracking/consumers/contracts/job.rb

@@ -6,7 +6,7 @@ module Karafka
       module Consumers
         module Contracts
           # Contract for the job reporting details
-          class Job < BaseContract
+          class Job < Tracking::Contracts::Base
             configure
 
             required(:consumer) { |val| val.is_a?(String) }
@@ -19,6 +19,9 @@ module Karafka
             required(:committed_offset) { |val| val.is_a?(Integer) }
             required(:type) { |val| %w[consume revoked shutdown].include?(val) }
             required(:tags) { |val| val.is_a?(Karafka::Core::Taggable::Tags) }
+            # -1 can be here for workless flows
+            required(:consumption_lag) { |val| val.is_a?(Integer) && (val >= 0 || val == -1) }
+            required(:processing_lag) { |val| val.is_a?(Integer) && (val >= 0 || val == -1) }
           end
         end
       end

data/lib/karafka/web/tracking/consumers/contracts/partition.rb

@@ -6,7 +6,7 @@ module Karafka
       module Consumers
         module Contracts
           # Partition metrics required for web to operate
-          class Partition < BaseContract
+          class Partition < Tracking::Contracts::Base
             configure
 
             required(:id) { |val| val.is_a?(Integer) && val >= 0 }

data/lib/karafka/web/tracking/consumers/contracts/report.rb

@@ -9,7 +9,7 @@ module Karafka
           #
           # Any outgoing reporting needs to match this format for it to work with the statuses
           # consumer.
-          class Report < BaseContract
+          class Report < Tracking::Contracts::Base
             configure
 
             required(:schema_version) { |val| val.is_a?(String) }

data/lib/karafka/web/tracking/consumers/contracts/subscription_group.rb

@@ -7,7 +7,7 @@ module Karafka
         module Contracts
           # Expected data for each subscription group
           # It's mostly about topics details
-          class SubscriptionGroup < BaseContract
+          class SubscriptionGroup < Tracking::Contracts::Base
             configure
 
             required(:id) { |val| val.is_a?(String) && !val.empty? }

data/lib/karafka/web/tracking/consumers/contracts/topic.rb

@@ -6,7 +6,9 @@ module Karafka
       module Consumers
         module Contracts
           # Expected topic information that needs to go out
-          class Topic < BaseContract
+          class Topic < Tracking::Contracts::Base
+            configure
+
             required(:name) { |val| val.is_a?(String) && !val.empty? }
             required(:partitions) { |val| val.is_a?(Hash) }
 

data/lib/karafka/web/tracking/consumers/listeners/base.rb

@@ -21,9 +21,9 @@ module Karafka
               @sampler ||= ::Karafka::Web.config.tracking.consumers.sampler
             end
 
-            # @return [Object] reported in use
+            # @return [Object] reporter in use
             def reporter
-              @reporter ||= ::Karafka::Web.config.tracking.reporter
+              @reporter ||= ::Karafka::Web.config.tracking.consumers.reporter
             end
           end
         end

data/lib/karafka/web/tracking/consumers/listeners/errors.rb

@@ -7,6 +7,13 @@ module Karafka
         module Listeners
           # Listener related to tracking errors, DLQs, and retries metrics for the Web UI
           class Errors < Base
+            include Tracking::Helpers::ErrorInfo
+
+            # Schema used by consumers error reporting
+            SCHEMA_VERSION = '1.0.0'
+
+            private_constant :SCHEMA_VERSION
+
             # Collects errors info and counts errors
             #
             # @param event [Karafka::Core::Monitoring::Event]
@@ -23,6 +30,7 @@ module Karafka
                 error_class, error_message, backtrace = extract_error_info(event[:error])
 
                 sampler.errors << {
+                  schema_version: SCHEMA_VERSION,
                   type: event[:type],
                   error_class: error_class,
                   error_message: error_message,
@@ -56,13 +64,6 @@ module Karafka
 
             private
 
-            # @return [Object] sampler for the metrics
-            # @note We use this sampler to get basic process details that we want to assign
-            #   to the error
-            def consumer_sampler
-              @consumer_sampler ||= ::Karafka::Web.config.tracking.consumers.sampler
-            end
-
             # @param consumer [::Karafka::BaseConsumer]
             # @return [Hash] hash with consumer specific info for details of error
             def extract_consumer_info(consumer)
@@ -77,43 +78,6 @@ module Karafka
                 tags: consumer.tags
               }
             end
-
-            # Extracts the basic error info
-            #
-            # @param error [StandardError] error that occurred
-            # @return [Array<String, String, String>] array with error name, message and backtrace
-            def extract_error_info(error)
-              app_root = "#{::Karafka.root}/"
-
-              gem_home = if ENV.key?('GEM_HOME')
-                           ENV['GEM_HOME']
-                         else
-                           File.expand_path(File.join(Karafka.gem_root.to_s, '../'))
-                         end
-
-              gem_home = "#{gem_home}/"
-
-              backtrace = error.backtrace || []
-              backtrace.map! { |line| line.gsub(app_root, '') }
-              backtrace.map! { |line| line.gsub(gem_home, '') }
-
-              [
-                error.class.name,
-                extract_exception_message(error),
-                backtrace.join("\n")
-              ]
-            end
-
-            # @param error [StandardError] error that occurred
-            # @return [String] formatted exception message
-            def extract_exception_message(error)
-              error_message = error.message.to_s[0, 10_000]
-              error_message.force_encoding('utf-8')
-              error_message.scrub! if error_message.respond_to?(:scrub!)
-              error_message
-            rescue StandardError
-              '!!! Error message extraction failed !!!'
-            end
           end
         end
       end

data/lib/karafka/web/tracking/consumers/listeners/processing.rb

@@ -147,6 +147,9 @@ module Karafka
             #
             # @param consumer [::Karafka::BaseConsumer] consumer instance
             # @param type [String] job type
+            # @note Be aware, that non consumption jobs may not have any messages (empty) in them
+            #   when certain filters or features are applied. Please refer to the Karafka docs for
+            #   more details.
             def job_details(consumer, type)
               {
                 started_at: float_now,
@@ -154,6 +157,8 @@ module Karafka
                 partition: consumer.partition,
                 first_offset: consumer.messages.metadata.first_offset,
                 last_offset: consumer.messages.metadata.last_offset,
+                processing_lag: consumer.messages.metadata.processing_lag,
+                consumption_lag: consumer.messages.metadata.consumption_lag,
                 committed_offset: consumer.coordinator.seek_offset - 1,
                 consumer: consumer.class.to_s,
                 consumer_group: consumer.topic.consumer_group.id,

data/lib/karafka/web/tracking/consumers/reporter.rb

@@ -0,0 +1,151 @@
+# frozen_string_literal: true
+
+module Karafka
+  module Web
+    module Tracking
+      module Consumers
+        # Reports the collected data about the process and sends it, so we can use it in the UI
+        class Reporter
+          include ::Karafka::Core::Helpers::Time
+          include ::Karafka::Helpers::Async
+
+          # Minimum number of messages to produce to produce them in sync mode
+          # This acts as a small back-off not to overload the system in case we would have
+          # extremely big number of errors happening
+          PRODUCE_SYNC_THRESHOLD = 25
+
+          private_constant :PRODUCE_SYNC_THRESHOLD
+
+          # This mutex is shared between tracker and samplers so there is no case where metrics
+          # would be collected same time tracker reports
+          MUTEX = Mutex.new
+
+          def initialize
+            # Move back so first report is dispatched fast to indicate, that the process is alive
+            @tracked_at = monotonic_now - 10_000
+            @report_contract = Consumers::Contracts::Report.new
+            @error_contract = Tracking::Contracts::Error.new
+          end
+
+          # Dispatches the current state from sampler to appropriate topics
+          #
+          # @param forced [Boolean] should we report bypassing the time frequency or should we
+          #   report only in case we would not send the report for long enough time.
+          def report(forced: false)
+            MUTEX.synchronize do
+              # Start background thread only when needed
+              # This prevents us from starting it too early or for non-consumer processes where
+              # Karafka is being included
+              async_call unless @running
+
+              return unless report?(forced)
+
+              @tracked_at = monotonic_now
+
+              report = sampler.to_report
+
+              @report_contract.validate!(report)
+
+              process_name = report[:process][:name]
+
+              # Report consumers statuses
+              messages = [
+                {
+                  topic: ::Karafka::Web.config.topics.consumers.reports,
+                  payload: report.to_json,
+                  key: process_name,
+                  partition: 0
+                }
+              ]
+
+              # Report errors that occurred (if any)
+              messages += sampler.errors.map do |error|
+                @error_contract.validate!(error)
+
+                {
+                  topic: Karafka::Web.config.topics.errors,
+                  payload: error.to_json,
+                  # Always dispatch errors from the same process to the same partition
+                  key: process_name
+                }
+              end
+
+              return if messages.empty?
+
+              produce(messages)
+
+              # Clear the sampler so it tracks new state changes without previous once impacting
+              # the data
+              sampler.clear
+            end
+          end
+
+          # Reports bypassing frequency check. This can be used to report when state changes in the
+          # process drastically. For example when process is stopping, we want to indicate this as
+          # fast as possible in the UI, etc.
+          def report!
+            report(forced: true)
+          end
+
+          private
+
+          # Reports the process state once in a while
+          def call
+            @running = true
+
+            loop do
+              report
+
+              # We won't track more often anyhow but want to try frequently not to miss a window
+              # We need to convert the sleep interval into seconds for sleep
+              sleep(::Karafka::Web.config.tracking.interval / 1_000 / 10)
+            end
+          end
+
+          # @param forced [Boolean] is this report forced. Forced means that as long as we can
+          #   flush we will flush
+          # @return [Boolean] Should we report or is it not yet time to do so
+          def report?(forced)
+            # We never report in initializing phase because things are not yet fully configured
+            return false if ::Karafka::App.initializing?
+            # We never report in the initialized because server is not yet ready until Karafka is
+            # fully running and some of the things like listeners are not yet available
+            return false if ::Karafka::App.initialized?
+
+            return true if forced
+
+            (monotonic_now - @tracked_at) >= ::Karafka::Web.config.tracking.interval
+          end
+
+          # @return [Object] sampler for the metrics
+          def sampler
+            @sampler ||= ::Karafka::Web.config.tracking.consumers.sampler
+          end
+
+          # Produces messages to Kafka.
+          #
+          # @param messages [Array<Hash>]
+          #
+          # @note We pick either sync or async dependent on number of messages. The trick here is,
+          #   that we do not want to end up overloading the internal queue with messages in case
+          #   someone has a lot of errors from processing or other errors. Producing sync will wait
+          #   for the delivery, hence will slow things down a little bit. On the other hand during
+          #   normal operations we should not have that many messages to dispatch and it should not
+          #   slowdown any processing.
+          def produce(messages)
+            if messages.count >= PRODUCE_SYNC_THRESHOLD
+              ::Karafka.producer.produce_many_sync(messages)
+            else
+              ::Karafka.producer.produce_many_async(messages)
+            end
+          # Since we run this in a background thread, there may be a case upon shutdown, where the
+          # producer is closed right before a potential dispatch. It is not worth dealing with this
+          # and we can just safely ignore this
+          rescue WaterDrop::Errors::ProducerClosedError
+            nil
+          end
+        end
+      end
+    end
+  end
+end

data/lib/karafka/web/tracking/contracts/base.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+module Karafka
+  module Web
+    module Tracking
+      # Namespace for contracts used by consumers and producers tracking
+      module Contracts
+        # Base for all the metric related contracts
+        class Base < ::Karafka::Core::Contractable::Contract
+          class << self
+            # This layer is not for users extensive feedback, thus we can easily use the minimum
+            # error messaging there is.
+            def configure
+              super do |config|
+                config.error_messages = YAML.safe_load(
+                  File.read(
+                    File.join(Karafka::Web.gem_root, 'config', 'locales', 'errors.yml')
+                  )
+                ).fetch('en').fetch('validations').fetch('web')
+              end
+            end
+          end
+
+          # @param data [Hash] data for validation
+          # @return [Boolean] true if all good
+          # @raise [Errors::ContractError] invalid report
+          def validate!(data)
+            super(data, Errors::Tracking::ContractError)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/karafka/web/tracking/contracts/error.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Karafka
+  module Web
+    module Tracking
+      module Contracts
+        # Contract for error reporting
+        # Since producers and consumers report their errors to the same topic, we need to have
+        # a unified contract for both
+        class Error < Base
+          configure
+
+          required(:schema_version) { |val| val.is_a?(String) }
+          required(:type) { |val| val.is_a?(String) && !val.empty? }
+          required(:error_class) { |val| val.is_a?(String) && !val.empty? }
+          required(:error_message) { |val| val.is_a?(String) }
+          required(:backtrace) { |val| val.is_a?(String) }
+          required(:details) { |val| val.is_a?(Hash) }
+          required(:occurred_at) { |val| val.is_a?(Float) }
+
+          nested(:process) do
+            required(:name) { |val| val.is_a?(String) && !val.empty? }
+            # Tags may not be present for producers because they may operate from outside of
+            # karafka taggable process
+            optional(:tags) { |val| val.is_a?(Karafka::Core::Taggable::Tags) }
+          end
+        end
+      end
+    end
+  end
+end