julewire-karafka 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 32290d2a9bb3d886763322d94fded8636dfe961d1c5378a265293acb36d7b932
+  data.tar.gz: 131b772cff7114eed297cebfa075c785c8b7bfc2afbb480684aab813acefc763
+SHA512:
+  metadata.gz: 56184e4abdb26392059b31b5f29323311c331374350d1c88969f98ce3fb3e4c8d3c1eb2d4dd26913244245777236f5d3d1e796ba661b65c354ac499b62088da8
+  data.tar.gz: 2693077aabaa0e6ad1bd28e9fe176e533e8e860d244fcc943a3a632a34b4972270713206bf594ad72f5c72206a82ed305882cbed3906bd744a5a6970694bbbe6

data/CHANGELOG.md

@@ -0,0 +1,6 @@
+## Unreleased
+
+## 1.0.0 - 2026-06-21
+
+- Initial release: Karafka and WaterDrop event capture, message execution
+  summaries, propagation carriers, and monitor health.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Alexander Grebennik
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,58 @@
+# Julewire Karafka
+
+Karafka and WaterDrop integration for Julewire.
+
+It uses structured integration points only: Karafka monitor events, WaterDrop
+middleware, WaterDrop monitor events, and explicit per-message context
+restoration.
+
+## Quickstart
+
+Consumer events:
+
+```ruby
+class KarafkaApp < Karafka::App
+  setup do |config|
+    Julewire::Karafka.install!(monitor: config.monitor)
+  end
+end
+```
+
+Message processing:
+
+```ruby
+def consume
+  messages.each do |message|
+    Julewire::Karafka.with_message(message) do
+      process(message)
+      mark_as_consumed(message)
+    end
+  end
+end
+```
+
+Producer headers and WaterDrop events:
+
+```ruby
+Julewire::Karafka.install!(consumer: false, producer: producer)
+```
+
+Default behavior:
+
+- important consumer and producer monitor events become point records
+- message headers carry Julewire propagation carriers
+- `with_message` restores message context and adds message attributes
+- Karafka fork hooks call `Julewire.after_fork!`
+- text logger listeners are not parsed or deduplicated
+
+Inbound Kafka carriers are trusted by default for internal service traffic. Set
+`carrier_filter` when consuming topics that external producers can write to.
+
+## Docs
+
+- [Configuration](docs/configuration.md)
+- [Advanced Configuration](docs/advanced-configuration.md)
+- [Consumer Events](docs/consumer-events.md)
+- [Message Context](docs/message-context.md)
+- [WaterDrop](docs/waterdrop.md)
+- [Silencing](docs/silencing.md)

data/docs/advanced-configuration.md

@@ -0,0 +1,8 @@
+# Advanced Configuration
+
+| Option | Default | Purpose |
+| --- | --- | --- |
+| `carrier_key` | `Julewire::Core::Propagation::Carrier::DEFAULT_KEY` | Carrier key inside propagation envelopes. |
+
+Change `carrier_key` only when an existing Kafka header contract requires a
+different envelope key.

data/docs/configuration.md

@@ -0,0 +1,49 @@
+# Configuration
+
+## Default Path
+
+| Option | Default | Purpose |
+| --- | --- | --- |
+| `enabled` | `true` | Install the integration. |
+| `consumer_events` | `true` | Subscribe Karafka monitor events. |
+| `producer_events` | `true` | Subscribe WaterDrop monitor events. |
+| `propagation` | `true` | Inject and restore Julewire carriers. |
+| `source` | `"karafka"` | Source value on Karafka and WaterDrop records. |
+
+## Common Knobs
+
+| Option | Default | Purpose |
+| --- | --- | --- |
+| `carrier_max_bytes` | `nil` | Omit oversized carriers from Kafka headers. |
+| `carrier_filter` | `nil` | Optional inbound header filter before restoring a message carrier. |
+| `consumer_event_names` | `:important` | Consumer monitor event profile or explicit event list. |
+| `producer_event_names` | `:important` | Producer monitor event profile or explicit event list. |
+
+`consumer_event_names` and `producer_event_names` accept:
+
+- `:important`
+- `:all`
+- an explicit event-name list
+
+`:all` uses the monitor's registered event list when available. If the monitor
+cannot expose that list, Julewire uses the important profile.
+
+Example:
+
+```ruby
+Julewire::Karafka.configure do |config|
+  config.consumer_event_names = :all
+  config.carrier_max_bytes = 16_384
+end
+```
+
+`carrier_filter` receives `(headers, message:)` and must return the headers to
+trust. Leave it unset for internal topics. Return `{}` for untrusted messages:
+
+```ruby
+Julewire::Karafka.configure do |config|
+  config.carrier_filter = ->(headers, message:) {
+    internal_topic?(message[:topic]) ? headers : {}
+  }
+end
+```

data/docs/consumer-events.md

@@ -0,0 +1,33 @@
+# Consumer Events
+
+`Julewire::Karafka.install!` subscribes to an `:important` event profile by default.
+Set `consumer_event_names = :all` to subscribe to Karafka's registered monitor
+events, or pass an explicit event list for application policy. If the monitor
+cannot expose its registered events, `:all` uses the important profile.
+
+`install!` also subscribes lightweight handlers for
+`swarm.node.after_fork` and `swarm.manager.after_fork`. Those handlers call
+`Julewire.after_fork!` so inherited mutexes, counters, async transports, and
+process-local context are reset in Karafka forked processes.
+
+Monitor events use the severity exposed by the event payload when Karafka
+provides one. Otherwise the listener follows Karafka's logger-listener severity
+conventions: normal lifecycle/consumer events are info, polling and swarm
+control events are debug, selected fatal framework errors are fatal, and other
+errors are error.
+
+Consumer batch lifecycle records such as `consumer.consumed` include batch
+metadata when Karafka exposes it: consumer class/id, group ids, topic,
+partition, message count, offset range, and lag metrics.
+
+Generic Kafka metadata also appears in the record's `neutral` section as
+`messaging.*` formatter-coordination fields. Full Karafka or WaterDrop event
+metadata remains under `attributes.karafka` or `attributes.waterdrop`.
+
+These records describe the batch lifecycle. They are not the propagation
+boundary. Generic array values in monitor payloads are summarized as
+`{ count: n }` to keep records bounded.
+
+Karafka `error.occurred` records use Julewire's normal top-level `error` shape,
+including the core backtrace policy. Error event attributes still carry Karafka
+metadata such as event type and consumer/batch details.

data/docs/message-context.md

@@ -0,0 +1,54 @@
+# Message Context
+
+Kafka batches can contain messages from different upstream requests. Do not use
+batch events as the propagation boundary. Wrap each message while processing it:
+
+```ruby
+def consume
+  messages.each do |message|
+    Julewire::Karafka.with_message(message) do
+      process(message)
+      mark_as_consumed(message)
+    end
+  end
+end
+```
+
+`with_message` restores the Julewire carrier from that message's headers and
+adds message attributes for records emitted inside the block. It does not emit a
+success log or summary by default.
+
+Generic Kafka metadata appears in the record's `neutral` section as
+`messaging.*` formatter-coordination fields. Full Karafka message metadata
+remains under `attributes.karafka`.
+
+By default, inbound Kafka carriers are trusted because most Kafka traffic is
+internal service-to-service traffic. Set `carrier_filter` when consuming topics
+that external producers can write to.
+
+Code inside the block can enqueue jobs, produce messages, or emit records. Core
+propagation carries upstream context and execution metadata, not the Karafka
+message attributes.
+
+Use `with_message_execution` only when consumer code chooses an explicit message
+unit of work and wants a summary for that block:
+
+```ruby
+def consume
+  messages.each do |message|
+    Julewire::Karafka.with_message_execution(message) do
+      process(message)
+    end
+  end
+end
+```
+
+This is intentionally opt-in. Batch-level summaries can conflate unrelated
+upstream request contexts, and per-message summaries are useful only when the
+consumer treats each message as an observable unit of work.
+
+`with_message_execution` accepts `type:`, `id:`, `emit_summary:`,
+`summary_event:`, `summary_severity:`, and `summary_source:`. Other keyword
+arguments become execution fields. By default, type is `:karafka_message`, id is
+`"topic:partition:offset"` when those message fields are present, and the
+summary event is `"message.completed"`.

data/docs/silencing.md

@@ -0,0 +1,35 @@
+# Silencing
+
+Karafka and WaterDrop logger listeners are application-installed monitor
+listeners, not global framework subscribers. The clean replacement is to remove
+the native logger-listener subscription where the application installs it:
+
+```ruby
+Karafka.monitor.subscribe(Karafka::Instrumentation::LoggerListener.new)
+```
+
+Replace that with:
+
+```ruby
+Julewire::Karafka.install!(monitor: Karafka.monitor)
+```
+
+If application boot code needs to keep the native listener for some
+environments, keep a reference and unsubscribe that same listener before
+installing Julewire:
+
+```ruby
+native_logger = Karafka::Instrumentation::LoggerListener.new
+Karafka.monitor.subscribe(native_logger)
+
+Karafka.monitor.unsubscribe(native_logger)
+Julewire::Karafka.install!(monitor: Karafka.monitor)
+```
+
+For WaterDrop, do the same with `WaterDrop::Instrumentation::LoggerListener`
+on the producer monitor and use
+`Julewire::Karafka.install!(consumer: false, producer: producer)`.
+
+This gem does not parse text logs or filter duplicate logger messages. Explicit
+logger calls made by application code remain normal Julewire `event: "log"`
+records.

data/docs/waterdrop.md

@@ -0,0 +1,24 @@
+# WaterDrop
+
+Add the middleware before producing messages:
+
+```ruby
+Julewire::Karafka.install!(consumer: false, producer: producer)
+```
+
+The middleware injects the current Julewire propagation carrier into message
+headers. The listener turns important WaterDrop monitor events into Julewire
+point records.
+
+Set `producer_event_names = :all` for every WaterDrop monitor event exposed by
+the monitor, or pass an explicit event-name list. If the monitor cannot expose
+its registered events, `:all` uses the important profile. WaterDrop event
+severity is read from the event payload when present; otherwise errors are
+error and normal producer events are info.
+
+Set `carrier_max_bytes` to omit oversized propagation carriers from Kafka
+headers. When omitted, the message is still produced normally; it just does not
+carry upstream Julewire context.
+
+Message keys, headers, and payload fields are raw unless an application installs
+a processor policy before output.

data/julewire-karafka.gemspec

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+require_relative "lib/julewire/karafka/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "julewire-karafka"
+  spec.version = Julewire::Karafka::VERSION
+  spec.authors = ["Alexander Grebennik"]
+  spec.email = ["slbug@users.noreply.github.com", "sl.bug.sl@gmail.com"]
+
+  spec.summary = "Karafka and WaterDrop integration for Julewire structured logging."
+  spec.description =
+    "Karafka monitor event capture, per-message context restoration, " \
+    "and WaterDrop propagation middleware for Julewire."
+  spec.homepage = "https://github.com/slbug/julewire"
+  spec.license = "MIT"
+  spec.required_ruby_version = ">= 3.4"
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = "https://github.com/slbug/julewire/tree/main/gems/karafka"
+  spec.metadata["changelog_uri"] = "https://github.com/slbug/julewire/blob/main/gems/karafka/CHANGELOG.md"
+
+  spec.metadata["rubygems_mfa_required"] = "true"
+
+  spec.files = Dir.chdir(__dir__) do
+    Dir[
+      "CHANGELOG.md",
+      "LICENSE.txt",
+      "README.md",
+      "docs/**/*.md",
+      "julewire-karafka.gemspec",
+      "lib/**/*.rb"
+    ]
+  end
+  spec.executables = []
+  spec.require_paths = ["lib"]
+
+  spec.add_dependency "julewire-core", ">= 1.0"
+  spec.add_dependency "karafka", ">= 2.5"
+  spec.add_dependency "waterdrop", ">= 2.10"
+  spec.add_dependency "zeitwerk", ">= 2.8.1"
+end

data/lib/julewire/karafka/configuration.rb

@@ -0,0 +1,52 @@
+# frozen_string_literal: true
+
+module Julewire
+  module Karafka
+    class Configuration
+      IMPORTANT_CONSUMER_EVENT_NAMES = %w[
+        app.initialized
+        app.running
+        app.quiet
+        app.stopped
+        consumer.consumed
+        consumer.revoked
+        consumer.shutdown
+        dead_letter_queue.dispatched
+        filtering.throttled
+        rebalance.partitions_assigned
+        rebalance.partitions_revoked
+        swarm.node.after_fork
+        worker.completed
+        error.occurred
+      ].freeze
+
+      IMPORTANT_PRODUCER_EVENT_NAMES = %w[
+        producer.connected
+        producer.closed
+        message.produced_async
+        message.produced_sync
+        message.acknowledged
+        messages.produced_async
+        messages.produced_sync
+        transaction.committed
+        transaction.aborted
+        buffer.flushed_async
+        buffer.flushed_sync
+        error.occurred
+      ].freeze
+
+      include Julewire::Core::Integration::Settings
+
+      setting :enabled, default: true, predicate: true
+      setting :consumer_events, default: true, predicate: true
+      setting :producer_events, default: true, predicate: true
+      setting :propagation, default: true, predicate: true
+      setting :carrier_filter
+      setting :carrier_key, default: Julewire::Core::Propagation::Carrier::DEFAULT_KEY
+      setting :carrier_max_bytes, validate: byte_limit
+      setting :source, default: "karafka"
+      setting :consumer_event_names, default: :important
+      setting :producer_event_names, default: :important
+    end
+  end
+end

data/lib/julewire/karafka/event_payload.rb

@@ -0,0 +1,91 @@
+# frozen_string_literal: true
+
+require "time"
+
+module Julewire
+  module Karafka
+    module EventPayload
+      CONSUMER_BATCH_EVENTS = %w[
+        consumer.consume
+        consumer.consumed
+      ].freeze
+      CONSUMER_ERROR_TYPE_PATTERN = /\Aconsumer\..*\.error\z/
+      private_constant :CONSUMER_BATCH_EVENTS, :CONSUMER_ERROR_TYPE_PATTERN
+
+      class << self
+        def call(name, event)
+          payload = event_payload(event)
+          return {} unless payload.is_a?(Hash)
+
+          payload.each_with_object(consumer_payload(name, payload)) do |(key, value), result|
+            safe = safe_value(key, value)
+            result[key] = safe unless safe.nil?
+          end
+        rescue StandardError => e
+          { payload_error: { exception_class: e.class.name } }
+        end
+
+        def event_payload(event)
+          return event.payload if event.respond_to?(:payload)
+
+          event.to_h if event.respond_to?(:to_h)
+        end
+
+        def error(event)
+          payload = event_payload(event)
+
+          PayloadReader.value(payload, :error) || PayloadReader.value(payload, :exception)
+        rescue StandardError
+          nil
+        end
+
+        def consumer_payload(name, payload)
+          return {} unless consumer_batch_event?(name, payload)
+
+          PayloadReader.consumer_payload(payload)
+        end
+
+        def consumer_batch_event?(name, payload)
+          event_name = name.to_s
+          return true if CONSUMER_BATCH_EVENTS.include?(event_name)
+          return false unless event_name == "error.occurred"
+
+          type = PayloadReader.value(payload, :type).to_s
+          type.match?(CONSUMER_ERROR_TYPE_PATTERN)
+        end
+
+        def safe_value(key, value)
+          case key.to_sym
+          when :caller
+            { class: value.class.name }
+          when :message
+            PayloadReader.message_payload(value)
+          when :messages
+            { count: Array(value).size }
+          when :error, :exception
+            nil
+          else
+            primitive_value(value)
+          end
+        end
+
+        def primitive_value(value)
+          case value
+          when nil, true, false, String, Symbol, Numeric
+            value
+          when Time
+            value.utc.iso8601(9)
+          when Hash
+            value.transform_keys(&:to_sym)
+          when Array
+            { count: value.size }
+          else
+            { class: value.class.name }
+          end
+        rescue StandardError
+          { class: value.class.name }
+        end
+      end
+    end
+  end
+end

data/lib/julewire/karafka/event_severity.rb

@@ -0,0 +1,108 @@
+# frozen_string_literal: true
+
+module Julewire
+  module Karafka
+    module EventSeverity
+      FATAL_ERROR_TYPES = %w[
+        runner.call.error
+        swarm.supervisor.error
+        worker.process.error
+      ].freeze
+
+      DEBUG_CONSUMER_EVENTS = %w[
+        connection.listener.fetch_loop
+        statistics.emitted
+        swarm.manager.before_fork
+        swarm.manager.control
+      ].freeze
+
+      ERROR_CONSUMER_EVENTS = %w[
+        swarm.manager.stopping
+        swarm.manager.terminating
+      ].freeze
+
+      DEBUG_PRODUCER_EVENTS = %w[
+        oauthbearer.token_refresh
+        statistics.emitted
+      ].freeze
+
+      class << self
+        def consumer(name, event:, payload:)
+          payload_severity(payload) || default_consumer_severity(name.to_s, event, payload)
+        end
+
+        def producer(name, payload)
+          payload_severity(payload) || default_producer_severity(name.to_s)
+        end
+
+        def payload_severity(payload)
+          value = payload_value(payload, :severity) || payload_value(payload, :level)
+          Julewire::Core::Records::Severity.normalize(value) if value
+        rescue StandardError
+          nil
+        end
+
+        def default_consumer_severity(name, event, payload)
+          return error_severity(event, payload) if name == "error.occurred"
+          return fetch_loop_received_severity(event, payload) if name == "connection.listener.fetch_loop.received"
+          return notice_signal_severity(event, payload) if name == "process.notice_signal"
+          return :error if ERROR_CONSUMER_EVENTS.include?(name)
+          return :debug if DEBUG_CONSUMER_EVENTS.include?(name)
+
+          :info
+        end
+
+        def default_producer_severity(name)
+          return :error if name == "error.occurred"
+          return :debug if DEBUG_PRODUCER_EVENTS.include?(name)
+
+          :info
+        end
+
+        def error_severity(event, payload)
+          type = event_value(event, payload, :type).to_s
+          FATAL_ERROR_TYPES.include?(type) ? :fatal : :error
+        end
+
+        def fetch_loop_received_severity(event, payload)
+          messages = event_value(event, payload, :messages_buffer)
+          count = collection_count(messages)
+
+          count&.zero? ? :debug : :info
+        end
+
+        def notice_signal_severity(event, payload)
+          signal = event_value(event, payload, :signal).to_s.upcase
+          signal.end_with?("TTIN") ? :warn : :info
+        end
+
+        def event_value(event, payload, key)
+          payload_value(payload, key) || raw_event_value(event, key)
+        end
+
+        def payload_value(payload, key)
+          Core::Integration::Values::Read.value(payload, key)
+        end
+
+        def raw_event_value(event, key)
+          raw = EventPayload.event_payload(event)
+          return raw[key] if raw.respond_to?(:key?) && raw.key?(key)
+          return raw[key.to_s] if raw.respond_to?(:key?) && raw.key?(key.to_s)
+          return event[key] if event.respond_to?(:[])
+
+          nil
+        rescue StandardError
+          nil
+        end
+
+        def collection_count(value)
+          return value[:count] if value.is_a?(Hash) && value.key?(:count)
+          return value["count"] if value.is_a?(Hash) && value.key?("count")
+          return value.size if value.respond_to?(:size)
+
+          nil
+        end
+      end
+    end
+  end
+end

data/lib/julewire/karafka/fork_hooks.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+module Julewire
+  module Karafka
+    module ForkHooks
+      EVENTS = %w[
+        swarm.node.after_fork
+        swarm.manager.after_fork
+      ].freeze
+      INSTALL_STATE = Core::Integration::IvarState.new(:@julewire_karafka_fork_hooks_state)
+      private_constant :EVENTS, :INSTALL_STATE
+
+      class << self
+        def subscribe!(monitor, configuration: Configuration.new)
+          return unless configuration.enabled?
+          return unless monitor.respond_to?(:subscribe)
+
+          state = INSTALL_STATE.fetch_or_store(monitor) { { events: [].freeze }.freeze }
+          subscribed_events = Array(state[:events]).dup
+          EVENTS.each do |event_name|
+            next if subscribed_events.include?(event_name)
+
+            subscribed_events << event_name if subscribe_event(monitor, event_name)
+          end
+          INSTALL_STATE.store(monitor, { events: subscribed_events.freeze }.freeze)
+          monitor
+        end
+
+        def handle(event_name, _event)
+          IntegrationHealth.with_failure_health(action: :after_fork, component: :fork_hooks, event: event_name) do
+            Julewire.after_fork!
+          end
+        end
+
+        private
+
+        def subscribe_event(monitor, event_name)
+          IntegrationHealth.with_failure_health(action: :subscribe, component: :fork_hooks, event: event_name) do
+            monitor.subscribe(event_name) { handle(event_name, it) }
+            true
+          end || false
+        end
+      end
+    end
+  end
+end

data/lib/julewire/karafka/installer.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Julewire
+  module Karafka
+    module Installer
+      class << self
+        def install!(app: nil, monitor: nil, configuration: Configuration.new)
+          return false unless configuration.enabled?
+
+          monitor ||= monitor_for(app)
+          raise Error, "Karafka monitor is not available" unless monitor
+
+          ForkHooks.subscribe!(monitor, configuration: configuration)
+          if configuration.consumer_events?
+            MonitorSubscription.install!(monitor, configuration: configuration, profile: :consumer)
+          end
+          monitor
+        end
+
+        private
+
+        def monitor_for(app)
+          app ||= defined?(::Karafka::App) ? ::Karafka::App : nil
+          app.config.monitor if app.respond_to?(:config)
+        rescue StandardError
+          nil
+        end
+      end
+    end
+  end
+end