dead_bro 0.2.28 → 0.2.30

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6b96d1dee673157c1b4c46ad86f7e0d4a4feb51774038de260c5464353ff292e
-  data.tar.gz: f922d71e3c1f132642bf5756aff381a5d372538672c3e0affd010cb31f30b397
+  metadata.gz: cccae4ac857f38e889e12c292943b25fad11f7e372d7f13e4eb838d13c6330ee
+  data.tar.gz: 4d459c4d049219747ed5c1b6f10e1417648282eca2e3e81c7eeef132e8a347bd
 SHA512:
-  metadata.gz: 20f9d81a03bb468cc4ded983ad77b1ad9191dde3ba5c5320b7dd1e11bb5295b5d90289f597df036770cb0d6740ec10f426cf75ba5c941a1922cce8785ee21fe8
-  data.tar.gz: c5325f9e3c8126236fb4386b9214d9637a4086176ae6e2675d4344958bc09d16bbb5718f6461f8bb8eaf75da4ef7e295d3ae012b15a5bb0ab1fbdea8ef0b8f48
+  metadata.gz: d12c33c0902c95f318cd716d7c88c78925543b235729ee90525d0453b16ed46521e2b00cc1daa5131b673abb7ee7e74e0968f299d56da055fd36452c2e314b90
+  data.tar.gz: e04ca78806e4d9a07757c0fedc177c2e4b77b295437d415fab94546d84add717ceab84791e34991f4955197ab1d8036ad498d679b089e795e5356beb0662d286

data/CHANGELOG.md

@@ -3,6 +3,11 @@
 ### Added
 - Monitor thread now sends a synchronous heartbeat on startup before the first collection tick. This ensures remote settings — including `monitor_enabled` — are applied from the very first reporting cycle, so Sidekiq workers and other non-web processes that have not yet sent any metrics still receive the correct configuration immediately on boot rather than waiting up to 60 seconds for the first scheduled tick.
 
+## [0.2.30] - 2026-06-19
+
+### Added
+- **`DeadBro.watch(label, **tags) { ... }`**: opt-in scoped timing for arbitrary code during instrumented web requests and background jobs. Records elapsed time, `start_offset_ms`, nesting depth, SQL count/duration attributable to the block, optional tags, and exception metadata (then re-raises). Spans are sent as `watch_events` in the normal APM payload for the Request Trace waterfall. Disabled by default; enable locally via `DeadBro.configure { |c| c.watch_enabled = true }` or remotely via the `watch_enabled` setting. Guardrails: max depth 10, max 50 spans per request, label length 200 chars, max 5 tags per span with tag values truncated to 100 chars. **Nested SQL:** each span diffs the request-wide SQL counter at enter/exit; parent spans include SQL from nested child spans (inclusive totals — use inner spans for per-block SQL, do not sum parent + child counts).
+
 ## [0.2.25] - 2026-06-14
 
 ### Added

data/lib/dead_bro/configuration.rb

@@ -17,7 +17,7 @@ module DeadBro
 
     # Remote-managed settings (overwritten by backend JSON `settings` on successful API responses)
     attr_accessor :memory_tracking_enabled, :allocation_tracking_enabled, :allocation_sample_rate,
-      :sample_rate, :slow_query_threshold_ms,
+      :sample_rate, :slow_query_threshold_ms, :watch_enabled,
       :monitor_enabled, :enable_db_stats, :enable_process_stats, :enable_system_stats,
       :max_sql_queries_to_send, :max_logs_to_send
 
@@ -67,7 +67,7 @@ module DeadBro
     REMOTE_SETTING_KEYS = %w[
       enabled sample_rate memory_tracking_enabled allocation_tracking_enabled allocation_sample_rate
       explain_analyze_enabled slow_query_threshold_ms max_sql_queries_to_send max_logs_to_send
-      excluded_controllers excluded_jobs exclusive_controllers exclusive_jobs
+      watch_enabled excluded_controllers excluded_jobs exclusive_controllers exclusive_jobs
       monitor_enabled enable_db_stats enable_process_stats enable_system_stats
       sample_rates_by_type
     ].freeze
@@ -102,6 +102,7 @@ module DeadBro
       @slow_query_threshold_ms = 500
       @max_sql_queries_to_send = 500
       @max_logs_to_send = 100
+      @watch_enabled = false
       self.excluded_controllers = []
       self.excluded_jobs = []
       self.exclusive_controllers = []
@@ -178,7 +179,7 @@ module DeadBro
             # must never be able to switch it on — only off. The local opt-in
             # (explain_analyze_enabled) stays untouched; see #explain_analyze_active?
             @remote_explain_analyze_enabled = !!value
-          when "enabled", "memory_tracking_enabled", "allocation_tracking_enabled",
+          when "enabled", "memory_tracking_enabled", "allocation_tracking_enabled", "watch_enabled",
                "monitor_enabled", "enable_db_stats", "enable_process_stats", "enable_system_stats"
             send(:"#{k}=", !!value)
           when "excluded_controllers", "excluded_jobs", "exclusive_controllers", "exclusive_jobs"
@@ -208,6 +209,10 @@ module DeadBro
       Time.now.utc < t
     end
 
+    def watch_enabled?
+      !!@watch_enabled
+    end
+
     def resolve_deploy_id
       explicit = @explicit_deploy_revision&.to_s&.strip
       return explicit unless explicit.nil? || explicit.empty?

data/lib/dead_bro/job_sql_tracking_middleware.rb

@@ -15,6 +15,17 @@ module DeadBro
 
         DeadBro::SqlSubscriber.start_request_tracking
 
+        # Start dependency tracking so HTTP / Redis / cache / view / Elasticsearch time
+        # spent inside the job is captured — mirrors the web SqlTrackingMiddleware. Each
+        # of those subscribers only records when its thread-local has been initialized;
+        # without this, they silently drop every event during a job and all that time
+        # collapses into the "Active Job" residual of the performance breakdown.
+        Thread.current[:dead_bro_http_events] = []
+        DeadBro::CacheSubscriber.start_request_tracking if defined?(DeadBro::CacheSubscriber)
+        DeadBro::RedisSubscriber.start_request_tracking if defined?(DeadBro::RedisSubscriber)
+        DeadBro::ElasticsearchSubscriber.start_request_tracking if defined?(DeadBro::ElasticsearchSubscriber)
+        DeadBro::ViewRenderingSubscriber.start_request_tracking if defined?(DeadBro::ViewRenderingSubscriber)
+
         # Start lightweight memory tracking for this job
         if defined?(DeadBro::LightweightMemoryTracker)
           DeadBro::LightweightMemoryTracker.start_request_tracking
@@ -29,6 +40,8 @@ module DeadBro
         if defined?(DeadBro::DbConnectionSubscriber)
           DeadBro::DbConnectionSubscriber.start_request_tracking
         end
+
+        DeadBro::WatchTracker.start_request_tracking if defined?(DeadBro::WatchTracker)
       end
     rescue
       # Never raise from instrumentation install

data/lib/dead_bro/job_subscriber.rb

@@ -59,7 +59,9 @@ module DeadBro
           DeadBro.logger.clear
           Thread.current[DeadBro::TRACKING_START_TIME_KEY] = Time.now
           DeadBro::SqlSubscriber.start_request_tracking
+          start_job_dependency_tracking
           DeadBro::DbConnectionSubscriber.start_request_tracking if defined?(DeadBro::DbConnectionSubscriber)
+          DeadBro::WatchTracker.start_request_tracking if defined?(DeadBro::WatchTracker)
           if DeadBro.configuration.allocation_tracking_enabled && defined?(DeadBro::MemoryTrackingSubscriber)
             DeadBro::MemoryTrackingSubscriber.start_request_tracking
           else
@@ -69,9 +71,11 @@ module DeadBro
 
         # Get SQL queries executed during this job
         sql_queries = DeadBro::SqlSubscriber.stop_request_tracking
+        dependency_events = job_dependency_payload
         db_connection_stats = defined?(DeadBro::DbConnectionSubscriber) ? DeadBro::DbConnectionSubscriber.stop_request_tracking : {}
         gc_pressure = defined?(DeadBro::GcTracker) ? DeadBro::GcTracker.stop_request_tracking : {}
         ar_instantiation_count = defined?(DeadBro::ArObjectTracker) ? DeadBro::ArObjectTracker.stop_request_tracking : nil
+        watch_events = defined?(DeadBro::WatchTracker) ? DeadBro::WatchTracker.stop_request_tracking : []
 
         # Stop memory tracking and get collected memory data
         if DeadBro.configuration.allocation_tracking_enabled && defined?(DeadBro::MemoryTrackingSubscriber)
@@ -122,8 +126,9 @@ module DeadBro
           gc_stats: gc_stats,
           memory_events: memory_events,
           memory_performance: memory_performance,
+          watch_events: watch_events,
           logs: DeadBro.logger.logs
-        }
+        }.merge(dependency_events)
 
         # force: true — the sampling decision above already accounted for any
         # per-job-type override; client#post_metric must not re-roll it globally.
@@ -158,7 +163,9 @@ module DeadBro
           DeadBro.logger.clear
           Thread.current[DeadBro::TRACKING_START_TIME_KEY] = Time.now
           DeadBro::SqlSubscriber.start_request_tracking
+          start_job_dependency_tracking
           DeadBro::DbConnectionSubscriber.start_request_tracking if defined?(DeadBro::DbConnectionSubscriber)
+          DeadBro::WatchTracker.start_request_tracking if defined?(DeadBro::WatchTracker)
           if DeadBro.configuration.allocation_tracking_enabled && defined?(DeadBro::MemoryTrackingSubscriber)
             DeadBro::MemoryTrackingSubscriber.start_request_tracking
           else
@@ -168,9 +175,11 @@ module DeadBro
 
         # Get SQL queries executed during this job
         sql_queries = DeadBro::SqlSubscriber.stop_request_tracking
+        dependency_events = job_dependency_payload
         db_connection_stats = defined?(DeadBro::DbConnectionSubscriber) ? DeadBro::DbConnectionSubscriber.stop_request_tracking : {}
         gc_pressure = defined?(DeadBro::GcTracker) ? DeadBro::GcTracker.stop_request_tracking : {}
         ar_instantiation_count = defined?(DeadBro::ArObjectTracker) ? DeadBro::ArObjectTracker.stop_request_tracking : nil
+        watch_events = defined?(DeadBro::WatchTracker) ? DeadBro::WatchTracker.stop_request_tracking : []
 
         # Stop memory tracking and get collected memory data
         if DeadBro.configuration.allocation_tracking_enabled && defined?(DeadBro::MemoryTrackingSubscriber)
@@ -224,8 +233,9 @@ module DeadBro
           gc_stats: gc_stats,
           memory_events: memory_events,
           memory_performance: memory_performance,
+          watch_events: watch_events,
           logs: DeadBro.logger.logs
-        }
+        }.merge(dependency_events)
 
         event_name = exception&.class&.name || "ActiveJob::Exception"
         client.post_metric(event_name: event_name, payload: payload, force: true)
@@ -239,6 +249,11 @@ module DeadBro
     def self.drain_job_tracking
       # wait_for_explains: false — result is discarded, don't block on pending plans.
       DeadBro::SqlSubscriber.stop_request_tracking(wait_for_explains: false) if defined?(DeadBro::SqlSubscriber)
+      Thread.current[:dead_bro_http_events] = nil
+      DeadBro::CacheSubscriber.stop_request_tracking if defined?(DeadBro::CacheSubscriber)
+      DeadBro::RedisSubscriber.stop_request_tracking if defined?(DeadBro::RedisSubscriber)
+      DeadBro::ElasticsearchSubscriber.stop_request_tracking if defined?(DeadBro::ElasticsearchSubscriber)
+      DeadBro::ViewRenderingSubscriber.stop_request_tracking if defined?(DeadBro::ViewRenderingSubscriber)
       DeadBro::DbConnectionSubscriber.stop_request_tracking if defined?(DeadBro::DbConnectionSubscriber)
       DeadBro::GcTracker.stop_request_tracking if defined?(DeadBro::GcTracker)
       DeadBro::ArObjectTracker.stop_request_tracking if defined?(DeadBro::ArObjectTracker)
@@ -246,10 +261,59 @@ module DeadBro
       if DeadBro.configuration.allocation_tracking_enabled && defined?(DeadBro::MemoryTrackingSubscriber)
         DeadBro::MemoryTrackingSubscriber.stop_request_tracking
       end
+      DeadBro::WatchTracker.stop_request_tracking if defined?(DeadBro::WatchTracker)
     rescue
       # Best effort
     end
 
+    # Start HTTP/Redis/cache/view/Elasticsearch tracking for job backends that never
+    # emit perform_start.active_job (so JobSqlTrackingMiddleware didn't run). Mirrors the
+    # dependency tracking the middleware normally arms — see its comment for why each
+    # thread-local must be initialized before its subscriber will record anything.
+    def self.start_job_dependency_tracking
+      Thread.current[:dead_bro_http_events] = []
+      DeadBro::CacheSubscriber.start_request_tracking if defined?(DeadBro::CacheSubscriber)
+      DeadBro::RedisSubscriber.start_request_tracking if defined?(DeadBro::RedisSubscriber)
+      DeadBro::ElasticsearchSubscriber.start_request_tracking if defined?(DeadBro::ElasticsearchSubscriber)
+      DeadBro::ViewRenderingSubscriber.start_request_tracking if defined?(DeadBro::ViewRenderingSubscriber)
+    rescue
+    end
+
+    # Snapshot and clear the per-job dependency events, returning the payload slice that
+    # mirrors what the web Subscriber sends. These feed the performance breakdown (the app
+    # derives http/redis/es duration columns from them) and the trace timeline.
+    def self.job_dependency_payload
+      http_outgoing = Thread.current[:dead_bro_http_events] || []
+      Thread.current[:dead_bro_http_events] = nil
+      cache_events = defined?(DeadBro::CacheSubscriber) ? DeadBro::CacheSubscriber.stop_request_tracking : []
+      redis_events = defined?(DeadBro::RedisSubscriber) ? DeadBro::RedisSubscriber.stop_request_tracking : []
+      elasticsearch_events = defined?(DeadBro::ElasticsearchSubscriber) ? DeadBro::ElasticsearchSubscriber.stop_request_tracking : []
+      view_events = defined?(DeadBro::ViewRenderingSubscriber) ? DeadBro::ViewRenderingSubscriber.stop_request_tracking : []
+      view_performance = defined?(DeadBro::ViewRenderingSubscriber) ? DeadBro::ViewRenderingSubscriber.analyze_view_performance(view_events) : {}
+
+      {
+        http_outgoing: http_outgoing,
+        cache_events: cache_events,
+        redis_events: redis_events,
+        elasticsearch_events: elasticsearch_events,
+        view_events: view_events,
+        view_performance: view_performance,
+        view_runtime_ms: sum_view_runtime_ms(view_events)
+      }
+    rescue
+      {}
+    end
+
+    def self.sum_view_runtime_ms(view_events)
+      return nil unless view_events.is_a?(Array) && view_events.any?
+      view_events.sum do |e|
+        next 0 unless e.is_a?(Hash)
+        (e[:total_duration_ms] || e["total_duration_ms"] || e[:duration_ms] || e["duration_ms"] || 0).to_f
+      end.round(2)
+    rescue
+      nil
+    end
+
     private
 
     def self.job_queue_duration_ms(job, perform_started)

data/lib/dead_bro/sql_subscriber.rb

@@ -50,6 +50,21 @@ module DeadBro
       stack.last
     end
 
+    # Sum of SQL counts and durations recorded so far in the active tracking
+    # context. Used by WatchTracker to attribute SQL to a DeadBro.watch block.
+    def self.current_sql_metrics
+      agg_stack = Thread.current[THREAD_LOCAL_AGGREGATES_KEY]
+      aggregates_h = (agg_stack.is_a?(Array) && agg_stack.any?) ? agg_stack.last : {}
+      return {count: 0, duration_ms: 0.0} unless aggregates_h.is_a?(Hash) && aggregates_h.any?
+
+      {
+        count: aggregates_h.values.sum { |a| (a[:count] || 0).to_i },
+        duration_ms: aggregates_h.values.sum { |a| (a[:total_duration_ms] || 0.0).to_f }.round(2)
+      }
+    rescue StandardError
+      {count: 0, duration_ms: 0.0}
+    end
+
     # Check if we should continue tracking based on count and time limits
     def self.should_continue_tracking?(current_queries_array, max_count)
       return false unless current_queries_array.is_a?(Array)

data/lib/dead_bro/sql_tracking_middleware.rb

@@ -82,6 +82,9 @@ module DeadBro
       # Start CPU time tracking for this request (thread-local clock)
       DeadBro::CpuTracker.start_request_tracking if defined?(DeadBro::CpuTracker)
 
+      # Start user-defined watch block tracking for this request
+      DeadBro::WatchTracker.start_request_tracking if defined?(DeadBro::WatchTracker)
+
       # Start outgoing HTTP accumulation for this request
       Thread.current[:dead_bro_http_events] = []
 

data/lib/dead_bro/subscriber.rb

@@ -80,6 +80,8 @@ module DeadBro
         view_events = DeadBro::ViewRenderingSubscriber.stop_request_tracking
         view_performance = DeadBro::ViewRenderingSubscriber.analyze_view_performance(view_events)
 
+        watch_events = defined?(DeadBro::WatchTracker) ? DeadBro::WatchTracker.stop_request_tracking : []
+
         # Per-phase allocation attribution (under memory tracking) — which phase
         # allocated the request's objects (sql / view / elasticsearch).
         allocation_phases = if DeadBro.configuration.memory_tracking_enabled && defined?(DeadBro::MemoryPhaseTracker)
@@ -222,6 +224,7 @@ module DeadBro
           gc_pressure: gc_pressure,
           ar_instantiation_count: ar_instantiation_count,
           cpu_time_ms: cpu_time_ms,
+          watch_events: watch_events,
           logs: DeadBro.logger.logs
         }
         # force: true — the sampling decision (global or per-request-type) was
@@ -254,6 +257,7 @@ module DeadBro
       DeadBro::GcTracker.stop_request_tracking if defined?(DeadBro::GcTracker)
       DeadBro::ArObjectTracker.stop_request_tracking if defined?(DeadBro::ArObjectTracker)
       DeadBro::CpuTracker.stop_request_tracking if defined?(DeadBro::CpuTracker)
+      DeadBro::WatchTracker.stop_request_tracking if defined?(DeadBro::WatchTracker)
     rescue
       # Best effort — draining must never raise from the notifications callback.
     end

data/lib/dead_bro/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module DeadBro
-  VERSION = "0.2.28"
+  VERSION = "0.2.30"
 end

data/lib/dead_bro/watch_tracker.rb

@@ -0,0 +1,189 @@
+# frozen_string_literal: true
+
+module DeadBro
+  # Records user-defined timed blocks (DeadBro.watch) during an instrumented
+  # request or background job. Spans include wall duration, nesting depth, and
+  # SQL attributable to the block (delta from SqlSubscriber aggregates).
+  #
+  # SQL attribution: each span diffs the global SQL counter at block entry vs
+  # exit. Nested blocks each get their own diff; a parent span's sql_count /
+  # sql_duration_ms therefore includes queries run inside nested child spans
+  # (inclusive totals, not exclusive). Compare inner spans when you need
+  # per-block SQL; do not sum parent and child counts.
+  #
+  # Unlike DeadBro.analyze, watch spans are sent to the DeadBro backend as part
+  # of the normal request/job payload when watch_enabled is on.
+  module WatchTracker
+    THREAD_LOCAL_EVENTS_KEY = :dead_bro_watch_events
+    THREAD_LOCAL_ACTIVE_STACK_KEY = :dead_bro_watch_active_stack
+
+    # Deepest depth index still recorded — depths 0..MAX_DEPTH are kept, anything
+    # nested below that yields untracked.
+    MAX_DEPTH = 10
+    MAX_SPANS_PER_REQUEST = 50
+    MAX_LABEL_LENGTH = 200
+    MAX_TAGS = 5
+    MAX_TAG_VALUE_LENGTH = 100
+
+    module_function
+
+    def enabled?
+      DeadBro.configuration.watch_enabled?
+    rescue StandardError
+      false
+    end
+
+    def tracking_active?
+      Thread.current[THREAD_LOCAL_EVENTS_KEY].is_a?(Array)
+    rescue StandardError
+      false
+    end
+
+    def start_request_tracking
+      Thread.current[THREAD_LOCAL_EVENTS_KEY] = []
+      Thread.current[THREAD_LOCAL_ACTIVE_STACK_KEY] = []
+    end
+
+    def stop_request_tracking
+      events = Thread.current[THREAD_LOCAL_EVENTS_KEY]
+      Thread.current[THREAD_LOCAL_EVENTS_KEY] = nil
+      Thread.current[THREAD_LOCAL_ACTIVE_STACK_KEY] = nil
+      # compact drops reserved slots whose block never completed — can't happen
+      # for a lexical block, but keeps a nil out of the payload if it ever does.
+      events.is_a?(Array) ? events.compact : []
+    rescue StandardError
+      []
+    end
+
+    def watch(label = nil, **tags)
+      return yield unless block_given?
+      return yield unless enabled?
+      return yield unless tracking_active?
+
+      events = Thread.current[THREAD_LOCAL_EVENTS_KEY]
+      active_stack = Thread.current[THREAD_LOCAL_ACTIVE_STACK_KEY]
+      return yield unless events.is_a?(Array) && active_stack.is_a?(Array)
+
+      depth = active_stack.length
+      return yield if depth > MAX_DEPTH
+      return yield if events.length >= MAX_SPANS_PER_REQUEST
+
+      sanitized_label = sanitize_label(label)
+      sanitized_tags = sanitize_tags(tags)
+      start_offset_ms = compute_start_offset_ms
+      sql_before = sql_metrics_snapshot
+      block_start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
+      frame = {
+        label: sanitized_label,
+        depth: depth,
+        tags: sanitized_tags,
+        start_offset_ms: start_offset_ms,
+        sql_before: sql_before,
+        block_start: block_start,
+        error: false,
+        exception_class: nil
+      }
+      # Reserve the slot on entry so spans come out in start order. Blocks only
+      # complete inner-first, so appending on completion would emit a nested span
+      # ahead of its own parent; the placeholder keeps parents before children.
+      # It also makes MAX_SPANS_PER_REQUEST count blocks still in flight.
+      frame[:slot] = events.length
+      events << nil
+      active_stack << frame
+
+      begin
+        yield
+      rescue StandardError => e
+        frame[:error] = true
+        # Anonymous classes (Class.new(StandardError)) have a nil name — leave the
+        # field unset rather than shipping an empty string the dashboard would
+        # render as a blank exception.
+        class_name = e.class.name
+        frame[:exception_class] = class_name[0, 200] if class_name
+        raise
+      ensure
+        active_stack.pop if active_stack.last.equal?(frame)
+        append_completed_span(events, frame)
+      end
+    end
+
+    def sanitize_label(label)
+      text = label.to_s.strip
+      text = "block" if text.empty?
+      (text.length > MAX_LABEL_LENGTH) ? text[0, MAX_LABEL_LENGTH] + "..." : text
+    rescue StandardError
+      "block"
+    end
+
+    def sanitize_tags(tags)
+      return {} unless tags.is_a?(Hash) && tags.any?
+
+      tags.first(MAX_TAGS).each_with_object({}) do |(key, value), out|
+        k = key.to_s.strip[0, 50]
+        next if k.empty?
+
+        v = value.to_s.strip
+        v = v[0, MAX_TAG_VALUE_LENGTH] + "..." if v.length > MAX_TAG_VALUE_LENGTH
+        out[k] = v
+      end
+    rescue StandardError
+      {}
+    end
+
+    def compute_start_offset_ms
+      tracking_start = Thread.current[DeadBro::TRACKING_START_TIME_KEY]
+      return 0.0 unless tracking_start
+
+      ((Time.now - tracking_start) * 1000.0).round(2)
+    rescue StandardError
+      0.0
+    end
+
+    def sql_metrics_snapshot
+      return {count: 0, duration_ms: 0.0} unless defined?(DeadBro::SqlSubscriber)
+
+      DeadBro::SqlSubscriber.current_sql_metrics
+    rescue StandardError
+      {count: 0, duration_ms: 0.0}
+    end
+
+    def append_completed_span(events, frame)
+      return unless events.is_a?(Array)
+
+      slot = frame[:slot]
+      return unless slot.is_a?(Integer) && slot < events.length
+
+      elapsed_ms = begin
+        ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - frame[:block_start]) * 1000.0).round(2)
+      rescue StandardError
+        0.0
+      end
+
+      sql_after = sql_metrics_snapshot
+      sql_before = frame[:sql_before] || {count: 0, duration_ms: 0.0}
+      sql_count = [(sql_after[:count] || 0).to_i - (sql_before[:count] || 0).to_i, 0].max
+      sql_duration_ms = [
+        (sql_after[:duration_ms] || 0.0).to_f - (sql_before[:duration_ms] || 0.0).to_f,
+        0.0
+      ].max.round(2)
+
+      span = {
+        label: frame[:label],
+        duration_ms: elapsed_ms,
+        start_offset_ms: frame[:start_offset_ms],
+        depth: frame[:depth],
+        sql_count: sql_count,
+        sql_duration_ms: sql_duration_ms,
+        error: frame[:error] == true
+      }
+      span[:exception_class] = frame[:exception_class] if frame[:exception_class]
+      tags = frame[:tags]
+      span[:tags] = tags if tags.is_a?(Hash) && tags.any?
+
+      events[slot] = span
+    rescue StandardError
+      # Never raise from instrumentation
+    end
+  end
+end

data/lib/dead_bro.rb

@@ -29,6 +29,7 @@ module DeadBro
   autoload :Monitor, "dead_bro/monitor"
   autoload :MemoryDetails, "dead_bro/memory_details"
   autoload :Logger, "dead_bro/logger"
+  autoload :WatchTracker, "dead_bro/watch_tracker"
   begin
     require "dead_bro/railtie"
   rescue LoadError
@@ -458,4 +459,28 @@ module DeadBro
     raise error if error
     analysis_result
   end
+
+  # Time a named block during an instrumented request or background job.
+  #
+  # When watch_enabled is on and tracking is active (web request or job),
+  # records elapsed time, optional tags, nested SQL count/duration, and errors
+  # as watch_events in the APM payload for the Request Trace waterfall.
+  #
+  # Nested watch blocks: each span's sql_count/sql_duration_ms is the SQL that
+  # ran while that block was active. Parent spans include SQL from nested child
+  # spans (inclusive). Inner spans show their own slice; do not add parent and
+  # child SQL counts together.
+  #
+  # When disabled or outside an instrumented context, yields with no overhead
+  # beyond a single config check.
+  #
+  # Usage:
+  #   DeadBro.watch("load users") { User.where(active: true).load }
+  #   DeadBro.watch("sync billing", customer_id: user.id) { sync!(user) }
+  #
+  # Compare to DeadBro.analyze — analyze is local-only console profiling;
+  # watch spans are sent to the DeadBro dashboard when enabled.
+  def self.watch(label = nil, **tags, &block)
+    WatchTracker.watch(label, **tags, &block)
+  end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: dead_bro
 version: !ruby/object:Gem::Version
-  version: 0.2.28
+  version: 0.2.30
 platform: ruby
 authors:
 - Emanuel Comsa
@@ -18,7 +18,6 @@ extensions: []
 extra_rdoc_files: []
 files:
 - CHANGELOG.md
-- FEATURES.md
 - README.md
 - lib/dead_bro.rb
 - lib/dead_bro/allocation_source_sampler.rb
@@ -59,6 +58,7 @@ files:
 - lib/dead_bro/subscriber.rb
 - lib/dead_bro/version.rb
 - lib/dead_bro/view_rendering_subscriber.rb
+- lib/dead_bro/watch_tracker.rb
 - lib/generators/dead_bro/install/install_generator.rb
 - lib/generators/dead_bro/install/templates/dead_bro.rb
 homepage: https://www.deadbro.com
@@ -80,7 +80,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.9
+rubygems_version: 4.0.10
 specification_version: 4
 summary: Minimal APM for Rails apps.
 test_files: []

data/FEATURES.md

@@ -1,333 +0,0 @@
-# ApmBro Feature List
-
-A comprehensive feature list for comparing ApmBro with other APM (Application Performance Monitoring) tools.
-
-## Core Architecture
-
-- **Rails Integration**: Automatic subscription to Rails events via ActiveSupport::Notifications
-- **Zero-Configuration Setup**: Works out of the box with minimal configuration
-- **Asynchronous Metrics Posting**: Non-blocking HTTP requests using background threads
-- **Thread-Local Storage**: Per-request metric collection using thread-local variables
-- **Circuit Breaker Pattern**: Built-in circuit breaker to prevent cascading failures when APM endpoint is down
-- **Deploy Tracking**: Automatic deploy ID resolution from multiple sources (Rails settings, ENV vars, Heroku, Git)
-
-## Request Tracking
-
-### Controller Action Monitoring
-- **Automatic Tracking**: Tracks all controller actions automatically
-- **Request Duration**: Measures total request processing time
-- **HTTP Method & Path**: Captures HTTP method and request path
-- **Status Codes**: Tracks HTTP response status codes
-- **View Runtime**: Separate tracking of view rendering time
-- **Database Runtime**: Separate tracking of database query time
-- **Request Parameters**: Captures request parameters (with sensitive data filtering)
-- **User Agent**: Tracks user agent strings
-- **User ID Extraction**: Extracts authenticated user ID (supports Warden)
-- **Environment Context**: Tracks Rails environment (development, staging, production)
-
-### Request Sampling
-- **Configurable Sample Rate**: Percentage-based sampling (1-100%)
-- **Random Sampling**: Each request has random chance of being tracked
-- **Consistent Per-Request**: Sampling decision applies to all metrics for a request
-- **Error Override**: Errors are always tracked regardless of sampling
-- **Cost Optimization**: Reduces data volume and costs for high-traffic applications
-
-### Exclusion Rules
-- **Controller Exclusion**: Exclude entire controllers from tracking
-- **Action Exclusion**: Exclude specific controller#action combinations
-- **Wildcard Support**: Pattern matching with `*` wildcards (e.g., `Admin::*`, `Admin::*#*`)
-- **Job Exclusion**: Exclude specific background jobs from tracking
-- **Flexible Configuration**: Configure via initializer, Rails settings, or environment variables
-
-## SQL Query Tracking
-
-### Query Details
-- **Full SQL Tracking**: Captures all SQL queries executed during requests and jobs
-- **Query Sanitization**: Automatically sanitizes SQL to remove sensitive data
-- **Query Name**: Tracks query names (e.g., "User Load", "User Update")
-- **Duration Measurement**: Precise query execution time in milliseconds
-- **Cache Detection**: Identifies cached queries
-- **Connection ID**: Tracks database connection ID
-- **Call Stack Traces**: Full backtrace showing where queries were executed
-- **Object Allocations**: Optional tracking of object allocations per query
-
-### Query Performance Analysis
-- **Slow Query Detection**: Configurable threshold for identifying slow queries
-- **EXPLAIN ANALYZE**: Automatic execution plan capture for slow queries
-- **Background Execution**: EXPLAIN ANALYZE runs in separate thread (non-blocking)
-- **Multi-Database Support**: Works with PostgreSQL, MySQL, SQLite, and others
-- **Smart Filtering**: Automatically skips transaction queries (BEGIN, COMMIT, ROLLBACK)
-- **Execution Plan Details**: 
-  - PostgreSQL: Full EXPLAIN ANALYZE with buffer usage statistics
-  - MySQL: EXPLAIN ANALYZE with actual execution times
-  - SQLite: EXPLAIN QUERY PLAN output
-- **Query Optimization Insights**: Helps identify missing indexes, full table scans, JOIN issues
-
-## View Rendering Tracking
-
-### View Performance
-- **Template Rendering**: Tracks main template rendering
-- **Partial Rendering**: Tracks partial template rendering with cache key information
-- **Collection Rendering**: Tracks collection rendering (partials in loops)
-- **Rendering Duration**: Precise timing for each view component
-- **Virtual Path Tracking**: Tracks view virtual paths
-- **Layout Information**: Captures layout usage
-
-### View Analysis
-- **Slow View Detection**: Identifies the slowest rendering views
-- **Frequency Analysis**: Tracks most frequently rendered views
-- **Cache Hit Rate**: Calculates cache hit rates for partials
-- **Collection Cache Analysis**: Tracks cache hit rates for collection rendering
-- **Performance Metrics**: 
-  - Total views rendered per request
-  - Total view rendering duration
-  - Average view rendering duration
-  - Breakdown by view type (template, partial, collection)
-
-## Memory Tracking & Leak Detection
-
-### Lightweight Memory Tracking (Default)
-- **Memory Usage Monitoring**: Tracks memory consumption per request using GC stats
-- **Memory Growth Tracking**: Measures memory growth during request processing
-- **GC Statistics**: Tracks garbage collection count and heap pages
-- **Minimal Performance Impact**: ~0.1ms overhead per request
-- **Memory Before/After**: Captures memory state at request start and end
-
-### Detailed Allocation Tracking (Optional)
-- **Object Allocation Tracking**: Detailed tracking of object allocations (disabled by default)
-- **Allocation Sampling**: Configurable sampling rate for allocations
-- **Large Object Detection**: Identifies objects larger than 1MB threshold
-- **Memory Snapshots**: Periodic memory snapshots during request processing
-- **Object Count Tracking**: Tracks object counts before and after requests
-- **Performance Impact**: ~2-5ms overhead per request (only when enabled)
-
-### Memory Leak Detection
-- **Pattern Detection**: Detects growing memory patterns over time
-- **GC Efficiency Analysis**: Monitors garbage collection effectiveness
-- **Heap Page Tracking**: Tracks heap page growth
-- **Request Correlation**: Correlates memory growth with specific controllers/actions
-
-## Background Job Tracking
-
-### Job Execution Monitoring
-- **ActiveJob Integration**: Automatic tracking when ActiveJob is available
-- **Job Class Tracking**: Tracks job class names
-- **Job ID**: Captures unique job identifiers
-- **Queue Name**: Tracks which queue processed the job
-- **Job Arguments**: Captures job arguments (with sensitive data filtering)
-- **Duration Measurement**: Precise job execution time in milliseconds
-- **Status Tracking**: Tracks job status (completed or failed)
-
-### Job Error Tracking
-- **Exception Capture**: Captures exceptions from failed jobs
-- **Exception Class**: Tracks exception class names
-- **Exception Messages**: Captures exception messages (truncated to 1000 chars)
-- **Backtraces**: Full exception backtraces (first 50 lines)
-- **SQL Query Context**: Includes SQL queries executed during failed jobs
-- **Memory Context**: Includes memory usage during job execution
-
-### Job SQL Tracking
-- **SQL Query Tracking**: Tracks all SQL queries executed during job processing
-- **Query Details**: Same detailed SQL tracking as request tracking
-- **Query Context**: Full context of database operations in background jobs
-
-## Cache Tracking
-
-### Cache Operations
-- **Read Operations**: Tracks cache read operations
-- **Write Operations**: Tracks cache write operations
-- **Delete Operations**: Tracks cache delete operations
-- **Existence Checks**: Tracks cache existence checks
-- **Fetch Operations**: Tracks cache fetch with hit/miss detection
-- **Multi-Read Operations**: Tracks cache read_multi operations
-- **Multi-Write Operations**: Tracks cache write_multi operations
-- **Cache Generation**: Tracks cache generation events
-
-### Cache Analysis
-- **Cache Hit Detection**: Identifies cache hits vs misses
-- **Cache Key Tracking**: Tracks cache keys (truncated to 200 chars)
-- **Store Information**: Identifies which cache store was used
-- **Namespace Tracking**: Tracks cache namespaces
-- **Duration Measurement**: Precise timing for each cache operation
-- **Hit Rate Calculation**: Calculates cache hit rates per request
-
-## Redis Tracking
-
-### Redis Command Tracking
-- **Command Monitoring**: Tracks all Redis commands executed
-- **Command Name**: Captures Redis command names (GET, SET, etc.)
-- **Key Tracking**: Tracks Redis keys (truncated to 200 chars)
-- **Argument Count**: Tracks number of arguments per command
-- **Database Selection**: Tracks which Redis database is used
-- **Duration Measurement**: Precise timing for each Redis command
-- **Error Tracking**: Captures Redis command errors
-
-### Advanced Redis Features
-- **Pipeline Support**: Tracks Redis pipeline operations with command counts
-- **Multi/Transaction Support**: Tracks Redis MULTI/EXEC transactions
-- **ActiveSupport Integration**: Subscribes to ActiveSupport::Notifications for Redis events
-- **Client Instrumentation**: Direct instrumentation of Redis::Client for comprehensive coverage
-
-## Error Tracking
-
-### Exception Handling
-- **Automatic Exception Capture**: Captures exceptions from controller actions
-- **Exception Class**: Tracks exception class names
-- **Exception Messages**: Captures exception messages (truncated to 1000 chars)
-- **Full Backtraces**: Captures complete exception backtraces (first 50 lines)
-- **Request Context**: Includes full request context with exceptions
-- **Error Flagging**: Errors are marked and always tracked (even with sampling)
-
-### Error Context
-- **Controller/Action**: Identifies where the error occurred
-- **Request Parameters**: Includes request parameters at time of error
-- **User Information**: Includes user ID if available
-- **SQL Queries**: Includes SQL queries executed before error
-- **Memory State**: Includes memory usage at time of error
-- **Log Messages**: Includes application logs captured during request
-
-## HTTP Instrumentation
-
-### Outgoing HTTP Tracking
-- **HTTP Request Tracking**: Tracks outgoing HTTP requests (via middleware)
-- **Request Context**: Captures HTTP request details
-- **Response Context**: Captures HTTP response details
-- **Duration Measurement**: Tracks HTTP request duration
-
-## Configuration & Flexibility
-
-### Configuration Options
-- **API Key Management**: Multiple sources (config, Rails credentials, ENV)
-- **Endpoint Configuration**: Configurable endpoint URL
-- **Timeout Settings**: Configurable open and read timeouts
-- **Enable/Disable Toggle**: Can be enabled/disabled via configuration
-- **Environment Detection**: Automatic Rails environment detection
-
-### Circuit Breaker Configuration
-- **Failure Threshold**: Configurable failure threshold (default: 3)
-- **Recovery Timeout**: Configurable recovery timeout (default: 60 seconds)
-- **Retry Timeout**: Configurable retry timeout (default: 300 seconds)
-- **Enable/Disable**: Can enable/disable circuit breaker
-
-### Memory Tracking Configuration
-- **Memory Tracking Toggle**: Enable/disable memory tracking
-- **Allocation Tracking Toggle**: Enable/disable detailed allocation tracking
-- **Sampling Configuration**: Configurable request sampling rate
-
-### Query Analysis Configuration
-- **Slow Query Threshold**: Configurable threshold in milliseconds (default: 500ms)
-- **EXPLAIN ANALYZE Toggle**: Enable/disable automatic EXPLAIN ANALYZE
-
-## Data Safety & Privacy
-
-### Data Sanitization
-- **SQL Sanitization**: Automatically sanitizes SQL queries
-- **Parameter Filtering**: Filters sensitive parameters (password, token, secret, key)
-- **Argument Truncation**: Limits and truncates job arguments
-- **Key Truncation**: Truncates cache and Redis keys to 200 characters
-- **Value Truncation**: Recursively truncates nested values to prevent huge payloads
-- **String Limits**: Limits string values (e.g., user agent to 200 chars, messages to 1000 chars)
-
-### Data Limits
-- **Array Limits**: Limits array sizes (e.g., first 10 job arguments, first 5 array elements)
-- **Hash Limits**: Limits hash key counts (e.g., first 20 hash keys, first 30 params)
-- **Backtrace Limits**: Limits backtraces to first 50 lines
-- **Allocation Limits**: Limits allocations tracked per request (max 1000)
-
-## Performance & Reliability
-
-### Performance Optimizations
-- **Asynchronous Posting**: Non-blocking HTTP requests
-- **Lightweight Default Mode**: Minimal overhead in default configuration
-- **Sampling Support**: Reduces data volume for high-traffic applications
-- **Thread-Local Storage**: Efficient per-request data collection
-- **Background EXPLAIN**: EXPLAIN ANALYZE runs in background thread
-
-### Reliability Features
-- **Circuit Breaker**: Prevents cascading failures
-- **Error Handling**: Comprehensive error handling to prevent instrumentation failures
-- **Graceful Degradation**: Continues working even if some features fail
-- **Timeout Protection**: Configurable timeouts prevent hanging requests
-
-## Integration & Compatibility
-
-### Framework Support
-- **Rails Integration**: Full Rails integration via Railtie
-- **ActiveSupport Notifications**: Uses ActiveSupport::Notifications for event subscription
-- **ActiveRecord Integration**: Tracks ActiveRecord SQL queries
-- **ActiveJob Integration**: Tracks ActiveJob background jobs
-- **ActionView Integration**: Tracks ActionView rendering
-
-### Database Support
-- **PostgreSQL**: Full support with EXPLAIN ANALYZE
-- **MySQL**: Full support with EXPLAIN ANALYZE
-- **SQLite**: Full support with EXPLAIN QUERY PLAN
-- **Other Databases**: Basic support with standard EXPLAIN
-
-### Cache Store Support
-- **All Cache Stores**: Works with any Rails cache store
-- **Multi-Store Support**: Tracks cache operations across different stores
-
-### Redis Support
-- **Redis Gem**: Works with redis gem
-- **Client Instrumentation**: Direct instrumentation of Redis::Client
-- **Pipeline Support**: Tracks Redis pipelines
-- **Transaction Support**: Tracks Redis MULTI/EXEC transactions
-
-## Logging & Debugging
-
-### Application Logging
-- **Log Capture**: Captures application logs during request processing
-- **Log Context**: Includes logs in metric payloads
-- **Debug Logging**: Optional debug logging for skipped requests
-
-## Deployment & Environment
-
-### Deploy Tracking
-- **Deploy ID Resolution**: Multiple sources for deploy identification (`Configuration#deploy_id=` wins when set, then ENV in `Configuration::DEPLOY_REVISION_ENV_KEYS` order—including `DEAD_BRO_DEPLOY_ID`, git/CI vars, `DD_VERSION`, etc.), otherwise a **per-process UUID** (fine for single dyno/process; unusable alone for fleets like ECS replicas)
-- **Revision Tracking**: Includes deploy/revision ID in all metric payloads
-
-### Environment Support
-- **Rails Environment**: Automatic Rails environment detection
-- **Rack Environment**: Fallback to RACK_ENV or RAILS_ENV
-- **Environment Context**: Includes environment in all metric payloads
-
-## Data Collection & Transmission
-
-### Metric Payload Structure
-- **Structured Data**: Well-structured JSON payloads
-- **Event Names**: Descriptive event names for different metric types
-- **Timestamp Tracking**: ISO8601 timestamps for all metrics
-- **Metadata**: Rich metadata including environment, host, deploy ID
-
-### HTTP Client
-- **HTTPS Support**: Secure HTTPS communication
-- **Bearer Token Auth**: API key authentication via Bearer tokens
-- **JSON Encoding**: JSON-encoded payloads
-- **Custom Headers**: Proper Content-Type and Authorization headers
-
-## Comparison-Ready Features
-
-### Unique Differentiators
-1. **Automatic EXPLAIN ANALYZE**: Background execution plan capture for slow queries
-2. **Lightweight Memory Tracking**: Low-overhead memory monitoring by default
-3. **Comprehensive Cache Tracking**: Detailed cache operation tracking
-4. **Redis Instrumentation**: Full Redis command tracking including pipelines
-5. **View Rendering Analysis**: Detailed view performance analysis with cache hit rates
-6. **Flexible Exclusion Rules**: Wildcard support for controller/job exclusion
-7. **Request Sampling**: Configurable percentage-based sampling
-8. **Circuit Breaker**: Built-in resilience for APM endpoint failures
-9. **Multi-Source Configuration**: Flexible configuration from multiple sources
-10. **Deploy Tracking**: Automatic deploy ID resolution from multiple sources
-
-### Standard APM Features
-- Request/response tracking
-- SQL query tracking
-- Error tracking
-- Background job tracking
-- Memory tracking
-- Performance metrics
-- Exception handling
-- User context
-- Environment tracking
-