dead_bro 0.2.37 → 0.2.38

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 85f75545759422dfd14db796ac17035d9d6722e8e68e9c31e9ef0382358e93f8
-  data.tar.gz: beed782f1043421e364b1548db3dcd29dd1389d54d46440c2ff65aaf81a4b6af
+  metadata.gz: 4258f98fde5b0e222593ebfadee095b6402576d94a33a2b57ac26c71f0b75d47
+  data.tar.gz: 4a268f4eb24b882bf2dddd36a45db694f664319d3fa9650c3587eb59dd37f2af
 SHA512:
-  metadata.gz: 1dd65b6b589c013f24d18a548491aa6203c990cf0522c23c0e9000939898aed0e6927a26793c1a8eb80e4fe96e7a9c72176436f2cf8d3ae84848562b2fb82c2d
-  data.tar.gz: c0c020935c96898d9c54bd1b9f39ee77bf7040b8ad0cf2a9d51487af14199dca91b46e6b89b66e99c73cce470452136cb71d49ae2535f725b05edcd8b1511bdc
+  metadata.gz: 94f6cccf6594f0c50467b61a5d2bf53d6b73e7e72e6f042c2eae3cea2bad12e674e8a2f525bb7880f51f3b6cc679e3f3951dfba224b07e9701b48669060ffd51
+  data.tar.gz: eae0e565ae2d1b43f20df7c3fd14311c7543f29b1680f0495c1fa1b4ca86a30c880b20dc94ba9cc147288dff41fa39eb7aae360dafb76139b751dccc380d02e4

data/CHANGELOG.md

@@ -1,5 +1,17 @@
 ## [Unreleased]
 
+## [0.2.38] - 2026-10-26
+
+### Added
+- **Nested Request Trace.** Each request now sends its span tree as `spans`: the Rack middleware chain, the router, the controller action, view renders (layouts, templates, partials and collections), `DeadBro.watch` blocks, the block a `Rails.cache.fetch` miss runs, and SQL queries, each with an `id` and the id of the span it ran inside (`parent_id`). Repeated calls on the same path share one span with a `count`: a partial rendered 50 times from one template is one span, and the query it runs each time is one span under it. A span's `total_ms` is its calls' summed time with the spans nested in it included, and `start_ms` / `end_ms` are its first call's start and its last call's end. Cache, Redis, HTTP, Elasticsearch and transaction events carry the `parent_id` of the span they ran in, the cache event of a `Rails.cache.fetch` block and `DeadBro.watch` events the `span_id` of their span, and `sql_queries` entries the `id` their SQL spans refer to. DeadBro shows the tree with each span's total time and self time (total minus the spans nested in it). Up to 500 spans are kept per request; `spans_dropped` counts the ones left out. Background jobs don't send `spans`. An action that uses `ActionController::Live` runs on a thread of its own; its spans are recorded there, without the middleware chain.
+- **Middleware timing.** Each middleware of the app's Rails middleware stack and the router it ends in are timed as spans, so the trace shows the time each one spends before and after the app. The dashboard's middleware spans setting turns their recording off and on without a restart, and `config.middleware_spans_enabled = false` in an initializer leaves the middleware stack as Rails builds it, without the probes. Either way, the trace then starts at the controller action.
+
+### Changed
+- A request's payload is sent once the outermost middleware has returned instead of when the controller action finishes, so that the middleware spans are complete. If that middleware never returns through DeadBro's probe, for example when a timeout outside the Rails middleware stack raises into the thread at that moment, the payload is sent when the thread's next request comes in, and the next request's spans start afresh.
+
+### Fixed
+- **An HTTP request made through Faraday was recorded twice:** once by DeadBro's Faraday middleware and once by its `Net::HTTP` or Typhoeus hook underneath, so its time was counted twice. It is now recorded once, by the Faraday middleware, which also sends the request's start offset and exception class as the `Net::HTTP` hook does. Elasticsearch requests sent through Faraday are recorded once too. A request the Faraday middleware leaves out (one to localhost) is still left to the `Net::HTTP` or Typhoeus hook, as before.
+
 ## [0.2.37] - 2026-10-26
 
 ### Fixed

data/README.md

@@ -118,6 +118,20 @@ DeadBro automatically tracks view rendering performance for each request. This i
 - **Slow view detection**: Identification of the slowest rendering views
 - **Frequency analysis**: Most frequently rendered views
 
+## Request Trace (span tree)
+
+Each web request carries its span tree as `spans`, which DeadBro shows as a nested Request Trace with each span's total and self time:
+
+- **Spans**: every middleware of the Rails middleware stack and the router it ends in, the controller action, view renders (layouts, templates, partials, collections), `DeadBro.watch` blocks, the block a `Rails.cache.fetch` miss runs, and SQL queries. Each has an `id`, the `parent_id` of the span it ran inside (`nil` at the top) and a `kind` (`middleware`, `endpoint`, `action`, `view`, `watch`, `cache`, `sql`).
+- **Repeated calls**: calls with the same name under the same parent share one span with a `count`. A partial rendered 50 times from one template is one span; the query it runs each time is one span under it. `Rails.cache.fetch` blocks whose keys differ only in their digits share one span, named by the key with each run of digits as `?`.
+- **Times**: `total_ms` is the calls' summed duration, nested spans included; `start_ms` and `end_ms` are the first call's start and the last call's end, as offsets from the request's tracking start like every `start_offset_ms` (middleware that runs before DeadBro's own has negative offsets).
+- **Other events**: cache, Redis, HTTP, Elasticsearch and transaction events stay in their own lists and carry the `parent_id` of the span they ran in. The cache event of a `Rails.cache.fetch` block and `DeadBro.watch` events carry the `span_id` of their span, and SQL spans refer to their `sql_queries` entry by `ref`.
+- **Limits**: up to 500 spans per request; `spans_dropped` counts the ones left out, whose time stays in their parent's self time.
+
+Middleware is timed by a probe Rails places around each middleware when it builds the middleware stack. The middleware stack returns after the controller action has finished, so a request's payload is sent once the outermost middleware has returned. An action that uses `ActionController::Live` runs on a thread of its own, and its spans are recorded there, without the middleware chain. Background jobs don't send `spans`.
+
+The dashboard's **Middleware spans** setting (`middleware_spans_enabled`, on by default) turns their recording off and on without a restart. While it's off, the probes stay in place but record nothing: the trace starts at the controller action, and a request's payload is sent when the action finishes. To leave the middleware stack as Rails builds it, without the probes, set `config.middleware_spans_enabled = false` in `config/initializers/dead_bro.rb` (default `true`). It's read when Rails builds the middleware stack, after the app's initializers have run, so a change takes effect when the app restarts; while it's `false`, the dashboard setting can't turn the spans on.
+
 ## Memory Tracking & Leak Detection
 
 DeadBro automatically tracks memory usage and detects memory leaks with minimal performance impact. This includes:

data/lib/dead_bro/cache_subscriber.rb

@@ -69,7 +69,7 @@ module DeadBro
     def self.build_event(name, data, duration_ms, start_offset_ms = nil)
       return nil unless data.is_a?(Hash)
 
-      {
+      event = {
         event: name,
         duration_ms: duration_ms,
         start_offset_ms: start_offset_ms,
@@ -80,6 +80,17 @@ module DeadBro
         namespace: safe_namespace(data[:namespace]),
         at: Time.now.utc.to_i
       }
+      # A Rails.cache.fetch block is a span of the request's tree: its event
+      # names that span, which ran in the span the event's parent_id gives.
+      node = DeadBro::SpanTracker.cache_node(data)
+      if node
+        event[:span_id] = node[:id]
+        event[:parent_id] = node[:parent_id] if node[:parent_id]
+      else
+        parent_id = DeadBro::SpanTracker.current_id
+        event[:parent_id] = parent_id if parent_id
+      end
+      event
     rescue
       nil
     end

data/lib/dead_bro/configuration.rb

@@ -19,6 +19,14 @@ module DeadBro
     alias_method :explain_analyze_enabled, :explain_enabled
     alias_method :explain_analyze_enabled=, :explain_enabled=
 
+    # Middleware spans. false in the app's initializer leaves the middleware
+    # stack as Rails builds it, without a MiddlewareProbe around each
+    # middleware and the router, so the span tree starts at the controller
+    # action; it's read when Rails builds the stack, after the initializers.
+    # With the probes in place, the dashboard's setting of the same name turns
+    # their recording on and off (see #middleware_spans_active?).
+    attr_accessor :middleware_spans_enabled
+
     # Remote-managed settings (overwritten by backend JSON `settings` on successful API responses)
     attr_accessor :memory_tracking_enabled, :allocation_tracking_enabled, :allocation_sample_rate,
       :sample_rate, :slow_query_threshold_ms, :watch_enabled,
@@ -73,7 +81,7 @@ module DeadBro
       explain_enabled slow_query_threshold_ms max_sql_queries_to_send max_logs_to_send
       watch_enabled excluded_controllers excluded_jobs exclusive_controllers exclusive_jobs
       monitor_enabled enable_db_stats enable_process_stats enable_system_stats
-      sample_rates_by_type report_routing_errors
+      sample_rates_by_type report_routing_errors middleware_spans_enabled
     ].freeze
 
     def initialize
@@ -89,6 +97,10 @@ module DeadBro
       @explicit_deploy_revision = nil
       @disk_paths = ["/"]
       @interfaces_ignore = %w[lo lo0 docker0]
+      @middleware_spans_enabled = true
+      # The dashboard's switch for recording middleware spans; on until a
+      # settings response says otherwise (older backends never send it).
+      @remote_middleware_spans_enabled = true
 
       # Remote-managed defaults (used until backend sends real values)
       @sample_rate = 100
@@ -193,6 +205,10 @@ module DeadBro
             # must never be able to switch it on — only off. The local opt-in
             # (explain_enabled) stays untouched; see #explain_active?
             @remote_explain_enabled = !!value
+          when "middleware_spans_enabled"
+            # The probes are already in place (or not) by now; this only turns
+            # their recording on and off. The local value stays as it was.
+            @remote_middleware_spans_enabled = !!value
           when "enabled", "memory_tracking_enabled", "allocation_tracking_enabled", "watch_enabled",
                "monitor_enabled", "enable_db_stats", "enable_process_stats", "enable_system_stats",
                "report_routing_errors"
@@ -212,6 +228,13 @@ module DeadBro
       !!(@explain_enabled && @remote_explain_enabled)
     end
 
+    # Middleware spans are recorded when the app hasn't turned them off in its
+    # initializer (which also leaves the stack without probes) and the
+    # dashboard hasn't either.
+    def middleware_spans_active?
+      !!(@middleware_spans_enabled && @remote_middleware_spans_enabled)
+    end
+
     alias_method :explain_analyze_active?, :explain_active?
 
     def heartbeat_due?

data/lib/dead_bro/elasticsearch_subscriber.rb

@@ -130,7 +130,7 @@ module DeadBro
       end
 
       def build_event(method, path, status, duration_ms, start_offset_ms = nil)
-        {
+        event = {
           method: method.to_s.upcase,
           path: sanitize_path(path),
           operation: extract_operation(method, path),
@@ -138,6 +138,9 @@ module DeadBro
           duration_ms: duration_ms,
           start_offset_ms: start_offset_ms
         }
+        parent_id = DeadBro::SpanTracker.current_id
+        event[:parent_id] = parent_id if parent_id
+        event
       end
     end
   end

data/lib/dead_bro/http_instrumentation.rb

@@ -8,6 +8,10 @@ module DeadBro
     EVENT_NAME = "outgoing.http"
     THREAD_LOCAL_KEY = :dead_bro_http_events
     MAX_TRACKED_EVENTS = 1000
+    # Set while DeadBro's Faraday middleware runs the request: the adapter's
+    # own Net::HTTP or Typhoeus call is that request, recorded once by the
+    # middleware.
+    IN_FARADAY_KEY = :dead_bro_http_in_faraday
 
     def self.install!(client: Client.new)
       install_net_http!(client)
@@ -50,7 +54,11 @@ module DeadBro
               tracking_start = Thread.current[DeadBro::TRACKING_START_TIME_KEY]
               start_offset_ms = tracking_start ? ((wall_start - tracking_start) * 1000.0).round(2) : nil
 
-              if is_es_host
+              if Thread.current[DeadBro::HttpInstrumentation::IN_FARADAY_KEY]
+                # Faraday sent it through this adapter; DeadBro's Faraday
+                # middleware around the request records it.
+                nil
+              elsif is_es_host
                 # Route to elasticsearch subscriber instead of http_outgoing
                 if Thread.current[DeadBro::ElasticsearchSubscriber::THREAD_LOCAL_KEY]
                   path = (uri && uri.path) || req.path
@@ -75,6 +83,8 @@ module DeadBro
                   start_offset_ms: start_offset_ms,
                   exception: error && error.class.name
                 }
+                parent_id = DeadBro::SpanTracker.current_id
+                payload[:parent_id] = parent_id if parent_id
                 if Thread.current[THREAD_LOCAL_KEY] && DeadBro::HttpInstrumentation.should_continue_tracking?
                   Thread.current[THREAD_LOCAL_KEY] << payload
                 end
@@ -106,7 +116,9 @@ module DeadBro
                 (respond_to?(:base_url) ? base_url : nil)
               end
 
-              skip_instrumentation = req_url && (req_url.include?("localhost:3100/apm/v1/metrics") || req_url.include?("deadbro.aberatii.com/apm/v1/metrics"))
+              # Inside DeadBro's Faraday middleware, which records the request.
+              skip_instrumentation = Thread.current[DeadBro::HttpInstrumentation::IN_FARADAY_KEY] ||
+                (req_url && (req_url.include?("localhost:3100/apm/v1/metrics") || req_url.include?("deadbro.aberatii.com/apm/v1/metrics")))
 
               unless skip_instrumentation
                 payload = {
@@ -116,6 +128,8 @@ module DeadBro
                   status: response && response.code,
                   duration_ms: duration_ms
                 }
+                parent_id = DeadBro::SpanTracker.current_id
+                payload[:parent_id] = parent_id if parent_id
                 if Thread.current[THREAD_LOCAL_KEY] && DeadBro::HttpInstrumentation.should_continue_tracking?
                   Thread.current[THREAD_LOCAL_KEY] << payload
                 end
@@ -137,10 +151,21 @@ module DeadBro
         middleware_klass = Class.new(::Faraday::Middleware) do
           def call(env)
             start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+            wall_start = Time.now
             response = nil
+            error = nil
+            # A request this middleware leaves out (below) is left to the
+            # adapter's own Net::HTTP or Typhoeus hook, as before.
+            records = DeadBro::HttpInstrumentation.faraday_records?(env)
+            in_faraday = Thread.current[DeadBro::HttpInstrumentation::IN_FARADAY_KEY]
+            Thread.current[DeadBro::HttpInstrumentation::IN_FARADAY_KEY] = true if records
             begin
               response = @app.call(env)
+            rescue => e
+              error = e
+              raise
             ensure
+              Thread.current[DeadBro::HttpInstrumentation::IN_FARADAY_KEY] = in_faraday
               finish_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
               duration_ms = ((finish_time - start_time) * 1000.0).round(2)
               begin
@@ -150,7 +175,9 @@ module DeadBro
                 url_str = url.to_s
 
                 is_es_host = DeadBro::HttpInstrumentation.elasticsearch_host?(host, port)
-                skip = !is_es_host && (url_str.include?("localhost") || url_str.include?("aberatii.com"))
+
+                tracking_start = Thread.current[DeadBro::TRACKING_START_TIME_KEY]
+                start_offset_ms = tracking_start ? ((wall_start - tracking_start) * 1000.0).round(2) : nil
 
                 if is_es_host
                   if Thread.current[DeadBro::ElasticsearchSubscriber::THREAD_LOCAL_KEY]
@@ -158,10 +185,11 @@ module DeadBro
                       method: env.method.to_s.upcase,
                       path: url.path,
                       status: response&.status,
-                      duration_ms: duration_ms
+                      duration_ms: duration_ms,
+                      start_offset_ms: start_offset_ms
                     )
                   end
-                elsif !skip
+                elsif records
                   lib = DeadBro::HttpInstrumentation.typesense_host?(host, port) ? "typesense" : "faraday"
                   payload = {
                     library: lib,
@@ -170,8 +198,12 @@ module DeadBro
                     host: host,
                     path: url.path,
                     status: response&.status,
-                    duration_ms: duration_ms
+                    duration_ms: duration_ms,
+                    start_offset_ms: start_offset_ms,
+                    exception: error&.class&.name
                   }
+                  parent_id = DeadBro::SpanTracker.current_id
+                  payload[:parent_id] = parent_id if parent_id
                   key = DeadBro::HttpInstrumentation::THREAD_LOCAL_KEY
                   if Thread.current[key] && DeadBro::HttpInstrumentation.should_continue_tracking?
                     Thread.current[key] << payload
@@ -202,6 +234,16 @@ module DeadBro
     rescue
     end
 
+    # Whether DeadBro's Faraday middleware records the request: everything but
+    # localhost and DeadBro's own endpoint, Elasticsearch hosts always.
+    def self.faraday_records?(env)
+      url = env.url
+      url_str = url.to_s
+      elasticsearch_host?(url.host.to_s, url.port) || !(url_str.include?("localhost") || url_str.include?("aberatii.com"))
+    rescue *DeadBro::CONTAINED_ERRORS
+      false
+    end
+
     def self.elasticsearch_host?(host, port)
       return false if host.nil?
       return true if port == 9200

data/lib/dead_bro/middleware_probe.rb

@@ -0,0 +1,85 @@
+# frozen_string_literal: true
+
+module DeadBro
+  # Times one Rack middleware — or the router the stack ends in — as a span of
+  # the request's tree (see SpanTracker). Rails wraps every middleware in one
+  # when it builds the app's middleware stack; the probe reads a clock on the
+  # way in and out and allocates nothing.
+  class MiddlewareProbe
+    attr_reader :span_name, :span_kind
+
+    def initialize(app, name, kind)
+      @app = app
+      @span_name = name
+      @span_kind = kind
+    end
+
+    def call(env)
+      frame = SpanTracker.rack_enter(self)
+      @app.call(env)
+    ensure
+      SpanTracker.rack_leave(frame) if frame
+    end
+
+    # Anything else asked of the middleware goes to it unchanged.
+    def respond_to_missing?(method_name, include_private = false)
+      @app.respond_to?(method_name, include_private) || super
+    end
+
+    def method_missing(method_name, *args, &block)
+      if @app.respond_to?(method_name)
+        @app.public_send(method_name, *args, &block)
+      else
+        super
+      end
+    end
+    ruby2_keywords(:method_missing) if respond_to?(:ruby2_keywords, true)
+
+    # ActionDispatch::MiddlewareStack::Middleware#build: one probe per middleware.
+    module MiddlewareBuild
+      def build(app)
+        return super unless MiddlewareProbe.enabled?
+
+        MiddlewareProbe.wrap(super, inspect, SpanTracker::MIDDLEWARE)
+      end
+    end
+
+    # ActionDispatch::MiddlewareStack#build: a probe around the endpoint (the
+    # app's routes), so routing and controller setup aren't counted as the
+    # innermost middleware's own time.
+    module StackBuild
+      def build(app = nil, &block)
+        endpoint = app || block
+        return super if endpoint.nil? || !MiddlewareProbe.enabled?
+
+        super(MiddlewareProbe.wrap(endpoint, endpoint.class.name, SpanTracker::ENDPOINT))
+      end
+    end
+
+    # config.middleware_spans_enabled, read when Rails builds the stack.
+    def self.enabled?
+      DeadBro.configuration.middleware_spans_enabled ? true : false
+    rescue
+      true
+    end
+
+    def self.wrap(app, name, kind)
+      new(app, name.to_s.dup.freeze, kind)
+    rescue
+      app
+    end
+
+    # Must run before Rails builds the middleware stack (the
+    # build_middleware_stack initializer); a stack built earlier has no probes.
+    def self.install!
+      return false unless defined?(::ActionDispatch::MiddlewareStack)
+
+      stack = ::ActionDispatch::MiddlewareStack
+      stack.prepend(StackBuild) unless stack.ancestors.include?(StackBuild)
+      stack::Middleware.prepend(MiddlewareBuild) unless stack::Middleware.ancestors.include?(MiddlewareBuild)
+      true
+    rescue
+      false
+    end
+  end
+end

data/lib/dead_bro/railtie.rb

@@ -53,6 +53,11 @@ if defined?(Rails) && defined?(Rails::Railtie)
           require "dead_bro/view_rendering_subscriber"
           DeadBro::ViewRenderingSubscriber.subscribe!(client: shared_client)
 
+          # Install the span tree (controller action and view spans; the
+          # middleware spans come from the probes installed below)
+          require "dead_bro/span_tracker"
+          DeadBro::SpanTracker.subscribe!
+
           # Install lightweight memory tracking (default)
           require "dead_bro/lightweight_memory_tracker"
           require "dead_bro/memory_leak_detector"
@@ -104,6 +109,17 @@ if defined?(Rails) && defined?(Rails::Railtie)
         # Never raise in Railtie init
       end
 
+      # Time every middleware and the router for the span tree. Railtie
+      # initializers run before the application's build_middleware_stack, so
+      # the probes are in place when Rails builds the stack.
+      initializer "dead_bro.middleware_probes" do
+        require "dead_bro/span_tracker"
+        require "dead_bro/middleware_probe"
+        DeadBro::MiddlewareProbe.install!
+      rescue
+        # Never raise in Railtie init
+      end
+
       # Insert SQL tracking middleware
       initializer "dead_bro.sql_tracking_middleware" do |app|
         require "dead_bro/sql_tracking_middleware"

data/lib/dead_bro/redis_subscriber.rb

@@ -90,6 +90,8 @@ module DeadBro
                 db: RedisSubscriber.safe_db(@db),
                 error: error ? error.class.name : nil
               }
+              parent_id = DeadBro::SpanTracker.current_id
+              event[:parent_id] = parent_id if parent_id
 
               if Thread.current[RedisSubscriber::THREAD_LOCAL_KEY] && RedisSubscriber.should_continue_tracking?
                 Thread.current[RedisSubscriber::THREAD_LOCAL_KEY] << event
@@ -122,6 +124,8 @@ module DeadBro
                 start_offset_ms: start_offset_ms,
                 db: RedisSubscriber.safe_db(@db)
               }
+              parent_id = DeadBro::SpanTracker.current_id
+              event[:parent_id] = parent_id if parent_id
 
               if Thread.current[RedisSubscriber::THREAD_LOCAL_KEY] && RedisSubscriber.should_continue_tracking?
                 Thread.current[RedisSubscriber::THREAD_LOCAL_KEY] << event
@@ -154,6 +158,8 @@ module DeadBro
                 start_offset_ms: start_offset_ms,
                 db: RedisSubscriber.safe_db(@db)
               }
+              parent_id = DeadBro::SpanTracker.current_id
+              event[:parent_id] = parent_id if parent_id
 
               if Thread.current[RedisSubscriber::THREAD_LOCAL_KEY] && RedisSubscriber.should_continue_tracking?
                 Thread.current[RedisSubscriber::THREAD_LOCAL_KEY] << event
@@ -220,7 +226,7 @@ module DeadBro
 
     def self.build_event(name, data, duration_ms, start_offset_ms = nil)
       cmd = extract_command(data)
-      {
+      event = {
         event: name.to_s,
         command: cmd[:command],
         key: cmd[:key],
@@ -229,6 +235,9 @@ module DeadBro
         start_offset_ms: start_offset_ms,
         db: safe_db(data[:db])
       }
+      parent_id = DeadBro::SpanTracker.current_id
+      event[:parent_id] = parent_id if parent_id
+      event
     rescue
       nil
     end

data/lib/dead_bro/span_tracker.rb

@@ -0,0 +1,650 @@
+# frozen_string_literal: true
+
+module DeadBro
+  # Builds a web request's span tree: every span gets an id and the id of the
+  # span it ran inside (parent_id), so the dashboard can nest the Request Trace
+  # and show each span's total (inclusive) and self (exclusive) time.
+  #
+  # Nodes are kept for the spans other spans run inside — Rack middleware and
+  # the router (MiddlewareProbe), the controller action, view renders,
+  # DeadBro.watch blocks and the block a Rails.cache.fetch miss runs — and for
+  # SQL queries, and are sent as the request's `spans`. Repeated calls on the
+  # same path share one node: a partial rendered 50 times inside one template
+  # is one node with count 50, and the query it runs each time is one child
+  # node with count 50. Cache, Redis, HTTP, Elasticsearch and transaction
+  # events stay in their own lists and carry the parent_id of the span they
+  # ran inside; the cache event of a fetch block carries its node's span_id.
+  #
+  # Node fields: id, parent_id (nil at the top), kind (middleware, endpoint,
+  # action, view, watch, cache, sql), name, count, total_ms — the summed
+  # duration of the node's calls, nested spans included — and start_ms /
+  # end_ms — the first call's start and the last call's end, as offsets from
+  # the tracking start like every other start_offset_ms. View nodes add type
+  # (template, layout, partial, collection) and cache nodes type generate;
+  # SQL nodes carry ref, the id of their sql_queries entry, instead of a name.
+  #
+  # The middleware chain returns after the controller action has finished, so
+  # a request that runs inside it has its payload held until the outermost
+  # middleware returns (see .post), and is sent with the middleware spans. If
+  # that middleware never returns through its probe, the payload is sent when
+  # the thread's next request comes in (see .rack_enter).
+  #
+  # The tree belongs to the thread that runs the request. ActionController::Live
+  # runs the action on a thread of its own, with the request thread's locals
+  # copied over; that thread records the action in a tree of its own (see
+  # .live_context).
+  module SpanTracker
+    THREAD_KEY = :dead_bro_span_context
+
+    # Nodes kept per request; past it, a span's time stays in its parent's self
+    # time and the request's spans_dropped counts what was left out.
+    MAX_NODES = 500
+    # Middleware/router frames kept per request (a Rails stack has ~25).
+    MAX_RACK_FRAMES = 200
+    # Payloads held for one middleware chain: the request's own, and the
+    # exceptions app's when Rails dispatches an error to it. Past this a
+    # payload is sent right away.
+    MAX_PENDING = 10
+
+    MIDDLEWARE = "middleware"
+    ENDPOINT = "endpoint"
+    ACTION = "action"
+    VIEW = "view"
+    WATCH = "watch"
+    CACHE = "cache"
+    SQL = "sql"
+
+    # Frame pushed for a span past MAX_NODES, so leaving it still pops the
+    # right frame.
+    DROPPED_VIEW = {kind: VIEW}.freeze
+    DROPPED_WATCH = {kind: WATCH}.freeze
+    DROPPED_CACHE = {kind: CACHE}.freeze
+
+    # Rails.cache.fetch runs its block inside this event when the key misses.
+    CACHE_GENERATE_EVENT = "cache_generate.active_support"
+    # Fetch blocks whose keys differ only in their digits share a node.
+    CACHE_KEY_DIGITS = /\d+/
+
+    VIEW_EVENTS = {
+      "render_layout.action_view" => "layout",
+      "render_template.action_view" => "template",
+      "render_partial.action_view" => "partial",
+      "render_collection.action_view" => "collection"
+    }.freeze
+
+    # One request's tree, reused by the thread's next request.
+    class Context
+      attr_reader :nodes, :dropped, :thread
+      attr_accessor :pending
+
+      def initialize
+        # The thread this tree is recorded on. Another thread holding this
+        # context (ActionController::Live copies the request thread's locals)
+        # records nothing into it.
+        @thread = Thread.current
+        # Open spans inside the middleware chain, innermost last: a node Hash
+        # or a DROPPED_* marker. The ids are what starts now gets as
+        # parent_id; the starts are Float seconds (CLOCK_REALTIME, the clock
+        # behind every other offset in the payload).
+        @frames = []
+        @frame_ids = []
+        @frame_starts = []
+        # parent_id => {key => node}, per kind, so repeat calls on a path find
+        # their node.
+        @view_index = {}
+        @watch_index = {}
+        @cache_index = {}
+        @sql_index = {}
+        # Each fetch block's event payload => its node, for CacheSubscriber,
+        # whose listener for the same event may run before or after ours.
+        @cache_payloads = {}.compare_by_identity
+        # Middleware frames, as parallel arrays: the chain runs on every
+        # request, so recording it allocates nothing. Frame i is span i + 1
+        # (the other spans are numbered after MAX_RACK_FRAMES); @rack_current
+        # is the innermost open one and @rack_parents each one's enclosing
+        # frame.
+        @rack_probes = []
+        @rack_parents = []
+        @rack_starts = []
+        @rack_ends = []
+        reset
+      end
+
+      def reset
+        # A new Array: the last one may still be in a payload waiting to be sent.
+        @nodes = []
+        @frames.clear
+        @frame_ids.clear
+        @frame_starts.clear
+        @view_index.clear
+        @watch_index.clear
+        @cache_index.clear
+        @sql_index.clear
+        @cache_payloads.clear
+        @rack_probes.clear
+        @rack_parents.clear
+        @rack_starts.clear
+        @rack_ends.clear
+        @rack_current = nil
+        @next_id = MAX_RACK_FRAMES
+        @base = nil
+        @tracking = false
+        @recording_rack = false
+        @action = nil
+        @pending = nil
+        @dropped = 0
+        @finalized = false
+      end
+
+      def tracking?
+        @tracking
+      end
+
+      def idle?
+        @frames.empty? && @pending.nil? && !rack_open?
+      end
+
+      # Left over from a request whose outermost middleware never returned
+      # through its probe — an exception raised into the thread from outside
+      # the Rails stack (an outer timeout) while the probe opened or closed its
+      # frame: nothing is open or running, or the outermost probe is entered
+      # again while its own frame still is.
+      def abandoned?(probe)
+        return false if idle?
+
+        (@rack_current.nil? && @frames.empty?) || (@rack_current == 0 && @rack_probes[0].equal?(probe))
+      end
+
+      def current_id
+        @frame_ids.last || (@rack_current && @rack_current + 1)
+      end
+
+      # The tracking start (TRACKING_START_TIME_KEY), which every offset is
+      # measured from.
+      def start_tracking(base)
+        @base = base
+        @tracking = true
+      end
+
+      def stop_tracking
+        @tracking = false
+      end
+
+      # True while the outermost middleware hasn't returned yet.
+      def rack_open?
+        !@rack_ends.empty? && @rack_ends[0].nil?
+      end
+
+      def start_rack
+        @recording_rack = true
+      end
+
+      # A Rack app called from inside the action isn't recorded: middleware
+      # frames are the outermost spans.
+      def rack_enter(probe, now)
+        return nil unless @recording_rack && @frames.empty?
+
+        i = @rack_probes.size
+        return nil if i >= MAX_RACK_FRAMES
+
+        @rack_probes << probe
+        @rack_parents << @rack_current
+        @rack_starts << now
+        @rack_ends << nil
+        @rack_current = i
+        i
+      end
+
+      def rack_leave(i, now)
+        @rack_ends[i] = now
+        @rack_current = @rack_parents[i]
+        # Anything still open inside it never finished.
+        pop_to(0) unless @frames.empty?
+      end
+
+      def enter_action(now)
+        node = new_node(ACTION, current_id)
+        return nil unless node
+
+        @action = node
+        push(node, node[:id], now)
+        node
+      end
+
+      # Closed with the times process_action reports, which every listener of
+      # the event sees, rather than when this listener happens to run.
+      def finish_action(started, finished)
+        j = innermost(ACTION)
+        return unless j
+
+        node = @frames[j]
+        pop_to(j)
+        add_call(node, started, finished)
+      end
+
+      def enter_view(identifier, type, now)
+        key = identifier || type
+        node = node_for(VIEW, @view_index, key) do |created|
+          created[:name] = DeadBro::ViewRenderingSubscriber.safe_identifier(identifier)
+          created[:type] = type
+        end
+        push(node || DROPPED_VIEW, node ? node[:id] : current_id, now)
+      end
+
+      def leave_view(now)
+        j = innermost(VIEW)
+        leave_frame(j, now) if j
+      end
+
+      def enter_watch(label, now)
+        node = node_for(WATCH, @watch_index, label) { |created| created[:name] = label }
+        frame = node || DROPPED_WATCH
+        push(frame, node ? node[:id] : current_id, now)
+        frame
+      end
+
+      def leave_watch(frame, now)
+        j = @frames.size - 1
+        j -= 1 while j >= 0 && !@frames[j].equal?(frame)
+        leave_frame(j, now) if j >= 0
+      end
+
+      # A Rails.cache.fetch miss: the block it runs nests in this span. Keys
+      # that differ only in their digits share a node, named by that pattern.
+      def enter_cache_generate(payload, key, now)
+        pattern = key.gsub(CACHE_KEY_DIGITS, "?")
+        node = node_for(CACHE, @cache_index, pattern) do |created|
+          created[:name] = key
+          created[:type] = "generate"
+        end
+        if node
+          node[:name] = pattern unless node[:name] == key
+          @cache_payloads[payload] = node
+        end
+        push(node || DROPPED_CACHE, node ? node[:id] : current_id, now)
+      end
+
+      def leave_cache_generate(now)
+        j = innermost(CACHE)
+        leave_frame(j, now) if j
+      end
+
+      def cache_node(payload)
+        @cache_payloads[payload]
+      end
+
+      def sql(key, ref, started, finished)
+        node = node_for(SQL, @sql_index, key) { |created| created[:ref] = ref }
+        add_call(node, started, finished) if node
+      end
+
+      # Rounds the totals and turns the middleware frames into nodes. Once per
+      # request, right before the payload is sent.
+      def finalize(action_name)
+        return if @finalized || @base.nil?
+
+        @finalized = true
+        @action[:name] ||= action_name if @action
+        @nodes.each { |node| node[:total_ms] = node[:total_ms].round(3) }
+        @rack_probes.each_with_index do |probe, i|
+          start = @rack_starts[i]
+          finish = @rack_ends[i] || open_frame_end(start)
+          parent = @rack_parents[i]
+          @nodes << {
+            id: i + 1,
+            parent_id: parent && parent + 1,
+            kind: probe.span_kind,
+            name: probe.span_name,
+            count: 1,
+            total_ms: ((finish - start) * 1000.0).round(3),
+            start_ms: offset_ms(start),
+            end_ms: offset_ms(finish)
+          }
+        end
+      end
+
+      private
+
+      def push(frame, id, now)
+        @frames << frame
+        @frame_ids << id
+        @frame_starts << now
+      end
+
+      # Pops frame j and every frame still open inside it.
+      def pop_to(j)
+        (@frames.size - j).times do
+          @frames.pop
+          @frame_ids.pop
+          @frame_starts.pop
+        end
+      end
+
+      # The innermost open frame of a kind (a node or a DROPPED_* marker).
+      # A loop rather than rindex with a block, which allocates.
+      def innermost(kind)
+        j = @frames.size - 1
+        while j >= 0
+          frame = @frames[j]
+          return j if frame.is_a?(Hash) && frame[:kind] == kind
+
+          j -= 1
+        end
+        nil
+      end
+
+      def leave_frame(j, now)
+        frame = @frames[j]
+        start = @frame_starts[j]
+        pop_to(j)
+        add_call(frame, start, now) unless frame.frozen?
+      end
+
+      # The node for key under the innermost open span, created on first use;
+      # nil past MAX_NODES.
+      def node_for(kind, index, key)
+        parent_id = current_id
+        by_key = (index[parent_id] ||= {})
+        node = by_key[key]
+        return node if node
+
+        node = new_node(kind, parent_id)
+        return nil unless node
+
+        yield node
+        by_key[key] = node
+      end
+
+      def new_node(kind, parent_id)
+        if @nodes.size >= MAX_NODES
+          @dropped += 1
+          return nil
+        end
+
+        node = {id: (@next_id += 1), parent_id: parent_id, kind: kind, count: 0, total_ms: 0.0, start_ms: nil, end_ms: nil}
+        @nodes << node
+        node
+      end
+
+      def add_call(node, started, finished)
+        node[:count] += 1
+        node[:total_ms] += (finished - started) * 1000.0
+        node[:start_ms] ||= offset_ms(started)
+        node[:end_ms] = offset_ms(finished)
+      end
+
+      def offset_ms(time)
+        ((time - @base) * 1000.0).round(3)
+      end
+
+      # A frame sent while still open (its middleware never returned through
+      # the probe) ends where the last frame that did close ended.
+      def open_frame_end(start)
+        finish = start
+        @rack_ends.each { |time| finish = time if time && time > finish }
+        finish
+      end
+    end
+
+    # Pushes and pops the view spans; start/finish listeners, so a view's
+    # children see it as their parent while it renders.
+    class ViewListener
+      def initialize(type)
+        @type = type
+      end
+
+      def start(_name, _id, payload)
+        SpanTracker.enter_view(payload[:identifier], @type)
+      end
+
+      def finish(_name, _id, _payload)
+        SpanTracker.leave_view
+      end
+    end
+
+    # start_processing.action_controller fires right before
+    # process_action.action_controller starts: the action span opens here and
+    # Subscriber closes it.
+    class ActionListener
+      def start(_name, _id, _payload)
+        SpanTracker.enter_action
+      end
+
+      def finish(_name, _id, _payload)
+      end
+    end
+
+    # cache_generate wraps the block of a Rails.cache.fetch miss, so what the
+    # block runs nests in its span instead of sitting next to it.
+    class CacheGenerateListener
+      def start(_name, _id, payload)
+        SpanTracker.enter_cache_generate(payload)
+      end
+
+      def finish(_name, _id, _payload)
+        SpanTracker.leave_cache_generate
+      end
+    end
+
+    module_function
+
+    def subscribe!
+      return unless defined?(ActiveSupport::Notifications)
+
+      VIEW_EVENTS.each do |event_name, type|
+        ActiveSupport::Notifications.subscribe(event_name, ViewListener.new(type))
+      end
+      ActiveSupport::Notifications.subscribe("start_processing.action_controller", ActionListener.new)
+      ActiveSupport::Notifications.subscribe(CACHE_GENERATE_EVENT, CacheGenerateListener.new)
+    rescue
+      # Never raise from instrumentation install
+    end
+
+    def now
+      Process.clock_gettime(Process::CLOCK_REALTIME)
+    end
+
+    # This thread's context, created on first use — and in place of one
+    # copied from another thread.
+    def context
+      own_context || (Thread.current[THREAD_KEY] = Context.new)
+    end
+
+    def own_context
+      ctx = Thread.current[THREAD_KEY]
+      ctx if ctx && ctx.thread.equal?(Thread.current)
+    end
+
+    # The context of a request being tracked on this thread, else nil.
+    def tracking_context
+      ctx = own_context
+      ctx if ctx&.tracking?
+    end
+
+    # ActionController::Live runs the action on a thread of its own, with the
+    # request thread's locals copied over — that thread's context and the
+    # tracking start among them. The request thread leaves the middleware
+    # chain once the response is committed, while the action may still run,
+    # so the action thread records the action in a context of its own,
+    # measured from the same tracking start. Nil anywhere else.
+    def live_context
+      copied = Thread.current[THREAD_KEY]
+      return nil if copied.nil? || copied.thread.equal?(Thread.current)
+
+      tracking_start = Thread.current[DeadBro::TRACKING_START_TIME_KEY]
+      return nil unless tracking_start
+
+      ctx = Thread.current[THREAD_KEY] = Context.new
+      ctx.start_tracking(tracking_start.to_f)
+      ctx
+    end
+
+    # The id of the span running now — the parent_id of an event recorded now.
+    def current_id
+      tracking_context&.current_id
+    end
+
+    # Called by SqlTrackingMiddleware once tracking is set up. Without
+    # MiddlewareProbe frames around it (no Rails middleware stack) the request
+    # starts here.
+    def start_request(tracking_start)
+      ctx = context
+      ctx.reset if ctx.idle?
+      ctx.start_tracking(tracking_start.to_f)
+    rescue *DeadBro::CONTAINED_ERRORS
+    end
+
+    def stop_request
+      own_context&.stop_tracking
+    rescue *DeadBro::CONTAINED_ERRORS
+    end
+
+    # Returns a frame token for rack_leave, or nil when nothing is recorded.
+    def rack_enter(probe)
+      config = DeadBro.configuration
+      return nil unless config.enabled
+
+      ctx = own_context
+      if ctx&.abandoned?(probe)
+        # Send what the abandoned request held, and start over — also while
+        # the dashboard has middleware spans off, so it doesn't linger.
+        begin
+          flush(ctx)
+        ensure
+          ctx.reset
+        end
+      end
+      return nil unless config.middleware_spans_active?
+
+      ctx ||= context
+      if ctx.idle?
+        ctx.reset
+        ctx.start_rack
+      end
+      ctx.rack_enter(probe, now)
+    rescue *DeadBro::CONTAINED_ERRORS
+      nil
+    end
+
+    # Leaving the outermost frame (0) sends the payloads held for it.
+    def rack_leave(frame)
+      ctx = own_context
+      return unless ctx
+
+      ctx.rack_leave(frame, now)
+      return unless frame == 0
+
+      begin
+        flush(ctx)
+      ensure
+        ctx.reset
+      end
+    rescue *DeadBro::CONTAINED_ERRORS
+    end
+
+    def enter_action
+      (tracking_context || live_context)&.enter_action(now)
+    rescue *DeadBro::CONTAINED_ERRORS
+    end
+
+    def finish_action(started, finished)
+      tracking_context&.finish_action(started.to_f, finished.to_f)
+    rescue *DeadBro::CONTAINED_ERRORS
+    end
+
+    def enter_view(identifier, type)
+      tracking_context&.enter_view(identifier, type, now)
+    rescue *DeadBro::CONTAINED_ERRORS
+    end
+
+    def leave_view
+      tracking_context&.leave_view(now)
+    rescue *DeadBro::CONTAINED_ERRORS
+    end
+
+    # Returns the frame to pass to leave_watch, nil when not tracking.
+    def enter_watch(label)
+      tracking_context&.enter_watch(label, now)
+    rescue *DeadBro::CONTAINED_ERRORS
+      nil
+    end
+
+    def leave_watch(frame)
+      return unless frame
+
+      tracking_context&.leave_watch(frame, now)
+    rescue *DeadBro::CONTAINED_ERRORS
+    end
+
+    def enter_cache_generate(payload)
+      ctx = tracking_context
+      return unless ctx
+
+      key = defined?(DeadBro::CacheSubscriber) ? DeadBro::CacheSubscriber.safe_key(payload[:key]) : payload[:key]
+      ctx.enter_cache_generate(payload, key.to_s, now)
+    rescue *DeadBro::CONTAINED_ERRORS
+    end
+
+    def leave_cache_generate
+      tracking_context&.leave_cache_generate(now)
+    rescue *DeadBro::CONTAINED_ERRORS
+    end
+
+    # The node of a fetch block's cache_generate event, by the event's
+    # payload; nil for every other event.
+    def cache_node(payload)
+      tracking_context&.cache_node(payload)
+    rescue *DeadBro::CONTAINED_ERRORS
+      nil
+    end
+
+    # A query of the request's own SQL tracking (sql_depth 1: an inline job's
+    # queries go to the job's payload, whose sql_queries ids are its own).
+    def sql(key, ref, started, finished, sql_depth)
+      return unless sql_depth == 1
+
+      tracking_context&.sql(key, ref, started.to_f, finished.to_f)
+    rescue *DeadBro::CONTAINED_ERRORS
+    end
+
+    # The request's nodes for the payload (finished by .post), or nil when the
+    # request isn't being tracked — e.g. the exceptions app Rails dispatches to
+    # after DeadBro's middleware has returned, whose context still holds the
+    # failed request's tree.
+    def payload_spans
+      ctx = tracking_context
+      return nil unless ctx && !ctx.nodes.empty?
+
+      ctx.nodes
+    end
+
+    def payload_spans_dropped
+      ctx = tracking_context
+      ctx.dropped if ctx && ctx.dropped > 0
+    end
+
+    # Sends a request's payload. While the middleware chain is still running
+    # the payload is held and sent when its outermost middleware returns, with
+    # the middleware spans closed.
+    def post(client, event_name, payload, action_name)
+      ctx = own_context
+      if ctx&.rack_open? && (ctx.pending.nil? || ctx.pending.size < MAX_PENDING)
+        (ctx.pending ||= []) << [client, event_name, payload, action_name]
+      else
+        ctx&.finalize(action_name)
+        client.post_metric(event_name: event_name, payload: payload, force: true)
+      end
+    end
+
+    def flush(ctx)
+      pending = ctx.pending
+      return unless pending
+
+      ctx.pending = nil
+      pending.each do |client, event_name, payload, action_name|
+        ctx.finalize(action_name)
+        client.post_metric(event_name: event_name, payload: payload, force: true)
+      end
+    end
+  end
+end

data/lib/dead_bro/sql_subscriber.rb

@@ -156,11 +156,14 @@ module DeadBro
       tracking_start = Thread.current[DeadBro::TRACKING_START_TIME_KEY]
       start_offset_ms = tracking_start ? ((started - tracking_start) * 1000.0).round(2) : nil
 
-      current << {
+      event = {
         event: name,
         duration_ms: ((finished - started) * 1000.0).round(2),
         start_offset_ms: start_offset_ms
       }
+      parent_id = DeadBro::SpanTracker.current_id
+      event[:parent_id] = parent_id if parent_id
+      current << event
     rescue
       nil
     end
@@ -251,7 +254,9 @@ module DeadBro
             agg[:n_plus_one]         = true if call_count >= N_PLUS_ONE_THRESHOLD
             agg[:backtrace]          = captured_trace if agg[:backtrace].empty? && !captured_trace.empty?
           else
-            aggregates_h[normalized_key] = {
+            agg = aggregates_h[normalized_key] = {
+              # What the request's span tree refers to this query by (ref).
+              id:                aggregates_h.size + 1,
               sql:               sanitized_sql,
               name:              data[:name],
               count:             1,
@@ -266,6 +271,7 @@ module DeadBro
               explain_plan:      nil
             }
           end
+          DeadBro::SpanTracker.sql(normalized_key, agg[:id], started, finished, agg_stack.length)
         end
 
         if should_continue_tracking?(current, MAX_TRACKED_QUERIES)

data/lib/dead_bro/sql_tracking_middleware.rb

@@ -38,6 +38,9 @@ module DeadBro
         DeadBro::SqlSubscriber.start_request_tracking
       end
 
+      # Start the span tree, measured from the same rack entry
+      DeadBro::SpanTracker.start_request(rack_entry) if defined?(DeadBro::SpanTracker)
+
       # Start cache tracking for this request
       if defined?(DeadBro::CacheSubscriber)
         DeadBro::CacheSubscriber.start_request_tracking
@@ -158,6 +161,9 @@ module DeadBro
     ensure
       Thread.current[:dead_bro_alloc_active] = nil
       Thread.current[DeadBro::TRACKING_START_TIME_KEY] = nil
+      # Keeps the spans (and a payload held for the middleware chain) until
+      # the outermost middleware returns.
+      DeadBro::SpanTracker.stop_request if defined?(DeadBro::SpanTracker)
     end
 
     def parse_queue_start(env, rack_entry)

data/lib/dead_bro/subscriber.rb

@@ -13,6 +13,10 @@ module DeadBro
         # drop with an excluded request) — ErrorMiddleware must not send it again.
         mark_reported(data)
 
+        # Closes the action span opened at start_processing, whether or not
+        # this request is sent.
+        DeadBro::SpanTracker.finish_action(started, finished)
+
         # When disabled remotely, fire a heartbeat at most once per minute so the gem
         # can detect when tracking has been re-enabled, then skip all tracking.
         unless DeadBro.configuration.enabled
@@ -191,6 +195,14 @@ module DeadBro
           watch_events: watch_events,
           logs: DeadBro.logger.logs
         }
+        # The span tree (see SpanTracker); the middleware spans are added when
+        # the payload is sent.
+        spans = DeadBro::SpanTracker.payload_spans
+        if spans
+          detail_fields[:spans] = spans
+          spans_dropped = DeadBro::SpanTracker.payload_spans_dropped
+          detail_fields[:spans_dropped] = spans_dropped if spans_dropped
+        end
 
         # Report exceptions attached to this action (e.g. controller/view errors)
         if data[:exception] || data[:exception_object]
@@ -223,7 +235,7 @@ module DeadBro
             )
 
             event_name = (exception_class || exception_obj&.class&.name || "exception").to_s
-            client.post_metric(event_name: event_name, payload: error_payload, force: true)
+            DeadBro::SpanTracker.post(client, event_name, error_payload, request_type_key)
           rescue
           ensure
             next
@@ -247,10 +259,12 @@ module DeadBro
           user_agent: safe_user_agent(data),
           user_id: extract_user_id(data)
         )
-        # force: true — the sampling decision (global or per-request-type) was
-        # already made above; client#post_metric must not re-roll it with the
-        # global-only rate, which would silently override a per-type sample rate.
-        client.post_metric(event_name: name, payload: payload, force: true)
+        # Sent with force: true — the sampling decision (global or
+        # per-request-type) was already made above; client#post_metric must not
+        # re-roll it with the global-only rate, which would silently override a
+        # per-type sample rate. Held until the middleware chain returns when the
+        # request runs inside one (see SpanTracker.post).
+        DeadBro::SpanTracker.post(client, name, payload, request_type_key)
       rescue *DeadBro::CONTAINED_ERRORS
         # Anything raised here would fail the host's request (see
         # DeadBro::CONTAINED_ERRORS) — drop this request's metrics instead.

data/lib/dead_bro/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module DeadBro
-  VERSION = "0.2.37"
+  VERSION = "0.2.38"
 end

data/lib/dead_bro/watch_tracker.rb

@@ -91,6 +91,8 @@ module DeadBro
       frame[:slot] = events.length
       events << nil
       active_stack << frame
+      # The block's span in the request's span tree (nil in a job).
+      span_frame = DeadBro::SpanTracker.enter_watch(sanitized_label)
 
       begin
         yield
@@ -104,6 +106,8 @@ module DeadBro
         raise
       ensure
         active_stack.pop if active_stack.last.equal?(frame)
+        DeadBro::SpanTracker.leave_watch(span_frame)
+        frame[:span_id] = span_frame[:id] if span_frame.is_a?(Hash)
         append_completed_span(events, frame)
       end
     end
@@ -178,6 +182,7 @@ module DeadBro
         error: frame[:error] == true
       }
       span[:exception_class] = frame[:exception_class] if frame[:exception_class]
+      span[:span_id] = frame[:span_id] if frame[:span_id]
       tags = frame[:tags]
       span[:tags] = tags if tags.is_a?(Hash) && tags.any?
 

data/lib/dead_bro.rb

@@ -30,6 +30,8 @@ module DeadBro
   autoload :MemoryDetails, "dead_bro/memory_details"
   autoload :Logger, "dead_bro/logger"
   autoload :WatchTracker, "dead_bro/watch_tracker"
+  autoload :SpanTracker, "dead_bro/span_tracker"
+  autoload :MiddlewareProbe, "dead_bro/middleware_probe"
   autoload :Sanitizer, "dead_bro/sanitizer"
   begin
     require "dead_bro/railtie"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: dead_bro
 version: !ruby/object:Gem::Version
-  version: 0.2.37
+  version: 0.2.38
 platform: ruby
 authors:
 - Emanuel Comsa
@@ -51,10 +51,12 @@ files:
 - lib/dead_bro/memory_leak_detector.rb
 - lib/dead_bro/memory_phase_tracker.rb
 - lib/dead_bro/memory_tracking_subscriber.rb
+- lib/dead_bro/middleware_probe.rb
 - lib/dead_bro/monitor.rb
 - lib/dead_bro/railtie.rb
 - lib/dead_bro/redis_subscriber.rb
 - lib/dead_bro/sanitizer.rb
+- lib/dead_bro/span_tracker.rb
 - lib/dead_bro/sql_subscriber.rb
 - lib/dead_bro/sql_tracking_middleware.rb
 - lib/dead_bro/subscriber.rb