forge_ops_tracker 0.11.0 → 0.12.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: af67d3b6d07322ccfd80d56256e5e600c264f60394058acab51b7bfc660435fa
-  data.tar.gz: 2716944aed397bb03c754900b2e34f3abc49247b43b6a97b411cec6b6a3ccbc2
+  metadata.gz: 70707edca2368561ce9b785b91dc9e8a26280071ec3db9ed34fa71db6c402fcf
+  data.tar.gz: a6fc1013df0404e46e0683749f2cedd688ac30273bda4ac113a7dd6334a19430
 SHA512:
-  metadata.gz: 2952a5218507cfceb871821b1f79add47d255b45577ee4bc67a782b574e3d41135d1f971d966846666c3bbcc1de6e86e3c3c7671e8a5ec04b31d9dabf4af0cc4
-  data.tar.gz: 32efc755d0d05fca0dd4f4354d00808caf135da2c0a22ad2b9d417017a3e73406b6d1b879da3e5862724340d52222fec9d1fbbfebc45efee58101e824b105373
+  metadata.gz: 420ff144de5e84636348c245fef836cf240c94fe6bccb49217f8d51455a3c08ce81c18dadcf7260ccc46f35dd7b2470d809b1eecd92d9bea70e17ecf149f1a78
+  data.tar.gz: 74fc60b3d7451827618e0d63deef8cd6563d29eab144c8556bc912e91bbb4a7c8467110ae25940a7e4a7457db8718cad8b763ff752311014ed325b22ce5c13b0

data/CHANGELOG.md

@@ -1,5 +1,36 @@
 # Changelog
 
+## 0.12.0
+
+- Every error event reported during a request now says where it happened: `transaction_name` (the
+  same `CheckoutsController#create` name performance samples and traces already use), `endpoint`
+  (the HTTP method plus the route as declared in your routes file, e.g. `POST /checkout` or
+  `GET /orders/:id`, never the literal path, so no ids or tokens; falls back to
+  `POST CheckoutsController#create` before Rails 7.1), and `trace_id` (the request's W3C trace id).
+  All three are set as soon as the controller starts handling the request, so an error reported
+  from inside the action with `Rails.error.handle`/`record` already has them. Left out entirely
+  outside a request (a background job, a console session). Structured fields, so they're never
+  PII-scrubbed.
+- Unhandled request errors keep their context. Rails 8.1 reports an unhandled request exception
+  from its executor, outside this gem's middleware, after the gem had already cleared its
+  per-request state, so the affected user and the breadcrumb trail were missing on exactly those
+  errors. A new innermost middleware, `ForgeOpsTracker::Middleware::RequestContext`, now copies the
+  user, breadcrumbs, transaction name, endpoint, and trace id onto the exception on its way out
+  (re-raising it unchanged), and the error subscriber falls back to that copy.
+- Distributed tracing across services, using the W3C Trace Context standard. A request that
+  arrives with a valid `traceparent` header continues that trace instead of starting its own, and
+  its root span points at the caller's span. Every outbound `Net::HTTP` call made during a request
+  now sends a `traceparent` header whose parent id is that call's own span, so the next service
+  (any service that understands the standard, not only ones running this gem) can continue the
+  trace. A `traceparent` you set yourself is never overwritten. New `config.propagate_traces`
+  (default true) and `config.trace_propagation_targets` (default nil, meaning every host; or an
+  array of host Strings, each matching that host and its subdomains, and/or Regexps) control where
+  the header goes. A trace id is created for every request even with `track_tracing` off, since it
+  is also what links an error to errors in other services.
+- An errored request's trace is always sent, however fast it was: an unhandled exception, or any
+  error reported through `Rails.error` while the request was running. Previously only requests
+  slower than `trace_capture_threshold_ms` were sent. Fast, successful requests are unchanged.
+
 ## 0.11.0
 
 - Database errors now say where to look. When an error comes from a database call

data/README.md

@@ -18,6 +18,10 @@ ForgeOpsTracker.configure do |config|
   config.dsn = ENV["FORGE_OPS_DSN"] # "https://<api_key>@getforgeops.net/api/v1/events"
   config.release = ENV["HEROKU_SLUG_COMMIT"] || `git rev-parse HEAD`.strip
   config.enabled_environments = %w[production staging] # default; reporting is a no-op elsewhere
+
+  # Trace propagation to the services you call (see "Tracing" below):
+  config.propagate_traces = true           # default; false stops sending the traceparent header
+  config.trace_propagation_targets = nil   # default: every host; or e.g. ["internal.example", /\.svc\z/]
 end
 ```
 
@@ -112,6 +116,29 @@ none of them (or with a signed-out `current_user`, as the `&.` above already han
 whatever was set. Shows up on an issue's own detail page, and as its own `affected_users_count`
 alongside the regular event count.
 
+## Where an error happened
+
+An error reported during a request carries three extra fields, filled in as soon as the
+controller starts handling the request (so an error reported from inside the action with
+`Rails.error.handle`/`.record` has them too):
+
+- `transaction_name`: the controller and action, `"CheckoutsController#create"`, the same name
+  performance monitoring and traces use.
+- `endpoint`: the HTTP method and the route as declared in your routes file, `"POST /checkout"` or
+  `"GET /orders/:id"`. Never the literal path, so an id or a token in the URL never ends up here.
+  Before Rails 7.1 (which added the route pattern to the request), this falls back to
+  `"POST CheckoutsController#create"`.
+- `trace_id`: the request's W3C trace id, which ForgeOps uses to link this error to errors that
+  other services reported for the same trace (see "Tracing" below).
+
+They're left out entirely outside a request (a background job, a console session), and they're
+never PII-scrubbed, like `environment` and `release`: they're structured fields, not free text.
+
+An unhandled exception keeps these, the affected user, and the breadcrumb trail, even though Rails
+reports it after this gem's middleware has already finished with the request: the innermost
+middleware, `ForgeOpsTracker::Middleware::RequestContext`, copies them onto the exception on its
+way out and re-raises it unchanged.
+
 ## Breadcrumbs
 
 A trail of what happened right before an error, on by default, no setup needed: every SQL query,
@@ -329,6 +356,61 @@ Requires a ForgeOps plan that includes performance monitoring, the same plan fea
 monitoring itself already requires; on a plan that doesn't, the periodic flushes are simply
 rejected server-side and dropped, exactly like any other delivery failure.
 
+## Tracing
+
+On by default: every request records a nested tree of spans (the controller action, SQL queries,
+outbound `Net::HTTP` calls, Redis calls, and anything you wrap in
+`ForgeOpsTracker.span("PaymentService#charge") { ... }`), kept in memory until the request ends.
+Only some requests' traces are actually sent:
+
+- a request at least `trace_capture_threshold_ms` slow (default 1000), and
+- **every request that errored, however fast**: an unhandled exception, or any error reported
+  through `Rails.error` (`handle`/`record`/`report`) while the request was running. The spans
+  leading up to an error are exactly what you want next to it.
+
+A fast, successful request never sends anything.
+
+```ruby
+ForgeOpsTracker.configure do |config|
+  config.track_tracing = false             # opt out of recording spans entirely
+  config.trace_capture_threshold_ms = 500  # default 1000
+end
+```
+
+### Following a request across services
+
+Traces use the [W3C Trace Context](https://www.w3.org/TR/trace-context/) standard, so a request
+can be followed from one service into the next, whichever language or tracing library the other
+service uses:
+
+- **Incoming**: a request that arrives with a valid `traceparent` header continues that trace
+  (same trace id), and its root span records the caller's span as its parent. A missing or
+  malformed header just starts a new trace.
+- **Outgoing**: every outbound `Net::HTTP` call made during a request (Faraday's `net_http`
+  adapter, HTTParty, and RestClient included) gets a `traceparent` header whose parent id is that
+  call's own span, so the next service's spans nest under it. A `traceparent` you set yourself is
+  never replaced.
+
+A trace id exists for every request even with `track_tracing` off, and the header is still sent,
+since the trace id is also what links an error here to an error in the service you called.
+Narrow or turn off where the header goes, for example if a third-party API rejects unknown
+headers:
+
+```ruby
+ForgeOpsTracker.configure do |config|
+  config.propagate_traces = false # never send traceparent (default true)
+
+  # Default nil: every host. A String matches that host and its subdomains ("internal.example"
+  # matches "orders.internal.example", not "notinternal.example"); a Regexp is matched against
+  # the host.
+  config.trace_propagation_targets = [ "internal.example", /\A10\.0\./ ]
+end
+```
+
+Requires a ForgeOps plan that includes distributed tracing; on a plan that doesn't, traces are
+simply rejected server-side and dropped, exactly like any other delivery failure. The `trace_id`
+on error events is sent either way.
+
 ## Custom metrics
 
 Unlike session/performance tracking above, there's no automatic instrumentation here at all: a

data/lib/forge_ops_tracker/configuration.rb

@@ -17,6 +17,7 @@ module ForgeOpsTracker
     attr_accessor :metric_flush_interval, :infrastructure_metric_flush_interval
     attr_accessor :gauge_poll_interval
     attr_accessor :track_tracing, :trace_capture_threshold_ms
+    attr_accessor :propagate_traces, :trace_propagation_targets
 
     def initialize
       @dsn = ENV["FORGE_OPS_DSN"]
@@ -124,6 +125,37 @@ module ForgeOpsTracker
       # PerformanceSample-derived threshold elsewhere in this gem: those describe a trend across
       # many requests, this describes a single one being outright slow.
       @trace_capture_threshold_ms = 1_000
+      # Adds a W3C `traceparent` header (https://www.w3.org/TR/trace-context/) to every outbound
+      # Net::HTTP request made during a request, so the service on the other end can continue this
+      # request's trace instead of starting its own (see Integrations::NetHTTP). On by default, the
+      # same as every other automatic behavior here, and independent of track_tracing: the header
+      # carries the trace id that links an error here to an error there, which is useful with or
+      # without spans. Never overwrites a traceparent the request already has.
+      @propagate_traces = true
+      # Which outbound hosts get that header. nil (the default) means every host. Otherwise an
+      # Array of Strings and/or Regexps: a String matches that exact host or any subdomain of it
+      # ("example.com" matches "example.com" and "api.example.com", but not "badexample.com"), a
+      # Regexp is matched against the host. Worth narrowing for an app that calls third-party APIs
+      # which reject unknown headers, or that shouldn't learn this app's trace ids at all.
+      @trace_propagation_targets = nil
+    end
+
+    # Whether an outbound request to host should carry a traceparent header; see
+    # propagate_traces/trace_propagation_targets above. Case-insensitive, since hostnames are.
+    def propagate_trace_to?(host)
+      return false unless propagate_traces
+      return true if trace_propagation_targets.nil?
+
+      host = host.to_s.downcase
+      Array(trace_propagation_targets).any? do |target|
+        case target
+        when Regexp
+          target.match?(host)
+        else
+          target = target.to_s.downcase.delete_prefix(".")
+          !target.empty? && (host == target || host.end_with?(".#{target}"))
+        end
+      end
     end
 
     def api_key

data/lib/forge_ops_tracker/error_subscriber.rb

@@ -28,11 +28,33 @@ module ForgeOpsTracker
       # hands this class the Rack env, so ForgeOpsTracker::Middleware::UserContext (Warden
       # auto-detection) and ForgeOpsTracker.set_user (the manual override/fallback) both
       # communicate with this exact same thread-local instead, the only channel available here.
-      # ForgeOpsTracker::Middleware::BreadcrumbContext's own per-request buffer works the same way.
+      # ForgeOpsTracker::Middleware::BreadcrumbContext's own per-request buffer works the same way,
+      # and so does Middleware::RequestContext's RequestState (trace id, transaction name,
+      # endpoint).
+      #
+      # Each live thread-local wins when it's set; the snapshot Middleware::RequestContext attached
+      # to the exception on its way out of the request is the fallback. That fallback is the whole
+      # story for an unhandled request error on Rails 8.1: ActionDispatch::Executor reports it from
+      # outside every middleware this gem installs, after all of their ensure blocks have already
+      # cleared every one of these thread-locals (see that middleware's own comment).
+      original_error = error
       error = unwrap_retry_control_flow(error)
-      user = Thread.current[:forge_ops_tracker_current_user]
-      breadcrumbs = Thread.current[:forge_ops_tracker_breadcrumbs]&.all || []
-      delivery_queue.push(event_builder.build(error, context: context, user: user, breadcrumbs: breadcrumbs))
+      snapshot = Middleware::RequestContext.snapshot_for(original_error) || Middleware::RequestContext.snapshot_for(error) || {}
+      state = Middleware::RequestContext.current
+
+      # A handled error reported during a request (Rails.error.handle/record/report) is what marks
+      # it errored here; an unhandled one was already marked by RequestContext's own rescue. Either
+      # way Middleware::SpanTracing then sends this request's trace however fast it was.
+      state&.mark_errored!
+
+      user = Thread.current[:forge_ops_tracker_current_user] || snapshot[:user]
+      breadcrumbs = Thread.current[:forge_ops_tracker_breadcrumbs]&.all || snapshot[:breadcrumbs] || []
+      delivery_queue.push(event_builder.build(
+        error, context: context, user: user, breadcrumbs: breadcrumbs,
+        transaction_name: state&.transaction_name || snapshot[:transaction_name],
+        endpoint: state&.endpoint || snapshot[:endpoint],
+        trace_id: state&.trace_id || snapshot[:trace_id]
+      ))
       nil
     rescue StandardError => e
       configuration.logger&.debug { "[ForgeOpsTracker] report failed: #{e.class}: #{e.message}" }

data/lib/forge_ops_tracker/event_builder.rb

@@ -27,7 +27,15 @@ module ForgeOpsTracker
       @configuration = configuration
     end
 
-    def build(error, context: {}, user: nil, breadcrumbs: [])
+    # transaction_name/endpoint/trace_id describe the request the error happened in (see
+    # RequestState): transaction_name is the same "CheckoutsController#create" name the request's
+    # root span and performance samples use, endpoint the human-readable route ("POST /checkout"),
+    # trace_id the request's 32-character lowercase hex W3C trace id, which is also what links this
+    # error to errors other services reported for the same trace. All three are nil outside a
+    # request (a background job, a console session) and each key is left out of the payload
+    # entirely when nil, the same "omit rather than send an empty value" treatment user and
+    # breadcrumbs already get.
+    def build(error, context: {}, user: nil, breadcrumbs: [], transaction_name: nil, endpoint: nil, trace_id: nil)
       payload = {
         exception_class: error.class.name,
         message: error.message.to_s,
@@ -42,6 +50,9 @@ module ForgeOpsTracker
       }
       payload[:user] = user if user && !user.empty?
       payload[:breadcrumbs] = breadcrumbs if breadcrumbs && !breadcrumbs.empty?
+      payload[:transaction_name] = transaction_name if transaction_name
+      payload[:endpoint] = endpoint if endpoint
+      payload[:trace_id] = trace_id if trace_id
       attach_sql(payload, error)
       scrub(payload)
     end
@@ -49,9 +60,11 @@ module ForgeOpsTracker
     private
       attr_reader :configuration
 
-      # exception_class/occurred_at/environment/release/server_name/sdk_name/user are left alone;
-      # structured fields this gem or the host app sets deliberately, not free text an exception
-      # or its context could accidentally spill sensitive data into. user specifically is a
+      # exception_class/occurred_at/environment/release/server_name/sdk_name/user/transaction_name/
+      # endpoint/trace_id are left alone; structured fields this gem or the host app sets
+      # deliberately, not free text an exception or its context could accidentally spill sensitive
+      # data into. endpoint in particular is a declared route pattern ("GET /orders/:id"), never
+      # the literal request path, so it has no id or token in it to redact. user specifically is a
       # deliberate exemption, not an oversight: the server's own PiiScrubber-equivalent would
       # otherwise redact the exact email address this field exists to carry. breadcrumbs is *not*
       # exempt, unlike user: console-style/query/request trail entries are exactly the kind of free

data/lib/forge_ops_tracker/integrations/net_http.rb

@@ -23,14 +23,25 @@ module ForgeOpsTracker
       # confirmed directly that Net::HTTP connects lazily, inside #request itself, the first time
       # it's called on a not-yet-started instance, so that path's own connection failures are
       # caught exactly like any other #request failure.
+      #
+      # Also propagates the current request's trace to the service being called, as a W3C
+      # `traceparent` header (see TraceParent, and Configuration#propagate_traces/
+      # trace_propagation_targets for turning it off or narrowing it to specific hosts). The
+      # header's parent id is this outbound call's own span id, generated before the call is made
+      # (the span itself can only be recorded afterward, once its duration is known) and then
+      # recorded with that exact id, so the downstream service's root span points at a span that
+      # really exists in this trace. Only inside a request (a RequestState to take the trace id
+      # from); a background job or a console session has no trace to continue.
       module Timing
         # The guard (return super unless ...) skips timing, but the ensure block below still runs
         # on every exit path regardless, same as any ensure; start staying nil is what actually
         # skips recording there, not a second guard duplicating this one.
         def request(req, body = nil, &block)
           configuration = ForgeOpsTracker.configuration
-          return super unless (configuration.track_performance || configuration.track_breadcrumbs || configuration.track_failures || configuration.track_tracing) && configuration.enabled?
+          return super unless (configuration.track_performance || configuration.track_breadcrumbs || configuration.track_failures || configuration.track_tracing || configuration.propagate_traces) && configuration.enabled?
 
+          span_id = ForgeOpsTracker::TraceParent.generate_span_id
+          propagate_trace(req, span_id, configuration)
           start = ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
           response = super
         rescue StandardError => e
@@ -102,12 +113,26 @@ module ForgeOpsTracker
               # it actually took.
               start_time = ::Time.now.utc - (duration_ms / 1000.0)
               buffer.record(
-                span_id: SecureRandom.hex(8), name: "#{req.method} #{address}", kind: "http",
+                span_id: span_id, name: "#{req.method} #{address}", kind: "http",
                 started_at: start_time, duration_ms: duration_ms, data: { status: status }.compact
               )
             end
           end
         end
+
+        private
+          # Leaves a traceparent the caller already set alone: whoever set it explicitly knows
+          # better than this gem which trace the call belongs to. Never raises, since a failure to
+          # add a header must never stop the host app's own request from going out.
+          def propagate_trace(req, span_id, configuration)
+            state = ForgeOpsTracker::Middleware::RequestContext.current
+            return unless state && configuration.propagate_trace_to?(address)
+            return if req[ForgeOpsTracker::TraceParent::HEADER]
+
+            req[ForgeOpsTracker::TraceParent::HEADER] = ForgeOpsTracker::TraceParent.build(trace_id: state.trace_id, span_id: span_id)
+          rescue StandardError
+            nil
+          end
       end
     end
   end

data/lib/forge_ops_tracker/middleware/request_context.rb

@@ -0,0 +1,90 @@
+module ForgeOpsTracker
+  module Middleware
+    # Publishes this request's RequestState (trace id, remote parent span id, transaction name,
+    # endpoint, errored flag; see that class) as a thread-local for the duration of the request,
+    # cleared in an ensure for the same "Puma reuses threads" reason every other middleware in this
+    # gem already gives. Not gated on track_tracing: the trace id links an error to errors in other
+    # services even when no spans are ever recorded.
+    #
+    # Its second job is the reason it has to be the *innermost* of this gem's middlewares (added
+    # last in Railtie, so app.middleware.use puts it closest to the app). Rails 8.1 reports an
+    # unhandled request exception from ActionDispatch::Executor, which sits outside all of this
+    # gem's middlewares; by the time ErrorSubscriber#report runs for that exception, every one of
+    # their ensure blocks has already cleared its thread-local, so the user, the breadcrumb trail,
+    # and the trace id/endpoint would all be gone for exactly the errors that matter most. Rescuing
+    # here, innermost, is the last moment all of those are still set: this copies them onto the
+    # exception object itself (see SNAPSHOT_IVAR) before re-raising it unchanged, and
+    # ErrorSubscriber#report falls back to that copy whenever the live thread-locals are gone.
+    #
+    # Rescues Exception, not StandardError, and always re-raises: this only ever observes an
+    # exception on its way out, it never handles one, so there's no reason to let anything slip
+    # past without its context attached.
+    class RequestContext
+      THREAD_KEY = :forge_ops_tracker_request_context
+
+      # An instance variable on the exception object, not a wrapper exception or a side table: the
+      # exception is the one thing guaranteed to travel from here to wherever Rails ends up
+      # reporting it, and a plain ivar leaves its class, message, and backtrace exactly as the host
+      # app raised them.
+      SNAPSHOT_IVAR = :@__forge_ops_tracker_request_snapshot
+
+      def initialize(app, configuration: ForgeOpsTracker.configuration)
+        @app = app
+        @configuration = configuration
+      end
+
+      def call(env)
+        return app.call(env) unless configuration.enabled?
+
+        state = RequestState.for(env)
+        Thread.current[THREAD_KEY] = state
+        app.call(env)
+      rescue Exception => e
+        if state
+          state.mark_errored!
+          self.class.attach_snapshot(e, state)
+        end
+        raise
+      ensure
+        Thread.current[THREAD_KEY] = nil
+      end
+
+      class << self
+        def current
+          Thread.current[THREAD_KEY]
+        end
+
+        # Never raises, whatever it's handed: a frozen exception (some libraries freeze theirs, and
+        # a frozen object rejects instance_variable_set with a FrozenError) just goes without a
+        # snapshot, and so does anything else unexpected, since failing here would replace the host
+        # app's own exception with one of this gem's. The first snapshot wins if the same exception
+        # object somehow passes through twice: the innermost one saw the request closest to where
+        # it actually failed.
+        def attach_snapshot(error, state)
+          return if error.frozen? || error.instance_variable_defined?(SNAPSHOT_IVAR)
+
+          error.instance_variable_set(SNAPSHOT_IVAR, {
+            user: Thread.current[:forge_ops_tracker_current_user],
+            breadcrumbs: Thread.current[:forge_ops_tracker_breadcrumbs]&.all,
+            transaction_name: state.transaction_name,
+            endpoint: state.endpoint,
+            trace_id: state.trace_id
+          }.freeze)
+        rescue StandardError
+          nil
+        end
+
+        def snapshot_for(error)
+          return nil unless error.instance_variable_defined?(SNAPSHOT_IVAR)
+
+          error.instance_variable_get(SNAPSHOT_IVAR)
+        rescue StandardError
+          nil
+        end
+      end
+
+      private
+        attr_reader :app, :configuration
+    end
+  end
+end

data/lib/forge_ops_tracker/middleware/span_tracing.rb

@@ -3,15 +3,29 @@ module ForgeOpsTracker
     # Gives every request its own fresh SpanBuffer (see that class for the stack/nesting
     # mechanics), the same "Thread.current, cleared in an ensure" shape BreadcrumbContext already
     # uses and for the identical reason: Puma reuses threads across requests, so a buffer left set
-    # would leak one request's spans into a later, unrelated one handled on the same thread.
+    # would leak one request's spans into a later, unrelated one handled on the same thread. The
+    # buffer's trace_id and remote parent span id come from the request's RequestState, never
+    # generated here, so a trace this sends and an error event from the same request always agree
+    # on which trace they belong to (see RequestState for why that object is shared through the
+    # Rack env rather than the thread-local RequestContext publishes: this middleware sits outside
+    # that one, and runs first).
     #
     # The actual send-or-don't decision lives here, not in SpanBuffer or SpanQueue: only once
     # app.call returns does this middleware know the root span's own final duration (recorded by
     # Railtie's process_action.action_controller subscription, which fires from *inside* that
     # call, after every nested query/HTTP/Redis call has already run), so this is the first point
     # that can compare it against configuration.trace_capture_threshold_ms and decide the trace was
-    # never worth sending in the first place - the entire reason a normal, fast request never costs
+    # never worth sending in the first place: the entire reason a normal, fast request never costs
     # a single byte over the wire, unlike PerformanceSample's own always-on aggregate reporting.
+    #
+    # An errored request's trace is always sent too, however fast it was, since the waterfall of
+    # what led up to an error is exactly what an issue page wants to show next to it. "Errored"
+    # means the RequestState was marked by either path: an unhandled exception passing through
+    # RequestContext on its way out, or ErrorSubscriber#report receiving a handled error during the
+    # request. Both have already happened by the time this ensure runs; the unhandled one is only
+    # *reported* later (by ActionDispatch::Executor, outside every middleware here), but it's
+    # marked on the way through, which is what counts. Requires the root span to have been
+    # recorded, see SpanBuffer#root_recorded? for why.
     class SpanTracing
       def initialize(app, configuration: ForgeOpsTracker.configuration)
         @app = app
@@ -21,11 +35,12 @@ module ForgeOpsTracker
       def call(env)
         return app.call(env) unless configuration.track_tracing && configuration.enabled?
 
-        buffer = SpanBuffer.new
+        state = RequestState.for(env)
+        buffer = SpanBuffer.new(trace_id: state.trace_id, parent_span_id: state.parent_span_id)
         Thread.current[:forge_ops_tracker_spans] = buffer
         app.call(env)
       ensure
-        if buffer&.slow?(configuration.trace_capture_threshold_ms)
+        if buffer && send_trace?(buffer, state)
           Railtie.span_queue.push(trace_id: buffer.trace_id, spans: buffer.spans)
         end
         Thread.current[:forge_ops_tracker_spans] = nil
@@ -33,6 +48,12 @@ module ForgeOpsTracker
 
       private
         attr_reader :app, :configuration
+
+        def send_trace?(buffer, state)
+          return true if buffer.slow?(configuration.trace_capture_threshold_ms)
+
+          state.errored? && buffer.root_recorded?
+        end
     end
   end
 end

data/lib/forge_ops_tracker/performance_instrumentation.rb

@@ -16,6 +16,37 @@ module ForgeOpsTracker
       "#{payload[:controller]}##{payload[:action]}"
     end
 
+    # A human-readable route for an error event's `endpoint` field, "<HTTP method> <route pattern>"
+    # ("POST /checkout", "GET /orders/:id"): the route as it was declared in config/routes.rb, never
+    # the literal request path, so it stays low-cardinality and never carries an id or a token, the
+    # same reasoning every transaction_name in this system already follows. payload[:request] is
+    # only present in Rails 7.1+, the same release that added
+    # ActionDispatch::Request#route_uri_pattern; on anything older (or a request with no matched
+    # route pattern at all) this falls back to the controller/action name instead
+    # ("GET CheckoutsController#create"), still meaningful, just less familiar-looking than a URL.
+    def controller_endpoint(payload)
+      target = route_pattern(payload[:request]) || controller_transaction_name(payload)
+      method = payload[:method]
+      method.nil? || method.to_s.empty? ? target : "#{method} #{target}"
+    end
+
+    # Rails appends an optional "(.:format)" segment to nearly every route pattern it generates;
+    # it's noise in a label a person reads, so it's dropped. Defensive about everything else, since
+    # this runs inside a notification subscriber on every single request: a request object without
+    # the method, a nil/empty pattern, or the method itself raising all just mean "no pattern," and
+    # the caller falls back.
+    def route_pattern(request)
+      return nil unless request.respond_to?(:route_uri_pattern)
+
+      pattern = request.route_uri_pattern
+      return nil if pattern.nil? || pattern.to_s.empty?
+
+      pattern = pattern.to_s.delete_suffix("(.:format)")
+      pattern.empty? ? "/" : pattern
+    rescue StandardError
+      nil
+    end
+
     # nil means "don't record this one": Rails' own sql.active_record fires for two kinds of
     # noise that would drown out real query timing otherwise. "SCHEMA" is Rails' own internal
     # introspection (column lookups, etc.), never something a customer's own code triggered;

data/lib/forge_ops_tracker/railtie.rb

@@ -73,6 +73,31 @@ module ForgeOpsTracker
       app.middleware.use ForgeOpsTracker::Middleware::SpanTracing, configuration: ForgeOpsTracker.configuration
     end
 
+    # Must stay the last app.middleware.use in this file, so RequestContext is the innermost of
+    # this gem's middlewares: its rescue is what copies the user, the breadcrumb trail, and the
+    # trace id/endpoint onto an unhandled exception before any of the middlewares above clear their
+    # thread-locals in their own ensure blocks, and ActionDispatch::Executor (which reports that
+    # exception in Rails 8.1) sits outside all of them. See that middleware's own comment.
+    # Initializers within one Railtie run in the order they're defined, and app.middleware.use
+    # appends, so being defined after track_tracing is what makes it innermost.
+    #
+    # start_processing.action_controller fires as the controller begins handling the request, before
+    # the action itself runs (unlike process_action.action_controller below, which only fires once
+    # it has finished), so a handled error reported from inside the action (Rails.error.handle/
+    # record) already knows which transaction and endpoint it happened in. Its payload carries
+    # :controller/:action/:method, plus :request from Rails 7.1 on (see
+    # PerformanceInstrumentation.controller_endpoint for what each is used for).
+    initializer "forge_ops_tracker.request_context" do |app|
+      configuration = ForgeOpsTracker.configuration
+      app.middleware.use ForgeOpsTracker::Middleware::RequestContext, configuration: configuration
+
+      ActiveSupport::Notifications.subscribe("start_processing.action_controller") do |*args|
+        next unless configuration.enabled?
+
+        ForgeOpsTracker::Middleware::RequestContext.current&.controller_started(args.last)
+      end
+    end
+
     # Rails already fires "process_action.action_controller" after every request with the
     # controller/action and how long it took (event.duration, in ms) built in, so unlike session
     # tracking above (which needed a Rack middleware because there's no single existing hook for
@@ -130,8 +155,8 @@ module ForgeOpsTracker
         # this exact call site already established for breadcrumbs): this is the one call in the
         # whole gem that ever passes root: true, since process_action.action_controller is the only
         # instrumentation point that wraps a genuinely whole request. See SpanBuffer#record's own
-        # comment for why root: true has to force parent_span_id nil explicitly here rather than
-        # reading it off the (already-popped-by-now) stack.
+        # comment for why root: true takes its parent_span_id from the incoming traceparent header
+        # (nil when there wasn't one) rather than reading it off the (already-popped-by-now) stack.
         if configuration.track_tracing && (buffer = Thread.current[:forge_ops_tracker_spans])
           buffer.pop
           buffer.record(

data/lib/forge_ops_tracker/request_state.rb

@@ -0,0 +1,69 @@
+module ForgeOpsTracker
+  # Everything this gem knows about the current request that isn't already owned by one of the
+  # older, single-purpose thread-locals (the user, the breadcrumb trail, the span buffer): which
+  # trace it belongs to, which remote span called it (if any), what it's called, and whether it
+  # errored. One instance per request, shared by every piece of this gem that needs any of those:
+  #
+  # - Middleware::RequestContext publishes it as a thread-local
+  #   (Thread.current[:forge_ops_tracker_request_context]) for the duration of the request, so
+  #   ErrorSubscriber#report, the start_processing.action_controller subscription, and the
+  #   Net::HTTP integration can reach it (none of them are ever handed the Rack env).
+  # - Middleware::SpanTracing reads the same instance's trace_id/parent_span_id for its SpanBuffer,
+  #   and its errored? flag once the request is over.
+  #
+  # The trace_id exists for every request, not only when span tracing is on: it's also what an
+  # error event carries so ForgeOps can link it to errors reported by other services taking part in
+  # the same trace, which works on any plan and with track_tracing off entirely.
+  #
+  # Memoized in the Rack env (see .for) rather than created by one specific middleware and handed to
+  # the other: SpanTracing sits *outside* RequestContext in the stack (RequestContext must be the
+  # innermost of this gem's middlewares; see its own comment for why), so SpanTracing runs first and
+  # needs the trace_id before RequestContext has run at all, then keeps needing this same object
+  # after RequestContext's own ensure has already cleared the thread-local. Whichever of the two
+  # reaches a given request first creates it; the other gets the identical instance back.
+  class RequestState
+    ENV_KEY = "forge_ops_tracker.request_state".freeze
+
+    attr_reader :trace_id, :parent_span_id
+    attr_accessor :transaction_name, :endpoint
+
+    def self.for(env)
+      env[ENV_KEY] ||= from_env(env)
+    end
+
+    # Continues the caller's trace when the request arrived with a usable W3C traceparent header,
+    # remembering the caller's span as this request's remote parent; starts a fresh trace otherwise.
+    def self.from_env(env)
+      incoming = TraceParent.parse(env[TraceParent::RACK_ENV_KEY])
+      if incoming
+        new(trace_id: incoming[:trace_id], parent_span_id: incoming[:parent_span_id])
+      else
+        new
+      end
+    end
+
+    def initialize(trace_id: TraceParent.generate_trace_id, parent_span_id: nil)
+      @trace_id = trace_id
+      @parent_span_id = parent_span_id
+      @transaction_name = nil
+      @endpoint = nil
+      @errored = false
+    end
+
+    # Called from the start_processing.action_controller subscription (see Railtie), the earliest
+    # point Rails names the controller/action handling this request: early enough that an error
+    # reported partway through the action itself (Rails.error.handle/record) already carries both.
+    def controller_started(payload)
+      self.transaction_name = PerformanceInstrumentation.controller_transaction_name(payload)
+      self.endpoint = PerformanceInstrumentation.controller_endpoint(payload)
+    end
+
+    def mark_errored!
+      @errored = true
+    end
+
+    def errored?
+      @errored
+    end
+  end
+end

data/lib/forge_ops_tracker/span_buffer.rb

@@ -3,9 +3,13 @@ require "securerandom"
 module ForgeOpsTracker
   # Accumulates one request's own nested call tree in-process; the tracing analog to
   # BreadcrumbBuffer's per-thread trail. ForgeOpsTracker::Middleware::SpanTracing resets this to a
-  # fresh buffer at the start of every request, with a freshly generated trace_id and root
-  # span_id, and reads it back once the request is over to decide whether this trace is worth
-  # sending at all (see that middleware's own comment for exactly where and why).
+  # fresh buffer at the start of every request, with a freshly generated root span_id and the
+  # request's own trace_id (and remote parent span id, when the request continued a trace another
+  # service started; see RequestState), and reads it back once the request is over to decide
+  # whether this trace is worth sending at all (see that middleware's own comment for exactly where
+  # and why). trace_id/parent_span_id are passed in rather than generated here: the trace_id
+  # belongs to the request, not to span tracing, since an error event carries it too, even with
+  # tracing off, and the two have to agree for ForgeOps to link them.
   #
   # The stack exists so nesting can be reconstructed correctly even though most of what's recorded
   # here (a query, an outbound HTTP call, a Redis call) only has a start+duration known *after* the
@@ -19,10 +23,11 @@ module ForgeOpsTracker
   # already run) naturally nests under it without the root having to exist as a real recorded span
   # first.
   class SpanBuffer
-    attr_reader :trace_id, :root_span_id, :spans
+    attr_reader :trace_id, :root_span_id, :parent_span_id, :spans
 
-    def initialize
-      @trace_id = SecureRandom.hex(16)
+    def initialize(trace_id: SecureRandom.hex(16), parent_span_id: nil)
+      @trace_id = trace_id
+      @parent_span_id = parent_span_id
       @root_span_id = SecureRandom.hex(8)
       @spans = []
       @stack = [ @root_span_id ]
@@ -42,12 +47,14 @@ module ForgeOpsTracker
     end
 
     # root: true only for the one call that records the request's own root span (see Railtie's
-    # process_action.action_controller subscription): forces parent_span_id nil explicitly, since
-    # by the time that fires, the root's own span_id has already been popped off the stack (there's
-    # nothing left on it to read back as "the parent" at that point - correctly, since the root has
-    # no parent of its own).
+    # process_action.action_controller subscription): takes its parent from the incoming
+    # traceparent header's span id instead of the stack, since by the time that fires, the root's
+    # own span_id has already been popped off it (there's nothing left on it to read back as "the
+    # parent" at that point, correctly: nothing inside this process is the root's parent). nil for
+    # a request that started its own trace; the calling service's span otherwise, which ForgeOps
+    # treats as a remote parent because it's never part of this same batch.
     def record(span_id:, name:, kind:, started_at:, duration_ms:, data: {}, root: false)
-      parent_span_id = root ? nil : current_parent_id
+      parent_span_id = root ? @parent_span_id : current_parent_id
       spans << {
         span_id: span_id, parent_span_id: parent_span_id, name: name, kind: kind,
         started_at: started_at.utc.iso8601(3), duration_ms: duration_ms, data: data
@@ -61,5 +68,13 @@ module ForgeOpsTracker
     def slow?(threshold_ms)
       @root_duration_ms && @root_duration_ms >= threshold_ms
     end
+
+    # Whether the root span was ever recorded at all. Middleware::SpanTracing only sends an errored
+    # request's trace when this is true: without the root, every other span in the buffer points at
+    # a parent that was never recorded, which ForgeOps would misread as a remote parent in some
+    # other service rather than a missing root in this one.
+    def root_recorded?
+      !@root_duration_ms.nil?
+    end
   end
 end

data/lib/forge_ops_tracker/trace_parent.rb

@@ -0,0 +1,62 @@
+require "securerandom"
+
+module ForgeOpsTracker
+  # Reads and writes the W3C Trace Context `traceparent` header
+  # (https://www.w3.org/TR/trace-context/), the vendor-neutral format for carrying one trace across
+  # service boundaries: `00-<32 hex trace-id>-<16 hex parent-id>-<2 hex flags>`. Used in both
+  # directions: Middleware::RequestContext parses an incoming one so this request continues the
+  # caller's trace instead of starting its own, and Integrations::NetHTTP builds an outgoing one so
+  # the next service along continues this request's.
+  #
+  # Deliberately strict on the way in, the same posture the spec itself asks receivers to take: a
+  # malformed value, uppercase hex, the reserved version "ff", or an all-zero trace/parent id are
+  # all treated as "no usable header at all" (parse returns nil and the request starts a fresh
+  # trace), never half-trusted. A version this gem doesn't know yet is still accepted as long as
+  # its first four fields have version 00's shape, which is exactly what the spec says a version-00
+  # parser should do with a future version; version 00 itself must have exactly four fields.
+  module TraceParent
+    HEADER = "traceparent".freeze
+    RACK_ENV_KEY = "HTTP_TRACEPARENT".freeze
+
+    PATTERN = /\A(?<version>[0-9a-f]{2})-(?<trace_id>[0-9a-f]{32})-(?<parent_id>[0-9a-f]{16})-(?<flags>[0-9a-f]{2})(?<rest>-.*)?\z/
+    INVALID_TRACE_ID = ("0" * 32).freeze
+    INVALID_PARENT_ID = ("0" * 16).freeze
+
+    # Always "01" (sampled) on the way out: this gem decides whether to actually send a trace only
+    # once the request is over (see Middleware::SpanTracing), long after this header has already
+    # gone out on an outbound call, so there's no honest earlier answer to give than "this may be
+    # recorded." A downstream service is free to make its own decision either way.
+    SAMPLED_FLAGS = "01".freeze
+
+    module_function
+
+    # Returns { trace_id:, parent_span_id: } for a usable header, nil for anything else (absent,
+    # blank, malformed, or one of the explicitly invalid values above).
+    def parse(value)
+      return nil unless value.is_a?(String)
+
+      match = PATTERN.match(value.strip)
+      return nil unless match
+      return nil if match[:version] == "ff"
+      return nil if match[:version] == "00" && match[:rest]
+      return nil if match[:trace_id] == INVALID_TRACE_ID || match[:parent_id] == INVALID_PARENT_ID
+
+      { trace_id: match[:trace_id], parent_span_id: match[:parent_id] }
+    end
+
+    def build(trace_id:, span_id:)
+      "00-#{trace_id}-#{span_id}-#{SAMPLED_FLAGS}"
+    end
+
+    # 32 lowercase hex characters, the W3C trace-id format; SecureRandom.hex already produces
+    # lowercase, which is the only case the spec allows.
+    def generate_trace_id
+      SecureRandom.hex(16)
+    end
+
+    # 16 lowercase hex characters, the W3C parent-id (span id) format.
+    def generate_span_id
+      SecureRandom.hex(8)
+    end
+  end
+end

data/lib/forge_ops_tracker/version.rb

@@ -1,3 +1,3 @@
 module ForgeOpsTracker
-  VERSION = "0.11.0"
+  VERSION = "0.12.0"
 end

data/lib/forge_ops_tracker.rb

@@ -18,10 +18,13 @@ require "forge_ops_tracker/periodic_poller"
 require "forge_ops_tracker/breadcrumb_buffer"
 require "forge_ops_tracker/span_buffer"
 require "forge_ops_tracker/span_queue"
+require "forge_ops_tracker/trace_parent"
+require "forge_ops_tracker/request_state"
 require "forge_ops_tracker/middleware/session_tracking"
 require "forge_ops_tracker/middleware/user_context"
 require "forge_ops_tracker/middleware/breadcrumb_context"
 require "forge_ops_tracker/middleware/span_tracing"
+require "forge_ops_tracker/middleware/request_context"
 
 module ForgeOpsTracker
   class << self

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: forge_ops_tracker
 version: !ruby/object:Gem::Version
-  version: 0.11.0
+  version: 0.12.0
 platform: ruby
 authors:
 - ForgeOps
@@ -106,6 +106,7 @@ files:
 - lib/forge_ops_tracker/integrations/solid_queue.rb
 - lib/forge_ops_tracker/metric_buffer.rb
 - lib/forge_ops_tracker/middleware/breadcrumb_context.rb
+- lib/forge_ops_tracker/middleware/request_context.rb
 - lib/forge_ops_tracker/middleware/session_tracking.rb
 - lib/forge_ops_tracker/middleware/span_tracing.rb
 - lib/forge_ops_tracker/middleware/user_context.rb
@@ -114,10 +115,12 @@ files:
 - lib/forge_ops_tracker/periodic_poller.rb
 - lib/forge_ops_tracker/pii_scrubber.rb
 - lib/forge_ops_tracker/railtie.rb
+- lib/forge_ops_tracker/request_state.rb
 - lib/forge_ops_tracker/session_flusher.rb
 - lib/forge_ops_tracker/span_buffer.rb
 - lib/forge_ops_tracker/span_queue.rb
 - lib/forge_ops_tracker/sql_statement.rb
+- lib/forge_ops_tracker/trace_parent.rb
 - lib/forge_ops_tracker/version.rb
 homepage: https://getforgeops.net
 licenses: