forge_ops_tracker 0.11.0 → 0.13.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: af67d3b6d07322ccfd80d56256e5e600c264f60394058acab51b7bfc660435fa
-  data.tar.gz: 2716944aed397bb03c754900b2e34f3abc49247b43b6a97b411cec6b6a3ccbc2
+  metadata.gz: ccbeed25a3172994d20455c19fa79a8336f3c73324ed667fd2f31d2cc05a19f5
+  data.tar.gz: 3cb7e1562daf2ccb8f09226bc915b7f00f6e8486ebbf361f8adf0a16c9851afe
 SHA512:
-  metadata.gz: 2952a5218507cfceb871821b1f79add47d255b45577ee4bc67a782b574e3d41135d1f971d966846666c3bbcc1de6e86e3c3c7671e8a5ec04b31d9dabf4af0cc4
-  data.tar.gz: 32efc755d0d05fca0dd4f4354d00808caf135da2c0a22ad2b9d417017a3e73406b6d1b879da3e5862724340d52222fec9d1fbbfebc45efee58101e824b105373
+  metadata.gz: 6287470f8bc768e95d1d1afe55f2c7a0e640a4d66ded8a0e3c518e0248031a7f5f76621adf0436d2de3bc7da029733b9debab07e31d2f1b37c7867cfd3aee7e5
+  data.tar.gz: b7c996fbf89c2c0a90f6c39c848542cde734242a3fa4efaf450a36b1e7362815322417ceafc89d05a71a453a9965d7b4ab0c4deb2eedad0fdafd6b21808ad790

data/CHANGELOG.md

@@ -1,5 +1,53 @@
 # Changelog
 
+## 0.13.0 (2026-09-25)
+
+- New `ForgeOpsTracker.record_change(kind:, title:, details: {}, environment: nil, service: nil,
+  actor: nil, url: nil, id: nil, occurred_at: nil)` records something that changed in your system
+  (a feature flag, a config value, a hand-run migration) so ForgeOps can show it next to the errors
+  that followed. `kind` is one of `feature_flag`, `config`, `migration`, `dependency`,
+  `infrastructure`, or `other`; anything else is sent as `other`. Sent from a background thread,
+  never raises, and a no-op wherever reporting is off.
+- Changes between deploys are now detected automatically. Once per process, after Rails finishes
+  booting (or on `ForgeOpsTracker.configure` outside Rails), a background thread sends a snapshot of
+  the Ruby version, the gem versions in `Gemfile.lock`, and the current Active Record migration
+  version, and ForgeOps records whatever changed since the previous boot. New
+  `config.detect_changes` (default true) turns this off.
+- New `config.track_env_var_names` (default false) adds environment variable names, never values,
+  to that snapshot. Host-specific names (`HOSTNAME`, `PATH`, `PORT`, `LC_*`, Kubernetes service
+  variables, and others) and the gem's own `FORGE_OPS_*` settings are always left out.
+
+## 0.12.0
+
+- Every error event reported during a request now says where it happened: `transaction_name` (the
+  same `CheckoutsController#create` name performance samples and traces already use), `endpoint`
+  (the HTTP method plus the route as declared in your routes file, e.g. `POST /checkout` or
+  `GET /orders/:id`, never the literal path, so no ids or tokens; falls back to
+  `POST CheckoutsController#create` before Rails 7.1), and `trace_id` (the request's W3C trace id).
+  All three are set as soon as the controller starts handling the request, so an error reported
+  from inside the action with `Rails.error.handle`/`record` already has them. Left out entirely
+  outside a request (a background job, a console session). Structured fields, so they're never
+  PII-scrubbed.
+- Unhandled request errors keep their context. Rails 8.1 reports an unhandled request exception
+  from its executor, outside this gem's middleware, after the gem had already cleared its
+  per-request state, so the affected user and the breadcrumb trail were missing on exactly those
+  errors. A new innermost middleware, `ForgeOpsTracker::Middleware::RequestContext`, now copies the
+  user, breadcrumbs, transaction name, endpoint, and trace id onto the exception on its way out
+  (re-raising it unchanged), and the error subscriber falls back to that copy.
+- Distributed tracing across services, using the W3C Trace Context standard. A request that
+  arrives with a valid `traceparent` header continues that trace instead of starting its own, and
+  its root span points at the caller's span. Every outbound `Net::HTTP` call made during a request
+  now sends a `traceparent` header whose parent id is that call's own span, so the next service
+  (any service that understands the standard, not only ones running this gem) can continue the
+  trace. A `traceparent` you set yourself is never overwritten. New `config.propagate_traces`
+  (default true) and `config.trace_propagation_targets` (default nil, meaning every host; or an
+  array of host Strings, each matching that host and its subdomains, and/or Regexps) control where
+  the header goes. A trace id is created for every request even with `track_tracing` off, since it
+  is also what links an error to errors in other services.
+- An errored request's trace is always sent, however fast it was: an unhandled exception, or any
+  error reported through `Rails.error` while the request was running. Previously only requests
+  slower than `trace_capture_threshold_ms` were sent. Fast, successful requests are unchanged.
+
 ## 0.11.0
 
 - Database errors now say where to look. When an error comes from a database call

data/README.md

@@ -18,6 +18,10 @@ ForgeOpsTracker.configure do |config|
   config.dsn = ENV["FORGE_OPS_DSN"] # "https://<api_key>@getforgeops.net/api/v1/events"
   config.release = ENV["HEROKU_SLUG_COMMIT"] || `git rev-parse HEAD`.strip
   config.enabled_environments = %w[production staging] # default; reporting is a no-op elsewhere
+
+  # Trace propagation to the services you call (see "Tracing" below):
+  config.propagate_traces = true           # default; false stops sending the traceparent header
+  config.trace_propagation_targets = nil   # default: every host; or e.g. ["internal.example", /\.svc\z/]
 end
 ```
 
@@ -112,6 +116,29 @@ none of them (or with a signed-out `current_user`, as the `&.` above already han
 whatever was set. Shows up on an issue's own detail page, and as its own `affected_users_count`
 alongside the regular event count.
 
+## Where an error happened
+
+An error reported during a request carries three extra fields, filled in as soon as the
+controller starts handling the request (so an error reported from inside the action with
+`Rails.error.handle`/`.record` has them too):
+
+- `transaction_name`: the controller and action, `"CheckoutsController#create"`, the same name
+  performance monitoring and traces use.
+- `endpoint`: the HTTP method and the route as declared in your routes file, `"POST /checkout"` or
+  `"GET /orders/:id"`. Never the literal path, so an id or a token in the URL never ends up here.
+  Before Rails 7.1 (which added the route pattern to the request), this falls back to
+  `"POST CheckoutsController#create"`.
+- `trace_id`: the request's W3C trace id, which ForgeOps uses to link this error to errors that
+  other services reported for the same trace (see "Tracing" below).
+
+They're left out entirely outside a request (a background job, a console session), and they're
+never PII-scrubbed, like `environment` and `release`: they're structured fields, not free text.
+
+An unhandled exception keeps these, the affected user, and the breadcrumb trail, even though Rails
+reports it after this gem's middleware has already finished with the request: the innermost
+middleware, `ForgeOpsTracker::Middleware::RequestContext`, copies them onto the exception on its
+way out and re-raises it unchanged.
+
 ## Breadcrumbs
 
 A trail of what happened right before an error, on by default, no setup needed: every SQL query,
@@ -329,6 +356,107 @@ Requires a ForgeOps plan that includes performance monitoring, the same plan fea
 monitoring itself already requires; on a plan that doesn't, the periodic flushes are simply
 rejected server-side and dropped, exactly like any other delivery failure.
 
+## Tracing
+
+On by default: every request records a nested tree of spans (the controller action, SQL queries,
+outbound `Net::HTTP` calls, Redis calls, and anything you wrap in
+`ForgeOpsTracker.span("PaymentService#charge") { ... }`), kept in memory until the request ends.
+Only some requests' traces are actually sent:
+
+- a request at least `trace_capture_threshold_ms` slow (default 1000), and
+- **every request that errored, however fast**: an unhandled exception, or any error reported
+  through `Rails.error` (`handle`/`record`/`report`) while the request was running. The spans
+  leading up to an error are exactly what you want next to it.
+
+A fast, successful request never sends anything.
+
+```ruby
+ForgeOpsTracker.configure do |config|
+  config.track_tracing = false             # opt out of recording spans entirely
+  config.trace_capture_threshold_ms = 500  # default 1000
+end
+```
+
+### Following a request across services
+
+Traces use the [W3C Trace Context](https://www.w3.org/TR/trace-context/) standard, so a request
+can be followed from one service into the next, whichever language or tracing library the other
+service uses:
+
+- **Incoming**: a request that arrives with a valid `traceparent` header continues that trace
+  (same trace id), and its root span records the caller's span as its parent. A missing or
+  malformed header just starts a new trace.
+- **Outgoing**: every outbound `Net::HTTP` call made during a request (Faraday's `net_http`
+  adapter, HTTParty, and RestClient included) gets a `traceparent` header whose parent id is that
+  call's own span, so the next service's spans nest under it. A `traceparent` you set yourself is
+  never replaced.
+
+A trace id exists for every request even with `track_tracing` off, and the header is still sent,
+since the trace id is also what links an error here to an error in the service you called.
+Narrow or turn off where the header goes, for example if a third-party API rejects unknown
+headers:
+
+```ruby
+ForgeOpsTracker.configure do |config|
+  config.propagate_traces = false # never send traceparent (default true)
+
+  # Default nil: every host. A String matches that host and its subdomains ("internal.example"
+  # matches "orders.internal.example", not "notinternal.example"); a Regexp is matched against
+  # the host.
+  config.trace_propagation_targets = [ "internal.example", /\A10\.0\./ ]
+end
+```
+
+Requires a ForgeOps plan that includes distributed tracing; on a plan that doesn't, traces are
+simply rejected server-side and dropped, exactly like any other delivery failure. The `trace_id`
+on error events is sent either way.
+
+## What changed
+
+ForgeOps can show what changed in your system next to the errors and slowdowns that followed it.
+Two ways in:
+
+**Record a change yourself** when something changes that no deploy captures, like a feature flag
+flipped, a config value edited, or a migration run by hand:
+
+```ruby
+ForgeOpsTracker.record_change(
+  kind: "feature_flag", # feature_flag, config, migration, dependency, infrastructure, or other
+  title: "Enabled new_checkout for 10% of users",
+  details: { flag: "new_checkout", rollout_percent: 10 },
+  actor: current_user.email,
+  url: "https://flags.example.com/new_checkout"
+)
+```
+
+`kind` and `title` are required; `details`, `environment` (defaults to `config.environment`),
+`service`, `actor`, `url`, `id` (an idempotency key, so sending the same change twice records it
+once), and `occurred_at` (defaults to now) are optional. An unknown `kind` is sent as `other`. It's
+queued and sent from a background thread, so it never slows down the caller, never raises, and is
+a no-op wherever reporting is off.
+
+**Changes between deploys are detected for you.** Once per process, right after Rails finishes
+booting, a background thread sends ForgeOps a snapshot of what the app is running: the Ruby
+version, every gem version in your `Gemfile.lock`, and the current Active Record migration version
+(left out if the database can't be reached). ForgeOps compares it with the previous boot's and
+records whatever changed, such as a gem upgrade or a new migration. Outside Rails the snapshot goes
+out when you call `ForgeOpsTracker.configure`, without the migration version.
+
+```ruby
+ForgeOpsTracker.configure do |config|
+  config.detect_changes = true       # default; false sends no startup snapshot
+  config.track_env_var_names = false # default; true also sends environment variable names
+end
+```
+
+With `track_env_var_names` on, the snapshot lists the names of your environment variables (never
+their values), so an added or removed variable shows up as a change. Names that differ from host to
+host, like `HOSTNAME`, `PATH`, `PORT`, `LC_*`, and Kubernetes service variables, are left out, as
+are the gem's own `FORGE_OPS_*` settings.
+
+Requires a ForgeOps plan that includes change tracking; on a plan that doesn't, both are rejected
+server-side and dropped, exactly like any other delivery failure.
+
 ## Custom metrics
 
 Unlike session/performance tracking above, there's no automatic instrumentation here at all: a

data/lib/forge_ops_tracker/change.rb

@@ -0,0 +1,47 @@
+require "time"
+
+module ForgeOpsTracker
+  # Builds the body for one ForgeOpsTracker.record_change call (POST /api/v1/changes): something
+  # that changed in a running system (a feature flag flipped, a config value edited, a migration
+  # run by hand), so ForgeOps can line it up against the errors and slowdowns around it.
+  module Change
+    # The server rejects any other kind outright (422), so an unknown one is sent as "other"
+    # rather than dropped: the change still gets recorded, just less specifically categorized.
+    KINDS = %w[feature_flag config migration dependency infrastructure other].freeze
+    MAX_TITLE_LENGTH = 200
+
+    module_function
+
+    # Returns nil when there's nothing sendable (a blank title): the server requires one, so
+    # posting it anyway would only ever come back 422.
+    def build(configuration, kind:, title:, details: {}, environment: nil, service: nil, actor: nil, url: nil, id: nil, occurred_at: nil)
+      title = title.to_s.strip
+      return nil if title.empty?
+
+      {
+        kind: normalize_kind(kind),
+        title: title[0, MAX_TITLE_LENGTH],
+        details: details.is_a?(Hash) ? details : {},
+        environment: (environment || configuration.environment)&.to_s,
+        service: service&.to_s,
+        actor: actor&.to_s,
+        url: url&.to_s,
+        id: id&.to_s,
+        occurred_at: iso8601(occurred_at || Time.now)
+      }.compact
+    end
+
+    def normalize_kind(kind)
+      kind = kind.to_s
+      KINDS.include?(kind) ? kind : "other"
+    end
+
+    def iso8601(value)
+      case value
+      when Time then value.utc.iso8601(3)
+      when String then value
+      else value.respond_to?(:to_time) ? value.to_time.utc.iso8601(3) : Time.now.utc.iso8601(3)
+      end
+    end
+  end
+end

data/lib/forge_ops_tracker/change_snapshot.rb

@@ -0,0 +1,133 @@
+require "thread"
+
+module ForgeOpsTracker
+  # Tells ForgeOps what this process sees at startup (the Ruby version, the locked gem versions, the
+  # current Active Record migration version, and optionally the names of its environment variables),
+  # once per process, from a background thread. The server diffs it against the last snapshot for
+  # the same project and environment and records whatever changed as a change, so this gem stays
+  # stateless: it never remembers what it sent last time.
+  #
+  # Any key that can't be determined reliably is left out rather than guessed at, since the server
+  # reads a missing key as "unknown", never as "everything was removed".
+  class ChangeSnapshot
+    MAX_DEPENDENCIES = 3000
+    MAX_NAME_LENGTH = 200
+    MAX_VERSION_LENGTH = 100
+    MAX_ENV_VAR_NAMES = 3000
+
+    # Host-specific variables that differ from one box to the next (or one boot to the next) without
+    # anything about the deploy having changed, so a fleet doesn't look like it's changing on every
+    # restart. The gem's own FORGE_OPS_* settings are dropped too.
+    ENV_VAR_DENYLIST = %w[
+      HOSTNAME HOST HOME PATH PWD OLDPWD SHLVL _ TERM USER LOGNAME SHELL LANG TMPDIR TZ PORT DYNO
+      INVOCATION_ID JOURNAL_STREAM
+    ].freeze
+    ENV_VAR_DENYLIST_PATTERNS = [
+      /\ALC_/, /\ASYSTEMD_/, /\AMEMORY_PRESSURE_/, /\AKUBERNETES_/, /\AFORGE_OPS_/,
+      /_SERVICE_HOST\z/, /_SERVICE_PORT/, /_PORT_.*_TCP/
+    ].freeze
+
+    MUTEX = Mutex.new
+
+    class << self
+      # Starts the one snapshot this process sends, on its own thread so it never delays boot. A
+      # second call is a no-op, so the Railtie and a plain ForgeOpsTracker.configure can't both
+      # send one. Does nothing (and doesn't use up the once) when the gem isn't enabled or
+      # detect_changes is off. Never raises.
+      def start(configuration)
+        return false unless configuration.enabled? && configuration.detect_changes
+
+        MUTEX.synchronize do
+          return false if @started
+
+          @started = true
+        end
+
+        thread = Thread.new { new(configuration).deliver }
+        thread.report_on_exception = false
+        true
+      rescue StandardError
+        false
+      end
+
+      # Names only, never values, sorted so the same set always serializes the same way.
+      def env_var_names(env = ENV)
+        env.keys.reject { |name| denied_env_var?(name) }.sort.first(MAX_ENV_VAR_NAMES)
+      end
+
+      def denied_env_var?(name)
+        ENV_VAR_DENYLIST.include?(name) || ENV_VAR_DENYLIST_PATTERNS.any? { |pattern| pattern.match?(name) }
+      end
+    end
+
+    def initialize(configuration, client: Client.new(configuration))
+      @configuration = configuration
+      @client = client
+    end
+
+    def deliver
+      client.deliver_change_snapshot(payload)
+    rescue StandardError => e
+      configuration.logger&.debug { "[ForgeOpsTracker] change snapshot failed: #{e.class}: #{e.message}" }
+      false
+    end
+
+    def payload
+      {
+        environment: configuration.environment.to_s,
+        state: {
+          schema_version: schema_version,
+          runtime: runtime,
+          dependencies: dependencies,
+          env_var_names: configuration.track_env_var_names ? self.class.env_var_names : nil
+        }.compact
+      }
+    end
+
+    def runtime
+      "#{RUBY_ENGINE} #{RUBY_ENGINE_VERSION}"
+    end
+
+    # The app's Gemfile.lock (through Bundler, so it's the exact resolved versions), falling back to
+    # whatever gems are actually activated when there's no bundle at all. nil, and so left out,
+    # when neither is available.
+    def dependencies
+      specs = bundler_specs || Gem.loaded_specs.values
+      return nil if specs.empty?
+
+      specs.each_with_object({}) do |spec, deps|
+        deps[spec.name.to_s[0, MAX_NAME_LENGTH]] = spec.version.to_s[0, MAX_VERSION_LENGTH]
+      end.sort.first(MAX_DEPENDENCIES).to_h
+    rescue StandardError
+      nil
+    end
+
+    # The newest migration Active Record has run against this app's database, as a string. Left
+    # out when Active Record isn't loaded, the database can't be reached, or nothing has been
+    # migrated yet: a snapshot must never fail (or fail boot) because the database is down.
+    def schema_version
+      return nil unless defined?(::ActiveRecord::Base)
+
+      version = ::ActiveRecord::Base.connection_pool.with_connection do |connection|
+        pool = ::ActiveRecord::Base.connection_pool
+        context = pool.respond_to?(:migration_context) ? pool.migration_context : connection.migration_context
+        context.current_version
+      end
+      version.to_i.positive? ? version.to_s : nil
+    rescue StandardError, ScriptError
+      nil
+    end
+
+    private
+      attr_reader :configuration, :client
+
+      def bundler_specs
+        return nil unless defined?(::Bundler)
+
+        specs = ::Bundler.locked_gems&.specs
+        specs.nil? || specs.empty? ? nil : specs
+      rescue StandardError
+        nil
+      end
+  end
+end

data/lib/forge_ops_tracker/client.rb

@@ -60,6 +60,17 @@ module ForgeOpsTracker
       post(configuration.spans_uri, { trace_id: trace_id, spans: spans })
     end
 
+    # Same delivery contract again; see Configuration#changes_uri and ForgeOpsTracker.record_change.
+    # One change per call. A plan without change tracking answers 403, which is just a false here.
+    def deliver_change(payload)
+      post(configuration.changes_uri, payload)
+    end
+
+    # Same again; see Configuration#change_snapshots_uri and ChangeSnapshot.
+    def deliver_change_snapshot(payload)
+      post(configuration.change_snapshots_uri, payload)
+    end
+
     private
       attr_reader :configuration
 

data/lib/forge_ops_tracker/configuration.rb

@@ -17,6 +17,8 @@ module ForgeOpsTracker
     attr_accessor :metric_flush_interval, :infrastructure_metric_flush_interval
     attr_accessor :gauge_poll_interval
     attr_accessor :track_tracing, :trace_capture_threshold_ms
+    attr_accessor :propagate_traces, :trace_propagation_targets
+    attr_accessor :detect_changes, :track_env_var_names
 
     def initialize
       @dsn = ENV["FORGE_OPS_DSN"]
@@ -124,6 +126,46 @@ module ForgeOpsTracker
       # PerformanceSample-derived threshold elsewhere in this gem: those describe a trend across
       # many requests, this describes a single one being outright slow.
       @trace_capture_threshold_ms = 1_000
+      # Adds a W3C `traceparent` header (https://www.w3.org/TR/trace-context/) to every outbound
+      # Net::HTTP request made during a request, so the service on the other end can continue this
+      # request's trace instead of starting its own (see Integrations::NetHTTP). On by default, the
+      # same as every other automatic behavior here, and independent of track_tracing: the header
+      # carries the trace id that links an error here to an error there, which is useful with or
+      # without spans. Never overwrites a traceparent the request already has.
+      @propagate_traces = true
+      # Which outbound hosts get that header. nil (the default) means every host. Otherwise an
+      # Array of Strings and/or Regexps: a String matches that exact host or any subdomain of it
+      # ("example.com" matches "example.com" and "api.example.com", but not "badexample.com"), a
+      # Regexp is matched against the host. Worth narrowing for an app that calls third-party APIs
+      # which reject unknown headers, or that shouldn't learn this app's trace ids at all.
+      @trace_propagation_targets = nil
+      # Sends one ChangeSnapshot per process at startup (Ruby version, locked gem versions, the
+      # current migration version), which ForgeOps diffs against the previous boot's to record what
+      # changed between deploys. On by default, like every other automatic behavior here.
+      @detect_changes = true
+      # Whether that snapshot also lists the names of this process's environment variables, so an
+      # added or removed variable shows up as a change. Off by default: names only, never values,
+      # but even names can describe more about an app than some teams want to share. See
+      # ChangeSnapshot::ENV_VAR_DENYLIST for the host-specific names that are always left out.
+      @track_env_var_names = false
+    end
+
+    # Whether an outbound request to host should carry a traceparent header; see
+    # propagate_traces/trace_propagation_targets above. Case-insensitive, since hostnames are.
+    def propagate_trace_to?(host)
+      return false unless propagate_traces
+      return true if trace_propagation_targets.nil?
+
+      host = host.to_s.downcase
+      Array(trace_propagation_targets).any? do |target|
+        case target
+        when Regexp
+          target.match?(host)
+        else
+          target = target.to_s.downcase.delete_prefix(".")
+          !target.empty? && (host == target || host.end_with?(".#{target}"))
+        end
+      end
     end
 
     def api_key
@@ -199,11 +241,30 @@ module ForgeOpsTracker
       uri
     end
 
+    def changes_uri
+      sibling_uri("/changes")
+    end
+
+    def change_snapshots_uri
+      sibling_uri("/change_snapshots")
+    end
+
     def enabled?
       !blank?(dsn) && !blank?(api_key) && enabled_environments.map(&:to_s).include?(environment.to_s)
     end
 
     private
+      # Same substitution every *_uri method above spells out by hand: a sibling path under the
+      # DSN's own /api/v1/events.
+      def sibling_uri(path)
+        uri = ingestion_uri
+        return nil unless uri
+
+        uri = uri.dup
+        uri.path = uri.path.sub(%r{/events\z}, path)
+        uri
+      end
+
       def parsed_dsn
         return nil if blank?(dsn)
 

data/lib/forge_ops_tracker/delivery_queue.rb

@@ -8,9 +8,12 @@ module ForgeOpsTracker
   # in each forked Puma/Passenger worker process rather than being carried
   # across a fork, which would leave it dead in the child.
   class DeliveryQueue
-    def initialize(configuration, client: Client.new(configuration))
+    # deliver_with names the Client method each payload goes through: :deliver (error events) by
+    # default, :deliver_change for ForgeOpsTracker.record_change's own queue.
+    def initialize(configuration, client: Client.new(configuration), deliver_with: :deliver)
       @configuration = configuration
       @client = client
+      @deliver_with = deliver_with
       @queue = SizedQueue.new(configuration.queue_size)
       @mutex = Mutex.new
       @thread = nil
@@ -44,7 +47,7 @@ module ForgeOpsTracker
 
       def run
         loop do
-          client.deliver(@queue.pop)
+          client.public_send(@deliver_with, @queue.pop)
         rescue StandardError => e
           log { "delivery thread error: #{e.class}: #{e.message}" }
         end

data/lib/forge_ops_tracker/error_subscriber.rb

@@ -28,11 +28,33 @@ module ForgeOpsTracker
       # hands this class the Rack env, so ForgeOpsTracker::Middleware::UserContext (Warden
       # auto-detection) and ForgeOpsTracker.set_user (the manual override/fallback) both
       # communicate with this exact same thread-local instead, the only channel available here.
-      # ForgeOpsTracker::Middleware::BreadcrumbContext's own per-request buffer works the same way.
+      # ForgeOpsTracker::Middleware::BreadcrumbContext's own per-request buffer works the same way,
+      # and so does Middleware::RequestContext's RequestState (trace id, transaction name,
+      # endpoint).
+      #
+      # Each live thread-local wins when it's set; the snapshot Middleware::RequestContext attached
+      # to the exception on its way out of the request is the fallback. That fallback is the whole
+      # story for an unhandled request error on Rails 8.1: ActionDispatch::Executor reports it from
+      # outside every middleware this gem installs, after all of their ensure blocks have already
+      # cleared every one of these thread-locals (see that middleware's own comment).
+      original_error = error
       error = unwrap_retry_control_flow(error)
-      user = Thread.current[:forge_ops_tracker_current_user]
-      breadcrumbs = Thread.current[:forge_ops_tracker_breadcrumbs]&.all || []
-      delivery_queue.push(event_builder.build(error, context: context, user: user, breadcrumbs: breadcrumbs))
+      snapshot = Middleware::RequestContext.snapshot_for(original_error) || Middleware::RequestContext.snapshot_for(error) || {}
+      state = Middleware::RequestContext.current
+
+      # A handled error reported during a request (Rails.error.handle/record/report) is what marks
+      # it errored here; an unhandled one was already marked by RequestContext's own rescue. Either
+      # way Middleware::SpanTracing then sends this request's trace however fast it was.
+      state&.mark_errored!
+
+      user = Thread.current[:forge_ops_tracker_current_user] || snapshot[:user]
+      breadcrumbs = Thread.current[:forge_ops_tracker_breadcrumbs]&.all || snapshot[:breadcrumbs] || []
+      delivery_queue.push(event_builder.build(
+        error, context: context, user: user, breadcrumbs: breadcrumbs,
+        transaction_name: state&.transaction_name || snapshot[:transaction_name],
+        endpoint: state&.endpoint || snapshot[:endpoint],
+        trace_id: state&.trace_id || snapshot[:trace_id]
+      ))
       nil
     rescue StandardError => e
       configuration.logger&.debug { "[ForgeOpsTracker] report failed: #{e.class}: #{e.message}" }

data/lib/forge_ops_tracker/event_builder.rb

@@ -27,7 +27,15 @@ module ForgeOpsTracker
       @configuration = configuration
     end
 
-    def build(error, context: {}, user: nil, breadcrumbs: [])
+    # transaction_name/endpoint/trace_id describe the request the error happened in (see
+    # RequestState): transaction_name is the same "CheckoutsController#create" name the request's
+    # root span and performance samples use, endpoint the human-readable route ("POST /checkout"),
+    # trace_id the request's 32-character lowercase hex W3C trace id, which is also what links this
+    # error to errors other services reported for the same trace. All three are nil outside a
+    # request (a background job, a console session) and each key is left out of the payload
+    # entirely when nil, the same "omit rather than send an empty value" treatment user and
+    # breadcrumbs already get.
+    def build(error, context: {}, user: nil, breadcrumbs: [], transaction_name: nil, endpoint: nil, trace_id: nil)
       payload = {
         exception_class: error.class.name,
         message: error.message.to_s,
@@ -42,6 +50,9 @@ module ForgeOpsTracker
       }
       payload[:user] = user if user && !user.empty?
       payload[:breadcrumbs] = breadcrumbs if breadcrumbs && !breadcrumbs.empty?
+      payload[:transaction_name] = transaction_name if transaction_name
+      payload[:endpoint] = endpoint if endpoint
+      payload[:trace_id] = trace_id if trace_id
       attach_sql(payload, error)
       scrub(payload)
     end
@@ -49,9 +60,11 @@ module ForgeOpsTracker
     private
       attr_reader :configuration
 
-      # exception_class/occurred_at/environment/release/server_name/sdk_name/user are left alone;
-      # structured fields this gem or the host app sets deliberately, not free text an exception
-      # or its context could accidentally spill sensitive data into. user specifically is a
+      # exception_class/occurred_at/environment/release/server_name/sdk_name/user/transaction_name/
+      # endpoint/trace_id are left alone; structured fields this gem or the host app sets
+      # deliberately, not free text an exception or its context could accidentally spill sensitive
+      # data into. endpoint in particular is a declared route pattern ("GET /orders/:id"), never
+      # the literal request path, so it has no id or token in it to redact. user specifically is a
       # deliberate exemption, not an oversight: the server's own PiiScrubber-equivalent would
       # otherwise redact the exact email address this field exists to carry. breadcrumbs is *not*
       # exempt, unlike user: console-style/query/request trail entries are exactly the kind of free

data/lib/forge_ops_tracker/integrations/net_http.rb

@@ -23,14 +23,25 @@ module ForgeOpsTracker
       # confirmed directly that Net::HTTP connects lazily, inside #request itself, the first time
       # it's called on a not-yet-started instance, so that path's own connection failures are
       # caught exactly like any other #request failure.
+      #
+      # Also propagates the current request's trace to the service being called, as a W3C
+      # `traceparent` header (see TraceParent, and Configuration#propagate_traces/
+      # trace_propagation_targets for turning it off or narrowing it to specific hosts). The
+      # header's parent id is this outbound call's own span id, generated before the call is made
+      # (the span itself can only be recorded afterward, once its duration is known) and then
+      # recorded with that exact id, so the downstream service's root span points at a span that
+      # really exists in this trace. Only inside a request (a RequestState to take the trace id
+      # from); a background job or a console session has no trace to continue.
       module Timing
         # The guard (return super unless ...) skips timing, but the ensure block below still runs
         # on every exit path regardless, same as any ensure; start staying nil is what actually
         # skips recording there, not a second guard duplicating this one.
         def request(req, body = nil, &block)
           configuration = ForgeOpsTracker.configuration
-          return super unless (configuration.track_performance || configuration.track_breadcrumbs || configuration.track_failures || configuration.track_tracing) && configuration.enabled?
+          return super unless (configuration.track_performance || configuration.track_breadcrumbs || configuration.track_failures || configuration.track_tracing || configuration.propagate_traces) && configuration.enabled?
 
+          span_id = ForgeOpsTracker::TraceParent.generate_span_id
+          propagate_trace(req, span_id, configuration)
           start = ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
           response = super
         rescue StandardError => e
@@ -102,12 +113,26 @@ module ForgeOpsTracker
               # it actually took.
               start_time = ::Time.now.utc - (duration_ms / 1000.0)
               buffer.record(
-                span_id: SecureRandom.hex(8), name: "#{req.method} #{address}", kind: "http",
+                span_id: span_id, name: "#{req.method} #{address}", kind: "http",
                 started_at: start_time, duration_ms: duration_ms, data: { status: status }.compact
               )
             end
           end
         end
+
+        private
+          # Leaves a traceparent the caller already set alone: whoever set it explicitly knows
+          # better than this gem which trace the call belongs to. Never raises, since a failure to
+          # add a header must never stop the host app's own request from going out.
+          def propagate_trace(req, span_id, configuration)
+            state = ForgeOpsTracker::Middleware::RequestContext.current
+            return unless state && configuration.propagate_trace_to?(address)
+            return if req[ForgeOpsTracker::TraceParent::HEADER]
+
+            req[ForgeOpsTracker::TraceParent::HEADER] = ForgeOpsTracker::TraceParent.build(trace_id: state.trace_id, span_id: span_id)
+          rescue StandardError
+            nil
+          end
       end
     end
   end

data/lib/forge_ops_tracker/middleware/request_context.rb

@@ -0,0 +1,90 @@
+module ForgeOpsTracker
+  module Middleware
+    # Publishes this request's RequestState (trace id, remote parent span id, transaction name,
+    # endpoint, errored flag; see that class) as a thread-local for the duration of the request,
+    # cleared in an ensure for the same "Puma reuses threads" reason every other middleware in this
+    # gem already gives. Not gated on track_tracing: the trace id links an error to errors in other
+    # services even when no spans are ever recorded.
+    #
+    # Its second job is the reason it has to be the *innermost* of this gem's middlewares (added
+    # last in Railtie, so app.middleware.use puts it closest to the app). Rails 8.1 reports an
+    # unhandled request exception from ActionDispatch::Executor, which sits outside all of this
+    # gem's middlewares; by the time ErrorSubscriber#report runs for that exception, every one of
+    # their ensure blocks has already cleared its thread-local, so the user, the breadcrumb trail,
+    # and the trace id/endpoint would all be gone for exactly the errors that matter most. Rescuing
+    # here, innermost, is the last moment all of those are still set: this copies them onto the
+    # exception object itself (see SNAPSHOT_IVAR) before re-raising it unchanged, and
+    # ErrorSubscriber#report falls back to that copy whenever the live thread-locals are gone.
+    #
+    # Rescues Exception, not StandardError, and always re-raises: this only ever observes an
+    # exception on its way out, it never handles one, so there's no reason to let anything slip
+    # past without its context attached.
+    class RequestContext
+      THREAD_KEY = :forge_ops_tracker_request_context
+
+      # An instance variable on the exception object, not a wrapper exception or a side table: the
+      # exception is the one thing guaranteed to travel from here to wherever Rails ends up
+      # reporting it, and a plain ivar leaves its class, message, and backtrace exactly as the host
+      # app raised them.
+      SNAPSHOT_IVAR = :@__forge_ops_tracker_request_snapshot
+
+      def initialize(app, configuration: ForgeOpsTracker.configuration)
+        @app = app
+        @configuration = configuration
+      end
+
+      def call(env)
+        return app.call(env) unless configuration.enabled?
+
+        state = RequestState.for(env)
+        Thread.current[THREAD_KEY] = state
+        app.call(env)
+      rescue Exception => e
+        if state
+          state.mark_errored!
+          self.class.attach_snapshot(e, state)
+        end
+        raise
+      ensure
+        Thread.current[THREAD_KEY] = nil
+      end
+
+      class << self
+        def current
+          Thread.current[THREAD_KEY]
+        end
+
+        # Never raises, whatever it's handed: a frozen exception (some libraries freeze theirs, and
+        # a frozen object rejects instance_variable_set with a FrozenError) just goes without a
+        # snapshot, and so does anything else unexpected, since failing here would replace the host
+        # app's own exception with one of this gem's. The first snapshot wins if the same exception
+        # object somehow passes through twice: the innermost one saw the request closest to where
+        # it actually failed.
+        def attach_snapshot(error, state)
+          return if error.frozen? || error.instance_variable_defined?(SNAPSHOT_IVAR)
+
+          error.instance_variable_set(SNAPSHOT_IVAR, {
+            user: Thread.current[:forge_ops_tracker_current_user],
+            breadcrumbs: Thread.current[:forge_ops_tracker_breadcrumbs]&.all,
+            transaction_name: state.transaction_name,
+            endpoint: state.endpoint,
+            trace_id: state.trace_id
+          }.freeze)
+        rescue StandardError
+          nil
+        end
+
+        def snapshot_for(error)
+          return nil unless error.instance_variable_defined?(SNAPSHOT_IVAR)
+
+          error.instance_variable_get(SNAPSHOT_IVAR)
+        rescue StandardError
+          nil
+        end
+      end
+
+      private
+        attr_reader :app, :configuration
+    end
+  end
+end

data/lib/forge_ops_tracker/middleware/span_tracing.rb

@@ -3,15 +3,29 @@ module ForgeOpsTracker
     # Gives every request its own fresh SpanBuffer (see that class for the stack/nesting
     # mechanics), the same "Thread.current, cleared in an ensure" shape BreadcrumbContext already
     # uses and for the identical reason: Puma reuses threads across requests, so a buffer left set
-    # would leak one request's spans into a later, unrelated one handled on the same thread.
+    # would leak one request's spans into a later, unrelated one handled on the same thread. The
+    # buffer's trace_id and remote parent span id come from the request's RequestState, never
+    # generated here, so a trace this sends and an error event from the same request always agree
+    # on which trace they belong to (see RequestState for why that object is shared through the
+    # Rack env rather than the thread-local RequestContext publishes: this middleware sits outside
+    # that one, and runs first).
     #
     # The actual send-or-don't decision lives here, not in SpanBuffer or SpanQueue: only once
     # app.call returns does this middleware know the root span's own final duration (recorded by
     # Railtie's process_action.action_controller subscription, which fires from *inside* that
     # call, after every nested query/HTTP/Redis call has already run), so this is the first point
     # that can compare it against configuration.trace_capture_threshold_ms and decide the trace was
-    # never worth sending in the first place - the entire reason a normal, fast request never costs
+    # never worth sending in the first place: the entire reason a normal, fast request never costs
     # a single byte over the wire, unlike PerformanceSample's own always-on aggregate reporting.
+    #
+    # An errored request's trace is always sent too, however fast it was, since the waterfall of
+    # what led up to an error is exactly what an issue page wants to show next to it. "Errored"
+    # means the RequestState was marked by either path: an unhandled exception passing through
+    # RequestContext on its way out, or ErrorSubscriber#report receiving a handled error during the
+    # request. Both have already happened by the time this ensure runs; the unhandled one is only
+    # *reported* later (by ActionDispatch::Executor, outside every middleware here), but it's
+    # marked on the way through, which is what counts. Requires the root span to have been
+    # recorded, see SpanBuffer#root_recorded? for why.
     class SpanTracing
       def initialize(app, configuration: ForgeOpsTracker.configuration)
         @app = app
@@ -21,11 +35,12 @@ module ForgeOpsTracker
       def call(env)
         return app.call(env) unless configuration.track_tracing && configuration.enabled?
 
-        buffer = SpanBuffer.new
+        state = RequestState.for(env)
+        buffer = SpanBuffer.new(trace_id: state.trace_id, parent_span_id: state.parent_span_id)
         Thread.current[:forge_ops_tracker_spans] = buffer
         app.call(env)
       ensure
-        if buffer&.slow?(configuration.trace_capture_threshold_ms)
+        if buffer && send_trace?(buffer, state)
           Railtie.span_queue.push(trace_id: buffer.trace_id, spans: buffer.spans)
         end
         Thread.current[:forge_ops_tracker_spans] = nil
@@ -33,6 +48,12 @@ module ForgeOpsTracker
 
       private
         attr_reader :app, :configuration
+
+        def send_trace?(buffer, state)
+          return true if buffer.slow?(configuration.trace_capture_threshold_ms)
+
+          state.errored? && buffer.root_recorded?
+        end
     end
   end
 end

data/lib/forge_ops_tracker/performance_instrumentation.rb

@@ -16,6 +16,37 @@ module ForgeOpsTracker
       "#{payload[:controller]}##{payload[:action]}"
     end
 
+    # A human-readable route for an error event's `endpoint` field, "<HTTP method> <route pattern>"
+    # ("POST /checkout", "GET /orders/:id"): the route as it was declared in config/routes.rb, never
+    # the literal request path, so it stays low-cardinality and never carries an id or a token, the
+    # same reasoning every transaction_name in this system already follows. payload[:request] is
+    # only present in Rails 7.1+, the same release that added
+    # ActionDispatch::Request#route_uri_pattern; on anything older (or a request with no matched
+    # route pattern at all) this falls back to the controller/action name instead
+    # ("GET CheckoutsController#create"), still meaningful, just less familiar-looking than a URL.
+    def controller_endpoint(payload)
+      target = route_pattern(payload[:request]) || controller_transaction_name(payload)
+      method = payload[:method]
+      method.nil? || method.to_s.empty? ? target : "#{method} #{target}"
+    end
+
+    # Rails appends an optional "(.:format)" segment to nearly every route pattern it generates;
+    # it's noise in a label a person reads, so it's dropped. Defensive about everything else, since
+    # this runs inside a notification subscriber on every single request: a request object without
+    # the method, a nil/empty pattern, or the method itself raising all just mean "no pattern," and
+    # the caller falls back.
+    def route_pattern(request)
+      return nil unless request.respond_to?(:route_uri_pattern)
+
+      pattern = request.route_uri_pattern
+      return nil if pattern.nil? || pattern.to_s.empty?
+
+      pattern = pattern.to_s.delete_suffix("(.:format)")
+      pattern.empty? ? "/" : pattern
+    rescue StandardError
+      nil
+    end
+
     # nil means "don't record this one": Rails' own sql.active_record fires for two kinds of
     # noise that would drown out real query timing otherwise. "SCHEMA" is Rails' own internal
     # introspection (column lookups, etc.), never something a customer's own code triggered;

data/lib/forge_ops_tracker/railtie.rb

@@ -73,6 +73,31 @@ module ForgeOpsTracker
       app.middleware.use ForgeOpsTracker::Middleware::SpanTracing, configuration: ForgeOpsTracker.configuration
     end
 
+    # Must stay the last app.middleware.use in this file, so RequestContext is the innermost of
+    # this gem's middlewares: its rescue is what copies the user, the breadcrumb trail, and the
+    # trace id/endpoint onto an unhandled exception before any of the middlewares above clear their
+    # thread-locals in their own ensure blocks, and ActionDispatch::Executor (which reports that
+    # exception in Rails 8.1) sits outside all of them. See that middleware's own comment.
+    # Initializers within one Railtie run in the order they're defined, and app.middleware.use
+    # appends, so being defined after track_tracing is what makes it innermost.
+    #
+    # start_processing.action_controller fires as the controller begins handling the request, before
+    # the action itself runs (unlike process_action.action_controller below, which only fires once
+    # it has finished), so a handled error reported from inside the action (Rails.error.handle/
+    # record) already knows which transaction and endpoint it happened in. Its payload carries
+    # :controller/:action/:method, plus :request from Rails 7.1 on (see
+    # PerformanceInstrumentation.controller_endpoint for what each is used for).
+    initializer "forge_ops_tracker.request_context" do |app|
+      configuration = ForgeOpsTracker.configuration
+      app.middleware.use ForgeOpsTracker::Middleware::RequestContext, configuration: configuration
+
+      ActiveSupport::Notifications.subscribe("start_processing.action_controller") do |*args|
+        next unless configuration.enabled?
+
+        ForgeOpsTracker::Middleware::RequestContext.current&.controller_started(args.last)
+      end
+    end
+
     # Rails already fires "process_action.action_controller" after every request with the
     # controller/action and how long it took (event.duration, in ms) built in, so unlike session
     # tracking above (which needed a Rack middleware because there's no single existing hook for
@@ -130,8 +155,8 @@ module ForgeOpsTracker
         # this exact call site already established for breadcrumbs): this is the one call in the
         # whole gem that ever passes root: true, since process_action.action_controller is the only
         # instrumentation point that wraps a genuinely whole request. See SpanBuffer#record's own
-        # comment for why root: true has to force parent_span_id nil explicitly here rather than
-        # reading it off the (already-popped-by-now) stack.
+        # comment for why root: true takes its parent_span_id from the incoming traceparent header
+        # (nil when there wasn't one) rather than reading it off the (already-popped-by-now) stack.
         if configuration.track_tracing && (buffer = Thread.current[:forge_ops_tracker_spans])
           buffer.pop
           buffer.record(
@@ -280,6 +305,14 @@ module ForgeOpsTracker
     # sidekiq, ActiveJob's redis queue adapter, ...), which this gem has no control over; deferring
     # the defined? check to after every railtie (including this app's own dependencies) has
     # finished initializing avoids a load-order-dependent false negative.
+    # After initialization, not in a plain initializer, so the app's own
+    # config/initializers/forge_ops_tracker.rb has already run (detect_changes/track_env_var_names
+    # and the DSN are final) and Active Record is configured enough to report the migration
+    # version. ChangeSnapshot does the actual work on its own thread, so boot never waits on it.
+    config.after_initialize do
+      ForgeOpsTracker::ChangeSnapshot.start(ForgeOpsTracker.configuration)
+    end
+
     config.after_initialize do
       next unless defined?(::RedisClient)
 

data/lib/forge_ops_tracker/request_state.rb

@@ -0,0 +1,69 @@
+module ForgeOpsTracker
+  # Everything this gem knows about the current request that isn't already owned by one of the
+  # older, single-purpose thread-locals (the user, the breadcrumb trail, the span buffer): which
+  # trace it belongs to, which remote span called it (if any), what it's called, and whether it
+  # errored. One instance per request, shared by every piece of this gem that needs any of those:
+  #
+  # - Middleware::RequestContext publishes it as a thread-local
+  #   (Thread.current[:forge_ops_tracker_request_context]) for the duration of the request, so
+  #   ErrorSubscriber#report, the start_processing.action_controller subscription, and the
+  #   Net::HTTP integration can reach it (none of them are ever handed the Rack env).
+  # - Middleware::SpanTracing reads the same instance's trace_id/parent_span_id for its SpanBuffer,
+  #   and its errored? flag once the request is over.
+  #
+  # The trace_id exists for every request, not only when span tracing is on: it's also what an
+  # error event carries so ForgeOps can link it to errors reported by other services taking part in
+  # the same trace, which works on any plan and with track_tracing off entirely.
+  #
+  # Memoized in the Rack env (see .for) rather than created by one specific middleware and handed to
+  # the other: SpanTracing sits *outside* RequestContext in the stack (RequestContext must be the
+  # innermost of this gem's middlewares; see its own comment for why), so SpanTracing runs first and
+  # needs the trace_id before RequestContext has run at all, then keeps needing this same object
+  # after RequestContext's own ensure has already cleared the thread-local. Whichever of the two
+  # reaches a given request first creates it; the other gets the identical instance back.
+  class RequestState
+    ENV_KEY = "forge_ops_tracker.request_state".freeze
+
+    attr_reader :trace_id, :parent_span_id
+    attr_accessor :transaction_name, :endpoint
+
+    def self.for(env)
+      env[ENV_KEY] ||= from_env(env)
+    end
+
+    # Continues the caller's trace when the request arrived with a usable W3C traceparent header,
+    # remembering the caller's span as this request's remote parent; starts a fresh trace otherwise.
+    def self.from_env(env)
+      incoming = TraceParent.parse(env[TraceParent::RACK_ENV_KEY])
+      if incoming
+        new(trace_id: incoming[:trace_id], parent_span_id: incoming[:parent_span_id])
+      else
+        new
+      end
+    end
+
+    def initialize(trace_id: TraceParent.generate_trace_id, parent_span_id: nil)
+      @trace_id = trace_id
+      @parent_span_id = parent_span_id
+      @transaction_name = nil
+      @endpoint = nil
+      @errored = false
+    end
+
+    # Called from the start_processing.action_controller subscription (see Railtie), the earliest
+    # point Rails names the controller/action handling this request: early enough that an error
+    # reported partway through the action itself (Rails.error.handle/record) already carries both.
+    def controller_started(payload)
+      self.transaction_name = PerformanceInstrumentation.controller_transaction_name(payload)
+      self.endpoint = PerformanceInstrumentation.controller_endpoint(payload)
+    end
+
+    def mark_errored!
+      @errored = true
+    end
+
+    def errored?
+      @errored
+    end
+  end
+end

data/lib/forge_ops_tracker/span_buffer.rb

@@ -3,9 +3,13 @@ require "securerandom"
 module ForgeOpsTracker
   # Accumulates one request's own nested call tree in-process; the tracing analog to
   # BreadcrumbBuffer's per-thread trail. ForgeOpsTracker::Middleware::SpanTracing resets this to a
-  # fresh buffer at the start of every request, with a freshly generated trace_id and root
-  # span_id, and reads it back once the request is over to decide whether this trace is worth
-  # sending at all (see that middleware's own comment for exactly where and why).
+  # fresh buffer at the start of every request, with a freshly generated root span_id and the
+  # request's own trace_id (and remote parent span id, when the request continued a trace another
+  # service started; see RequestState), and reads it back once the request is over to decide
+  # whether this trace is worth sending at all (see that middleware's own comment for exactly where
+  # and why). trace_id/parent_span_id are passed in rather than generated here: the trace_id
+  # belongs to the request, not to span tracing, since an error event carries it too, even with
+  # tracing off, and the two have to agree for ForgeOps to link them.
   #
   # The stack exists so nesting can be reconstructed correctly even though most of what's recorded
   # here (a query, an outbound HTTP call, a Redis call) only has a start+duration known *after* the
@@ -19,10 +23,11 @@ module ForgeOpsTracker
   # already run) naturally nests under it without the root having to exist as a real recorded span
   # first.
   class SpanBuffer
-    attr_reader :trace_id, :root_span_id, :spans
+    attr_reader :trace_id, :root_span_id, :parent_span_id, :spans
 
-    def initialize
-      @trace_id = SecureRandom.hex(16)
+    def initialize(trace_id: SecureRandom.hex(16), parent_span_id: nil)
+      @trace_id = trace_id
+      @parent_span_id = parent_span_id
       @root_span_id = SecureRandom.hex(8)
       @spans = []
       @stack = [ @root_span_id ]
@@ -42,12 +47,14 @@ module ForgeOpsTracker
     end
 
     # root: true only for the one call that records the request's own root span (see Railtie's
-    # process_action.action_controller subscription): forces parent_span_id nil explicitly, since
-    # by the time that fires, the root's own span_id has already been popped off the stack (there's
-    # nothing left on it to read back as "the parent" at that point - correctly, since the root has
-    # no parent of its own).
+    # process_action.action_controller subscription): takes its parent from the incoming
+    # traceparent header's span id instead of the stack, since by the time that fires, the root's
+    # own span_id has already been popped off it (there's nothing left on it to read back as "the
+    # parent" at that point, correctly: nothing inside this process is the root's parent). nil for
+    # a request that started its own trace; the calling service's span otherwise, which ForgeOps
+    # treats as a remote parent because it's never part of this same batch.
     def record(span_id:, name:, kind:, started_at:, duration_ms:, data: {}, root: false)
-      parent_span_id = root ? nil : current_parent_id
+      parent_span_id = root ? @parent_span_id : current_parent_id
       spans << {
         span_id: span_id, parent_span_id: parent_span_id, name: name, kind: kind,
         started_at: started_at.utc.iso8601(3), duration_ms: duration_ms, data: data
@@ -61,5 +68,13 @@ module ForgeOpsTracker
     def slow?(threshold_ms)
       @root_duration_ms && @root_duration_ms >= threshold_ms
     end
+
+    # Whether the root span was ever recorded at all. Middleware::SpanTracing only sends an errored
+    # request's trace when this is true: without the root, every other span in the buffer points at
+    # a parent that was never recorded, which ForgeOps would misread as a remote parent in some
+    # other service rather than a missing root in this one.
+    def root_recorded?
+      !@root_duration_ms.nil?
+    end
   end
 end

data/lib/forge_ops_tracker/trace_parent.rb

@@ -0,0 +1,62 @@
+require "securerandom"
+
+module ForgeOpsTracker
+  # Reads and writes the W3C Trace Context `traceparent` header
+  # (https://www.w3.org/TR/trace-context/), the vendor-neutral format for carrying one trace across
+  # service boundaries: `00-<32 hex trace-id>-<16 hex parent-id>-<2 hex flags>`. Used in both
+  # directions: Middleware::RequestContext parses an incoming one so this request continues the
+  # caller's trace instead of starting its own, and Integrations::NetHTTP builds an outgoing one so
+  # the next service along continues this request's.
+  #
+  # Deliberately strict on the way in, the same posture the spec itself asks receivers to take: a
+  # malformed value, uppercase hex, the reserved version "ff", or an all-zero trace/parent id are
+  # all treated as "no usable header at all" (parse returns nil and the request starts a fresh
+  # trace), never half-trusted. A version this gem doesn't know yet is still accepted as long as
+  # its first four fields have version 00's shape, which is exactly what the spec says a version-00
+  # parser should do with a future version; version 00 itself must have exactly four fields.
+  module TraceParent
+    HEADER = "traceparent".freeze
+    RACK_ENV_KEY = "HTTP_TRACEPARENT".freeze
+
+    PATTERN = /\A(?<version>[0-9a-f]{2})-(?<trace_id>[0-9a-f]{32})-(?<parent_id>[0-9a-f]{16})-(?<flags>[0-9a-f]{2})(?<rest>-.*)?\z/
+    INVALID_TRACE_ID = ("0" * 32).freeze
+    INVALID_PARENT_ID = ("0" * 16).freeze
+
+    # Always "01" (sampled) on the way out: this gem decides whether to actually send a trace only
+    # once the request is over (see Middleware::SpanTracing), long after this header has already
+    # gone out on an outbound call, so there's no honest earlier answer to give than "this may be
+    # recorded." A downstream service is free to make its own decision either way.
+    SAMPLED_FLAGS = "01".freeze
+
+    module_function
+
+    # Returns { trace_id:, parent_span_id: } for a usable header, nil for anything else (absent,
+    # blank, malformed, or one of the explicitly invalid values above).
+    def parse(value)
+      return nil unless value.is_a?(String)
+
+      match = PATTERN.match(value.strip)
+      return nil unless match
+      return nil if match[:version] == "ff"
+      return nil if match[:version] == "00" && match[:rest]
+      return nil if match[:trace_id] == INVALID_TRACE_ID || match[:parent_id] == INVALID_PARENT_ID
+
+      { trace_id: match[:trace_id], parent_span_id: match[:parent_id] }
+    end
+
+    def build(trace_id:, span_id:)
+      "00-#{trace_id}-#{span_id}-#{SAMPLED_FLAGS}"
+    end
+
+    # 32 lowercase hex characters, the W3C trace-id format; SecureRandom.hex already produces
+    # lowercase, which is the only case the spec allows.
+    def generate_trace_id
+      SecureRandom.hex(16)
+    end
+
+    # 16 lowercase hex characters, the W3C parent-id (span id) format.
+    def generate_span_id
+      SecureRandom.hex(8)
+    end
+  end
+end

data/lib/forge_ops_tracker/version.rb

@@ -1,3 +1,3 @@
 module ForgeOpsTracker
-  VERSION = "0.11.0"
+  VERSION = "0.13.0"
 end

data/lib/forge_ops_tracker.rb

@@ -6,6 +6,8 @@ require "forge_ops_tracker/sql_statement"
 require "forge_ops_tracker/event_builder"
 require "forge_ops_tracker/client"
 require "forge_ops_tracker/delivery_queue"
+require "forge_ops_tracker/change"
+require "forge_ops_tracker/change_snapshot"
 require "forge_ops_tracker/error_subscriber"
 require "forge_ops_tracker/session_flusher"
 require "forge_ops_tracker/histogram_bucketer"
@@ -18,10 +20,13 @@ require "forge_ops_tracker/periodic_poller"
 require "forge_ops_tracker/breadcrumb_buffer"
 require "forge_ops_tracker/span_buffer"
 require "forge_ops_tracker/span_queue"
+require "forge_ops_tracker/trace_parent"
+require "forge_ops_tracker/request_state"
 require "forge_ops_tracker/middleware/session_tracking"
 require "forge_ops_tracker/middleware/user_context"
 require "forge_ops_tracker/middleware/breadcrumb_context"
 require "forge_ops_tracker/middleware/span_tracing"
+require "forge_ops_tracker/middleware/request_context"
 
 module ForgeOpsTracker
   class << self
@@ -29,8 +34,33 @@ module ForgeOpsTracker
       @configuration ||= Configuration.new
     end
 
+    # Outside Rails this is also where the startup ChangeSnapshot goes out, since there's no
+    # Railtie to send it once the app has booted; under Rails the Railtie sends it instead, after
+    # initialization, when Active Record can report the migration version.
     def configure
       yield configuration
+      ChangeSnapshot.start(configuration) unless defined?(::Rails::Railtie)
+    end
+
+    # Records one thing that changed in a running system, e.g.
+    # `ForgeOpsTracker.record_change(kind: "feature_flag", title: "Enabled new_checkout for 10%")`,
+    # so ForgeOps can show it next to the errors and slowdowns that followed. kind is one of
+    # Change::KINDS (anything else is sent as "other"); environment defaults to the configured one.
+    # Delivered from a background thread like error events, so it never blocks the caller, and a
+    # no-op when the gem isn't enabled. Never raises: returns false when nothing was queued.
+    def record_change(kind:, title:, details: {}, environment: nil, service: nil, actor: nil, url: nil, id: nil, occurred_at: nil)
+      return false unless configuration.enabled?
+
+      payload = Change.build(
+        configuration, kind: kind, title: title, details: details, environment: environment,
+        service: service, actor: actor, url: url, id: id, occurred_at: occurred_at
+      )
+      return false unless payload
+
+      change_queue.push(payload)
+    rescue StandardError => e
+      configuration.logger&.debug { "[ForgeOpsTracker] record_change failed: #{e.class}: #{e.message}" }
+      false
     end
 
     # Records a named business metric (a signup, a payment, anything a customer wants to name),
@@ -117,6 +147,10 @@ module ForgeOpsTracker
     end
 
     private
+      def change_queue
+        @change_queue ||= DeliveryQueue.new(configuration, deliver_with: :deliver_change)
+      end
+
       def metric_buffer
         @metric_buffer ||= MetricBuffer.new(configuration)
       end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: forge_ops_tracker
 version: !ruby/object:Gem::Version
-  version: 0.11.0
+  version: 0.13.0
 platform: ruby
 authors:
 - ForgeOps
@@ -90,6 +90,8 @@ files:
 - README.md
 - lib/forge_ops_tracker.rb
 - lib/forge_ops_tracker/breadcrumb_buffer.rb
+- lib/forge_ops_tracker/change.rb
+- lib/forge_ops_tracker/change_snapshot.rb
 - lib/forge_ops_tracker/client.rb
 - lib/forge_ops_tracker/configuration.rb
 - lib/forge_ops_tracker/delivery_queue.rb
@@ -106,6 +108,7 @@ files:
 - lib/forge_ops_tracker/integrations/solid_queue.rb
 - lib/forge_ops_tracker/metric_buffer.rb
 - lib/forge_ops_tracker/middleware/breadcrumb_context.rb
+- lib/forge_ops_tracker/middleware/request_context.rb
 - lib/forge_ops_tracker/middleware/session_tracking.rb
 - lib/forge_ops_tracker/middleware/span_tracing.rb
 - lib/forge_ops_tracker/middleware/user_context.rb
@@ -114,10 +117,12 @@ files:
 - lib/forge_ops_tracker/periodic_poller.rb
 - lib/forge_ops_tracker/pii_scrubber.rb
 - lib/forge_ops_tracker/railtie.rb
+- lib/forge_ops_tracker/request_state.rb
 - lib/forge_ops_tracker/session_flusher.rb
 - lib/forge_ops_tracker/span_buffer.rb
 - lib/forge_ops_tracker/span_queue.rb
 - lib/forge_ops_tracker/sql_statement.rb
+- lib/forge_ops_tracker/trace_parent.rb
 - lib/forge_ops_tracker/version.rb
 homepage: https://getforgeops.net
 licenses: