@forge-ops/tracker 0.10.0 → 0.11.0

package/README.md

@@ -113,6 +113,28 @@ themselves, long before it would ever reach here.
 Every failure mode (network errors, timeouts, a full queue, a malformed DSN) is caught and
 dropped rather than thrown, so a broken or unreachable tracker can never take down the host app.
 
+## Where an error happened
+
+An error reported during a request carries three extra fields:
+
+- `transaction_name`: the method and the route pattern, `"GET /orders/:id"`, the same name
+  performance monitoring and traces use.
+- `endpoint`: the method and the route as declared, `"GET /orders/:id"`. Never the literal path, so
+  an id or a token in the URL never ends up here. On Express, it's the route as declared on the
+  router that matched it: for a router mounted with `app.use("/api", router)`, that's the part
+  declared on the router, without `/api`, since the mount path Express exposes is the literal
+  matched one.
+- `trace_id`: the request's W3C trace id, which ForgeOps uses to link this error to errors that
+  other services reported for the same trace (see "Following a request across services" below).
+
+Register `forgeOpsTrackerTracingExpressMiddleware` before your routes, or call
+`registerForgeOpsTrackerTracing(app)` on Fastify (see "Distributed tracing" below), for these to be
+set on an error you capture yourself inside a handler; Fastify knows the route before any handler
+runs, Express as soon as the matched handler starts. `forgeOpsTrackerExpressMiddleware` and
+`registerForgeOpsTracker` add them to an unhandled error on their own. They're left out entirely
+outside a request (a script, a worker), and they're never PII-scrubbed, like `environment` and
+`release`: they're structured fields, not free text.
+
 ## Identifying users
 
 ```js
@@ -355,7 +377,7 @@ batch behind it. Requires a ForgeOps plan that includes custom metrics / infrast
 
 ## Distributed tracing
 
-For one slow request, `forgeOpsTrackerTracingExpressMiddleware` (Express) and
+For one slow or errored request, `forgeOpsTrackerTracingExpressMiddleware` (Express) and
 `registerForgeOpsTrackerTracing()` (Fastify, `./integrations/tracing`) capture its full nested
 call tree: the route span, plus every outbound HTTP call and manually-wrapped span nested under
 it, so ForgeOps can render a waterfall for that one request.
@@ -363,7 +385,7 @@ it, so ForgeOps can render a waterfall for that one request.
 ```js
 import { forgeOpsTrackerTracingExpressMiddleware } from "@forge-ops/tracker/integrations/tracing";
 
-app.use(forgeOpsTrackerTracingExpressMiddleware);
+app.use(forgeOpsTrackerTracingExpressMiddleware); // before your routes
 ```
 
 ```js
@@ -375,7 +397,9 @@ registerForgeOpsTrackerTracing(fastify);
 This is the whole point of the feature, so it's worth being explicit about: a request's own trace
 is only ever built, let alone sent, once its own root span's duration crosses a threshold, decided
 entirely client-side before a single byte goes over the wire. A normal, fast request costs
-nothing extra.
+nothing extra. A request that errored (an unhandled error, or any `captureException()` during it)
+is the one exception: its trace is always sent, however fast it was, since the spans leading up to
+an error are exactly what you want next to it.
 
 ```js
 forgeOpsTracker.init({
@@ -413,6 +437,58 @@ with `trackTracing` off: it just runs the callback and records nothing, never th
 Requires a ForgeOps plan that includes distributed tracing; on a plan that doesn't, a captured
 trace is simply rejected server-side and dropped, exactly like any other delivery failure.
 
+### Following a request across services
+
+Traces use the [W3C Trace Context](https://www.w3.org/TR/trace-context/) standard, so a request can
+be followed from one service into the next, whichever language or tracing library the other
+service uses:
+
+- **Incoming**: a request that arrives with a valid `traceparent` header continues that trace (same
+  trace id), and its root span records the caller's span as its parent. A missing or malformed
+  header just starts a new trace.
+- **Outgoing**: every outbound call made through Node's `http`/`https` modules during a request
+  gets a `traceparent` header whose parent id is that call's own span, so the next service's spans
+  nest under it. A `traceparent` you set yourself is never replaced, and this client's own
+  deliveries to ForgeOps never get one. The global `fetch()` doesn't go through `http`/`https`, so
+  it isn't instrumented and gets no header.
+
+A trace id exists for every request even with `trackTracing` off, and the header is still sent,
+since the trace id is also what links an error here to an error in the service you called. Seeing
+the two errors connected in ForgeOps needs both services' projects linked there.
+
+```js
+import http from "node:http";
+import express from "express";
+import * as forgeOpsTracker from "@forge-ops/tracker";
+import { forgeOpsTrackerExpressMiddleware } from "@forge-ops/tracker/integrations/express";
+import { forgeOpsTrackerTracingExpressMiddleware } from "@forge-ops/tracker/integrations/tracing";
+
+forgeOpsTracker.init({
+  dsn: "https://<api_key>@getforgeops.net/api/v1/events",
+  propagateTraces: true, // default true; false never sends traceparent
+  // Default null: every host. A string matches that host and its subdomains on a dot boundary
+  // ("internal.example" matches "orders.internal.example", not "notinternal.example"); a RegExp is
+  // tested against the host, without the port.
+  tracePropagationTargets: ["internal.example", /^10\.0\./],
+});
+
+const app = express();
+app.use(forgeOpsTrackerTracingExpressMiddleware);
+
+app.post("/checkout", (req, res, next) => {
+  // Carries traceparent: 00-<this request's trace id>-<this call's span id>-01
+  const call = http.request("http://orders.internal.example/orders", { method: "POST" }, (response) => {
+    response.resume();
+    res.status(response.statusCode === 201 ? 201 : 502).end();
+  });
+  call.on("error", next);
+  call.end(JSON.stringify({ sku: "A1" }));
+});
+
+app.use(forgeOpsTrackerExpressMiddleware);
+app.listen(3000);
+```
+
 **Known gaps:** no database span capture (this SDK has no existing query/ORM instrumentation hook
 of any kind yet to extend) and no Redis span capture (no existing Redis hook or dependency exists
 here either); a database call or Redis call inside a traced request just won't show up as its own

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@forge-ops/tracker",
-  "version": "0.10.0",
+  "version": "0.11.0",
   "description": "ForgeOps error tracking client: captures unhandled exceptions (Express/Fastify integration, plus explicit capture anywhere else) and delivers them to ForgeOps over HTTP.",
   "type": "module",
   "main": "src/index.js",

package/src/configuration.js

@@ -116,6 +116,26 @@ export class Configuration {
    * gems/forge_ops_tracker's own Configuration#trace_capture_threshold_ms default. */
   traceCaptureThresholdMs = 1000;
 
+  /**
+   * Adds a W3C `traceparent` header (https://www.w3.org/TR/trace-context/) to every outbound
+   * http/https request made during a request the Express/Fastify tracing integration is handling
+   * (see httpTracing.js), so a service that reports to ForgeOps too (or anything else that
+   * understands the standard) continues this request's trace instead of starting its own. On by
+   * default, the same as gems/forge_ops_tracker's propagate_traces: the header also carries the
+   * trace id that links an error here to an error there, which is useful with or without spans.
+   * Never overwrites a traceparent the request already has.
+   */
+  propagateTraces = true;
+  /**
+   * Which hosts get that header. null (the default) means every host. Otherwise an array whose
+   * entries are either a host string, matching that host and its subdomains on a dot boundary
+   * ("example.com" matches "api.example.com", not "badexample.com"; a leading dot is ignored), or a
+   * RegExp tested against the host (no port). Narrow it for a third-party API that rejects headers
+   * it doesn't know, or that shouldn't learn this app's trace ids at all.
+   * @type {Array<string | RegExp> | null}
+   */
+  tracePropagationTargets = null;
+
   /** @returns {string | null} */
   apiKey() {
     const parsed = this.#parsedDsn();
@@ -203,6 +223,51 @@ export class Configuration {
     return uri.replace(/\/events$/, "/spans");
   }
 
+  /**
+   * Whether an outbound request to `host` should carry a traceparent header; see
+   * propagateTraces/tracePropagationTargets above. Case-insensitive, since hostnames are.
+   * @param {string} host
+   * @returns {boolean}
+   */
+  propagatesTraceTo(host) {
+    if (!this.propagateTraces) {
+      return false;
+    }
+    if (this.tracePropagationTargets === null || this.tracePropagationTargets === undefined) {
+      return true;
+    }
+    const normalized = String(host ?? "").toLowerCase();
+    return this.tracePropagationTargets.some((target) => {
+      if (target instanceof RegExp) {
+        target.lastIndex = 0; // a /g or /y pattern would otherwise carry state from the last test
+        return target.test(normalized);
+      }
+      const suffix = String(target).toLowerCase().replace(/^\./, "");
+      return suffix !== "" && (normalized === suffix || normalized.endsWith(`.${suffix}`));
+    });
+  }
+
+  /**
+   * Whether a request to `host`:`port` is one of this client's own deliveries to ForgeOps (same
+   * host and port as the DSN), so the outbound-HTTP wrapper never adds a traceparent to it. The
+   * Client itself uses fetch(), which that wrapper never sees, so this is a second guard, not the
+   * only one.
+   * @param {string} host
+   * @param {number | string | null | undefined} port
+   * @param {string} protocol "http:" or "https:"
+   * @returns {boolean}
+   */
+  isOwnHost(host, port, protocol) {
+    const parsed = this.#parsedDsn();
+    if (parsed === null || !host) {
+      return false;
+    }
+    const defaultPort = (scheme) => (scheme === "http:" ? "80" : scheme === "https:" ? "443" : "");
+    const ownPort = parsed.port || defaultPort(parsed.protocol);
+    const targetPort = port === null || port === undefined || port === "" ? defaultPort(protocol) : String(port);
+    return String(host).toLowerCase() === parsed.hostname.toLowerCase() && targetPort === ownPort;
+  }
+
   /** @returns {boolean} */
   isEnabled() {
     return Boolean(this.dsn) && this.apiKey() !== null && this.enabledEnvironments.has(this.environment);

package/src/eventBuilder.js

@@ -49,8 +49,14 @@ export class EventBuilder {
    * @param {Record<string, unknown>} [context]
    * @param {Record<string, unknown> | null} [user]
    * @param {Array<Record<string, unknown>>} [breadcrumbs]
+   * @param {{ transactionName?: string | null, endpoint?: string | null, traceId?: string | null }} [request]
+   *   Where the error happened (see RequestState in requestState.js): transactionName is the same
+   *   "GET /orders/:id" name the request's root span and performance samples use, endpoint the
+   *   human-readable route ("GET /orders/:id"), traceId the request's 32-character lowercase hex
+   *   W3C trace id, which is also what links this error to errors other services reported for the
+   *   same trace. All absent outside a request, and a missing one is left out of the payload.
    */
-  build(error, context = {}, user = null, breadcrumbs = []) {
+  build(error, context = {}, user = null, breadcrumbs = [], request = {}) {
     const payload = {
       exception_class: error?.name ?? "Error",
       message: error?.message ?? "",
@@ -69,13 +75,24 @@ export class EventBuilder {
     if (breadcrumbs && breadcrumbs.length > 0) {
       payload.breadcrumbs = [...breadcrumbs];
     }
+    if (request.transactionName) {
+      payload.transaction_name = request.transactionName;
+    }
+    if (request.endpoint) {
+      payload.endpoint = request.endpoint;
+    }
+    if (request.traceId) {
+      payload.trace_id = request.traceId;
+    }
     this.#attachSql(payload, error);
 
     return this.#configuration.scrubPii ? this.#scrub(payload) : payload;
   }
 
-  // exception_class/occurred_at/environment/release/server_name/user are
-  // left alone: structured fields this client or the host app sets
+  // exception_class/occurred_at/environment/release/server_name/user/
+  // transaction_name/endpoint/trace_id are left alone (endpoint is a
+  // declared route pattern, never the literal path, so there's no id or
+  // token in it to scrub): structured fields this client or the host app sets
   // deliberately, not free text an exception or its context could
   // accidentally spill sensitive data into. user specifically is a
   // deliberate exemption, not an oversight: the scrubber's own email

package/src/httpTracing.js

@@ -1,5 +1,6 @@
 import http from "node:http";
 import https from "node:https";
+import { generateSpanId, TRACEPARENT_HEADER } from "./traceParent.js";
 
 let installed = false;
 let originalHttpRequest = null;
@@ -20,19 +21,28 @@ let originalHttpsRequest = null;
  * recordSpan is passed in as a plain parameter rather than imported directly from index.js: this
  * module gets installed FROM index.js's own init(), so importing index.js back from here would
  * create a circular module dependency index.js has never needed before, for no real benefit over
- * just passing the one function actually used.
+ * just passing the functions actually used.
  *
- * @param {(name: string, kind: string, startedAt: Date, durationMs: number, data?: Record<string, unknown>) => void} recordSpan
+ * Also propagates the current request's trace to the service being called, as a W3C `traceparent`
+ * header (see traceParent.js, and Configuration#propagateTraces/tracePropagationTargets for turning
+ * it off or narrowing it to specific hosts). The header's parent id is this outbound call's own
+ * span id, generated before the call is made (the span itself can only be recorded afterward, once
+ * its duration is known) and then recorded with that exact id, so the downstream service's root
+ * span points at a span that really exists in this trace. traceparentFor decides whether there is
+ * a trace to continue at all (only inside a request) and returns null otherwise.
+ *
+ * @param {(name: string, kind: string, startedAt: Date, durationMs: number, data?: Record<string, unknown>, spanId?: string) => void} recordSpan
+ * @param {(host: string, port: number | string | null | undefined, protocol: string, spanId: string) => string | null} [traceparentFor]
  */
-export function installHttpTracing(recordSpan) {
+export function installHttpTracing(recordSpan, traceparentFor = () => null) {
   if (installed) {
     return;
   }
   installed = true;
   originalHttpRequest = http.request;
   originalHttpsRequest = https.request;
-  http.request = wrap(originalHttpRequest, recordSpan);
-  https.request = wrap(originalHttpsRequest, recordSpan);
+  http.request = wrap(originalHttpRequest, recordSpan, traceparentFor, "http:");
+  https.request = wrap(originalHttpsRequest, recordSpan, traceparentFor, "https:");
 }
 
 /** @internal test-only: restores http/https.request exactly as installHttpTracing found them. */
@@ -47,12 +57,14 @@ export function _uninstallHttpTracing() {
   originalHttpsRequest = null;
 }
 
-function wrap(originalRequest, recordSpan) {
+function wrap(originalRequest, recordSpan, traceparentFor, defaultProtocol) {
   return function patchedRequest(...args) {
+    const spanId = generateSpanId();
     const startedAt = new Date();
     const start = performance.now();
     const req = originalRequest.apply(this, args);
-    const { method, host } = describeRequest(args);
+    const { method, host, port, protocol } = describeRequest(args, defaultProtocol);
+    propagateTrace(req, () => traceparentFor(host, port, protocol, spanId));
 
     // "response" and "error" are mutually exclusive on a real ClientRequest, but guarded anyway:
     // recording twice for the same outbound call would double it up in the waterfall.
@@ -62,7 +74,7 @@ function wrap(originalRequest, recordSpan) {
         return;
       }
       finished = true;
-      recordSpan(`${method} ${host}`, "http", startedAt, performance.now() - start, status === undefined ? {} : { status });
+      recordSpan(`${method} ${host}`, "http", startedAt, performance.now() - start, status === undefined ? {} : { status }, spanId);
     };
 
     // No status at all means the request itself failed (DNS, connection refused, timeout) before
@@ -75,6 +87,31 @@ function wrap(originalRequest, recordSpan) {
   };
 }
 
+/**
+ * Adds the traceparent header to a request that was just created and hasn't sent its headers yet
+ * (the caller only writes or ends it after http.request returns). Leaves a traceparent the caller
+ * already set alone: whoever set it explicitly knows better than this client which trace the call
+ * belongs to. Skipped when headers were already flushed (a raw-array `headers` option sends them at
+ * construction). Never throws, since a failure to add a header must never stop the host app's own
+ * request from going out.
+ *
+ * @param {import("node:http").ClientRequest} req
+ * @param {() => string | null} traceparent
+ */
+function propagateTrace(req, traceparent) {
+  try {
+    if (req.headersSent || req.hasHeader(TRACEPARENT_HEADER)) {
+      return;
+    }
+    const value = traceparent();
+    if (value !== null) {
+      req.setHeader(TRACEPARENT_HEADER, value);
+    }
+  } catch {
+    // See this function's own comment: the request goes out without the header.
+  }
+}
+
 /**
  * Best-effort parse of whatever calling convention produced this request (a bare options object,
  * a URL string/object plus an optional options object, either form optionally followed by a
@@ -82,15 +119,21 @@ function wrap(originalRequest, recordSpan) {
  * path or query string a customer's own request carries (that could hold a customer's own id or
  * a secret), the same reasoning every other transaction_name/span name in this client already
  * follows.
+ *
+ * port/protocol are only ever used to recognize this client's own deliveries to ForgeOps (see
+ * Configuration#isOwnHost), never in a label.
  * @param {unknown[]} args
+ * @param {string} defaultProtocol
  */
-function describeRequest(args) {
+function describeRequest(args, defaultProtocol) {
   const [first, second] = args;
   const options = {};
 
   if (typeof first === "string" || first instanceof URL) {
     const url = typeof first === "string" ? new URL(first) : first;
     options.host = url.hostname;
+    options.port = url.port;
+    options.protocol = url.protocol;
     if (second && typeof second === "object") {
       Object.assign(options, second);
     }
@@ -100,5 +143,5 @@ function describeRequest(args) {
 
   const method = String(options.method ?? "GET").toUpperCase();
   const host = options.host ?? options.hostname ?? "unknown";
-  return { method, host };
+  return { method, host, port: options.port, protocol: options.protocol ?? defaultProtocol };
 }

package/src/index.js

@@ -8,8 +8,10 @@ import { installHttpTracing, _uninstallHttpTracing } from "./httpTracing.js";
 import { MetricBuffer } from "./metricBuffer.js";
 import { PerformanceFlusher } from "./performanceFlusher.js";
 import { Reporter } from "./reporter.js";
+import { requestStateFor } from "./requestState.js";
 import { SessionFlusher } from "./sessionFlusher.js";
 import { randomSpanId, SpanBuffer } from "./spanBuffer.js";
+import { buildTraceparent } from "./traceParent.js";
 
 export { Configuration };
 export { withSql } from "./sqlStatement.js";
@@ -78,6 +80,14 @@ const spanStorage = new AsyncLocalStorage();
 // SpanBuffer at all" as the real signal for "not inside a traced request").
 const spanParentStorage = new AsyncLocalStorage();
 
+// Request-scoped RequestState (trace id, remote parent span id, transaction name, endpoint, errored
+// flag; see requestState.js), published by the Express/Fastify integrations for the duration of a
+// request through _runWithRequestState below. Not gated on trackTracing: the trace id links an
+// error to errors in other services even when no spans are ever recorded. Unset outside a request,
+// where captureException, span tracing, and the outbound-HTTP wrapper all behave exactly as they
+// did before this existed.
+const requestStorage = new AsyncLocalStorage();
+
 function getConfiguration() {
   if (configuration === null) {
     configuration = new Configuration();
@@ -242,6 +252,64 @@ export function _runWithBreadcrumbTrail(callback) {
   return breadcrumbStorage.run(new BreadcrumbBuffer(config), callback);
 }
 
+/**
+ * Internal; called by the Express/Fastify integrations with their own request object and its
+ * incoming traceparent header's raw value. Returns the RequestState memoized on that request (see
+ * requestState.js), continuing the incoming trace or starting a fresh one, or null when the client
+ * isn't enabled at all, the same "skip entirely" gating every other automatic source applies.
+ *
+ * @param {object} request an Express req or a Fastify request
+ * @param {unknown} traceparent
+ * @returns {import("./requestState.js").RequestState | null}
+ */
+export function _requestStateFor(request, traceparent) {
+  if (!getConfiguration().isEnabled()) {
+    return null;
+  }
+  return requestStateFor(request, traceparent);
+}
+
+/**
+ * Internal; runs callback with `state` published as the current request's for its whole async
+ * call chain. Just calls callback when state is null, or already the current one (two integrations
+ * wrapping the same request), so it never nests a second identical context.
+ *
+ * @template T
+ * @param {import("./requestState.js").RequestState | null} state
+ * @param {() => T} callback
+ * @returns {T}
+ */
+export function _runWithRequestState(state, callback) {
+  if (!state || requestStorage.getStore() === state) {
+    return callback();
+  }
+  return requestStorage.run(state, callback);
+}
+
+/**
+ * Internal; called by the outbound-HTTP wrapper (httpTracing.js) just before a request to
+ * `host`:`port` goes out. Returns the traceparent header value naming `spanId` (that request's own
+ * span) as the parent, or null when this request shouldn't carry one: outside a request, with
+ * propagateTraces off or `host` not a propagation target, or for a delivery to ForgeOps itself.
+ *
+ * @param {string} host
+ * @param {number | string | null | undefined} port
+ * @param {string} protocol
+ * @param {string} spanId
+ * @returns {string | null}
+ */
+export function _traceparentFor(host, port, protocol, spanId) {
+  const state = requestStorage.getStore();
+  if (!state) {
+    return null;
+  }
+  const config = getConfiguration();
+  if (config.isOwnHost(host, port, protocol) || !config.propagatesTraceTo(host)) {
+    return null;
+  }
+  return buildTraceparent(state.traceId, spanId);
+}
+
 /**
  * Internal; called by the Express/Fastify tracing integration (integrations/tracing.js), never
  * by host app code directly. Gives callback's own async call chain a fresh SpanBuffer, with its
@@ -261,7 +329,10 @@ export function _runWithSpanTrace(callback) {
   if (!config.trackTracing || !config.isEnabled()) {
     return callback();
   }
-  const buffer = new SpanBuffer(config);
+  // The trace id and remote parent come from the request's RequestState (see _runWithRequestState
+  // above), so this trace and any error event from the same request agree on the trace.
+  const state = requestStorage.getStore();
+  const buffer = new SpanBuffer(config, state ? { traceId: state.traceId, parentSpanId: state.parentSpanId } : {});
   return spanStorage.run(buffer, () => spanParentStorage.run(buffer.rootSpanId, callback));
 }
 
@@ -285,7 +356,11 @@ export function _finishSpanTrace(name, startedAt, durationMs) {
     return;
   }
   buffer.record({ spanId: buffer.rootSpanId, name, kind: "controller", startedAt, durationMs, root: true });
-  if (buffer.isSlow(getConfiguration().traceCaptureThresholdMs)) {
+  // An errored request's trace is always sent too, however fast it was, since the waterfall of
+  // what led up to an error is exactly what an issue page wants to show next to it. "Errored" means
+  // captureException ran during the request (the Express/Fastify error integrations call it too).
+  const errored = requestStorage.getStore()?.errored === true;
+  if (buffer.isSlow(getConfiguration().traceCaptureThresholdMs) || (errored && buffer.rootRecorded)) {
     getSpanQueue().push({ trace_id: buffer.traceId, spans: buffer.spans });
   }
 }
@@ -303,15 +378,17 @@ export function _finishSpanTrace(name, startedAt, durationMs) {
  * @param {Date} startedAt
  * @param {number} durationMs
  * @param {Record<string, unknown>} [data]
+ * @param {string} [spanId] only passed by the outbound-HTTP wrapper, which has to pick it before
+ *   the call goes out so the traceparent header it sends names this exact span
  */
-export function _recordSpan(name, kind, startedAt, durationMs, data = {}) {
+export function _recordSpan(name, kind, startedAt, durationMs, data = {}, spanId = randomSpanId()) {
   const config = getConfiguration();
   const buffer = spanStorage.getStore();
   if (!config.trackTracing || !buffer) {
     return;
   }
   const parentSpanId = spanParentStorage.getStore() ?? buffer.rootSpanId;
-  buffer.record({ spanId: randomSpanId(), parentSpanId, name, kind, startedAt, durationMs, data });
+  buffer.record({ spanId, parentSpanId, name, kind, startedAt, durationMs, data });
 }
 
 /**
@@ -363,7 +440,7 @@ export function init(options = {}) {
   // gems/forge_ops_tracker's own Railtie always applies Net::HTTP.prepend(Timing) regardless of
   // config, checking configuration.trackTracing fresh on every actual outbound call instead (see
   // _recordSpan above), never at install time.
-  installHttpTracing(_recordSpan);
+  installHttpTracing(_recordSpan, _traceparentFor);
 
   return config;
 }
@@ -373,13 +450,20 @@ export function init(options = {}) {
  * established for this async call chain, if anything; pass one explicitly to override that for
  * this one report.
  *
+ * Inside a request the Express/Fastify integrations are handling, the event also carries the
+ * request's transaction_name, endpoint, and trace_id, and the request's trace is sent however fast
+ * it was.
+ *
  * @param {Error} error
  * @param {Record<string, unknown>} [context]
  * @param {Record<string, unknown> | null} [user]
  */
 export function captureException(error, context = {}, user = null) {
   const breadcrumbs = breadcrumbStorage.getStore()?.all() ?? [];
-  getReporter().report(error, context, user ?? userStorage.getStore()?.user ?? null, breadcrumbs);
+  const state = requestStorage.getStore();
+  state?.markErrored();
+  const request = state ? { transactionName: state.transactionName, endpoint: state.endpoint, traceId: state.traceId } : {};
+  getReporter().report(error, context, user ?? userStorage.getStore()?.user ?? null, breadcrumbs, request);
 }
 
 /**
@@ -565,6 +649,7 @@ export function _resetForTesting() {
   // sharing the same root context. disable() drops it and leaves the instance perfectly usable
   // again for the next run()/enterWith() call, confirmed directly, not assumed.
   breadcrumbStorage.disable();
+  requestStorage.disable();
   // Re-installed fresh on the very next init() call in whatever test runs next: without this,
   // http.request/https.request would stay wrapped by a closure over a *previous* test's own
   // _recordSpan reference forever (installHttpTracing's own guard only ever installs once per

package/src/integrations/express.js

@@ -1,4 +1,5 @@
-import { captureException, runWithUser } from "../index.js";
+import { _requestStateFor, _runWithRequestState, captureException, runWithUser } from "../index.js";
+import { describeExpressRoute } from "./tracing.js";
 
 /**
  * Express error-handling middleware. Register last, after all routes:
@@ -12,6 +13,12 @@ import { captureException, runWithUser } from "../index.js";
  * async route handlers to error-handling middleware automatically,
  * verified directly against a real async handler, not assumed. Only an
  * exception your own code catches and handles is invisible to this.
+ *
+ * The event carries the request's trace id, transaction name, and
+ * endpoint: the RequestState forgeOpsTrackerTracingExpressMiddleware
+ * already stashed on req when that's installed, or one created here from
+ * the request's own traceparent header when it isn't, so an unhandled
+ * error is linked across services either way.
  */
 export function forgeOpsTrackerExpressMiddleware(err, req, res, next) {
   // See integrations/sessionTracking.js's own comment: marks the request crashed before
@@ -20,10 +27,14 @@ export function forgeOpsTrackerExpressMiddleware(err, req, res, next) {
   // the time it checks.
   req._forgeOpsSessionCrashed = true;
 
-  captureException(err, {
-    path: req.path,
-    method: req.method,
-  });
+  const state = _requestStateFor(req, req.headers?.traceparent);
+  state?.describeWith(() => describeExpressRoute(req));
+  _runWithRequestState(state, () =>
+    captureException(err, {
+      path: req.path,
+      method: req.method,
+    }),
+  );
   next(err);
 }
 

package/src/integrations/fastify.js

@@ -1,4 +1,5 @@
-import { captureException } from "../index.js";
+import { _requestStateFor, _runWithRequestState, captureException } from "../index.js";
+import { fastifyRoutePattern } from "./tracing.js";
 
 /**
  * Called directly with your Fastify instance: not registered via
@@ -15,13 +16,25 @@ import { captureException } from "../index.js";
  * continues exactly as if this hook weren't registered. Only an
  * exception your own code catches and handles is invisible to this.
  *
+ * The event carries the request's trace id, transaction name, and
+ * endpoint: the RequestState registerForgeOpsTrackerTracing already
+ * created for this request when that's registered, or one created here
+ * from the request's own traceparent header when it isn't.
+ *
  * @param {import("fastify").FastifyInstance} fastify
  */
 export function registerForgeOpsTracker(fastify) {
   fastify.addHook("onError", async (request, reply, error) => {
-    captureException(error, {
-      path: request.url,
-      method: request.method,
-    });
+    const state = _requestStateFor(request, request.headers?.traceparent);
+    const route = fastifyRoutePattern(request);
+    if (state && route && state.transactionName === null) {
+      state.setRoute(`${request.method} ${route}`, `${request.method} ${route}`);
+    }
+    _runWithRequestState(state, () =>
+      captureException(error, {
+        path: request.url,
+        method: request.method,
+      }),
+    );
   });
 }

package/src/integrations/tracing.js

@@ -1,4 +1,23 @@
-import { _finishSpanTrace, _runWithSpanTrace } from "../index.js";
+import { _finishSpanTrace, _requestStateFor, _runWithRequestState, _runWithSpanTrace } from "../index.js";
+
+/**
+ * How an Express request is named once a route has matched: transactionName the same
+ * "<HTTP method> <route pattern>" the root span and performance samples use, endpoint the same
+ * route pattern. req.route.path is the pattern as declared on the router that matched it; for a
+ * router mounted with app.use("/api", router), that's the part declared on the router, without
+ * "/api": req.baseUrl is the literal matched mount path, which for a parameterized mount
+ * ("/users/:userId") would put a real id into a field that must never carry one. null before a
+ * route has matched (the event then leaves both fields out).
+ *
+ * @param {import("express").Request} req
+ */
+export function describeExpressRoute(req) {
+  if (!req.route) {
+    return null;
+  }
+  const name = `${req.method} ${req.route.path}`;
+  return { transactionName: name, endpoint: name };
+}
 
 /**
  * Express distributed-tracing middleware: wraps the rest of the request in _runWithSpanTrace
@@ -20,20 +39,33 @@ import { _finishSpanTrace, _runWithSpanTrace } from "../index.js";
  * actually finished and the root span's real total duration is known: nothing about a normal,
  * fast request costs a single byte over the wire, matching gems/forge_ops_tracker's own
  * Middleware::SpanTracing.
+ *
+ * Also publishes this request's RequestState (see requestState.js) whether or not trackTracing is
+ * on, so every request gets a trace id, continuing an incoming W3C traceparent header when there
+ * is one (its root span's parent is then the caller's span), and an error captured anywhere in
+ * the request carries it plus the matched route. Register it before your routes for that: an
+ * error thrown by a route registered ahead of it never sees this request's trace. Express only
+ * sets req.route once the matched handler starts, so the route is read lazily, whenever an error
+ * needs it (see RequestState#describeWith).
  */
 export function forgeOpsTrackerTracingExpressMiddleware(req, res, next) {
-  _runWithSpanTrace(() => {
-    const startedAt = new Date();
-    const start = performance.now();
+  const state = _requestStateFor(req, req.headers?.traceparent);
+  state?.describeWith(() => describeExpressRoute(req));
+
+  _runWithRequestState(state, () =>
+    _runWithSpanTrace(() => {
+      const startedAt = new Date();
+      const start = performance.now();
 
-    res.on("finish", () => {
-      const durationMs = performance.now() - start;
-      const transactionName = `${req.method} ${req.route?.path ?? req.path}`;
-      _finishSpanTrace(transactionName, startedAt, durationMs);
-    });
+      res.on("finish", () => {
+        const durationMs = performance.now() - start;
+        const transactionName = `${req.method} ${req.route?.path ?? req.path}`;
+        _finishSpanTrace(transactionName, startedAt, durationMs);
+      });
 
-    next();
-  });
+      next();
+    }),
+  );
 }
 
 /**
@@ -55,15 +87,29 @@ export function forgeOpsTrackerTracingExpressMiddleware(req, res, next) {
  * matched a route at all (a 404), the same "pattern first, literal path as the 404 fallback"
  * shape the Express integration above already takes.
  *
+ * Also publishes this request's RequestState (see requestState.js) whether or not trackTracing is
+ * on, the same as the Express middleware above: a trace id for every request, continuing an
+ * incoming W3C traceparent header. Fastify has already matched the route by onRequest, so the
+ * transaction name and endpoint are set right here, before any handler runs, and only from the
+ * matched pattern: a request that matched no route (a 404) leaves both out.
+ *
  * @param {import("fastify").FastifyInstance} fastify
  */
 export function registerForgeOpsTrackerTracing(fastify) {
   fastify.addHook("onRequest", (request, reply, done) => {
-    _runWithSpanTrace(() => {
-      request._forgeOpsSpanStartedAt = new Date();
-      request._forgeOpsSpanStart = performance.now();
-      done();
-    });
+    const state = _requestStateFor(request, request.headers?.traceparent);
+    const route = fastifyRoutePattern(request);
+    if (state && route) {
+      state.setRoute(`${request.method} ${route}`, `${request.method} ${route}`);
+    }
+
+    _runWithRequestState(state, () =>
+      _runWithSpanTrace(() => {
+        request._forgeOpsSpanStartedAt = new Date();
+        request._forgeOpsSpanStart = performance.now();
+        done();
+      }),
+    );
   });
 
   fastify.addHook("onResponse", async (request) => {
@@ -78,3 +124,14 @@ export function registerForgeOpsTrackerTracing(fastify) {
     _finishSpanTrace(transactionName, request._forgeOpsSpanStartedAt, durationMs);
   });
 }
+
+/**
+ * Fastify's matched route pattern ("/orders/:id", prefixes included), or undefined for a request
+ * that matched no route: request.routeOptions.url, falling back to the older request.routerPath.
+ *
+ * @param {import("fastify").FastifyRequest} request
+ * @returns {string | undefined}
+ */
+export function fastifyRoutePattern(request) {
+  return request.routeOptions?.url ?? request.routerPath ?? undefined;
+}

package/src/reporter.js

@@ -22,14 +22,15 @@ export class Reporter {
    * @param {Record<string, unknown>} [context]
    * @param {Record<string, unknown> | null} [user]
    * @param {Array<Record<string, unknown>>} [breadcrumbs]
+   * @param {{ transactionName?: string | null, endpoint?: string | null, traceId?: string | null }} [request] see EventBuilder#build
    */
-  report(error, context = {}, user = null, breadcrumbs = []) {
+  report(error, context = {}, user = null, breadcrumbs = [], request = {}) {
     try {
       if (!this.#configuration.isEnabled()) {
         return;
       }
 
-      const payload = this.#eventBuilder.build(error, context, user, breadcrumbs);
+      const payload = this.#eventBuilder.build(error, context, user, breadcrumbs, request);
       this.#deliveryQueue.push(payload);
     } catch (e) {
       this.#configuration.log(`[forge-ops-tracker] report failed: ${e.name}: ${e.message}`);

package/src/requestState.js

@@ -0,0 +1,117 @@
+import { generateTraceId, parseTraceparent } from "./traceParent.js";
+
+/**
+ * Everything this client knows about the current request that isn't already owned by one of the
+ * older, single-purpose AsyncLocalStorage instances (the user, the breadcrumb trail, the span
+ * buffer): which trace it belongs to, which remote span called it (if any), what it's called, and
+ * whether it errored. One instance per request, published by the Express/Fastify integrations
+ * (see _runWithRequestState in index.js) so captureException, the tracing integration, and the
+ * outbound-HTTP wrapper can all reach it. Ported from
+ * gems/forge_ops_tracker/lib/forge_ops_tracker/request_state.rb.
+ *
+ * The traceId exists for every request, not only when trackTracing is on: it's also what an error
+ * event carries so ForgeOps can link it to errors reported by other services taking part in the
+ * same trace, which works on any plan and with tracing off entirely.
+ *
+ * Also stashed on the framework's own request object (see requestStateFor), the Node equivalent of
+ * the Ruby gem memoizing it in the Rack env: an error-handling middleware that runs outside the
+ * tracing middleware's async context still finds the same instance there.
+ */
+export class RequestState {
+  traceId;
+  parentSpanId;
+  errored = false;
+  #describe;
+  #transactionName = null;
+  #endpoint = null;
+
+  /**
+   * @param {{ traceId?: string, parentSpanId?: string | null }} [ids]
+   */
+  constructor({ traceId = generateTraceId(), parentSpanId = null } = {}) {
+    this.traceId = traceId;
+    this.parentSpanId = parentSpanId;
+  }
+
+  /**
+   * Continues the caller's trace when the request arrived with a usable W3C traceparent header,
+   * remembering the caller's span as this request's remote parent; starts a fresh trace otherwise.
+   * @param {unknown} traceparent
+   */
+  static fromTraceparent(traceparent) {
+    const incoming = parseTraceparent(traceparent);
+    return incoming === null ? new RequestState() : new RequestState(incoming);
+  }
+
+  /**
+   * How to name this request, read lazily, each time an error needs it: Express only sets req.route
+   * once the matched route's own handler starts running, with no app-level hook in between, so the
+   * integration hands over a function that reads it then instead of a name it can't know yet at
+   * request start. Returns null (and the event leaves both fields out) until a route has matched.
+   * @param {() => { transactionName: string, endpoint: string } | null} describe
+   */
+  describeWith(describe) {
+    this.#describe = describe;
+  }
+
+  /**
+   * Sets both names outright, for a framework that already knows the matched route when the
+   * request starts (Fastify's onRequest).
+   * @param {string} transactionName
+   * @param {string} endpoint
+   */
+  setRoute(transactionName, endpoint) {
+    this.#transactionName = transactionName;
+    this.#endpoint = endpoint;
+  }
+
+  /** @returns {string | null} */
+  get transactionName() {
+    return this.#transactionName ?? this.#described()?.transactionName ?? null;
+  }
+
+  /** @returns {string | null} */
+  get endpoint() {
+    return this.#endpoint ?? this.#described()?.endpoint ?? null;
+  }
+
+  markErrored() {
+    this.errored = true;
+  }
+
+  #described() {
+    try {
+      return this.#describe?.() ?? null;
+    } catch {
+      return null;
+    }
+  }
+}
+
+const REQUEST_STATE_KEY = Symbol.for("forge-ops-tracker.requestState");
+
+/**
+ * The RequestState memoized on `request` (an Express req or a Fastify request), created from its
+ * traceparent header on first use: whichever integration reaches a request first creates it, and
+ * every other one gets the identical instance back, so they all agree on one trace id. A symbol
+ * key, not a plain property: invisible to the app's own enumeration and serialization of req.
+ *
+ * @param {object} request
+ * @param {unknown} traceparent the incoming traceparent header's raw value
+ * @returns {RequestState}
+ */
+export function requestStateFor(request, traceparent) {
+  if (request[REQUEST_STATE_KEY] === undefined) {
+    request[REQUEST_STATE_KEY] = RequestState.fromTraceparent(traceparent);
+  }
+  return request[REQUEST_STATE_KEY];
+}
+
+/**
+ * The RequestState already memoized on `request`, if any integration created one.
+ * @param {object} request
+ * @returns {RequestState | undefined}
+ */
+export function existingRequestState(request) {
+  return request?.[REQUEST_STATE_KEY];
+}

package/src/spanBuffer.js

@@ -1,15 +1,15 @@
-import { randomBytes } from "node:crypto";
+import { generateSpanId, generateTraceId } from "./traceParent.js";
 
 /**
- * A short, unique-enough identifier for one span: 8 random bytes as hex, matching
- * gems/forge_ops_tracker/lib/forge_ops_tracker/span_buffer.rb's own SecureRandom.hex(8). The
+ * A short, unique-enough identifier for one span: a W3C span id (16 lowercase hex characters,
+ * never all zeros; see traceParent.js), the same id a traceparent header's parent-id carries. The
  * server only ever needs these to be unique within one trace's own array (see
  * Api::V1::SpansController on the Rails side), never a real database id, so this is deliberately
  * cheap rather than a full UUID.
  * @returns {string}
  */
 export function randomSpanId() {
-  return randomBytes(8).toString("hex");
+  return generateSpanId();
 }
 
 /**
@@ -33,17 +33,31 @@ export class SpanBuffer {
   rootSpanId;
   spans = [];
   #rootDurationMs = null;
+  #remoteParentSpanId;
 
-  constructor(configuration) {
+  /**
+   * traceId/parentSpanId come from the request's own state (see _runWithRequestState in index.js)
+   * when there is one, never generated separately here, so a trace this sends and an error event
+   * from the same request always agree on which trace they belong to. parentSpanId is the calling
+   * service's span (from an incoming traceparent header), recorded as the root span's parent: the
+   * server treats a parent that isn't in this trace's own batch as remote, and nests this
+   * service's root under the caller's outgoing HTTP span.
+   *
+   * @param {import("./configuration.js").Configuration} configuration
+   * @param {{ traceId?: string, parentSpanId?: string | null }} [ids]
+   */
+  constructor(configuration, { traceId = generateTraceId(), parentSpanId = null } = {}) {
     this.#configuration = configuration;
-    this.traceId = randomBytes(16).toString("hex");
+    this.traceId = traceId;
     this.rootSpanId = randomSpanId();
+    this.#remoteParentSpanId = parentSpanId;
   }
 
   /**
    * root: true only for the one call that records the request's own root span: forces
-   * parent_span_id null explicitly, the same reasoning span_buffer.rb's own #record documents for
-   * why that can't just be read off of "whatever's currently open" for the root specifically.
+   * parent_span_id to the remote parent (null unless the request arrived with a traceparent
+   * header) explicitly, the same reasoning span_buffer.rb's own #record documents for why that
+   * can't just be read off of "whatever's currently open" for the root specifically.
    *
    * @param {{
    *   spanId: string,
@@ -59,7 +73,7 @@ export class SpanBuffer {
   record({ spanId, parentSpanId = null, name, kind, startedAt, durationMs, data = {}, root = false }) {
     this.spans.push({
       span_id: spanId,
-      parent_span_id: root ? null : parentSpanId,
+      parent_span_id: root ? this.#remoteParentSpanId : parentSpanId,
       name,
       kind,
       // Date#toISOString() already yields millisecond precision UTC ("...sssZ"), exactly the wire
@@ -76,6 +90,15 @@ export class SpanBuffer {
     }
   }
 
+  /**
+   * Whether the request's own root span has been recorded: an errored request's trace is only sent
+   * once it has, since spans with no root to hang from would render as a waterfall with no top.
+   * @returns {boolean}
+   */
+  get rootRecorded() {
+    return this.#rootDurationMs !== null;
+  }
+
   /**
    * null (never sent) until the root span has actually been recorded: a request whose own
    * tracing integration never got the chance to call back at all has no duration to compare

package/src/traceParent.js

@@ -0,0 +1,88 @@
+import { randomBytes } from "node:crypto";
+
+/**
+ * Reads and writes the W3C Trace Context `traceparent` header
+ * (https://www.w3.org/TR/trace-context/), the vendor-neutral format for carrying one trace across
+ * service boundaries: `00-<32 hex trace-id>-<16 hex parent-id>-<2 hex flags>`. Used in both
+ * directions: the Express/Fastify integrations parse an incoming one so a request continues the
+ * caller's trace instead of starting its own, and httpTracing.js builds an outgoing one so the next
+ * service along continues this request's. Ported from
+ * gems/forge_ops_tracker/lib/forge_ops_tracker/trace_parent.rb.
+ *
+ * Deliberately strict on the way in, the same posture the spec itself asks receivers to take: a
+ * malformed value, uppercase hex, the reserved version "ff", or an all-zero trace/parent id are all
+ * treated as "no usable header at all" (parseTraceparent returns null and the request starts a
+ * fresh trace), never half-trusted. A version this client doesn't know yet is still accepted as
+ * long as its first four fields have version 00's shape, which is exactly what the spec says a
+ * version-00 parser should do with a future version; version 00 itself must have exactly four
+ * fields.
+ */
+
+export const TRACEPARENT_HEADER = "traceparent";
+
+const PATTERN = /^([0-9a-f]{2})-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})(-.*)?$/s;
+const INVALID_TRACE_ID = "0".repeat(32);
+const INVALID_PARENT_ID = "0".repeat(16);
+
+// Always "01" (sampled) on the way out: this client decides whether to actually send a trace only
+// once the request is over (see _finishSpanTrace in index.js), long after this header has already
+// gone out on an outbound call, so there's no honest earlier answer to give than "this may be
+// recorded." A downstream service is free to make its own decision either way.
+const SAMPLED_FLAGS = "01";
+
+/**
+ * @param {unknown} value an incoming header's raw value
+ * @returns {{ traceId: string, parentSpanId: string } | null} null for anything that isn't a usable header
+ */
+export function parseTraceparent(value) {
+  if (typeof value !== "string") {
+    return null;
+  }
+  const match = PATTERN.exec(value.trim());
+  if (match === null) {
+    return null;
+  }
+  const [, version, traceId, parentSpanId, , rest] = match;
+  if (version === "ff" || (version === "00" && rest !== undefined)) {
+    return null;
+  }
+  if (traceId === INVALID_TRACE_ID || parentSpanId === INVALID_PARENT_ID) {
+    return null;
+  }
+  return { traceId, parentSpanId };
+}
+
+/**
+ * @param {string} traceId
+ * @param {string} spanId
+ * @returns {string}
+ */
+export function buildTraceparent(traceId, spanId) {
+  return `00-${traceId}-${spanId}-${SAMPLED_FLAGS}`;
+}
+
+/**
+ * 32 lowercase hex characters, the W3C trace-id format, never all zeros (the spec reserves that as
+ * invalid, and a receiver drops the whole header over it): vanishingly unlikely from 16 random
+ * bytes, but free to rule out.
+ * @returns {string}
+ */
+export function generateTraceId() {
+  return randomId(16);
+}
+
+/**
+ * 16 lowercase hex characters, the W3C parent-id (span id) format, never all zeros.
+ * @returns {string}
+ */
+export function generateSpanId() {
+  return randomId(8);
+}
+
+function randomId(bytes) {
+  let id = randomBytes(bytes).toString("hex");
+  while (/^0+$/.test(id)) {
+    id = randomBytes(bytes).toString("hex");
+  }
+  return id;
+}