@heystack/otel 0.9.1 → 0.10.0

package/README.md

@@ -22,7 +22,7 @@ HEYSTACK_API_KEY=sk_live_…
 | Next.js — **any** deploy target (Vercel/Node **and** Cloudflare/OpenNext) | `@heystack/otel/next` | `registerHeystack` in `instrumentation.ts`. Auto-detects Node vs Cloudflare workerd and picks the right exporter. No-op on Edge. |
 | Standalone Cloudflare Workers (hand-written `export default { fetch }`) | `@heystack/otel/workers` | `instrument()` wraps your handler. Fetch-based exporter, flushes via `ctx.waitUntil`. |
 | Node / Express / Fastify / NestJS (long-running server) | `@heystack/otel/node` | `initHeystack`: auto-instrumentations + graceful shutdown. |
-| Browser (SPA / any web frontend) | `@heystack/otel/web` | `instrumentWeb`: session replay + W3C `traceparent` on outgoing fetch. No-op on the server (SSR-safe). |
+| Browser (SPA / any web frontend) | `@heystack/otel/web` | `instrumentWeb`: session replay + opt-in browser distributed tracing (`tracing: true`) that emits CLIENT spans + propagates W3C `traceparent` (browser→API shows as one trace). No-op on the server (SSR-safe). |
 | Anywhere (pure helpers) | `@heystack/otel` | `buildExporterConfig`, types. No Node SDK loaded. |
 
 ## Node / Express / etc.
@@ -306,11 +306,16 @@ The default export still needs to be wrapped with `instrument()` (or `initHeysta
 
 For a browser frontend (any SPA / web app), `instrumentWeb` records **session replay** and injects a W3C `traceparent` header on outgoing `fetch` calls, so replays correlate with the backend traces they triggered. It is a **no-op on the server** (SSR-safe), so it's safe to call from code that also runs during server rendering.
 
+The rrweb recorder **ships inside this package** — there is nothing else to install. Uploads go cross-origin to the Heystack ingest endpoint and **work out of the box** (no CORS configuration on your side).
+
 ```ts
 import { instrumentWeb } from "@heystack/otel/web";
 
 const stop = await instrumentWeb({
-  apiKey: process.env.HEYSTACK_API_KEY, // same ingest key as the rest of the SDK
+  // A BROWSER-exposed ingest key — it ships to the client, like an analytics
+  // write key. Use a public env var (below), ideally a dedicated key you can
+  // rotate independently of your server-side key. NOT your server secret.
+  apiKey: import.meta.env.VITE_HEYSTACK_API_KEY, // Vite; Next.js: process.env.NEXT_PUBLIC_HEYSTACK_API_KEY
   service: "my-web-app",
 });
 
@@ -320,17 +325,36 @@ stop();
 
 `instrumentWeb` returns a `stop()` function that ends the recording session.
 
+**Where to call it.** It must run in the **browser**. In a Vite/CRA SPA, call it once in your client entry (`main.tsx`). In **Next.js (App Router)**, wrap it in a small `"use client"` component that calls it from a `useEffect` and mount that once in your root layout (server components can't call it). Recording is **server-gated**: nothing is captured until you enable replay for the app in the console, so it's safe to ship this before flipping the switch.
+
 ### Options
 
 | Option | Type | Notes |
 | --- | --- | --- |
-| `apiKey` | `string` | **Required.** The same Heystack ingest key used by the rest of the SDK. |
+| `apiKey` | `string` | **Required.** A Heystack ingest key, **exposed to the browser** (public env var). Prefer a dedicated key you can rotate — not your server-side secret. |
 | `service` | `string` | **Required.** The OTel service name (matches the app's service in the console). |
 | `userId` | `string?` | Optional app-supplied identifier stamped on the session. |
 | `endpoint` | `string?` | Optional ingest endpoint override (defaults to the Heystack ingest endpoint). |
 | `sampleRate` | `number?` | Optional **local** override for the recording sample rate (0–1). By default sampling is controlled from the console. |
 | `flushIntervalMs` | `number?` | How often buffered events are flushed (default 5000ms). |
 | `flushEveryEvents` | `number?` | Max buffered events before an early flush (default 200). |
+| `tracing` | `boolean?` | Opt in to **browser distributed tracing** (default off). Emits a real CLIENT span per outbound `fetch` and propagates W3C trace context, so browser→backend calls show as one connected trace + a service-map edge. Independent of replay. |
+| `traceSampleRate` | `number?` | Head sample rate for browser tracing (0–1, default 1 when `tracing` is on). Lower it to cap span volume/cost on busy apps. |
+
+### Browser distributed tracing
+
+By default `/web` only records session replay. Set `tracing: true` to also trace the browser: each outbound `fetch` becomes a CLIENT span, and the injected `traceparent` makes the downstream service's SERVER span its child — so a browser→API call renders as **one connected trace** and a **service-map edge** (`web → api`). This is separate from replay (it works even with replay off).
+
+```ts
+await instrumentWeb({
+  apiKey: import.meta.env.VITE_HEYSTACK_API_KEY,
+  service: "my-web-app",
+  tracing: true,
+  traceSampleRate: 0.25, // sample 25% of requests — tune for cost
+});
+```
+
+It's **cost-aware and safe by design**: off unless you opt in; head-sampled (an unsampled request still propagates `traceparent` with the sampled flag cleared, so the backend makes the same keep/drop decision — no orphaned server spans); and the exporter posts through the *original* `fetch`, never tracing its own upload (no self-export loop). Spans post to `/v1/traces` cross-origin with no CORS setup on your side.
 
 ### Sampling & masking come from the console
 
@@ -364,6 +388,7 @@ As belt-and-suspenders the exporter also drops any span whose HTTP target points
 
 ## Migration / versioning
 
+- **`0.9.2`** — **`/workers`: `instrument()` no longer breaks WebSocket / Durable Object upgrades.** The fetch wrapper rebuilt every response with `new Response(...)`, which **drops the `webSocket` property** on a `101 Switching Protocols` upgrade — so every WebSocket connection on an instrumented Worker failed *whenever tracing was active* (an API key set), while passing in dev/staging where `instrument()` is a passthrough. `instrument()` now detects an upgrade (`response.webSocket` present, or `status === 101`) and returns the **original response untouched** (span closed immediately; no `traceparent` header injected — headers on a 101 aren't delivered anyway). Fixes failed `wss://` connections behind `routeAgentRequest` / any `new WebSocketPair()` handler. No API change; upgrade and redeploy.
 - **`0.9.1`** — **`/workers`: binding spans no longer orphan without `nodejs_compat`.** On a workerd runtime without `nodejs_compat` (no `globalThis.AsyncLocalStorage`), the synchronous fallback context manager cannot carry the active span across an `await` — so a D1/KV/R2/Vectorize/AI/Queue/Service span created *after* an `await` in your handler was emitted as the root of its own single-span trace instead of a child of the request. `instrument()` now captures the request's SERVER context per-request and uses it as an explicit parent fallback when no span is active, so binding operations are always children of the request trace. `nodejs_compat` is still recommended for cross-`await` parenting of outbound-fetch CLIENT spans and manual `withSpan` spans. No API change; upgrade and redeploy.
 - **`0.9.0`** — **`/workers`: automatic LLM gen_ai enrichment for outbound API calls.** Outbound `fetch` calls to known LLM providers (OpenAI, Anthropic, Cloudflare AI Gateway, Google) automatically gain `gen_ai.*` OTel semantic-convention attributes on the CLIENT span — model, token counts, finish reason, response ID — with no extra code. New optional `WorkersConfig.ai` option: `captureContent: true` also captures prompt/completion text (off by default; **strongly recommended for AI-app RCA**), with `redact` for scrubbing and `maxContentChars` for length capping. The original request/response bodies are never consumed (request read only when already a string; response via `response.clone()`). Streaming responses skip response enrichment. No breaking changes.
 - **`0.8.0`** — **`/workers`: Workers AI, Queue producer, and Service binding instrumentation.** `instrumentBindings: true` now auto-wraps three additional binding types: `env.AI.run()` emits CLIENT spans with `gen_ai.system`, `gen_ai.request.model`, and `gen_ai.usage.input_tokens`/`output_tokens` (streaming results are never consumed); Queue `.send`/`.sendBatch` emit PRODUCER spans with `messaging.*` attributes including batch size; Service binding `.fetch` emits a CLIENT span and injects a W3C `traceparent` header into the outgoing request so calls to other Workers appear in the same distributed trace. `startSpan` factory now accepts an optional `SpanKind` for correct CLIENT/PRODUCER categorisation. No breaking changes.

package/dist/web.d.ts

@@ -65,11 +65,21 @@ export interface InstrumentWebOptions {
     flushIntervalMs?: number;
     /** Max buffered events before an early flush. */
     flushEveryEvents?: number;
+    /**
+     * Opt in to browser distributed tracing: emit a real CLIENT span per outbound
+     * fetch and propagate W3C trace context, so browser→backend calls show as one
+     * connected trace (and a service-map edge). Off by default — it adds span
+     * volume (backend cost). Independent of session replay.
+     */
+    tracing?: boolean;
+    /** Head sample rate for browser tracing (0–1, default 1 when `tracing` is on).
+     *  Lower it to control span volume/cost on high-traffic apps. */
+    traceSampleRate?: number;
 }
 /** Entry point: fetch config, decide sampling, start rrweb, stream chunks.
  *  Returns a stop() function. Safe to call in any browser; no-ops on the server. */
 export declare function instrumentWeb(opts: InstrumentWebOptions): Promise<() => void>;
-export declare function makeTraceparent(traceId: string, spanId: string): string;
+export declare function makeTraceparent(traceId: string, spanId: string, sampled?: boolean): string;
 /** Collects trace ids observed during a session (deduped, capped). */
 export declare class TraceIdCollector {
     private readonly cap;
@@ -81,4 +91,49 @@ export declare class TraceIdCollector {
 /** Patch window.fetch to inject traceparent on outgoing calls and record the
  *  trace id for correlation. Returns an unpatch function. */
 export declare function patchFetchForCorrelation(collector: TraceIdCollector): () => void;
+/** One recorded browser CLIENT span (an outbound fetch). */
+export interface BrowserClientSpan {
+    traceId: string;
+    spanId: string;
+    name: string;
+    startMs: number;
+    endMs: number;
+    method: string;
+    url: string;
+    /** HTTP response status; 0 for a network error / throw. */
+    statusCode: number;
+    error: boolean;
+}
+/** Build an OTLP/JSON ExportTraceServiceRequest for a batch of client spans. */
+export declare function buildTraceExport(service: string, spans: BrowserClientSpan[]): Record<string, unknown>;
+export interface TraceExporterOpts {
+    endpoint: string;
+    apiKey: string;
+    service: string;
+    /** MUST be the ORIGINAL fetch captured before patching, or the export POST
+     *  self-traces and loops (cost guardrail #1). */
+    fetchImpl: typeof fetch;
+    maxBatch?: number;
+}
+/** Buffers browser CLIENT spans and POSTs them as OTLP/JSON to /v1/traces. */
+export declare class BrowserTraceExporter {
+    private readonly o;
+    private buf;
+    constructor(o: TraceExporterOpts);
+    add(span: BrowserClientSpan): void;
+    flush(keepalive?: boolean): Promise<void>;
+}
+export interface TracingPatchOpts {
+    onSpan: (s: BrowserClientSpan) => void;
+    sampleRate: number;
+    /** Bare ingest hostname; calls to it are never traced (self-export loop guard). */
+    ingestHost: string;
+    /** Optional: also record the real trace id for replay↔trace correlation. */
+    collector?: TraceIdCollector;
+    rng?: () => number;
+}
+/** Patch window.fetch to emit a real CLIENT span per outbound call, inject that
+ *  span's W3C traceparent, and (head-sampled) hand the finished span to onSpan.
+ *  Returns an unpatch function. */
+export declare function patchFetchForTracing(o: TracingPatchOpts): () => void;
 export {};

package/dist/web.js

@@ -1,4 +1,8 @@
 import { DEFAULT_ENDPOINT } from "./core.js";
+// safeHostname is pure string logic (no runtime imports) — safe in the browser
+// bundle. Used to skip tracing our OWN telemetry POSTs (the self-export loop that
+// caused the June 2026 cost incident); see CLAUDE.md → "Cost guardrails".
+import { safeHostname } from "./self-span.js";
 /** Pure: decide once per session whether to record. rng defaults to Math.random. */
 export function shouldRecord(cfg, rng = Math.random) {
     if (!cfg.enabled)
@@ -93,10 +97,41 @@ export async function instrumentWeb(opts) {
     catch { /* offline / blocked - fall back to disabled */ }
     if (opts.sampleRate !== undefined)
         cfg = { ...cfg, sample_rate: opts.sampleRate };
-    // 2. Session-level sampling decision.
+    // Capture the ORIGINAL fetch before any patching — telemetry exporters MUST use
+    // it so their own POSTs to the ingest endpoint aren't traced/looped (guardrail #1).
+    const originalFetch = fetch.bind(globalThis);
+    const ingestHost = safeHostname(endpoint);
+    const traces = new TraceIdCollector();
+    // 2. Browser distributed tracing (opt-in) — INDEPENDENT of replay sampling. Emits
+    //    a real CLIENT span per outbound fetch + propagates W3C context to the backend.
+    let stopTracing = () => { };
+    let fetchPatched = false;
+    if (opts.tracing) {
+        const exporter = new BrowserTraceExporter({
+            endpoint, apiKey: opts.apiKey, service: opts.service, fetchImpl: originalFetch,
+        });
+        const unpatchTrace = patchFetchForTracing({
+            onSpan: (s) => exporter.add(s),
+            sampleRate: opts.traceSampleRate ?? 1,
+            ingestHost,
+            collector: traces, // real trace ids also tag the replay session (better correlation)
+        });
+        fetchPatched = true;
+        const traceFlush = setInterval(() => void exporter.flush(), opts.flushIntervalMs ?? DEFAULT_FLUSH_MS);
+        const onHideTrace = () => { if (document.visibilityState === "hidden")
+            void exporter.flush(true); };
+        document.addEventListener("visibilitychange", onHideTrace);
+        window.addEventListener("pagehide", () => void exporter.flush(true));
+        stopTracing = () => {
+            clearInterval(traceFlush);
+            document.removeEventListener("visibilitychange", onHideTrace);
+            unpatchTrace();
+            void exporter.flush(true);
+        };
+    }
+    // 3. Session replay — gated on the replay sampling decision (independent of tracing).
     if (!shouldRecord(cfg))
-        return () => { };
-    // 3. Start the recorder.
+        return () => { stopTracing(); };
     const { record } = await import("rrweb");
     // An element marked `data-hs-unmask` (or any descendant of one) is recorded in
     // cleartext; everything else is masked. rrweb 2.0.1's real opt-out hooks are
@@ -109,12 +144,11 @@ export async function instrumentWeb(opts) {
     const maskInputFn = (text, element) => reveal(text, element) ? text : "*".repeat(text.length);
     const maskTextFn = (text, element) => reveal(text, element) ? text : "*".repeat(text.length);
     const sessionId = crypto.randomUUID();
-    const transport = new ReplayTransport({ endpoint, apiKey: opts.apiKey, sessionId });
-    // Install the fetch patch AFTER the transport is constructed so the recorder's
-    // own upload POSTs (which captured the original fetch at construction time via
-    // fetch.bind(globalThis)) are not self-traced.
-    const traces = new TraceIdCollector();
-    const unpatch = patchFetchForCorrelation(traces);
+    const transport = new ReplayTransport({ endpoint, apiKey: opts.apiKey, fetchImpl: originalFetch, sessionId });
+    // Only patch fetch for replay correlation if tracing didn't already patch it
+    // (tracing's patch injects real context AND feeds `traces`). Avoids double-wrapping
+    // window.fetch. Uses the original fetch for uploads (self-span suppression).
+    const unpatch = fetchPatched ? () => { } : patchFetchForCorrelation(traces);
     let buffer = [];
     let errorCount = 0;
     const browser = navigator.userAgent;
@@ -159,6 +193,7 @@ export async function instrumentWeb(opts) {
         stopRecord?.();
         unpatch();
         flush(true);
+        stopTracing();
     };
 }
 function matchMediaDevice() {
@@ -171,8 +206,11 @@ function randHex(bytes) {
     crypto.getRandomValues(a);
     return Array.from(a, (b) => b.toString(16).padStart(2, "0")).join("");
 }
-export function makeTraceparent(traceId, spanId) {
-    return `00-${traceId}-${spanId}-01`;
+export function makeTraceparent(traceId, spanId, sampled = true) {
+    // The trace-flags byte's low bit is "sampled". When head sampling drops a
+    // request we still inject the context (00) so the downstream service makes the
+    // SAME keep/drop decision — coordinated sampling, no orphaned server spans.
+    return `00-${traceId}-${spanId}-${sampled ? "01" : "00"}`;
 }
 /** Collects trace ids observed during a session (deduped, capped). */
 export class TraceIdCollector {
@@ -209,3 +247,117 @@ export function patchFetchForCorrelation(collector) {
     });
     return () => { window.fetch = orig; };
 }
+const kvStr = (key, value) => ({ key, value: { stringValue: value } });
+const kvInt = (key, n) => ({ key, value: { intValue: String(n) } });
+const msToNano = (ms) => `${Math.trunc(ms)}000000`;
+function spanToOtlp(s) {
+    const attributes = [
+        kvStr("http.request.method", s.method),
+        kvStr("url.full", s.url),
+        kvStr("server.address", safeHostname(s.url)),
+    ];
+    if (s.statusCode > 0)
+        attributes.push(kvInt("http.response.status_code", s.statusCode));
+    return {
+        traceId: s.traceId,
+        spanId: s.spanId,
+        name: s.name,
+        kind: 3, // SPAN_KIND CLIENT
+        startTimeUnixNano: msToNano(s.startMs),
+        endTimeUnixNano: msToNano(s.endMs),
+        attributes,
+        status: { code: s.error ? 2 : 1 }, // STATUS_CODE ERROR : OK
+    };
+}
+/** Build an OTLP/JSON ExportTraceServiceRequest for a batch of client spans. */
+export function buildTraceExport(service, spans) {
+    return {
+        resourceSpans: [
+            {
+                resource: { attributes: [kvStr("service.name", service)] },
+                scopeSpans: [{ scope: { name: "@heystack/otel/web" }, spans: spans.map(spanToOtlp) }],
+            },
+        ],
+    };
+}
+/** Buffers browser CLIENT spans and POSTs them as OTLP/JSON to /v1/traces. */
+export class BrowserTraceExporter {
+    o;
+    buf = [];
+    constructor(o) {
+        this.o = o;
+    }
+    add(span) {
+        this.buf.push(span);
+        if (this.buf.length >= (this.o.maxBatch ?? 50))
+            void this.flush();
+    }
+    async flush(keepalive = false) {
+        if (this.buf.length === 0)
+            return;
+        const spans = this.buf;
+        this.buf = [];
+        const body = JSON.stringify(buildTraceExport(this.o.service, spans));
+        await this.o
+            .fetchImpl(`${this.o.endpoint}/v1/traces`, {
+            method: "POST",
+            keepalive,
+            headers: { authorization: `Bearer ${this.o.apiKey}`, "content-type": "application/json" },
+            body,
+        })
+            .catch(() => { });
+    }
+}
+function fetchUrl(input) {
+    if (typeof input === "string")
+        return input;
+    if (input instanceof URL)
+        return input.href;
+    return input.url ?? "";
+}
+function fetchMethod(input, init) {
+    const m = init?.method ?? (input instanceof Request ? input.method : undefined) ?? "GET";
+    return m.toUpperCase();
+}
+function pathOf(url) {
+    try {
+        return new URL(url).pathname;
+    }
+    catch {
+        return url;
+    }
+}
+/** Patch window.fetch to emit a real CLIENT span per outbound call, inject that
+ *  span's W3C traceparent, and (head-sampled) hand the finished span to onSpan.
+ *  Returns an unpatch function. */
+export function patchFetchForTracing(o) {
+    if (typeof window === "undefined" || !window.fetch)
+        return () => { };
+    const orig = window.fetch.bind(window);
+    const rng = o.rng ?? Math.random;
+    window.fetch = ((input, init) => {
+        const url = fetchUrl(input);
+        // Never trace our own telemetry POSTs (trace export, replay upload, config) —
+        // they'd create spans that export → re-trace → loop. The exporter already uses
+        // the original fetch; this is the belt-and-suspenders host-match guard.
+        if (o.ingestHost && safeHostname(url) === o.ingestHost)
+            return orig(input, init);
+        const sampled = rng() < o.sampleRate;
+        const traceId = randHex(16);
+        const spanId = randHex(8);
+        if (sampled && o.collector)
+            o.collector.add(traceId);
+        const headers = new Headers(init?.headers ?? (input instanceof Request ? input.headers : undefined));
+        if (!headers.has("traceparent"))
+            headers.set("traceparent", makeTraceparent(traceId, spanId, sampled));
+        const method = fetchMethod(input, init);
+        const startMs = Date.now();
+        const emit = (statusCode, error) => {
+            if (!sampled)
+                return;
+            o.onSpan({ traceId, spanId, name: `${method} ${pathOf(url)}`, startMs, endMs: Date.now(), method, url, statusCode, error });
+        };
+        return orig(input, { ...init, headers }).then((res) => { emit(res.status, res.status >= 400); return res; }, (err) => { emit(0, true); throw err; });
+    });
+    return () => { window.fetch = orig; };
+}

package/dist/workers.js

@@ -915,6 +915,22 @@ export function instrument(handler, config) {
             }
             try {
                 const response = await context.with(serverCtx, () => originalFetch(req, handlerEnv, ctx));
+                // WebSocket / connection upgrades: a 101 Switching Protocols response
+                // carries a live server-side `webSocket` that the Workers Runtime hands
+                // to the client ONLY if that exact Response object is returned. Rebuilding
+                // it (as the body branches below do) drops `.webSocket` — silently breaking
+                // every upgrade on an instrumented Worker whenever tracing is active. Pass
+                // the ORIGINAL response through untouched and close the span now (there is
+                // no body to time). We do NOT inject the `traceparent` header — headers on a
+                // 101 aren't delivered to the client anyway. `webSocket` is the reliable
+                // signal; `status === 101` is a belt-and-braces fallback.
+                if (response.webSocket || response.status === 101) {
+                    span.setAttribute("http.response.status_code", response.status);
+                    span.setStatus({ code: SpanStatusCode.UNSET });
+                    span.end();
+                    drain(provider, ctx);
+                    return response;
+                }
                 const headers = new Headers(response.headers);
                 headers.set("traceparent", traceparent);
                 appendExposeHeader(headers, "traceparent");

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@heystack/otel",
-  "version": "0.9.1",
+  "version": "0.10.0",
   "description": "Runtime-aware OpenTelemetry tracing that exports to Heystack (Node, Next.js, Workers).",
   "license": "MIT",
   "type": "module",