@heystack/otel 0.5.0 → 0.7.0

package/README.md

@@ -117,9 +117,40 @@ Set the key as a secret: `wrangler secret put HEYSTACK_API_KEY`.
 | `apiKey` | `string?` | Defaults to `env.HEYSTACK_API_KEY`. |
 | `getUser` | `(req: Request) => { id?, session?, requestId? } \| undefined` | Called per request. `id` → `enduser.id`, `session` → `session.id`, `requestId` → `http.request.id` (falls back to the `cf-ray` header). |
 | `instrumentBindings` | `boolean \| string[]` | `true` = auto child spans for all detected D1/KV/R2/Vectorize bindings; `string[]` = only the named bindings. Default `false`. |
+| `sampling` | `{ rate?: number } \| { remote: true }` | Head-sampling configuration. `{ rate }`: keep a deterministic fraction of fresh root traces (0–1; default `1` = keep all). `{ remote: true }`: fetch the rate from the Heystack config endpoint instead — lets you change it centrally without redeploying. Cold isolates keep all traffic until the first config fetch resolves; fails open if the config can't be reached. Parent-respecting in both modes: a request arriving with a sampled `traceparent` is always recorded. See [Head sampling](#head-sampling) below. |
 | `waitUntil` | `(p: Promise<unknown>) => void` | Override the isolate keep-alive hook; defaults to the auto-detected `ctx.waitUntil`. |
 | `endpoint` | `string?` | Override the ingest endpoint (advanced). |
 
+### Head sampling
+
+`sampling: { rate }` performs **head sampling** — the keep/drop decision is made at the start of each fresh root trace, before any export. Traces that are dropped never leave the worker (no egress, no ingest cost, no storage). Traces that are kept are exported in full.
+
+```ts
+export default instrument(worker, {
+  service: "my-api",
+  sampling: { rate: 0.2 }, // keep ~20% of traffic
+});
+```
+
+The sampler is **deterministic by trace ID**: the same trace always makes the same decision, so parent→child consistency is maintained when this worker is in a call chain sampled at the same rate by another service.
+
+**Tradeoff:** head sampling can't guarantee keeping error traces — the drop/keep decision is made before the response status is known. If you need every error captured, keep `rate: 1` (the default). Heystack's ingest-side keep rules (error and slow retention) apply among traces that _are_ exported; they can't rescue a trace that was dropped in the worker before export.
+
+The `rate` governs **fresh root traces only** (no inbound `traceparent`, or `traceparent` with `sampled=0`). A request arriving with a sampled inbound `traceparent` (`sampled=1`) is always recorded — the parent's decision is respected, so a distributed trace is never split mid-flight by the child's sample rate.
+
+### Remote sampling
+
+`sampling: { remote: true }` lets Heystack control the rate centrally — change it from the dashboard without redeploying:
+
+```ts
+export default instrument(worker, {
+  service: "my-api",
+  sampling: { remote: true },
+});
+```
+
+On startup the worker fetches its configured rate from the Heystack config endpoint. **Cold isolates keep all traffic until that first fetch resolves** (fails open — nothing is dropped while loading). If the config endpoint can't be reached, the worker keeps everything. The same parent-respecting rule applies: a request with an inbound sampled `traceparent` is always recorded regardless of the fetched rate.
+
 ### Automatic tracing
 
 `instrument()` traces the following automatically, with no additional config:
@@ -274,6 +305,8 @@ As belt-and-suspenders the exporter also drops any span whose HTTP target points
 
 ## Migration / versioning
 
+- **`0.7.0`** — **`/workers`: remote sampling (`sampling: { remote: true }`).** New `sampling` variant that fetches the head-sampling rate from the Heystack config endpoint at runtime, so you can change it from the console without redeploying. Cold isolates keep all traffic until the first config fetch resolves (fails open). If the config endpoint is unreachable, the worker keeps everything. Same parent-respecting rule as `sampling: { rate }`. No breaking changes; existing `sampling: { rate }` configs are unchanged.
+- **`0.6.0`** — **`/workers`: head sampling (`sampling: { rate }`).** New optional `WorkersConfig` field: `sampling.rate` (0–1, default `1`). Keeps a deterministic fraction of fresh root traces — the drop decision is made in the worker before export (no egress, no ingest cost). Parent-respecting: requests arriving with a sampled `traceparent` are always recorded. Consistent with server-side sampling (same trace-ID hash). No breaking changes; all new options are optional. See [Head sampling](#head-sampling).
 - **`0.5.0`** — **`/workers`: identity enrichment, binding tracing, outbound-fetch tracing, manual span helpers.** New `WorkersConfig` options: `getUser` (attach `enduser.id`/`session.id`/`http.request.id` per request from a synchronous callback), `instrumentBindings` (auto child spans for D1/KV/R2/Vectorize — `true` = all detected, or a `string[]` to select). Outbound `fetch` calls made inside a traced handler automatically get CLIENT child spans with `traceparent` injection (distributed tracing across services). New ergonomic exports from `/workers`: `withSpan(name, attrs?, fn)` runs a function inside a named child span (auto-parented, exceptions recorded, `span.end()` in `finally`); `addEvent(name, attrs?)` adds an event to the active span. No breaking changes; all new options are optional.
 - **`0.4.3`** — **feedback-loop guard extended to the Node path (cost fix).** The self-instrumentation loop the Workers path fixed in 0.3.1/0.3.2 was still live on plain Node / Next-on-Node: the OTLP-over-`http` exporter's `POST /v1/traces` was auto-instrumented and re-exported, so ~77% of ingested spans in production were the exporter tracing itself — needless ingest + ClickHouse compute. 0.4.3 ignores ingest-host requests in the HTTP/undici auto-instrumentations **and** filters self-spans at the exporter boundary (covers caller-supplied `instrumentations` too). The hostname matcher is now a shared module used by both `/node` and `/workers`. No API change. **Action: upgrade and redeploy any Node/Next-on-Node app** — it cuts ingested span volume sharply.
 - **`0.3.5`** — **type-constraint fix (Workers).** A Worker whose `queue` consumer is typed with a concrete message body — `queue(batch: MessageBatch<MyJob>, …)`, the normal case — failed to compile against `instrument()` in 0.3.4 (`TS2345`: `MessageBatch<unknown>` not assignable to `MessageBatch<MyJob>`). The `WorkerHandler` constraint declared its entrypoints as arrow properties, whose parameters are checked **contravariantly** under `strictFunctionTypes`, so a narrowed handler wasn't assignable. 0.3.5 declares them with **method syntax** (bivariant parameters) and widens the batch to `MessageBatch<any>`, mirroring Cloudflare's own `ExportedHandler` — a typed-queue Worker now type-checks with a bare `instrument(handler, cfg)`. Runtime behaviour unchanged. Also adds a **consumer type-check gate** (`pnpm consumer-typecheck`, run in `check` and `prepublishOnly`) that compiles a fully-typed Worker against the built `dist` through the public `exports` map and asserts `satisfies ExportedHandler<Env>` — the regression that escaped in 0.3.3/0.3.4 now fails the build before publish.

package/dist/workers-sampler.d.ts

@@ -0,0 +1,35 @@
+import { type Sampler, type SamplingResult } from "@opentelemetry/sdk-trace-base";
+import type { Context, Attributes, Link, SpanKind } from "@opentelemetry/api";
+export declare function fnv01(s: string): number;
+export declare function traceKept(traceId: string, rate: number): boolean;
+/** Test-only: set the remote rate directly. */
+export declare function __setRemoteRate(n: number): void;
+/** Test-only: read the current remote rate. */
+export declare function __getRemoteRate(): number;
+export declare class HeystackRatioSampler implements Sampler {
+    private readonly rate;
+    constructor(rate: number | (() => number));
+    shouldSample(_ctx: Context, traceId: string, _name: string, _kind: SpanKind, _attrs: Attributes, _links: Link[]): SamplingResult;
+    toString(): string;
+}
+/**
+ * Fetch the sampling rate from the Heystack ingest config endpoint and update
+ * the module-level `_remoteRate` ref. Call once per isolate (guarded in
+ * `workers.ts` via `_remoteSamplingKicked`). Uses the provided `fetchImpl`
+ * (the captured pre-patch fetch) so the GET is never re-entered by outbound
+ * fetch instrumentation and is wrapped in `suppressTracing` at the call site
+ * in `workers.ts` (belt-and-suspenders against self-tracing).
+ *
+ * Fail-open: any network failure, non-200, or parse error leaves `_remoteRate`
+ * at its current value (initially 1 = keep-all). Remote sampling must never
+ * drop telemetry due to a config-service outage.
+ */
+export declare function loadRemoteSamplingRate(opts: {
+    endpoint: string;
+    apiKey: string;
+    fetchImpl: typeof fetch;
+}): Promise<void>;
+export declare function makeSampler(sampling?: {
+    rate?: number;
+    remote?: boolean;
+}): Sampler;

package/dist/workers-sampler.js

@@ -0,0 +1,89 @@
+import { SamplingDecision, ParentBasedSampler, AlwaysOnSampler, } from "@opentelemetry/sdk-trace-base";
+// FNV-1a 32-bit → [0,1). MUST stay byte-identical to apps/ingest/src/sampling.ts
+// so the SDK and ingest agree on which traces to keep.
+export function fnv01(s) {
+    let h = 0x811c9dc5;
+    for (let i = 0; i < s.length; i++) {
+        h ^= s.charCodeAt(i);
+        h = Math.imul(h, 0x01000193);
+    }
+    return (h >>> 0) / 0x100000000;
+}
+export function traceKept(traceId, rate) {
+    if (rate >= 1)
+        return true;
+    if (rate <= 0)
+        return false;
+    return fnv01(traceId) < rate;
+}
+// Module-level mutable ref for the remote-loaded rate. Starts at 1 (keep-all)
+// so cold-isolate requests are fully preserved until the config fetch resolves.
+const _remoteRate = { value: 1 };
+/** Test-only: set the remote rate directly. */
+export function __setRemoteRate(n) {
+    _remoteRate.value = n;
+}
+/** Test-only: read the current remote rate. */
+export function __getRemoteRate() {
+    return _remoteRate.value;
+}
+export class HeystackRatioSampler {
+    rate;
+    constructor(rate) {
+        this.rate = rate;
+    }
+    shouldSample(_ctx, traceId, _name, _kind, _attrs, _links) {
+        const rate = typeof this.rate === "function" ? this.rate() : this.rate;
+        return {
+            decision: traceKept(traceId, rate)
+                ? SamplingDecision.RECORD_AND_SAMPLED
+                : SamplingDecision.NOT_RECORD,
+        };
+    }
+    toString() {
+        const r = typeof this.rate === "function" ? this.rate() : this.rate;
+        return `HeystackRatioSampler{${r}}`;
+    }
+}
+/**
+ * Fetch the sampling rate from the Heystack ingest config endpoint and update
+ * the module-level `_remoteRate` ref. Call once per isolate (guarded in
+ * `workers.ts` via `_remoteSamplingKicked`). Uses the provided `fetchImpl`
+ * (the captured pre-patch fetch) so the GET is never re-entered by outbound
+ * fetch instrumentation and is wrapped in `suppressTracing` at the call site
+ * in `workers.ts` (belt-and-suspenders against self-tracing).
+ *
+ * Fail-open: any network failure, non-200, or parse error leaves `_remoteRate`
+ * at its current value (initially 1 = keep-all). Remote sampling must never
+ * drop telemetry due to a config-service outage.
+ */
+export async function loadRemoteSamplingRate(opts) {
+    try {
+        const url = `${opts.endpoint.replace(/\/+$/, "")}/v1/sampling/config`;
+        const res = await opts.fetchImpl(url, {
+            headers: { Authorization: `Bearer ${opts.apiKey}` },
+        });
+        if (!res.ok)
+            return; // fail open — keep current rate
+        const cfg = (await res.json());
+        const r = Number(cfg?.trace_sample_rate);
+        if (Number.isFinite(r) && r >= 0 && r <= 1)
+            _remoteRate.value = r;
+    }
+    catch {
+        /* fail open: leave rate at keep-all */
+    }
+}
+export function makeSampler(sampling) {
+    if (sampling?.remote) {
+        // Dynamic rate: reads from the per-isolate ref that loadRemoteSamplingRate sets.
+        // Starts at 1 (keep-all) until the config fetch resolves on the first request.
+        return new ParentBasedSampler({ root: new HeystackRatioSampler(() => _remoteRate.value) });
+    }
+    const rate = sampling?.rate;
+    if (rate === undefined || rate >= 1)
+        return new AlwaysOnSampler();
+    // ParentBased: a sampled/dropped PARENT (local or remote via traceparent) wins,
+    // so a distributed trace samples consistently; only the root uses the ratio.
+    return new ParentBasedSampler({ root: new HeystackRatioSampler(rate) });
+}

package/dist/workers.d.ts

@@ -159,6 +159,25 @@ export interface WorkersConfig {
      * Defaults to `false` (no binding tracing). Consumed by a later task.
      */
     instrumentBindings?: boolean | string[];
+    /**
+     * Head-sampling configuration. When omitted (or `rate` >= 1), all traces are
+     * kept. When `rate` < 1, a deterministic FNV-1a hash of the trace ID decides
+     * keep/drop at the root span — consistent with the ingest-side sampler so the
+     * SDK and backend agree. Error / slow-span retention is an ingest concern.
+     *
+     * Note: head sampling is parent-respecting, so an incoming request carrying a
+     * sampled `traceparent` is still recorded even at `rate: 0` (it is not an
+     * absolute kill-switch; it governs only fresh/root traces).
+     *
+     * When `remote: true`, the sampling rate is fetched once per isolate from
+     * `{endpoint}/v1/sampling/config` on the first request. Cold-isolate spans
+     * are kept (rate=1) until the fetch resolves. On any failure the rate stays
+     * at keep-all (fail-open). Incompatible with inline `rate` (remote wins).
+     */
+    sampling?: {
+        rate?: number;
+        remote?: boolean;
+    };
 }
 /**
  * A `BasicTracerProvider` with the underlying `HeystackSpanExporter` attached so
@@ -178,7 +197,11 @@ export type HeystackTracerProvider = BasicTracerProvider & {
  * must drain the exporter — `await provider.heystackExporter.forceFlush()` (or
  * `flushHeystack()`), not just `provider.forceFlush()`.
  */
-export declare function createTracerProvider(config: HeystackOptions): HeystackTracerProvider;
+export declare function createTracerProvider(config: HeystackOptions & {
+    sampling?: {
+        rate?: number;
+    };
+}): HeystackTracerProvider;
 /**
  * An AsyncLocalStorage-backed ContextManager for the /workers entry. When the
  * runtime exposes `globalThis.AsyncLocalStorage` (workerd and Node do), this

package/dist/workers.js

@@ -13,9 +13,10 @@ import { ROOT_CONTEXT } from "@opentelemetry/api";
 import { Resource } from "@opentelemetry/resources";
 import { BasicTracerProvider, SimpleSpanProcessor, } from "@opentelemetry/sdk-trace-base";
 import { ATTR_SERVICE_NAME } from "@opentelemetry/semantic-conventions";
-import { buildExporterConfig } from "./core.js";
+import { buildExporterConfig, DEFAULT_ENDPOINT } from "./core.js";
 import { isSelfSpanAttrs, safeHostname } from "./self-span.js";
 import { instrumentEnv } from "./workers-bindings.js";
+import { makeSampler, loadRemoteSamplingRate } from "./workers-sampler.js";
 // `ExportResult` / `ExportResultCode` mirror `@opentelemetry/core`. We define
 // them inline (structurally identical) rather than import them: core is only a
 // transitive dep of sdk-trace-base and isn't reliably resolvable, and keeping it
@@ -523,6 +524,7 @@ export function createTracerProvider(config) {
     const provider = new BasicTracerProvider({
         resource: new Resource({ [ATTR_SERVICE_NAME]: config.service }),
         spanProcessors: [new SimpleSpanProcessor(exporter)],
+        sampler: makeSampler(config.sampling),
     });
     // Attach the exporter so flush paths can await its in-flight fetches.
     return Object.assign(provider, { heystackExporter: exporter });
@@ -694,7 +696,10 @@ export async function flushHeystack() {
 /** Reset the singleton global provider. Internal/testing helper. */
 export function __resetProvider() {
     _provider = null;
+    _remoteSamplingKicked = false;
 }
+/** Guard: the remote sampling config GET fires at most once per isolate. */
+let _remoteSamplingKicked = false;
 let warnedNoKey = false;
 function warnOnceNoKey() {
     if (warnedNoKey)
@@ -723,6 +728,7 @@ export function instrument(handler, config) {
             service: config.service,
             endpoint: config.endpoint,
             waitUntil: config.waitUntil,
+            sampling: config.sampling,
         });
         return { provider, tracer: trace.getTracer("heystack") };
     };
@@ -760,6 +766,22 @@ export function instrument(handler, config) {
             if (!s)
                 return originalFetch(req, env, ctx);
             const { provider, tracer } = s;
+            // Once per isolate: kick off the remote sampling config GET so the rate
+            // is available for subsequent requests without a redeploy. Uses the
+            // captured pre-patch fetch under suppressTracing — never self-traced,
+            // never looped. Fail-open: any error leaves the rate at 1 (keep-all).
+            if (config.sampling?.remote && !_remoteSamplingKicked) {
+                _remoteSamplingKicked = true;
+                const resolvedKey = config.apiKey ?? env?.HEYSTACK_API_KEY;
+                if (resolvedKey) {
+                    const ep = config.endpoint ?? DEFAULT_ENDPOINT;
+                    ctx.waitUntil(context.with(suppressTracing(context.active()), () => loadRemoteSamplingRate({
+                        endpoint: ep,
+                        apiKey: resolvedKey,
+                        fetchImpl: _originalFetch ?? fetch,
+                    })));
+                }
+            }
             const url = new URL(req.url);
             // FR5: continue an inbound W3C traceparent so tap→server is one trace.
             const parent = parseTraceparent(req.headers.get("traceparent"));

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@heystack/otel",
-  "version": "0.5.0",
+  "version": "0.7.0",
   "description": "Runtime-aware OpenTelemetry tracing that exports to Heystack (Node, Next.js, Workers).",
   "license": "MIT",
   "type": "module",