@heystack/otel 0.6.0 → 0.7.0

package/README.md

@@ -117,7 +117,7 @@ Set the key as a secret: `wrangler secret put HEYSTACK_API_KEY`.
 | `apiKey` | `string?` | Defaults to `env.HEYSTACK_API_KEY`. |
 | `getUser` | `(req: Request) => { id?, session?, requestId? } \| undefined` | Called per request. `id` → `enduser.id`, `session` → `session.id`, `requestId` → `http.request.id` (falls back to the `cf-ray` header). |
 | `instrumentBindings` | `boolean \| string[]` | `true` = auto child spans for all detected D1/KV/R2/Vectorize bindings; `string[]` = only the named bindings. Default `false`. |
-| `sampling` | `{ rate?: number }` | Head-sampling rate 0–1. `1` (default) keeps everything. `0.2` keeps ~20% of fresh root traces. Parent-respecting: a request that arrives with an inbound sampled `traceparent` is always recorded regardless of the rate. See [Head sampling](#head-sampling) below. |
+| `sampling` | `{ rate?: number } \| { remote: true }` | Head-sampling configuration. `{ rate }`: keep a deterministic fraction of fresh root traces (0–1; default `1` = keep all). `{ remote: true }`: fetch the rate from the Heystack config endpoint instead — lets you change it centrally without redeploying. Cold isolates keep all traffic until the first config fetch resolves; fails open if the config can't be reached. Parent-respecting in both modes: a request arriving with a sampled `traceparent` is always recorded. See [Head sampling](#head-sampling) below. |
 | `waitUntil` | `(p: Promise<unknown>) => void` | Override the isolate keep-alive hook; defaults to the auto-detected `ctx.waitUntil`. |
 | `endpoint` | `string?` | Override the ingest endpoint (advanced). |
 
@@ -138,6 +138,19 @@ The sampler is **deterministic by trace ID**: the same trace always makes the sa
 
 The `rate` governs **fresh root traces only** (no inbound `traceparent`, or `traceparent` with `sampled=0`). A request arriving with a sampled inbound `traceparent` (`sampled=1`) is always recorded — the parent's decision is respected, so a distributed trace is never split mid-flight by the child's sample rate.
 
+### Remote sampling
+
+`sampling: { remote: true }` lets Heystack control the rate centrally — change it from the dashboard without redeploying:
+
+```ts
+export default instrument(worker, {
+  service: "my-api",
+  sampling: { remote: true },
+});
+```
+
+On startup the worker fetches its configured rate from the Heystack config endpoint. **Cold isolates keep all traffic until that first fetch resolves** (fails open — nothing is dropped while loading). If the config endpoint can't be reached, the worker keeps everything. The same parent-respecting rule applies: a request with an inbound sampled `traceparent` is always recorded regardless of the fetched rate.
+
 ### Automatic tracing
 
 `instrument()` traces the following automatically, with no additional config:
@@ -292,6 +305,7 @@ As belt-and-suspenders the exporter also drops any span whose HTTP target points
 
 ## Migration / versioning
 
+- **`0.7.0`** — **`/workers`: remote sampling (`sampling: { remote: true }`).** New `sampling` variant that fetches the head-sampling rate from the Heystack config endpoint at runtime, so you can change it from the console without redeploying. Cold isolates keep all traffic until the first config fetch resolves (fails open). If the config endpoint is unreachable, the worker keeps everything. Same parent-respecting rule as `sampling: { rate }`. No breaking changes; existing `sampling: { rate }` configs are unchanged.
 - **`0.6.0`** — **`/workers`: head sampling (`sampling: { rate }`).** New optional `WorkersConfig` field: `sampling.rate` (0–1, default `1`). Keeps a deterministic fraction of fresh root traces — the drop decision is made in the worker before export (no egress, no ingest cost). Parent-respecting: requests arriving with a sampled `traceparent` are always recorded. Consistent with server-side sampling (same trace-ID hash). No breaking changes; all new options are optional. See [Head sampling](#head-sampling).
 - **`0.5.0`** — **`/workers`: identity enrichment, binding tracing, outbound-fetch tracing, manual span helpers.** New `WorkersConfig` options: `getUser` (attach `enduser.id`/`session.id`/`http.request.id` per request from a synchronous callback), `instrumentBindings` (auto child spans for D1/KV/R2/Vectorize — `true` = all detected, or a `string[]` to select). Outbound `fetch` calls made inside a traced handler automatically get CLIENT child spans with `traceparent` injection (distributed tracing across services). New ergonomic exports from `/workers`: `withSpan(name, attrs?, fn)` runs a function inside a named child span (auto-parented, exceptions recorded, `span.end()` in `finally`); `addEvent(name, attrs?)` adds an event to the active span. No breaking changes; all new options are optional.
 - **`0.4.3`** — **feedback-loop guard extended to the Node path (cost fix).** The self-instrumentation loop the Workers path fixed in 0.3.1/0.3.2 was still live on plain Node / Next-on-Node: the OTLP-over-`http` exporter's `POST /v1/traces` was auto-instrumented and re-exported, so ~77% of ingested spans in production were the exporter tracing itself — needless ingest + ClickHouse compute. 0.4.3 ignores ingest-host requests in the HTTP/undici auto-instrumentations **and** filters self-spans at the exporter boundary (covers caller-supplied `instrumentations` too). The hostname matcher is now a shared module used by both `/node` and `/workers`. No API change. **Action: upgrade and redeploy any Node/Next-on-Node app** — it cuts ingested span volume sharply.

package/dist/workers-sampler.d.ts

@@ -2,12 +2,34 @@ import { type Sampler, type SamplingResult } from "@opentelemetry/sdk-trace-base
 import type { Context, Attributes, Link, SpanKind } from "@opentelemetry/api";
 export declare function fnv01(s: string): number;
 export declare function traceKept(traceId: string, rate: number): boolean;
+/** Test-only: set the remote rate directly. */
+export declare function __setRemoteRate(n: number): void;
+/** Test-only: read the current remote rate. */
+export declare function __getRemoteRate(): number;
 export declare class HeystackRatioSampler implements Sampler {
     private readonly rate;
-    constructor(rate: number);
+    constructor(rate: number | (() => number));
     shouldSample(_ctx: Context, traceId: string, _name: string, _kind: SpanKind, _attrs: Attributes, _links: Link[]): SamplingResult;
     toString(): string;
 }
+/**
+ * Fetch the sampling rate from the Heystack ingest config endpoint and update
+ * the module-level `_remoteRate` ref. Call once per isolate (guarded in
+ * `workers.ts` via `_remoteSamplingKicked`). Uses the provided `fetchImpl`
+ * (the captured pre-patch fetch) so the GET is never re-entered by outbound
+ * fetch instrumentation and is wrapped in `suppressTracing` at the call site
+ * in `workers.ts` (belt-and-suspenders against self-tracing).
+ *
+ * Fail-open: any network failure, non-200, or parse error leaves `_remoteRate`
+ * at its current value (initially 1 = keep-all). Remote sampling must never
+ * drop telemetry due to a config-service outage.
+ */
+export declare function loadRemoteSamplingRate(opts: {
+    endpoint: string;
+    apiKey: string;
+    fetchImpl: typeof fetch;
+}): Promise<void>;
 export declare function makeSampler(sampling?: {
     rate?: number;
+    remote?: boolean;
 }): Sampler;

package/dist/workers-sampler.js

@@ -16,23 +16,70 @@ export function traceKept(traceId, rate) {
         return false;
     return fnv01(traceId) < rate;
 }
+// Module-level mutable ref for the remote-loaded rate. Starts at 1 (keep-all)
+// so cold-isolate requests are fully preserved until the config fetch resolves.
+const _remoteRate = { value: 1 };
+/** Test-only: set the remote rate directly. */
+export function __setRemoteRate(n) {
+    _remoteRate.value = n;
+}
+/** Test-only: read the current remote rate. */
+export function __getRemoteRate() {
+    return _remoteRate.value;
+}
 export class HeystackRatioSampler {
     rate;
     constructor(rate) {
         this.rate = rate;
     }
     shouldSample(_ctx, traceId, _name, _kind, _attrs, _links) {
+        const rate = typeof this.rate === "function" ? this.rate() : this.rate;
         return {
-            decision: traceKept(traceId, this.rate)
+            decision: traceKept(traceId, rate)
                 ? SamplingDecision.RECORD_AND_SAMPLED
                 : SamplingDecision.NOT_RECORD,
         };
     }
     toString() {
-        return `HeystackRatioSampler{${this.rate}}`;
+        const r = typeof this.rate === "function" ? this.rate() : this.rate;
+        return `HeystackRatioSampler{${r}}`;
+    }
+}
+/**
+ * Fetch the sampling rate from the Heystack ingest config endpoint and update
+ * the module-level `_remoteRate` ref. Call once per isolate (guarded in
+ * `workers.ts` via `_remoteSamplingKicked`). Uses the provided `fetchImpl`
+ * (the captured pre-patch fetch) so the GET is never re-entered by outbound
+ * fetch instrumentation and is wrapped in `suppressTracing` at the call site
+ * in `workers.ts` (belt-and-suspenders against self-tracing).
+ *
+ * Fail-open: any network failure, non-200, or parse error leaves `_remoteRate`
+ * at its current value (initially 1 = keep-all). Remote sampling must never
+ * drop telemetry due to a config-service outage.
+ */
+export async function loadRemoteSamplingRate(opts) {
+    try {
+        const url = `${opts.endpoint.replace(/\/+$/, "")}/v1/sampling/config`;
+        const res = await opts.fetchImpl(url, {
+            headers: { Authorization: `Bearer ${opts.apiKey}` },
+        });
+        if (!res.ok)
+            return; // fail open — keep current rate
+        const cfg = (await res.json());
+        const r = Number(cfg?.trace_sample_rate);
+        if (Number.isFinite(r) && r >= 0 && r <= 1)
+            _remoteRate.value = r;
+    }
+    catch {
+        /* fail open: leave rate at keep-all */
     }
 }
 export function makeSampler(sampling) {
+    if (sampling?.remote) {
+        // Dynamic rate: reads from the per-isolate ref that loadRemoteSamplingRate sets.
+        // Starts at 1 (keep-all) until the config fetch resolves on the first request.
+        return new ParentBasedSampler({ root: new HeystackRatioSampler(() => _remoteRate.value) });
+    }
     const rate = sampling?.rate;
     if (rate === undefined || rate >= 1)
         return new AlwaysOnSampler();

package/dist/workers.d.ts

@@ -168,9 +168,15 @@ export interface WorkersConfig {
      * Note: head sampling is parent-respecting, so an incoming request carrying a
      * sampled `traceparent` is still recorded even at `rate: 0` (it is not an
      * absolute kill-switch; it governs only fresh/root traces).
+     *
+     * When `remote: true`, the sampling rate is fetched once per isolate from
+     * `{endpoint}/v1/sampling/config` on the first request. Cold-isolate spans
+     * are kept (rate=1) until the fetch resolves. On any failure the rate stays
+     * at keep-all (fail-open). Incompatible with inline `rate` (remote wins).
      */
     sampling?: {
         rate?: number;
+        remote?: boolean;
     };
 }
 /**

package/dist/workers.js

@@ -13,10 +13,10 @@ import { ROOT_CONTEXT } from "@opentelemetry/api";
 import { Resource } from "@opentelemetry/resources";
 import { BasicTracerProvider, SimpleSpanProcessor, } from "@opentelemetry/sdk-trace-base";
 import { ATTR_SERVICE_NAME } from "@opentelemetry/semantic-conventions";
-import { buildExporterConfig } from "./core.js";
+import { buildExporterConfig, DEFAULT_ENDPOINT } from "./core.js";
 import { isSelfSpanAttrs, safeHostname } from "./self-span.js";
 import { instrumentEnv } from "./workers-bindings.js";
-import { makeSampler } from "./workers-sampler.js";
+import { makeSampler, loadRemoteSamplingRate } from "./workers-sampler.js";
 // `ExportResult` / `ExportResultCode` mirror `@opentelemetry/core`. We define
 // them inline (structurally identical) rather than import them: core is only a
 // transitive dep of sdk-trace-base and isn't reliably resolvable, and keeping it
@@ -696,7 +696,10 @@ export async function flushHeystack() {
 /** Reset the singleton global provider. Internal/testing helper. */
 export function __resetProvider() {
     _provider = null;
+    _remoteSamplingKicked = false;
 }
+/** Guard: the remote sampling config GET fires at most once per isolate. */
+let _remoteSamplingKicked = false;
 let warnedNoKey = false;
 function warnOnceNoKey() {
     if (warnedNoKey)
@@ -763,6 +766,22 @@ export function instrument(handler, config) {
             if (!s)
                 return originalFetch(req, env, ctx);
             const { provider, tracer } = s;
+            // Once per isolate: kick off the remote sampling config GET so the rate
+            // is available for subsequent requests without a redeploy. Uses the
+            // captured pre-patch fetch under suppressTracing — never self-traced,
+            // never looped. Fail-open: any error leaves the rate at 1 (keep-all).
+            if (config.sampling?.remote && !_remoteSamplingKicked) {
+                _remoteSamplingKicked = true;
+                const resolvedKey = config.apiKey ?? env?.HEYSTACK_API_KEY;
+                if (resolvedKey) {
+                    const ep = config.endpoint ?? DEFAULT_ENDPOINT;
+                    ctx.waitUntil(context.with(suppressTracing(context.active()), () => loadRemoteSamplingRate({
+                        endpoint: ep,
+                        apiKey: resolvedKey,
+                        fetchImpl: _originalFetch ?? fetch,
+                    })));
+                }
+            }
             const url = new URL(req.url);
             // FR5: continue an inbound W3C traceparent so tap→server is one trace.
             const parent = parseTraceparent(req.headers.get("traceparent"));

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@heystack/otel",
-  "version": "0.6.0",
+  "version": "0.7.0",
   "description": "Runtime-aware OpenTelemetry tracing that exports to Heystack (Node, Next.js, Workers).",
   "license": "MIT",
   "type": "module",