@heystack/otel 0.14.0 → 0.16.0

package/README.md

@@ -13,7 +13,7 @@ Always read your key from the environment — never paste it into source:
 HEYSTACK_API_KEY=sk_live_…
 ```
 
-> **Use `@heystack/otel` `>=0.14.0`.** 0.14 makes LLM `gen_ai.*` telemetry work under Cloudflare native tracing (it was silently absent before), adds Mistral + AI-gateway (`ai.gateways`) detection and streaming token usage, and moves to the OpenTelemetry 2.x SDK (clears downstream Dependabot alerts). 0.13 made Cloudflare's native trace tree authoritative. See [Migration](#migration--versioning) below.
+> **Use `@heystack/otel` `>=0.16.0`.** 0.16 extends direct Workers instrumentation to Durable Objects, Workflows, and the remaining Worker lifecycle entry points while preserving host-object return semantics. Cloudflare native OTel remains the preferred path for Workers; direct SDK export is a compatibility fallback when native tracing is unavailable. See [Migration](#migration--versioning) below.
 
 ## Runtime matrix
 
@@ -21,6 +21,8 @@ HEYSTACK_API_KEY=sk_live_…
 | --- | --- | --- |
 | Next.js — **any** deploy target (Vercel/Node **and** Cloudflare/OpenNext) | `@heystack/otel/next` | `registerHeystack` in `instrumentation.ts`. Auto-detects Node vs Cloudflare workerd and picks the right exporter. No-op on Edge. |
 | Standalone Cloudflare Workers (hand-written `export default { fetch }`) | `@heystack/otel/workers` | Native-first `instrument()` wrapper; Cloudflare owns OTLP export and automatic platform spans. |
+| Cloudflare Python Workers | No `@heystack/otel` import | Enable Cloudflare native traces/logs and an OTLP destination. Python runs inside workerd, so platform tracing covers handler and supported binding operations without a JavaScript SDK. |
+| WebAssembly modules used by a Worker | Instrument the JavaScript/TypeScript host, if present | Enable Cloudflare native traces/logs for the Worker. Wasm execution is part of its host invocation; this package does not inject language-level spans inside Rust/Go/C Wasm code. |
 | Cloudflare Agents SDK / Agents Tail Worker | `@heystack/otel/workers/agents` | Structured Agents lifecycle events, trace correlation, redaction, and Tail Worker adapter. |
 | Node / Express / Fastify / NestJS (long-running server) | `@heystack/otel/node` | `initHeystack`: auto-instrumentations + graceful shutdown. |
 | Browser (SPA / any web frontend) | `@heystack/otel/web` | `instrumentWeb`: session replay + opt-in browser distributed tracing (`tracing: true`) that emits CLIENT spans + propagates W3C `traceparent` (browser→API shows as one trace). No-op on the server (SSR-safe). |
@@ -147,6 +149,12 @@ export default instrument(worker, {
 });
 ```
 
+Native export is the portable path across Workers languages. The dashboard destination sends OTLP/HTTP traces and logs, so the same configuration applies to JavaScript/TypeScript, [Python Workers](https://developers.cloudflare.com/workers/languages/python/), and Workers whose handlers call [WebAssembly modules](https://developers.cloudflare.com/workers/runtime-apis/webassembly/). Cloudflare's [OTel export documentation](https://developers.cloudflare.com/workers/observability/exporting-opentelemetry-data/) currently says infrastructure and custom metrics export is unavailable.
+
+For a Python Worker, configure the same `observability.traces` and `observability.logs` blocks and emit structured application logs through the runtime's `console`. There is no Python entry point in this npm package and no claim of Python function-level auto-instrumentation. For WebAssembly, native tracing observes the Worker invocation and Cloudflare operations around the module. Add explicit spans in the JavaScript/TypeScript host with `withCloudflareSpan` when you need timings around a Wasm call; spans inside the compiled module require that language's own workerd-compatible instrumentation.
+
+Cloudflare native tracing automatically covers handler invocations, fetch subrequests, and supported platform operations such as KV, R2, and Durable Objects. The direct JavaScript binding proxies below are a compatibility and enrichment layer; they are not required when native tracing is active.
+
 The wrapper also enters native spans for default-export queue and scheduled handlers. `withCloudflareSpan(ctx, ...)` remains native across arbitrary `await` boundaries and does not require `nodejs_compat`. `withSpan(...)` routes to the native tracer whenever its active context is available.
 
 ### Cloudflare Agents SDK
@@ -184,7 +192,7 @@ export default instrument(
     getUser: (req) => ({
       id: req.headers.get("x-user-id") ?? undefined,
     }),
-    instrumentBindings: true,  // auto-trace D1/KV/R2/Vectorize/AI/Queues/Service bindings
+    instrumentBindings: true,  // direct-mode spans for supported binding operations
   },
 );
 ```
@@ -201,7 +209,7 @@ Set the key as a secret: `wrangler secret put HEYSTACK_API_KEY`, and set `native
 > ```toml
 > compatibility_flags = ["nodejs_compat"]
 > ```
-> Without it the SDK falls back to a synchronous stack-based context manager — suppression still works, and **binding spans (D1/KV/R2/Vectorize/AI/Queue/Service) are still parented to the request's SERVER span** (they fall back to the request context explicitly, so they never orphan into their own single-span traces). What still degrades to best-effort without `nodejs_compat` is cross-`await` parenting for *outbound-fetch CLIENT spans* and *manual `withSpan` spans* — so `nodejs_compat` remains recommended for a fully-nested trace tree.
+> Without it the SDK falls back to a synchronous stack-based context manager — suppression still works, and **instrumented binding spans are still parented to the request's SERVER span** (they fall back to the request context explicitly, so they never orphan into their own single-span traces). What still degrades to best-effort without `nodejs_compat` is cross-`await` parenting for *outbound-fetch CLIENT spans* and *manual `withSpan` spans* — so `nodejs_compat` remains recommended for a fully-nested trace tree.
 
 ### `WorkersConfig` options
 
@@ -211,7 +219,7 @@ Set the key as a secret: `wrangler secret put HEYSTACK_API_KEY`, and set `native
 | `nativeTracing` | `"auto" \| "native" \| "direct"` | `auto` (default) uses Cloudflare's native tree when `ctx.tracing` exists. `direct` forces the compatibility exporter. |
 | `apiKey` | `string?` | Defaults to `env.HEYSTACK_API_KEY`. |
 | `getUser` | `(req: Request) => { id?, session?, requestId? } \| undefined` | Called per request. `id` → `enduser.id`, `session` → `session.id`, `requestId` → `http.request.id` (falls back to the `cf-ray` header). |
-| `instrumentBindings` | `boolean \| string[]` | `true` = auto child spans for all detected D1 / KV / R2 / Vectorize / Workers AI / Queue producer / Service binding bindings; `string[]` = only the named bindings. Default `false`. |
+| `instrumentBindings` | `boolean \| string[]` | Direct mode only. `true` = child spans for all detected supported bindings; `string[]` = only named bindings. Default `false`. See the operation matrix below. |
 | `sampling` | `{ rate?: number } \| { remote: true }` | Head-sampling configuration. `{ rate }`: keep a deterministic fraction of fresh root traces (0–1; default `1` = keep all). `{ remote: true }`: fetch the rate from the Heystack config endpoint instead — lets you change it centrally without redeploying. Cold isolates keep all traffic until the first config fetch resolves; fails open if the config can't be reached. Parent-respecting in both modes: a request arriving with a sampled `traceparent` is always recorded. See [Head sampling](#head-sampling) below. |
 | `ai` | `{ captureContent?: boolean; redact?: (text: string) => string; maxContentChars?: number; gateways?: Array<{ hostSuffix: string; pathPattern: string }> }` | LLM/gen_ai capture for outbound calls to known providers (works under native tracing and the direct exporter). `gateways` declares AI gateways/proxies you call instead of the provider host (e.g. Wafer). See [AI / LLM observability](#ai--llm-observability) below. |
 | `waitUntil` | `(p: Promise<unknown>) => void` | Override the isolate keep-alive hook; defaults to the auto-detected `ctx.waitUntil`. |
@@ -257,7 +265,22 @@ On startup the worker fetches its configured rate from the Heystack config endpo
 - **Outbound `fetch`** — each outbound subrequest while a request span is active gets a CLIENT child span (`http.request.method`, `url.full`, `server.address`, `http.response.status_code`). A W3C `traceparent` header is injected into the subrequest so a downstream Heystack-instrumented service continues the same trace (distributed tracing across services). The exporter's own ingest POST is never traced.
 - **Queue consumers (`queue`)** — a CONSUMER span per batch, with `messaging.destination.name` (queue name) and `messaging.batch.message_count`.
 - **Scheduled handlers (`scheduled`)** — an INTERNAL span per invocation, with `controller.cron`.
-- **Binding calls** (when `instrumentBindings` is set) — a child span for every D1 query (`db.statement`), KV read/write, R2 operation, Vectorize query, Workers AI inference (`gen_ai.*` attributes including model name and token usage), Queue `.send`/`.sendBatch` (PRODUCER spans with `messaging.*` attributes), and Service binding `.fetch` calls (CLIENT spans with `traceparent` injected so calls to other Workers stitch into the same distributed trace).
+- **Binding calls** (direct mode, when `instrumentBindings` is set) — child spans for the stable operations listed below. Service binding `.fetch` calls receive a W3C `traceparent` header so another instrumented Worker can continue the trace.
+
+| Binding | Direct-mode operations covered |
+| --- | --- |
+| D1 | prepared statement `all`, `first`, `run`, `raw`; database/session `batch`; database `exec`; statements created through `withSession` |
+| KV | `get`, `getWithMetadata`, `put`, `list`, `delete` |
+| R2 | `get`, `put`, `head`, `delete`, `list`; multipart create/resume plus `uploadPart`, `complete`, `abort` |
+| Vectorize | `query`, `upsert`, `insert`, `deleteByIds`, `getByIds` |
+| Workers AI | `run`, including model and token usage when returned without consuming streams |
+| Queues producer | `send`, `sendBatch`, `metrics` |
+| Workflows | `get`, `create`, `createBatch`, `deleteBatch` |
+| Rate Limiting | `limit`; the caller's rate-limit key is deliberately not attached |
+| Analytics Engine | synchronous `writeDataPoint`; datapoint values are deliberately not attached |
+| Fetch-compatible bindings | `fetch` on Service, Browser Rendering, Assets, and other Fetcher-shaped bindings |
+
+Some bindings intentionally remain outside generic direct-mode proxying. Durable Object namespaces return arbitrary RPC stubs whose method surface is application-defined; Hyperdrive exposes a connection string and the actual calls belong to the selected database driver. Email and Pipelines both expose only a generic `send` method; Analytics SQL exposes only `query`; Media begins with a generic `input`; blindly identifying those shapes could wrap an unrelated application object. Images' chained transform handles and hosted-image handles are also not proxied. Secrets, version metadata, and static-data bindings do not perform observable operations. Use [Cloudflare native tracing](https://developers.cloudflare.com/workers/observability/traces/) where it supports the binding, instrument the database/client library, or wrap a known application operation with `withSpan`. RPC methods beyond `.fetch()` need explicit application spans because automatically proxying arbitrary properties can change RPC stub semantics.
 
 ### Client enrichment
 
@@ -592,6 +615,8 @@ As belt-and-suspenders the exporter also drops any span whose HTTP target points
 
 ## Migration / versioning
 
+- **`0.16.0`** — **complete direct Workers lifecycle and binding coverage.** Durable Object namespace stubs now trace `fetch` and RPC calls with W3C trace propagation; Workflow bindings trace creation, batch creation, lookup, instance lifecycle methods, and deletion operations. Direct instrumentation also covers scheduled, queue, email, Tail Worker, and related Worker entry points, while preserving synchronous values, native promises, custom thenables, upgrade responses, host-object identity, and callback-scoped native span ownership. Service-binding propagation now respects caller-supplied request headers. No breaking changes.
+- **`0.15.0`** — **direct Workers D1 session and batch coverage.** With `nativeTracing: "direct"` and `instrumentBindings` enabled, prepared D1 statements, database/session `batch`, database `exec`, and statements created through `withSession` now receive child spans. Native Cloudflare OTel remains the recommended Workers path because it preserves Cloudflare's platform trace/log tree; use direct export only where `ctx.tracing` is unavailable. No breaking changes.
 - **`0.14.0`** — **gen_ai enrichment under Cloudflare native tracing, Mistral + AI-gateway detection, streaming usage; OpenTelemetry JS SDK 2.x.** (1) `/workers`: under `nativeTracing: "auto" | "native"` `instrument()` never installed the fetch wrapper, so native-mode Workers emitted **no** `gen_ai.*` telemetry. It now installs a lean LLM-only wrapper that opens a native `<operation> <provider>` span (e.g. `chat mistral`) with request/response `gen_ai.*` for detected providers and passes every other subrequest through untouched. (2) New provider **Mistral** (`api.mistral.ai`) and a new `ai.gateways` option (`[{ hostSuffix, pathPattern }]`, e.g. `{ hostSuffix: "wafersecurity.ai", pathPattern: "/p/:tenant/:provider" }`) that resolves the provider from an AI-gateway URL and stamps `heystack.ai.gateway`; `gemini` → `google`, `workers-ai` → `cloudflare`. New `gen_ai.operation.name`. (3) **Streaming (`text/event-stream`) responses now yield usage**: the body is `tee()`d and scanned line-by-line for `usage`/`usageMetadata`, model, id and finish reason only (never content); the span stays open until the stream closes. Google (`usageMetadata`) and Mistral response shapes added to the JSON path too. (4) New `/workers` exports `genAiRequestAttrs` / `genAiResponseAttrs` for manual `withSpan` spans around `env.AI.run` or any non-`fetch` LLM call. Native `error.type` now keeps the exception class instead of being overwritten with `error`. (5) All `@opentelemetry/*` dependencies move to the 2.x SDK line (`core`/`sdk-trace-base`/`resources` ^2.11, `sdk-node`/`exporter-trace-otlp-http` ^0.222, `auto-instrumentations-node` ^0.80, `semantic-conventions` ^1.43), clearing the downstream Dependabot alerts the 1.x line surfaced through this package (GHSA-8988-4f7v-96qf in `@opentelemetry/core`, GHSA-q7rr-3cgh-j5r3 in `sdk-node` / `auto-instrumentations-node` / `exporter-prometheus`, GHSA-45rx-2jwx-cxfr in `propagator-jaeger`). **Node ≥ 18.19 (or ≥ 20.6) is now required on `/node` and `/next`** — the 2.x SDK dropped Node 14/16. If you pass your own `instrumentations` array to `initHeystack`, upgrade those `@opentelemetry/instrumentation-*` packages to a release that peers on `@opentelemetry/core@^2` (the auto-instrumentations 0.80 set already does). `/workers` is still WinterCG-safe (no `node:*` imports). No other API changes; all new options are optional.
 - **`0.13.0`** — **Cloudflare-native tracing and complete Workers platform coverage.** On runtimes exposing `ctx.tracing`, `/workers` now joins Cloudflare's authoritative invocation trace instead of creating and exporting a duplicate tree. `nativeTracing: "auto"` is the default; `withCloudflareSpan(ctx, ...)` adds async-safe business spans, while fetch, queue, and scheduled handlers are enriched in the native tree. The new `/workers/agents` entry maps Cloudflare Agents observability callbacks and Tail Worker diagnostics to structured, redacted telemetry. Direct OTLP remains available with `nativeTracing: "direct"` for runtimes without the native API. Heystack ingest now preserves full OTLP trace/log fidelity, accepts OTLP metrics, and can import Workers Analytics metrics through the app-scoped Cloudflare connector. Native mode needs no Worker API-key secret and must not be combined with direct export.
 - **`0.11.0`** — **`/web`: browser error + console collection, in-app bug reports, and seekable replay keyframes.** `instrumentWeb` now captures uncaught errors (`window.onerror` + `unhandledrejection`) and `console.error` as OTLP logs by default (`event.name=browser.error` / `browser.console`, with `exception.*` semconv, `url.full`, `session_id`, and the active trace/span ids when `tracing` is on). New options: `errors?: boolean` (default `true`), `captureConsole?: 'error' | 'warn' | false` (default `'error'`). New **`reportBug({ message, email?, context? })`** API — a headless in-app bug reporter that POSTs to `/v1/bug-reports` and auto-attaches the URL, user agent, `session_id`, active `trace_id`, `release`/`build`, and the last ≤20 captured browser errors (also emitted as a `user.bug_report` log); reports triage in the console **Bugs** tab. The `version` / `build` options are now stamped as `service.version` / `service.build` on exported browser logs (release attribution). Console capture is recursion-guarded + rate-capped (no self-export loop, no flood). The recorder also checkpoints a keyframe (~60s) so the console player seeks from the nearest snapshot instead of loading the whole session. No breaking changes; all new options are optional.
@@ -624,3 +649,13 @@ As belt-and-suspenders the exporter also drops any span whose HTTP target points
 - Short-lived processes: `initHeystack` registers `SIGTERM`/`SIGINT` handlers automatically, so the SDK flushes and shuts down before exit and you don't lose the last batch.
 
 Then run your app and make a request — traces appear in your Heystack dashboard within seconds.
+
+## Investigate with your coding agent
+
+Once telemetry is flowing, connect your coding agent to the hosted **Heystack MCP server** and it can verify the setup (`heystack_verify_setup`), investigate incidents from your traces, sessions and releases, and propose the fix in your code:
+
+```bash
+claude mcp add --transport http heystack https://mcp.heystack.dev/mcp
+```
+
+One-click links for Cursor and VS Code, commands for Codex/Gemini CLI, personal access tokens for CI, and the full tool reference are in the docs: **https://heystack.dev/docs/mcp**.

package/dist/traceparent.d.ts

@@ -0,0 +1,8 @@
+interface TraceparentSpanContext {
+    traceId: string;
+    spanId: string;
+    traceFlags: number;
+}
+/** Encode a span context as W3C traceparent, preserving the low flags byte. */
+export declare function makeTraceparent(context: TraceparentSpanContext): string;
+export {};

package/dist/traceparent.js

@@ -0,0 +1,5 @@
+/** Encode a span context as W3C traceparent, preserving the low flags byte. */
+export function makeTraceparent(context) {
+    const flags = (context.traceFlags & 0xff).toString(16).padStart(2, "0");
+    return `00-${context.traceId}-${context.spanId}-${flags}`;
+}

package/dist/workers-agents.d.ts

@@ -1,5 +1,5 @@
 import { type Span } from "@opentelemetry/api";
-import { type CloudflareNativeTracing } from "./workers";
+import { type CloudflareNativeTracing } from "./workers.js";
 /** Structural equivalent of the Agents SDK event; no `agents` peer required. */
 export interface CloudflareAgentEvent {
     type: string;

package/dist/workers-agents.js

@@ -1,5 +1,5 @@
 import { trace } from "@opentelemetry/api";
-import { withCloudflareSpan, } from "./workers";
+import { withCloudflareSpan, } from "./workers.js";
 const SENSITIVE_KEY = /(?:authorization|cookie|password|secret|token|api[_-]?key|prompt|completion|content|message)/i;
 const ERROR_EVENT = /(?::error|:failed|:stalled|:interrupted|:exhausted)$/;
 function sanitize(value, extraKeys, depth = 0) {

package/dist/workers-bindings.d.ts

@@ -10,11 +10,19 @@ export interface InstrumentBindingsOpts {
      */
     startSpan: (name: string, attrs: Record<string, unknown>, kind?: SpanKind) => Span;
     /**
-     * `true`  → auto-detect and wrap all D1/KV/R2/Vectorize/AI/Queue/Service bindings.
+     * `true`  → auto-detect and wrap every supported Cloudflare binding.
      * `string[]` → only wrap bindings whose env key is listed.
      */
     select: boolean | string[];
 }
+/**
+ * Run `fn`, ending `span` in `finally` and recording exceptions + ERROR status
+ * on failure (re-throwing). Internal helper shared with the class-level
+ * wrappers in workers.ts.
+ */
+export declare function runWithSpan<T>(span: Span, fn: () => Promise<T>): Promise<T>;
+/** Internal proxy builder shared with the class-level wrappers in workers.ts. */
+export declare function makeProxy(target: any, handlers: Record<string, Function>): any;
 /**
  * Build a new args array for `fetch(input, init?)` with a `traceparent` header
  * injected WITHOUT mutating the caller's original Request or init object.
@@ -34,6 +42,11 @@ export declare function injectTraceparentArgs(args: any[], traceparent: string):
  * - Vectorize: `query`+`upsert`
  * - Workers AI: `run` (without `prepare`)
  * - Queue producer: `send`+`sendBatch`
+ * - Workflow: `get`+`create`+`createBatch`
+ * - Rate limit: `limit`
+ * - Analytics Engine: `writeDataPoint`
+ * - Durable Object namespace: `idFromName`+`get`+`newUniqueId` (stub fetch/RPC spans)
+ * - Workflow binding: `create`+`createBatch`+`get` and not a DO namespace
  * - Service binding / Fetcher: `fetch` (last branch; most generic)
  *
  * Unrecognised bindings are passed through unchanged.