@gr8ful/spf 0.17.0 → 0.18.0

package/assets/skill/references/config.md

@@ -110,11 +110,13 @@ the two runs' cost/tokens/gates together.
 |---|---|---|
 | `db` | path \| `{kind: sqlite, path?}` \| `{kind: d1, database_id, account_id_env?, api_token_env?}` | Where the trace db lives. A bare string (or the equivalent spelled-out `{kind: sqlite, ...}`) is a repo-relative local sqlite path, default `.spf/data/spf.db`. `{kind: d1, ...}` points at a remote Cloudflare D1 database instead — `database_id` is required; `account_id_env`/`api_token_env` default to `CLOUDFLARE_ACCOUNT_ID`/`CLOUDFLARE_API_TOKEN` (the same env vars the `cloudflare` model provider reads), naming the env vars to read the credentials from, never the credentials themselves. **Egress**: `kind: d1` sends the complete trace — raw request text, tool arguments/results, envelope payloads, gate violation details — to Cloudflare's D1 HTTP API on every run; `kind: sqlite` (the default) never leaves the machine. See `observability.md`'s "Two stores, one truth" and "D1 ships the complete trace off-box" sections for what changes (and what deliberately doesn't) with a D1-backed trace db — notably, the local backend's WAL live-read guarantee does not hold for D1. |
 | `poll_ms` | int | UI live-poll cadence. Default `500`. |
-| `otel.endpoint` | string | OTLP/HTTP collector endpoint (e.g., `https://your-host/v1/traces`). Omit to disable OTel export. |
+| `otel.endpoint` | string | OTLP/HTTP collector endpoint (e.g., `https://your-host/v1/traces`). Omit to disable OTel export (traces AND metrics — both share this one switch). |
 | `otel.headers` | object | Optional HTTP headers (e.g., auth tokens). Each value is a string. |
-| `otel.service_name` | string | Optional service name in exported spans. Default `spf`. |
+| `otel.service_name` | string | Optional service name in exported spans/metrics. Default `spf`. |
+| `otel.metrics` | bool | Default `true`. Set `false` to keep trace export on while opting out of the process-scoped metrics pipeline (`spf.tokens`, `spf.cost_usd`, `spf.phase.duration`, `spf.gate.result`, `spf.agent.calls` — see `observability.md`). Has no effect on activation either way. |
+| `otel.allow_env` | bool | Default `false`. When `true` AND `otel.endpoint` is ALREADY set here, the standard `OTEL_EXPORTER_OTLP_ENDPOINT`/`OTEL_EXPORTER_OTLP_HEADERS` env vars may SUPPLEMENT this block (e.g. a CI-injected collector token) — config-declared values win on conflict, and this can never activate export on its own. |
 
-**No ambient env activation**: OTEL export requires explicit `observability.otel` config — the `OTEL_EXPORTER_OTLP_ENDPOINT` shell variable is never consulted. An unrelated shell env variable must not become a data-egress switch.
+**No ambient env activation**: OTEL export requires explicit `observability.otel` config — the `OTEL_EXPORTER_OTLP_ENDPOINT` shell variable is never consulted UNLESS `otel.allow_env: true` and the block above is already active (see that row).
 
 **`spf init`'s interview** asks local-vs-D1 only inside the "advanced" gate (declining advanced settings, or `--yes`, leaves `observability.db` unset — local sqlite at its default path, today's behavior, unchanged). Choosing D1 asks for `database_id` (required) and, unless this same interview run already collected `CLOUDFLARE_ACCOUNT_ID`/`CLOUDFLARE_API_TOKEN` for the Cloudflare Workers AI provider, asks for those too — reusing them instead of asking twice when it did. The generated config omits `account_id_env`/`api_token_env` (they match the schema's own defaults above), so a D1 choice writes just `db: {kind: d1, database_id: ...}`.
 
@@ -545,6 +547,7 @@ when `.gitmodules` is present.
 | `writes` | no | What this agent may modify **in the repo**, enforced after every call. `undefined`/`null` = unrestricted (still barred from `protected_files`); `[]` = no repo writes; a list = only those paths (trailing `/` = directory prefix, `*` = one path segment, `**` = crosses segments, anything else = exact path). |
 | `env_allowlist` | no | Opt-in filter on the environment handed to this agent's subprocess/sandbox. `undefined` (default) = the full operator environment, unchanged. A list = only those keys, plus the baseline (`PATH`, `HOME`, `USER`, `LANG`, `TERM`, `TMPDIR`) either backend keeps regardless. |
 | `sandbox` | no | Per-agent backend override — see `sandbox` below. `undefined`/`null` = inherit `sandbox.backend`; `"local"` = force local for this agent regardless of the repo-wide setting; a backend name = use that backend for this agent only. A bare name, never a nested block — everything else (`image`, `egress`, `transport`, credentials *policy*) stays repo-wide. |
+| `lora_adapter` | no | Explicit override for the `spf.lora_adapter` OTel span attribute (see `observability.md`). Unset (the common case) falls back to parsing `model` — `provider/base:adapter` (the suffix after the last `:`), else `provider/adapter-name` when the id contains `-lora-` (this org's own vLLM/Switchyard served-model convention). Purely descriptive: never changes routing, dispatch, or which model actually serves the call. |
 
 Output types are deliberately absent from config: an entry defines who an
 agent *is*; the call site defines how it's *used*.

package/assets/skill/references/observability.md

@@ -75,9 +75,123 @@ reading `parent_id`.
 `spf.config.yaml`, a lossy, allowlisted projection of phase/agent/tool spans
 (status, model, token/cost counts, gate results — never prompts, envelopes,
 tool arguments, or source code) is also pushed to an OTLP/HTTP collector,
-fire-and-forget. SQLite remains the source of truth regardless; see
+fire-and-forget, via the real `@opentelemetry/sdk-trace-base` +
+`exporter-trace-otlp-http` packages (SPF's own bespoke sha256 trace/span-id
+scheme and attribute allowlist ride on top of the real SDK's encoder, not a
+hand-rolled one — see `core/otel.ts`'s header for exactly what that means for
+the wire bytes). SQLite remains the source of truth regardless; see
 `config.md`'s `observability.otel.*` rows for the field reference.
 
+**Optional OTel metrics.** The same `observability.otel.endpoint` also gates
+a second, PROCESS-scoped pipeline (`core/otel_metrics.ts`, built on
+`@opentelemetry/sdk-metrics` + `exporter-metrics-otlp-http`) — set
+`observability.otel.metrics: false` to keep trace export on while opting out
+of metrics specifically. Five instruments: `spf.tokens` (counter; attrs
+`kind` — input/output/cache_read/cache_write — `agent`, `model`),
+`spf.cost_usd` (counter; `agent`, `model`), `spf.phase.duration` (histogram,
+seconds; `kind`, `owner`, `status`), `spf.gate.result` (counter; `gate`,
+`result` — pass/fail), and `spf.agent.calls` (counter; `agent`, `model`,
+`coding_agent`). Unlike the span exporter (one per run), there is exactly
+one `MeterProvider` for the life of the `spf` process — see `otel_metrics.
+ts`'s own header for why that matters under `spf watch`'s daemon loop.
+
+**Outbound trace-context propagation.** When otel is configured, SPF also
+tries to carry `traceparent` onto the OUTBOUND model calls each agent makes,
+so this run's spans join whatever trace the model-serving stack itself
+produces (see "Tracing across the inference stack" below). The three coding
+agent backends get different-confidence treatment:
+  - `coding_agent: claude_code` — a real, verified guarantee.
+    `agent_cc.ts`'s single `spawn()` choke point sets `TRACEPARENT` and
+    `ANTHROPIC_CUSTOM_HEADERS` (`traceparent`/`x-request-id`, newline-
+    separated `Name: Value` pairs — the CLI's own documented format,
+    requires `claude` CLI >= 2.1.227) on the `claude` subprocess's
+    environment for every call.
+  - `coding_agent: opencode` — config-verified, behavior best-effort.
+    `agent_opencode.ts` injects `TRACEPARENT` onto the subprocess env
+    (parity with `claude_code`; unverified whether the opencode CLI itself
+    reads it) AND writes `traceparent`/`x-request-id` as static
+    `provider.<id>.options.headers` into the temp `opencode.json` —
+    opencode's documented per-provider options surface, honored by the AI
+    SDK on that provider's requests. Static is correct because one
+    `opencode run` subprocess is exactly one agent call. Two caveats,
+    documented in the module's own doc comment: a bare model id with no
+    `provider/` prefix skips the headers (SPF never guesses a provider id),
+    and a target repo's own `opencode.json` merges at higher precedence and
+    can override them.
+  - `coding_agent: flue` — best-effort. `@flue/opentelemetry`'s own docs say
+    plainly that `dispatch()` "does not propagate trace context" on its own,
+    so SPF additionally registers `@opentelemetry/instrumentation-http` +
+    `-undici` globally, with a propagator that carries both the standard W3C
+    `traceparent` and a custom `x-request-id`. This reaches any provider
+    whose Node SDK issues requests through `http`/`https`/`undici` (verified)
+    — it does NOT reach a provider transport that bypasses both (unverified
+    for the Anthropic/Google/Mistral SDKs' internal transports specifically;
+    flagged, not assumed). See `core/otel_propagation.ts`'s header for the
+    full mechanism. Since #80, flue's spans additionally JOIN SPF's
+    deterministic trace: `agent_flue.ts` registers each run's session id
+    (== flue's instance id) against that call's traceparent, and the
+    installed instrumentation's `resolveRootContext` option — consulted per
+    root span, keyed on the event context's instance id — roots each
+    session's spans under the right agent-call span. This is done per-span
+    rather than by wrapping `dispatch()` in a context because flue's node
+    runtime executes all submissions in one process-lifetime claim loop
+    whose async context is captured once — a dispatch-time wrap would
+    mis-attribute every agent after the first into the first agent's trace
+    (verified against `@flue/runtime`'s dist; see `core/otel_propagation
+    .ts`'s header). Unmapped sessions (backlog restarts, post-settlement
+    stragglers) degrade to the old separate-trace behavior — never
+    mis-attributed, never an error; correlate those by `x-request-id`/
+    `spf.adw_id`/time window.
+
+## Tracing across the inference stack
+
+A run's SPF spans are not the only spans in play once a `claude_code` agent
+is pointed at a self-hosted model through Switchyard/vLLM (this repo's own
+inference-platform-aws stack): the `claude` CLI's own request, Switchyard's
+routing hop, and vLLM's own serving span can ALL be emitted to the same Tempo
+(or any OTLP-compatible) backend. What joins them:
+
+  - **The `traceparent` SPF injects** (see above) is a real W3C header on the
+    actual HTTP request `claude` makes to `ANTHROPIC_BASE_URL` — whatever
+    receives that request (Switchyard, vLLM directly, an Envoy AI Gateway
+    hop) that is ALSO instrumented with OTel and honors an inbound
+    `traceparent` will parent its own span under SPF's agent-call span,
+    landing in the SAME trace.
+  - **SPF's OWN span export is a separate, deterministic trace** — one
+    `adw_id` = one trace id (`sha256(adw_id)`), independent of whatever trace
+    id the model-serving hop's own OTel SDK would otherwise mint. Since SPF
+    controls the OUTBOUND `traceparent` it sends (not merely observes one),
+    a `claude_code` call's downstream spans (Switchyard, vLLM) land as
+    CHILDREN of SPF's own deterministic trace id, not the other way around —
+    a Tempo query for `sha256(adw_id)` finds the whole cross-service picture
+    for that run, agent call down through the model server.
+  - **The `flue` backend joins the same trace** (since #80): Flue's own
+    spans (`flue.coordinator`, `invoke_agent`, `chat <model>`,
+    `execute_tool`) inherit SPF's deterministic trace id (SDK-random span
+    ids underneath) via the instrumentation's per-span `resolveRootContext`
+    rooted on the run's registered session-id -> traceparent map, and the
+    `traceparent` the http/undici instrumentations inject onto Flue's
+    provider requests carries it too — downstream Switchyard/vLLM spans
+    land in the same trace, exactly as the `claude_code` path described
+    above. Unmapped submissions (a restarted process draining a durable
+    backlog, post-settlement bookkeeping) fall back to a separate trace —
+    never MIS-attributed into another agent's trace — correlatable by
+    `x-request-id`/`spf.adw_id`/time window; see the "Outbound
+    trace-context propagation" section above.
+  - **`x-request-id`** rides alongside `traceparent` specifically so a
+    collector/log pipeline that correlates by individual REQUEST (rather
+    than by trace) has a stable id to key on — it is this call's own
+    (SPF-side) span id for `claude_code`, and the currently-active span's id
+    for `flue`.
+  - **vLLM's own cache-read pass-through**
+    (`usage.prompt_tokens_details.cached_tokens`, surfaced here as both
+    `spf.tokens.cache_read` and `gen_ai.usage.cache_read.input_tokens`)
+    requires the upstream vLLM server to be started with
+    `--enable-prompt-tokens-details` (off by default) — SPF has no way to
+    detect this at config-validation time; a correctly-wired pipeline reads
+    `0` forever against a server that hasn't set the flag, with no bug
+    anywhere in SPF's own code.
+
 **Spend is itemized per phase.** `agent_end.usage` carries tokens *and*
 dollars for each component Flue reports (matching pi-ai's field names
 one-for-one) — `input`, `output`, `cache_read`, `cache_write` — summed

package/dist/cli/index.js

@@ -11,6 +11,7 @@ import * as agentFlue from "../core/agent_flue.js";
 import * as agentOpencode from "../core/agent_opencode.js";
 import * as notify from "../core/notify/notifier.js";
 import * as otel from "../core/otel.js";
+import * as otelMetrics from "../core/otel_metrics.js";
 import * as paths from "../core/paths.js";
 import * as sandbox from "../core/sandbox.js";
 import { findChain, registerRepoChains, repoChainProblems } from "../chains/index.js";
@@ -261,5 +262,10 @@ export async function main() {
         // deadline, so a fast-exiting command neither drops spans mid-flight nor
         // waits on an unreachable collector. Never throws (core/otel.ts).
         await otel.flushAll();
+        // Shuts down the process-scoped OTel MeterProvider (a no-op if metrics
+        // were never configured/resolved this process) — see otel_metrics.ts's
+        // own module header for why this is process-scoped rather than per-run
+        // like the span exporter above. Never throws.
+        await otelMetrics.shutdownOtelMetrics();
     }
 }

package/dist/core/agent_cc.d.ts

@@ -115,6 +115,17 @@ export declare class CcToolCallTracker {
     private finish;
 }
 export declare function isKnownToolName(name: string): boolean;
+/**
+ * Merges `TRACEPARENT`/`ANTHROPIC_CUSTOM_HEADERS` into `baseEnv` for
+ * outbound OTel propagation, or returns `baseEnv` UNCHANGED when
+ * `otel` is absent (the common case — `observability.otel` not configured).
+ * `ANTHROPIC_CUSTOM_HEADERS`'s verified format is newline-separated
+ * `Name: Value` pairs (see this module's own header for the citation); an
+ * operator-supplied value already present in `baseEnv` is kept and appended
+ * to, not overwritten — a real header injected via config/settings must
+ * still reach the wire alongside this module's own.
+ */
+export declare function injectOtelEnv(baseEnv: Record<string, string>, otel: AgentRequest["otel"] | undefined): Record<string, string>;
 /**
  * Resolve `SPF_CLAUDE_CMD` for THIS call, substituting a literal `{model}`
  * token with `model` — see the module doc comment for why a fixed tag baked

package/dist/core/agent_cc.js

@@ -248,6 +248,27 @@ const EFFORT_MAP = {
     xhigh: "xhigh",
     max: "max",
 };
+/**
+ * Merges `TRACEPARENT`/`ANTHROPIC_CUSTOM_HEADERS` into `baseEnv` for
+ * outbound OTel propagation, or returns `baseEnv` UNCHANGED when
+ * `otel` is absent (the common case — `observability.otel` not configured).
+ * `ANTHROPIC_CUSTOM_HEADERS`'s verified format is newline-separated
+ * `Name: Value` pairs (see this module's own header for the citation); an
+ * operator-supplied value already present in `baseEnv` is kept and appended
+ * to, not overwritten — a real header injected via config/settings must
+ * still reach the wire alongside this module's own.
+ */
+export function injectOtelEnv(baseEnv, otel) {
+    if (!otel)
+        return baseEnv;
+    const ownHeaders = [`traceparent: ${otel.traceparent}`, `x-request-id: ${otel.x_request_id}`].join("\n");
+    const existing = baseEnv.ANTHROPIC_CUSTOM_HEADERS;
+    return {
+        ...baseEnv,
+        TRACEPARENT: otel.traceparent,
+        ANTHROPIC_CUSTOM_HEADERS: existing ? `${existing}\n${ownHeaders}` : ownHeaders,
+    };
+}
 /**
  * Resolve `SPF_CLAUDE_CMD` for THIS call, substituting a literal `{model}`
  * token with `model` — see the module doc comment for why a fixed tag baked
@@ -337,7 +358,24 @@ export async function run(request, onEvent, onSpawn, onExit) {
     const isOllamaLaunch = isOllamaLaunchCmd(cmdSpec);
     const needsOllamaLaunchSeparator = isOllamaLaunch && !cmdArgs.includes("--");
     const fullArgs = needsOllamaLaunchSeparator ? [...cmdArgs, "--", ...args] : [...cmdArgs, ...args];
-    const child = spawn(cmd, fullArgs, { cwd: request.cwd, env: request.env ?? operatorEnv() });
+    // Outbound OTel propagation (SPF's otel-sdk extension): `request.otel` is
+    // set only when `observability.otel` is configured for this run (see
+    // `agents.ts`'s `send()`) — absent otherwise, so `env` below is
+    // byte-identical to before this existed for every repo that hasn't
+    // configured otel. `TRACEPARENT` is the standard W3C env var the `claude`
+    // CLI's own subprocesses/telemetry already look for; `ANTHROPIC_CUSTOM_
+    // HEADERS` additionally puts `traceparent` (redundant with the env var,
+    // but this is the only way to reach the ACTUAL outbound HTTP request the
+    // CLI itself makes to its configured `ANTHROPIC_BASE_URL`) and
+    // `x-request-id` (this agent call's own span id) onto that request's
+    // headers. Format verified against Claude Code's own docs (https://
+    // code.claude.com/docs/en/env-vars, fetched live for this feature —
+    // requires CLI >= 2.1.227): "Custom headers to add to requests (`Name:
+    // Value` format, newline-separated for multiple headers)". An
+    // operator-supplied `ANTHROPIC_CUSTOM_HEADERS` already present in
+    // `request.env` is PRESERVED, not clobbered — this appends to it.
+    const env = injectOtelEnv(request.env ?? operatorEnv(), request.otel);
+    const child = spawn(cmd, fullArgs, { cwd: request.cwd, env });
     // The prompt travels as a positional argv element, not stdin — closing it
     // immediately avoids a real, observed ~3s "no stdin data received" stall
     // where `claude` otherwise waits to see whether anything is piped in.

package/dist/core/agent_flue.js

@@ -30,6 +30,7 @@ import { registerOllamaModel } from "./ollama_provider.js";
 import { registerCloudflareModel } from "./cloudflare_provider.js";
 import * as sandbox from "./sandbox.js";
 import { nowIso, operatorEnv } from "./utils.js";
+import { installFluePropagation, registerFlueSessionTrace, unregisterFlueSessionTrace } from "./otel_propagation.js";
 const RESULT_SNIPPET_CHARS = 20_000; // tool output rides along whole; clip only guards pathological cases
 const ARG_VALUE_CHARS = 20_000; // args too — the UI scrolls, it must not be handed cut-off data
 const LABEL_CHARS = 80; // "bash: <command>" shown as the event name
@@ -402,6 +403,14 @@ export async function run(request, onEvent, onSpawn, onExit) {
     // real-Bearer-token (not dummy-key) auth.
     if (provider === "cloudflare")
         await registerCloudflareModel(modelId);
+    // Outbound OTel propagation (SPF's otel-sdk extension) — see
+    // `otel_propagation.ts`'s own header for what this does and does not
+    // guarantee. `request.otel` is set only when `observability.otel` is
+    // configured for this run (see `agents.ts`'s `send()`); the installer is
+    // itself a no-op on `undefined` AND idempotent across every later call in
+    // this same process, so this costs nothing for a repo that hasn't
+    // configured otel and installs at most once for one that has.
+    installFluePropagation(request.otel);
     await ensureRuntime(request.flue_db_path);
     REGISTRY.set(request.session_id, {
         model: request.model,
@@ -417,6 +426,19 @@ export async function run(request, onEvent, onSpawn, onExit) {
     const pid = process.pid ?? -1;
     onSpawn?.(pid);
     const handle = init(SfAgent, { id: request.session_id });
+    // Join this session's flue spans to SPF's deterministic trace: the session
+    // id (== flue's instance id, which every flue observability event's ctx.id
+    // carries — see otel_propagation.ts's header) is registered against this
+    // call's traceparent, and the installed instrumentation resolves root
+    // spans through that map per span. WRAPPING dispatch() in a context here
+    // would be WRONG, not just redundant (verified against @flue/runtime's
+    // node dist): execution runs in ONE process-lifetime claim loop whose
+    // async context is captured by the first dispatch that started it, so a
+    // dispatch-time context wrap would silently mis-attribute every later
+    // agent's spans into the FIRST agent's trace. No-op when otel is
+    // unconfigured.
+    if (request.otel)
+        registerFlueSessionTrace(request.session_id, request.otel.traceparent);
     const receipt = await handle.dispatch(request.prompt);
     const slot = { usage: new UsageBreakdown(), context_tokens: 0 };
     pendingUsage.set(receipt.submissionId, slot);
@@ -455,6 +477,10 @@ export async function run(request, onEvent, onSpawn, onExit) {
     }
     finally {
         pendingUsage.delete(receipt.submissionId);
+        // Post-settlement bookkeeping spans flue mints after this point resolve
+        // to an unparented root (separate trace) — by design; see
+        // otel_propagation.ts's unregisterFlueSessionTrace.
+        unregisterFlueSessionTrace(request.session_id);
         onExit?.(pid);
     }
 }

package/dist/core/agent_opencode.d.ts

@@ -65,9 +65,48 @@
  * docs, a target repo's OWN project-level `opencode.json` (if one exists)
  * merges at HIGHER precedence than the file `OPENCODE_CONFIG` points at.
  * That means a repo carrying its own `opencode.json` can silently override
- * (widen or narrow) the restriction this module writes — a real, documented
- * gap in this backend's tool-restriction guarantee, not a bug this module
- * can paper over from the outside.
+ * (widen or narrow) the restriction this module writes, and likewise
+ * override the OTel propagation headers described below — a real,
+ * documented gap in both guarantees, not a bug this module can paper over
+ * from the outside.
+ *
+ * OUTBOUND OTEL PROPAGATION [OFFICIAL config surface; best-effort
+ * behavior]: when `request.otel` is present (`agents.ts`'s `send()` threads
+ * it only when `observability.otel` is configured — absent otherwise, and
+ * this whole path is byte-identical to before), this module propagates
+ * trace context two ways:
+ *
+ *   1. `TRACEPARENT` on the child's env — the standard W3C env var, same
+ *      shape `agent_cc.ts` sets. No published statement confirms the
+ *      opencode CLI itself reads it (UNVERIFIED either way; set for parity
+ *      and for any opencode-spawned subprocess telemetry, at zero cost).
+ *   2. `traceparent` + `x-request-id` as STATIC provider headers in the
+ *      temp `opencode.json` (`provider.<id>.options.headers` — opencode's
+ *      documented per-provider options surface;
+ *      https://opencode.ai/docs/providers/). Static values are CORRECT
+ *      here, unlike the general case, because one `opencode run` subprocess
+ *      IS exactly one SPF agent call — the traceparent can never go stale
+ *      mid-run. This is the header that actually reaches the wire:
+ *      opencode's provider requests flow through the AI SDK, which honors
+ *      `options.headers` for that provider's requests. Subject to the
+ *      CONFIG PRECEDENCE limitation above: a repo's own `opencode.json` can
+ *      override these headers.
+ *
+ * `injectOtelEnv()` / `otelProviderHeaders()` / `tempConfigContents()` are
+ * exported pure functions so every fragment is unit-testable without
+ * spawning a real subprocess — same discipline as the rest of this module.
+ *
+ * OPERATOR-CONFIG MERGE: when a caller-provided `OPENCODE_CONFIG` already
+ * exists in the base env (operatorEnv() passthrough or an agent's
+ * env_allowlist) AND this module needs a temp config of its own (a tools:
+ * restriction or the propagation headers), the operator's file is READ and
+ * its contents MERGED into the temp file (SPF's own blocks win on
+ * conflict — see `mergeOperatorConfig()`), rather than the pre-existing
+ * replace-it-outright behavior. This closes a real regression class:
+ * enabling OTel or a tools: list on an already-configured workflow used to
+ * silently drop the operator's provider routing/credentials config. An
+ * unreadable or unparseable operator file falls back to the old replace
+ * behavior (and is left untouched on disk) rather than failing the run.
  *
  * `write`/`apply_patch` ARE GATED THROUGH `edit` [OFFICIAL]: opencode's own
  * docs say these two are not independent permission keys — both ride the
@@ -206,6 +245,69 @@ export declare class OcToolCallTracker {
     observe(message: OcStreamMessage): Record<string, any> | null;
 }
 export declare function isKnownToolName(name: string): boolean;
+/**
+ * Merges `TRACEPARENT` into `baseEnv` for outbound OTel propagation, or
+ * returns `baseEnv` UNCHANGED when `otel` is absent (the common case —
+ * `observability.otel` not configured). See the module doc comment's
+ * OUTBOUND OTEL PROPAGATION section: unlike `agent_cc.ts`'s sibling, no
+ * published statement confirms the opencode CLI itself reads this var, so
+ * the REAL propagation path is the temp-config headers below; this env var
+ * is set for parity at zero cost.
+ */
+export declare function injectOtelEnv(baseEnv: Record<string, string>, otel: AgentRequest["otel"] | undefined): Record<string, string>;
+export interface OpencodeOtelHeaders {
+    provider: string;
+    headers: {
+        traceparent: string;
+        "x-request-id": string;
+    };
+}
+/**
+ * Builds the `provider.<id>.options.headers` fragment for a temp
+ * `opencode.json` (see the module doc comment's OUTBOUND OTEL PROPAGATION
+ * section for why static values are correct here). Returns `null` when
+ * `otel` is absent, or when the model id carries no `provider/` prefix —
+ * opencode's own `--model` vocabulary is documented as `provider/model-id`,
+ * but a bare model name has no provider id this module could key headers
+ * under, and guessing the wrong provider id would write a config block
+ * opencode merges onto a DIFFERENT provider than the one being called.
+ */
+export declare function otelProviderHeaders(model: string, otel: AgentRequest["otel"] | undefined): OpencodeOtelHeaders | null;
+/**
+ * Builds the JSON CONTENT of the temp `opencode.json`: a `permission` map
+ * when `toolNames` is an array (including `[]` — restrict to exactly those,
+ * mirroring CC's `--tools`), a `provider` block when `otelHeaders` is
+ * present, both when both apply. `null` when NEITHER applies — no config
+ * file needed at all, `--auto` alone (every tool usable, mirroring CC's
+ * `"default"`). Exported as its own pure function, same reasoning as
+ * `resolveOpencodeCmdSpec`: this shape is unit-testable without touching
+ * the filesystem or spawning a subprocess.
+ */
+export declare function tempConfigContents(toolNames: string[] | null | undefined, otelHeaders: OpencodeOtelHeaders | null): Record<string, unknown> | null;
+/**
+ * Recursive plain-object merge of SPF's own temp-config blocks ONTO the
+ * operator's existing `OPENCODE_CONFIG` contents — SPF's values win on
+ * conflicting plain-object leaves/deeper keys (a `tools:` restriction is a
+ * safety gate and the propagation headers are this call's own, so neither
+ * may be silently watered down), while every key the operator set and SPF
+ * never touches (provider `baseURL`/`apiKey`/model routing, MCP servers,
+ * agents) passes through intact. Arrays are replaced, not concatenated:
+ * this module never writes arrays itself, so a replaced array is always
+ * one of the OPERATOR's own values being deliberately overridden by an
+ * SPF block — the same "SPF wins on conflict" rule.
+ */
+export declare function mergeOperatorConfig(base: Record<string, unknown>, overlay: Record<string, unknown>): Record<string, unknown>;
+/**
+ * Reads the operator's existing `OPENCODE_CONFIG` file for merging, or
+ * returns `null` when there is nothing usable to merge with — file missing
+ * / unreadable / not valid JSON with a plain-object root. JSON.parse is
+ * deliberately used as-is: opencode's own docs advertise plain-JSON config,
+ * and silently treating a JSONC file (comments) as replaceable would repeat
+ * the pre-existing clobber the merge exists to fix — so an unparseable file
+ * simply falls through to the pre-existing replace behavior, with the
+ * operator's own file untouched on disk.
+ */
+export declare function readOperatorConfig(configPath: string): Record<string, unknown> | null;
 /**
  * Resolve `SPF_OPENCODE_CMD` for THIS call, substituting a literal `{model}`
  * token with `model` — same mechanism and rationale as `agent_cc.ts`'s

package/dist/core/agent_opencode.js

@@ -65,9 +65,48 @@
  * docs, a target repo's OWN project-level `opencode.json` (if one exists)
  * merges at HIGHER precedence than the file `OPENCODE_CONFIG` points at.
  * That means a repo carrying its own `opencode.json` can silently override
- * (widen or narrow) the restriction this module writes — a real, documented
- * gap in this backend's tool-restriction guarantee, not a bug this module
- * can paper over from the outside.
+ * (widen or narrow) the restriction this module writes, and likewise
+ * override the OTel propagation headers described below — a real,
+ * documented gap in both guarantees, not a bug this module can paper over
+ * from the outside.
+ *
+ * OUTBOUND OTEL PROPAGATION [OFFICIAL config surface; best-effort
+ * behavior]: when `request.otel` is present (`agents.ts`'s `send()` threads
+ * it only when `observability.otel` is configured — absent otherwise, and
+ * this whole path is byte-identical to before), this module propagates
+ * trace context two ways:
+ *
+ *   1. `TRACEPARENT` on the child's env — the standard W3C env var, same
+ *      shape `agent_cc.ts` sets. No published statement confirms the
+ *      opencode CLI itself reads it (UNVERIFIED either way; set for parity
+ *      and for any opencode-spawned subprocess telemetry, at zero cost).
+ *   2. `traceparent` + `x-request-id` as STATIC provider headers in the
+ *      temp `opencode.json` (`provider.<id>.options.headers` — opencode's
+ *      documented per-provider options surface;
+ *      https://opencode.ai/docs/providers/). Static values are CORRECT
+ *      here, unlike the general case, because one `opencode run` subprocess
+ *      IS exactly one SPF agent call — the traceparent can never go stale
+ *      mid-run. This is the header that actually reaches the wire:
+ *      opencode's provider requests flow through the AI SDK, which honors
+ *      `options.headers` for that provider's requests. Subject to the
+ *      CONFIG PRECEDENCE limitation above: a repo's own `opencode.json` can
+ *      override these headers.
+ *
+ * `injectOtelEnv()` / `otelProviderHeaders()` / `tempConfigContents()` are
+ * exported pure functions so every fragment is unit-testable without
+ * spawning a real subprocess — same discipline as the rest of this module.
+ *
+ * OPERATOR-CONFIG MERGE: when a caller-provided `OPENCODE_CONFIG` already
+ * exists in the base env (operatorEnv() passthrough or an agent's
+ * env_allowlist) AND this module needs a temp config of its own (a tools:
+ * restriction or the propagation headers), the operator's file is READ and
+ * its contents MERGED into the temp file (SPF's own blocks win on
+ * conflict — see `mergeOperatorConfig()`), rather than the pre-existing
+ * replace-it-outright behavior. This closes a real regression class:
+ * enabling OTel or a tools: list on an already-configured workflow used to
+ * silently drop the operator's provider routing/credentials config. An
+ * unreadable or unparseable operator file falls back to the old replace
+ * behavior (and is left untouched on disk) rather than failing the run.
  *
  * `write`/`apply_patch` ARE GATED THROUGH `edit` [OFFICIAL]: opencode's own
  * docs say these two are not independent permission keys — both ride the
@@ -182,7 +221,7 @@
 import { spawn } from "node:child_process";
 import { createInterface } from "node:readline";
 import { randomUUID } from "node:crypto";
-import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
 import os from "node:os";
 import path from "node:path";
 import { UsageBreakdown, makeAgentResult } from "./data_types.js";
@@ -359,16 +398,114 @@ function permissionMapFor(toolNames) {
     return permission;
 }
 /**
- * `request.tools` is `null`/`undefined` -> no config file at all, `--auto`
- * alone (every tool usable, mirroring CC's `"default"`). An array (including
- * `[]`) -> a real temp `opencode.json` restricting to exactly what was
- * asked for. Caller is responsible for `rmSync`-ing `dir` when done — see
- * `run()`'s `finally`.
+ * Merges `TRACEPARENT` into `baseEnv` for outbound OTel propagation, or
+ * returns `baseEnv` UNCHANGED when `otel` is absent (the common case —
+ * `observability.otel` not configured). See the module doc comment's
+ * OUTBOUND OTEL PROPAGATION section: unlike `agent_cc.ts`'s sibling, no
+ * published statement confirms the opencode CLI itself reads this var, so
+ * the REAL propagation path is the temp-config headers below; this env var
+ * is set for parity at zero cost.
+ */
+export function injectOtelEnv(baseEnv, otel) {
+    if (!otel)
+        return baseEnv;
+    return { ...baseEnv, TRACEPARENT: otel.traceparent };
+}
+/**
+ * Builds the `provider.<id>.options.headers` fragment for a temp
+ * `opencode.json` (see the module doc comment's OUTBOUND OTEL PROPAGATION
+ * section for why static values are correct here). Returns `null` when
+ * `otel` is absent, or when the model id carries no `provider/` prefix —
+ * opencode's own `--model` vocabulary is documented as `provider/model-id`,
+ * but a bare model name has no provider id this module could key headers
+ * under, and guessing the wrong provider id would write a config block
+ * opencode merges onto a DIFFERENT provider than the one being called.
+ */
+export function otelProviderHeaders(model, otel) {
+    if (!otel)
+        return null;
+    const slash = model.indexOf("/");
+    if (slash <= 0)
+        return null;
+    return { provider: model.slice(0, slash), headers: { traceparent: otel.traceparent, "x-request-id": otel.x_request_id } };
+}
+/**
+ * Builds the JSON CONTENT of the temp `opencode.json`: a `permission` map
+ * when `toolNames` is an array (including `[]` — restrict to exactly those,
+ * mirroring CC's `--tools`), a `provider` block when `otelHeaders` is
+ * present, both when both apply. `null` when NEITHER applies — no config
+ * file needed at all, `--auto` alone (every tool usable, mirroring CC's
+ * `"default"`). Exported as its own pure function, same reasoning as
+ * `resolveOpencodeCmdSpec`: this shape is unit-testable without touching
+ * the filesystem or spawning a subprocess.
+ */
+export function tempConfigContents(toolNames, otelHeaders) {
+    const contents = {};
+    if (toolNames != null)
+        contents.permission = permissionMapFor(toolNames);
+    if (otelHeaders)
+        contents.provider = { [otelHeaders.provider]: { options: { headers: otelHeaders.headers } } };
+    return Object.keys(contents).length === 0 ? null : contents;
+}
+/**
+ * Recursive plain-object merge of SPF's own temp-config blocks ONTO the
+ * operator's existing `OPENCODE_CONFIG` contents — SPF's values win on
+ * conflicting plain-object leaves/deeper keys (a `tools:` restriction is a
+ * safety gate and the propagation headers are this call's own, so neither
+ * may be silently watered down), while every key the operator set and SPF
+ * never touches (provider `baseURL`/`apiKey`/model routing, MCP servers,
+ * agents) passes through intact. Arrays are replaced, not concatenated:
+ * this module never writes arrays itself, so a replaced array is always
+ * one of the OPERATOR's own values being deliberately overridden by an
+ * SPF block — the same "SPF wins on conflict" rule.
+ */
+export function mergeOperatorConfig(base, overlay) {
+    const out = { ...base };
+    for (const [key, value] of Object.entries(overlay)) {
+        const existing = out[key];
+        out[key] =
+            existing && typeof existing === "object" && !Array.isArray(existing) && value && typeof value === "object" && !Array.isArray(value)
+                ? mergeOperatorConfig(existing, value)
+                : value;
+    }
+    return out;
+}
+/**
+ * Reads the operator's existing `OPENCODE_CONFIG` file for merging, or
+ * returns `null` when there is nothing usable to merge with — file missing
+ * / unreadable / not valid JSON with a plain-object root. JSON.parse is
+ * deliberately used as-is: opencode's own docs advertise plain-JSON config,
+ * and silently treating a JSONC file (comments) as replaceable would repeat
+ * the pre-existing clobber the merge exists to fix — so an unparseable file
+ * simply falls through to the pre-existing replace behavior, with the
+ * operator's own file untouched on disk.
+ */
+export function readOperatorConfig(configPath) {
+    try {
+        const parsed = JSON.parse(readFileSync(configPath, "utf8"));
+        return parsed && typeof parsed === "object" && !Array.isArray(parsed) ? parsed : null;
+    }
+    catch {
+        return null;
+    }
+}
+/**
+ * Writes `contents` (see `tempConfigContents`) to a temp `opencode.json`.
+ * Caller is responsible for `rmSync`-ing `dir` when done — see `run()`'s
+ * `finally`.
  */
-function writeTempPermissionConfig(toolNames) {
+function writeTempConfig(contents) {
     const dir = mkdtempSync(path.join(os.tmpdir(), "spf-opencode-"));
     const configPath = path.join(dir, "opencode.json");
-    writeFileSync(configPath, JSON.stringify({ permission: permissionMapFor(toolNames) }, null, 2));
+    try {
+        writeFileSync(configPath, JSON.stringify(contents, null, 2));
+    }
+    catch (err) {
+        // This runs BEFORE run()'s try/finally — a failed write would otherwise
+        // leak the just-minted temp dir with nobody responsible for it.
+        rmSync(dir, { recursive: true, force: true });
+        throw err;
+    }
     return { dir, configPath };
 }
 // SPF's off|minimal|low|medium|high|xhigh|max -> opencode's --variant.
@@ -464,12 +601,27 @@ export async function run(request, onEvent, onSpawn, onExit) {
     const [cmd, ...cmdArgs] = cmdTokens;
     const fullArgs = [...cmdArgs, ...args];
     const baseEnv = request.env ?? operatorEnv();
-    // `request.tools` null/undefined -> every tool, no config file written at
-    // all (see writeTempPermissionConfig's own doc comment). An array
-    // (including []) -> a real temp opencode.json, pointed at via
-    // OPENCODE_CONFIG on the CHILD's env only — never mutates process.env.
-    const tempConfig = request.tools != null ? writeTempPermissionConfig(request.tools) : null;
-    const childEnv = tempConfig ? { ...baseEnv, OPENCODE_CONFIG: tempConfig.configPath } : baseEnv;
+    // `request.tools` null/undefined AND no otel config -> every tool, no
+    // config file written at all (see tempConfigContents' own doc comment).
+    // Otherwise -> a real temp opencode.json (tool restriction, OTel provider
+    // headers, or both), pointed at via OPENCODE_CONFIG on the CHILD's env
+    // only — never mutates process.env.
+    const otelHeaders = otelProviderHeaders(request.model, request.otel);
+    let configContents = tempConfigContents(request.tools, otelHeaders);
+    // A caller-provided OPENCODE_CONFIG (operatorEnv() passthrough or an
+    // agent's env_allowlist) is MERGED into the temp file, never replaced —
+    // SPF's own blocks win on conflict (see mergeOperatorConfig), but
+    // everything else the operator configured (provider baseURL/apiKey/model
+    // routing, MCP servers) survives. Without this, enabling OTel or a tools:
+    // list on an already-configured workflow would silently re-route or break
+    // agent calls.
+    if (configContents && baseEnv.OPENCODE_CONFIG) {
+        const operatorCfg = readOperatorConfig(baseEnv.OPENCODE_CONFIG);
+        if (operatorCfg)
+            configContents = mergeOperatorConfig(operatorCfg, configContents);
+    }
+    const tempConfig = configContents ? writeTempConfig(configContents) : null;
+    const childEnv = injectOtelEnv(tempConfig ? { ...baseEnv, OPENCODE_CONFIG: tempConfig.configPath } : baseEnv, request.otel);
     try {
         const child = spawn(cmd, fullArgs, { cwd: request.cwd, env: childEnv });
         // See the module doc comment's STDIN note — closed immediately to avoid