@tangle-network/agent-eval 0.20.12 → 0.22.0

package/CHANGELOG.md

@@ -1,5 +1,182 @@
 # Changelog
 
+## 0.22.0 — EvalCampaign + replay + always-valid + outcome calibration
+
+0.21 shipped the four capture-integrity primitives as opt-in. Every consumer still had to wire them by hand, and the bug class blueprint-agent reported (forgotten wiring → silent partial-capture) reappears the moment a new consumer adopts agent-eval cold. **0.22 makes the right thing the default path** — and adds three primitives that compound on top of standardized capture: replay-from-raw-events, anytime-valid sequential evaluation, and rubric predictive validity. The four primitives together turn agent-eval from a TS framework into research-grade evaluation infrastructure.
+
+### Added
+
+#### `runEvalCampaign` — capture integrity by construction
+
+Opinionated matrix runner that wires the four directives by construction. Inputs: variants, scenarios, seeds, an `LlmClientOptions`, factories for `TraceStore` and `RawProviderSink`, and a `runner(ctx)` callback. Outputs: per-cell `RunRecord[]`, `RunIntegrityReport[]`, optional `researchReport`, and a campaign fingerprint.
+
+- **Preflight:** `assertLlmRoute` is called once before any work, with `{ requireExplicitBaseUrl: true, requireAuth: true }` defaults. Misconfigured routes never burn a run.
+- **Per run:** the campaign constructs the `TraceStore`, `RawProviderSink`, and `TraceEmitter` (with `onRunComplete` hooks attached), then hands the runner an `LlmClientOptions` already pre-wired with `rawSink` + `traceContext`. The runner cannot accidentally call an LLM without capture.
+- **Run-completion:** `assertRunCaptured` runs after every `endRun` with `{ llmSpansMin: 1, requireRawCoverageOfLlmSpans: true, requireOutcome: true }` defaults. Failures are routed via `onIntegrityFailure: 'throw' | 'mark_failed' | 'log'` (default `'mark_failed'`).
+- **End of campaign:** if `report.comparator` is set, computes `researchReport` over the collected `RunRecord`s and embeds the campaign fingerprint + `preregistrationHash`.
+- **Concurrency:** local async worker pool, default 1, configurable via `concurrency`.
+- **Determinism:** the default `runId` generator is a stable hash of `(campaignId, variantId, scenarioId, seed)`, so re-running the same campaign produces the same ids; override `runId` for non-deterministic generation.
+
+Exported from the root barrel and the `@tangle-network/agent-eval/optimization` subpath: `runEvalCampaign`, `CampaignRunner`, `CampaignRunContext`, `CampaignRunOutcome`, `CampaignVariant`, `CampaignScenario`, `EvalCampaignOptions`, `EvalCampaignResult`, `FailedRun`, `CampaignIntegrityPolicy`, `CampaignFactoryParams`.
+
+#### Replay-from-raw-events
+
+Every campaign run is now a re-runnable artifact. `ReplayCache.fromSink(sink)` turns a populated `RawProviderSink` into a deterministic `(canonicalised request → cached response)` map; `createReplayFetch(cache)` returns a `fetch`-shaped function that satisfies `/chat/completions` calls out of the cache and passes other URLs through.
+
+```ts
+const cache = await ReplayCache.fromSink(yesterdayRawSink)
+const replayFetch = createReplayFetch(cache, { onMiss: 'fail-closed' })
+await callLlm(req, { ...llmOpts, fetch: replayFetch }) // zero LLM cost
+```
+
+Use cases:
+
+- Post-hoc judging — apply a new judge or scorer to last week's runs without burning a single token.
+- Determinism audits — replay a campaign and verify the responses match byte-for-byte.
+- Free judge calibration — run two judges on identical responses and measure agreement.
+
+`onMiss` is `'throw' | 'fallback' | 'fail-closed'`. The cache hashes a canonical projection (`model + messages + temperature + max_tokens|max_completion_tokens + response_format`) so insertion-order quirks don't cause spurious misses.
+
+Exported from root and `@tangle-network/agent-eval/traces`: `ReplayCache`, `createReplayFetch`, `iterateRawCalls`, `ReplayCacheEntry`, `ReplayCacheStats`, `ReplayFetchOptions`, `ReplayCacheMissError`.
+
+#### Always-valid sequential evaluation
+
+`pairedEvalueSequence(deltas, opts)` and `evaluateInterimReleaseConfidence({ deltaSeries })` ship the predictable plug-in betting martingale of Waudby-Smith & Ramdas (2024) for paired bounded outcomes, plus the empirical Bernstein confidence sequence of Howard et al. (2021) for the running mean. Both are *anytime-valid* — type-I error is bounded by α at every stopping time, no peeking penalty.
+
+```ts
+const verdict = evaluateInterimReleaseConfidence({
+  deltaSeries: [{ candidateId: 'cand', deltas }],
+  alpha: 0.05,
+  rope: { low: -0.02, high: 0.02 },
+})
+// → { recommendation: { decision: 'promote_now' | 'continue' | 'reject_now' | 'equivalent', candidateId } }
+```
+
+This closes the methodological hole flagged in the 0.21 methodology doc as out-of-scope. Consumers running rolling campaigns can now ship the moment evidence is decisive, stop-early on dead-on-arrival variants, and accumulate evidence across partial runs without spending the FDR budget. Tested under-the-null at α=0.05 on 100 synthetic series; false-rejection rate stays below the bound.
+
+Exported from root and `@tangle-network/agent-eval/reporting`: `pairedEvalueSequence`, `evaluateInterimReleaseConfidence`, `PairedEvalueOptions`, `PairedEvalueSequence`, `PairedEvalueStep`, `InterimReleaseConfidence`, `InterimReleaseConfidenceInput`, `SequentialDecision`.
+
+#### Rubric predictive validity
+
+`rubricPredictiveValidity({ runs, outcomes, outcomeMetrics })` joins canonical campaign `RunRecord`s to a `DeploymentOutcomeStore` and reports per-rubric Pearson + Spearman + bootstrap CI against each outcome metric. Verdict bucketing: `'load_bearing' | 'informative' | 'decorative'` based on `|spearman|`. **Without this loop every rubric is faith-based;** with it, you know which rubrics earn their promotion power and which are decoration.
+
+```ts
+const validity = await rubricPredictiveValidity({
+  runs: lastQuarterRuns,
+  outcomes: shipFlagOutcomeStore,
+  outcomeMetrics: ['revenue_lift', 'retention_30d', 'csat'],
+})
+for (const r of validity.ranked) {
+  console.log(`${r.rubric} → ${r.bestOutcome}: ρ=${r.spearman.toFixed(2)} (${r.verdict})`)
+}
+```
+
+Builds on the existing `correlationStudy` primitive but works directly off `RunRecord` (the canonical campaign artifact) rather than `Run` from a `TraceStore`, so it composes cleanly with `runEvalCampaign`'s output. Returns a per-rubric ranking + every (rubric, outcome) pair tested + a list of rubrics that produced no usable data.
+
+Exported from root and `@tangle-network/agent-eval/reporting`: `rubricPredictiveValidity`, `RubricOutcomePair`, `RubricRanking`, `RubricPredictiveValidityInput`, `RubricPredictiveValidityReport`. The existing `correlationStudy`, `OutcomeStore`, `InMemoryOutcomeStore`, `FileSystemOutcomeStore` continue to work unchanged.
+
+#### `NoopRawProviderSink.list()` returns `[]`
+
+Explicit opt-out from capture is no longer flagged by `assertRunCaptured` as `no_raw_sink`. Opt-out remains a deliberate choice; the campaign still requires the matching integrity overrides.
+
+### Why
+
+Every consumer that adopted agent-eval before 0.22 wrote their own matrix runner, and every one of them re-introduced the same forgettable wiring (raw sink, route guard, integrity assertion, analyst hook). 0.21 documented the pattern; 0.22 owns it. The four new primitives compound:
+
+- `runEvalCampaign` standardises the artifact (`RunRecord` + raw events + fingerprint).
+- Replay turns every past run into free training/validation data for new judges.
+- Sequential evaluation makes "ship-when-evidence-says-so" mathematically defensible.
+- Predictive validity converts evals from belief-based to outcome-anchored.
+
+`runMultiShotOptimization` remains the right primitive for trajectory-shaped GEPA optimization sweeps; `runPromptEvolution` for prompt + code evolution loops with sandbox pools; `runEvalCampaign` for the "compare N variants on M scenarios with K seeds and tell me which to ship" case that makes up the bulk of consumer evals.
+
+### References
+
+- Howard, S. R., Ramdas, A., McAuliffe, J., Sekhon, J. (2021). Time-uniform, nonparametric, nonasymptotic confidence sequences. *Annals of Statistics*, 49(2), 1055–1080.
+- Waudby-Smith, I., Ramdas, A. (2024). Estimating means of bounded random variables by betting. *JRSS B*, 86(1), 1–27.
+
+### Migration
+
+Existing consumers do not need to change. All four primitives are additive. Recommended path: on the next eval-runner refactor, replace hand-rolled matrix loops with `runEvalCampaign`. Use `evaluateInterimReleaseConfidence` for any campaign you run on a recurring cadence. Wire `rubricPredictiveValidity` once you have ≥ 30 deployment outcomes joinable by `runId`. Replay is a free win — once campaigns are running, every eval R&D loop drops to CPU-bound.
+
+## 0.21.0 — capture integrity + launch-grade reporting
+
+This release closes the layer-1 gap a downstream consumer surfaced: better
+post-run statistics don't help if the underlying data wasn't captured. 0.21
+adds first-class raw provider-event capture, a fail-loud route guard, a
+run-completion integrity check, and run-complete hooks (with a trace-analyst
+auto-execution helper) so a direct matrix run produces complete forensics
+without out-of-band glue.
+
+### Added
+
+- **`RawProviderSink` (capture).** First-class persistence for HTTP-level
+  provider request / response / error payloads alongside the structured
+  `LlmSpan`. `InMemoryRawProviderSink`, `FileSystemRawProviderSink` (NDJSON,
+  rolls at 32 MiB), and `NoopRawProviderSink` ship in core. Default redactor
+  strips `Authorization` / `X-Api-Key` / `Cookie` headers and credential-shaped
+  body fields (`apiKey`, `bearer`, `password`, `secret`, `token`); redacted
+  paths are recorded on `event.redactedFields` so a reviewer can see what was
+  stripped without exposing values. Wired into `callLlm` via
+  `LlmClientOptions.rawSink` — every retry attempt produces a `request` and
+  either a `response` or `error` event with the attempt index attached.
+- **`assertLlmRoute` (route guard).** Pure function that throws
+  `LlmRouteAssertionError` when the configured client doesn't match the
+  caller's route requirements: `requireExplicitBaseUrl`, `allowedBaseUrls`,
+  `blockedBaseUrls`, `requireAuth`, `expectedProvider`. Designed for the
+  matrix-runner preflight — fail loud at the boundary instead of silently
+  falling back to the public/free-tier router.
+- **`assertRunCaptured` (integrity check).** Read-only check on
+  `(store, runId, expectations)` that returns a structured
+  `RunIntegrityReport` with issue codes (`missing_llm_spans`,
+  `missing_raw_events`, `orphan_llm_span`, `no_raw_sink`, `missing_outcome`,
+  …). Pair with the new `requireRawCoverageOfLlmSpans` to assert every
+  `LlmSpan` has a matching raw `request` event. Use directly or via
+  `throwIfRunIncomplete` for strict mode.
+- **`onRunComplete` hooks on `TraceEmitter`.** New
+  `TraceEmitterOptions.onRunComplete` array fires after `endRun` / `abortRun`
+  with full run context (run id, outcome, status, store, emitter). Errors are
+  swallowed and recorded as `log` events by default; opt into propagation via
+  `hookErrors: 'throw'`. `addRunCompleteHook` attaches hooks after construction.
+- **`traceAnalystOnRunComplete` factory.** Drop-in run-complete hook that
+  runs `analyzeTraces` after each run and persists the result. Resolves the
+  "trace analyst never ran on this matrix sweep" complaint by making
+  auto-execution declarative.
+- **`researchReport`** — executive research-report layer for coding-vertical
+  benchmark runs (originally landed in #34, elevated in #35). Composes
+  `summaryTable`, `paretoChart`, `gainHistogram`, held-out gate decisions,
+  and optional `failureClusterView` output into one structured artifact:
+  promote / hold / equivalent / reject / needs-more-data guidance with
+  rationale, risks, next actions, markdown, HTML, and JSON chart specs.
+  - Decisions are made on paired evidence — never on marginal means alone.
+  - ROPE (Region of Practical Equivalence) supported via the `rope` option.
+  - Bayesian-bootstrap-style `Pr(Δ>0)` and `Pr(Δ∈ROPE)` summaries (Rubin 1981).
+  - Per-candidate minimum detectable paired effect via `pairedMde`.
+  - SHA-256 `runFingerprint` and optional `preregistrationHash` linking a
+    signed `HypothesisManifest`.
+  - Embedded methodology + `docs/research-report-methodology.md` companion.
+- **`pairedMde`** in `power-analysis`: closed-form minimum detectable paired
+  effect (inverse to the paired-t / sign-rank power formula).
+
+### Changed
+
+- `researchReport` is async (uses Web Crypto via `hashJson` for the run
+  fingerprint).
+- Default `researchReport.minPairs` is 20 (soft floor); hard floor of 6 is
+  enforced regardless via `RESEARCH_REPORT_HARD_PAIR_FLOOR`.
+
+### Wire-protocol consumers
+
+No wire-protocol changes. The new capture / integrity / hook primitives are
+TypeScript-only; cross-language consumers continue to use the existing RPC
+surface.
+
+### Python client
+
+The PyPI distribution renamed from `tangle-agent-eval` to **`agent-eval-rpc`**, and the import path from `tangle_agent_eval` to `agent_eval_rpc`. The new name accurately describes the package — it is a thin RPC client over the Node runtime, not a Python re-implementation of the eval logic — and the npm scope (`@tangle-network/agent-eval`) already provides the namespacing the `tangle-` prefix was substituting for. No prior PyPI version ever shipped under the old name (Trusted Publisher misconfiguration; see issue #40), so this rename is a clean first publish rather than a migration.
+
+Locked at `agent-eval-rpc==0.21.0` to match the npm package.
+
 ## 0.20.10 — hardening audit follow-up
 
 ### Fixed

package/README.md

@@ -111,9 +111,51 @@ import { renderReleaseReport } from '@tangle-network/agent-eval/reporting'
 | Compare prompt/tool/retrieval policies over full trajectories | `runMultiShotOptimization` |
 | Gate releases with paired evidence and holdouts | `evaluateReleaseConfidence`, `HeldOutGate` |
 | Explain regressions across trace corpora | `TraceAnalyst` / `analyzeTraces` |
-| Report a launch decision | `renderReleaseReport`, `summaryTable`, `paretoChart`, `gainHistogram` |
+| Report a launch decision | `renderReleaseReport`, `researchReport`, `summaryTable`, `paretoChart`, `gainHistogram` |
+| Capture every provider HTTP request / response for forensics | `RawProviderSink`, `LlmClientOptions.rawSink` |
+| Fail loud if an eval would silently use the wrong route | `assertLlmRoute` |
+| Assert at run-end that the artifact is complete | `assertRunCaptured`, `throwIfRunIncomplete` |
+| Auto-execute the trace analyst on every run | `traceAnalystOnRunComplete` + `TraceEmitterOptions.onRunComplete` |
+| Run a matrix of variants × scenarios × seeds with capture integrity by construction | `runEvalCampaign` |
+| Re-judge / determinism-audit a past campaign for free | `ReplayCache`, `createReplayFetch` |
+| Ship-when-decisive with anytime-valid α across rolling looks | `pairedEvalueSequence`, `evaluateInterimReleaseConfidence` |
+| Tell load-bearing rubrics from decorative ones using deployment outcomes | `rubricPredictiveValidity` |
 | Model missing context separately from bad reasoning | `KnowledgeRequirement`, `KnowledgeBundle` |
 
+### Capture integrity (0.21+)
+
+Launch-grade benchmark runs need four things that are easy to forget in glue
+code: (1) raw HTTP capture alongside the structured spans so a reviewer can
+verify which route answered, (2) a preflight assertion that the configured
+client points at the intended provider, (3) a run-end assertion that the
+expected events were actually written, and (4) auto-execution of the trace
+analyst as part of the run lifecycle. The wiring fits in a few lines:
+
+```ts
+import {
+  TraceEmitter, FileSystemRawProviderSink, callLlm, assertLlmRoute,
+  assertRunCaptured, throwIfRunIncomplete,
+} from '@tangle-network/agent-eval'
+import { traceAnalystOnRunComplete } from '@tangle-network/agent-eval/traces'
+
+const sink = new FileSystemRawProviderSink({ dir: `${workDir}/raw-events` })
+assertLlmRoute(llmOpts, { requireExplicitBaseUrl: true, allowedBaseUrls, requireAuth: true })
+
+const emitter = new TraceEmitter(store, {
+  onRunComplete: [traceAnalystOnRunComplete({ analyze: analystOpts, save })],
+})
+await emitter.startRun(/* ... */)
+// LLM calls flow through callLlm with `{ rawSink: sink, traceContext: { runId, spanId } }`.
+await emitter.endRun({ pass, score })
+
+throwIfRunIncomplete(await assertRunCaptured(store, emitter.runId, {
+  llmSpansMin: 1, rawSink: sink, requireRawCoverageOfLlmSpans: true, requireOutcome: true,
+}))
+```
+
+Directives, rationale, and shipped-bug context are in
+[`SKILL.md` § Capture integrity](./.claude/skills/agent-eval/SKILL.md#capture-integrity-required-for-launch-grade-adoption).
+
 ## Examples
 
 Runnable examples live in

package/dist/{chunk-KWUAAIHR.js → chunk-4W4NCYM2.js}

@@ -1,3 +1,8 @@
+import {
+  canonicalize,
+  hashJson
+} from "./chunk-6M774GY6.js";
+
 // src/trace/store.ts
 var InMemoryTraceStore = class {
   runs = /* @__PURE__ */ new Map();
@@ -497,6 +502,140 @@ function runToTraceId(run) {
   return cleaned.slice(0, 32).padEnd(32, "0");
 }
 
+// src/replay.ts
+var ReplayCacheMissError = class extends Error {
+  constructor(url, requestKey2, message) {
+    super(message ?? `replay cache miss for ${url} (key=${requestKey2})`);
+    this.url = url;
+    this.requestKey = requestKey2;
+    this.name = "ReplayCacheMissError";
+  }
+  url;
+  requestKey;
+};
+var ReplayCache = class _ReplayCache {
+  byKey = /* @__PURE__ */ new Map();
+  orphans = 0;
+  byProvider = {};
+  byModel = {};
+  /**
+   * Build a cache from a sink's events. The sink must implement `list()`.
+   * Filter by `runId` / `spanId` to scope to a specific replay.
+   */
+  static async fromSink(sink, filter = {}) {
+    if (!sink.list) {
+      throw new Error("ReplayCache.fromSink: sink must implement list() to be replayable.");
+    }
+    const events = await sink.list(filter);
+    return _ReplayCache.fromEvents(events);
+  }
+  /** Build a cache from an in-memory event list. */
+  static async fromEvents(events) {
+    const cache = new _ReplayCache();
+    const groups = /* @__PURE__ */ new Map();
+    for (const e of events) {
+      const k = `${e.runId ?? ""}::${e.spanId ?? ""}::${e.attemptIndex}`;
+      const g = groups.get(k) ?? {};
+      if (e.direction === "request") g.req = e;
+      else g.res = e;
+      groups.set(k, g);
+    }
+    for (const g of groups.values()) {
+      if (!g.req) continue;
+      if (!g.res) {
+        cache.orphans += 1;
+        continue;
+      }
+      const key = await requestKey(g.req);
+      cache.byKey.set(key, { request: g.req, response: g.res });
+      cache.byProvider[g.req.provider] = (cache.byProvider[g.req.provider] ?? 0) + 1;
+      cache.byModel[g.req.model] = (cache.byModel[g.req.model] ?? 0) + 1;
+    }
+    return cache;
+  }
+  /** Number of cacheable (request, response) pairs in the cache. */
+  size() {
+    return this.byKey.size;
+  }
+  stats() {
+    return {
+      total: this.byKey.size,
+      byProvider: { ...this.byProvider },
+      byModel: { ...this.byModel },
+      orphanRequests: this.orphans
+    };
+  }
+  /**
+   * Look up a cached response by hashing the (model, messages, temperature,
+   * maxTokens, response_format) shape. Returns `undefined` on miss; the
+   * caller decides whether to throw, fall back to the network, or skip.
+   */
+  async lookup(requestBody) {
+    const key = await keyFromBody(requestBody);
+    return this.byKey.get(key);
+  }
+};
+function createReplayFetch(cache, opts = {}) {
+  const onMiss = opts.onMiss ?? "throw";
+  const fallback = opts.fallbackFetch ?? globalThis.fetch?.bind(globalThis);
+  return (async (input, init) => {
+    const url = typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url;
+    if (!/\/chat\/completions(?:[?#].*)?$/.test(url)) {
+      if (!fallback) throw new Error(`replay fetch: non-completions URL ${url} but no fallbackFetch configured`);
+      return fallback(input, init);
+    }
+    let bodyParsed;
+    if (init?.body && typeof init.body === "string") {
+      try {
+        bodyParsed = JSON.parse(init.body);
+      } catch {
+      }
+    }
+    const hit = bodyParsed === void 0 ? void 0 : await cache.lookup(bodyParsed);
+    if (hit) {
+      opts.onHit?.({ url, provider: hit.request.provider, model: hit.request.model });
+      const status = hit.response.statusCode ?? 200;
+      const headers = new Headers(Object.entries(hit.response.responseHeaders ?? { "Content-Type": "application/json" }));
+      const bodyText = typeof hit.response.responseBody === "string" ? hit.response.responseBody : JSON.stringify(hit.response.responseBody ?? {});
+      return new Response(bodyText, { status, headers });
+    }
+    opts.onMissNotify?.({ url, requestBody: bodyParsed });
+    if (onMiss === "throw") {
+      const key = bodyParsed === void 0 ? "<unparseable>" : await keyFromBody(bodyParsed);
+      throw new ReplayCacheMissError(url, key);
+    }
+    if (onMiss === "fail-closed") {
+      return new Response(JSON.stringify({ error: "replay_cache_miss" }), { status: 599 });
+    }
+    if (!fallback) throw new Error("replay fetch: onMiss=fallback but no fallbackFetch configured");
+    return fallback(input, init);
+  });
+}
+async function* iterateRawCalls(sink, filter = {}) {
+  if (!sink.list) {
+    throw new Error("iterateRawCalls: sink must implement list().");
+  }
+  const events = await sink.list(filter);
+  const cache = await ReplayCache.fromEvents(events);
+  for (const entry of cache["byKey"].values()) yield entry;
+}
+async function requestKey(event) {
+  return keyFromBody(event.requestBody);
+}
+async function keyFromBody(body) {
+  if (body == null || typeof body !== "object") return hashJson({ raw: String(body) });
+  const b = body;
+  const reduced = canonicalize({
+    model: b.model ?? null,
+    messages: b.messages ?? null,
+    temperature: b.temperature ?? null,
+    max_tokens: b.max_tokens ?? null,
+    max_completion_tokens: b.max_completion_tokens ?? null,
+    response_format: b.response_format ?? null
+  });
+  return hashJson(reduced);
+}
+
 // src/trace-analyst/types.ts
 var DEFAULT_TRACE_ANALYST_BUDGETS = {
   perCallByteCeiling: 15e4,
@@ -1468,6 +1607,43 @@ function normalizeRecordArray(value) {
   return value.map((item) => item && typeof item === "object" ? { ...item } : { value: item });
 }
 
+// src/trace-analyst/hook.ts
+var DEFAULT_QUESTION = "Summarise what happened in this run. Surface any failure modes, surprising findings, or evidence that the run's verdict is wrong.";
+function traceAnalystOnRunComplete(opts) {
+  return async (ctx) => {
+    if (opts.shouldRun && !opts.shouldRun(ctx)) return;
+    const source = opts.analyze.source;
+    if (source === void 0) {
+      await ctx.store.appendEvent({
+        eventId: `analyst-skip-${ctx.runId}`,
+        runId: ctx.runId,
+        kind: "log",
+        timestamp: Date.now(),
+        payload: { source: "trace_analyst_hook", reason: "no source configured" }
+      });
+      return;
+    }
+    const result = await analyzeTraces(
+      { question: opts.question ?? DEFAULT_QUESTION },
+      { ...opts.analyze, source }
+    );
+    if (opts.save) await opts.save(result, ctx);
+    if (opts.gateOn && !opts.gateOn(result, ctx)) {
+      await ctx.store.appendEvent({
+        eventId: `analyst-gate-${ctx.runId}`,
+        runId: ctx.runId,
+        kind: "log",
+        timestamp: Date.now(),
+        payload: {
+          source: "trace_analyst_hook",
+          reason: "analyst_gate_failed",
+          findings: result.findings
+        }
+      });
+    }
+  };
+}
+
 // src/trace-analyst/insights.ts
 var DOMAIN_STOP_WORDS = /* @__PURE__ */ new Set([
   "and",
@@ -1739,6 +1915,10 @@ export {
   redactValue,
   OTEL_AGENT_EVAL_SCOPE,
   exportRunAsOtlp,
+  ReplayCacheMissError,
+  ReplayCache,
+  createReplayFetch,
+  iterateRawCalls,
   DEFAULT_TRACE_ANALYST_BUDGETS,
   TRACE_ANALYST_TRUNCATION_MARKER_PREFIX,
   OtlpFileTraceStore,
@@ -1751,6 +1931,7 @@ export {
   buildTraceAnalystTools,
   traceAnalystFunctionGroup,
   analyzeTraces,
+  traceAnalystOnRunComplete,
   tokenizeDomainWords,
   inferDomainKeywords,
   domainEvidencePattern,
@@ -1761,4 +1942,4 @@ export {
   defaultTraceInsightPanel,
   buildTraceInsightPrompt
 };
-//# sourceMappingURL=chunk-KWUAAIHR.js.map
+//# sourceMappingURL=chunk-4W4NCYM2.js.map