@tangle-network/agent-eval 0.21.0 → 0.22.0

package/CHANGELOG.md

@@ -1,5 +1,104 @@
 # Changelog
 
+## 0.22.0 — EvalCampaign + replay + always-valid + outcome calibration
+
+0.21 shipped the four capture-integrity primitives as opt-in. Every consumer still had to wire them by hand, and the bug class blueprint-agent reported (forgotten wiring → silent partial-capture) reappears the moment a new consumer adopts agent-eval cold. **0.22 makes the right thing the default path** — and adds three primitives that compound on top of standardized capture: replay-from-raw-events, anytime-valid sequential evaluation, and rubric predictive validity. The four primitives together turn agent-eval from a TS framework into research-grade evaluation infrastructure.
+
+### Added
+
+#### `runEvalCampaign` — capture integrity by construction
+
+Opinionated matrix runner that wires the four directives by construction. Inputs: variants, scenarios, seeds, an `LlmClientOptions`, factories for `TraceStore` and `RawProviderSink`, and a `runner(ctx)` callback. Outputs: per-cell `RunRecord[]`, `RunIntegrityReport[]`, optional `researchReport`, and a campaign fingerprint.
+
+- **Preflight:** `assertLlmRoute` is called once before any work, with `{ requireExplicitBaseUrl: true, requireAuth: true }` defaults. Misconfigured routes never burn a run.
+- **Per run:** the campaign constructs the `TraceStore`, `RawProviderSink`, and `TraceEmitter` (with `onRunComplete` hooks attached), then hands the runner an `LlmClientOptions` already pre-wired with `rawSink` + `traceContext`. The runner cannot accidentally call an LLM without capture.
+- **Run-completion:** `assertRunCaptured` runs after every `endRun` with `{ llmSpansMin: 1, requireRawCoverageOfLlmSpans: true, requireOutcome: true }` defaults. Failures are routed via `onIntegrityFailure: 'throw' | 'mark_failed' | 'log'` (default `'mark_failed'`).
+- **End of campaign:** if `report.comparator` is set, computes `researchReport` over the collected `RunRecord`s and embeds the campaign fingerprint + `preregistrationHash`.
+- **Concurrency:** local async worker pool, default 1, configurable via `concurrency`.
+- **Determinism:** the default `runId` generator is a stable hash of `(campaignId, variantId, scenarioId, seed)`, so re-running the same campaign produces the same ids; override `runId` for non-deterministic generation.
+
+Exported from the root barrel and the `@tangle-network/agent-eval/optimization` subpath: `runEvalCampaign`, `CampaignRunner`, `CampaignRunContext`, `CampaignRunOutcome`, `CampaignVariant`, `CampaignScenario`, `EvalCampaignOptions`, `EvalCampaignResult`, `FailedRun`, `CampaignIntegrityPolicy`, `CampaignFactoryParams`.
+
+#### Replay-from-raw-events
+
+Every campaign run is now a re-runnable artifact. `ReplayCache.fromSink(sink)` turns a populated `RawProviderSink` into a deterministic `(canonicalised request → cached response)` map; `createReplayFetch(cache)` returns a `fetch`-shaped function that satisfies `/chat/completions` calls out of the cache and passes other URLs through.
+
+```ts
+const cache = await ReplayCache.fromSink(yesterdayRawSink)
+const replayFetch = createReplayFetch(cache, { onMiss: 'fail-closed' })
+await callLlm(req, { ...llmOpts, fetch: replayFetch }) // zero LLM cost
+```
+
+Use cases:
+
+- Post-hoc judging — apply a new judge or scorer to last week's runs without burning a single token.
+- Determinism audits — replay a campaign and verify the responses match byte-for-byte.
+- Free judge calibration — run two judges on identical responses and measure agreement.
+
+`onMiss` is `'throw' | 'fallback' | 'fail-closed'`. The cache hashes a canonical projection (`model + messages + temperature + max_tokens|max_completion_tokens + response_format`) so insertion-order quirks don't cause spurious misses.
+
+Exported from root and `@tangle-network/agent-eval/traces`: `ReplayCache`, `createReplayFetch`, `iterateRawCalls`, `ReplayCacheEntry`, `ReplayCacheStats`, `ReplayFetchOptions`, `ReplayCacheMissError`.
+
+#### Always-valid sequential evaluation
+
+`pairedEvalueSequence(deltas, opts)` and `evaluateInterimReleaseConfidence({ deltaSeries })` ship the predictable plug-in betting martingale of Waudby-Smith & Ramdas (2024) for paired bounded outcomes, plus the empirical Bernstein confidence sequence of Howard et al. (2021) for the running mean. Both are *anytime-valid* — type-I error is bounded by α at every stopping time, no peeking penalty.
+
+```ts
+const verdict = evaluateInterimReleaseConfidence({
+  deltaSeries: [{ candidateId: 'cand', deltas }],
+  alpha: 0.05,
+  rope: { low: -0.02, high: 0.02 },
+})
+// → { recommendation: { decision: 'promote_now' | 'continue' | 'reject_now' | 'equivalent', candidateId } }
+```
+
+This closes the methodological hole flagged in the 0.21 methodology doc as out-of-scope. Consumers running rolling campaigns can now ship the moment evidence is decisive, stop-early on dead-on-arrival variants, and accumulate evidence across partial runs without spending the FDR budget. Tested under-the-null at α=0.05 on 100 synthetic series; false-rejection rate stays below the bound.
+
+Exported from root and `@tangle-network/agent-eval/reporting`: `pairedEvalueSequence`, `evaluateInterimReleaseConfidence`, `PairedEvalueOptions`, `PairedEvalueSequence`, `PairedEvalueStep`, `InterimReleaseConfidence`, `InterimReleaseConfidenceInput`, `SequentialDecision`.
+
+#### Rubric predictive validity
+
+`rubricPredictiveValidity({ runs, outcomes, outcomeMetrics })` joins canonical campaign `RunRecord`s to a `DeploymentOutcomeStore` and reports per-rubric Pearson + Spearman + bootstrap CI against each outcome metric. Verdict bucketing: `'load_bearing' | 'informative' | 'decorative'` based on `|spearman|`. **Without this loop every rubric is faith-based;** with it, you know which rubrics earn their promotion power and which are decoration.
+
+```ts
+const validity = await rubricPredictiveValidity({
+  runs: lastQuarterRuns,
+  outcomes: shipFlagOutcomeStore,
+  outcomeMetrics: ['revenue_lift', 'retention_30d', 'csat'],
+})
+for (const r of validity.ranked) {
+  console.log(`${r.rubric} → ${r.bestOutcome}: ρ=${r.spearman.toFixed(2)} (${r.verdict})`)
+}
+```
+
+Builds on the existing `correlationStudy` primitive but works directly off `RunRecord` (the canonical campaign artifact) rather than `Run` from a `TraceStore`, so it composes cleanly with `runEvalCampaign`'s output. Returns a per-rubric ranking + every (rubric, outcome) pair tested + a list of rubrics that produced no usable data.
+
+Exported from root and `@tangle-network/agent-eval/reporting`: `rubricPredictiveValidity`, `RubricOutcomePair`, `RubricRanking`, `RubricPredictiveValidityInput`, `RubricPredictiveValidityReport`. The existing `correlationStudy`, `OutcomeStore`, `InMemoryOutcomeStore`, `FileSystemOutcomeStore` continue to work unchanged.
+
+#### `NoopRawProviderSink.list()` returns `[]`
+
+Explicit opt-out from capture is no longer flagged by `assertRunCaptured` as `no_raw_sink`. Opt-out remains a deliberate choice; the campaign still requires the matching integrity overrides.
+
+### Why
+
+Every consumer that adopted agent-eval before 0.22 wrote their own matrix runner, and every one of them re-introduced the same forgettable wiring (raw sink, route guard, integrity assertion, analyst hook). 0.21 documented the pattern; 0.22 owns it. The four new primitives compound:
+
+- `runEvalCampaign` standardises the artifact (`RunRecord` + raw events + fingerprint).
+- Replay turns every past run into free training/validation data for new judges.
+- Sequential evaluation makes "ship-when-evidence-says-so" mathematically defensible.
+- Predictive validity converts evals from belief-based to outcome-anchored.
+
+`runMultiShotOptimization` remains the right primitive for trajectory-shaped GEPA optimization sweeps; `runPromptEvolution` for prompt + code evolution loops with sandbox pools; `runEvalCampaign` for the "compare N variants on M scenarios with K seeds and tell me which to ship" case that makes up the bulk of consumer evals.
+
+### References
+
+- Howard, S. R., Ramdas, A., McAuliffe, J., Sekhon, J. (2021). Time-uniform, nonparametric, nonasymptotic confidence sequences. *Annals of Statistics*, 49(2), 1055–1080.
+- Waudby-Smith, I., Ramdas, A. (2024). Estimating means of bounded random variables by betting. *JRSS B*, 86(1), 1–27.
+
+### Migration
+
+Existing consumers do not need to change. All four primitives are additive. Recommended path: on the next eval-runner refactor, replace hand-rolled matrix loops with `runEvalCampaign`. Use `evaluateInterimReleaseConfidence` for any campaign you run on a recurring cadence. Wire `rubricPredictiveValidity` once you have ≥ 30 deployment outcomes joinable by `runId`. Replay is a free win — once campaigns are running, every eval R&D loop drops to CPU-bound.
+
 ## 0.21.0 — capture integrity + launch-grade reporting
 
 This release closes the layer-1 gap a downstream consumer surfaced: better
@@ -74,7 +173,9 @@ surface.
 
 ### Python client
 
-Locked at `tangle-agent-eval==0.21.0` to match the npm package.
+The PyPI distribution renamed from `tangle-agent-eval` to **`agent-eval-rpc`**, and the import path from `tangle_agent_eval` to `agent_eval_rpc`. The new name accurately describes the package — it is a thin RPC client over the Node runtime, not a Python re-implementation of the eval logic — and the npm scope (`@tangle-network/agent-eval`) already provides the namespacing the `tangle-` prefix was substituting for. No prior PyPI version ever shipped under the old name (Trusted Publisher misconfiguration; see issue #40), so this rename is a clean first publish rather than a migration.
+
+Locked at `agent-eval-rpc==0.21.0` to match the npm package.
 
 ## 0.20.10 — hardening audit follow-up
 

package/README.md

@@ -116,6 +116,10 @@ import { renderReleaseReport } from '@tangle-network/agent-eval/reporting'
 | Fail loud if an eval would silently use the wrong route | `assertLlmRoute` |
 | Assert at run-end that the artifact is complete | `assertRunCaptured`, `throwIfRunIncomplete` |
 | Auto-execute the trace analyst on every run | `traceAnalystOnRunComplete` + `TraceEmitterOptions.onRunComplete` |
+| Run a matrix of variants × scenarios × seeds with capture integrity by construction | `runEvalCampaign` |
+| Re-judge / determinism-audit a past campaign for free | `ReplayCache`, `createReplayFetch` |
+| Ship-when-decisive with anytime-valid α across rolling looks | `pairedEvalueSequence`, `evaluateInterimReleaseConfidence` |
+| Tell load-bearing rubrics from decorative ones using deployment outcomes | `rubricPredictiveValidity` |
 | Model missing context separately from bad reasoning | `KnowledgeRequirement`, `KnowledgeBundle` |
 
 ### Capture integrity (0.21+)

package/dist/{chunk-WOK2RTWG.js → chunk-4W4NCYM2.js}

@@ -1,3 +1,8 @@
+import {
+  canonicalize,
+  hashJson
+} from "./chunk-6M774GY6.js";
+
 // src/trace/store.ts
 var InMemoryTraceStore = class {
   runs = /* @__PURE__ */ new Map();
@@ -497,119 +502,138 @@ function runToTraceId(run) {
   return cleaned.slice(0, 32).padEnd(32, "0");
 }
 
-// src/trace/integrity.ts
-var RunIntegrityError = class extends Error {
-  constructor(report) {
-    super(
-      `Run ${report.runId} failed integrity check: ${report.issues.map((i) => i.code).join(", ")}`
-    );
-    this.report = report;
-    this.name = "RunIntegrityError";
-  }
-  report;
+// src/replay.ts
+var ReplayCacheMissError = class extends Error {
+  constructor(url, requestKey2, message) {
+    super(message ?? `replay cache miss for ${url} (key=${requestKey2})`);
+    this.url = url;
+    this.requestKey = requestKey2;
+    this.name = "ReplayCacheMissError";
+  }
+  url;
+  requestKey;
 };
-async function assertRunCaptured(store, runId, expectations = {}) {
-  const issues = [];
-  const run = await store.getRun(runId);
-  if (!run) {
-    return {
-      ok: false,
-      runId,
-      llmSpanCount: 0,
-      judgeSpanCount: 0,
-      toolSpanCount: 0,
-      rawProviderEventCount: 0,
-      rawSpanCoverage: { covered: 0, total: 0 },
-      issues: [{ code: "no_run", message: `Run ${runId} not found in store.` }]
-    };
+var ReplayCache = class _ReplayCache {
+  byKey = /* @__PURE__ */ new Map();
+  orphans = 0;
+  byProvider = {};
+  byModel = {};
+  /**
+   * Build a cache from a sink's events. The sink must implement `list()`.
+   * Filter by `runId` / `spanId` to scope to a specific replay.
+   */
+  static async fromSink(sink, filter = {}) {
+    if (!sink.list) {
+      throw new Error("ReplayCache.fromSink: sink must implement list() to be replayable.");
+    }
+    const events = await sink.list(filter);
+    return _ReplayCache.fromEvents(events);
+  }
+  /** Build a cache from an in-memory event list. */
+  static async fromEvents(events) {
+    const cache = new _ReplayCache();
+    const groups = /* @__PURE__ */ new Map();
+    for (const e of events) {
+      const k = `${e.runId ?? ""}::${e.spanId ?? ""}::${e.attemptIndex}`;
+      const g = groups.get(k) ?? {};
+      if (e.direction === "request") g.req = e;
+      else g.res = e;
+      groups.set(k, g);
+    }
+    for (const g of groups.values()) {
+      if (!g.req) continue;
+      if (!g.res) {
+        cache.orphans += 1;
+        continue;
+      }
+      const key = await requestKey(g.req);
+      cache.byKey.set(key, { request: g.req, response: g.res });
+      cache.byProvider[g.req.provider] = (cache.byProvider[g.req.provider] ?? 0) + 1;
+      cache.byModel[g.req.model] = (cache.byModel[g.req.model] ?? 0) + 1;
+    }
+    return cache;
   }
-  const spans = await store.spans({ runId });
-  const llmSpans2 = spans.filter((s) => s.kind === "llm");
-  const judgeSpans2 = spans.filter((s) => s.kind === "judge");
-  const toolSpans2 = spans.filter((s) => s.kind === "tool");
-  const llmMin = expectations.llmSpansMin ?? 0;
-  const judgeMin = expectations.judgeSpansMin ?? 0;
-  const toolMin = expectations.toolSpansMin ?? 0;
-  if (llmSpans2.length < llmMin) {
-    issues.push({
-      code: "missing_llm_spans",
-      message: `Expected \u2265 ${llmMin} LLM spans, found ${llmSpans2.length}.`,
-      detail: { expected: llmMin, found: llmSpans2.length }
-    });
+  /** Number of cacheable (request, response) pairs in the cache. */
+  size() {
+    return this.byKey.size;
   }
-  if (judgeSpans2.length < judgeMin) {
-    issues.push({
-      code: "missing_judge_spans",
-      message: `Expected \u2265 ${judgeMin} judge spans, found ${judgeSpans2.length}.`,
-      detail: { expected: judgeMin, found: judgeSpans2.length }
-    });
+  stats() {
+    return {
+      total: this.byKey.size,
+      byProvider: { ...this.byProvider },
+      byModel: { ...this.byModel },
+      orphanRequests: this.orphans
+    };
   }
-  if (toolSpans2.length < toolMin) {
-    issues.push({
-      code: "missing_tool_spans",
-      message: `Expected \u2265 ${toolMin} tool spans, found ${toolSpans2.length}.`,
-      detail: { expected: toolMin, found: toolSpans2.length }
-    });
+  /**
+   * Look up a cached response by hashing the (model, messages, temperature,
+   * maxTokens, response_format) shape. Returns `undefined` on miss; the
+   * caller decides whether to throw, fall back to the network, or skip.
+   */
+  async lookup(requestBody) {
+    const key = await keyFromBody(requestBody);
+    return this.byKey.get(key);
   }
-  let rawEventCount = 0;
-  let coverage = { covered: 0, total: llmSpans2.length };
-  if (expectations.rawSink) {
-    if (!expectations.rawSink.list) {
-      issues.push({
-        code: "no_raw_sink",
-        message: "Provided rawSink does not implement list(); cannot verify capture."
-      });
-    } else {
-      const events = await expectations.rawSink.list({ runId });
-      rawEventCount = events.length;
-      const rawMin = expectations.rawProviderEventsMin ?? 1;
-      if (rawEventCount < rawMin) {
-        issues.push({
-          code: "missing_raw_events",
-          message: `Expected \u2265 ${rawMin} raw provider events, found ${rawEventCount}.`,
-          detail: { expected: rawMin, found: rawEventCount }
-        });
-      }
-      if (expectations.requireRawCoverageOfLlmSpans) {
-        const requestEventsBySpan = new Set(
-          events.filter((e) => e.direction === "request" && e.spanId).map((e) => e.spanId)
-        );
-        const orphaned = llmSpans2.filter((s) => !requestEventsBySpan.has(s.spanId));
-        coverage = { covered: llmSpans2.length - orphaned.length, total: llmSpans2.length };
-        if (orphaned.length > 0) {
-          issues.push({
-            code: "orphan_llm_span",
-            message: `${orphaned.length} LLM span(s) have no matching raw provider request event.`,
-            detail: { orphanedSpanIds: orphaned.map((s) => s.spanId) }
-          });
-        }
+};
+function createReplayFetch(cache, opts = {}) {
+  const onMiss = opts.onMiss ?? "throw";
+  const fallback = opts.fallbackFetch ?? globalThis.fetch?.bind(globalThis);
+  return (async (input, init) => {
+    const url = typeof input === "string" ? input : input instanceof URL ? input.toString() : input.url;
+    if (!/\/chat\/completions(?:[?#].*)?$/.test(url)) {
+      if (!fallback) throw new Error(`replay fetch: non-completions URL ${url} but no fallbackFetch configured`);
+      return fallback(input, init);
+    }
+    let bodyParsed;
+    if (init?.body && typeof init.body === "string") {
+      try {
+        bodyParsed = JSON.parse(init.body);
+      } catch {
       }
     }
-  } else if (expectations.requireRawCoverageOfLlmSpans || expectations.rawProviderEventsMin) {
-    issues.push({
-      code: "no_raw_sink",
-      message: "Raw coverage required but no rawSink supplied to the integrity check."
-    });
-  }
-  if (expectations.requireOutcome && (run.outcome === void 0 || run.outcome === null)) {
-    issues.push({
-      code: "missing_outcome",
-      message: `Run ${runId} has no outcome recorded.`
-    });
-  }
-  return {
-    ok: issues.length === 0,
-    runId,
-    llmSpanCount: llmSpans2.length,
-    judgeSpanCount: judgeSpans2.length,
-    toolSpanCount: toolSpans2.length,
-    rawProviderEventCount: rawEventCount,
-    rawSpanCoverage: coverage,
-    issues
-  };
+    const hit = bodyParsed === void 0 ? void 0 : await cache.lookup(bodyParsed);
+    if (hit) {
+      opts.onHit?.({ url, provider: hit.request.provider, model: hit.request.model });
+      const status = hit.response.statusCode ?? 200;
+      const headers = new Headers(Object.entries(hit.response.responseHeaders ?? { "Content-Type": "application/json" }));
+      const bodyText = typeof hit.response.responseBody === "string" ? hit.response.responseBody : JSON.stringify(hit.response.responseBody ?? {});
+      return new Response(bodyText, { status, headers });
+    }
+    opts.onMissNotify?.({ url, requestBody: bodyParsed });
+    if (onMiss === "throw") {
+      const key = bodyParsed === void 0 ? "<unparseable>" : await keyFromBody(bodyParsed);
+      throw new ReplayCacheMissError(url, key);
+    }
+    if (onMiss === "fail-closed") {
+      return new Response(JSON.stringify({ error: "replay_cache_miss" }), { status: 599 });
+    }
+    if (!fallback) throw new Error("replay fetch: onMiss=fallback but no fallbackFetch configured");
+    return fallback(input, init);
+  });
 }
-function throwIfRunIncomplete(report) {
-  if (!report.ok) throw new RunIntegrityError(report);
+async function* iterateRawCalls(sink, filter = {}) {
+  if (!sink.list) {
+    throw new Error("iterateRawCalls: sink must implement list().");
+  }
+  const events = await sink.list(filter);
+  const cache = await ReplayCache.fromEvents(events);
+  for (const entry of cache["byKey"].values()) yield entry;
+}
+async function requestKey(event) {
+  return keyFromBody(event.requestBody);
+}
+async function keyFromBody(body) {
+  if (body == null || typeof body !== "object") return hashJson({ raw: String(body) });
+  const b = body;
+  const reduced = canonicalize({
+    model: b.model ?? null,
+    messages: b.messages ?? null,
+    temperature: b.temperature ?? null,
+    max_tokens: b.max_tokens ?? null,
+    max_completion_tokens: b.max_completion_tokens ?? null,
+    response_format: b.response_format ?? null
+  });
+  return hashJson(reduced);
 }
 
 // src/trace-analyst/types.ts
@@ -1891,9 +1915,10 @@ export {
   redactValue,
   OTEL_AGENT_EVAL_SCOPE,
   exportRunAsOtlp,
-  RunIntegrityError,
-  assertRunCaptured,
-  throwIfRunIncomplete,
+  ReplayCacheMissError,
+  ReplayCache,
+  createReplayFetch,
+  iterateRawCalls,
   DEFAULT_TRACE_ANALYST_BUDGETS,
   TRACE_ANALYST_TRUNCATION_MARKER_PREFIX,
   OtlpFileTraceStore,
@@ -1917,4 +1942,4 @@ export {
   defaultTraceInsightPanel,
   buildTraceInsightPrompt
 };
-//# sourceMappingURL=chunk-WOK2RTWG.js.map
+//# sourceMappingURL=chunk-4W4NCYM2.js.map