@tangle-network/agent-eval 0.40.2 → 0.40.4

package/docs/design/product-self-improvement-loop.md

@@ -0,0 +1,146 @@
+# The product self-improvement loop — the finish-line target
+
+This is the **end state** every Tangle product agent (gtm, legal, tax, creative,
+agent-builder, blueprint, physim) converges to. It is the target the consumer
+migrations build toward — not a 1:1 port of whatever eval/improvement code a
+product has today.
+
+**Thesis.** A product agent is *one closed, automated self-improvement loop*
+that makes the agent measurably better over time while humans only approve
+PRs. A product should NOT have a "production loop" *and* a pile of `eval/*`
+CLIs *and* bespoke optimization orchestration. It has **one** loop, composed
+from the substrate. Everything else is deleted.
+
+Primitives reference: [`primitives-integration-spec.md`](./primitives-integration-spec.md).
+Engine internals: [`self-improvement-engine.md`](./self-improvement-engine.md).
+
+---
+
+## The loop (7 steps, exact substrate composition)
+
+```
+1. SAMPLE the eval matrix
+   scenarios = cartesian(
+     profileVariants,        // the surface(s) under test: baseline + candidates
+     productScenarios,       // the hard product tasks (gtm: attribution honesty, …)
+     personas,               // simulated users / drivers
+   )  ∪  productionFailures  // real failures pulled from the LabeledScenarioStore
+                             // (the flywheel: prod traces become eval scenarios)
+
+2. MEASURE — runCampaign
+   dispatch(scenario) = runMultishot({           // the multi-turn challenging flow
+     persona: scenario.persona,                  //   driver = simulated user
+     profile: productAgentProfile(surface),      //   worker = the agent under test
+     shape:   scenario.flow,                     //   the real task, many turns
+     tools:   productTools,                      //   real tools, real side-effects
+   }) → transcript artifact
+   judges = product ensemble (domain dimensions) → scorecard + bootstrap CIs
+   labeledStore: capture EVERY cell (scenario, artifact, score, source) → the dataset
+
+3. ANALYZE — trace analysts (runAnalystLoop / AnalystRegistry)
+   read the campaign traces → a research report (failure modes, why, where).
+   This REPLACES bespoke "failure clustering": the analyst is the richer,
+   LLM-driven version of "what should we improve and why".
+
+4. IMPROVE — runImprovementLoop( improvementDriver + agenticGenerator )
+   driver.propose({ report, dataset, … }) → candidate surfaces.
+   The agentic generator runs a coding harness in a worktree, reading the
+   report + the codebase, making REAL product changes — prompt, tools, AND
+   code — not just an addendum string. Each candidate is measured on a
+   HELD-OUT slice of the matrix.
+
+5. GATE — defaultProductionGate (+ domain gates, composed)
+   heldout-delta + budget + red-team + reward-hacking + canary, plus any
+   product-specific gate (e.g. anti-fabrication) and an overfit-gap check.
+   Verdict ∈ ship | hold | need_more_work | model_ceiling | arch_ceiling.
+
+6. PROMOTE — openAutoPr
+   the winning worktree → a PR against the product repo. Human approves → ships.
+   (autoOnPromote: 'pr'. Live self-mutation is deferred behind the full safety
+   stack.)
+
+7. LOOP
+   the shipped, improved agent runs in production → emits traces → the dataset
+   grows → back to (1). The loop is scheduled (cron) and/or triggered when the
+   analyst report crosses a severity threshold. Autonomous between PR approvals.
+```
+
+**One entry point, no new abstractions.** A product exposes a single
+`run<Product>ImprovementCycle()` that *composes* the substrate primitives
+above. It does NOT define `runFooPromptEvolution`, `FooOptimizer`,
+`FooProductionLoop`, etc. The substrate carries every name; the product only
+wires its domain pieces into the seams.
+
+---
+
+## What each product OWNS vs DELETES vs COMPOSES
+
+**OWNS (domain — stays, this is the product's value):**
+- `productScenarios` — the hard tasks the agent must handle.
+- `personas` — the simulated users that drive the multi-shot flows.
+- `judges` / rubrics / dimension weights — how "good" is defined.
+- `productTools` — the real tools the agent uses.
+- deterministic checks (anti-slop, format, forbidden-claim) — fast pre-judges.
+- domain gates (e.g. anti-fabrication) — composed into the gate.
+
+**DELETES (orchestration the substrate now owns):**
+- every `for (gen of generations)` mutate→score→select loop.
+- bespoke prompt-evolution / production-loop / analyst-loop wrappers.
+- trial-matrix construction, frontier tracking, seed plumbing, manifest
+  hashing, cell caching, scorecard aggregation, CI math.
+- PR-opening scaffolding, worktree git plumbing.
+- parallel `eval/*` CLIs that each re-implement a slice of the above.
+
+**COMPOSES (the substrate, in the one cycle):**
+- `runCampaign` (matrix measurement) · `runMultishot` (the dispatch flow) ·
+  `FsLabeledScenarioStore` (dataset) · analysts (report) ·
+  `runImprovementLoop` + `improvementDriver` + `agenticGenerator` (improve) ·
+  `defaultProductionGate` + `composeGate` (gate) · `openAutoPr` (promote).
+
+---
+
+## Definition of done (a product is "at the finish line" when)
+
+1. **One cycle, one entry.** A single `run<Product>ImprovementCycle()` composes
+   the substrate; the old eval/improvement systems are deleted, not coexisting.
+2. **Matrix eval is real.** `dispatch` runs genuine multi-shot persona↔agent
+   flows with real tools — not single-turn projections, not stubbed workers
+   (non-zero token usage is asserted).
+3. **The dataset is fed.** Every cell captures to `LabeledScenarioStore` with
+   correct provenance; production failures flow back in as scenarios.
+4. **Improvement is code-real.** The agentic generator produces worktree
+   changes (prompt/tools/code), measured on holdout — not just addendum-string
+   mutation.
+5. **The gate is honest.** Composed `defaultProductionGate` + domain gates +
+   overfit-gap; fails closed; holdout never overlaps train.
+6. **Promotion is a PR.** `openAutoPr` opens it; a human approves; nothing
+   auto-deploys.
+7. **It's scheduled + triggered.** Runs on cadence and/or when the analyst
+   report crosses severity; autonomous between approvals.
+8. **Tests + a real proof run.** Contract tests assert the wiring; one real
+   end-to-end cycle produces a scorecard and (on a shipping gate) a PR.
+
+Anything short of this is mid-migration, not done.
+
+---
+
+## gtm-agent — the worked instantiation (first reference build)
+
+| Loop step | gtm wiring |
+|---|---|
+| SAMPLE | profile variants of `OPERATOR_CEO_SYSTEM_PROMPT` + addendum; `GTM_LOOP_HOLDOUT_SCENARIOS` + `eval/business-owner/personas.json`; production failures from the trace store |
+| MEASURE | `dispatch` = `runMultishot(persona ↔ gtm-agent via runChatThroughRuntime, real tools)`; judges = the 3-model ensemble (`attribution_honesty`, `proposal_grounding`) + canonical 12-dim |
+| ANALYZE | trace analysts over the campaign traces → report (supersedes `FailureClusterConfig` clustering) |
+| IMPROVE | `improvementDriver` + `agenticGenerator` (claude harness) edits prompt/tools/code in a worktree, fed the report |
+| GATE | `composeGate(defaultProductionGate, antiFabricationGate, overfitGapGate)` |
+| PROMOTE | `openAutoPr` → PR against `tangle-network/gtm-agent` |
+
+**Deleted:** `eval/run-prompt-evolution.ts`, `eval/analyst-loop.ts`,
+`eval/optimization-campaign.ts`, `scripts/evals/*`, the orchestration body of
+`production-loop/index.ts` and `eval/canonical.ts`.
+**Kept:** scenarios, personas, judges, tools, deterministic checks, the
+`composeProductionLoopSystemPrompt` wiring.
+**Result:** one `runGtmImprovementCycle()`; ~3–4k LOC of scattered orchestration
+gone, replaced by a substrate composition.
+
+This gtm build is the reference the other six products copy.

package/docs/design/self-improvement-engine.md

@@ -57,17 +57,27 @@ run that produces traces, and the analysts that turn traces into a report.
 ## `propose()` — the plan step, recursively agentic
 
 `propose()` does NOT run the worker and does NOT measure. It returns N
-candidate surfaces to measure next. *How* it produces them is per-driver and
-spans a cost spectrum:
+candidate surfaces to measure next.
 
-| Driver | `propose()` mechanism | Sandbox? | Output |
+There is ONE driver, not several. agent-runtime ships a single
+`improvementDriver` that implements this contract and owns the candidate
+lifecycle (worktree create → generate → finalize/discard, × `populationSize`);
+it delegates the only thing that varies — *how* a candidate change is produced
+— to a pluggable `CandidateGenerator`. The generators span a cost spectrum:
+
+| Generator | mechanism | sandbox? | output |
 |---|---|---|---|
-| `evolutionaryDriver` | mutate current surface text into N variants | no | `string[]` |
-| `analystDriver` (reflective) | LLM reads the report → drafts edits | LLM call | `string[]` / `CodeSurface[]` |
-| `autoresearchDriver` (code-tier) | **full sandbox runLoop** (≤ `maxImprovementShots`) reads report+traces+codebase → commits in one worktree | **yes** | `CodeSurface[]` |
+| `evolutionaryDriver` (agent-eval) | mutate current surface text into N variants | no | `string[]` |
+| `reflectiveGenerator` (agent-runtime) | LLM drafts patches from the report → applies them | LLM call | `CodeSurface` |
+| `agenticGenerator` (agent-runtime) | a real coding harness in the worktree (≤ `maxImprovementShots`) reads report+codebase → edits in place | **yes** | `CodeSurface` |
+
+`evolutionaryDriver` is a standalone `ImprovementDriver` (pure, agent-eval).
+The reflective + agentic paths are two *generators* of the one
+`improvementDriver` — the same operation at two settings of the cost dial, not
+two separate drivers.
 
-The recursion: generating *one* candidate (autoresearch `propose`) is itself a
-driver↔worker-in-a-sandbox loop, nested inside the *measurement* of that
+The recursion: generating *one* candidate (the agentic generator) is itself a
+harness-agent editing a worktree, nested inside the *measurement* of that
 candidate (Phase 4), nested inside the improvement loop. "A loop whose step
 contains a loop."
 
@@ -90,8 +100,7 @@ So:
 | **VCS-pluggable worktree adapter** | agent-eval | pure git/FS, no sandbox; produces `CodeSurface` |
 | `runOptimization` / `runImprovementLoop` | agent-eval | driver-agnostic loop body + gated shell |
 | `defaultProductionGate` | agent-eval | measurement-side safety |
-| **`autoresearchDriver`** (sandbox-spawning `propose`) | agent-runtime | needs the sandbox SDK + `runLoop` |
-| `analystDriver` (wraps the improvement adapter) | agent-runtime | depends on `runAnalystLoop` machinery |
+| **`improvementDriver`** + `reflectiveGenerator` / `agenticGenerator` | agent-runtime | needs the coding-harness runner + worktree on a real FS |
 | trace analysts / `runAnalystLoop` (Phase 2) | agent-runtime | runs agents to analyze |
 
 ## The worktree adapter (VCS-pluggable)
@@ -119,12 +128,13 @@ becomes the PR branch.
 
 ## Build sequence
 
-1. **agent-eval 0.40.2**: widen `propose()` input (additive optional
-   `report` / `traces` / `dataset` / `maxImprovementShots`); add the
-   VCS-pluggable worktree adapter with a git impl; multi-sink trace fan-out
-   helper.
-2. **agent-runtime 0.25.0**: `analystDriver` (wraps the improvement adapter,
-   fed the Phase-2 report); `autoresearchDriver` (sandbox runLoop `propose`);
-   default-on multi-sink tracing in `handleChatTurn`.
+1. **agent-eval 0.40.2** ✅: widen `propose()` input (additive optional
+   `report` / `dataset` / `maxImprovementShots`); VCS-pluggable worktree
+   adapter with a git impl.
+2. **agent-runtime 0.25.0** ✅: `improvementDriver` (the one driver) +
+   `reflectiveGenerator` (shots=1, no sandbox) + `agenticGenerator` (coding
+   harness in the worktree, ≤ `maxImprovementShots`), all fed the Phase-2
+   report. Default tracing was descoped — the flywheel's production capture is
+   already served by agent-runtime's OTEL export + `runCampaign`'s labeledStore.
 3. Wire one consumer end-to-end (Phase 4 of the broader rollout), prove it,
    then fan out.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tangle-network/agent-eval",
-  "version": "0.40.2",
+  "version": "0.40.4",
   "description": "Substrate for self-improving agents: traces, verifiable rewards, preferences, GEPA / reflective mutation, auto-research, replay, sequential anytime-valid stats, and release gates.",
   "homepage": "https://github.com/tangle-network/agent-eval#readme",
   "repository": {

package/dist/chunk-TMXPFWC7.js.map

@@ -1 +0,0 @@
-{"version":3,"sources":["../src/campaign/run-campaign.ts"],"sourcesContent":["/**\n * @experimental\n *\n * `runCampaign` — Pass A substrate primitive. ONE function that orchestrates\n * scenarios → dispatch → artifacts → judges → aggregates, with full\n * reproducibility (seed + manifest hash), cell-level resumability, bootstrap\n * CIs, and the `LabeledScenarioStore` capture flywheel.\n *\n * Improvement loops (optimizer / gate / autoOnPromote) ride on top of this\n * primitive but live in `presets/run-improvement-loop.ts`. This file keeps\n * the core orchestrator minimal — Phase 1 of the Pass A track.\n */\n\nimport { createHash } from 'node:crypto'\nimport { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'\nimport { join } from 'node:path'\nimport { confidenceInterval } from '../statistics'\nimport type {\n  CampaignAggregates,\n  CampaignArtifactWriter,\n  CampaignCellResult,\n  CampaignCostMeter,\n  CampaignResult,\n  CampaignTraceWriter,\n  DispatchContext,\n  DispatchFn,\n  JudgeAggregate,\n  JudgeConfig,\n  JudgeScore,\n  LabeledScenarioStore,\n  Scenario,\n  ScenarioAggregate,\n  TraceSpan,\n} from './types'\n\nexport interface RunCampaignOptions<TScenario extends Scenario, TArtifact> {\n  scenarios: TScenario[]\n  dispatch: DispatchFn<TScenario, TArtifact>\n  judges?: JudgeConfig<TArtifact, TScenario>[]\n  /** Required for reproducibility. Default 42. */\n  seed?: number\n  /** Per-scenario replicates for CI bands. Default 1; raise to 5+ for\n   *  bootstrap-tight intervals on critical eval. */\n  reps?: number\n  /** When true (default), completed cells are cached by\n   *  (manifestHash, scenarioId, rep, generation). Re-runs skip cached cells. */\n  resumable?: boolean\n  /** Optional store — when present, every artifact + judge score is captured\n   *  with the configured `captureSource`. Capture is default ON; pass `'off'`\n   *  to disable. */\n  labeledStore?: LabeledScenarioStore | 'off'\n  captureSource?: 'production-trace' | 'eval-run' | 'manual' | 'red-team' | 'synthetic'\n  captureSourceVersionHash?: string\n  /** Wall-clock cost cap across all cells. Cells beyond ceiling are skipped. */\n  costCeiling?: number\n  /** Max concurrent cells. Default 2. */\n  maxConcurrency?: number\n  /** Required: where artifacts + traces land. */\n  runDir: string\n  /** Tracing posture. Default is the substrate's `FileSystemTraceStore` rooted\n   *  at `<runDir>/traces/`. `'off'` disables capture entirely — substrate\n   *  refuses this when the caller wires `autoOnPromote !== 'none'`. */\n  tracing?: 'on' | 'off'\n  /** Test seam — override the wall clock for deterministic tests. */\n  now?: () => Date\n  /** Test seam — override per-cell trace writer factory. */\n  buildTraceWriter?: (cellId: string, dir: string) => CampaignTraceWriter\n}\n\nexport async function runCampaign<TScenario extends Scenario, TArtifact>(\n  opts: RunCampaignOptions<TScenario, TArtifact>,\n): Promise<CampaignResult<TArtifact, TScenario>> {\n  const seed = opts.seed ?? 42\n  const reps = opts.reps ?? 1\n  const resumable = opts.resumable ?? true\n  const maxConcurrency = opts.maxConcurrency ?? 2\n  const now = opts.now ?? (() => new Date())\n  const judges = opts.judges ?? []\n\n  if (!existsSync(opts.runDir)) mkdirSync(opts.runDir, { recursive: true })\n\n  const manifestHash = computeManifestHash({\n    scenarios: opts.scenarios,\n    judges: judges as unknown as JudgeConfig<unknown>[],\n    dispatchRef: opts.dispatch.name || 'anonymous',\n    seed,\n    reps,\n  })\n\n  const startedAt = now()\n  const cells: CampaignCellResult<TArtifact>[] = []\n  const artifactsByPath: Record<string, string> = {}\n\n  // Build the cell schedule (scenario × rep).\n  const schedule: Array<{ scenario: TScenario; rep: number; cellId: string; cellSeed: number }> = []\n  let cellIndex = 0\n  for (const scenario of opts.scenarios) {\n    for (let rep = 0; rep < reps; rep++) {\n      const cellId = `${scenario.id}:${rep}`\n      const cellSeed = seed + cellIndex\n      schedule.push({ scenario, rep, cellId, cellSeed })\n      cellIndex += 1\n    }\n  }\n\n  // Concurrency-limited execution.\n  let totalCostUsd = 0\n  let costCeilingReached = false\n  const abortController = new AbortController()\n  // Concurrency lanes that drain the cell schedule. Named \"lanes\" — not\n  // \"workers\" — to avoid clashing with the taxonomy's worker (= the agent\n  // harness in a sandbox, invoked behind `dispatch`). See loop-taxonomy.md.\n  const lanes: Promise<void>[] = []\n  let nextIdx = 0\n  const cellsRef = cells\n\n  for (let i = 0; i < maxConcurrency; i++) {\n    lanes.push(\n      (async () => {\n        while (true) {\n          const myIdx = nextIdx++\n          if (myIdx >= schedule.length) return\n          const slot = schedule[myIdx]!\n          if (costCeilingReached) {\n            cellsRef.push(skippedCell(slot, 'cost_ceiling_reached'))\n            continue\n          }\n          const result = await executeCell({\n            slot,\n            opts,\n            manifestHash,\n            resumable,\n            now,\n            buildTraceWriter: opts.buildTraceWriter ?? defaultBuildTraceWriter,\n            signal: abortController.signal,\n          })\n          cellsRef.push(result.cell)\n          totalCostUsd += result.cell.costUsd\n          Object.assign(artifactsByPath, result.artifactsByPath)\n          if (opts.costCeiling !== undefined && totalCostUsd >= opts.costCeiling) {\n            costCeilingReached = true\n          }\n          // Capture into LabeledScenarioStore unless explicitly disabled.\n          if (opts.labeledStore && opts.labeledStore !== 'off' && !result.cell.error) {\n            await captureToStore({\n              store: opts.labeledStore,\n              cell: result.cell,\n              scenario: slot.scenario,\n              opts,\n              now,\n            }).catch((err) => {\n              // Capture failures are non-fatal — log but don't crash the campaign.\n              // (Trace would normally land here.)\n              console.warn(\n                `[runCampaign] capture failed for ${result.cell.cellId}: ${err instanceof Error ? err.message : String(err)}`,\n              )\n            })\n          }\n        }\n      })(),\n    )\n  }\n  await Promise.all(lanes)\n\n  const endedAt = now()\n  cellsRef.sort((a, b) => a.cellId.localeCompare(b.cellId))\n\n  const aggregates = computeAggregates(\n    cellsRef,\n    judges as unknown as JudgeConfig<TArtifact>[],\n    seed,\n  )\n\n  return {\n    manifestHash,\n    seed,\n    startedAt: startedAt.toISOString(),\n    endedAt: endedAt.toISOString(),\n    durationMs: endedAt.getTime() - startedAt.getTime(),\n    cells: cellsRef,\n    aggregates,\n    runDir: opts.runDir,\n    artifactsByPath,\n    scenarios: opts.scenarios.map((s) => ({ id: s.id, kind: s.kind })),\n  }\n}\n\n// ── Internals ─────────────────────────────────────────────────────────\n\ninterface ExecuteCellArgs<TScenario extends Scenario, TArtifact> {\n  slot: { scenario: TScenario; rep: number; cellId: string; cellSeed: number }\n  opts: RunCampaignOptions<TScenario, TArtifact>\n  manifestHash: string\n  resumable: boolean\n  now: () => Date\n  buildTraceWriter: (cellId: string, dir: string) => CampaignTraceWriter\n  signal: AbortSignal\n}\n\nasync function executeCell<TScenario extends Scenario, TArtifact>(\n  args: ExecuteCellArgs<TScenario, TArtifact>,\n): Promise<{ cell: CampaignCellResult<TArtifact>; artifactsByPath: Record<string, string> }> {\n  const cellDir = join(args.opts.runDir, args.slot.cellId.replace(/[^a-zA-Z0-9_-]/g, '_'))\n  if (!existsSync(cellDir)) mkdirSync(cellDir, { recursive: true })\n\n  // Resumability: cache key = (manifestHash, scenarioId, rep)\n  const cachePath = join(cellDir, 'cached-result.json')\n  if (args.resumable && existsSync(cachePath)) {\n    try {\n      const cached = JSON.parse(readFileSync(cachePath, 'utf8')) as CampaignCellResult<TArtifact>\n      if (cached.cellId === args.slot.cellId) {\n        return { cell: { ...cached, cached: true }, artifactsByPath: {} }\n      }\n    } catch {\n      // Corrupt cache — fall through to re-run.\n    }\n  }\n\n  const startMs = Date.now()\n  const trace = args.buildTraceWriter(args.slot.cellId, cellDir)\n  const artifactsByPath: Record<string, string> = {}\n  const artifacts: CampaignArtifactWriter = {\n    async write(path, content) {\n      const fullPath = join(cellDir, path)\n      const dir = join(fullPath, '..')\n      if (!existsSync(dir)) mkdirSync(dir, { recursive: true })\n      writeFileSync(fullPath, content as Uint8Array)\n      artifactsByPath[`${args.slot.cellId}/${path}`] = fullPath\n      return fullPath\n    },\n    async writeJson(path, value) {\n      return artifacts.write(path, JSON.stringify(value, null, 2))\n    },\n  }\n  let costSoFar = 0\n  const cost: CampaignCostMeter = {\n    observe(amount, source) {\n      costSoFar += amount\n      trace.span(`cost.${source}`, { amountUsd: amount }).end()\n    },\n    current() {\n      return costSoFar\n    },\n  }\n\n  const ctx: DispatchContext = {\n    cellId: args.slot.cellId,\n    rep: args.slot.rep,\n    seed: args.slot.cellSeed,\n    signal: args.signal,\n    trace,\n    artifacts,\n    cost,\n  }\n\n  let artifact: TArtifact | undefined\n  let errorMessage: string | undefined\n  try {\n    artifact = await args.opts.dispatch(args.slot.scenario, ctx)\n  } catch (err) {\n    errorMessage = err instanceof Error ? err.message : String(err)\n  }\n\n  // Run judges (only if we have an artifact).\n  const judgeScores: Record<string, JudgeScore> = {}\n  if (artifact !== undefined) {\n    for (const judge of args.opts.judges ?? []) {\n      if (judge.appliesTo && !judge.appliesTo(args.slot.scenario)) continue\n      try {\n        const score = await runJudgeCell(judge, { artifact, scenario: args.slot.scenario })\n        judgeScores[judge.name] = score\n      } catch (err) {\n        judgeScores[judge.name] = {\n          dimensions: {},\n          composite: 0,\n          notes: `judge failed: ${err instanceof Error ? err.message : String(err)}`,\n        }\n      }\n    }\n  }\n\n  await trace.flush()\n\n  const cell: CampaignCellResult<TArtifact> = {\n    cellId: args.slot.cellId,\n    scenarioId: args.slot.scenario.id,\n    rep: args.slot.rep,\n    artifact: (artifact ?? null) as TArtifact,\n    judgeScores,\n    costUsd: costSoFar,\n    durationMs: Date.now() - startMs,\n    seed: args.slot.cellSeed,\n    cached: false,\n    error: errorMessage,\n  }\n\n  if (!errorMessage && args.resumable) {\n    writeFileSync(cachePath, JSON.stringify(cell))\n  }\n\n  return { cell, artifactsByPath }\n}\n\nasync function runJudgeCell<TArtifact, TScenario extends Scenario>(\n  _judge: JudgeConfig<TArtifact, TScenario>,\n  _input: { artifact: TArtifact; scenario: TScenario },\n): Promise<JudgeScore> {\n  // Phase 1 stub — wires to the existing 0.38 runJudge in Phase 2.\n  // Returns a zero-score for now; consumer wiring + preset uses this.\n  return { dimensions: {}, composite: 0, notes: 'phase-1-stub' }\n}\n\nfunction defaultBuildTraceWriter(cellId: string, dir: string): CampaignTraceWriter {\n  const spans: Array<Record<string, unknown>> = []\n  return {\n    span(name, attributes) {\n      const startMs = Date.now()\n      const record: Record<string, unknown> = { name, cellId, startMs, ...(attributes ?? {}) }\n      const finish: TraceSpan = {\n        end(endAttrs) {\n          record.durationMs = Date.now() - startMs\n          if (endAttrs) Object.assign(record, endAttrs)\n          spans.push(record)\n        },\n        setAttribute(key, value) {\n          record[key] = value\n        },\n      }\n      return finish\n    },\n    async flush() {\n      const path = join(dir, 'spans.jsonl')\n      writeFileSync(path, spans.map((s) => JSON.stringify(s)).join('\\n'))\n    },\n  }\n}\n\nfunction skippedCell<TScenario extends Scenario, TArtifact>(\n  slot: { scenario: TScenario; rep: number; cellId: string; cellSeed: number },\n  reason: string,\n): CampaignCellResult<TArtifact> {\n  return {\n    cellId: slot.cellId,\n    scenarioId: slot.scenario.id,\n    rep: slot.rep,\n    artifact: null as unknown as TArtifact,\n    judgeScores: {},\n    costUsd: 0,\n    durationMs: 0,\n    seed: slot.cellSeed,\n    cached: false,\n    error: `skipped: ${reason}`,\n  }\n}\n\ninterface CaptureArgs<TScenario extends Scenario, TArtifact> {\n  store: LabeledScenarioStore\n  cell: CampaignCellResult<TArtifact>\n  scenario: TScenario\n  opts: RunCampaignOptions<TScenario, TArtifact>\n  now: () => Date\n}\n\nasync function captureToStore<TScenario extends Scenario, TArtifact>(\n  args: CaptureArgs<TScenario, TArtifact>,\n): Promise<void> {\n  await args.store.observe({\n    scenario: args.scenario,\n    artifact: args.cell.artifact,\n    judgeScores: args.cell.judgeScores,\n    source: args.opts.captureSource ?? 'eval-run',\n    sourceVersionHash: args.opts.captureSourceVersionHash ?? 'unknown',\n    capturedAt: args.now().toISOString(),\n    redactionStatus: 'raw',\n  })\n}\n\n// ── Aggregates + manifest hash ────────────────────────────────────────\n\nfunction computeManifestHash(input: {\n  scenarios: Scenario[]\n  judges: JudgeConfig<unknown>[]\n  dispatchRef: string\n  seed: number\n  reps: number\n}): string {\n  const canonical = {\n    scenarios: input.scenarios.map((s) => ({ id: s.id, kind: s.kind })),\n    judges: input.judges.map((j) => ({ name: j.name, dims: j.dimensions.map((d) => d.key) })),\n    dispatch: input.dispatchRef,\n    seed: input.seed,\n    reps: input.reps,\n  }\n  return createHash('sha256').update(JSON.stringify(canonical)).digest('hex')\n}\n\nfunction computeAggregates<TArtifact>(\n  cells: CampaignCellResult<TArtifact>[],\n  judges: JudgeConfig<TArtifact>[],\n  seed: number,\n): CampaignAggregates {\n  const byJudge: Record<string, JudgeAggregate> = {}\n  for (const judge of judges) {\n    const scores: number[] = []\n    for (const cell of cells) {\n      const s = cell.judgeScores[judge.name]\n      if (s !== undefined) scores.push(s.composite)\n    }\n    byJudge[judge.name] = aggregate(scores, seed)\n  }\n  const byScenario: Record<string, ScenarioAggregate> = {}\n  const scenarioGroups = new Map<string, number[]>()\n  for (const cell of cells) {\n    const composites = Object.values(cell.judgeScores).map((s) => s.composite)\n    if (composites.length === 0) continue\n    const mean = composites.reduce((a, b) => a + b, 0) / composites.length\n    const arr = scenarioGroups.get(cell.scenarioId) ?? []\n    arr.push(mean)\n    scenarioGroups.set(cell.scenarioId, arr)\n  }\n  for (const [scenarioId, samples] of scenarioGroups) {\n    const ag = aggregate(samples, seed)\n    byScenario[scenarioId] = { meanComposite: ag.mean, ci95: ag.ci95, n: ag.n }\n  }\n  return {\n    byJudge,\n    byScenario,\n    totalCostUsd: cells.reduce((a, c) => a + c.costUsd, 0),\n    cellsExecuted: cells.filter((c) => !c.error).length,\n    cellsSkipped: cells.filter((c) => c.error?.startsWith('skipped:')).length,\n    cellsCached: cells.filter((c) => c.cached).length,\n    cellsFailed: cells.filter((c) => c.error && !c.error.startsWith('skipped:')).length,\n  }\n}\n\n// Percentile bootstrap CI95 via seeded resampling. Deterministic for a given\n// seed — same campaign re-run produces identical CI bands. Falls back to\n// degenerate intervals at n<=1 (the bootstrap is undefined there).\nfunction aggregate(samples: number[], seed: number): JudgeAggregate {\n  const n = samples.length\n  if (n === 0) return { mean: 0, stdev: 0, ci95: [0, 0], n: 0 }\n  const mean = samples.reduce((a, b) => a + b, 0) / n\n  const variance = samples.reduce((a, b) => a + (b - mean) ** 2, 0) / Math.max(1, n - 1)\n  const stdev = Math.sqrt(variance)\n  const ci = confidenceInterval(samples, 0.95, { seed, resamples: 1000 })\n  return { mean, stdev, ci95: [ci.lower, ci.upper], n }\n}\n"],"mappings":";;;;;AAaA,SAAS,kBAAkB;AAC3B,SAAS,YAAY,WAAW,cAAc,qBAAqB;AACnE,SAAS,YAAY;AAsDrB,eAAsB,YACpB,MAC+C;AAC/C,QAAM,OAAO,KAAK,QAAQ;AAC1B,QAAM,OAAO,KAAK,QAAQ;AAC1B,QAAM,YAAY,KAAK,aAAa;AACpC,QAAM,iBAAiB,KAAK,kBAAkB;AAC9C,QAAM,MAAM,KAAK,QAAQ,MAAM,oBAAI,KAAK;AACxC,QAAM,SAAS,KAAK,UAAU,CAAC;AAE/B,MAAI,CAAC,WAAW,KAAK,MAAM,EAAG,WAAU,KAAK,QAAQ,EAAE,WAAW,KAAK,CAAC;AAExE,QAAM,eAAe,oBAAoB;AAAA,IACvC,WAAW,KAAK;AAAA,IAChB;AAAA,IACA,aAAa,KAAK,SAAS,QAAQ;AAAA,IACnC;AAAA,IACA;AAAA,EACF,CAAC;AAED,QAAM,YAAY,IAAI;AACtB,QAAM,QAAyC,CAAC;AAChD,QAAM,kBAA0C,CAAC;AAGjD,QAAM,WAA0F,CAAC;AACjG,MAAI,YAAY;AAChB,aAAW,YAAY,KAAK,WAAW;AACrC,aAAS,MAAM,GAAG,MAAM,MAAM,OAAO;AACnC,YAAM,SAAS,GAAG,SAAS,EAAE,IAAI,GAAG;AACpC,YAAM,WAAW,OAAO;AACxB,eAAS,KAAK,EAAE,UAAU,KAAK,QAAQ,SAAS,CAAC;AACjD,mBAAa;AAAA,IACf;AAAA,EACF;AAGA,MAAI,eAAe;AACnB,MAAI,qBAAqB;AACzB,QAAM,kBAAkB,IAAI,gBAAgB;AAI5C,QAAM,QAAyB,CAAC;AAChC,MAAI,UAAU;AACd,QAAM,WAAW;AAEjB,WAAS,IAAI,GAAG,IAAI,gBAAgB,KAAK;AACvC,UAAM;AAAA,OACH,YAAY;AACX,eAAO,MAAM;AACX,gBAAM,QAAQ;AACd,cAAI,SAAS,SAAS,OAAQ;AAC9B,gBAAM,OAAO,SAAS,KAAK;AAC3B,cAAI,oBAAoB;AACtB,qBAAS,KAAK,YAAY,MAAM,sBAAsB,CAAC;AACvD;AAAA,UACF;AACA,gBAAM,SAAS,MAAM,YAAY;AAAA,YAC/B;AAAA,YACA;AAAA,YACA;AAAA,YACA;AAAA,YACA;AAAA,YACA,kBAAkB,KAAK,oBAAoB;AAAA,YAC3C,QAAQ,gBAAgB;AAAA,UAC1B,CAAC;AACD,mBAAS,KAAK,OAAO,IAAI;AACzB,0BAAgB,OAAO,KAAK;AAC5B,iBAAO,OAAO,iBAAiB,OAAO,eAAe;AACrD,cAAI,KAAK,gBAAgB,UAAa,gBAAgB,KAAK,aAAa;AACtE,iCAAqB;AAAA,UACvB;AAEA,cAAI,KAAK,gBAAgB,KAAK,iBAAiB,SAAS,CAAC,OAAO,KAAK,OAAO;AAC1E,kBAAM,eAAe;AAAA,cACnB,OAAO,KAAK;AAAA,cACZ,MAAM,OAAO;AAAA,cACb,UAAU,KAAK;AAAA,cACf;AAAA,cACA;AAAA,YACF,CAAC,EAAE,MAAM,CAAC,QAAQ;AAGhB,sBAAQ;AAAA,gBACN,oCAAoC,OAAO,KAAK,MAAM,KAAK,eAAe,QAAQ,IAAI,UAAU,OAAO,GAAG,CAAC;AAAA,cAC7G;AAAA,YACF,CAAC;AAAA,UACH;AAAA,QACF;AAAA,MACF,GAAG;AAAA,IACL;AAAA,EACF;AACA,QAAM,QAAQ,IAAI,KAAK;AAEvB,QAAM,UAAU,IAAI;AACpB,WAAS,KAAK,CAAC,GAAG,MAAM,EAAE,OAAO,cAAc,EAAE,MAAM,CAAC;AAExD,QAAM,aAAa;AAAA,IACjB;AAAA,IACA;AAAA,IACA;AAAA,EACF;AAEA,SAAO;AAAA,IACL;AAAA,IACA;AAAA,IACA,WAAW,UAAU,YAAY;AAAA,IACjC,SAAS,QAAQ,YAAY;AAAA,IAC7B,YAAY,QAAQ,QAAQ,IAAI,UAAU,QAAQ;AAAA,IAClD,OAAO;AAAA,IACP;AAAA,IACA,QAAQ,KAAK;AAAA,IACb;AAAA,IACA,WAAW,KAAK,UAAU,IAAI,CAAC,OAAO,EAAE,IAAI,EAAE,IAAI,MAAM,EAAE,KAAK,EAAE;AAAA,EACnE;AACF;AAcA,eAAe,YACb,MAC2F;AAC3F,QAAM,UAAU,KAAK,KAAK,KAAK,QAAQ,KAAK,KAAK,OAAO,QAAQ,mBAAmB,GAAG,CAAC;AACvF,MAAI,CAAC,WAAW,OAAO,EAAG,WAAU,SAAS,EAAE,WAAW,KAAK,CAAC;AAGhE,QAAM,YAAY,KAAK,SAAS,oBAAoB;AACpD,MAAI,KAAK,aAAa,WAAW,SAAS,GAAG;AAC3C,QAAI;AACF,YAAM,SAAS,KAAK,MAAM,aAAa,WAAW,MAAM,CAAC;AACzD,UAAI,OAAO,WAAW,KAAK,KAAK,QAAQ;AACtC,eAAO,EAAE,MAAM,EAAE,GAAG,QAAQ,QAAQ,KAAK,GAAG,iBAAiB,CAAC,EAAE;AAAA,MAClE;AAAA,IACF,QAAQ;AAAA,IAER;AAAA,EACF;AAEA,QAAM,UAAU,KAAK,IAAI;AACzB,QAAM,QAAQ,KAAK,iBAAiB,KAAK,KAAK,QAAQ,OAAO;AAC7D,QAAM,kBAA0C,CAAC;AACjD,QAAM,YAAoC;AAAA,IACxC,MAAM,MAAM,MAAM,SAAS;AACzB,YAAM,WAAW,KAAK,SAAS,IAAI;AACnC,YAAM,MAAM,KAAK,UAAU,IAAI;AAC/B,UAAI,CAAC,WAAW,GAAG,EAAG,WAAU,KAAK,EAAE,WAAW,KAAK,CAAC;AACxD,oBAAc,UAAU,OAAqB;AAC7C,sBAAgB,GAAG,KAAK,KAAK,MAAM,IAAI,IAAI,EAAE,IAAI;AACjD,aAAO;AAAA,IACT;AAAA,IACA,MAAM,UAAU,MAAM,OAAO;AAC3B,aAAO,UAAU,MAAM,MAAM,KAAK,UAAU,OAAO,MAAM,CAAC,CAAC;AAAA,IAC7D;AAAA,EACF;AACA,MAAI,YAAY;AAChB,QAAM,OAA0B;AAAA,IAC9B,QAAQ,QAAQ,QAAQ;AACtB,mBAAa;AACb,YAAM,KAAK,QAAQ,MAAM,IAAI,EAAE,WAAW,OAAO,CAAC,EAAE,IAAI;AAAA,IAC1D;AAAA,IACA,UAAU;AACR,aAAO;AAAA,IACT;AAAA,EACF;AAEA,QAAM,MAAuB;AAAA,IAC3B,QAAQ,KAAK,KAAK;AAAA,IAClB,KAAK,KAAK,KAAK;AAAA,IACf,MAAM,KAAK,KAAK;AAAA,IAChB,QAAQ,KAAK;AAAA,IACb;AAAA,IACA;AAAA,IACA;AAAA,EACF;AAEA,MAAI;AACJ,MAAI;AACJ,MAAI;AACF,eAAW,MAAM,KAAK,KAAK,SAAS,KAAK,KAAK,UAAU,GAAG;AAAA,EAC7D,SAAS,KAAK;AACZ,mBAAe,eAAe,QAAQ,IAAI,UAAU,OAAO,GAAG;AAAA,EAChE;AAGA,QAAM,cAA0C,CAAC;AACjD,MAAI,aAAa,QAAW;AAC1B,eAAW,SAAS,KAAK,KAAK,UAAU,CAAC,GAAG;AAC1C,UAAI,MAAM,aAAa,CAAC,MAAM,UAAU,KAAK,KAAK,QAAQ,EAAG;AAC7D,UAAI;AACF,cAAM,QAAQ,MAAM,aAAa,OAAO,EAAE,UAAU,UAAU,KAAK,KAAK,SAAS,CAAC;AAClF,oBAAY,MAAM,IAAI,IAAI;AAAA,MAC5B,SAAS,KAAK;AACZ,oBAAY,MAAM,IAAI,IAAI;AAAA,UACxB,YAAY,CAAC;AAAA,UACb,WAAW;AAAA,UACX,OAAO,iBAAiB,eAAe,QAAQ,IAAI,UAAU,OAAO,GAAG,CAAC;AAAA,QAC1E;AAAA,MACF;AAAA,IACF;AAAA,EACF;AAEA,QAAM,MAAM,MAAM;AAElB,QAAM,OAAsC;AAAA,IAC1C,QAAQ,KAAK,KAAK;AAAA,IAClB,YAAY,KAAK,KAAK,SAAS;AAAA,IAC/B,KAAK,KAAK,KAAK;AAAA,IACf,UAAW,YAAY;AAAA,IACvB;AAAA,IACA,SAAS;AAAA,IACT,YAAY,KAAK,IAAI,IAAI;AAAA,IACzB,MAAM,KAAK,KAAK;AAAA,IAChB,QAAQ;AAAA,IACR,OAAO;AAAA,EACT;AAEA,MAAI,CAAC,gBAAgB,KAAK,WAAW;AACnC,kBAAc,WAAW,KAAK,UAAU,IAAI,CAAC;AAAA,EAC/C;AAEA,SAAO,EAAE,MAAM,gBAAgB;AACjC;AAEA,eAAe,aACb,QACA,QACqB;AAGrB,SAAO,EAAE,YAAY,CAAC,GAAG,WAAW,GAAG,OAAO,eAAe;AAC/D;AAEA,SAAS,wBAAwB,QAAgB,KAAkC;AACjF,QAAM,QAAwC,CAAC;AAC/C,SAAO;AAAA,IACL,KAAK,MAAM,YAAY;AACrB,YAAM,UAAU,KAAK,IAAI;AACzB,YAAM,SAAkC,EAAE,MAAM,QAAQ,SAAS,GAAI,cAAc,CAAC,EAAG;AACvF,YAAM,SAAoB;AAAA,QACxB,IAAI,UAAU;AACZ,iBAAO,aAAa,KAAK,IAAI,IAAI;AACjC,cAAI,SAAU,QAAO,OAAO,QAAQ,QAAQ;AAC5C,gBAAM,KAAK,MAAM;AAAA,QACnB;AAAA,QACA,aAAa,KAAK,OAAO;AACvB,iBAAO,GAAG,IAAI;AAAA,QAChB;AAAA,MACF;AACA,aAAO;AAAA,IACT;AAAA,IACA,MAAM,QAAQ;AACZ,YAAM,OAAO,KAAK,KAAK,aAAa;AACpC,oBAAc,MAAM,MAAM,IAAI,CAAC,MAAM,KAAK,UAAU,CAAC,CAAC,EAAE,KAAK,IAAI,CAAC;AAAA,IACpE;AAAA,EACF;AACF;AAEA,SAAS,YACP,MACA,QAC+B;AAC/B,SAAO;AAAA,IACL,QAAQ,KAAK;AAAA,IACb,YAAY,KAAK,SAAS;AAAA,IAC1B,KAAK,KAAK;AAAA,IACV,UAAU;AAAA,IACV,aAAa,CAAC;AAAA,IACd,SAAS;AAAA,IACT,YAAY;AAAA,IACZ,MAAM,KAAK;AAAA,IACX,QAAQ;AAAA,IACR,OAAO,YAAY,MAAM;AAAA,EAC3B;AACF;AAUA,eAAe,eACb,MACe;AACf,QAAM,KAAK,MAAM,QAAQ;AAAA,IACvB,UAAU,KAAK;AAAA,IACf,UAAU,KAAK,KAAK;AAAA,IACpB,aAAa,KAAK,KAAK;AAAA,IACvB,QAAQ,KAAK,KAAK,iBAAiB;AAAA,IACnC,mBAAmB,KAAK,KAAK,4BAA4B;AAAA,IACzD,YAAY,KAAK,IAAI,EAAE,YAAY;AAAA,IACnC,iBAAiB;AAAA,EACnB,CAAC;AACH;AAIA,SAAS,oBAAoB,OAMlB;AACT,QAAM,YAAY;AAAA,IAChB,WAAW,MAAM,UAAU,IAAI,CAAC,OAAO,EAAE,IAAI,EAAE,IAAI,MAAM,EAAE,KAAK,EAAE;AAAA,IAClE,QAAQ,MAAM,OAAO,IAAI,CAAC,OAAO,EAAE,MAAM,EAAE,MAAM,MAAM,EAAE,WAAW,IAAI,CAAC,MAAM,EAAE,GAAG,EAAE,EAAE;AAAA,IACxF,UAAU,MAAM;AAAA,IAChB,MAAM,MAAM;AAAA,IACZ,MAAM,MAAM;AAAA,EACd;AACA,SAAO,WAAW,QAAQ,EAAE,OAAO,KAAK,UAAU,SAAS,CAAC,EAAE,OAAO,KAAK;AAC5E;AAEA,SAAS,kBACP,OACA,QACA,MACoB;AACpB,QAAM,UAA0C,CAAC;AACjD,aAAW,SAAS,QAAQ;AAC1B,UAAM,SAAmB,CAAC;AAC1B,eAAW,QAAQ,OAAO;AACxB,YAAM,IAAI,KAAK,YAAY,MAAM,IAAI;AACrC,UAAI,MAAM,OAAW,QAAO,KAAK,EAAE,SAAS;AAAA,IAC9C;AACA,YAAQ,MAAM,IAAI,IAAI,UAAU,QAAQ,IAAI;AAAA,EAC9C;AACA,QAAM,aAAgD,CAAC;AACvD,QAAM,iBAAiB,oBAAI,IAAsB;AACjD,aAAW,QAAQ,OAAO;AACxB,UAAM,aAAa,OAAO,OAAO,KAAK,WAAW,EAAE,IAAI,CAAC,MAAM,EAAE,SAAS;AACzE,QAAI,WAAW,WAAW,EAAG;AAC7B,UAAM,OAAO,WAAW,OAAO,CAAC,GAAG,MAAM,IAAI,GAAG,CAAC,IAAI,WAAW;AAChE,UAAM,MAAM,eAAe,IAAI,KAAK,UAAU,KAAK,CAAC;AACpD,QAAI,KAAK,IAAI;AACb,mBAAe,IAAI,KAAK,YAAY,GAAG;AAAA,EACzC;AACA,aAAW,CAAC,YAAY,OAAO,KAAK,gBAAgB;AAClD,UAAM,KAAK,UAAU,SAAS,IAAI;AAClC,eAAW,UAAU,IAAI,EAAE,eAAe,GAAG,MAAM,MAAM,GAAG,MAAM,GAAG,GAAG,EAAE;AAAA,EAC5E;AACA,SAAO;AAAA,IACL;AAAA,IACA;AAAA,IACA,cAAc,MAAM,OAAO,CAAC,GAAG,MAAM,IAAI,EAAE,SAAS,CAAC;AAAA,IACrD,eAAe,MAAM,OAAO,CAAC,MAAM,CAAC,EAAE,KAAK,EAAE;AAAA,IAC7C,cAAc,MAAM,OAAO,CAAC,MAAM,EAAE,OAAO,WAAW,UAAU,CAAC,EAAE;AAAA,IACnE,aAAa,MAAM,OAAO,CAAC,MAAM,EAAE,MAAM,EAAE;AAAA,IAC3C,aAAa,MAAM,OAAO,CAAC,MAAM,EAAE,SAAS,CAAC,EAAE,MAAM,WAAW,UAAU,CAAC,EAAE;AAAA,EAC/E;AACF;AAKA,SAAS,UAAU,SAAmB,MAA8B;AAClE,QAAM,IAAI,QAAQ;AAClB,MAAI,MAAM,EAAG,QAAO,EAAE,MAAM,GAAG,OAAO,GAAG,MAAM,CAAC,GAAG,CAAC,GAAG,GAAG,EAAE;AAC5D,QAAM,OAAO,QAAQ,OAAO,CAAC,GAAG,MAAM,IAAI,GAAG,CAAC,IAAI;AAClD,QAAM,WAAW,QAAQ,OAAO,CAAC,GAAG,MAAM,KAAK,IAAI,SAAS,GAAG,CAAC,IAAI,KAAK,IAAI,GAAG,IAAI,CAAC;AACrF,QAAM,QAAQ,KAAK,KAAK,QAAQ;AAChC,QAAM,KAAK,mBAAmB,SAAS,MAAM,EAAE,MAAM,WAAW,IAAK,CAAC;AACtE,SAAO,EAAE,MAAM,OAAO,MAAM,CAAC,GAAG,OAAO,GAAG,KAAK,GAAG,EAAE;AACtD;","names":[]}