@tangle-network/agent-eval 0.61.0 → 0.63.0

package/CHANGELOG.md

@@ -4,24 +4,64 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
 
 ---
 
-## [0.61.0] — 2026-05-30 — `runProfileMatrix` (profile × scenario × persona matrix with integrity by construction)
+## [0.63.0] — 2026-05-30 — the full optimizer drivers: GEPA Pareto + SkillOpt + a head-to-head lift benchmark
+
+Closes the optimizer-completeness gap (#101/#100). `gepaDriver` was reflection-only; the SOTA SkillOpt technique was roadmapped but unbuilt; and there was no head-to-head benchmark, so optimizer quality was measurement-invisible — a simplified driver could ship unnoticed. This release ships both drivers in full and the forcing function that keeps them honest.
 
 ### Added
 
-- **`runProfileMatrix({ profiles, scenarios, dispatch, judges, reps, integrity, personaOf })`** (`@tangle-network/agent-eval/campaign`) — the keystone that lets a consumer express a multi-profile × scenario/persona eval as **one** call instead of a hand-rolled `eval:*` script. Fans `profiles` (axis 3) over the scenario/persona corpus (axis 1), runs `runCampaign` per profile (reusing its seeds / reps / bootstrap CIs / resumability / `LabeledScenarioStore` capture flywheel), maps every cell to a validated `RunRecord` carrying real `tokenUsage`, and runs **`assertRealBackend` by construction** (`integrity: 'assert' | 'warn' | 'off'`, default `assert`). Returns `{ records, byProfile, byScenario, byPersona, integrity, campaigns }`.
-- **`ProfileMatrixError`** — thrown at preflight (before any LLM spend) when a profile's model lacks a snapshot version or the profile/scenario lists are empty.
+- **GEPA Pareto frontier + combine-complementary-lessons (#101).** `runOptimization` now accumulates every scored surface as a per-scenario objective vector and recomputes the non-dominated set before each generation, handing it to the driver as `ctx.paretoParents` (new `ParetoParent` type). A surface uniquely best on one hard scenario survives even when its mean composite is lower. `gepaDriver` spends one population slot merging the frontier parents' complementary strengths (toggle via `combineParents`, default on; fires only when the frontier has >1 member). `RunOptimizationResult.paretoFrontier` exposes the final frontier. Dominance is computed by the package-canonical `paretoFrontier` (`src/pareto.ts`) — the parallel `src/campaign/pareto.ts` fork has been deleted (one dominance implementation).
+- **SkillOpt patch-mode driver + `runSkillOpt` preset (#100)** (Microsoft, arXiv:2605.23904). `skillOptDriver` proposes BOUNDED add/delete/replace patches to one skill document (`applySkillPatch`, `SkillPatch`); `runSkillOpt` is the held-out-gated epoch hill-climb: reflect on TRAIN weaknesses → propose ≤ `editBudget` ops → score on the held-out split → ACCEPT only on STRICT held-out improvement, else buffer the rejected edit; with edit-budget annealing (the "textual learning rate") and a slow-update meta note. The held-out composite is monotonically non-decreasing by construction — a regression can never ship. Proposals reflect on train evidence only (no held-out leakage).
+- **`compareDrivers` head-to-head lift benchmark (the forcing function).** Runs N optimizer entries on ONE corpus, scores the baseline + every promoted surface UNIFORMLY on the same held-out scenarios, and reports per-driver lift + paired-bootstrap CI + pairwise "which driver wins" CIs, ranked (cost breaks a lift tie). Ships `gepaReflectionEntry` / `gepaParetoEntry` / `skillOptEntry` to wire the real optimizers. Optimizer quality is now a number with a confidence interval — a driver regression turns a build red instead of going invisible.
+- **`campaignMeanComposite` / `campaignBreakdown`** (`score-utils`) — the one definition of "composite of a campaign" + per-scenario/dimension breakdown, now shared by `runOptimization`, `runSkillOpt`, and `compareDrivers` (extracted from `runOptimization`'s private copies).
 
-### Fixed / closed gap
+### Changed
 
-- **Token usage is now captured by `runCampaign`** — `CampaignCostMeter` gains `observeTokens(usage)` + `tokens()`, and `CampaignCellResult` gains `tokenUsage`. Previously a campaign cell carried `costUsd` but no token counts, so `assertRealBackend`/`summarizeBackendIntegrity` (which key on `tokenUsage`) could not run on a `CampaignResult`. This closes the integrity gap for **every** campaign consumer, not just `runProfileMatrix`.
+- `gepaDriver`'s docstring + new `combineParents`/`combineMaxParents` options reflect the now-complete GEPA mapping (reflection + Pareto + combine).
 
-### Why this matters
+---
+
+## [0.62.0] — 2026-05-30 — eval↔runtime boundary hardening (honest cost meter + per-cell stub guard)
+
+From the agent-eval ↔ agent-runtime boundary critique. Builds on `runProfileMatrix` (0.61.0).
+
+### Fixed
+
+- **`CampaignCostMeter` docstring no longer lies.** It claimed "Substrate auto-tracks LLM costs via the cost-ledger backend hooks" — false (the meter mutates only on explicit `observe`/`observeTokens`), and it contradicted `observeTokens`' own doc. That doc was the root cause of consumers skipping `observeTokens`, getting `{0,0}` stub cells, and building `RunRecord`s on a side-channel. The doc now states plainly: nothing is captured automatically; the dispatch MUST report.
+
+### Added
+
+- **`runCampaign({ expectUsage })`** — per-cell stub guard, the early/fine-grained sibling of batch `assertRealBackend`. A cell that produced an artifact but reported `costUsd === 0` AND zero tokens is a stub. Modes: `'warn'` (default, non-breaking), `'assert'` (throw `BackendIntegrityError` on the first stub cell), `'off'` (replay/offline). Errored/skipped cells and deterministic judge-only runs are not flagged.
+
+### Changed
+
+- **`CampaignTokenUsage` is now `type CampaignTokenUsage = RunTokenUsage`** (one source of truth; a field added to `RunTokenUsage` is a compile error here, not silent drift across the three hand-synced copies the audit found).
+- **multishot aliases sandbox's `AgentProfile` → `SandboxAgentProfile`** so it no longer collides with the eval-harness `AgentProfile` the root exports.
+
+### Boundary
+
+- **`tests/boundary-integrity.test.ts`** — mechanically enforces the zero-upward-dependency rule (agent-eval must never import agent-runtime/agent-knowledge). The CLAUDE.md rule was prose-only; it is now a red build.
+
+### Notes
+
+Pure additive/doc surface (`expectUsage` defaults to non-breaking `'warn'`). Full suite 1538/1538 green. Consumes-side: agent-runtime `loopDispatch` (0.32.0) turns the whole seam into one un-mis-wireable call.
+
+---
+
+## [0.61.0] — 2026-05-30 — `runProfileMatrix` (profile × scenario × persona matrix with integrity by construction)
+
+### Added
+
+- **`runProfileMatrix({ profiles, scenarios, dispatch, judges, reps, integrity, personaOf })`** (`@tangle-network/agent-eval/campaign`) — the keystone that lets a consumer express a multi-profile × scenario/persona eval as **one** call instead of a hand-rolled `eval:*` script. Fans `profiles` over the scenario/persona corpus, runs `runCampaign` per profile, maps every cell to a validated `RunRecord` carrying real `tokenUsage`, and runs **`assertRealBackend` by construction**. Returns `{ records, byProfile, byScenario, byPersona, integrity, campaigns }`.
+- **`ProfileMatrixError`** — thrown at preflight (before any LLM spend) when a profile's model lacks a snapshot version or the lists are empty.
+
+### Fixed / closed gap
 
-A fleet eval-surface audit found every consumer hand-rolls the same matrix→dispatch→`RunRecord`→integrity bridge as a bespoke script, because no primitive produced integrity-checkable `RunRecord`s from a profile matrix. `runProfileMatrix` is that bridge, once — so the adoption skills can mandate "one matrix harness, axes as flags" with a real primitive behind it. Consumers wrap their runtime (e.g. agent-runtime `runLoop` + `reportLoopUsage`) in `dispatch`; the integrity guard then sees real LLM activity.
+- **Token usage captured by `runCampaign`** — `CampaignCostMeter` gains `observeTokens()`/`tokens()` and `CampaignCellResult` gains `tokenUsage`, so the integrity guards can run on a `CampaignResult` (they key on `tokenUsage`). Closes the gap for **every** campaign consumer.
 
 ### Notes
 
-Pure additive surface (the `CampaignCostMeter` additions are new optional methods). 7 new tests under `tests/campaign/run-profile-matrix.test.ts` — the keystone being the **stub→throws** regression (a zero-token dispatch fails the matrix loudly instead of reporting a clean 0/N leaderboard). Full suite 1527/1527 green.
+7 new tests; the keystone is the **stub→throws** regression. Full suite 1527/1527 green at release.
 
 ---
 

package/dist/adapters/http.d.ts

@@ -1,4 +1,7 @@
-import { S as Scenario, D as DispatchFn, b as DispatchContext } from '../types-Beb6KPqZ.js';
+import { S as Scenario, D as DispatchFn, b as DispatchContext } from '../types-c2R2kfmv.js';
+import '../run-record-BgTFzO2r.js';
+import '../errors-Dwqw-T_m.js';
+import '../schema-m0gsnbt3.js';
 
 /**
  * # `@tangle-network/agent-eval/adapters/http` — distributed Dispatch over HTTP.

package/dist/adapters/langchain.d.ts

@@ -1,4 +1,7 @@
-import { S as Scenario, J as JudgeScore, D as DispatchFn, a as JudgeConfig } from '../types-Beb6KPqZ.js';
+import { S as Scenario, J as JudgeScore, D as DispatchFn, a as JudgeConfig } from '../types-c2R2kfmv.js';
+import '../run-record-BgTFzO2r.js';
+import '../errors-Dwqw-T_m.js';
+import '../schema-m0gsnbt3.js';
 
 /**
  * # `@tangle-network/agent-eval/adapters/langchain` — wrap any LangChain

package/dist/adapters/otel.d.ts

@@ -1,9 +1,9 @@
-import { T as TraceSpanEvent, H as HostedClient } from '../index-D9dwa00f.js';
-import '../types-Beb6KPqZ.js';
-import '../summary-report-BQvXpvaR.js';
-import '../run-record-DgUVo5pw.js';
+import { T as TraceSpanEvent, H as HostedClient } from '../index-GISRh500.js';
+import '../types-c2R2kfmv.js';
+import '../run-record-BgTFzO2r.js';
 import '../errors-Dwqw-T_m.js';
 import '../schema-m0gsnbt3.js';
+import '../summary-report-ByiOUrHj.js';
 import '../failure-cluster-CL7IVgkJ.js';
 import '../store-CKUAgsJz.js';
 import '../judge-calibration-DilmB3Ml.js';

package/dist/{agent-profile-9J9hxdm2.d.ts → agent-profile-DzcPHR1Z.d.ts}

@@ -1,5 +1,5 @@
 import { A as AgentEvalError } from './errors-Dwqw-T_m.js';
-import { R as RunRecord } from './run-record-DgUVo5pw.js';
+import { R as RunRecord } from './run-record-BgTFzO2r.js';
 
 /**
  * Backend-integrity guard: distinguish "agent failed" from "eval ran against

package/dist/benchmarks/index.d.ts

@@ -1,4 +1,4 @@
-export { B as BENCHMARK_SPLIT_SEED, a as BenchmarkAdapter, b as BenchmarkDatasetItem, c as BenchmarkEvaluation, d as deterministicSplit, e as routing } from '../index-Bvk35ils.js';
-import '../run-record-DgUVo5pw.js';
+export { B as BENCHMARK_SPLIT_SEED, a as BenchmarkAdapter, b as BenchmarkDatasetItem, c as BenchmarkEvaluation, d as deterministicSplit, e as routing } from '../index-DsnOpCO6.js';
+import '../run-record-BgTFzO2r.js';
 import '../errors-Dwqw-T_m.js';
 import '../schema-m0gsnbt3.js';

package/dist/campaign/index.d.ts

@@ -1,20 +1,167 @@
-import { C as CampaignStorage } from '../provenance-D0WeCXt1.js';
-export { B as BuildLoopProvenanceArgs, D as DefaultProductionGateOptions, E as EmitLoopProvenanceArgs, a as EmitLoopProvenanceResult, b as EvolutionaryDriverOptions, G as GepaDriverConstraints, c as GepaDriverOptions, H as HeldOutGateOptions, L as LoopProvenanceBackend, d as LoopProvenanceCandidate, e as LoopProvenanceRecord, O as OpenAutoPrOptions, f as OpenAutoPrResult, R as RunCampaignOptions, g as RunEvalOptions, h as RunImprovementLoopOptions, i as RunImprovementLoopResult, j as RunOptimizationOptions, k as RunOptimizationResult, l as buildLoopProvenanceRecord, m as composeGate, n as countSentenceEdits, o as defaultProductionGate, p as defaultRenderDiff, q as emitLoopProvenance, r as evolutionaryDriver, s as extractH2Sections, t as fsCampaignStorage, u as gepaDriver, v as heldOutGate, w as inMemoryCampaignStorage, x as loopProvenanceSpans, y as openAutoPr, z as provenanceRecordPath, A as provenanceSpansPath, F as runCampaign, I as runEval, J as runImprovementLoop, K as runOptimization, M as surfaceContentHash, N as surfaceHash } from '../provenance-D0WeCXt1.js';
-import { L as LabeledScenarioStore, c as LabeledScenarioWrite, d as LabeledScenarioSampleArgs, e as LabeledScenarioRecord, f as LabelTrust, S as Scenario, b as DispatchContext, a as JudgeConfig, g as LabeledScenarioSource, C as CampaignResult, h as CodeSurface } from '../types-Beb6KPqZ.js';
-export { i as CampaignAggregates, j as CampaignArtifactWriter, k as CampaignCellResult, l as CampaignCostMeter, m as CampaignTokenUsage, n as CampaignTraceWriter, D as DispatchFn, G as Gate, o as GateContext, p as GateDecision, q as GateResult, r as GenerationCandidate, s as GenerationRecord, I as ImprovementDriver, t as JudgeAggregate, u as JudgeDimension, J as JudgeScore, M as MutableSurface, v as Mutator, O as OptimizerConfig, P as ProposeContext, w as ProposedCandidate, R as RedactionStatus, x as ScenarioAggregate, y as SessionScript, T as TraceSpan, z as isProposedCandidate, A as labelTrustRank } from '../types-Beb6KPqZ.js';
-import { A as AgentProfile, B as BackendIntegrityReport } from '../agent-profile-9J9hxdm2.js';
+import { a as RunCampaignOptions, C as CampaignStorage } from '../provenance-cUnovpWV.js';
+export { B as BuildLoopProvenanceArgs, D as DefaultProductionGateOptions, m as EmitLoopProvenanceArgs, n as EmitLoopProvenanceResult, E as EvolutionaryDriverOptions, o as GepaDriverConstraints, G as GepaDriverOptions, H as HeldOutGateOptions, p as LoopProvenanceBackend, q as LoopProvenanceCandidate, L as LoopProvenanceRecord, O as OpenAutoPrOptions, s as OpenAutoPrResult, b as RunEvalOptions, c as RunImprovementLoopOptions, R as RunImprovementLoopResult, t as RunOptimizationOptions, u as RunOptimizationResult, v as buildLoopProvenanceRecord, d as composeGate, w as countSentenceEdits, e as defaultProductionGate, x as defaultRenderDiff, y as emitLoopProvenance, f as evolutionaryDriver, z as extractH2Sections, g as fsCampaignStorage, h as gepaDriver, i as heldOutGate, j as inMemoryCampaignStorage, A as loopProvenanceSpans, F as openAutoPr, I as provenanceRecordPath, J as provenanceSpansPath, r as runCampaign, k as runEval, l as runImprovementLoop, K as runOptimization, M as surfaceContentHash, N as surfaceHash } from '../provenance-cUnovpWV.js';
+import { L as LlmClientOptions } from '../llm-client-DbjLfz-K.js';
+import { I as ImprovementDriver, L as LabeledScenarioStore, q as LabeledScenarioWrite, r as LabeledScenarioSampleArgs, s as LabeledScenarioRecord, t as LabelTrust, S as Scenario, M as MutableSurface, b as DispatchContext, a as JudgeConfig, u as LabeledScenarioSource, f as CampaignResult, h as CodeSurface } from '../types-c2R2kfmv.js';
+export { C as CampaignAggregates, c as CampaignArtifactWriter, d as CampaignCellResult, e as CampaignCostMeter, v as CampaignTokenUsage, g as CampaignTraceWriter, D as DispatchFn, G as Gate, i as GateContext, j as GateDecision, k as GateResult, l as GenerationCandidate, m as GenerationRecord, w as JudgeAggregate, n as JudgeDimension, J as JudgeScore, o as Mutator, O as OptimizerConfig, P as ParetoParent, x as ProposeContext, y as ProposedCandidate, R as RedactionStatus, z as ScenarioAggregate, p as SessionScript, T as TraceSpan, A as isProposedCandidate, B as labelTrustRank } from '../types-c2R2kfmv.js';
+import { A as AgentProfile, B as BackendIntegrityReport } from '../agent-profile-DzcPHR1Z.js';
 import { A as AgentEvalError } from '../errors-Dwqw-T_m.js';
-import { a as RunSplitTag, R as RunRecord } from '../run-record-DgUVo5pw.js';
-import '../llm-client-DbjLfz-K.js';
-import '../raw-provider-sink-C46HDghv.js';
+import { b as RunSplitTag, R as RunRecord } from '../run-record-BgTFzO2r.js';
 import '../red-team-DW9Ca_tj.js';
 import '../dataset-B2kL-fSM.js';
 import '../store-CKUAgsJz.js';
 import '../schema-m0gsnbt3.js';
-import '../index-D9dwa00f.js';
-import '../summary-report-BQvXpvaR.js';
+import '../index-GISRh500.js';
+import '../summary-report-ByiOUrHj.js';
 import '../failure-cluster-CL7IVgkJ.js';
 import '../judge-calibration-DilmB3Ml.js';
+import '../raw-provider-sink-C46HDghv.js';
+
+/**
+ * @experimental
+ *
+ * SkillOpt patch primitives (Microsoft, arXiv:2605.23904 — "Executive
+ * Strategy for Self-Evolving Agent Skills"). Where GEPA regenerates a surface
+ * by reflection, SkillOpt emits BOUNDED, anchored edits to ONE skill document
+ * — add / delete / replace — and accepts an edit only if it strictly improves
+ * a held-out score. Bounded edits are the "textual learning rate": small,
+ * reversible, and cheap to accept/reject, so a good rule introduced earlier is
+ * not overwritten by a later sweeping rewrite.
+ *
+ * This module applies a patch deterministically and reports, per op, what
+ * applied and what could not (a missing anchor is a rejected op, never a
+ * silently dropped one). Pure, no I/O.
+ */
+/** A single bounded edit against a skill surface.
+ *   - `add`     — insert `text` after the first line containing `after`
+ *                 (append to the end when `after` is absent/empty).
+ *   - `delete`  — remove the first line containing `anchor`.
+ *   - `replace` — replace the first line containing `anchor` with `text`.
+ *  `text` may be multi-line; it is spliced in as multiple lines. Anchors match
+ *  the FIRST line that contains the substring (deterministic; SkillOpt is
+ *  expected to anchor on unique text). */
+type SkillPatchOp = {
+    op: 'add';
+    after?: string;
+    text: string;
+} | {
+    op: 'delete';
+    anchor: string;
+} | {
+    op: 'replace';
+    anchor: string;
+    text: string;
+};
+/** A named, attributable bundle of ops the optimizer proposes as one edit. */
+interface SkillPatch {
+    label: string;
+    rationale: string;
+    ops: SkillPatchOp[];
+}
+interface SkillPatchRejection {
+    op: SkillPatchOp;
+    reason: string;
+}
+interface ApplySkillPatchResult {
+    surface: string;
+    /** Count of ops that mutated the surface. */
+    applied: number;
+    /** Ops that could not apply (unanchored / empty), with the reason. The
+     *  surface still reflects every APPLIED op — partial application is honest,
+     *  and the caller decides whether a partial patch is worth scoring. */
+    rejected: SkillPatchRejection[];
+}
+/**
+ * Apply a SkillOpt patch to a text surface. Ops apply in array order against
+ * the evolving line buffer (an `add after X` followed by a `delete X` sees the
+ * inserted lines). A missing anchor rejects only that op; the rest still apply.
+ */
+declare function applySkillPatch(surface: string, patch: SkillPatch): ApplySkillPatchResult;
+/** Total ops in a patch — the edit-budget axis (SkillOpt's "textual learning
+ *  rate" caps this per epoch). */
+declare function patchEditCount(patch: SkillPatch): number;
+
+/**
+ * @experimental
+ *
+ * `skillOptDriver` — a patch-mode `ImprovementDriver` implementing SkillOpt
+ * (Microsoft, arXiv:2605.23904). Where `gepaDriver` regenerates the whole
+ * surface by reflection, SkillOpt proposes BOUNDED, anchored edits
+ * (add/delete/replace) to ONE skill document, so a good rule introduced
+ * earlier is not clobbered by a later sweeping rewrite. The edit budget is the
+ * paper's "textual learning rate"; a rejected-edit buffer + a slow-update
+ * meta-note steer the optimizer away from dead ends.
+ *
+ * This module is the PROPOSER — the LLM call that turns evidence into
+ * structured patches. The accept-only-if-held-out-improves loop, the budget
+ * annealing, and the rejected buffer live in the `runSkillOpt` preset, which
+ * owns the epoch hill-climb. The driver also conforms to `ImprovementDriver`
+ * (`propose` applies its patches to the current surface and returns the
+ * candidate surfaces) so it is a drop-in for `runOptimization` and a fair
+ * entrant in `compareDrivers`.
+ */
+
+/** Evidence the optimizer reflects on: where the current surface is weakest.
+ *  Computed by the caller (the preset uses a TRAIN campaign so proposals never
+ *  see the held-out split; the generic loop derives it from history). */
+interface SkillOptEvidence {
+    /** Lowest-scoring scenarios (drives WHICH behavior to patch). */
+    weakScenarios: Array<{
+        scenarioId: string;
+        composite: number;
+    }>;
+    /** Lowest-scoring judge dimensions (drives WHAT to patch for). */
+    weakDimensions: Array<{
+        dimension: string;
+        score: number;
+    }>;
+}
+/** A patch that was tried and not accepted — fed back to the model so it does
+ *  not re-propose a dead end (SkillOpt's rejected-edit buffer). */
+interface RejectedEdit {
+    label: string;
+    rationale: string;
+    reason: string;
+}
+interface ProposePatchesArgs {
+    surface: string;
+    evidence: SkillOptEvidence;
+    /** Max ops per patch this round (the annealed textual learning rate). */
+    editBudget: number;
+    rejectedBuffer: RejectedEdit[];
+    /** Slow-update meta guidance accumulated across epochs. */
+    metaNote?: string;
+    /** How many candidate patches to propose. */
+    count: number;
+    signal: AbortSignal;
+}
+interface SkillOptDriverOptions {
+    llm: LlmClientOptions;
+    model: string;
+    /** What the skill document governs — orients the prompt. */
+    target: string;
+    /** Default ops-per-patch cap when used as a bare `ImprovementDriver`. The
+     *  `runSkillOpt` preset overrides this per epoch as it anneals. Default 3. */
+    editBudget?: number;
+    temperature?: number;
+    maxTokens?: number;
+    /** Top-K weak scenarios/dimensions surfaced as evidence. Default 3. */
+    evidenceK?: number;
+}
+interface SkillOptDriver extends ImprovementDriver {
+    /** Patch-native path used by `runSkillOpt` (the SkillOpt epoch loop owns
+     *  acceptance/budget/buffer). Returns structured patches, NOT surfaces. */
+    proposePatches(args: ProposePatchesArgs): Promise<SkillPatch[]>;
+}
+declare function skillOptDriver(opts: SkillOptDriverOptions): SkillOptDriver;
+/** Parse + validate the patch response. Throws `SkillPatchParseError` when the
+ *  response is not valid JSON at all (a router/model failure the caller must
+ *  see — never a silent no-op epoch). Returns `[]` only for the legitimate
+ *  "valid JSON, zero usable patches" case. Malformed ops within a patch are
+ *  dropped (not silently mutated); each patch is truncated to the edit budget. */
+declare class SkillPatchParseError extends Error {
+    constructor(message: string);
+}
+declare function parseSkillPatchResponse(raw: string, maxPatches: number, editBudget: number): SkillPatch[];
 
 /**
  * @experimental
@@ -77,6 +224,114 @@ declare class FsLabeledScenarioStore implements LabeledScenarioStore {
     private pathForSource;
 }
 
+/**
+ * @experimental
+ *
+ * `compareDrivers` — a head-to-head lift benchmark across optimizer drivers on
+ * ONE corpus. This is the forcing function: optimizer quality (GEPA reflection
+ * vs GEPA+Pareto vs SkillOpt) becomes a NUMBER with a confidence interval, so a
+ * driver regression — or shipping a simplified driver and calling it the real
+ * one — turns a build red instead of going measurement-invisible.
+ *
+ * Every entrant is scored the SAME way: each driver returns the surface it
+ * promoted, then `compareDrivers` scores the baseline + every winner on the
+ * SAME held-out scenarios with the SAME judges. Apples-to-apples by
+ * construction — the comparison never depends on how a driver measured itself.
+ * The per-scenario held-out composites feed a paired bootstrap (`statistics.ts`)
+ * for each driver's lift CI and for the pairwise "which driver wins" CI.
+ */
+
+/** What an optimizer produced: the surface it promoted + what it cost to get
+ *  there. `compareDrivers` does the held-out scoring itself, so an entry only
+ *  needs to run its loop and hand back the winner. */
+interface DriverEntry {
+    name: string;
+    optimize: () => Promise<{
+        winnerSurface: MutableSurface;
+        costUsd: number;
+        durationMs?: number;
+    }>;
+}
+interface DriverScore {
+    name: string;
+    /** Mean held-out composite of the baseline (identical across drivers). */
+    baselineComposite: number;
+    /** Mean held-out composite of this driver's promoted surface. */
+    winnerComposite: number;
+    /** Mean per-scenario held-out lift (winner − baseline). */
+    lift: number;
+    /** Paired-bootstrap CI of the per-scenario lift. `low > 0` ⇒ a real gain. */
+    liftCi: {
+        low: number;
+        high: number;
+    };
+    costUsd: number;
+    durationMs?: number;
+    winnerSurface: MutableSurface;
+    /** 1-based, by descending lift. */
+    rank: number;
+}
+interface DriverPairwise {
+    /** Higher-ranked driver. */
+    a: string;
+    b: string;
+    /** Mean per-scenario held-out delta (a − b). */
+    deltaMean: number;
+    low: number;
+    high: number;
+    /** `a` if the CI clears 0, `b` if it is entirely negative, else `'tie'`. */
+    favored: string;
+}
+interface DriverComparison {
+    /** Sorted by descending lift; `rank` set accordingly. */
+    scores: DriverScore[];
+    best: DriverScore;
+    /** Best vs each other driver, paired-bootstrap on the held-out winners. */
+    pairwise: DriverPairwise[];
+    holdoutScenarioIds: string[];
+}
+interface CompareDriversOptions<TScenario extends Scenario, TArtifact> extends Omit<RunCampaignOptions<TScenario, TArtifact>, 'dispatch' | 'scenarios'> {
+    drivers: DriverEntry[];
+    baselineSurface: MutableSurface;
+    /** The held-out scenarios every winner is scored on. */
+    holdoutScenarios: TScenario[];
+    /** Scores a surface on a scenario — the same dispatcher the drivers used. */
+    dispatchWithSurface: (surface: MutableSurface, scenario: TScenario, ctx: DispatchContext) => Promise<TArtifact>;
+    /** Bootstrap resamples for the lift CIs. Default 2000. */
+    resamples?: number;
+    /** CI confidence. Default 0.95. */
+    confidence?: number;
+}
+declare function compareDrivers<TScenario extends Scenario, TArtifact>(opts: CompareDriversOptions<TScenario, TArtifact>): Promise<DriverComparison>;
+/** Shared corpus + transport for the three built-in optimizer entries. */
+interface OptimizerEntryConfig<TScenario extends Scenario, TArtifact> {
+    baselineSurface: string;
+    /** Training scenarios the drivers reflect on. */
+    trainScenarios: TScenario[];
+    /** Held-out scenarios (the gate axis + the benchmark scoring axis). */
+    holdoutScenarios: TScenario[];
+    dispatchWithSurface: (surface: MutableSurface, scenario: TScenario, ctx: DispatchContext) => Promise<TArtifact>;
+    judges: JudgeConfig<TArtifact, TScenario>[];
+    llm: LlmClientOptions;
+    model: string;
+    target: string;
+    runDir: string;
+    seed?: number;
+    /** GEPA population per generation. Default 2. */
+    populationSize?: number;
+    /** GEPA generations. Default 3. */
+    maxGenerations?: number;
+    /** SkillOpt epochs. Default 6. */
+    maxEpochs?: number;
+    mutationPrimitives?: string[];
+}
+/** GEPA, reflection-only (single-parent, no Pareto combine). */
+declare function gepaReflectionEntry<TScenario extends Scenario, TArtifact>(config: OptimizerEntryConfig<TScenario, TArtifact>, name?: string): DriverEntry;
+/** GEPA with the Pareto frontier + combine-complementary-lessons. */
+declare function gepaParetoEntry<TScenario extends Scenario, TArtifact>(config: OptimizerEntryConfig<TScenario, TArtifact>, name?: string): DriverEntry;
+/** SkillOpt patch-mode hill-climb. */
+declare function skillOptEntry<TScenario extends Scenario, TArtifact>(config: OptimizerEntryConfig<TScenario, TArtifact>, name?: string): DriverEntry;
+
 /**
  * @experimental
  *
@@ -215,6 +470,128 @@ interface RunProfileMatrixResult<TArtifact, TScenario extends Scenario> {
 }
 declare function runProfileMatrix<TScenario extends Scenario, TArtifact>(opts: RunProfileMatrixOptions<TScenario, TArtifact>): Promise<RunProfileMatrixResult<TArtifact, TScenario>>;
 
+/**
+ * @experimental
+ *
+ * `runSkillOpt` — the SkillOpt epoch hill-climb (Microsoft, arXiv:2605.23904).
+ * Unlike `runOptimization`'s population/promote-top-K search, SkillOpt is a
+ * sequential, held-out-gated hill-climb on ONE skill document:
+ *
+ *   each epoch:
+ *     1. reflect on the CURRENT surface's weakest TRAIN scenarios/dimensions
+ *        (never the held-out split — proposals must not see the acceptance axis)
+ *     2. propose ≤ `patchesPerEpoch` bounded patches (≤ `editBudget` ops each)
+ *     3. apply each; score the candidate on the HELD-OUT split
+ *     4. ACCEPT the first patch that STRICTLY improves the held-out composite;
+ *        otherwise push it to the rejected-edit buffer (fed back so the model
+ *        does not re-propose dead ends)
+ *     5. anneal the edit budget down after consecutive rejections (the
+ *        "textual learning rate" decay); refresh the slow-update meta note
+ *     6. stop at `maxEpochs` or after `patience` epochs with no acceptance
+ *
+ * The accept-only-if-held-out-improves rule is the same discipline as
+ * `HeldOutGate`/`defaultProductionGate`, applied per edit instead of once at
+ * the end — which is why the held-out composite is monotonically
+ * non-decreasing and a regression can never ship. `runCampaign` is the
+ * measurement; `applySkillPatch` applies the edits; `skillOptDriver` proposes
+ * them.
+ */
+
+interface RunSkillOptOptions<TScenario extends Scenario, TArtifact> extends Omit<RunCampaignOptions<TScenario, TArtifact>, 'dispatch' | 'scenarios'> {
+    /** The skill document being optimized. */
+    baselineSurface: string;
+    /** Dispatcher taking the CURRENT skill surface + scenario → artifact. */
+    dispatchWithSurface: (surface: string, scenario: TScenario, ctx: DispatchContext) => Promise<TArtifact>;
+    driver: SkillOptDriver;
+    /** Scenarios the optimizer reflects on for evidence. MUST be disjoint from
+     *  `holdoutScenarios` — proposals never see the acceptance axis. */
+    trainScenarios: TScenario[];
+    /** Held-out scenarios. An edit is accepted ONLY if it strictly improves the
+     *  mean composite here. */
+    holdoutScenarios: TScenario[];
+    maxEpochs: number;
+    /** Candidate patches proposed per epoch. Default 2. */
+    patchesPerEpoch?: number;
+    /** Initial ops-per-patch cap (the textual learning rate). Default 3. */
+    editBudget?: number;
+    /** Strict acceptance margin: accept iff the held-out composite improves by
+     *  MORE than this. Default 0 (any strict improvement). */
+    minImprovement?: number;
+    /** Stop after this many consecutive epochs with no acceptance. Default =
+     *  `maxEpochs` (never early-stop). */
+    patience?: number;
+    /** Shrink the edit budget by 1 after 2 consecutive rejected epochs (min 1).
+     *  Default true. */
+    budgetAnneal?: boolean;
+    /** Cap on the rejected-edit buffer (most-recent kept). Default 12. */
+    rejectedBufferSize?: number;
+    /** Refresh the slow-update meta note every N epochs. Default 2. 0 disables. */
+    slowMetaEvery?: number;
+    /** Top-K weak scenarios/dimensions surfaced as evidence each epoch. Default 3. */
+    evidenceK?: number;
+    /** Abort signal forwarded to the patch-proposing LLM calls. */
+    signal?: AbortSignal;
+}
+interface AcceptedEdit {
+    epoch: number;
+    label: string;
+    rationale: string;
+    /** Held-out composite improvement vs the surface before this edit. */
+    holdoutDelta: number;
+}
+interface SkillOptEpochRecord {
+    epoch: number;
+    editBudget: number;
+    proposed: number;
+    /** The accepted edit this epoch, or null if every proposal was rejected. */
+    accepted: AcceptedEdit | null;
+    rejected: RejectedEdit[];
+    /** Held-out composite of the CURRENT surface at the END of the epoch. */
+    holdoutComposite: number;
+}
+interface RunSkillOptResult {
+    winnerSurface: string;
+    baselineHoldoutComposite: number;
+    winnerHoldoutComposite: number;
+    /** `winnerHoldoutComposite - baselineHoldoutComposite` — monotonically ≥ 0
+     *  by construction (only strictly-improving edits are accepted). */
+    lift: number;
+    acceptedEdits: AcceptedEdit[];
+    rejectedEdits: RejectedEdit[];
+    epochsRun: number;
+    history: SkillOptEpochRecord[];
+    /** Total cost across every scoring campaign (train evidence + holdout
+     *  acceptance) the hill-climb ran. */
+    totalCostUsd: number;
+}
+declare function runSkillOpt<TScenario extends Scenario, TArtifact>(opts: RunSkillOptOptions<TScenario, TArtifact>): Promise<RunSkillOptResult>;
+
+/**
+ * @experimental
+ *
+ * Shared campaign-score reductions used by every optimizer preset
+ * (`runOptimization`, `runSkillOpt`, `compareDrivers`). ONE definition of
+ * "composite of a campaign" and "per-scenario / per-dimension breakdown" so
+ * the optimizers cannot drift on how a surface's score is computed.
+ */
+
+/** Mean composite across a campaign: per cell, the mean of its judges'
+ *  composites; then the mean across cells. Cells with no judge scores are
+ *  skipped. Empty ⇒ 0. */
+declare function campaignMeanComposite<TArtifact, TScenario extends Scenario>(campaign: CampaignResult<TArtifact, TScenario>): number;
+interface CampaignBreakdown {
+    /** Mean score per judge dimension across all cells. */
+    dimensions: Record<string, number>;
+    /** Per-scenario composite (mean over reps + judges). */
+    scenarios: Array<{
+        scenarioId: string;
+        composite: number;
+    }>;
+}
+/** Per-candidate evidence a reflective/patch driver grounds its next proposal
+ *  on: mean score per judge dimension + per-scenario composite. */
+declare function campaignBreakdown<TArtifact, TScenario extends Scenario>(campaign: CampaignResult<TArtifact, TScenario>): CampaignBreakdown;
+
 /**
  * @experimental
  *
@@ -270,4 +647,4 @@ declare function gitWorktreeAdapter(opts: GitWorktreeAdapterOptions): WorktreeAd
  *  as a ref under the adapter's worktree dir. */
 declare function resolveWorktreePath(surface: CodeSurface, worktreeDir?: string): string;
 
-export { CampaignResult, CampaignStorage, CodeSurface, DispatchContext, FsLabeledScenarioStore, type FsLabeledScenarioStoreOptions, type GitWorktreeAdapterOptions, JudgeConfig, LabelTrust, LabeledScenarioRecord, LabeledScenarioSampleArgs, LabeledScenarioSource, LabeledScenarioStore, LabeledScenarioStoreError, LabeledScenarioWrite, type ProfileDispatchFn, ProfileMatrixError, type ProfileSummary, type RunProfileMatrixOptions, type RunProfileMatrixResult, Scenario, type ScenarioRollup, type Worktree, type WorktreeAdapter, WorktreeAdapterError, gitWorktreeAdapter, resolveWorktreePath, runProfileMatrix };
+export { type AcceptedEdit, type ApplySkillPatchResult, type CampaignBreakdown, CampaignResult, CampaignStorage, CodeSurface, type CompareDriversOptions, DispatchContext, type DriverComparison, type DriverEntry, type DriverPairwise, type DriverScore, FsLabeledScenarioStore, type FsLabeledScenarioStoreOptions, type GitWorktreeAdapterOptions, ImprovementDriver, JudgeConfig, LabelTrust, LabeledScenarioRecord, LabeledScenarioSampleArgs, LabeledScenarioSource, LabeledScenarioStore, LabeledScenarioStoreError, LabeledScenarioWrite, MutableSurface, type OptimizerEntryConfig, type ProfileDispatchFn, ProfileMatrixError, type ProfileSummary, type ProposePatchesArgs, type RejectedEdit, RunCampaignOptions, type RunProfileMatrixOptions, type RunProfileMatrixResult, type RunSkillOptOptions, type RunSkillOptResult, Scenario, type ScenarioRollup, type SkillOptDriver, type SkillOptDriverOptions, type SkillOptEpochRecord, type SkillOptEvidence, type SkillPatch, type SkillPatchOp, SkillPatchParseError, type SkillPatchRejection, type Worktree, type WorktreeAdapter, WorktreeAdapterError, applySkillPatch, campaignBreakdown, campaignMeanComposite, compareDrivers, gepaParetoEntry, gepaReflectionEntry, gitWorktreeAdapter, parseSkillPatchResponse, patchEditCount, resolveWorktreePath, runProfileMatrix, runSkillOpt, skillOptDriver, skillOptEntry };