@tangle-network/agent-eval 0.125.0 → 0.126.1

package/docs/design/loop-taxonomy.md

@@ -1,212 +1,164 @@
-# Loop taxonomy: execution driver, worker, measurement, and surface proposer
+# Evaluation And Improvement Model
 
-This is the canonical vocabulary for the Tangle agent stack. It exists because
-the same word ("loop", "shot", "worker") was being used at three different
-layers, and the layers were getting conflated. Every role below has exactly
-one meaning. Use these words and nothing else.
+This document defines the package's internal terms and ownership boundaries.
+Public examples should use concrete words such as prompt, skill, case, and agent whenever possible.
 
-Cross-links: [`concepts.md`](../concepts.md) (eval mental model),
-[`campaign-proposers.md`](../campaign-proposers.md) (proposer catalog), and
-[`multi-shot-optimization.md`](../multi-shot-optimization.md) (GEPA).
+## The Evaluation Unit
 
-## Core Roles
+One evaluation cell contains:
 
-| Role | Definition | Lives at |
-|---|---|---|
-| **Execution driver** | The thing that decides or routes the next turn/action inside a sandbox or worker conversation. | Inner layer only |
-| **Surface proposer** | The thing that proposes the next prompt/config/code surface for the improvement loop to measure. | Outer layer only |
-| **Worker** | An agent harness instance (Claude Code, Codex, OpenCode, …) running inside a sandbox. Does the actual work; responds in chat. | Inner layer only |
-| **Sandbox** | A multi-harness VM. Hosts **1..N workers**, which can share a workspace. Not an agent: the substrate an agent runs in. | Inner layer only |
-| **Measurement** | Runs the worker over a set of scenarios and judges the outputs into a scorecard with confidence intervals. This is `runCampaign`. | Outer layer |
+```text
+candidate surface
++ scenario
++ repetition
++ dispatch
++ judges
+= artifact, scores, trace, usage, cost, and status
+```
 
-Two facts that trip people up:
+The dispatch runs the system under test.
+A judge converts one artifact into dimension scores and a composite score.
+A campaign repeats this process across cases and candidates.
 
-1. **A sandbox is not a worker.** One sandbox can hold ten workers: a driver
-   can coordinate CC + Codex + OpenCode siblings sharing one workspace, or a
-   fleet spread across machines. `runLoop`'s placement encodes exactly this:
-   `{ sibling, sandboxId }` = co-located workers; `{ fleet, fleetId,
-   machineId, sandboxId }` = workers across machines.
+## The Value Being Changed
 
-2. **"Driver" is reserved for execution.** The outer loop uses a
-   **surface proposer**: it proposes the next prompt / tool config / code
-   surface for the measurement loop.
+`MutableSurface` is the API type for the value under optimization.
+It can be:
 
-## The nesting
+- a prompt string,
+- a serialized configuration,
+- a code surface owned by a runtime,
+- a named component map for multi-part GEPA optimization.
 
-There are two loops. The outer one improves the thing the inner one runs.
+A campaign measures surfaces but does not decide how a product stores or activates them.
 
-```
-runImprovementLoop                         OUTER loop: improve the agent over time
-│
-├─ PROPOSER = SurfaceProposer              proposes a candidate SURFACE
-│           (evolutionary mutator |        (the worker's system prompt / tools / config)
-│            reflective analyst)           : NOT a conversation turn
-│
-└─ for each candidate surface:
-     │
-     runCampaign                           a MEASUREMENT: scores ONE surface
-     │
-     └─ for each scenario × rep:
-          │
-          dispatch(scenario)               THE SEAM: topology-opaque, returns an artifact
-          │
-          └─ runLoop / runMultishot        INNER loop: one conversation
-               ├─ DRIVER  = persona / user / planner    chats with ↓
-               └─ WORKERS = 1..N agent harnesses in 1..M sandboxes
-               │
-               → transcript / artifact
-        judge(artifact) → score
-   → scorecard + CIs
-  gate(winner vs baseline) → PR
-```
+## Two Candidate Paths
+
+### Complete Optimization Method
+
+`OptimizationMethod` owns search, candidate history, selection, and stopping.
+It receives train and selection cases and returns one selected surface.
+
+Use this path for:
+
+- `gepaOptimizationMethod()`, which calls official GEPA engines and composition functions,
+- `skillOptOptimizationMethod()`, which calls Microsoft's official `ReflACTTrainer`,
+- another optimizer that already owns its search behavior.
+
+Complete methods run through `compareOptimizationMethods()`.
+The comparison function scores selected surfaces on final cases after all optimization finishes.
 
-### `dispatch` is the topology-opaque seam
+### Caller-Owned Candidate Generator
 
-`dispatch(scenario) → artifact` is the boundary between the measurement layer
-and the execution layer. The measurement does **not** know or care how the
-artifact was produced. Behind the seam can be:
+`SurfaceProposer` suggests candidates inside Agent Eval's `runImprovementLoop()`.
+It receives the current surface, prior campaign history, findings, generation number, requested population size, and cancellation signal.
 
-- one LLM call,
-- one worker (CC) in one sandbox,
-- a conversation driver coordinating 10 workers (CC + Codex + OpenCode)
-  sharing a workspace in one sandbox,
-- a fleet across machines.
+Use this path when:
 
-All of it is invisible to `runCampaign`. This is why the substrate has no
-opinion about execution topology: the topology lives inside `dispatch`.
+- product rules generate candidates,
+- agent-runtime delegates candidate creation to a worker,
+- a human-authored list defines possible edits.
 
-### Corrected statements (things that were said backwards)
+Do not reproduce an upstream optimizer behind this interface.
+Use its complete method adapter so the upstream package retains control of its own search state.
 
-- The worker is the agent in the sandbox. The driver talks to it. ✓
-- `runCampaign` is a **measurement**, not a worker. It *runs the worker* (via
-  `dispatch`); the worker does not "run the eval".
-- The outer improvement loop has **no single worker**: its proposer proposes a
-  *surface*, and each surface is scored by a *measurement* that drives the
-  inner workers.
+## Three Data Partitions
 
-## The dataset flywheel: why every loop run matters
+| Partition | May author candidates | May select candidates | May rank final methods |
+|---|---:|---:|---:|
+| Train | yes | yes | no |
+| Selection | yes | yes | no |
+| Final test | no | no | yes |
 
-**Every loop run, regardless of why it ran, feeds the same dataset.** This is
-the through-line that ties measurement and improvement together.
+An `OptimizationMethodInput` has no final-test field.
+This structural omission prevents an optimizer from receiving final cases through the normal API.
+Official methods receive serialized train and selection cases, so both partitions are optimizer-visible.
 
-When `runCampaign` runs with a `labeledStore`, each cell captures
-`(scenario, artifact, judgeScore, source)` into the `LabeledScenarioStore`.
-The `source` discriminates *why* the run happened: but the captured tuple is
-identical in shape:
+Change authors must still avoid indirect leaks through files, environment variables, cached artifacts, or custom scenario descriptions.
 
-| `captureSource` | The run that produced it |
+## Main APIs
+
+| API | Responsibility |
 |---|---|
-| `'eval-run'` | a plain evaluation campaign |
-| `'production-trace'` | a real user conversation in production |
-| `'red-team'` | an adversarial probe |
-| `'synthetic'` | a generated scenario |
-| `'manual'` | a human-curated example |
-
-That captured corpus **is the GEPA training set.** A basic eval run, a
-production conversation, and an autoresearch loop all deposit the same
-`(input, output, reward)` tuples. The optimization proposer later samples from
-that corpus to evolve the surface. So:
-
-> Running *any* loop: even one whose purpose is not optimization: builds the
-> dataset that optimization needs. The flywheel turns whether or not you are
-> currently optimizing.
-
-This is enforced, not aspirational: `runImprovementLoop` **refuses**
-`tracing: 'off'` whenever a proposer is wired, precisely because a loop that
-doesn't feed the dataset is a loop that breaks the flywheel.
-
-Temporal-split discipline (train vs holdout, `capturedBefore`) and
-default-off-for-training of `production-trace` are enforced at the
-`LabeledScenarioStore.sample()` boundary so the flywheel cannot contaminate
-the holdout it is judged against. See `src/campaign/labeled-store/`.
-
-## One improvement loop, pluggable proposers
-
-The improvement loop is **proposer-agnostic**. `runOptimization` (the loop body)
-and `runImprovementLoop` (the gated-promotion shell) call
-`proposer.propose(...)` → measure → `proposer.decide(...)`. They do not know
-which strategy proposed the candidate. The API interface is `SurfaceProposer`:
-
-```ts
-interface SurfaceProposer<TFindings = unknown> {
-  kind: string
-  propose(args: {
-    currentSurface: MutableSurface
-    history: GenerationRecord[]   // what's been tried + scored
-    findings: TFindings[]         // external signal (e.g. analyst output)
-    populationSize: number
-    generation: number
-    signal: AbortSignal
-  }): Promise<MutableSurface[]>
-  decide?(args: { history: GenerationRecord[] }): { stop: boolean; reason?: string }
-}
-```
+| `runCampaign()` | Execute and score a fixed set of candidate cells. |
+| `runImprovementLoop()` | Search with a caller-owned `SurfaceProposer` and apply a release rule. |
+| `compareOptimizationMethods()` | Run complete methods and compare selected surfaces on shared final cases. |
+| `gepaOptimizationMethod()` | Adapt official GEPA recipes to Agent Eval execution and scoring. |
+| `skillOptOptimizationMethod()` | Adapt official SkillOpt training to Agent Eval execution and scoring. |
+
+## Runtime Ownership
+
+Agent Eval owns measurement:
+
+- scenarios,
+- dispatch contracts,
+- judges,
+- run records,
+- cost receipts,
+- statistics,
+- method comparison.
+
+Agent Runtime owns execution policy:
+
+- agent sessions,
+- worker creation,
+- steering,
+- code edits,
+- process placement,
+- activation in a running product.
+
+Agent Knowledge owns knowledge state:
+
+- sources,
+- retrieval,
+- memory adapters,
+- knowledge writes,
+- freshness and provenance.
+
+Runtime and knowledge packages can expose their values as candidate surfaces and use Agent Eval to measure them.
+Agent Eval must not import either consumer package.
+
+## Resume And Parallel Work
+
+Campaign storage keeps cell-level results and cost receipts.
+Official optimizer adapters add their own compatible-run identity and process lock.
+
+A compatible official run includes:
+
+- upstream package revision,
+- optimizer recipe or trainer settings,
+- starting surface,
+- train and selection descriptions,
+- evaluation ID for execution and scoring behavior,
+- seed,
+- work limits.
+
+SkillOpt and a direct GEPA engine can restore official state.
+Composed GEPA recipes restart and report `resumed: false`.
+Method-level concurrency and candidate-level concurrency are separate controls.
+Each method receives its own run directory and optimization spend account.
+
+## Cost Accounting
+
+Agent and judge calls must report receipts through `DispatchContext.cost`.
+Unknown spend remains unknown.
+
+Standard GEPA and SkillOpt model calls pass through Agent Eval's local proxy.
+The proxy enforces limits and records provider usage at caller-supplied rates.
+Other GEPA engines can report their own spend, but that amount remains incomplete because Agent Eval did not observe those model calls.
+Missing usage remains unknown instead of being treated as zero.
+
+## Promotion
+
+Optimization returns a candidate.
+Product activation remains a separate caller decision.
+
+The caller should require:
+
+- a calibrated score,
+- improvement on cases not used to author candidates,
+- acceptable regressions by dimension,
+- complete enough cost data for the decision,
+- an inspectable exact change.
 
-| Implementation | Strategy | How it proposes | Where it lives |
-|---|---|---|---|
-| `evolutionaryProposer` | Evolutionary (GEPA / AxGEPA) | Standalone `SurfaceProposer`. Mutates the current best surface into N candidates, blind to history beyond the current best. Optimizes against the dataset's rewards. | **agent-eval** (pure: dataset → surface, no sandbox) |
-| Runtime reflective proposer | Reflective | Cheap generator: drafts patches from the report and applies them into a worktree (shots=1, no sandbox). | **agent-runtime**: implements agent-eval's proposer contract |
-| Runtime agentic proposer | Agentic | Full generator: runs a coding harness in the worktree (≤ `maxImprovementShots`) to edit in place. | **agent-runtime** |
-
-This resolves the prior duplication where `runImprovementLoop` (evolutionary,
-agent-eval) and `runAnalystLoop` (reflective, agent-runtime) were two parallel
-loops doing "propose change → measure → gate → PR". There is **one loop** and
-one proposer contract. The reflective and agentic paths are two settings of the
-same cost dial, not separate outer loops. The dependency direction permits this
-cleanly: agent-eval is the leaf and owns the proposer contract; agent-runtime
-imports agent-eval and implements it.
-
-## What "the surface" is: improvement tiers
-
-`MutableSurface` is the thing the proposer changes. It has tiers, least → most
-invasive. `MutableSurface = string | CodeSurface` spans all of them: `string`
-for tiers 1–2, and a finalized `CodeSurface` for tier 4. A code surface's
-worktree path is only its locator; exact base/candidate commits, final tree,
-and binary-patch digest are its portable identity. Call `verifyCodeSurface`
-before executing the checkout so a moved ref or post-finalization mutation
-fails before measurement. Verification hashes raw files and executable modes
-without Git filters and rejects external symlinks or submodules whose bytes are
-not represented by the candidate tree.
-
-| Tier | Surface | Generator that changes it | Blast radius |
-|---|---|---|---|
-| 1 | System prompt / prompt-signature addendum | `evolutionaryProposer` (GEPA), `reflectiveGenerator` | prompt only |
-| 2 | Tool config / tool signatures | `reflectiveGenerator` | which tools, their schemas |
-| 3 | Knowledge (wiki / knowledge graph) | agent-knowledge's knowledge adapter | what the agent *knows* |
-| 4 | Code / scaffolding | `agenticGenerator` (coding harness reads codebase + report) → worktree / PR | the implementation itself |
-
-The cost/capability distinction:
-
-- **`reflectiveGenerator`** updates the *signatures*: prompt + tool surface
-  (tiers 1–2). Cheap (drafts patches, no sandbox), reversible, measured
-  directly against the dataset.
-- **`agenticGenerator`** updates the *code* (tier 4). A coding harness reads
-  the repository + the report, edits in a worktree, iterates up to
-  `maxImprovementShots`: measured by re-running the inner loop against the
-  changed code.
-
-Both are implementations of the one proposer contract (propose → measure → gate
-→ PR). They differ only in *what* they edit and *how invasive* it is: and both
-consume the **same dataset** the flywheel builds.
-
-## Vocabulary quick reference
-
-- **shot**: one conversational turn (driver says X, worker responds Y). Used
-  in `runMultishot`. Never used to mean a whole eval run.
-- **runMultishot**: many shots in one conversation; persona-driver ↔ one
-  router-agent. agent-eval.
-- **runLoop**: driver ↔ workers in sandboxes; topology-agnostic execution.
-  agent-runtime.
-- **runCampaign**: a measurement: a surface scored over N scenarios × M reps.
-  agent-eval. (A "campaign" = a coordinated batch of measurements.)
-- **runOptimization**: the improvement loop body: proposer suggests surfaces, each is measured, and only a candidate that beats the global incumbent is promoted. agent-eval.
-- **runImprovementLoop**: `runOptimization` + holdout re-score + release gate
-  + optional PR. agent-eval.
-- **runAnalystLoop**: reflective autoresearch: findings + knowledge updates +
-  improvement proposals. agent-runtime.
-- **SurfaceProposer**: the contract a surface proposer implements.
-  `evolutionaryProposer` (agent-eval) is one; agent-runtime can provide
-  reflective or agentic implementations.
-- **CandidateGenerator**: the byte-producing seam inside a runtime proposer;
-  `reflectiveGenerator` (cheap, no sandbox) and `agenticGenerator` (coding
-  harness in the worktree) are the two cost settings. agent-runtime.
+The package records the decision inputs but does not deploy a prompt, skill, model, code change, memory, or knowledge base.

package/docs/design.md

@@ -14,7 +14,7 @@ agent-runtime    Runs agents (chat turns, one-shot tasks, multi-attempt loops),
                  run as a trace, and exposes improve(), which composes agent-eval's improvement
                  loop. Produces the RunRecords + traces agent-eval scores. Depends on agent-eval.
 
-agent-eval       selfImprove, analyzeRuns, runCampaign + surface proposers (GEPA proposer, …),
+agent-eval       selfImprove, analyzeRuns, runCampaign, official optimizer adapters,
    (this repo)   the gates (heldOutGate, defaultProductionGate, paretoSignificanceGate), the
                  InsightReport, the RL bridge, the wire protocol. Depends on neither consumer.
 

package/docs/distributed-driver.md

@@ -1,10 +1,10 @@
 # Distributed campaign execution: coordinator-on-A, workers-on-B
 
-The coordinator process (running `runCampaign` / `runImprovementLoop` /
-`gepaProposer`) and the worker (running your actual agent) **do not have to live in the
-same process, machine, region, or cloud.** `Dispatch` is just a
-function: scenario in, artifact out. Whatever returns the artifact is
-the worker: local, remote, sandboxed, or fanned out across a fleet.
+The coordinator process runs `runCampaign()`, `runImprovementLoop()`, or `compareOptimizationMethods()`.
+The worker runs your actual agent.
+They do not have to live in the same process, machine, region, or cloud.
+`Dispatch` is a function from scenario to artifact.
+The implementation can be local, remote, sandboxed, or spread across a worker fleet.
 
 ## Why you'd want this
 
@@ -161,12 +161,9 @@ and using `cellPlacement` to fan across many of them.
   stay on the worker. We need to forward them in the response body so
   `defaultProductionGate`'s `budgetUsd` ceiling reflects total spend, not
   coordinator-side spend. Tracked as a 0.45.x follow-up.
-- **Per-cell artifact streaming**: when the worker writes intermediate
-  artifacts via `ctx.artifacts.write`, those land on the worker's
-  storage. For multi-worker campaigns you'll want a shared object store
-  (S3/GCS) reachable from both sides; today consumers wire that as a
-  `CampaignStorage` impl. A reference S3-backed storage is on the
-  roadmap.
+- **Per-cell artifact streaming**: when the worker writes intermediate artifacts through `ctx.artifacts.write`, those files remain in worker storage.
+  Multi-worker campaigns need a shared `CampaignStorage` implementation reachable from both sides.
+  This package does not include an S3 or GCS implementation.
 - **gRPC / NATS / Temporal transports**: the wire is HTTP today by
   default because everything speaks HTTP. Other transports can ship as
   additional adapters; the `Dispatch` interface itself is

package/docs/feature-guide.md

@@ -5,9 +5,8 @@ then shows when to use each one.
 
 ## Overview
 
-LLM agents can write code, drafts, research, plans, and actions. The hard part
-is knowing whether they actually did a good job, whether they should keep
-trying, and whether a change made them better or worse.
+LLM agents can write code, drafts, research, plans, and actions.
+Evaluation measures whether the result met its requirements, whether another attempt is warranted, and whether a change improved outcomes.
 
 `agent-eval` gives you reusable tools for that:
 
@@ -18,8 +17,7 @@ trying, and whether a change made them better or worse.
 - **Feedback trajectories** turn normal user approvals/rejections into training
   and eval data.
 - **Datasets and holdouts** keep examples organized so you do not overfit.
-- **Optimizers and mutation loops** try prompt/signature/code variants and keep
-  the ones that really improve.
+- **Optimizers and mutation loops** try prompt, signature, or code variants and retain measured improvements.
 - **Traces and telemetry** show what happened, step by step.
 
 ## Which Primitive Should I Use?
@@ -27,15 +25,15 @@ trying, and whether a change made them better or worse.
 | Problem | Use | Why |
 | --- | --- | --- |
 | “Did this single answer/draft pass?” | Judge or rubric | Fast quality signal for one artifact. |
-| “Does generated code actually work?” | `BuilderSession`, `MultiLayerVerifier`, sandbox harness | Build/test/runtime gates catch failures judges miss. |
+| “Does generated code work?” | `BuilderSession`, `MultiLayerVerifier` | Build, test, and runtime checks catch failures judges miss. |
 | “Should the agent keep trying?” | `runAgentControlLoop` | Budgeted `observe -> validate -> decide -> act` runtime. |
 | “The agent should propose, verify, review, and revise.” | `runProposeReviewAsControlLoop` | Reusable preset over the generic control loop. |
 | “Human feedback should become reusable eval data.” | `FeedbackTrajectory` | Captures approvals, rejections, edits, choices, metrics, and policy blocks. |
 | “I want Vercel-style eval folders agents can add quickly.” | `loadEvalFixtureScenarios`, `planEvalFixtureRun`, `runCampaign` | Folder-per-eval DX with campaign scoring and cache-safe dry runs. |
 | “Can this action run, or does it need approval?” | `evaluateActionPolicy` | Generic preflight for side effects, budgets, and required evidence. |
 | “I need train/dev/test/holdout examples.” | `Dataset` plus feedback trajectory conversion | Stable splits and contamination control. |
-| “Which prompt or signature wins?” | `runImprovementLoop`, steering optimizers | Runs variants on scenarios and compares scores. |
-| “Improve a multi-turn agent over real task traces.” | `runImprovementLoop` | GEPA-style trajectory optimization with ASI and held-out promotion. |
+| “Which optimization procedure wins?” | `compareOptimizationMethods` | Runs complete methods on shared train and selection cases, then compares them on separate final cases. |
+| “Improve a multi-turn agent with candidates from my runtime.” | `runImprovementLoop` | Evaluates caller-generated candidates and applies a separate release rule. |
 | “Improve prompts, then code if prompts plateau.” | `runPromptEvolution`, composite mutator, code mutator | Bounded evolution with telemetry and lineage. |
 | “Find why a regression happened.” | bisector, traces, run records | Narrows changes and preserves evidence. |
 | “Expose evals to another language.” | Wire protocol and Python client | HTTP/RPC boundary for non-TypeScript apps. |
@@ -102,21 +100,24 @@ Result:
 generated code -> build/test/runtime gates -> score -> ship or revise
 ```
 
-### Prompt/Signature Optimizer
+### Prompt And Skill Optimization
 
-Use when you want Ax/GEPA-style improvement.
+Use `gepaOptimizationMethod()` for official GEPA recipes.
+Use `skillOptOptimizationMethod()` for Microsoft's official SkillOpt trainer.
+For DSPy programs, use official DSPy optimizers with Python's `DspyJudgeMetric`.
+Run complete methods through `compareOptimizationMethods()`.
+Use `runImprovementLoop()` only when your application or runtime owns candidate generation through `SurfaceProposer`.
 
-1. For variable-length agent tasks, use `runImprovementLoop`.
-2. Build search/dev/test/holdout splits from the real product loop.
-3. Score full trajectories, not just final text.
-4. Emit actionable side information for failures the mutator can fix.
-5. Promote only `promotedVariant`, never a rejected `searchBestVariant`.
-6. Keep run records with prompt hash, model, config, cost, and commit.
+1. Build train, selection, and final splits from the real product path.
+2. Score complete trajectories rather than final text alone.
+3. Give optimizers actionable evidence from train and selection failures.
+4. Keep final cases out of candidate generation and method selection.
+5. Record prompt hash, model, configuration, cost status, and source revision.
 
 Result:
 
 ```text
-candidate variant -> repeated evals -> statistical comparison -> promotion gate
+candidate variant -> repeated evaluations -> statistical comparison -> release decision
 ```
 
 Do not optimize a toy harness if users run a different product loop. Build
@@ -158,9 +159,9 @@ Store as `FeedbackTrajectory`, then derive:
 | Action policy | `evaluateActionPolicy` | Approval/budget preflight | Blocks or labels actions before `act()`. |
 | Datasets | `Dataset`, holdout tools, canaries | Train/dev/test/holdout corpora | Keeps optimization honest. |
 | Eval fixtures | `discoverEvalFixtures`, `loadEvalFixtureScenarios`, `planEvalFixtureRun` | Coding-agent eval folders | Vercel-style input shape; still executes through `runCampaign`. |
-| Optimization | `runImprovementLoop`, steering optimizers | Prompt/signature comparison | Use held-out gates before promotion. |
+| Optimization | `compareOptimizationMethods`, official optimizer adapters, `runImprovementLoop` | Prompt and skill comparison | Reserve separate final cases for method ranking. |
 | Evolution | prompt/code mutators, sandbox pool, telemetry | Autoresearch and mutation loops | Use budgets and lineage; do not run unbounded. |
-| Telemetry | `TraceStore`, OTLP, file sinks | Audit and replay | Treat traces as evidence, not just logs. |
+| Telemetry | `TraceStore`, OTLP, file sinks | Evidence and replay | Store traces as inspectable run evidence. |
 | Reporting | summaries, pareto, cost tracker | Decision support | Useful for PRs, launch gates, research notes. |
 
 ## Guardrails

package/docs/knowledge-readiness.md

@@ -77,8 +77,5 @@ Knowledge-related failures use the normal failure taxonomy:
 - `contradictory_evidence`
 - `ambiguous_user_intent`
 
-For optimization, scorers should use responsible surfaces such as
-`knowledge-requirements`, `data-acquisition`, `retrieval-policy`, and
-`user-question-policy` in actionable side information. That lets GEPA-style
-loops improve data acquisition and retrieval policy instead of blaming every
-failure on the prompt.
+For optimization, scorers should identify responsible surfaces such as `knowledge-requirements`, `data-acquisition`, `retrieval-policy`, and `user-question-policy`.
+That evidence lets an official optimization method or caller-owned `SurfaceProposer` improve acquisition and retrieval policy instead of blaming every failure on the prompt.

package/docs/multi-shot-optimization.md

@@ -1,32 +1,45 @@
 # Improve One Surface
 
-`runImprovementLoop` generates candidate prompts or configs, runs them on training scenarios, and checks the selected candidate on separate holdout scenarios.
+`runImprovementLoop()` evaluates candidates from a `SurfaceProposer`, selects one on training data, and compares it with the baseline on separate holdout cases.
 
-Use it when the question is: "Can this method improve my current surface?"
-Use [`compareOptimizationMethods`](./campaign-proposers.md#compare-complete-methods) when the question is: "Which complete optimization method performs best?"
+Use it when your application or runtime owns candidate generation.
+Use [`compareOptimizationMethods()`](./campaign-proposers.md) when GEPA, SkillOpt, or another external system owns the complete search procedure.
 
 ## Inputs
 
 | Input | Meaning |
 |---|---|
-| `baselineSurface` | Current prompt or config. |
-| `scenarios` | Training scenarios used to generate and score candidates. |
-| `holdoutScenarios` | Separate scenarios used by the release rule after search. |
-| `dispatchWithSurface` | Runs one scenario with one candidate surface. |
+| `baselineSurface` | Current prompt or serialized configuration. |
+| `scenarios` | Training cases used to generate and score candidates. |
+| `holdoutScenarios` | Separate cases used for the release decision after search. |
+| `dispatchWithSurface` | Runs one case with one candidate surface. |
 | `judges` | Scores the returned artifact. |
-| `proposer` | Generates candidate surfaces. |
-| `gate` | Applies the caller's release rule to baseline and winner results. |
-| `runDir` | Stores run artifacts, traces, and resumable state. |
+| `proposer` | Caller-owned candidate generator. |
+| `gate` | Caller-owned release rule for the baseline and selected candidate. |
+| `runDir` | Directory for run artifacts, traces, and resumable state. |
 
 ## Example
 
 ```ts
 import {
   defaultProductionGate,
-  gepaProposer,
   runImprovementLoop,
+  type SurfaceProposer,
 } from '@tangle-network/agent-eval/campaign'
 
+const proposer: SurfaceProposer = {
+  kind: 'product-rules',
+  async propose({ currentSurface, populationSize }) {
+    return [
+      {
+        surface: `${String(currentSurface)}\nReturn JSON only.`,
+        label: 'json-only',
+        rationale: 'Training failures contained prose around the JSON object.',
+      },
+    ].slice(0, populationSize)
+  },
+}
+
 const result = await runImprovementLoop({
   baselineSurface: currentSystemPrompt,
   scenarios: trainScenarios,
@@ -34,13 +47,9 @@ const result = await runImprovementLoop({
   dispatchWithSurface: async (surface, scenario, ctx) =>
     runYourAgent({ prompt: String(surface), scenario, signal: ctx.signal }),
   judges: [qualityJudge],
-  proposer: gepaProposer({
-    llm: { apiKey, baseUrl },
-    model,
-    target: 'the complete system prompt',
-  }),
-  populationSize: 4,
-  maxGenerations: 4,
+  proposer,
+  populationSize: 1,
+  maxGenerations: 1,
   gate: defaultProductionGate({
     holdoutScenarios,
     deltaThreshold: 0,
@@ -54,14 +63,13 @@ if (result.gateResult.decision === 'ship') {
 }
 ```
 
-## Behavior
+## Rules
 
-- Training and holdout scenario IDs must be disjoint.
-- Candidate generation cannot read holdout judge scores through `SurfaceProposer`.
-- The selected candidate is measured against the baseline on holdout scenarios.
-- A selected surface identical to the baseline is held instead of treating model variance as lift.
-- `result.cost` includes worker, candidate-generation, and judge calls recorded through the shared cost ledger.
-- `result.promotedDiff` describes the exact selected surface change.
+- Training and holdout case IDs must be disjoint.
+- Candidate generation cannot read holdout scores through `SurfaceProposer`.
+- An unchanged selected surface does not receive credit for model variance.
+- `result.cost` includes calls recorded through the shared cost ledger.
+- `result.promotedDiff` identifies the exact selected change.
 
-The release decision is only as useful as the scenarios and judges supplied by the caller.
-Calibrate the judge on known strong and weak outputs before using it for promotion.
+Calibrate the judge on known strong and weak artifacts before using its decision in production.
+The runnable offline example is [`examples/multi-shot-optimization`](../examples/multi-shot-optimization/).