@tangle-network/agent-eval 0.123.1 → 0.123.2

package/docs/design.md

@@ -0,0 +1,70 @@
+# Design rationale
+
+Why this package is shaped the way it is.
+This is background reading, not reference.
+The API itself is documented in [`concepts.md`](./concepts.md) and the [`README`](../README.md).
+
+## Composition with the Tangle agent stack
+
+`agent-eval` is one package in a larger stack.
+It sits at the bottom of the layering: consumers depend on it, it depends on none of them.
+
+```
+agent-runtime    Runs agents (chat turns, one-shot tasks, multi-attempt loops), captures every
+                 run as a trace, and exposes improve(), which composes agent-eval's improvement
+                 loop. Produces the RunRecords + traces agent-eval scores. Depends on agent-eval.
+
+agent-eval       selfImprove, analyzeRuns, runCampaign + surface proposers (GEPA proposer, …),
+   (this repo)   the gates (heldOutGate, defaultProductionGate, paretoSignificanceGate), the
+                 InsightReport, the RL bridge, the wire protocol. Depends on neither consumer.
+
+agent-knowledge  proposeKnowledgeWrites / applyKnowledgeWriteBlocks. agent-eval's analyst
+                 findings feed it; the knowledge gate consumes them. Depends on agent-eval.
+
+sandbox          Sandbox.create, streamPrompt. One execution environment the runtime's
+                 loops run on; agent-eval scores what comes back.
+```
+
+None of the sibling packages are required to use `agent-eval`; the library stands alone.
+The stack context matters only if you adopt more of it later.
+
+## The dependency rule
+
+This section is the public rationale.
+The enforceable maintainer rule lives in [`CLAUDE.md`](../CLAUDE.md#repo-layering--this-package-is-the-substrate).
+
+**`agent-eval` has zero upward dependencies on a consumer.**
+This is what keeps the package reusable outside our own stack: nothing in here imports from `agent-runtime`, `agent-knowledge`, or `sandbox`, whether at runtime, in development dependencies, or as type-only imports.
+
+The placement test for any shared type: *does this concept make sense without a running agent loop?*
+
+- Yes: it lives here. A judge score, a run record, a scenario, and a pass/fail verdict are all meaningful for a pile of logs with no agent attached.
+- No: it lives in the runtime layer. A validation context carrying an abort signal and a concrete sandbox session only exist mid-run.
+
+When in doubt, the type moves down into `agent-eval`: subtracting a dependency from a consumer is always cheaper than adding one here.
+Agent profile shape is the shared `@tangle-network/agent-interface` contract, so neither layer owns it.
+
+## Why "surface"
+
+The improvement loop needs one word for "the thing being changed", because it deliberately does not care what that thing is: a system prompt, a config object, a skill file, a set of few-shot examples.
+Proposers emit candidate surfaces, campaigns measure them, gates compare them against the baseline surface.
+Where a doc can say "prompt" concretely, it should; `surface` appears in API names where the generality is the point.
+
+## Why the report never invents signal
+
+Every section of the `analyzeRuns()` report is opt-in based on what the input data supports.
+If runs carry no judge scores, the judge section is empty rather than defaulted.
+If there is no baseline/candidate split, no lift is reported.
+Missing evidence is never scored as zero; a judge that throws is recorded as a failed cell, not silently folded into the average.
+The reasoning: a fabricated zero poisons every statistic downstream, and an eval library that quietly fabricates is worse than no eval at all.
+
+## Maintainer docs
+
+These files record operating conventions for maintainers and internal agents.
+They are not adoption reference:
+
+- [`building-doctrine.md`](./building-doctrine.md): conventions our agents follow when consuming this package (reachable model defaults, probe-before-debug, experiment integrity checklist)
+- [`self-improvement-map.md`](./self-improvement-map.md): how the one improvement loop is pointed at different surfaces, and the proposer bake-off narrative
+- [`design/loop-taxonomy.md`](./design/loop-taxonomy.md): the internal vocabulary for execution drivers, workers, measurements, and proposers
+- [`research-report-methodology.md`](./research-report-methodology.md): the evidence standard our own research reports are held to
+- [`.claude/skills/agent-eval/SKILL.md`](../.claude/skills/agent-eval/SKILL.md): directives for LLM agents writing integration code, encoding bug classes we have already shipped and fixed once

package/docs/distributed-driver.md

@@ -1,10 +1,10 @@
-# Distributed campaign execution — coordinator-on-A, workers-on-B
+# Distributed campaign execution: coordinator-on-A, workers-on-B
 
 The coordinator process (running `runCampaign` / `runImprovementLoop` /
 `gepaProposer`) and the worker (running your actual agent) **do not have to live in the
 same process, machine, region, or cloud.** `Dispatch` is just a
 function: scenario in, artifact out. Whatever returns the artifact is
-the worker — local, remote, sandboxed, or fanned out across a fleet.
+the worker: local, remote, sandboxed, or fanned out across a fleet.
 
 ## Why you'd want this
 
@@ -13,7 +13,7 @@ the worker — local, remote, sandboxed, or fanned out across a fleet.
 | **Coordinator on your VPC, workers on our sandbox fleet** | Coordinator holds secrets, training data, prompt corpus; workers stay stateless and scale horizontally |
 | **Multi-region campaigns** | Each cell runs in the region closest to its target API (latency, compliance, data residency) |
 | **Coordinator-as-a-service** | Long-running optimization process; reuses across many short-lived worker invocations |
-| **Heterogeneous workers** | One cell on a CPU container, another on a GPU box, another against a third-party API — same Dispatch shape, different placement |
+| **Heterogeneous workers** | One cell on a CPU container, another on a GPU box, another against a third-party API: same Dispatch shape, different placement |
 | **Budget-isolated workers** | Worker boxes get scoped, time-bounded credentials; coordinator never holds production keys |
 
 ## Two new pieces in 0.45.0
@@ -22,7 +22,7 @@ the worker — local, remote, sandboxed, or fanned out across a fleet.
 |---|---|
 | **`DispatchContext.placement?: string`** | Opaque placement key the substrate forwards to the Dispatch. |
 | **`RunCampaignOptions.cellPlacement?(input) → string \| undefined`** | Strategy function the substrate calls per cell to compute the placement key. |
-| **`@tangle-network/agent-eval/adapters/http`** | `httpDispatch` (client) + `runDispatchServer` (server) — wire shape for HTTP-based remote workers. |
+| **`@tangle-network/agent-eval/adapters/http`** | `httpDispatch` (client) + `runDispatchServer` (server): wire shape for HTTP-based remote workers. |
 
 Both ends of the wire are in the same package; no peer dep, no separate
 install. The substrate doesn't strategy-pick; you provide the
@@ -31,7 +31,7 @@ Dispatch reads it. Clean seam, no policy baked in.
 
 ## The three reference topologies
 
-### 1. In-process (the default — what you already have)
+### 1. In-process (the default: what you already have)
 
 ```ts
 await runCampaign({
@@ -74,7 +74,7 @@ import { runDispatchServer } from '@tangle-network/agent-eval/adapters/http'
 
 const handle = await runDispatchServer<MyScenario, MyArtifact>({
   dispatch: async (scenario, ctx) => {
-    // your agent — call OpenAI, LangChain, your sandbox, anything.
+    // your agent: call OpenAI, LangChain, your sandbox, anything.
     const artifact = await runMyAgent(scenario, ctx.signal)
     return artifact
   },
@@ -85,7 +85,7 @@ console.log(`worker listening on ${handle.port}`)
 ```
 
 Cancellation, retries on 5xx / 408 / 429, bounded timeouts, optional
-custom auth headers, optional `fetchImpl` override — all there.
+custom auth headers, optional `fetchImpl` override: all there.
 
 ### 3. Multi-region fan-out
 
@@ -121,7 +121,7 @@ await runCampaign({
 })
 ```
 
-`cellPlacement` is a pure function the substrate calls per cell — no
+`cellPlacement` is a pure function the substrate calls per cell: no
 state. Use whatever signal you want (tags, hash of scenario id,
 round-robin, region-affinity from a previous run, scheduling table).
 
@@ -134,7 +134,7 @@ round-robin, region-affinity from a previous run, scheduling table).
 | **Retries** | Idempotent retries on 5xx / 408 / 429 with exponential backoff + jitter. Coordinator aborts never retry. |
 | **Auth** | Bearer token on `Authorization`; pluggable via `auth: string \| () => string \| Promise<string>` for rotation/refresh. |
 | **Payload size** | Server enforces `maxBodyBytes` (default 10 MB). |
-| **Traces** | Both ends emit OTel — if both point at the same OTLP collector, you get a unified trace per cell. See `docs/adapters-observability.md`. |
+| **Traces** | Both ends emit OTel: if both point at the same OTLP collector, you get a unified trace per cell. See `docs/adapters-observability.md`. |
 | **Cost** | Worker's `ctx.cost.runPaidCall(...)` writes durable receipts in the worker process. Roll up those receipts server-side and attach them to worker telemetry; they are not forwarded to the coordinator automatically. |
 
 ## Running the reference example
@@ -142,10 +142,10 @@ round-robin, region-affinity from a previous run, scheduling table).
 See `examples/distributed-driver/`:
 
 ```sh
-# Terminal 1 — worker
+# Terminal 1: worker
 pnpm tsx examples/distributed-driver/worker.ts
 
-# Terminal 2 — coordinator
+# Terminal 2: coordinator
 WORKER_URL=http://localhost:8080/dispatch \
 WORKER_TOKEN=dev-token \
 pnpm tsx examples/distributed-driver/driver.ts
@@ -157,17 +157,17 @@ and using `cellPlacement` to fan across many of them.
 
 ## Known gaps + follow-ups
 
-- **Cost roll-up across the wire** — worker-side `ctx.cost` observations
+- **Cost roll-up across the wire**: worker-side `ctx.cost` observations
   stay on the worker. We need to forward them in the response body so
   `defaultProductionGate`'s `budgetUsd` ceiling reflects total spend, not
   coordinator-side spend. Tracked as a 0.45.x follow-up.
-- **Per-cell artifact streaming** — when the worker writes intermediate
+- **Per-cell artifact streaming**: when the worker writes intermediate
   artifacts via `ctx.artifacts.write`, those land on the worker's
   storage. For multi-worker campaigns you'll want a shared object store
   (S3/GCS) reachable from both sides; today consumers wire that as a
   `CampaignStorage` impl. A reference S3-backed storage is on the
   roadmap.
-- **gRPC / NATS / Temporal transports** — the wire is HTTP today by
+- **gRPC / NATS / Temporal transports**: the wire is HTTP today by
   default because everything speaks HTTP. Other transports can ship as
   additional adapters; the `Dispatch` interface itself is
   transport-agnostic.

package/docs/eval-surface-map.md

@@ -1,7 +1,7 @@
-# Eval surface map — which primitive, when
+# Eval surface map: which primitive, when
 
 The eval surface is a small set of orthogonal primitives. They compose; they do
-not overlap. If two seem interchangeable, read the "use when" — the distinction
+not overlap. If two seem interchangeable, read the "use when": the distinction
 is real and load-bearing. **Do not add a new wrapper to bridge two of these; the
 composition point already exists (see Produced-state grading below).**
 
@@ -10,15 +10,15 @@ composition point already exists (see Produced-state grading below).**
 | Primitive | Use when | Returns |
 |---|---|---|
 | `runCampaign` | The measurement primitive. Run a dispatch over scenarios × seeds × reps, score each with judges, aggregate. Caller owns the dispatch. | `CampaignResult` |
-| `runEval` | The simplest preset over `runCampaign` — just score, no loop, no gate. The 80% "I want a scorecard" case. | `CampaignResult` |
+| `runEval` | The simplest preset over `runCampaign`: just score, no loop, no gate. The 80% "I want a scorecard" case. | `CampaignResult` |
 | `runProfileMatrix` | Factor the SAME scenarios across N agent **profiles** (models / prompt variants), with RunRecord stamping + a real-backend integrity guard. | `RunRecord[]` |
 | `runOptimization` | GENERATE: measured or validated premeasured baseline → N generations of propose → measure → rank → promote. No release gate. | generations + winner |
 | `runImprovementLoop` | The release-gate shell around `runOptimization`: adds a held-out re-score + a promotion gate (+ optional auto-PR). | gate decision + winner |
-| `runEvalCampaign` | Inversion-of-control variant of `runCampaign` — the runner is handed a pre-wired trace/sink/emitter and integrity gating as a precondition. Use when you need full capture by construction. | `CampaignResult` + records |
+| `runEvalCampaign` | Inversion-of-control variant of `runCampaign`: the runner is handed a pre-wired trace/sink/emitter and integrity gating as a precondition. Use when you need full capture by construction. | `CampaignResult` + records |
 
 When variants of the same task run inside one `runCampaign`, give those scenarios the same `seedGroup` so each repetition uses common randomness.
 Use `runProfileMatrix` instead when profiles are separate campaign axes.
-| `runAgentMatrix` | The bare N-axis cartesian scheduler with concurrency control. The layer beneath the eval surface — reach for it only when you need raw scheduling, not eval semantics. | cell results |
+| `runAgentMatrix` | The bare N-axis cartesian scheduler with concurrency control. The layer beneath the eval surface: reach for it only when you need raw scheduling, not eval semantics. | cell results |
 
 Mental model: **measure** (`runCampaign`/`runEval`) → **factor** (`runProfileMatrix`) →
 **generate** (`runOptimization`) → **gate** (`runImprovementLoop`). `runEvalCampaign`
@@ -27,11 +27,11 @@ is `runCampaign` with capture inverted; `runAgentMatrix` is the scheduler undern
 Merging any two of these conflates distinct mental models (measure ≠ search ≠
 release-gate). Keep them separate; pick by the table.
 
-## Produced-state grading — there is NO persona-dispatch wrapper
+## Produced-state grading: there is NO persona-dispatch wrapper
 
 To grade what an agent actually **produced** (filed the proposal, wrote the
 artifact) rather than what it said, the composition point is a **judge that wraps
-`verifyCompletion`** — not a dedicated runner. The pipeline:
+`verifyCompletion`**: not a dedicated runner. The pipeline:
 
 ```
 runtime/app-tool events ──► extractProducedState(events) ──► ProducedState
@@ -43,20 +43,20 @@ runtime/app-tool events ──► extractProducedState(events) ──► Produce
 
 `extractProducedState` is a pure function over the produced-event stream; the
 judge calls it inline. This is why **`runProducedStatePersonaDispatch` does not
-exist and should not be built** — it would be a fourth layer over a composition
+exist and should not be built**: it would be a fourth layer over a composition
 that is already one judge. (Archetype: `playback.ts` `scoreUserStory`.)
 
 ### The in-band body contract
 
-Produced events carry their **body in-band** — the grader never reaches into a
+Produced events carry their **body in-band**: the grader never reaches into a
 product database to recover it:
 
 - `artifact` events carry `content` (the persisted file body).
-- `proposal_created` events carry `content` (the `submit_proposal` description) —
+- `proposal_created` events carry `content` (the `submit_proposal` description) -
   same role, same field name. A title-only filing omits it; a content-less
   proposal is graded presence-only (and, by the completion oracle's rule, does
   not count as a completed deliverable).
 
 A consumer that finds itself re-fetching a deliverable's body from its own DB to
-grade it is working around a thin event — fix the event (carry `content`), don't
+grade it is working around a thin event: fix the event (carry `content`), don't
 add an enrichment band-aid.

package/docs/hosted-ingest-spec.md

@@ -1,4 +1,4 @@
-# Hosted-ingest wire spec — `2026-05-26.v1`
+# Hosted-ingest wire spec: `2026-05-26.v1`
 
 The schema **every** orchestrator (ours, partners' self-hosted ones,
 any future open implementation) must accept. Frozen under semver:
@@ -149,7 +149,7 @@ Any orchestrator implementing this spec MUST:
 5. **Persist eval-runs durably**: at least the event + cell scores must
    survive an orchestrator restart. Trace spans MAY be best-effort.
 6. **Provide read access**: GET endpoints for the tenant to list + fetch
-   their own runs. Wire format for reads is NOT part of this spec — each
+   their own runs. Wire format for reads is NOT part of this spec: each
    orchestrator can pick its own (REST + JSON, gRPC, GraphQL).
 
 Servers SHOULD also:
@@ -163,7 +163,7 @@ Servers SHOULD also:
 
 ## Reference implementation
 
-`examples/hosted-ingest-server/` — a minimal hono-based receiver. ~200
+`examples/hosted-ingest-server/`: a minimal hono-based receiver. ~200
 LOC. Validates auth, accepts ingest, stores in memory, exposes a
 read endpoint. Runs anywhere Node runs.
 
@@ -200,4 +200,4 @@ receiver's `GET /v1/runs` lists it back.
 Servers MUST reject requests with `X-Tangle-Wire-Version` they don't
 support, with a 400 listing the versions they DO accept.
 
-The version string IS the spec id — pin against it.
+The version string IS the spec id: pin against it.

package/docs/improvement-glossary.md

@@ -1,12 +1,12 @@
 # Improvement glossary + proposer chooser + composition
 
-> **In plain terms:** this is the dictionary for the *improvement* half of the stack — the words that show up when you optimize an agent (proposer, surface, candidate, generation, holdout, lift, gate…) rather than when you *run* one (driver, worker, iteration — those live in [`agent-runtime/docs/glossary.md`](../../agent-runtime/docs/glossary.md)).
+> **In plain terms:** this is the dictionary for the *improvement* half of the stack: the words that show up when you optimize an agent (proposer, surface, candidate, generation, holdout, lift, gate…) rather than when you *run* one (driver, worker, iteration: those live in [`agent-runtime/docs/glossary.md`](../../agent-runtime/docs/glossary.md)).
 > Read this once and you can read any improvement result, pick a proposer, and wire two of them together.
 
 **Who this is for.**
 A **novice** (never seen the repo) should be able to read a `CampaignResult`, a `ProposeContext`, and a proposer chooser table without opening the source.
-An **expert** who knows DSPy/GEPA should be able to map their existing mental model onto our names in about a minute — every core term below carries a *"if you know DSPy/GEPA"* line.
-If code and this file disagree, the code wins — fix this file the same turn (the anti-staleness law).
+An **expert** who knows DSPy/GEPA should be able to map their existing mental model onto our names in about a minute: every core term below carries a *"if you know DSPy/GEPA"* line.
+If code and this file disagree, the code wins: fix this file the same turn (the anti-staleness law).
 
 Neighbors, so this page does not duplicate them: [`concepts.md`](./concepts.md) (eval mental model), [`self-improvement-map.md`](./self-improvement-map.md) (one loop / four roles / proposer catalog), [`campaign-proposers.md`](./campaign-proposers.md) (proposer ELI5), [`eval-surface-map.md`](./eval-surface-map.md) (which `run*` primitive), [`design/loop-taxonomy.md`](./design/loop-taxonomy.md) (execution vs proposer layering).
 
@@ -24,63 +24,63 @@ baseline surface
    repeat for maxGenerations
 ```
 
-## Glossary — one plain sentence each
+## Glossary: one plain sentence each
 
 Grounded to `agent-eval/src/campaign/types.ts` unless noted.
 
 | Term | Plain sentence | If you know DSPy/GEPA |
 |---|---|---|
-| **surface** | The one thing being changed this run — a prompt string, a JSON config string, or a finalized code candidate (`MutableSurface = string \| CodeSurface`, `types.ts`). | The optimized artifact: a signature/predictor's instruction text, or the module config. |
-| **proposer** | The strategy that, given the current surface + what failed, proposes the next batch of candidate surfaces to measure — it does **not** run the agent or score anything (`SurfaceProposer.propose`, `types.ts:286`). | The optimizer / teleprompter (MIPRO, BootstrapFewShot, GEPA's reflective proposer). |
+| **surface** | The one thing being changed this run: a prompt string, a JSON config string, or a finalized code candidate (`MutableSurface = string \| CodeSurface`, `types.ts`). | The optimized artifact: a signature/predictor's instruction text, or the module config. |
+| **proposer** | The strategy that, given the current surface + what failed, proposes the next batch of candidate surfaces to measure: it does **not** run the agent or score anything (`SurfaceProposer.propose`, `types.ts:286`). | The optimizer / teleprompter (MIPRO, BootstrapFewShot, GEPA's reflective proposer). |
 | **candidate** | One proposed surface plus its human `label` and `rationale`, ready to be measured (`ProposedCandidate`, `types.ts:166`). | One trial instruction/program the optimizer wants to evaluate. |
 | **generation** | One round of *propose → measure → rank → promote*; the loop runs up to `maxGenerations` of them (`GenerationRecord`, `types.ts:549`). | One GEPA iteration / optimization step. |
-| **populationSize** | BREADTH — how many candidate surfaces the proposer returns *this* generation (`ProposeContext.populationSize`, `types.ts:237`). Paired with `maxGenerations` (DEPTH) as the search budget. | Beam width / number of minibatch candidates per step. |
-| **population budget** | Informal name for the pair `{ populationSize, maxGenerations }` — the total candidates the search may evaluate (breadth × depth). | Optimizer trial budget. |
+| **populationSize** | BREADTH: how many candidate surfaces the proposer returns *this* generation (`ProposeContext.populationSize`, `types.ts:237`). Paired with `maxGenerations` (DEPTH) as the search budget. | Beam width / number of minibatch candidates per step. |
+| **population budget** | Informal name for the pair `{ populationSize, maxGenerations }`: the total candidates the search may evaluate (breadth × depth). | Optimizer trial budget. |
 | **campaign** | One measurement: run a dispatch over scenarios × seeds × reps, judge each output, aggregate → `CampaignResult` (`runCampaign`; a "campaign" = a coordinated batch of measurements). | One evaluation pass of a candidate over a valset. |
-| **cell** | The atomic measurement unit — exactly one `(scenario, rep)` execution producing one artifact + its judge scores + its cost (`CampaignCellResult`, `types.ts:509`; `DispatchContext.cellId/rep`). | One (example, seed) evaluation datapoint. |
+| **cell** | The atomic measurement unit: exactly one `(scenario, rep)` execution producing one artifact + its judge scores + its cost (`CampaignCellResult`, `types.ts:509`; `DispatchContext.cellId/rep`). | One (example, seed) evaluation datapoint. |
 | **scenario** | One input case with a stable `id` + `kind` (consumers attach their payload: persona, task, requirement) (`Scenario`, `types.ts:23`). | One dataset example / `dspy.Example`. |
-| **rep** | The repetition index — the same scenario run more than once so noise/variance is measurable rather than mistaken for signal (`ctx.rep`). | Repeated sampling of the same example (temperature/seed variance). |
+| **rep** | The repetition index: the same scenario run more than once so noise/variance is measurable rather than mistaken for signal (`ctx.rep`). | Repeated sampling of the same example (temperature/seed variance). |
 | **judge** | A scorer: given an artifact, return dimensions + a single `composite` + free-form `notes`; it throws on failure rather than silently scoring 0 (`JudgeConfig` → `JudgeScore`, `types.ts:94/116`). | The metric function, but pluggable (LLM-judge, deterministic checks, or an ensemble). |
-| **composite** | The single 0..1 number that combines all judge dimensions — the number you rank and gate on (`JudgeScore.composite`, `types.ts:116`). | The scalar metric value. |
-| **findings** | The failure analysis handed to the proposer so it edits from evidence, not guesses — worst cells + judge reasons, or a trace-analyst's clusters (`ProposeContext.findings`). | GEPA's reflective feedback / the "textual gradient". |
+| **composite** | The single 0..1 number that combines all judge dimensions: the number you rank and gate on (`JudgeScore.composite`, `types.ts:116`). | The scalar metric value. |
+| **findings** | The failure analysis handed to the proposer so it edits from evidence, not guesses: worst cells + judge reasons, or a trace-analyst's clusters (`ProposeContext.findings`). | GEPA's reflective feedback / the "textual gradient". |
 | **baseline** | The starting surface (the current prompt/config) every candidate is measured against (`baselineSurface`). | The unoptimized program you compare lift against. |
-| **holdout** | A separate scenario split the winner is re-scored on and that the proposer is *never* allowed to see — a compile-time firewall (`ProposeContext.judgeScores: never`, `types.ts:265`) keeps held-out verdicts out of proposal so the optimizer can't game the acceptance axis. | The held-out valset/testset — but here it is *write-only* to the optimizer. |
-| **lift** | Winner minus baseline `composite` on the holdout — the actual improvement, reported with a bootstrap confidence interval (`ImproveResult.lift`, `ProposerScore.lift`). | Δ metric between optimized and baseline program. |
-| **MDE** | Minimum Detectable Effect — the smallest lift your budget could statistically distinguish from noise, computed up front from baseline cells; a structurally-hopeless budget warns before you spend (surfaced as `result.power` / "power preflight", `agent-optimization-map.md`). Not spelled "MDE" in code yet — this is the standard name for it. | Power analysis on the eval set; the reason a tiny valset can't certify a small gain. |
-| **Pareto frontier** | The set of surfaces that are non-dominated across the per-scenario score vectors — a candidate worse on the mean but uniquely best on one hard scenario survives, so its lesson is not discarded (`ParetoParent`, `types.ts:198`; GEPA, arXiv:2507.19457). | Exactly GEPA's Pareto candidate pool — same paper, same idea. |
+| **holdout** | A separate scenario split the winner is re-scored on and that the proposer is *never* allowed to see: a compile-time firewall (`ProposeContext.judgeScores: never`, `types.ts:265`) keeps held-out verdicts out of proposal so the optimizer can't game the acceptance axis. | The held-out valset/testset: but here it is *write-only* to the optimizer. |
+| **lift** | Winner minus baseline `composite` on the holdout: the actual improvement, reported with a bootstrap confidence interval (`ImproveResult.lift`, `ProposerScore.lift`). | Δ metric between optimized and baseline program. |
+| **MDE** | Minimum Detectable Effect: the smallest lift your budget could statistically distinguish from noise, computed up front from baseline cells; a structurally-hopeless budget warns before you spend (surfaced as `result.power` / "power preflight", `agent-optimization-map.md`). Not spelled "MDE" in code yet: this is the standard name for it. | Power analysis on the eval set; the reason a tiny valset can't certify a small gain. |
+| **Pareto frontier** | The set of surfaces that are non-dominated across the per-scenario score vectors: a candidate worse on the mean but uniquely best on one hard scenario survives, so its lesson is not discarded (`ParetoParent`, `types.ts:198`; GEPA, arXiv:2507.19457). | Exactly GEPA's Pareto candidate pool: same paper, same idea. |
 | **gate** | The promotion decision: does the winner beat baseline on the holdout with significance, returning one of five verdicts `ship / hold / need_more_work / model_ceiling / arch_ceiling` (`Gate`, `GateDecision`, `types.ts:315/340`). | The accept/reject rule on the held-out valset, plus a significance test and ceiling diagnosis. |
-| **promotion** | What happens on a `ship` verdict — the winning surface is written back into the profile field it came from (`GenerationRecord.promoted`; `applyWinnerToProfile`, `improve.ts`). | Committing the optimized program as the new default. |
+| **promotion** | What happens on a `ship` verdict: the winning surface is written back into the profile field it came from (`GenerationRecord.promoted`; `applyWinnerToProfile`, `improve.ts`). | Committing the optimized program as the new default. |
 
-## Which proposer — the chooser
+## Which proposer: the chooser
 
 Every optimizer is a factory `xProposer(opts): SurfaceProposer`, all exported from `@tangle-network/agent-eval/campaign`.
 Start at the top row and only move down when the row's *"reach for it when"* matches your failure mode.
 
 | Proposer factory | Reach for it when | Surface it edits | Wired to the paved path? |
 |---|---|---|---|
-| `gepaProposer` | You want the strong default: reflective full-surface prompt rewrites, grounded in findings, keeping a Pareto frontier of complementary winners. | prompt string | **Yes — `improve({ surface: 'prompt' })` default.** Proven live. |
-| `skillOptProposer` | You are editing a structured `SKILL.md`/runbook and want small anchored add/delete/replace patches that preserve earlier rules. | skill/prompt string | **Yes — `improve({ surface: 'skills' })` default.** Not yet proven live. |
-| `parameterSweepProposer` | The likely fix is a config knob, not words — `retrieval.k`, `temperature`, `max_tokens`. You give it candidate patches; it applies them to a JSON surface. | JSON config string | Yes, but you supply the candidate list. |
-| `fapoProposer` | You want *evidence to decide when to escalate*: try prompt edits first, move to parameters, then to structural code — one scoped change per cycle, only escalating when the cheaper level is exhausted. | whatever its level proposers return | Exported; you wire the level proposers. |
+| `gepaProposer` | You want the strong default: reflective full-surface prompt rewrites, grounded in findings, keeping a Pareto frontier of complementary winners. | prompt string | **Yes: `improve({ surface: 'prompt' })` default.** Proven live. |
+| `skillOptProposer` | You are editing a structured `SKILL.md`/runbook and want small anchored add/delete/replace patches that preserve earlier rules. | skill/prompt string | **Yes: `improve({ surface: 'skills' })` default.** Not yet proven live. |
+| `parameterSweepProposer` | The likely fix is a config knob, not words: `retrieval.k`, `temperature`, `max_tokens`. You give it candidate patches; it applies them to a JSON surface. | JSON config string | Yes, but you supply the candidate list. |
+| `fapoProposer` | You want *evidence to decide when to escalate*: try prompt edits first, move to parameters, then to structural code: one scoped change per cycle, only escalating when the cheaper level is exhausted. | whatever its level proposers return | Exported; you wire the level proposers. |
 | `compositeProposer` | You want several proposers to share one candidate-generation budget in the same round. It allocates the population by declared weights, preserves member provenance, deduplicates surfaces, and isolates a member failure unless every member fails. | whatever its member proposers return | Exported; you wire the member proposers. |
 | `aceProposer` | You are accumulating hard-won lessons into a playbook and must **never** summarize an old lesson away (append-only, provenance-tagged). | playbook string | Exported. |
 | `memoryCurationProposer` | Same as ACE but you want a compact, deduped, re-ranked memory instead of append-only growth. | memory string | Exported. |
-| `evolutionaryProposer` | You want blind population search (mutate → measure → select) with no reflection over findings — a cheap control or a baseline to beat. | any string | Exported. |
+| `evolutionaryProposer` | You want blind population search (mutate → measure → select) with no reflection over findings: a cheap control or a baseline to beat. | any string | Exported. |
 | `traceAnalystProposer` | Bench-only: race our trace-analysis evidence engine head-to-head inside `compareProposers`. | prompt string | Bench-only. |
 | `haloProposer` | Bench-only: race the external `halo-engine` analysis against ours. | prompt string | Bench-only, external. |
 
 Default path: `gepaProposer` for prompts; add `parameterSweepProposer` when a config knob is the suspect; wrap levels in `fapoProposer` when the loop should decide *when* to escalate, or use `compositeProposer` when multiple proposer families must split one fixed population budget.
 
-## Composing proposers — four distinct shapes
+## Composing proposers: four distinct shapes
 
 Choose the shape that matches the experiment:
 
-1. **Portfolio** — `compositeProposer` splits one generation's population across member proposers by fixed weights and returns one provenance-labelled pool.
-2. **Escalate** — `fapoProposer` wraps prompt + parameter + structural levels into one proposer and spends on the cheapest level until evidence says to escalate.
-3. **Race** — `compareProposers` gives proposers separate loops, then re-scores their winners on one holdout and returns per-proposer lift intervals plus pairwise results.
-4. **Plug in** — hand any proposer to `runImprovementLoop({ proposer })`, or use `improve({ surface, generator })` in `@tangle-network/agent-runtime`.
+1. **Portfolio**: `compositeProposer` splits one generation's population across member proposers by fixed weights and returns one provenance-labelled pool.
+2. **Escalate**: `fapoProposer` wraps prompt + parameter + structural levels into one proposer and spends on the cheapest level until evidence says to escalate.
+3. **Race**: `compareProposers` gives proposers separate loops, then re-scores their winners on one holdout and returns per-proposer lift intervals plus pairwise results.
+4. **Plug in**: hand any proposer to `runImprovementLoop({ proposer })`, or use `improve({ surface, generator })` in `@tangle-network/agent-runtime`.
 
-### 2 + 4 — compose by escalation, then run the improvement loop
+### 2 + 4: compose by escalation, then run the improvement loop
 
 ```ts
 import {
@@ -110,8 +110,8 @@ const proposer = fapoProposer({
   }),
 })
 
-// dispatchWithSurface scores ONE surface on ONE scenario — this is the topology-
-// opaque seam (one LLM call, one worker, or a whole fleet — the loop can't tell).
+// dispatchWithSurface scores ONE surface on ONE scenario: this is the topology-
+// opaque seam (one LLM call, one worker, or a whole fleet: the loop can't tell).
 const dispatchWithSurface = async (surface, scenario, ctx) =>
   runYourAgent({ surface, scenario, signal: ctx.signal })   // returns the artifact to judge
 
@@ -148,7 +148,7 @@ const out = await improve(profile, findings, {
 if (out.shipped) deploy(out.profile)  // out.lift is the held-out winner − baseline
 ```
 
-### 3 — race proposers head-to-head for a lift CI
+### 3: race proposers head-to-head for a lift CI
 
 ```ts
 import {
@@ -186,19 +186,19 @@ const comparison = await compareProposers({
 })
 
 comparison.best          // highest-lift proposer
-comparison.scores        // per-proposer { lift, liftCi:{low,high}, cost } — low>0 ⇒ real gain
+comparison.scores        // per-proposer { lift, liftCi:{low,high}, cost }: low>0 ⇒ real gain
 comparison.pairwise      // best vs each other, paired-bootstrap: 'a' | 'b' | 'tie'
 ```
 
-Every entrant is re-scored on the **same** holdout with the **same** judges, so the comparison never trusts how a proposer measured itself — the only variable is proposal quality.
+Every entrant is re-scored on the **same** holdout with the **same** judges, so the comparison never trusts how a proposer measured itself: the only variable is proposal quality.
 
 ## Common traps (they cost the most)
 
-- Do not put eval logic inside a proposer — scoring lives in `dispatch` + `judges`, proposing lives in the proposer.
-- Do not let a proposer read held-out judge scores — `ProposeContext` makes that a compile error on purpose; a proposer that games the acceptance axis is an oracle, not an optimizer.
-- Do not read `lift` without `result.power`/MDE — a "+4" on a valset too small to detect +4 is noise wearing a number.
+- Do not put eval logic inside a proposer: scoring lives in `dispatch` + `judges`, proposing lives in the proposer.
+- Do not let a proposer read held-out judge scores: `ProposeContext` makes that a compile error on purpose; a proposer that games the acceptance axis is an oracle, not an optimizer.
+- Do not read `lift` without `result.power`/MDE: a "+4" on a valset too small to detect +4 is noise wearing a number.
 - Do not confuse `compositeProposer` with `fapoProposer`: the former allocates one fixed population across peers, while the latter escalates through ordered levels from cheaper to more structural changes.
 
-### neutralizationGate — the placebo / content-causality control
+### neutralizationGate: the placebo / content-causality control
 
 Standard gates prove a candidate *beat baseline*. `neutralizationGate` proves the candidate's **content** caused the lift, not the extra prompt bytes: it blanks the candidate's added content to byte-length-matched filler, holds everything else fixed, and requires the lift to vanish. A fully-neutralized candidate that still scores is decorative and is rejected. Exports: `neutralizationGate` (`src/campaign/gates/neutralization-gate.ts`), `neutralizeText` (`src/campaign/neutralize.ts`). Since 0.107.0. Pair it with `heldOutGate` in any loop that promotes authored artifacts (prompts, tool docs, knowledge) so a lift that is really just added prompt size cannot be shipped.