@tangle-network/agent-eval 0.123.1 → 0.123.3

package/docs/building-doctrine.md

@@ -4,7 +4,7 @@ How every fleet agent that consumes `agent-eval` is built. Each rule is mechanic
 
 ## 1. Defaults must be provably reachable
 
-Every hard-coded model id or endpoint default is verifiable against the live router. Membership in `{baseUrl}/models` is the free check; an optional 1-token probe per model confirms the router will actually serve it. A default the router cannot serve is a config bug caught before the run, not a runtime surprise that silently degrades into a stub. Backend ids are namespaced by binding: cli-bridge ids (`claude-code/*`, `kimi-code/*`, `opencode/*`) never appear as defaults in code reachable from production — bridge use is an explicit env opt-in, never an implicit fallback.
+Every hard-coded model id or endpoint default is verifiable against the live router. Membership in `{baseUrl}/models` is the free check; an optional 1-token probe per model confirms the router will actually serve it. A default the router cannot serve is a config bug caught before the run, not a runtime surprise that silently degrades into a stub. Backend ids are namespaced by binding: cli-bridge ids (`claude-code/*`, `kimi-code/*`, `opencode/*`) never appear as defaults in code reachable from production: bridge use is an explicit env opt-in, never an implicit fallback.
 
 Enforced by: `preflightModels` (membership + optional probe) and `assertModelsServed` (gate that names every unreachable id with status + detail).
 
@@ -12,13 +12,13 @@ Enforced by: `preflightModels` (membership + optional probe) and `assertModelsSe
 
 When a request fails, one direct call against the live endpoint bisects platform-versus-client before any code-level debugging begins. A 401 from the router on a `model_not_found` is the platform telling you the default is dead; a connection refused is the platform being unreachable. Establish which side is at fault with a probe first, then debug only the side that is actually broken.
 
-Enforced by: `preflightModels({ probe: true })` — the probe is the platform-side bisection, carrying the router's own `error.message` back to the caller.
+Enforced by: `preflightModels({ probe: true })`: the probe is the platform-side bisection, carrying the router's own `error.message` back to the caller.
 
 ## 3. Agent-produced findings are hypotheses
 
 Enumeration of candidate problems may fan out to agents, but agent output is not evidence. Truth comes from probes against ground truth, not from an agent's assertion. Every classification carries quoted evidence, and nothing unverified is merged or reported as fact. A confident-sounding agent claim with no probe behind it is a hypothesis awaiting falsification.
 
-Enforced by: `assertRealBackend` over the resulting `RunRecord[]` — an agent that claims success while the backend was never called reads as a stub, not a pass.
+Enforced by: `assertRealBackend` over the resulting `RunRecord[]`: an agent that claims success while the backend was never called reads as a stub, not a pass.
 
 ## 4. Experiment integrity checklist
 
@@ -30,7 +30,7 @@ Any lift or benchmark claim satisfies all of the following before it is reported
 - The same scorer and the same items on both sides of any comparison.
 - A leakage check from builder inputs into the evaluation set.
 - Cross-family judge panels, with inter-rater reliability reported and gated.
-- Missing evidence is never scored as zero — fail loud over fabricate.
+- Missing evidence is never scored as zero: fail loud over fabricate.
 - No optional stopping: the stopping rule is fixed before the run.
 
 Enforced by: `pairedBootstrap` (CI), `assertCrossFamily` (panel diversity), `interRaterReliability` (agreement), and `assertRealBackend` (no stub run masquerading as a result).
@@ -39,4 +39,4 @@ Enforced by: `pairedBootstrap` (CI), `assertCrossFamily` (panel diversity), `int
 
 A drifted default is the symptom of a missing convention. The fix ships the convention and its guard alongside the one-line correction, so the same drift cannot recur silently. Patching the single dead id without adding the preflight gate leaves the class open; the next default rots the same way.
 
-Enforced by: `assertModelsServed` wired into the campaign preflight — the guard that turns "this one model was dead" into "no campaign spends tokens against an unreachable default."
+Enforced by: `assertModelsServed` wired into the campaign preflight: the guard that turns "this one model was dead" into "no campaign spends tokens against an unreachable default."

package/docs/campaign-proposers.md

@@ -1,170 +1,181 @@
-# Campaign Proposers, ELI5
+# Candidate Generation and Method Comparison
 
-A campaign proposer is the part of an improvement loop that says: "try this
-candidate next."
+`SurfaceProposer` generates candidate prompts or configs.
+It does not run an agent, score output, or choose a winner.
 
-It does not run your agent. It does not score anything. It only proposes a new
-surface to measure. A surface is the thing you are changing: a prompt string, a
-serialized config string, or a code/worktree surface.
+`OptimizationMethod` runs a complete search procedure on train and selection data.
+It returns one selected surface plus its optimization cost.
 
-Use **proposer** for this role. Older optimizer APIs used "driver"; that word is
-now reserved for execution, sandbox, and router agents that actually drive
-workers.
+`compareOptimizationMethods` runs multiple methods, waits for every method to finish, then scores their selected surfaces on the same final test data.
 
-## The Loop
-
-```text
-current surface
-  -> proposer suggests candidate surfaces
-  -> runCampaign runs each candidate on scenarios
-  -> judges score the artifacts
-  -> runOptimization picks the best candidate
-  -> runImprovementLoop re-scores on holdout and gates release
-```
-
-## Proposer Input
-
-Every `SurfaceProposer.propose(ctx)` receives:
-
-| Field | Plain meaning |
+| API | Responsibility |
 |---|---|
-| `currentSurface` | The prompt/config/code surface currently being improved. |
-| `history` | What candidates were tried before and how they scored. |
-| `findings` | Failure analysis or analyst findings from traces/eval runs. |
-| `populationSize` | How many candidates the loop asks for this generation. |
-| `generation` | Which generation number this is. |
-| `signal` | Abort signal for cancellation. |
-| `report` | Optional larger analysis report. |
-| `dataset` | Optional labeled scenario store. |
-| `paretoParents` | Optional non-dominated surfaces from prior generations. |
+| `SurfaceProposer` | Suggest the next candidate surface. |
+| `runOptimization` | Run and score candidates on training scenarios. |
+| `runImprovementLoop` | Optimize one surface and apply a release rule on separate scenarios. |
+| `OptimizationMethod` | Adapt one complete optimization procedure for comparison. |
+| `compareOptimizationMethods` | Compare selected surfaces on shared final test data. |
 
-## Proposer Output
+## Candidate Generators
 
-A proposer returns candidates:
+Every `SurfaceProposer.propose(ctx)` receives the current surface, prior candidate scores, findings, requested candidate count, generation number, and cancellation signal.
+It may also receive a larger analysis report, a labeled scenario store, or Pareto parents when the caller provides them.
+
+A proposer may return a bare surface or a labeled candidate:
 
 ```ts
 {
-  surface: 'the full new prompt or config',
-  label: 'short name',
-  rationale: 'why this change should help'
+  surface: 'the complete new prompt or config',
+  label: 'require-citations',
+  rationale: 'three training failures omitted source references',
 }
 ```
 
-Bare surfaces are still accepted, but `label` and `rationale` make results
-auditable, so new proposers should return `ProposedCandidate`.
-
-## Which Proposer To Use
+Use a labeled `ProposedCandidate` when you need the result to retain why the candidate was generated.
 
-| Proposer factory | Best when | Output surface |
+| Factory | Use it for | Surface |
 |---|---|---|
-| `gepaProposer` | You want a strong prompt rewrite driven by prior scores and findings. | prompt string |
-| `skillOptProposer` | You are editing a structured skill/runbook and want anchored small patches. | prompt/skill string |
-| `aceProposer` | You want append-only lessons from findings, preserving every distinct lesson. | prompt/playbook string |
-| `memoryCurationProposer` | You want compact deduped lessons from findings. | prompt/playbook string |
-| `parameterSweepProposer` | You want FAPO-style config/parameter edits from a JSON config surface. | JSON string |
-| `fapoProposer` | You want the FAPO policy: prompt first, then parameter, then structural only when evidence supports escalation. | whatever its level proposer returns |
-
-## FAPO Proposer
+| `gepaProposer` | Rewrite a prompt from prior scores and findings. | string |
+| `skillOptProposer` | Apply bounded edits to a structured skill or runbook. | string |
+| `aceProposer` | Append distinct lessons from findings. | string |
+| `memoryCurationProposer` | Deduplicate and compact lessons from findings. | string |
+| `parameterSweepProposer` | Apply declared changes to a JSON config. | JSON string |
+| `fapoProposer` | Try prompt, parameter, and optional structural changes under one escalation policy. | caller-defined |
 
-FAPO is not "another prompt mutator." The paper describes a reviewed escalation
-policy:
+## Compare Complete Methods
 
-1. evaluate the current workflow,
-2. attribute failures to prompt, parameter/config, or structure,
-3. propose one scoped change,
-4. review the change for scope/leakage/compatibility,
-5. measure it,
-6. keep moving or escalate only when the cheaper level is exhausted.
-
-The simplest useful setup is prompt plus JSON config. Structural/code edits are
-optional and should be injected by the app or runtime layer.
+The following call owns the shared baseline, runner, judges, directories, and three data sets.
+Method configuration contains only settings that differ by method.
 
 ```ts
 import {
-  fapoProposer,
-  gepaProposer,
-  parameterSweepProposer,
-  runImprovementLoop,
+  type BuiltinOptimizationMethodConfig,
+  compareOptimizationMethods,
+  gepaParetoMethod,
+  gepaReflectionMethod,
+  skillOptMethod,
 } from '@tangle-network/agent-eval/campaign'
 
-const proposer = fapoProposer({
-  scope: { allowedLevels: ['prompt', 'parameter'] },
-  promptProposer: gepaProposer({ llm, model, target: 'agent prompt' }),
-  parameterProposer: parameterSweepProposer({
-    candidates: [
-      {
-        label: 'raise-retrieval-k',
-        rationale: 'retrieval misses indicate the search budget may be too low',
-        changes: [{ path: 'retrieval.k', value: 10 }],
-      },
-    ],
-  }),
-})
+const methodConfig: BuiltinOptimizationMethodConfig<MyScenario, MyArtifact> = {
+  llm,
+  model,
+  target: 'the complete prompt or config being improved',
+  populationSize: 2,
+  maxGenerations: 3,
+  maxEpochs: 6,
+}
 
-await runImprovementLoop({
-  scenarios: trainScenarios,
-  holdoutScenarios,
-  baselineSurface: JSON.stringify(currentConfig),
+const comparison = await compareOptimizationMethods<MyScenario, MyArtifact>({
+  methods: [
+    gepaReflectionMethod(methodConfig),
+    gepaParetoMethod(methodConfig),
+    skillOptMethod(methodConfig),
+  ],
+  baselineSurface,
+  trainScenarios,
+  selectionScenarios,
+  testScenarios,
   dispatchWithSurface,
   judges,
-  proposer,
-  gate,
-  autoOnPromote: 'none',
   runDir,
-  populationSize: 1,
-  maxGenerations: 10,
+  optimizationRunOptions: {
+    costCeiling: 5,
+    dispatchTimeoutMs: 60_000,
+    maxConcurrency: 4,
+  },
+  optimizationConcurrency: 2,
+  costCeiling: 2,
+  maxConcurrency: 4,
+  confidence: 0.95,
 })
 ```
 
-If you do have a real code/worktree proposer, pass it as `structuralProposer`.
-`agent-eval` intentionally does not provide that proposer because this package
-measures candidates; the runtime or app owns code generation.
+The runnable version is [`examples/compare-optimization-methods`](../examples/compare-optimization-methods/).
+
+## Data Use
+
+| Set | Who can read it | Purpose |
+|---|---|---|
+| Train | Optimization methods and candidate generators | Generate and fit candidates. |
+| Selection | Optimization methods | Accept candidates, stop early, and select one surface per method. |
+| Test | `compareOptimizationMethods` only | Estimate final lift and rank methods. |
+
+All three sets must be non-empty and pairwise disjoint by scenario ID.
+Test must contain at least two scenarios.
+Two is only an API minimum; use enough scenarios to detect the effect size that matters for your product.
+
+Each method receives independent copies of train and selection scenarios.
+The final test set is absent from `OptimizationMethodInput`.
+Every method finishes before the first test call starts.
+When `optimizationConcurrency` is greater than one, the shared runner and judges must support concurrent calls.
+
+## Execution And Cost
+
+`optimizationConcurrency` controls how many methods run at once.
+`optimizationRunOptions.maxConcurrency` controls scenario calls inside each method.
+Top-level `maxConcurrency` controls scenario calls during final test scoring.
 
-For side-by-side experiments with existing proposers, use the compare entry:
+`optimizationRunOptions.costCeiling` is a separate limit for each method.
+Top-level `costCeiling` is one shared limit across baseline and selected-surface scoring on final test.
+
+The result reports three cost objects:
 
 ```ts
-import {
-  compareProposers,
-  fapoEscalationEntry,
-  gepaParetoEntry,
-} from '@tangle-network/agent-eval/campaign'
+comparison.optimizationCost
+comparison.testCost
+comparison.totalCost
+```
 
-await compareProposers({
-  proposers: [
-    gepaParetoEntry(config),
-    fapoEscalationEntry({
-      ...config,
-      parameterCandidates: [
-        {
-          label: 'raise-retrieval-k',
-          rationale: 'retrieval misses indicate the search budget may be too low',
-          changes: [{ path: 'retrieval.k', value: 10 }],
-        },
-      ],
-    }),
-  ],
-  baselineSurface,
-  holdoutScenarios,
-  dispatchWithSurface,
-  judges,
-  runDir,
-})
+Each object contains `totalCostUsd`, `accountingComplete`, and `incompleteReasons`.
+An unknown provider charge therefore cannot appear as a trustworthy zero-dollar total.
+Cost breaks a lift tie only when every method in that tied group reports complete accounting.
+
+## Read The Result
+
+```ts
+for (const method of comparison.scores) {
+  console.log({
+    rank: method.rank,
+    name: method.name,
+    lift: method.lift,
+    interval: method.liftCi,
+    scenarios: method.scenarioScores,
+    optimizationCostUsd: method.optimizationCost.totalCostUsd,
+    costComplete: method.optimizationCost.accountingComplete,
+  })
+}
 ```
 
-## Common Mistakes
+`rank` orders methods by estimated lift, then by cost only when every method with that lift has complete cost accounting.
+It does not mean the higher-ranked method is conclusively better.
+Read `liftCi` and `comparison.pairwise[].favored` before making that claim.
+`scenarioScores` contains the paired values used to compute each method's result.
+
+Repetitions are averaged within each test scenario before scenarios are resampled.
+The intervals assume scenarios are the independent sampling units.
+
+`confidence: 0.95` applies to the complete family of method-vs-baseline and possible method-vs-method contrasts.
+The implementation adjusts each interval for that family and raises the default resample count when more methods require finer interval tails.
+An explicit resample count that is too small is rejected before optimization starts.
+
+The final test data is spent when this function ranks methods.
+If you choose a method from this result and later claim its deployed effect, confirm that claim on new data that was not used for this ranking.
+
+## FAPO
+
+`fapoProposer` can move from prompt edits to declared parameter edits and then to an injected structural proposer.
+Every level must accept and return the same surface representation.
 
-- Do not put eval logic inside a proposer. Put it in `dispatch` and `judges`.
-- Do not let a proposer read held-out judge scores. `ProposeContext` makes this
-  a type-level firewall.
-- Do not call FAPO a prompt-only optimizer. Its main value is evidence-based
-  escalation beyond prompt edits.
-- Do not put Claude Code or sandbox-specific code in `agent-eval`. Structural
-  code generation should be supplied as an injected `SurfaceProposer` from the
-  runtime/app layer.
+`agent-eval` does not generate repository code itself.
+Pass a code-capable `structuralProposer` from your runtime or application when structural edits are part of the comparison.
 
-## Simpler Mental Model
+Use `fapoEscalationMethod(config)` to compare the complete FAPO procedure with other methods.
 
-Use this sentence when wiring a loop:
+## Common Errors
 
-> The proposer chooses candidates; the campaign measures them; the gate decides
-> whether the measured winner is safe to promote.
+- Do not pass a raw `SurfaceProposer` to `compareOptimizationMethods`.
+- Do not let a custom `OptimizationMethod` load final test rows from another source.
+- Do not compare methods with different runners, judges, or final test scenarios.
+- Do not read `method.optimizationCost` as total comparison cost.
+- Do not report a dollar total as complete when `accountingComplete` is false.
+- Do not reuse the final test set for repeated method selection and continue calling it untouched.

package/docs/concepts.md

@@ -1,50 +1,47 @@
 # Concepts
 
-`agent-eval` is for deciding whether an agent run should pass, keep working, be
-replayed, be optimized, or be promoted.
+`agent-eval` records agent runs, scores their outputs, compares variants, and applies caller-defined release rules.
 
-It exists because agent output is not evidence. A model can say a task is done
-while the build fails, the browser flow is broken, the integration was never
-connected, or the answer lacks required sources. The package gives products a
-shared way to record runs, check outcomes, classify failures, compare variants,
-and make release decisions.
+A model can say a task is complete while the build fails, a browser flow is broken, an integration is disconnected, or required sources are missing.
+This package lets code, model judges, and human feedback check those outcomes through the same run format.
 
 ## The top-level functions
 
-Everything funnels through `/contract`. Start with `defineAgentEval()` when you
-can; drop to the raw functions when you need lower-level control.
+Start with `/contract` and `defineAgentEval()` for a new integration.
+Use the lower-level functions when you need direct control over execution, storage, or statistics.
 
 | Function | When to call it | What you give it | What you get back |
 |---|---|---|---|
-| **`defineAgentEval()`** | You have scenarios, an agent, a judge, and a baseline surface, and you want one object you can score or improve. | scenarios, agent, judge, baseline surface | `{ evaluate(), improve() }` where `evaluate()` returns a campaign result and `improve()` returns a decision packet |
-| **`selfImprove()`** | You have a closed loop — scenarios, judge, agent in hand, and you want the substrate to propose better candidates + gate them. | scenarios, agent, judge, baseline surface | `SelfImproveResult.insight: InsightReport` + ship/hold verdict + winner surface |
+| **`defineAgentEval()`** | You have scenarios, an agent, a judge, and a baseline surface, and you want one object you can score or improve. | scenarios, agent, judge, baseline surface | `{ evaluate(), improve() }` where `evaluate()` returns a campaign result and `improve()` returns a report |
+| **`selfImprove()`** | You want candidate generation, scoring, and a release decision in one call. | scenarios, agent, judge, baseline surface | report, ship/hold decision, winner surface |
 | **`loadEvalFixtureScenarios()`** | You want agents to add evals as folders with `PROMPT.md`, checks, and starter files. | `evals/<name>/PROMPT.md + EVAL.ts + package.json` | `Scenario[]` that runs through `runCampaign`; pair with `planEvalFixtureRun()` before spending tokens |
-| **`analyzeRuns()`** | You have observed runs (production traces, an approve/reject corpus, a CSV gold set) and want the same rigor packet without invoking an agent. | `RunRecord[]` + optional flags | `InsightReport` |
-| **Intake adapters** (`fromFeedbackTable`, `fromOtelSpans`) | Your data isn't already in `RunRecord` shape — it's in Obsidian, Sheets, an OTel collector, etc. | source-specific input | `RunRecord[]` ready to pipe into `analyzeRuns()` |
+| **`analyzeRuns()`** | You have existing runs and do not need to invoke an agent. | `RunRecord[]` and options | `InsightReport` |
+| **Intake adapters** (`fromFeedbackTable`, `fromOtelSpans`) | Your data isn't already in `RunRecord` shape: it's in Obsidian, Sheets, an OTel collector, etc. | source-specific input | `RunRecord[]` ready to pipe into `analyzeRuns()` |
 
-The customer maturity stages — logs only → ratings → closed loop — map to these
-entry points. See [`customer-journeys.md`](./customer-journeys.md) for the
-runnable walkthroughs.
+See [`customer-journeys.md`](./customer-journeys.md) for runnable paths from existing logs, human ratings, and a callable agent.
 
-The shape of the answer — `InsightReport` — is identical across all three paths. Distributional summary, paired-bootstrap lift CI, judge stats, inter-rater agreement, cost-quality Pareto, failure clusters, contamination check, outcome correlation, release axes, and a ranked recommendations array. Walked through section-by-section in [`insight-report.md`](./insight-report.md).
+`analyzeRuns()` and the high-level contract return the same `InsightReport` shape.
+It contains score distributions, paired lift intervals, judge agreement, cost, failure clusters, contamination checks, outcome correlation, and recommendations.
+[`insight-report.md`](./insight-report.md) defines every field.
 
-## The layering rule
+## Package Boundary
 
-`agent-eval` is the **substrate** at the bottom of the Tangle agent stack. `agent-runtime` and `agent-knowledge` depend on it; `agent-eval` MUST NOT import from either. Primitives that "feel like" they belong in a consumer but are actually substrate-shaped (validator verdicts, run records, scenarios, judge scores) live here. Primitives that genuinely require a running agent loop (`ValidationCtx` with iteration + signal + traceEmitter, sandbox `AgentRunSpec`) stay in `agent-runtime`.
+`agent-runtime` and `agent-knowledge` may import `agent-eval`.
+`agent-eval` must not import either package.
 
-The test: *does this concept make sense WITHOUT a running agent loop?* If yes, it's substrate. If no, it's runtime. The full rule is in [`/CLAUDE.md`](../CLAUDE.md#repo-layering--this-package-is-the-substrate).
+Run records, scenarios, judge scores, statistics, and release decisions belong here because they work without an agent runtime.
+Agent sessions, worker coordination, sandbox execution, and runtime-specific profiles belong in `agent-runtime`.
 
 ## Main Objects
 
 | Thing | What it is | One-line example |
 |---|---|---|
 | **Judge** | A function that scores one piece of output. | "Did this scaffold implement async fetching?" |
-| **Rubric** | The recipe a judge uses — what to score on, with what weights. | "Score on buyer_quality (0.5), voice (0.3), signal (0.2)." |
+| **Rubric** | The recipe a judge uses: what to score on, with what weights. | "Score on buyer_quality (0.5), voice (0.3), signal (0.2)." |
 | **Verifier** | A pipeline of judges run in order, with dependencies. | "install → typecheck → build → semantic" |
 | **Feedback trajectory** | A multi-shot record of attempts, approvals, rejections, edits, metrics, and policy outcomes. | "draft → user rejects → revised draft → approved → measured" |
 
-Everything else exists to make those objects useful in real product loops:
-traces, datasets, control runtime, optimizers, statistics, and reports.
+Traces, datasets, optimization, statistics, and reports build on these objects.
 
 When the thing being evaluated is an agent that should keep working, use
 [`runAgentControlLoop`](./control-runtime.md). It turns validators into a
@@ -52,30 +49,28 @@ runtime loop: observe typed state, validate it, decide the next action, act,
 and repeat until the task passes, blocks, times out, spends too much, or stops
 making progress.
 
-When normal agent usage should become reusable training/eval signal, use
+When normal agent usage should become reusable training or eval data, use
 [`FeedbackTrajectory`](./feedback-trajectories.md). It captures approvals,
 rejections, edits, option choices, metrics, and policy blocks as portable data
 that can seed memory, replay scenarios, and optimization.
 
-## Vocabulary, plain English
+## Terms
 
 | Term | Plain English |
 |---|---|
 | **Artifact** | The thing being judged. Often a workdir of files, sometimes a string of text. |
-| **Snapshot** | A frozen view of an artifact (every file path → content). What the judge actually reads. |
+| **Snapshot** | A frozen view of an artifact (every file path → content). This is the input the judge reads. |
 | **Harness** | A description of *how to run* the artifact: setup command, test command, working dir, timeout. |
-| **Sandbox driver** | The thing that actually executes commands inside the harness. Local subprocess, or remote container. |
+| **Sandbox driver** | Executes commands inside the harness, using a local subprocess or remote container. |
 | **Layer** | One stage of a verifier pipeline (install, typecheck, build, semantic, …). |
-| **Finding** | A specific issue a judge found — file, line, severity, message. |
+| **Finding** | A specific issue a judge found: file, line, severity, message. |
 | **Trace store** | The append-only log of every span/event during a run. Replay = read this back. |
 | **Composite score** | A 0..1 number combining all dimensions. The single number you gate on. |
 | **Rubric version** | A stable hash of the rubric. Scores from different rubric versions are not comparable. |
-| **Muffled gate** | A check that should fail loud but silently passes (e.g. `command || true`). The most expensive bug class in this codebase. |
 
 ## The feedback trajectory loop
 
-For agentic systems, the highest-quality labels often come from normal review
-workflow, not a separate labeling UI:
+Normal review activity can provide labels without a separate labeling interface:
 
 ```text
 agent proposes -> user approves/rejects/edits/selects -> agent revises -> outcome is measured
@@ -103,30 +98,31 @@ L1  app-build      Does the artifact build / typecheck / test?
                               │
                               ▼
 L2  app-runtime    Does the artifact actually run end-to-end?
-                   (Dynamic signal — only worth checking if L1 passed.)
+                   (Dynamic signal: only worth checking if L1 passed.)
 ```
 
 `BuilderSession` orchestrates this. It opens at `startChat`, runs the build at `ship`, runs the runtime check at `runAppScenario`. Each layer emits a trace span. Composite score aggregates them with `scoreProject`.
 
 Why three? Because each catches a different failure mode:
-- L0 misses — agent crashed mid-generation, you have a half-written file.
-- L1 misses — files exist but typecheck fails. LLM judges can't reliably catch this.
-- L2 misses — code compiles but does the wrong thing at runtime.
+- L0 misses: agent crashed mid-generation, you have a half-written file.
+- L1 misses: files exist but typecheck fails. LLM judges can't reliably catch this.
+- L2 misses: code compiles but does the wrong thing at runtime.
 
 If you only check one layer, you ship the bugs that the other two layers would have caught.
 
 ## How rubrics work
 
 A rubric describes:
-1. **Dimensions** — the axes you score on (e.g. `buyer_quality`, `voice`, `signal`).
-2. **Weights** — how to combine dimensions into a composite (`0.5 * buyer_quality + 0.3 * voice + 0.2 * signal`).
-3. **Failure modes** — named patterns the judge looks for ("ai-cadence", "vague-claim").
-4. **Wins** — named positive patterns ("specific-component", "earned-detail").
-5. **System prompt** — what to tell the judging LLM about the persona and the task.
+1. **Dimensions**: the axes you score on (e.g. `buyer_quality`, `voice`, `signal`).
+2. **Weights**: how to combine dimensions into a composite (`0.5 * buyer_quality + 0.3 * voice + 0.2 * signal`).
+3. **Failure modes**: named patterns the judge looks for ("ai-cadence", "vague-claim").
+4. **Wins**: named positive patterns ("specific-component", "earned-detail").
+5. **System prompt**: what to tell the judging LLM about the persona and the task.
 
-Built-in rubrics ship in `src/wire/rubrics.ts` (e.g. `anti-slop` for technical-buyer voice). You can also pass a rubric inline — the same shape, just defined at the call site.
+Built-in rubrics ship in `src/wire/rubrics.ts`, including `anti-slop` for technical-buyer voice.
+You can also pass the same rubric shape inline at the call site.
 
-A rubric is plain data. The hash of that data is the `rubricVersion`. Two scores are only comparable if they used the same `rubricVersion` — change the rubric and you start a new comparison series.
+A rubric is plain data. The hash of that data is the `rubricVersion`. Two scores are only comparable if they used the same `rubricVersion`: change the rubric and you start a new comparison series.
 
 ## How verifiers work
 
@@ -141,14 +137,14 @@ const verifier = new MultiLayerVerifier([
 ])
 
 const report = await verifier.run({ env: { runner, workdir, ... } })
-report.allPass        // boolean — every layer passed
-report.blendedScore   // 0..1 — weighted aggregate
+report.allPass        // boolean: every layer passed
+report.blendedScore   // 0..1: weighted aggregate
 report.layers         // per-layer status, findings, duration
 ```
 
 Two rules that will save you bugs:
 
-1. **Run both gates.** Build gates catch code that doesn't compile; structural assertions catch missing files. Run both unconditionally — they catch orthogonal failures.
+1. **Run both gates.** Build gates catch code that doesn't compile; structural assertions catch missing files. Run both unconditionally: they catch orthogonal failures.
 
 2. **Pair LLM judges with build outcomes.** An LLM judge will rate non-compiling code as "looks right" (0.8). Always short-circuit on `buildOutcome.passed === false` before any LLM judging.
 
@@ -159,7 +155,7 @@ Two questions to answer before trusting any LLM judge:
 1. **Does it agree with humans?** `calibrateJudge(golden, candidate)` reports Pearson, MAE, integer-rounded κ, and worst-N miscalibrations vs a human golden set.
 2. **Does it agree with itself / other judges?** `continuousAgreement(scores)` and `calibrateJudgeContinuous(golden, candidate)` report κ_w + ICC(2,1) + Pearson + Spearman with bootstrap 95% CIs on the raw [0,1] scores.
 
-Why two κ flavours: the original `calibrateJudge` rounds scores to ints before computing κ. For fine-grained judges that loses information — 0.78 vs 0.81 both round to "1" and look perfectly agreed. Use `calibrateJudgeContinuous` (or `continuousAgreement` for N≥2 raters) when scores are continuous. ICC(2,1) catches systematic bias that Pearson misses: if judge B scores 2× judge A, Pearson stays ≈ 1 while ICC drops — that's the signal.
+Why two κ flavours: the original `calibrateJudge` rounds scores to ints before computing κ. For fine-grained judges that loses information: 0.78 vs 0.81 both round to "1" and look perfectly agreed. Use `calibrateJudgeContinuous` (or `continuousAgreement` for N≥2 raters) when scores are continuous. ICC(2,1) catches systematic bias that Pearson misses: if judge B scores 2× judge A, Pearson stays ≈ 1 while ICC drops: that's the signal.
 
 Bias probes (`positionalBias`, `verbosityBias`, `selfPreference`) cover the orthogonal failure modes: position-dependent scoring, length-correlated scoring, and judge-prefers-its-own-family.
 
@@ -178,7 +174,7 @@ builder-session                 [span]
     └── scenario.run            [span]
 ```
 
-Spans are append-only and have stable ids — replay is reading the same store back. OTLP export ships them out for distributed tracing.
+Spans are append-only and have stable ids: replay is reading the same store back. OTLP export ships them out for distributed tracing.
 
 You usually should not build this tree by hand. Product runtimes,
 `runAgentControlLoop`, harnesses, and verifiers should emit it while they run.
@@ -187,12 +183,12 @@ release decision.
 
 ## Where to go next
 
-- **Confused by "GEPA / HALO / trace analysis / proposers everywhere"?** → [self-improvement-map.md](./self-improvement-map.md) — one loop, four roles, the proposer catalog (production vs bench-only), and why `gepa-refine` is the same loop on a test bench.
-- **Which `run*` primitive do I use, and how do I grade produced state?** → [eval-surface-map.md](./eval-surface-map.md) — the campaign/matrix/optimization/gate primitives as a pick-by-"use-when" table, plus the produced-state grading composition (verifyCompletion-as-judge — there is no persona-dispatch wrapper) and the in-band body contract.
-- **Need the layman feature map?** → [feature-guide.md](./feature-guide.md) — what each primitive does, when to use it, integration patterns, and guardrails.
-- **Just want to score a string against a rubric?** → [wire-protocol.md](./wire-protocol.md) — HTTP/RPC interface, pluggable from any language.
-- **Need a reusable driver/worker/evaluator loop?** → [control-runtime.md](./control-runtime.md) — generic runtime plus coding, browser, computer-use, and research integration patterns.
-- **Want review feedback to become eval/optimization data?** → [feedback-trajectories.md](./feedback-trajectories.md) — turn feedback into datasets, optimizer rows, and preference memory.
+- **Choosing a candidate-generation method?** Read [campaign-proposers.md](./campaign-proposers.md) for the available methods, their inputs, and runnable composition examples.
+- **Choosing a `run*` function or grading produced state?** Read [eval-surface-map.md](./eval-surface-map.md) for a use-case table and complete grading composition.
+- **Need the feature map?** Read [feature-guide.md](./feature-guide.md) for integration patterns and operational limits.
+- **Scoring a string from another language?** Read [wire-protocol.md](./wire-protocol.md) for the HTTP/RPC interface.
+- **Building a driver and worker loop?** Read [control-runtime.md](./control-runtime.md) for coding, browser, computer-use, and research patterns.
+- **Turning review feedback into reusable data?** Read [feedback-trajectories.md](./feedback-trajectories.md) for dataset, optimization, and preference-memory examples.
 - **Building a code-generator eval?** → Start with `BuilderSession`, `SandboxHarness`, and `MultiLayerVerifier`.
 - **Multi-layer verifier?** → Use [control-runtime.md](./control-runtime.md) and `MultiLayerVerifier` for ordered gates with dependencies.
 - **Adding a new judge or rubric?** → `src/wire/rubrics.ts` for the cross-language path; `src/anti-slop.ts` and `src/judges.ts` for the in-process path.