@tangle-network/agent-eval 0.123.0 → 0.123.2

package/docs/eval-surface-map.md

@@ -1,7 +1,7 @@
-# Eval surface map — which primitive, when
+# Eval surface map: which primitive, when
 
 The eval surface is a small set of orthogonal primitives. They compose; they do
-not overlap. If two seem interchangeable, read the "use when" — the distinction
+not overlap. If two seem interchangeable, read the "use when": the distinction
 is real and load-bearing. **Do not add a new wrapper to bridge two of these; the
 composition point already exists (see Produced-state grading below).**
 
@@ -10,15 +10,15 @@ composition point already exists (see Produced-state grading below).**
 | Primitive | Use when | Returns |
 |---|---|---|
 | `runCampaign` | The measurement primitive. Run a dispatch over scenarios × seeds × reps, score each with judges, aggregate. Caller owns the dispatch. | `CampaignResult` |
-| `runEval` | The simplest preset over `runCampaign` — just score, no loop, no gate. The 80% "I want a scorecard" case. | `CampaignResult` |
+| `runEval` | The simplest preset over `runCampaign`: just score, no loop, no gate. The 80% "I want a scorecard" case. | `CampaignResult` |
 | `runProfileMatrix` | Factor the SAME scenarios across N agent **profiles** (models / prompt variants), with RunRecord stamping + a real-backend integrity guard. | `RunRecord[]` |
 | `runOptimization` | GENERATE: measured or validated premeasured baseline → N generations of propose → measure → rank → promote. No release gate. | generations + winner |
 | `runImprovementLoop` | The release-gate shell around `runOptimization`: adds a held-out re-score + a promotion gate (+ optional auto-PR). | gate decision + winner |
-| `runEvalCampaign` | Inversion-of-control variant of `runCampaign` — the runner is handed a pre-wired trace/sink/emitter and integrity gating as a precondition. Use when you need full capture by construction. | `CampaignResult` + records |
+| `runEvalCampaign` | Inversion-of-control variant of `runCampaign`: the runner is handed a pre-wired trace/sink/emitter and integrity gating as a precondition. Use when you need full capture by construction. | `CampaignResult` + records |
 
 When variants of the same task run inside one `runCampaign`, give those scenarios the same `seedGroup` so each repetition uses common randomness.
 Use `runProfileMatrix` instead when profiles are separate campaign axes.
-| `runAgentMatrix` | The bare N-axis cartesian scheduler with concurrency control. The layer beneath the eval surface — reach for it only when you need raw scheduling, not eval semantics. | cell results |
+| `runAgentMatrix` | The bare N-axis cartesian scheduler with concurrency control. The layer beneath the eval surface: reach for it only when you need raw scheduling, not eval semantics. | cell results |
 
 Mental model: **measure** (`runCampaign`/`runEval`) → **factor** (`runProfileMatrix`) →
 **generate** (`runOptimization`) → **gate** (`runImprovementLoop`). `runEvalCampaign`
@@ -27,11 +27,11 @@ is `runCampaign` with capture inverted; `runAgentMatrix` is the scheduler undern
 Merging any two of these conflates distinct mental models (measure ≠ search ≠
 release-gate). Keep them separate; pick by the table.
 
-## Produced-state grading — there is NO persona-dispatch wrapper
+## Produced-state grading: there is NO persona-dispatch wrapper
 
 To grade what an agent actually **produced** (filed the proposal, wrote the
 artifact) rather than what it said, the composition point is a **judge that wraps
-`verifyCompletion`** — not a dedicated runner. The pipeline:
+`verifyCompletion`**: not a dedicated runner. The pipeline:
 
 ```
 runtime/app-tool events ──► extractProducedState(events) ──► ProducedState
@@ -43,20 +43,20 @@ runtime/app-tool events ──► extractProducedState(events) ──► Produce
 
 `extractProducedState` is a pure function over the produced-event stream; the
 judge calls it inline. This is why **`runProducedStatePersonaDispatch` does not
-exist and should not be built** — it would be a fourth layer over a composition
+exist and should not be built**: it would be a fourth layer over a composition
 that is already one judge. (Archetype: `playback.ts` `scoreUserStory`.)
 
 ### The in-band body contract
 
-Produced events carry their **body in-band** — the grader never reaches into a
+Produced events carry their **body in-band**: the grader never reaches into a
 product database to recover it:
 
 - `artifact` events carry `content` (the persisted file body).
-- `proposal_created` events carry `content` (the `submit_proposal` description) —
+- `proposal_created` events carry `content` (the `submit_proposal` description) -
   same role, same field name. A title-only filing omits it; a content-less
   proposal is graded presence-only (and, by the completion oracle's rule, does
   not count as a completed deliverable).
 
 A consumer that finds itself re-fetching a deliverable's body from its own DB to
-grade it is working around a thin event — fix the event (carry `content`), don't
+grade it is working around a thin event: fix the event (carry `content`), don't
 add an enrichment band-aid.

package/docs/hosted-ingest-spec.md

@@ -1,4 +1,4 @@
-# Hosted-ingest wire spec — `2026-05-26.v1`
+# Hosted-ingest wire spec: `2026-05-26.v1`
 
 The schema **every** orchestrator (ours, partners' self-hosted ones,
 any future open implementation) must accept. Frozen under semver:
@@ -149,7 +149,7 @@ Any orchestrator implementing this spec MUST:
 5. **Persist eval-runs durably**: at least the event + cell scores must
    survive an orchestrator restart. Trace spans MAY be best-effort.
 6. **Provide read access**: GET endpoints for the tenant to list + fetch
-   their own runs. Wire format for reads is NOT part of this spec — each
+   their own runs. Wire format for reads is NOT part of this spec: each
    orchestrator can pick its own (REST + JSON, gRPC, GraphQL).
 
 Servers SHOULD also:
@@ -163,7 +163,7 @@ Servers SHOULD also:
 
 ## Reference implementation
 
-`examples/hosted-ingest-server/` — a minimal hono-based receiver. ~200
+`examples/hosted-ingest-server/`: a minimal hono-based receiver. ~200
 LOC. Validates auth, accepts ingest, stores in memory, exposes a
 read endpoint. Runs anywhere Node runs.
 
@@ -200,4 +200,4 @@ receiver's `GET /v1/runs` lists it back.
 Servers MUST reject requests with `X-Tangle-Wire-Version` they don't
 support, with a 400 listing the versions they DO accept.
 
-The version string IS the spec id — pin against it.
+The version string IS the spec id: pin against it.

package/docs/improvement-glossary.md

@@ -1,12 +1,12 @@
 # Improvement glossary + proposer chooser + composition
 
-> **In plain terms:** this is the dictionary for the *improvement* half of the stack — the words that show up when you optimize an agent (proposer, surface, candidate, generation, holdout, lift, gate…) rather than when you *run* one (driver, worker, iteration — those live in [`agent-runtime/docs/glossary.md`](../../agent-runtime/docs/glossary.md)).
+> **In plain terms:** this is the dictionary for the *improvement* half of the stack: the words that show up when you optimize an agent (proposer, surface, candidate, generation, holdout, lift, gate…) rather than when you *run* one (driver, worker, iteration: those live in [`agent-runtime/docs/glossary.md`](../../agent-runtime/docs/glossary.md)).
 > Read this once and you can read any improvement result, pick a proposer, and wire two of them together.
 
 **Who this is for.**
 A **novice** (never seen the repo) should be able to read a `CampaignResult`, a `ProposeContext`, and a proposer chooser table without opening the source.
-An **expert** who knows DSPy/GEPA should be able to map their existing mental model onto our names in about a minute — every core term below carries a *"if you know DSPy/GEPA"* line.
-If code and this file disagree, the code wins — fix this file the same turn (the anti-staleness law).
+An **expert** who knows DSPy/GEPA should be able to map their existing mental model onto our names in about a minute: every core term below carries a *"if you know DSPy/GEPA"* line.
+If code and this file disagree, the code wins: fix this file the same turn (the anti-staleness law).
 
 Neighbors, so this page does not duplicate them: [`concepts.md`](./concepts.md) (eval mental model), [`self-improvement-map.md`](./self-improvement-map.md) (one loop / four roles / proposer catalog), [`campaign-proposers.md`](./campaign-proposers.md) (proposer ELI5), [`eval-surface-map.md`](./eval-surface-map.md) (which `run*` primitive), [`design/loop-taxonomy.md`](./design/loop-taxonomy.md) (execution vs proposer layering).
 
@@ -24,63 +24,63 @@ baseline surface
    repeat for maxGenerations
 ```
 
-## Glossary — one plain sentence each
+## Glossary: one plain sentence each
 
 Grounded to `agent-eval/src/campaign/types.ts` unless noted.
 
 | Term | Plain sentence | If you know DSPy/GEPA |
 |---|---|---|
-| **surface** | The one thing being changed this run — a prompt string, a JSON config string, or a finalized code candidate (`MutableSurface = string \| CodeSurface`, `types.ts`). | The optimized artifact: a signature/predictor's instruction text, or the module config. |
-| **proposer** | The strategy that, given the current surface + what failed, proposes the next batch of candidate surfaces to measure — it does **not** run the agent or score anything (`SurfaceProposer.propose`, `types.ts:286`). | The optimizer / teleprompter (MIPRO, BootstrapFewShot, GEPA's reflective proposer). |
+| **surface** | The one thing being changed this run: a prompt string, a JSON config string, or a finalized code candidate (`MutableSurface = string \| CodeSurface`, `types.ts`). | The optimized artifact: a signature/predictor's instruction text, or the module config. |
+| **proposer** | The strategy that, given the current surface + what failed, proposes the next batch of candidate surfaces to measure: it does **not** run the agent or score anything (`SurfaceProposer.propose`, `types.ts:286`). | The optimizer / teleprompter (MIPRO, BootstrapFewShot, GEPA's reflective proposer). |
 | **candidate** | One proposed surface plus its human `label` and `rationale`, ready to be measured (`ProposedCandidate`, `types.ts:166`). | One trial instruction/program the optimizer wants to evaluate. |
 | **generation** | One round of *propose → measure → rank → promote*; the loop runs up to `maxGenerations` of them (`GenerationRecord`, `types.ts:549`). | One GEPA iteration / optimization step. |
-| **populationSize** | BREADTH — how many candidate surfaces the proposer returns *this* generation (`ProposeContext.populationSize`, `types.ts:237`). Paired with `maxGenerations` (DEPTH) as the search budget. | Beam width / number of minibatch candidates per step. |
-| **population budget** | Informal name for the pair `{ populationSize, maxGenerations }` — the total candidates the search may evaluate (breadth × depth). | Optimizer trial budget. |
+| **populationSize** | BREADTH: how many candidate surfaces the proposer returns *this* generation (`ProposeContext.populationSize`, `types.ts:237`). Paired with `maxGenerations` (DEPTH) as the search budget. | Beam width / number of minibatch candidates per step. |
+| **population budget** | Informal name for the pair `{ populationSize, maxGenerations }`: the total candidates the search may evaluate (breadth × depth). | Optimizer trial budget. |
 | **campaign** | One measurement: run a dispatch over scenarios × seeds × reps, judge each output, aggregate → `CampaignResult` (`runCampaign`; a "campaign" = a coordinated batch of measurements). | One evaluation pass of a candidate over a valset. |
-| **cell** | The atomic measurement unit — exactly one `(scenario, rep)` execution producing one artifact + its judge scores + its cost (`CampaignCellResult`, `types.ts:509`; `DispatchContext.cellId/rep`). | One (example, seed) evaluation datapoint. |
+| **cell** | The atomic measurement unit: exactly one `(scenario, rep)` execution producing one artifact + its judge scores + its cost (`CampaignCellResult`, `types.ts:509`; `DispatchContext.cellId/rep`). | One (example, seed) evaluation datapoint. |
 | **scenario** | One input case with a stable `id` + `kind` (consumers attach their payload: persona, task, requirement) (`Scenario`, `types.ts:23`). | One dataset example / `dspy.Example`. |
-| **rep** | The repetition index — the same scenario run more than once so noise/variance is measurable rather than mistaken for signal (`ctx.rep`). | Repeated sampling of the same example (temperature/seed variance). |
+| **rep** | The repetition index: the same scenario run more than once so noise/variance is measurable rather than mistaken for signal (`ctx.rep`). | Repeated sampling of the same example (temperature/seed variance). |
 | **judge** | A scorer: given an artifact, return dimensions + a single `composite` + free-form `notes`; it throws on failure rather than silently scoring 0 (`JudgeConfig` → `JudgeScore`, `types.ts:94/116`). | The metric function, but pluggable (LLM-judge, deterministic checks, or an ensemble). |
-| **composite** | The single 0..1 number that combines all judge dimensions — the number you rank and gate on (`JudgeScore.composite`, `types.ts:116`). | The scalar metric value. |
-| **findings** | The failure analysis handed to the proposer so it edits from evidence, not guesses — worst cells + judge reasons, or a trace-analyst's clusters (`ProposeContext.findings`). | GEPA's reflective feedback / the "textual gradient". |
+| **composite** | The single 0..1 number that combines all judge dimensions: the number you rank and gate on (`JudgeScore.composite`, `types.ts:116`). | The scalar metric value. |
+| **findings** | The failure analysis handed to the proposer so it edits from evidence, not guesses: worst cells + judge reasons, or a trace-analyst's clusters (`ProposeContext.findings`). | GEPA's reflective feedback / the "textual gradient". |
 | **baseline** | The starting surface (the current prompt/config) every candidate is measured against (`baselineSurface`). | The unoptimized program you compare lift against. |
-| **holdout** | A separate scenario split the winner is re-scored on and that the proposer is *never* allowed to see — a compile-time firewall (`ProposeContext.judgeScores: never`, `types.ts:265`) keeps held-out verdicts out of proposal so the optimizer can't game the acceptance axis. | The held-out valset/testset — but here it is *write-only* to the optimizer. |
-| **lift** | Winner minus baseline `composite` on the holdout — the actual improvement, reported with a bootstrap confidence interval (`ImproveResult.lift`, `ProposerScore.lift`). | Δ metric between optimized and baseline program. |
-| **MDE** | Minimum Detectable Effect — the smallest lift your budget could statistically distinguish from noise, computed up front from baseline cells; a structurally-hopeless budget warns before you spend (surfaced as `result.power` / "power preflight", `agent-optimization-map.md`). Not spelled "MDE" in code yet — this is the standard name for it. | Power analysis on the eval set; the reason a tiny valset can't certify a small gain. |
-| **Pareto frontier** | The set of surfaces that are non-dominated across the per-scenario score vectors — a candidate worse on the mean but uniquely best on one hard scenario survives, so its lesson is not discarded (`ParetoParent`, `types.ts:198`; GEPA, arXiv:2507.19457). | Exactly GEPA's Pareto candidate pool — same paper, same idea. |
+| **holdout** | A separate scenario split the winner is re-scored on and that the proposer is *never* allowed to see: a compile-time firewall (`ProposeContext.judgeScores: never`, `types.ts:265`) keeps held-out verdicts out of proposal so the optimizer can't game the acceptance axis. | The held-out valset/testset: but here it is *write-only* to the optimizer. |
+| **lift** | Winner minus baseline `composite` on the holdout: the actual improvement, reported with a bootstrap confidence interval (`ImproveResult.lift`, `ProposerScore.lift`). | Δ metric between optimized and baseline program. |
+| **MDE** | Minimum Detectable Effect: the smallest lift your budget could statistically distinguish from noise, computed up front from baseline cells; a structurally-hopeless budget warns before you spend (surfaced as `result.power` / "power preflight", `agent-optimization-map.md`). Not spelled "MDE" in code yet: this is the standard name for it. | Power analysis on the eval set; the reason a tiny valset can't certify a small gain. |
+| **Pareto frontier** | The set of surfaces that are non-dominated across the per-scenario score vectors: a candidate worse on the mean but uniquely best on one hard scenario survives, so its lesson is not discarded (`ParetoParent`, `types.ts:198`; GEPA, arXiv:2507.19457). | Exactly GEPA's Pareto candidate pool: same paper, same idea. |
 | **gate** | The promotion decision: does the winner beat baseline on the holdout with significance, returning one of five verdicts `ship / hold / need_more_work / model_ceiling / arch_ceiling` (`Gate`, `GateDecision`, `types.ts:315/340`). | The accept/reject rule on the held-out valset, plus a significance test and ceiling diagnosis. |
-| **promotion** | What happens on a `ship` verdict — the winning surface is written back into the profile field it came from (`GenerationRecord.promoted`; `applyWinnerToProfile`, `improve.ts`). | Committing the optimized program as the new default. |
+| **promotion** | What happens on a `ship` verdict: the winning surface is written back into the profile field it came from (`GenerationRecord.promoted`; `applyWinnerToProfile`, `improve.ts`). | Committing the optimized program as the new default. |
 
-## Which proposer — the chooser
+## Which proposer: the chooser
 
 Every optimizer is a factory `xProposer(opts): SurfaceProposer`, all exported from `@tangle-network/agent-eval/campaign`.
 Start at the top row and only move down when the row's *"reach for it when"* matches your failure mode.
 
 | Proposer factory | Reach for it when | Surface it edits | Wired to the paved path? |
 |---|---|---|---|
-| `gepaProposer` | You want the strong default: reflective full-surface prompt rewrites, grounded in findings, keeping a Pareto frontier of complementary winners. | prompt string | **Yes — `improve({ surface: 'prompt' })` default.** Proven live. |
-| `skillOptProposer` | You are editing a structured `SKILL.md`/runbook and want small anchored add/delete/replace patches that preserve earlier rules. | skill/prompt string | **Yes — `improve({ surface: 'skills' })` default.** Not yet proven live. |
-| `parameterSweepProposer` | The likely fix is a config knob, not words — `retrieval.k`, `temperature`, `max_tokens`. You give it candidate patches; it applies them to a JSON surface. | JSON config string | Yes, but you supply the candidate list. |
-| `fapoProposer` | You want *evidence to decide when to escalate*: try prompt edits first, move to parameters, then to structural code — one scoped change per cycle, only escalating when the cheaper level is exhausted. | whatever its level proposers return | Exported; you wire the level proposers. |
+| `gepaProposer` | You want the strong default: reflective full-surface prompt rewrites, grounded in findings, keeping a Pareto frontier of complementary winners. | prompt string | **Yes: `improve({ surface: 'prompt' })` default.** Proven live. |
+| `skillOptProposer` | You are editing a structured `SKILL.md`/runbook and want small anchored add/delete/replace patches that preserve earlier rules. | skill/prompt string | **Yes: `improve({ surface: 'skills' })` default.** Not yet proven live. |
+| `parameterSweepProposer` | The likely fix is a config knob, not words: `retrieval.k`, `temperature`, `max_tokens`. You give it candidate patches; it applies them to a JSON surface. | JSON config string | Yes, but you supply the candidate list. |
+| `fapoProposer` | You want *evidence to decide when to escalate*: try prompt edits first, move to parameters, then to structural code: one scoped change per cycle, only escalating when the cheaper level is exhausted. | whatever its level proposers return | Exported; you wire the level proposers. |
 | `compositeProposer` | You want several proposers to share one candidate-generation budget in the same round. It allocates the population by declared weights, preserves member provenance, deduplicates surfaces, and isolates a member failure unless every member fails. | whatever its member proposers return | Exported; you wire the member proposers. |
 | `aceProposer` | You are accumulating hard-won lessons into a playbook and must **never** summarize an old lesson away (append-only, provenance-tagged). | playbook string | Exported. |
 | `memoryCurationProposer` | Same as ACE but you want a compact, deduped, re-ranked memory instead of append-only growth. | memory string | Exported. |
-| `evolutionaryProposer` | You want blind population search (mutate → measure → select) with no reflection over findings — a cheap control or a baseline to beat. | any string | Exported. |
+| `evolutionaryProposer` | You want blind population search (mutate → measure → select) with no reflection over findings: a cheap control or a baseline to beat. | any string | Exported. |
 | `traceAnalystProposer` | Bench-only: race our trace-analysis evidence engine head-to-head inside `compareProposers`. | prompt string | Bench-only. |
 | `haloProposer` | Bench-only: race the external `halo-engine` analysis against ours. | prompt string | Bench-only, external. |
 
 Default path: `gepaProposer` for prompts; add `parameterSweepProposer` when a config knob is the suspect; wrap levels in `fapoProposer` when the loop should decide *when* to escalate, or use `compositeProposer` when multiple proposer families must split one fixed population budget.
 
-## Composing proposers — four distinct shapes
+## Composing proposers: four distinct shapes
 
 Choose the shape that matches the experiment:
 
-1. **Portfolio** — `compositeProposer` splits one generation's population across member proposers by fixed weights and returns one provenance-labelled pool.
-2. **Escalate** — `fapoProposer` wraps prompt + parameter + structural levels into one proposer and spends on the cheapest level until evidence says to escalate.
-3. **Race** — `compareProposers` gives proposers separate loops, then re-scores their winners on one holdout and returns per-proposer lift intervals plus pairwise results.
-4. **Plug in** — hand any proposer to `runImprovementLoop({ proposer })`, or use `improve({ surface, generator })` in `@tangle-network/agent-runtime`.
+1. **Portfolio**: `compositeProposer` splits one generation's population across member proposers by fixed weights and returns one provenance-labelled pool.
+2. **Escalate**: `fapoProposer` wraps prompt + parameter + structural levels into one proposer and spends on the cheapest level until evidence says to escalate.
+3. **Race**: `compareProposers` gives proposers separate loops, then re-scores their winners on one holdout and returns per-proposer lift intervals plus pairwise results.
+4. **Plug in**: hand any proposer to `runImprovementLoop({ proposer })`, or use `improve({ surface, generator })` in `@tangle-network/agent-runtime`.
 
-### 2 + 4 — compose by escalation, then run the improvement loop
+### 2 + 4: compose by escalation, then run the improvement loop
 
 ```ts
 import {
@@ -110,8 +110,8 @@ const proposer = fapoProposer({
   }),
 })
 
-// dispatchWithSurface scores ONE surface on ONE scenario — this is the topology-
-// opaque seam (one LLM call, one worker, or a whole fleet — the loop can't tell).
+// dispatchWithSurface scores ONE surface on ONE scenario: this is the topology-
+// opaque seam (one LLM call, one worker, or a whole fleet: the loop can't tell).
 const dispatchWithSurface = async (surface, scenario, ctx) =>
   runYourAgent({ surface, scenario, signal: ctx.signal })   // returns the artifact to judge
 
@@ -148,7 +148,7 @@ const out = await improve(profile, findings, {
 if (out.shipped) deploy(out.profile)  // out.lift is the held-out winner − baseline
 ```
 
-### 3 — race proposers head-to-head for a lift CI
+### 3: race proposers head-to-head for a lift CI
 
 ```ts
 import {
@@ -186,19 +186,19 @@ const comparison = await compareProposers({
 })
 
 comparison.best          // highest-lift proposer
-comparison.scores        // per-proposer { lift, liftCi:{low,high}, cost } — low>0 ⇒ real gain
+comparison.scores        // per-proposer { lift, liftCi:{low,high}, cost }: low>0 ⇒ real gain
 comparison.pairwise      // best vs each other, paired-bootstrap: 'a' | 'b' | 'tie'
 ```
 
-Every entrant is re-scored on the **same** holdout with the **same** judges, so the comparison never trusts how a proposer measured itself — the only variable is proposal quality.
+Every entrant is re-scored on the **same** holdout with the **same** judges, so the comparison never trusts how a proposer measured itself: the only variable is proposal quality.
 
 ## Common traps (they cost the most)
 
-- Do not put eval logic inside a proposer — scoring lives in `dispatch` + `judges`, proposing lives in the proposer.
-- Do not let a proposer read held-out judge scores — `ProposeContext` makes that a compile error on purpose; a proposer that games the acceptance axis is an oracle, not an optimizer.
-- Do not read `lift` without `result.power`/MDE — a "+4" on a valset too small to detect +4 is noise wearing a number.
+- Do not put eval logic inside a proposer: scoring lives in `dispatch` + `judges`, proposing lives in the proposer.
+- Do not let a proposer read held-out judge scores: `ProposeContext` makes that a compile error on purpose; a proposer that games the acceptance axis is an oracle, not an optimizer.
+- Do not read `lift` without `result.power`/MDE: a "+4" on a valset too small to detect +4 is noise wearing a number.
 - Do not confuse `compositeProposer` with `fapoProposer`: the former allocates one fixed population across peers, while the latter escalates through ordered levels from cheaper to more structural changes.
 
-### neutralizationGate — the placebo / content-causality control
+### neutralizationGate: the placebo / content-causality control
 
 Standard gates prove a candidate *beat baseline*. `neutralizationGate` proves the candidate's **content** caused the lift, not the extra prompt bytes: it blanks the candidate's added content to byte-length-matched filler, holds everything else fixed, and requires the lift to vanish. A fully-neutralized candidate that still scores is decorative and is rejected. Exports: `neutralizationGate` (`src/campaign/gates/neutralization-gate.ts`), `neutralizeText` (`src/campaign/neutralize.ts`). Since 0.107.0. Pair it with `heldOutGate` in any loop that promotes authored artifacts (prompts, tool docs, knowledge) so a lift that is really just added prompt size cannot be shipped.

package/docs/insight-report.md

@@ -1,11 +1,11 @@
-# `InsightReport` — the decision packet
+# `InsightReport`: the report
 
 The single shape every analysis call returns. `selfImprove()` embeds it in `SelfImproveResult.insight`; `analyzeRuns()` returns it directly. The hosted-tier wire format carries it on `EvalRunEvent.insightReport?`.
 
 Use `summarizeExecution({ runs })` when observed traces have no task-quality labels.
 It returns only `execution` and `costProvenance`, so callers do not need to fabricate a quality score to report runtime facts.
 
-Every section is **opt-in based on what your data supports** — the function never invents signal. If your runs don't carry judge scores, `judges` is empty. If there's no baseline/candidate split, `lift` is undefined. The shape is consistent; population is honest.
+Every section is **opt-in based on what your data supports**: the function never invents signal. If your runs don't carry judge scores, `judges` is empty. If there's no baseline/candidate split, `lift` is undefined. The shape is consistent; population is honest.
 
 This page walks every section with a real (synthetic) example and explains how to act on it.
 
@@ -27,13 +27,13 @@ interface InsightReport {
   contamination?: ContaminationInsight   // when canaryScenarios supplied
   outcomeCorrelation?: OutcomeCorrelationInsight   // when outcomeSignal supplied
   release: ReleaseSummary                // always
-  recommendations: Recommendation[]      // always — read this FIRST
+  recommendations: Recommendation[]      // always: read this FIRST
 }
 ```
 
 ---
 
-## `execution` — runtime facts, separate from quality
+## `execution`: runtime facts, separate from quality
 
 Always present.
 It reports duration, optional queue time, direct input, output, reasoning, cache-read, and cache-write tokens, model-call coverage, model cohorts, explicit failures, and separately reported orchestration aggregates.
@@ -73,7 +73,7 @@ Cost remains in `costQuality`, where observed, estimated, and uncaptured USD sta
 
 ---
 
-## `n` + `composite` + `perDimension` — distributional summary
+## `n` + `composite` + `perDimension`: distributional summary
 
 Always present. The basic "where are my numbers" view.
 
@@ -99,13 +99,13 @@ Always present. The basic "where are my numbers" view.
 
 **Read first:** the `composite.mean`. If it's < 0.5, your agent has a ceiling problem, not a tuning problem.
 
-**Read next:** `perDimension`. If `clarity` is high but `concision` is low, your prompts get the right ideas in too many words — different fix than "wrong ideas."
+**Read next:** `perDimension`. If `clarity` is high but `concision` is low, your prompts get the right ideas in too many words: different fix than "wrong ideas."
 
 **Use the histogram for:** finding bimodal failure modes. A bin with `count > 0` near zero and another > 0 near 1 means your agent has two distinct behaviors, not one noisy one.
 
 ---
 
-## `costQuality` — cost-vs-quality Pareto
+## `costQuality`: cost-vs-quality Pareto
 
 Always present. `cost.histogram` is the per-run cost distribution; `pareto` is the substrate's `ParetoFigureSpec`.
 
@@ -131,11 +131,11 @@ Always present. `cost.histogram` is the per-run cost distribution; `pareto` is t
 
 **Use this when:** comparing prompts, models, or candidate surfaces. The Pareto frontier is your menu of "best you can do at each cost level."
 
-**Render with:** any chart library — `points` is plain JSON. Hosted-tier dashboards render this as a scatter with the frontier highlighted.
+**Render with:** any chart library: `points` is plain JSON. Hosted-tier dashboards render this as a scatter with the frontier highlighted.
 
 ---
 
-## `judges` — per-judge mean
+## `judges`: per-judge mean
 
 Populated when run records carry `outcome.judgeScores`.
 
@@ -154,20 +154,23 @@ The substrate's full judge-calibration suite (positional bias, self-preference,
 
 ---
 
-## `interRater` — multi-rater agreement + disagreement triage
+## `interRater`: multi-rater agreement and disagreement review
 
-Populated when `analyzeRuns({ raterScores })` is supplied — typically via `fromFeedbackTable()`.
+Populated when `analyzeRuns({ raterScores })` is supplied: typically via `fromFeedbackTable()`.
 
 ```jsonc
 {
   "interRater": {
     "raters": 3,
     "jointlyRated": 30,
-    "kappa": 0.71,
+    "kappa": 0.40,
+    "icc": 0.42,
+    "pearson": 0.43,
+    "spearman": 0.41,
     "perPair": {
-      "alice::bob":   0.78,
-      "alice::carol": 0.65,
-      "bob::carol":   0.69
+      "alice::bob":   0.53,
+      "alice::carol": 0.47,
+      "bob::carol":   0.19
     },
     "disagreementCases": [
       { "runId": "claim-7", "range": 1.00,
@@ -180,13 +183,15 @@ Populated when `analyzeRuns({ raterScores })` is supplied — typically via `fro
 }
 ```
 
-**Read first:** the mean `kappa`. < 0.5 means raters disagree on what "good" looks like — surface the disagreement cases at the next review meeting.
+**Read first:** `kappa` and `icc`, which measure absolute agreement.
+Pearson and Spearman measure correlation and can remain high when raters use different score levels.
+When absolute agreement is low, review the largest disagreement cases before automating the rubric.
 
 **Use this when:** building per-rater LLM judges. Each rater's individual scores are the gold signal you calibrate against. Once a calibrated LLM matches the human ≥85%, you can auto-grade and escalate only the disagreement cases.
 
 ---
 
-## `lift` — paired-bootstrap statistical lift
+## `lift`: paired-bootstrap statistical lift
 
 Populated when baseline + candidate candidates are present (auto-detected from two distinct `candidateId`s, or explicit via `baselineCandidateId` + `candidateCandidateId`).
 
@@ -211,13 +216,13 @@ Populated when baseline + candidate candidates are present (auto-detected from t
 - `ci95[0] ≤ threshold < ci95[1]` → **INCONCLUSIVE.** Expand the corpus or wait for more data.
 - `ci95[1] ≤ threshold` → **HOLD.** No evidence the candidate is better.
 
-The `recommendations` array surfaces exactly this decision (`kind: 'ship' | 'hold' | 'expand-corpus'`) — that's what consumers should read.
+The `recommendations` array surfaces exactly this decision (`kind: 'ship' | 'hold' | 'expand-corpus'`): that's what consumers should read.
 
 **Why bootstrap, not t-test alone:** paired bootstrap is distribution-free. Your judge scores are bounded in [0,1] and almost never normal; the bootstrap CI is the honest one.
 
 ---
 
-## `failureClusters` — grouped failure modes
+## `failureClusters`: grouped failure modes
 
 Populated when an `AnalystRegistry` is passed via `analyzeRuns({ analyst })`. The substrate runs each failed run through the registered analysts and groups findings by `analyst_id` / `area`.
 
@@ -245,7 +250,7 @@ Populated when an `AnalystRegistry` is passed via `analyzeRuns({ analyst })`. Th
 
 ---
 
-## `contamination` — canary check
+## `contamination`: canary check
 
 Populated when canary scenarios are passed via `analyzeRuns({ canaryScenarios })`. Each canary carries a sentinel string the agent should never emit; the report counts leaks.
 
@@ -277,7 +282,7 @@ When `leaks > 0`:
 
 ---
 
-## `outcomeCorrelation` — closing the loop on real outcomes
+## `outcomeCorrelation`: closing the loop on real outcomes
 
 Populated when `outcomeSignal: { metric, valueByRunId }` is supplied.
 
@@ -307,9 +312,9 @@ This is the layer that says **"does my judge's taste actually predict the metric
 
 ---
 
-## `release` — pass/warn/fail axes
+## `release`: pass/warn/fail axes
 
-Always present. Roll-up across three axes — quality lift, contamination, composite distribution.
+Always present. Roll-up across three axes: quality lift, contamination, composite distribution.
 
 ```jsonc
 {
@@ -334,7 +339,7 @@ Overall `status` is `fail` if any axis fails; `warn` if any warn; `pass` otherwi
 
 ---
 
-## `recommendations` — the actionable layer
+## `recommendations`: the actionable layer
 
 Always present. Read this first.
 
@@ -342,7 +347,7 @@ Always present. Read this first.
 {
   "recommendations": [
     { "priority": "critical", "kind": "ship",
-      "title": "Ship — lift 0.070 (95% CI 0.040..0.100)",
+      "title": "Ship: lift 0.070 (95% CI 0.040..0.100)",
       "detail": "Holdout lift exceeds threshold 0.02 with 95% bootstrap confidence (n=40, p=0.0008, d=0.41).",
       "evidencePath": "lift" },
     { "priority": "high", "kind": "investigate",
@@ -357,7 +362,7 @@ Always present. Read this first.
 |---|---|
 | `ship` | lift CI lower bound > threshold |
 | `hold` | lift CI upper bound ≤ threshold |
-| `expand-corpus` | lift CI straddles threshold — more data needed |
+| `expand-corpus` | lift CI straddles threshold: more data needed |
 | `fix` | canary contamination detected |
 | `recalibrate` | inter-rater κ < 0.5, OR outcome correlation < 0.3 |
 | `investigate` | top failure cluster > some-share |
@@ -378,4 +383,4 @@ Always present. Read this first.
 | `contamination` | `canaryScenarios` passed in |
 | `outcomeCorrelation` | `outcomeSignal` passed in |
 
-All sections beyond the always-present ones are `T | undefined`, never empty objects. If a section is missing, your inputs didn't support it — the report is honest about that.
+All sections beyond the always-present ones are `T | undefined`, never empty objects. If a section is missing, your inputs didn't support it: the report is honest about that.

package/docs/multi-shot-optimization.md

@@ -6,7 +6,7 @@
 > live API; see also [feature-guide.md](./feature-guide.md) and [concepts.md](./concepts.md).
 
 `runImprovementLoop` is the public entry for GEPA-style optimization over a whole
-task trajectory — the thing you improve is not a single model call but an agent
+task trajectory: the thing you improve is not a single model call but an agent
 system prompt, tool descriptions, a routing policy, or any scaffolding that affects
 the entire run. It is the OUTER loop: it improves the SURFACE the inner workers run.
 
@@ -15,13 +15,13 @@ the entire run. It is the OUTER loop: it improves the SURFACE the inner workers
 You own a few seams; the loop owns the release-critical glue (paired seeds, the
 held-out re-score, the promotion gate, provenance):
 
-- **`baselineSurface`** — the current surface (a prompt string, or a `CodeSurface`).
-- **`dispatchWithSurface(surface, scenario, ctx)`** — run one task to completion
+- **`baselineSurface`**: the current surface (a prompt string, or a `CodeSurface`).
+- **`dispatchWithSurface(surface, scenario, ctx)`**: run one task to completion
   under a candidate surface; return the artifact the judges score.
-- **`judges`** — score the artifact (`{ composite, dimensions }`).
-- **`proposer`** — proposes candidate surfaces each generation: `gepaProposer`
+- **`judges`**: score the artifact (`{ composite, dimensions }`).
+- **`proposer`**: proposes candidate surfaces each generation: `gepaProposer`
   (reflective + Pareto frontier) or `evolutionaryProposer` (mutator).
-- **`gate`** — `defaultProductionGate` (held-out significance + red-team +
+- **`gate`**: `defaultProductionGate` (held-out significance + red-team +
   reward-hacking + canary). Ships ONLY on a CI-lower-bound held-out lift.
 
 ## Minimal example
@@ -36,7 +36,7 @@ import {
 const result = await runImprovementLoop({
   baselineSurface: currentSystemPrompt,
   scenarios: trainScenarios, // optimizer-visible
-  holdoutScenarios, // DISJOINT — only the gate sees these
+  holdoutScenarios, // DISJOINT: only the gate sees these
   dispatchWithSurface: async (surface, scenario) =>
     runYourAgentToCompletion({ scenario, prompt: String(surface) }),
   judges: [myJudge],
@@ -64,7 +64,7 @@ if (result.gateResult.decision === 'ship') {
   paired-bootstrap CI lower bound clears `deltaThreshold`; a few-instance swing at
   thin `n` is held (`few_runs`), not promoted.
 - **No-op never ships.** If no candidate beats the baseline, the winner IS the
-  baseline (empty diff) and the loop forces `hold` — it does not score
+  baseline (empty diff) and the loop forces `hold`: it does not score
   baseline-vs-itself and read model noise as lift.
 - **Provenance falls out.** `result.promotedDiff` + `emitLoopProvenance` give the
   auditable candidate→gate→promote chain (rationale, content hashes, a held-out lift

package/docs/research-report-methodology.md

@@ -1,4 +1,4 @@
-# researchReport — methodology
+# researchReport: methodology
 
 This document is the methodological brief for `researchReport` (exported from
 `@tangle-network/agent-eval` and `@tangle-network/agent-eval/reporting`). It
@@ -11,14 +11,14 @@ corresponding function or option so the doc and the code don't drift.
 
 ## Inputs
 
-- `runs: RunRecord[]` — every record carries `runId`, `candidateId`, `seed`,
+- `runs: RunRecord[]`: every record carries `runId`, `candidateId`, `seed`,
   `experimentId`, `splitTag`, and an `outcome` with the configured score.
-- `comparator: string` — the candidate id treated as the null reference. Must
+- `comparator: string`: the candidate id treated as the null reference. Must
   be selected before data inspection; `preregistrationHash` should pin this.
-- `split: 'search' | 'holdout'` — defaults to `holdout`. Decisions on `search`
+- `split: 'search' | 'holdout'`: defaults to `holdout`. Decisions on `search`
   are descriptive only; promotion calls require the holdout.
-- `rope: { low, high }` — Region of Practical Equivalence on the paired delta,
-  in score units. Must come from the domain owner — there is no
+- `rope: { low, high }`: Region of Practical Equivalence on the paired delta,
+  in score units. Must come from the domain owner: there is no
   statistically-defensible default.
 - `minPairs` (soft floor, default 20) and `RESEARCH_REPORT_HARD_PAIR_FLOOR`
   (hard floor, 6). Below the soft floor, the verdict is `needs_more_data` and
@@ -35,7 +35,7 @@ with non-finite scores are dropped before pairing.
 
 ## Decision rule
 
-In order — first match wins:
+In order: first match wins:
 
 1. `comparator` itself → `hold` (baseline).
 2. No comparator → `hold` if on the cost/quality Pareto frontier, else
@@ -46,7 +46,7 @@ In order — first match wins:
    "below hard floor" reason. Bootstrap CIs degenerate at this size.
 5. ROPE configured AND paired-delta CI ⊂ ROPE → `equivalent`.
 6. Paired-delta CI upper bound < 0 → `reject` (CI excludes a non-negative
-   effect). Note: this uses **paired delta only** — not the marginal mean.
+   effect). Note: this uses **paired delta only**: not the marginal mean.
 7. Paired N < `minPairs` (soft floor) → `needs_more_data` with the MDE at
    current N attached so the verdict is actionable.
 8. BH-adjusted q ≤ `fdr` AND CI lower bound > 0 → `promote`. The BH q-value
@@ -71,7 +71,7 @@ The Pr(Δ>0) and Pr(Δ∈ROPE) summaries use the bootstrap-prior duality of
 [Rubin 1981]: under a non-informative Dirichlet prior, the bootstrap
 distribution of a sample statistic is its posterior. We expose these as
 posterior summaries on the **mean** delta and the bootstrap CI on the
-**median** delta — the median is more robust to the heavy-tailed score
+**median** delta: the median is more robust to the heavy-tailed score
 distributions seen in agent benchmarks; the mean lets us read off the
 Bayesian-style probability of superiority in a single number.