@tangle-network/agent-eval 0.123.1 → 0.123.3

package/docs/eval-surface-map.md

@@ -1,7 +1,7 @@
-# Eval surface map — which primitive, when
+# Eval surface map: which primitive, when
 
 The eval surface is a small set of orthogonal primitives. They compose; they do
-not overlap. If two seem interchangeable, read the "use when" — the distinction
+not overlap. If two seem interchangeable, read the "use when": the distinction
 is real and load-bearing. **Do not add a new wrapper to bridge two of these; the
 composition point already exists (see Produced-state grading below).**
 
@@ -10,15 +10,15 @@ composition point already exists (see Produced-state grading below).**
 | Primitive | Use when | Returns |
 |---|---|---|
 | `runCampaign` | The measurement primitive. Run a dispatch over scenarios × seeds × reps, score each with judges, aggregate. Caller owns the dispatch. | `CampaignResult` |
-| `runEval` | The simplest preset over `runCampaign` — just score, no loop, no gate. The 80% "I want a scorecard" case. | `CampaignResult` |
+| `runEval` | The simplest preset over `runCampaign`: just score, no loop, no gate. The 80% "I want a scorecard" case. | `CampaignResult` |
 | `runProfileMatrix` | Factor the SAME scenarios across N agent **profiles** (models / prompt variants), with RunRecord stamping + a real-backend integrity guard. | `RunRecord[]` |
 | `runOptimization` | GENERATE: measured or validated premeasured baseline → N generations of propose → measure → rank → promote. No release gate. | generations + winner |
 | `runImprovementLoop` | The release-gate shell around `runOptimization`: adds a held-out re-score + a promotion gate (+ optional auto-PR). | gate decision + winner |
-| `runEvalCampaign` | Inversion-of-control variant of `runCampaign` — the runner is handed a pre-wired trace/sink/emitter and integrity gating as a precondition. Use when you need full capture by construction. | `CampaignResult` + records |
+| `runEvalCampaign` | Inversion-of-control variant of `runCampaign`: the runner is handed a pre-wired trace/sink/emitter and integrity gating as a precondition. Use when you need full capture by construction. | `CampaignResult` + records |
 
 When variants of the same task run inside one `runCampaign`, give those scenarios the same `seedGroup` so each repetition uses common randomness.
 Use `runProfileMatrix` instead when profiles are separate campaign axes.
-| `runAgentMatrix` | The bare N-axis cartesian scheduler with concurrency control. The layer beneath the eval surface — reach for it only when you need raw scheduling, not eval semantics. | cell results |
+| `runAgentMatrix` | The bare N-axis cartesian scheduler with concurrency control. The layer beneath the eval surface: reach for it only when you need raw scheduling, not eval semantics. | cell results |
 
 Mental model: **measure** (`runCampaign`/`runEval`) → **factor** (`runProfileMatrix`) →
 **generate** (`runOptimization`) → **gate** (`runImprovementLoop`). `runEvalCampaign`
@@ -27,11 +27,11 @@ is `runCampaign` with capture inverted; `runAgentMatrix` is the scheduler undern
 Merging any two of these conflates distinct mental models (measure ≠ search ≠
 release-gate). Keep them separate; pick by the table.
 
-## Produced-state grading — there is NO persona-dispatch wrapper
+## Produced-state grading: there is NO persona-dispatch wrapper
 
 To grade what an agent actually **produced** (filed the proposal, wrote the
 artifact) rather than what it said, the composition point is a **judge that wraps
-`verifyCompletion`** — not a dedicated runner. The pipeline:
+`verifyCompletion`**: not a dedicated runner. The pipeline:
 
 ```
 runtime/app-tool events ──► extractProducedState(events) ──► ProducedState
@@ -43,20 +43,20 @@ runtime/app-tool events ──► extractProducedState(events) ──► Produce
 
 `extractProducedState` is a pure function over the produced-event stream; the
 judge calls it inline. This is why **`runProducedStatePersonaDispatch` does not
-exist and should not be built** — it would be a fourth layer over a composition
+exist and should not be built**: it would be a fourth layer over a composition
 that is already one judge. (Archetype: `playback.ts` `scoreUserStory`.)
 
 ### The in-band body contract
 
-Produced events carry their **body in-band** — the grader never reaches into a
+Produced events carry their **body in-band**: the grader never reaches into a
 product database to recover it:
 
 - `artifact` events carry `content` (the persisted file body).
-- `proposal_created` events carry `content` (the `submit_proposal` description) —
+- `proposal_created` events carry `content` (the `submit_proposal` description) -
   same role, same field name. A title-only filing omits it; a content-less
   proposal is graded presence-only (and, by the completion oracle's rule, does
   not count as a completed deliverable).
 
 A consumer that finds itself re-fetching a deliverable's body from its own DB to
-grade it is working around a thin event — fix the event (carry `content`), don't
+grade it is working around a thin event: fix the event (carry `content`), don't
 add an enrichment band-aid.

package/docs/hosted-ingest-spec.md

@@ -1,4 +1,4 @@
-# Hosted-ingest wire spec — `2026-05-26.v1`
+# Hosted-ingest wire spec: `2026-05-26.v1`
 
 The schema **every** orchestrator (ours, partners' self-hosted ones,
 any future open implementation) must accept. Frozen under semver:
@@ -149,7 +149,7 @@ Any orchestrator implementing this spec MUST:
 5. **Persist eval-runs durably**: at least the event + cell scores must
    survive an orchestrator restart. Trace spans MAY be best-effort.
 6. **Provide read access**: GET endpoints for the tenant to list + fetch
-   their own runs. Wire format for reads is NOT part of this spec — each
+   their own runs. Wire format for reads is NOT part of this spec: each
    orchestrator can pick its own (REST + JSON, gRPC, GraphQL).
 
 Servers SHOULD also:
@@ -163,7 +163,7 @@ Servers SHOULD also:
 
 ## Reference implementation
 
-`examples/hosted-ingest-server/` — a minimal hono-based receiver. ~200
+`examples/hosted-ingest-server/`: a minimal hono-based receiver. ~200
 LOC. Validates auth, accepts ingest, stores in memory, exposes a
 read endpoint. Runs anywhere Node runs.
 
@@ -200,4 +200,4 @@ receiver's `GET /v1/runs` lists it back.
 Servers MUST reject requests with `X-Tangle-Wire-Version` they don't
 support, with a 400 listing the versions they DO accept.
 
-The version string IS the spec id — pin against it.
+The version string IS the spec id: pin against it.

package/docs/insight-report.md

@@ -1,11 +1,11 @@
-# `InsightReport` — the decision packet
+# `InsightReport`: the report
 
 The single shape every analysis call returns. `selfImprove()` embeds it in `SelfImproveResult.insight`; `analyzeRuns()` returns it directly. The hosted-tier wire format carries it on `EvalRunEvent.insightReport?`.
 
 Use `summarizeExecution({ runs })` when observed traces have no task-quality labels.
 It returns only `execution` and `costProvenance`, so callers do not need to fabricate a quality score to report runtime facts.
 
-Every section is **opt-in based on what your data supports** — the function never invents signal. If your runs don't carry judge scores, `judges` is empty. If there's no baseline/candidate split, `lift` is undefined. The shape is consistent; population is honest.
+Every section is **opt-in based on what your data supports**: the function never invents signal. If your runs don't carry judge scores, `judges` is empty. If there's no baseline/candidate split, `lift` is undefined. The shape is consistent; population is honest.
 
 This page walks every section with a real (synthetic) example and explains how to act on it.
 
@@ -27,13 +27,13 @@ interface InsightReport {
   contamination?: ContaminationInsight   // when canaryScenarios supplied
   outcomeCorrelation?: OutcomeCorrelationInsight   // when outcomeSignal supplied
   release: ReleaseSummary                // always
-  recommendations: Recommendation[]      // always — read this FIRST
+  recommendations: Recommendation[]      // always: read this FIRST
 }
 ```
 
 ---
 
-## `execution` — runtime facts, separate from quality
+## `execution`: runtime facts, separate from quality
 
 Always present.
 It reports duration, optional queue time, direct input, output, reasoning, cache-read, and cache-write tokens, model-call coverage, model cohorts, explicit failures, and separately reported orchestration aggregates.
@@ -73,7 +73,7 @@ Cost remains in `costQuality`, where observed, estimated, and uncaptured USD sta
 
 ---
 
-## `n` + `composite` + `perDimension` — distributional summary
+## `n` + `composite` + `perDimension`: distributional summary
 
 Always present. The basic "where are my numbers" view.
 
@@ -99,13 +99,13 @@ Always present. The basic "where are my numbers" view.
 
 **Read first:** the `composite.mean`. If it's < 0.5, your agent has a ceiling problem, not a tuning problem.
 
-**Read next:** `perDimension`. If `clarity` is high but `concision` is low, your prompts get the right ideas in too many words — different fix than "wrong ideas."
+**Read next:** `perDimension`. If `clarity` is high but `concision` is low, your prompts get the right ideas in too many words: different fix than "wrong ideas."
 
 **Use the histogram for:** finding bimodal failure modes. A bin with `count > 0` near zero and another > 0 near 1 means your agent has two distinct behaviors, not one noisy one.
 
 ---
 
-## `costQuality` — cost-vs-quality Pareto
+## `costQuality`: cost-vs-quality Pareto
 
 Always present. `cost.histogram` is the per-run cost distribution; `pareto` is the substrate's `ParetoFigureSpec`.
 
@@ -131,11 +131,11 @@ Always present. `cost.histogram` is the per-run cost distribution; `pareto` is t
 
 **Use this when:** comparing prompts, models, or candidate surfaces. The Pareto frontier is your menu of "best you can do at each cost level."
 
-**Render with:** any chart library — `points` is plain JSON. Hosted-tier dashboards render this as a scatter with the frontier highlighted.
+**Render with:** any chart library: `points` is plain JSON. Hosted-tier dashboards render this as a scatter with the frontier highlighted.
 
 ---
 
-## `judges` — per-judge mean
+## `judges`: per-judge mean
 
 Populated when run records carry `outcome.judgeScores`.
 
@@ -154,20 +154,23 @@ The substrate's full judge-calibration suite (positional bias, self-preference,
 
 ---
 
-## `interRater` — multi-rater agreement + disagreement triage
+## `interRater`: multi-rater agreement and disagreement review
 
-Populated when `analyzeRuns({ raterScores })` is supplied — typically via `fromFeedbackTable()`.
+Populated when `analyzeRuns({ raterScores })` is supplied: typically via `fromFeedbackTable()`.
 
 ```jsonc
 {
   "interRater": {
     "raters": 3,
     "jointlyRated": 30,
-    "kappa": 0.71,
+    "kappa": 0.40,
+    "icc": 0.42,
+    "pearson": 0.43,
+    "spearman": 0.41,
     "perPair": {
-      "alice::bob":   0.78,
-      "alice::carol": 0.65,
-      "bob::carol":   0.69
+      "alice::bob":   0.53,
+      "alice::carol": 0.47,
+      "bob::carol":   0.19
     },
     "disagreementCases": [
       { "runId": "claim-7", "range": 1.00,
@@ -180,13 +183,15 @@ Populated when `analyzeRuns({ raterScores })` is supplied — typically via `fro
 }
 ```
 
-**Read first:** the mean `kappa`. < 0.5 means raters disagree on what "good" looks like — surface the disagreement cases at the next review meeting.
+**Read first:** `kappa` and `icc`, which measure absolute agreement.
+Pearson and Spearman measure correlation and can remain high when raters use different score levels.
+When absolute agreement is low, review the largest disagreement cases before automating the rubric.
 
 **Use this when:** building per-rater LLM judges. Each rater's individual scores are the gold signal you calibrate against. Once a calibrated LLM matches the human ≥85%, you can auto-grade and escalate only the disagreement cases.
 
 ---
 
-## `lift` — paired-bootstrap statistical lift
+## `lift`: paired-bootstrap statistical lift
 
 Populated when baseline + candidate candidates are present (auto-detected from two distinct `candidateId`s, or explicit via `baselineCandidateId` + `candidateCandidateId`).
 
@@ -211,13 +216,13 @@ Populated when baseline + candidate candidates are present (auto-detected from t
 - `ci95[0] ≤ threshold < ci95[1]` → **INCONCLUSIVE.** Expand the corpus or wait for more data.
 - `ci95[1] ≤ threshold` → **HOLD.** No evidence the candidate is better.
 
-The `recommendations` array surfaces exactly this decision (`kind: 'ship' | 'hold' | 'expand-corpus'`) — that's what consumers should read.
+The `recommendations` array surfaces exactly this decision (`kind: 'ship' | 'hold' | 'expand-corpus'`): that's what consumers should read.
 
 **Why bootstrap, not t-test alone:** paired bootstrap is distribution-free. Your judge scores are bounded in [0,1] and almost never normal; the bootstrap CI is the honest one.
 
 ---
 
-## `failureClusters` — grouped failure modes
+## `failureClusters`: grouped failure modes
 
 Populated when an `AnalystRegistry` is passed via `analyzeRuns({ analyst })`. The substrate runs each failed run through the registered analysts and groups findings by `analyst_id` / `area`.
 
@@ -245,7 +250,7 @@ Populated when an `AnalystRegistry` is passed via `analyzeRuns({ analyst })`. Th
 
 ---
 
-## `contamination` — canary check
+## `contamination`: canary check
 
 Populated when canary scenarios are passed via `analyzeRuns({ canaryScenarios })`. Each canary carries a sentinel string the agent should never emit; the report counts leaks.
 
@@ -277,7 +282,7 @@ When `leaks > 0`:
 
 ---
 
-## `outcomeCorrelation` — closing the loop on real outcomes
+## `outcomeCorrelation`: closing the loop on real outcomes
 
 Populated when `outcomeSignal: { metric, valueByRunId }` is supplied.
 
@@ -307,9 +312,9 @@ This is the layer that says **"does my judge's taste actually predict the metric
 
 ---
 
-## `release` — pass/warn/fail axes
+## `release`: pass/warn/fail axes
 
-Always present. Roll-up across three axes — quality lift, contamination, composite distribution.
+Always present. Roll-up across three axes: quality lift, contamination, composite distribution.
 
 ```jsonc
 {
@@ -334,7 +339,7 @@ Overall `status` is `fail` if any axis fails; `warn` if any warn; `pass` otherwi
 
 ---
 
-## `recommendations` — the actionable layer
+## `recommendations`: the actionable layer
 
 Always present. Read this first.
 
@@ -342,7 +347,7 @@ Always present. Read this first.
 {
   "recommendations": [
     { "priority": "critical", "kind": "ship",
-      "title": "Ship — lift 0.070 (95% CI 0.040..0.100)",
+      "title": "Ship: lift 0.070 (95% CI 0.040..0.100)",
       "detail": "Holdout lift exceeds threshold 0.02 with 95% bootstrap confidence (n=40, p=0.0008, d=0.41).",
       "evidencePath": "lift" },
     { "priority": "high", "kind": "investigate",
@@ -357,7 +362,7 @@ Always present. Read this first.
 |---|---|
 | `ship` | lift CI lower bound > threshold |
 | `hold` | lift CI upper bound ≤ threshold |
-| `expand-corpus` | lift CI straddles threshold — more data needed |
+| `expand-corpus` | lift CI straddles threshold: more data needed |
 | `fix` | canary contamination detected |
 | `recalibrate` | inter-rater κ < 0.5, OR outcome correlation < 0.3 |
 | `investigate` | top failure cluster > some-share |
@@ -378,4 +383,4 @@ Always present. Read this first.
 | `contamination` | `canaryScenarios` passed in |
 | `outcomeCorrelation` | `outcomeSignal` passed in |
 
-All sections beyond the always-present ones are `T | undefined`, never empty objects. If a section is missing, your inputs didn't support it — the report is honest about that.
+All sections beyond the always-present ones are `T | undefined`, never empty objects. If a section is missing, your inputs didn't support it: the report is honest about that.

package/docs/multi-shot-optimization.md

@@ -1,75 +1,67 @@
-# Multi-Shot Optimization
+# Improve One Surface
 
-> **Renamed.** `runMultiShotOptimization` was retired. The live API is
-> `runImprovementLoop` (proposer-agnostic, gated promotion) driven by `gepaProposer`,
-> with `compareProposers` for head-to-head proposer lift. This doc was rewritten to the
-> live API; see also [feature-guide.md](./feature-guide.md) and [concepts.md](./concepts.md).
+`runImprovementLoop` generates candidate prompts or configs, runs them on training scenarios, and checks the selected candidate on separate holdout scenarios.
 
-`runImprovementLoop` is the public entry for GEPA-style optimization over a whole
-task trajectory — the thing you improve is not a single model call but an agent
-system prompt, tool descriptions, a routing policy, or any scaffolding that affects
-the entire run. It is the OUTER loop: it improves the SURFACE the inner workers run.
+Use it when the question is: "Can this method improve my current surface?"
+Use [`compareOptimizationMethods`](./campaign-proposers.md#compare-complete-methods) when the question is: "Which complete optimization method performs best?"
 
-## The shape
+## Inputs
 
-You own a few seams; the loop owns the release-critical glue (paired seeds, the
-held-out re-score, the promotion gate, provenance):
+| Input | Meaning |
+|---|---|
+| `baselineSurface` | Current prompt or config. |
+| `scenarios` | Training scenarios used to generate and score candidates. |
+| `holdoutScenarios` | Separate scenarios used by the release rule after search. |
+| `dispatchWithSurface` | Runs one scenario with one candidate surface. |
+| `judges` | Scores the returned artifact. |
+| `proposer` | Generates candidate surfaces. |
+| `gate` | Applies the caller's release rule to baseline and winner results. |
+| `runDir` | Stores run artifacts, traces, and resumable state. |
 
-- **`baselineSurface`** — the current surface (a prompt string, or a `CodeSurface`).
-- **`dispatchWithSurface(surface, scenario, ctx)`** — run one task to completion
-  under a candidate surface; return the artifact the judges score.
-- **`judges`** — score the artifact (`{ composite, dimensions }`).
-- **`proposer`** — proposes candidate surfaces each generation: `gepaProposer`
-  (reflective + Pareto frontier) or `evolutionaryProposer` (mutator).
-- **`gate`** — `defaultProductionGate` (held-out significance + red-team +
-  reward-hacking + canary). Ships ONLY on a CI-lower-bound held-out lift.
-
-## Minimal example
+## Example
 
 ```ts
 import {
-  runImprovementLoop,
-  gepaProposer,
   defaultProductionGate,
-} from '@tangle-network/agent-eval/contract'
+  gepaProposer,
+  runImprovementLoop,
+} from '@tangle-network/agent-eval/campaign'
 
 const result = await runImprovementLoop({
   baselineSurface: currentSystemPrompt,
-  scenarios: trainScenarios, // optimizer-visible
-  holdoutScenarios, // DISJOINT — only the gate sees these
-  dispatchWithSurface: async (surface, scenario) =>
-    runYourAgentToCompletion({ scenario, prompt: String(surface) }),
-  judges: [myJudge],
+  scenarios: trainScenarios,
+  holdoutScenarios,
+  dispatchWithSurface: async (surface, scenario, ctx) =>
+    runYourAgent({ prompt: String(surface), scenario, signal: ctx.signal }),
+  judges: [qualityJudge],
   proposer: gepaProposer({
     llm: { apiKey, baseUrl },
-    model: 'gpt-5',
-    target: 'enforce a strict output schema',
+    model,
+    target: 'the complete system prompt',
   }),
   populationSize: 4,
   maxGenerations: 4,
-  gate: defaultProductionGate({ holdoutScenarios, deltaThreshold: 0 }),
-  autoOnPromote: 'none', // or 'pr' (+ ghOwner/ghRepo) to open a PR on ship
+  gate: defaultProductionGate({
+    holdoutScenarios,
+    deltaThreshold: 0,
+  }),
+  autoOnPromote: 'none',
   runDir,
 })
 
 if (result.gateResult.decision === 'ship') {
-  deploy(result.winnerSurface) // the proposer's candidate, gated on a real held-out lift
+  deploy(result.winnerSurface)
 }
 ```
 
-## Discipline (what makes it trustworthy)
+## Behavior
 
-- **Holdout is disjoint + gated.** `holdoutScenarios` must not overlap the training
-  pool. The gate re-scores baseline vs winner on the holdout and ships only when the
-  paired-bootstrap CI lower bound clears `deltaThreshold`; a few-instance swing at
-  thin `n` is held (`few_runs`), not promoted.
-- **No-op never ships.** If no candidate beats the baseline, the winner IS the
-  baseline (empty diff) and the loop forces `hold` — it does not score
-  baseline-vs-itself and read model noise as lift.
-- **Provenance falls out.** `result.promotedDiff` + `emitLoopProvenance` give the
-  auditable candidate→gate→promote chain (rationale, content hashes, a held-out lift
-  recomputable from the emitted record).
+- Training and holdout scenario IDs must be disjoint.
+- Candidate generation cannot read holdout judge scores through `SurfaceProposer`.
+- The selected candidate is measured against the baseline on holdout scenarios.
+- A selected surface identical to the baseline is held instead of treating model variance as lift.
+- `result.cost` includes worker, candidate-generation, and judge calls recorded through the shared cost ledger.
+- `result.promotedDiff` describes the exact selected surface change.
 
-Reach for `compareProposers` when the question is "which proposer wins" rather than
-"improve this surface", and see `tests/campaign/presets.test.ts` for the executable
-contract (no-op guard, fail-loud holdout, gate promotion).
+The release decision is only as useful as the scenarios and judges supplied by the caller.
+Calibrate the judge on known strong and weak outputs before using it for promotion.

package/docs/research-report-methodology.md

@@ -1,4 +1,4 @@
-# researchReport — methodology
+# researchReport: methodology
 
 This document is the methodological brief for `researchReport` (exported from
 `@tangle-network/agent-eval` and `@tangle-network/agent-eval/reporting`). It
@@ -11,14 +11,14 @@ corresponding function or option so the doc and the code don't drift.
 
 ## Inputs
 
-- `runs: RunRecord[]` — every record carries `runId`, `candidateId`, `seed`,
+- `runs: RunRecord[]`: every record carries `runId`, `candidateId`, `seed`,
   `experimentId`, `splitTag`, and an `outcome` with the configured score.
-- `comparator: string` — the candidate id treated as the null reference. Must
+- `comparator: string`: the candidate id treated as the null reference. Must
   be selected before data inspection; `preregistrationHash` should pin this.
-- `split: 'search' | 'holdout'` — defaults to `holdout`. Decisions on `search`
+- `split: 'search' | 'holdout'`: defaults to `holdout`. Decisions on `search`
   are descriptive only; promotion calls require the holdout.
-- `rope: { low, high }` — Region of Practical Equivalence on the paired delta,
-  in score units. Must come from the domain owner — there is no
+- `rope: { low, high }`: Region of Practical Equivalence on the paired delta,
+  in score units. Must come from the domain owner: there is no
   statistically-defensible default.
 - `minPairs` (soft floor, default 20) and `RESEARCH_REPORT_HARD_PAIR_FLOOR`
   (hard floor, 6). Below the soft floor, the verdict is `needs_more_data` and
@@ -35,7 +35,7 @@ with non-finite scores are dropped before pairing.
 
 ## Decision rule
 
-In order — first match wins:
+In order: first match wins:
 
 1. `comparator` itself → `hold` (baseline).
 2. No comparator → `hold` if on the cost/quality Pareto frontier, else
@@ -46,7 +46,7 @@ In order — first match wins:
    "below hard floor" reason. Bootstrap CIs degenerate at this size.
 5. ROPE configured AND paired-delta CI ⊂ ROPE → `equivalent`.
 6. Paired-delta CI upper bound < 0 → `reject` (CI excludes a non-negative
-   effect). Note: this uses **paired delta only** — not the marginal mean.
+   effect). Note: this uses **paired delta only**: not the marginal mean.
 7. Paired N < `minPairs` (soft floor) → `needs_more_data` with the MDE at
    current N attached so the verdict is actionable.
 8. BH-adjusted q ≤ `fdr` AND CI lower bound > 0 → `promote`. The BH q-value
@@ -71,7 +71,7 @@ The Pr(Δ>0) and Pr(Δ∈ROPE) summaries use the bootstrap-prior duality of
 [Rubin 1981]: under a non-informative Dirichlet prior, the bootstrap
 distribution of a sample statistic is its posterior. We expose these as
 posterior summaries on the **mean** delta and the bootstrap CI on the
-**median** delta — the median is more robust to the heavy-tailed score
+**median** delta: the median is more robust to the heavy-tailed score
 distributions seen in agent benchmarks; the mean lets us read off the
 Bayesian-style probability of superiority in a single number.
 

package/docs/trace-analysis.md

@@ -51,7 +51,7 @@ The analyst runs one Ax executor loop and accepts only an explicit structured `f
 ## Deterministic failure coverage (no LLM)
 
 Before (or alongside) the LLM analyst, `OtlpFileTraceStore.getOverview()` returns a
-`DatasetOverview` whose `error_clusters` are computed deterministically — error
+`DatasetOverview` whose `error_clusters` are computed deterministically: error
 spans are grouped by a normalized failure signature (uuids / hex ids / numbers /
 absolute paths / durations collapsed), each cluster carrying its prevalence,
 exemplar `trace_id`/`span_id`, and a verbatim sample. This is a zero-LLM,
@@ -60,7 +60,7 @@ reproducible failure checklist the analyst then explains and closes:
 ```ts
 const overview = await store.getOverview()
 for (const c of overview.error_clusters) {
-  console.log(`${c.trace_count}× ${c.signature} — e.g. trace ${c.exemplar_trace_ids[0]}`)
+  console.log(`${c.trace_count}× ${c.signature}: e.g. trace ${c.exemplar_trace_ids[0]}`)
 }
 ```
 

package/docs/wire-protocol.md

@@ -29,9 +29,9 @@ Both transports talk to identical handlers. If you need a sustained connection (
 
 ## Methods
 
-The current surface is the smallest useful slice. Adding a method is mechanical — see [§Adding a method](#adding-a-method).
+The current surface is the smallest useful slice. Adding a method is mechanical: see [§Adding a method](#adding-a-method).
 
-### `judge` — score content against a rubric
+### `judge`: score content against a rubric
 
 ```http
 POST /v1/judge
@@ -63,7 +63,7 @@ Pass either `rubricName` (built-in) or `rubric` (inline definition). Not both. T
 
 `rubricVersion` is the stable hash of the rubric used. Scores are only comparable across runs when this matches.
 
-### `listRubrics` — discover what's registered
+### `listRubrics`: discover what's registered
 
 ```http
 GET /v1/rubrics
@@ -87,7 +87,7 @@ GET /v1/rubrics
 }
 ```
 
-### `version` — server + wire-protocol versions
+### `version`: server + wire-protocol versions
 
 ```http
 GET /v1/version
@@ -102,13 +102,13 @@ GET /v1/version
 }
 ```
 
-`version` matches the package version. `wireVersion` bumps independently — only on breaking request/response schema changes. Package versions can differ across releases as long as `wireVersion` matches.
+`version` matches the package version. `wireVersion` bumps independently: only on breaking request/response schema changes. Package versions can differ across releases as long as `wireVersion` matches.
 
-### `GET /healthz` — liveness
+### `GET /healthz`: liveness
 
 For probing whether a server is up. Returns `{ "status": "ok", "uptimeSec": <number> }`.
 
-### `GET /openapi.json` — full spec
+### `GET /openapi.json`: full spec
 
 Auto-generated from the Zod schemas. This is what code generators consume to produce typed clients in other languages.
 
@@ -168,11 +168,11 @@ echo '{}' | agent-eval rpc listRubrics
 # judge (one-shot)
 echo '{"rubricName":"anti-slop","content":"…"}' | agent-eval rpc judge
 
-# JSONL batch — one request per line
+# JSONL batch: one request per line
 cat requests.jsonl | agent-eval rpc-batch judge > results.jsonl
 ```
 
-Each invocation is one process — Node startup adds ~500 ms. For more than a few calls, stand up a server.
+Each invocation is one process: Node startup adds ~500 ms. For more than a few calls, stand up a server.
 
 ## Clients
 
@@ -182,13 +182,13 @@ Each invocation is one process — Node startup adds ~500 ms. For more than a fe
 
 ## Adding a method
 
-1. **Schema** — define `XRequestSchema` and `XResponseSchema` in `src/wire/schemas.ts`. Every field gets a `.describe()` so docs flow through to OpenAPI.
-2. **Handler** — pure function in `src/wire/handlers.ts`. Throws `WireError` for caller-fixable issues.
-3. **Server route** — `app.post('/v1/x', …)` in `src/wire/server.ts`.
-4. **RPC case** — add `case 'x':` in `dispatchRpc` in `src/wire/rpc.ts`.
-5. **OpenAPI route** — register in `src/wire/openapi.ts` so it shows up in the spec.
-6. **Test** — add to `tests/wire/`. At minimum: schema validation, happy-path, error-path.
-7. **Python client** — add a method on `Client` in `clients/python/src/agent_eval_rpc/client.py`, plus pydantic models in `models.py` mirroring the new schemas.
+1. **Schema**: define `XRequestSchema` and `XResponseSchema` in `src/wire/schemas.ts`. Every field gets a `.describe()` so docs flow through to OpenAPI.
+2. **Handler**: pure function in `src/wire/handlers.ts`. Throws `WireError` for caller-fixable issues.
+3. **Server route**: `app.post('/v1/x', …)` in `src/wire/server.ts`.
+4. **RPC case**: add `case 'x':` in `dispatchRpc` in `src/wire/rpc.ts`.
+5. **OpenAPI route**: register in `src/wire/openapi.ts` so it shows up in the spec.
+6. **Test**: add to `tests/wire/`. At minimum: schema validation, happy-path, error-path.
+7. **Python client**: add a method on `Client` in `clients/python/src/agent_eval_rpc/client.py`, plus pydantic models in `models.py` mirroring the new schemas.
 
 The pattern is mechanical. When the surface grows past ~10 methods, swap the hand-written Python models for `datamodel-code-generator -i openapi.json -o models.py`.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tangle-network/agent-eval",
-  "version": "0.123.1",
+  "version": "0.123.3",
   "description": "Evaluate and improve AI agents from runs, traces, judges, and feedback. Compare candidates, cluster failures, measure lift, and gate releases.",
   "homepage": "https://github.com/tangle-network/agent-eval#readme",
   "repository": {
@@ -142,6 +142,7 @@
     "test": "vitest run",
     "test:watch": "vitest",
     "typecheck": "tsc --noEmit",
+    "typecheck:examples": "tsc -p tsconfig.examples.json",
     "lint": "biome check src",
     "format": "biome format --write src",
     "openapi": "node dist/cli.js openapi --out dist/openapi.json",