@tangle-network/agent-eval 0.123.1 → 0.123.2

package/docs/insight-report.md

@@ -1,11 +1,11 @@
-# `InsightReport` — the decision packet
+# `InsightReport`: the report
 
 The single shape every analysis call returns. `selfImprove()` embeds it in `SelfImproveResult.insight`; `analyzeRuns()` returns it directly. The hosted-tier wire format carries it on `EvalRunEvent.insightReport?`.
 
 Use `summarizeExecution({ runs })` when observed traces have no task-quality labels.
 It returns only `execution` and `costProvenance`, so callers do not need to fabricate a quality score to report runtime facts.
 
-Every section is **opt-in based on what your data supports** — the function never invents signal. If your runs don't carry judge scores, `judges` is empty. If there's no baseline/candidate split, `lift` is undefined. The shape is consistent; population is honest.
+Every section is **opt-in based on what your data supports**: the function never invents signal. If your runs don't carry judge scores, `judges` is empty. If there's no baseline/candidate split, `lift` is undefined. The shape is consistent; population is honest.
 
 This page walks every section with a real (synthetic) example and explains how to act on it.
 
@@ -27,13 +27,13 @@ interface InsightReport {
   contamination?: ContaminationInsight   // when canaryScenarios supplied
   outcomeCorrelation?: OutcomeCorrelationInsight   // when outcomeSignal supplied
   release: ReleaseSummary                // always
-  recommendations: Recommendation[]      // always — read this FIRST
+  recommendations: Recommendation[]      // always: read this FIRST
 }
 ```
 
 ---
 
-## `execution` — runtime facts, separate from quality
+## `execution`: runtime facts, separate from quality
 
 Always present.
 It reports duration, optional queue time, direct input, output, reasoning, cache-read, and cache-write tokens, model-call coverage, model cohorts, explicit failures, and separately reported orchestration aggregates.
@@ -73,7 +73,7 @@ Cost remains in `costQuality`, where observed, estimated, and uncaptured USD sta
 
 ---
 
-## `n` + `composite` + `perDimension` — distributional summary
+## `n` + `composite` + `perDimension`: distributional summary
 
 Always present. The basic "where are my numbers" view.
 
@@ -99,13 +99,13 @@ Always present. The basic "where are my numbers" view.
 
 **Read first:** the `composite.mean`. If it's < 0.5, your agent has a ceiling problem, not a tuning problem.
 
-**Read next:** `perDimension`. If `clarity` is high but `concision` is low, your prompts get the right ideas in too many words — different fix than "wrong ideas."
+**Read next:** `perDimension`. If `clarity` is high but `concision` is low, your prompts get the right ideas in too many words: different fix than "wrong ideas."
 
 **Use the histogram for:** finding bimodal failure modes. A bin with `count > 0` near zero and another > 0 near 1 means your agent has two distinct behaviors, not one noisy one.
 
 ---
 
-## `costQuality` — cost-vs-quality Pareto
+## `costQuality`: cost-vs-quality Pareto
 
 Always present. `cost.histogram` is the per-run cost distribution; `pareto` is the substrate's `ParetoFigureSpec`.
 
@@ -131,11 +131,11 @@ Always present. `cost.histogram` is the per-run cost distribution; `pareto` is t
 
 **Use this when:** comparing prompts, models, or candidate surfaces. The Pareto frontier is your menu of "best you can do at each cost level."
 
-**Render with:** any chart library — `points` is plain JSON. Hosted-tier dashboards render this as a scatter with the frontier highlighted.
+**Render with:** any chart library: `points` is plain JSON. Hosted-tier dashboards render this as a scatter with the frontier highlighted.
 
 ---
 
-## `judges` — per-judge mean
+## `judges`: per-judge mean
 
 Populated when run records carry `outcome.judgeScores`.
 
@@ -154,20 +154,23 @@ The substrate's full judge-calibration suite (positional bias, self-preference,
 
 ---
 
-## `interRater` — multi-rater agreement + disagreement triage
+## `interRater`: multi-rater agreement and disagreement review
 
-Populated when `analyzeRuns({ raterScores })` is supplied — typically via `fromFeedbackTable()`.
+Populated when `analyzeRuns({ raterScores })` is supplied: typically via `fromFeedbackTable()`.
 
 ```jsonc
 {
   "interRater": {
     "raters": 3,
     "jointlyRated": 30,
-    "kappa": 0.71,
+    "kappa": 0.40,
+    "icc": 0.42,
+    "pearson": 0.43,
+    "spearman": 0.41,
     "perPair": {
-      "alice::bob":   0.78,
-      "alice::carol": 0.65,
-      "bob::carol":   0.69
+      "alice::bob":   0.53,
+      "alice::carol": 0.47,
+      "bob::carol":   0.19
     },
     "disagreementCases": [
       { "runId": "claim-7", "range": 1.00,
@@ -180,13 +183,15 @@ Populated when `analyzeRuns({ raterScores })` is supplied — typically via `fro
 }
 ```
 
-**Read first:** the mean `kappa`. < 0.5 means raters disagree on what "good" looks like — surface the disagreement cases at the next review meeting.
+**Read first:** `kappa` and `icc`, which measure absolute agreement.
+Pearson and Spearman measure correlation and can remain high when raters use different score levels.
+When absolute agreement is low, review the largest disagreement cases before automating the rubric.
 
 **Use this when:** building per-rater LLM judges. Each rater's individual scores are the gold signal you calibrate against. Once a calibrated LLM matches the human ≥85%, you can auto-grade and escalate only the disagreement cases.
 
 ---
 
-## `lift` — paired-bootstrap statistical lift
+## `lift`: paired-bootstrap statistical lift
 
 Populated when baseline + candidate candidates are present (auto-detected from two distinct `candidateId`s, or explicit via `baselineCandidateId` + `candidateCandidateId`).
 
@@ -211,13 +216,13 @@ Populated when baseline + candidate candidates are present (auto-detected from t
 - `ci95[0] ≤ threshold < ci95[1]` → **INCONCLUSIVE.** Expand the corpus or wait for more data.
 - `ci95[1] ≤ threshold` → **HOLD.** No evidence the candidate is better.
 
-The `recommendations` array surfaces exactly this decision (`kind: 'ship' | 'hold' | 'expand-corpus'`) — that's what consumers should read.
+The `recommendations` array surfaces exactly this decision (`kind: 'ship' | 'hold' | 'expand-corpus'`): that's what consumers should read.
 
 **Why bootstrap, not t-test alone:** paired bootstrap is distribution-free. Your judge scores are bounded in [0,1] and almost never normal; the bootstrap CI is the honest one.
 
 ---
 
-## `failureClusters` — grouped failure modes
+## `failureClusters`: grouped failure modes
 
 Populated when an `AnalystRegistry` is passed via `analyzeRuns({ analyst })`. The substrate runs each failed run through the registered analysts and groups findings by `analyst_id` / `area`.
 
@@ -245,7 +250,7 @@ Populated when an `AnalystRegistry` is passed via `analyzeRuns({ analyst })`. Th
 
 ---
 
-## `contamination` — canary check
+## `contamination`: canary check
 
 Populated when canary scenarios are passed via `analyzeRuns({ canaryScenarios })`. Each canary carries a sentinel string the agent should never emit; the report counts leaks.
 
@@ -277,7 +282,7 @@ When `leaks > 0`:
 
 ---
 
-## `outcomeCorrelation` — closing the loop on real outcomes
+## `outcomeCorrelation`: closing the loop on real outcomes
 
 Populated when `outcomeSignal: { metric, valueByRunId }` is supplied.
 
@@ -307,9 +312,9 @@ This is the layer that says **"does my judge's taste actually predict the metric
 
 ---
 
-## `release` — pass/warn/fail axes
+## `release`: pass/warn/fail axes
 
-Always present. Roll-up across three axes — quality lift, contamination, composite distribution.
+Always present. Roll-up across three axes: quality lift, contamination, composite distribution.
 
 ```jsonc
 {
@@ -334,7 +339,7 @@ Overall `status` is `fail` if any axis fails; `warn` if any warn; `pass` otherwi
 
 ---
 
-## `recommendations` — the actionable layer
+## `recommendations`: the actionable layer
 
 Always present. Read this first.
 
@@ -342,7 +347,7 @@ Always present. Read this first.
 {
   "recommendations": [
     { "priority": "critical", "kind": "ship",
-      "title": "Ship — lift 0.070 (95% CI 0.040..0.100)",
+      "title": "Ship: lift 0.070 (95% CI 0.040..0.100)",
       "detail": "Holdout lift exceeds threshold 0.02 with 95% bootstrap confidence (n=40, p=0.0008, d=0.41).",
       "evidencePath": "lift" },
     { "priority": "high", "kind": "investigate",
@@ -357,7 +362,7 @@ Always present. Read this first.
 |---|---|
 | `ship` | lift CI lower bound > threshold |
 | `hold` | lift CI upper bound ≤ threshold |
-| `expand-corpus` | lift CI straddles threshold — more data needed |
+| `expand-corpus` | lift CI straddles threshold: more data needed |
 | `fix` | canary contamination detected |
 | `recalibrate` | inter-rater κ < 0.5, OR outcome correlation < 0.3 |
 | `investigate` | top failure cluster > some-share |
@@ -378,4 +383,4 @@ Always present. Read this first.
 | `contamination` | `canaryScenarios` passed in |
 | `outcomeCorrelation` | `outcomeSignal` passed in |
 
-All sections beyond the always-present ones are `T | undefined`, never empty objects. If a section is missing, your inputs didn't support it — the report is honest about that.
+All sections beyond the always-present ones are `T | undefined`, never empty objects. If a section is missing, your inputs didn't support it: the report is honest about that.

package/docs/multi-shot-optimization.md

@@ -6,7 +6,7 @@
 > live API; see also [feature-guide.md](./feature-guide.md) and [concepts.md](./concepts.md).
 
 `runImprovementLoop` is the public entry for GEPA-style optimization over a whole
-task trajectory — the thing you improve is not a single model call but an agent
+task trajectory: the thing you improve is not a single model call but an agent
 system prompt, tool descriptions, a routing policy, or any scaffolding that affects
 the entire run. It is the OUTER loop: it improves the SURFACE the inner workers run.
 
@@ -15,13 +15,13 @@ the entire run. It is the OUTER loop: it improves the SURFACE the inner workers
 You own a few seams; the loop owns the release-critical glue (paired seeds, the
 held-out re-score, the promotion gate, provenance):
 
-- **`baselineSurface`** — the current surface (a prompt string, or a `CodeSurface`).
-- **`dispatchWithSurface(surface, scenario, ctx)`** — run one task to completion
+- **`baselineSurface`**: the current surface (a prompt string, or a `CodeSurface`).
+- **`dispatchWithSurface(surface, scenario, ctx)`**: run one task to completion
   under a candidate surface; return the artifact the judges score.
-- **`judges`** — score the artifact (`{ composite, dimensions }`).
-- **`proposer`** — proposes candidate surfaces each generation: `gepaProposer`
+- **`judges`**: score the artifact (`{ composite, dimensions }`).
+- **`proposer`**: proposes candidate surfaces each generation: `gepaProposer`
   (reflective + Pareto frontier) or `evolutionaryProposer` (mutator).
-- **`gate`** — `defaultProductionGate` (held-out significance + red-team +
+- **`gate`**: `defaultProductionGate` (held-out significance + red-team +
   reward-hacking + canary). Ships ONLY on a CI-lower-bound held-out lift.
 
 ## Minimal example
@@ -36,7 +36,7 @@ import {
 const result = await runImprovementLoop({
   baselineSurface: currentSystemPrompt,
   scenarios: trainScenarios, // optimizer-visible
-  holdoutScenarios, // DISJOINT — only the gate sees these
+  holdoutScenarios, // DISJOINT: only the gate sees these
   dispatchWithSurface: async (surface, scenario) =>
     runYourAgentToCompletion({ scenario, prompt: String(surface) }),
   judges: [myJudge],
@@ -64,7 +64,7 @@ if (result.gateResult.decision === 'ship') {
   paired-bootstrap CI lower bound clears `deltaThreshold`; a few-instance swing at
   thin `n` is held (`few_runs`), not promoted.
 - **No-op never ships.** If no candidate beats the baseline, the winner IS the
-  baseline (empty diff) and the loop forces `hold` — it does not score
+  baseline (empty diff) and the loop forces `hold`: it does not score
   baseline-vs-itself and read model noise as lift.
 - **Provenance falls out.** `result.promotedDiff` + `emitLoopProvenance` give the
   auditable candidate→gate→promote chain (rationale, content hashes, a held-out lift

package/docs/research-report-methodology.md

@@ -1,4 +1,4 @@
-# researchReport — methodology
+# researchReport: methodology
 
 This document is the methodological brief for `researchReport` (exported from
 `@tangle-network/agent-eval` and `@tangle-network/agent-eval/reporting`). It
@@ -11,14 +11,14 @@ corresponding function or option so the doc and the code don't drift.
 
 ## Inputs
 
-- `runs: RunRecord[]` — every record carries `runId`, `candidateId`, `seed`,
+- `runs: RunRecord[]`: every record carries `runId`, `candidateId`, `seed`,
   `experimentId`, `splitTag`, and an `outcome` with the configured score.
-- `comparator: string` — the candidate id treated as the null reference. Must
+- `comparator: string`: the candidate id treated as the null reference. Must
   be selected before data inspection; `preregistrationHash` should pin this.
-- `split: 'search' | 'holdout'` — defaults to `holdout`. Decisions on `search`
+- `split: 'search' | 'holdout'`: defaults to `holdout`. Decisions on `search`
   are descriptive only; promotion calls require the holdout.
-- `rope: { low, high }` — Region of Practical Equivalence on the paired delta,
-  in score units. Must come from the domain owner — there is no
+- `rope: { low, high }`: Region of Practical Equivalence on the paired delta,
+  in score units. Must come from the domain owner: there is no
   statistically-defensible default.
 - `minPairs` (soft floor, default 20) and `RESEARCH_REPORT_HARD_PAIR_FLOOR`
   (hard floor, 6). Below the soft floor, the verdict is `needs_more_data` and
@@ -35,7 +35,7 @@ with non-finite scores are dropped before pairing.
 
 ## Decision rule
 
-In order — first match wins:
+In order: first match wins:
 
 1. `comparator` itself → `hold` (baseline).
 2. No comparator → `hold` if on the cost/quality Pareto frontier, else
@@ -46,7 +46,7 @@ In order — first match wins:
    "below hard floor" reason. Bootstrap CIs degenerate at this size.
 5. ROPE configured AND paired-delta CI ⊂ ROPE → `equivalent`.
 6. Paired-delta CI upper bound < 0 → `reject` (CI excludes a non-negative
-   effect). Note: this uses **paired delta only** — not the marginal mean.
+   effect). Note: this uses **paired delta only**: not the marginal mean.
 7. Paired N < `minPairs` (soft floor) → `needs_more_data` with the MDE at
    current N attached so the verdict is actionable.
 8. BH-adjusted q ≤ `fdr` AND CI lower bound > 0 → `promote`. The BH q-value
@@ -71,7 +71,7 @@ The Pr(Δ>0) and Pr(Δ∈ROPE) summaries use the bootstrap-prior duality of
 [Rubin 1981]: under a non-informative Dirichlet prior, the bootstrap
 distribution of a sample statistic is its posterior. We expose these as
 posterior summaries on the **mean** delta and the bootstrap CI on the
-**median** delta — the median is more robust to the heavy-tailed score
+**median** delta: the median is more robust to the heavy-tailed score
 distributions seen in agent benchmarks; the mean lets us read off the
 Bayesian-style probability of superiority in a single number.
 

package/docs/self-improvement-map.md

@@ -1,7 +1,7 @@
 # The self-improvement map
 
 One loop. Four roles. A proposer catalog of pluggable strategies. A bench rig that
-proves the loop produces real lift. Nothing here is duplicated — it is one engine
+proves the loop produces real lift. Nothing here is duplicated: it is one engine
 pointed at different surfaces. This map exists because the surface count makes it
 *look* like many competing systems when it is one.
 
@@ -26,11 +26,11 @@ product imports the same function. Each generation it does four things:
                               repeat N generations
 ```
 
-## The four roles — keep them separate and the confusion clears
+## The four roles: keep them separate and the confusion clears
 
 | Role | What it is | Plain meaning |
 |---|---|---|
-| **Surface** | a *string* — an agent directive, a `SKILL.md`, a playbook, a memory, a judge rubric | **what** gets improved |
+| **Surface** | a *string*: an agent directive, a `SKILL.md`, a playbook, a memory, a judge rubric | **what** gets improved |
 | **Proposer** | a `SurfaceProposer` (the catalog below) | **how** candidate surfaces are proposed |
 | **Gate** | held-out split + significance (`paretoSignificanceGate` / `heldOutGate` / `defaultProductionGate`) | **did it actually get better**, vs noise |
 | **Judge** | scores a run | **how good** any version is |
@@ -44,7 +44,7 @@ surface; **bench-only proposers** exist solely to be raced inside
 
 | Proposer factory | Surface | Strategy | Role | Notes |
 |---|---|---|---|---|
-| `gepaProposer` | prompt | reflective full-surface rewrite + Pareto frontier | **production default** | consumes trace-analysis findings — see below |
+| `gepaProposer` | prompt | reflective full-surface rewrite + Pareto frontier | **production default** | consumes trace-analysis findings: see below |
 | `fapoProposer` | prompt/config/code | reviewed escalation policy over prompt → parameter → structural proposers | production, benchmark | encodes FAPO's scope + reviewer + prompt-first escalation rules; structural generator is injected |
 | `parameterSweepProposer` | config | JSON config patch/sweep | production, benchmark | middle FAPO level for parameter/config edits such as `retrieval.k`, `temperature`, `max_tokens` |
 | `skillOptProposer` | skill-doc | anchored add/delete/replace patch | production | preserves earlier rules; edit budget = "textual learning rate" |
@@ -58,29 +58,29 @@ Default choice: start with `gepaProposer` for prompt surfaces, add
 `parameterSweepProposer` when config knobs are the likely failure mode, and wrap
 them with `fapoProposer` when evidence should decide when to escalate.
 
-## Trace analysis — what it is and the three places it is used
+## Trace analysis: what it is and the three places it is used
 
 "Trace analysis" is the **evidence layer**: it turns raw OTLP traces into "here is
 exactly *why* the agent failed" (failure clusters → findings). The engine is
 `analyzeRuns()` + the analyst registry (`src/contract/analyze-runs.ts`). It is used in
-three places — this is the answer to "if GEPA does its own thing, what is trace
+three places: this is the answer to "if GEPA does its own thing, what is trace
 analysis *for*?":
 
-1. **Ships to customers** — `analyzeRuns()` → `InsightReport`, the Intelligence product.
-2. **Feeds the proposer** — `gepaProposer` calls `renderAnalystEvidence(ctx.findings,
+1. **Ships to customers**: `analyzeRuns()` → `InsightReport`, the Intelligence product.
+2. **Feeds the proposer**: `gepaProposer` calls `renderAnalystEvidence(ctx.findings,
    ctx.report)` (`src/campaign/proposers/gepa.ts`). GEPA's rewrites are grounded
    in the diagnosis instead of guessing blind. Trace analysis
    **is** on the GEPA side.
-3. **Races HALO** — wrapped as `traceAnalystProposer` so our analysis competes
+3. **Races HALO**: wrapped as `traceAnalystProposer` so our analysis competes
    head-to-head with the external SOTA inside `compareProposers`.
 
 ## Where HALO fits (and why it feels "removed")
 
 `haloProposer` is alive (`src/campaign/proposers/halo.ts`, exported from the campaign
 barrel) but it is **never in the product loop**. It shells out to an *external* engine
-(`halo-engine`) — so the analysis genuinely lives outside this repo; we only wrap it.
+(`halo-engine`): so the analysis genuinely lives outside this repo; we only wrap it.
 
-Its only job is the **bake-off**. HALO's real opponent is **not** `gepaProposer` — it is
+Its only job is the **bake-off**. HALO's real opponent is **not** `gepaProposer`: it is
 `traceAnalystProposer`. `compareProposers` holds the apply step identical (same
 `APPLY_SYSTEM`, same `traces.jsonl`, same held-out scoring) so the only variable is
 **analysis quality: HALO vs ours.** A measuring stick, like a benchmark baseline.
@@ -90,7 +90,7 @@ Its only job is the **bake-off**. HALO's real opponent is **not** `gepaProposer`
 `agent-runtime/bench/src/gepa-refine.ts` runs **this same loop** against a *public
 benchmark* (AppWorld, CAD, …) instead of product data. Why a separate rig:
 
-- On product traces, "+4 lift" can be model noise or a judge flattering itself — no
+- On product traces, "+4 lift" can be model noise or a judge flattering itself: no
   ground truth.
 - On a benchmark the score is **objective and ungameable** (AppWorld runs the agent's
   code against its own unit tests). If a GEPA-optimized directive beats a deliberately
@@ -110,6 +110,6 @@ The code is well-factored; the confusion is narrative:
 - **The real gap is the missing proof, not the design.** The loop kept being proved on
   benchmarks too easy to show value: when a capable model ceilings an extraction task,
   **0 findings fire** and the whole trace-analysis→optimizer apparatus is inert. It
-  earns its keep only on **hard agentic tasks** — which is why the AppWorld REPL run
+  earns its keep only on **hard agentic tasks**: which is why the AppWorld REPL run
   (multi-turn, real tool execution, unbounded turns) is the one that can finally
   separate the evidence-grounded optimizer from baseline.

package/docs/trace-analysis.md

@@ -51,7 +51,7 @@ The analyst runs one Ax executor loop and accepts only an explicit structured `f
 ## Deterministic failure coverage (no LLM)
 
 Before (or alongside) the LLM analyst, `OtlpFileTraceStore.getOverview()` returns a
-`DatasetOverview` whose `error_clusters` are computed deterministically — error
+`DatasetOverview` whose `error_clusters` are computed deterministically: error
 spans are grouped by a normalized failure signature (uuids / hex ids / numbers /
 absolute paths / durations collapsed), each cluster carrying its prevalence,
 exemplar `trace_id`/`span_id`, and a verbatim sample. This is a zero-LLM,
@@ -60,7 +60,7 @@ reproducible failure checklist the analyst then explains and closes:
 ```ts
 const overview = await store.getOverview()
 for (const c of overview.error_clusters) {
-  console.log(`${c.trace_count}× ${c.signature} — e.g. trace ${c.exemplar_trace_ids[0]}`)
+  console.log(`${c.trace_count}× ${c.signature}: e.g. trace ${c.exemplar_trace_ids[0]}`)
 }
 ```
 

package/docs/wire-protocol.md

@@ -29,9 +29,9 @@ Both transports talk to identical handlers. If you need a sustained connection (
 
 ## Methods
 
-The current surface is the smallest useful slice. Adding a method is mechanical — see [§Adding a method](#adding-a-method).
+The current surface is the smallest useful slice. Adding a method is mechanical: see [§Adding a method](#adding-a-method).
 
-### `judge` — score content against a rubric
+### `judge`: score content against a rubric
 
 ```http
 POST /v1/judge
@@ -63,7 +63,7 @@ Pass either `rubricName` (built-in) or `rubric` (inline definition). Not both. T
 
 `rubricVersion` is the stable hash of the rubric used. Scores are only comparable across runs when this matches.
 
-### `listRubrics` — discover what's registered
+### `listRubrics`: discover what's registered
 
 ```http
 GET /v1/rubrics
@@ -87,7 +87,7 @@ GET /v1/rubrics
 }
 ```
 
-### `version` — server + wire-protocol versions
+### `version`: server + wire-protocol versions
 
 ```http
 GET /v1/version
@@ -102,13 +102,13 @@ GET /v1/version
 }
 ```
 
-`version` matches the package version. `wireVersion` bumps independently — only on breaking request/response schema changes. Package versions can differ across releases as long as `wireVersion` matches.
+`version` matches the package version. `wireVersion` bumps independently: only on breaking request/response schema changes. Package versions can differ across releases as long as `wireVersion` matches.
 
-### `GET /healthz` — liveness
+### `GET /healthz`: liveness
 
 For probing whether a server is up. Returns `{ "status": "ok", "uptimeSec": <number> }`.
 
-### `GET /openapi.json` — full spec
+### `GET /openapi.json`: full spec
 
 Auto-generated from the Zod schemas. This is what code generators consume to produce typed clients in other languages.
 
@@ -168,11 +168,11 @@ echo '{}' | agent-eval rpc listRubrics
 # judge (one-shot)
 echo '{"rubricName":"anti-slop","content":"…"}' | agent-eval rpc judge
 
-# JSONL batch — one request per line
+# JSONL batch: one request per line
 cat requests.jsonl | agent-eval rpc-batch judge > results.jsonl
 ```
 
-Each invocation is one process — Node startup adds ~500 ms. For more than a few calls, stand up a server.
+Each invocation is one process: Node startup adds ~500 ms. For more than a few calls, stand up a server.
 
 ## Clients
 
@@ -182,13 +182,13 @@ Each invocation is one process — Node startup adds ~500 ms. For more than a fe
 
 ## Adding a method
 
-1. **Schema** — define `XRequestSchema` and `XResponseSchema` in `src/wire/schemas.ts`. Every field gets a `.describe()` so docs flow through to OpenAPI.
-2. **Handler** — pure function in `src/wire/handlers.ts`. Throws `WireError` for caller-fixable issues.
-3. **Server route** — `app.post('/v1/x', …)` in `src/wire/server.ts`.
-4. **RPC case** — add `case 'x':` in `dispatchRpc` in `src/wire/rpc.ts`.
-5. **OpenAPI route** — register in `src/wire/openapi.ts` so it shows up in the spec.
-6. **Test** — add to `tests/wire/`. At minimum: schema validation, happy-path, error-path.
-7. **Python client** — add a method on `Client` in `clients/python/src/agent_eval_rpc/client.py`, plus pydantic models in `models.py` mirroring the new schemas.
+1. **Schema**: define `XRequestSchema` and `XResponseSchema` in `src/wire/schemas.ts`. Every field gets a `.describe()` so docs flow through to OpenAPI.
+2. **Handler**: pure function in `src/wire/handlers.ts`. Throws `WireError` for caller-fixable issues.
+3. **Server route**: `app.post('/v1/x', …)` in `src/wire/server.ts`.
+4. **RPC case**: add `case 'x':` in `dispatchRpc` in `src/wire/rpc.ts`.
+5. **OpenAPI route**: register in `src/wire/openapi.ts` so it shows up in the spec.
+6. **Test**: add to `tests/wire/`. At minimum: schema validation, happy-path, error-path.
+7. **Python client**: add a method on `Client` in `clients/python/src/agent_eval_rpc/client.py`, plus pydantic models in `models.py` mirroring the new schemas.
 
 The pattern is mechanical. When the surface grows past ~10 methods, swap the hand-written Python models for `datamodel-code-generator -i openapi.json -o models.py`.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tangle-network/agent-eval",
-  "version": "0.123.1",
+  "version": "0.123.2",
   "description": "Evaluate and improve AI agents from runs, traces, judges, and feedback. Compare candidates, cluster failures, measure lift, and gate releases.",
   "homepage": "https://github.com/tangle-network/agent-eval#readme",
   "repository": {
@@ -142,6 +142,7 @@
     "test": "vitest run",
     "test:watch": "vitest",
     "typecheck": "tsc --noEmit",
+    "typecheck:examples": "tsc -p tsconfig.examples.json",
     "lint": "biome check src",
     "format": "biome format --write src",
     "openapi": "node dist/cli.js openapi --out dist/openapi.json",