@tangle-network/agent-eval 0.123.1 → 0.123.2

package/docs/building-doctrine.md

@@ -4,7 +4,7 @@ How every fleet agent that consumes `agent-eval` is built. Each rule is mechanic
 
 ## 1. Defaults must be provably reachable
 
-Every hard-coded model id or endpoint default is verifiable against the live router. Membership in `{baseUrl}/models` is the free check; an optional 1-token probe per model confirms the router will actually serve it. A default the router cannot serve is a config bug caught before the run, not a runtime surprise that silently degrades into a stub. Backend ids are namespaced by binding: cli-bridge ids (`claude-code/*`, `kimi-code/*`, `opencode/*`) never appear as defaults in code reachable from production — bridge use is an explicit env opt-in, never an implicit fallback.
+Every hard-coded model id or endpoint default is verifiable against the live router. Membership in `{baseUrl}/models` is the free check; an optional 1-token probe per model confirms the router will actually serve it. A default the router cannot serve is a config bug caught before the run, not a runtime surprise that silently degrades into a stub. Backend ids are namespaced by binding: cli-bridge ids (`claude-code/*`, `kimi-code/*`, `opencode/*`) never appear as defaults in code reachable from production: bridge use is an explicit env opt-in, never an implicit fallback.
 
 Enforced by: `preflightModels` (membership + optional probe) and `assertModelsServed` (gate that names every unreachable id with status + detail).
 
@@ -12,13 +12,13 @@ Enforced by: `preflightModels` (membership + optional probe) and `assertModelsSe
 
 When a request fails, one direct call against the live endpoint bisects platform-versus-client before any code-level debugging begins. A 401 from the router on a `model_not_found` is the platform telling you the default is dead; a connection refused is the platform being unreachable. Establish which side is at fault with a probe first, then debug only the side that is actually broken.
 
-Enforced by: `preflightModels({ probe: true })` — the probe is the platform-side bisection, carrying the router's own `error.message` back to the caller.
+Enforced by: `preflightModels({ probe: true })`: the probe is the platform-side bisection, carrying the router's own `error.message` back to the caller.
 
 ## 3. Agent-produced findings are hypotheses
 
 Enumeration of candidate problems may fan out to agents, but agent output is not evidence. Truth comes from probes against ground truth, not from an agent's assertion. Every classification carries quoted evidence, and nothing unverified is merged or reported as fact. A confident-sounding agent claim with no probe behind it is a hypothesis awaiting falsification.
 
-Enforced by: `assertRealBackend` over the resulting `RunRecord[]` — an agent that claims success while the backend was never called reads as a stub, not a pass.
+Enforced by: `assertRealBackend` over the resulting `RunRecord[]`: an agent that claims success while the backend was never called reads as a stub, not a pass.
 
 ## 4. Experiment integrity checklist
 
@@ -30,7 +30,7 @@ Any lift or benchmark claim satisfies all of the following before it is reported
 - The same scorer and the same items on both sides of any comparison.
 - A leakage check from builder inputs into the evaluation set.
 - Cross-family judge panels, with inter-rater reliability reported and gated.
-- Missing evidence is never scored as zero — fail loud over fabricate.
+- Missing evidence is never scored as zero: fail loud over fabricate.
 - No optional stopping: the stopping rule is fixed before the run.
 
 Enforced by: `pairedBootstrap` (CI), `assertCrossFamily` (panel diversity), `interRaterReliability` (agreement), and `assertRealBackend` (no stub run masquerading as a result).
@@ -39,4 +39,4 @@ Enforced by: `pairedBootstrap` (CI), `assertCrossFamily` (panel diversity), `int
 
 A drifted default is the symptom of a missing convention. The fix ships the convention and its guard alongside the one-line correction, so the same drift cannot recur silently. Patching the single dead id without adding the preflight gate leaves the class open; the next default rots the same way.
 
-Enforced by: `assertModelsServed` wired into the campaign preflight — the guard that turns "this one model was dead" into "no campaign spends tokens against an unreachable default."
+Enforced by: `assertModelsServed` wired into the campaign preflight: the guard that turns "this one model was dead" into "no campaign spends tokens against an unreachable default."

package/docs/concepts.md

@@ -16,17 +16,17 @@ can; drop to the raw functions when you need lower-level control.
 
 | Function | When to call it | What you give it | What you get back |
 |---|---|---|---|
-| **`defineAgentEval()`** | You have scenarios, an agent, a judge, and a baseline surface, and you want one object you can score or improve. | scenarios, agent, judge, baseline surface | `{ evaluate(), improve() }` where `evaluate()` returns a campaign result and `improve()` returns a decision packet |
-| **`selfImprove()`** | You have a closed loop — scenarios, judge, agent in hand, and you want the substrate to propose better candidates + gate them. | scenarios, agent, judge, baseline surface | `SelfImproveResult.insight: InsightReport` + ship/hold verdict + winner surface |
+| **`defineAgentEval()`** | You have scenarios, an agent, a judge, and a baseline surface, and you want one object you can score or improve. | scenarios, agent, judge, baseline surface | `{ evaluate(), improve() }` where `evaluate()` returns a campaign result and `improve()` returns a report |
+| **`selfImprove()`** | You have a closed loop: scenarios, judge, agent in hand, and you want the substrate to propose better candidates + gate them. | scenarios, agent, judge, baseline surface | `SelfImproveResult.insight: InsightReport` + ship/hold verdict + winner surface |
 | **`loadEvalFixtureScenarios()`** | You want agents to add evals as folders with `PROMPT.md`, checks, and starter files. | `evals/<name>/PROMPT.md + EVAL.ts + package.json` | `Scenario[]` that runs through `runCampaign`; pair with `planEvalFixtureRun()` before spending tokens |
 | **`analyzeRuns()`** | You have observed runs (production traces, an approve/reject corpus, a CSV gold set) and want the same rigor packet without invoking an agent. | `RunRecord[]` + optional flags | `InsightReport` |
-| **Intake adapters** (`fromFeedbackTable`, `fromOtelSpans`) | Your data isn't already in `RunRecord` shape — it's in Obsidian, Sheets, an OTel collector, etc. | source-specific input | `RunRecord[]` ready to pipe into `analyzeRuns()` |
+| **Intake adapters** (`fromFeedbackTable`, `fromOtelSpans`) | Your data isn't already in `RunRecord` shape: it's in Obsidian, Sheets, an OTel collector, etc. | source-specific input | `RunRecord[]` ready to pipe into `analyzeRuns()` |
 
-The customer maturity stages — logs only → ratings → closed loop — map to these
+The customer maturity stages: logs only → ratings → closed loop: map to these
 entry points. See [`customer-journeys.md`](./customer-journeys.md) for the
 runnable walkthroughs.
 
-The shape of the answer — `InsightReport` — is identical across all three paths. Distributional summary, paired-bootstrap lift CI, judge stats, inter-rater agreement, cost-quality Pareto, failure clusters, contamination check, outcome correlation, release axes, and a ranked recommendations array. Walked through section-by-section in [`insight-report.md`](./insight-report.md).
+The shape of the answer: `InsightReport`: is identical across all three paths. Distributional summary, paired-bootstrap lift CI, judge stats, inter-rater agreement, cost-quality Pareto, failure clusters, contamination check, outcome correlation, release axes, and a ranked recommendations array. Walked through section-by-section in [`insight-report.md`](./insight-report.md).
 
 ## The layering rule
 
@@ -39,7 +39,7 @@ The test: *does this concept make sense WITHOUT a running agent loop?* If yes, i
 | Thing | What it is | One-line example |
 |---|---|---|
 | **Judge** | A function that scores one piece of output. | "Did this scaffold implement async fetching?" |
-| **Rubric** | The recipe a judge uses — what to score on, with what weights. | "Score on buyer_quality (0.5), voice (0.3), signal (0.2)." |
+| **Rubric** | The recipe a judge uses: what to score on, with what weights. | "Score on buyer_quality (0.5), voice (0.3), signal (0.2)." |
 | **Verifier** | A pipeline of judges run in order, with dependencies. | "install → typecheck → build → semantic" |
 | **Feedback trajectory** | A multi-shot record of attempts, approvals, rejections, edits, metrics, and policy outcomes. | "draft → user rejects → revised draft → approved → measured" |
 
@@ -66,7 +66,7 @@ that can seed memory, replay scenarios, and optimization.
 | **Harness** | A description of *how to run* the artifact: setup command, test command, working dir, timeout. |
 | **Sandbox driver** | The thing that actually executes commands inside the harness. Local subprocess, or remote container. |
 | **Layer** | One stage of a verifier pipeline (install, typecheck, build, semantic, …). |
-| **Finding** | A specific issue a judge found — file, line, severity, message. |
+| **Finding** | A specific issue a judge found: file, line, severity, message. |
 | **Trace store** | The append-only log of every span/event during a run. Replay = read this back. |
 | **Composite score** | A 0..1 number combining all dimensions. The single number you gate on. |
 | **Rubric version** | A stable hash of the rubric. Scores from different rubric versions are not comparable. |
@@ -103,30 +103,30 @@ L1  app-build      Does the artifact build / typecheck / test?
                               │
                               ▼
 L2  app-runtime    Does the artifact actually run end-to-end?
-                   (Dynamic signal — only worth checking if L1 passed.)
+                   (Dynamic signal: only worth checking if L1 passed.)
 ```
 
 `BuilderSession` orchestrates this. It opens at `startChat`, runs the build at `ship`, runs the runtime check at `runAppScenario`. Each layer emits a trace span. Composite score aggregates them with `scoreProject`.
 
 Why three? Because each catches a different failure mode:
-- L0 misses — agent crashed mid-generation, you have a half-written file.
-- L1 misses — files exist but typecheck fails. LLM judges can't reliably catch this.
-- L2 misses — code compiles but does the wrong thing at runtime.
+- L0 misses: agent crashed mid-generation, you have a half-written file.
+- L1 misses: files exist but typecheck fails. LLM judges can't reliably catch this.
+- L2 misses: code compiles but does the wrong thing at runtime.
 
 If you only check one layer, you ship the bugs that the other two layers would have caught.
 
 ## How rubrics work
 
 A rubric describes:
-1. **Dimensions** — the axes you score on (e.g. `buyer_quality`, `voice`, `signal`).
-2. **Weights** — how to combine dimensions into a composite (`0.5 * buyer_quality + 0.3 * voice + 0.2 * signal`).
-3. **Failure modes** — named patterns the judge looks for ("ai-cadence", "vague-claim").
-4. **Wins** — named positive patterns ("specific-component", "earned-detail").
-5. **System prompt** — what to tell the judging LLM about the persona and the task.
+1. **Dimensions**: the axes you score on (e.g. `buyer_quality`, `voice`, `signal`).
+2. **Weights**: how to combine dimensions into a composite (`0.5 * buyer_quality + 0.3 * voice + 0.2 * signal`).
+3. **Failure modes**: named patterns the judge looks for ("ai-cadence", "vague-claim").
+4. **Wins**: named positive patterns ("specific-component", "earned-detail").
+5. **System prompt**: what to tell the judging LLM about the persona and the task.
 
-Built-in rubrics ship in `src/wire/rubrics.ts` (e.g. `anti-slop` for technical-buyer voice). You can also pass a rubric inline — the same shape, just defined at the call site.
+Built-in rubrics ship in `src/wire/rubrics.ts` (e.g. `anti-slop` for technical-buyer voice). You can also pass a rubric inline: the same shape, just defined at the call site.
 
-A rubric is plain data. The hash of that data is the `rubricVersion`. Two scores are only comparable if they used the same `rubricVersion` — change the rubric and you start a new comparison series.
+A rubric is plain data. The hash of that data is the `rubricVersion`. Two scores are only comparable if they used the same `rubricVersion`: change the rubric and you start a new comparison series.
 
 ## How verifiers work
 
@@ -141,14 +141,14 @@ const verifier = new MultiLayerVerifier([
 ])
 
 const report = await verifier.run({ env: { runner, workdir, ... } })
-report.allPass        // boolean — every layer passed
-report.blendedScore   // 0..1 — weighted aggregate
+report.allPass        // boolean: every layer passed
+report.blendedScore   // 0..1: weighted aggregate
 report.layers         // per-layer status, findings, duration
 ```
 
 Two rules that will save you bugs:
 
-1. **Run both gates.** Build gates catch code that doesn't compile; structural assertions catch missing files. Run both unconditionally — they catch orthogonal failures.
+1. **Run both gates.** Build gates catch code that doesn't compile; structural assertions catch missing files. Run both unconditionally: they catch orthogonal failures.
 
 2. **Pair LLM judges with build outcomes.** An LLM judge will rate non-compiling code as "looks right" (0.8). Always short-circuit on `buildOutcome.passed === false` before any LLM judging.
 
@@ -159,7 +159,7 @@ Two questions to answer before trusting any LLM judge:
 1. **Does it agree with humans?** `calibrateJudge(golden, candidate)` reports Pearson, MAE, integer-rounded κ, and worst-N miscalibrations vs a human golden set.
 2. **Does it agree with itself / other judges?** `continuousAgreement(scores)` and `calibrateJudgeContinuous(golden, candidate)` report κ_w + ICC(2,1) + Pearson + Spearman with bootstrap 95% CIs on the raw [0,1] scores.
 
-Why two κ flavours: the original `calibrateJudge` rounds scores to ints before computing κ. For fine-grained judges that loses information — 0.78 vs 0.81 both round to "1" and look perfectly agreed. Use `calibrateJudgeContinuous` (or `continuousAgreement` for N≥2 raters) when scores are continuous. ICC(2,1) catches systematic bias that Pearson misses: if judge B scores 2× judge A, Pearson stays ≈ 1 while ICC drops — that's the signal.
+Why two κ flavours: the original `calibrateJudge` rounds scores to ints before computing κ. For fine-grained judges that loses information: 0.78 vs 0.81 both round to "1" and look perfectly agreed. Use `calibrateJudgeContinuous` (or `continuousAgreement` for N≥2 raters) when scores are continuous. ICC(2,1) catches systematic bias that Pearson misses: if judge B scores 2× judge A, Pearson stays ≈ 1 while ICC drops: that's the signal.
 
 Bias probes (`positionalBias`, `verbosityBias`, `selfPreference`) cover the orthogonal failure modes: position-dependent scoring, length-correlated scoring, and judge-prefers-its-own-family.
 
@@ -178,7 +178,7 @@ builder-session                 [span]
     └── scenario.run            [span]
 ```
 
-Spans are append-only and have stable ids — replay is reading the same store back. OTLP export ships them out for distributed tracing.
+Spans are append-only and have stable ids: replay is reading the same store back. OTLP export ships them out for distributed tracing.
 
 You usually should not build this tree by hand. Product runtimes,
 `runAgentControlLoop`, harnesses, and verifiers should emit it while they run.
@@ -187,12 +187,12 @@ release decision.
 
 ## Where to go next
 
-- **Confused by "GEPA / HALO / trace analysis / proposers everywhere"?** → [self-improvement-map.md](./self-improvement-map.md) — one loop, four roles, the proposer catalog (production vs bench-only), and why `gepa-refine` is the same loop on a test bench.
-- **Which `run*` primitive do I use, and how do I grade produced state?** → [eval-surface-map.md](./eval-surface-map.md) — the campaign/matrix/optimization/gate primitives as a pick-by-"use-when" table, plus the produced-state grading composition (verifyCompletion-as-judge — there is no persona-dispatch wrapper) and the in-band body contract.
-- **Need the layman feature map?** → [feature-guide.md](./feature-guide.md) — what each primitive does, when to use it, integration patterns, and guardrails.
-- **Just want to score a string against a rubric?** → [wire-protocol.md](./wire-protocol.md) — HTTP/RPC interface, pluggable from any language.
-- **Need a reusable driver/worker/evaluator loop?** → [control-runtime.md](./control-runtime.md) — generic runtime plus coding, browser, computer-use, and research integration patterns.
-- **Want review feedback to become eval/optimization data?** → [feedback-trajectories.md](./feedback-trajectories.md) — turn feedback into datasets, optimizer rows, and preference memory.
+- **Confused by "GEPA / HALO / trace analysis / proposers everywhere"?** → [self-improvement-map.md](./self-improvement-map.md): one loop, four roles, the proposer catalog (production vs bench-only), and why `gepa-refine` is the same loop on a test bench.
+- **Which `run*` primitive do I use, and how do I grade produced state?** → [eval-surface-map.md](./eval-surface-map.md): the campaign/matrix/optimization/gate primitives as a pick-by-"use-when" table, plus the produced-state grading composition (verifyCompletion-as-judge: there is no persona-dispatch wrapper) and the in-band body contract.
+- **Need the layman feature map?** → [feature-guide.md](./feature-guide.md): what each primitive does, when to use it, integration patterns, and guardrails.
+- **Just want to score a string against a rubric?** → [wire-protocol.md](./wire-protocol.md): HTTP/RPC interface, pluggable from any language.
+- **Need a reusable driver/worker/evaluator loop?** → [control-runtime.md](./control-runtime.md): generic runtime plus coding, browser, computer-use, and research integration patterns.
+- **Want review feedback to become eval/optimization data?** → [feedback-trajectories.md](./feedback-trajectories.md): turn feedback into datasets, optimizer rows, and preference memory.
 - **Building a code-generator eval?** → Start with `BuilderSession`, `SandboxHarness`, and `MultiLayerVerifier`.
 - **Multi-layer verifier?** → Use [control-runtime.md](./control-runtime.md) and `MultiLayerVerifier` for ordered gates with dependencies.
 - **Adding a new judge or rubric?** → `src/wire/rubrics.ts` for the cross-language path; `src/anti-slop.ts` and `src/judges.ts` for the in-process path.

package/docs/customer-journeys.md

@@ -1,210 +1,135 @@
-# Customer journeys
+# Adoption Paths
 
-Three end-to-end journeys covering the surface of `@tangle-network/agent-eval`. Each one is a runnable example under `examples/` — clone the repo and `pnpm tsx examples/<journey>/index.ts` to see the actual output.
+Choose the path that matches the data and code you already have.
 
-The three journeys map to three customer-maturity stages:
+| Starting point | API | Result |
+|---|---|---|
+| Completed OpenTelemetry spans | `fromOtelSpans()` and `analyzeRuns()` | Failure, score, token, and cost summaries |
+| Human ratings | `fromFeedbackTable()` and `analyzeRuns()` | Reviewer agreement, largest disagreements, and score distributions |
+| A runnable agent, scenarios, and a judge | `defineAgentEval()` | Repeatable evaluation and optional prompt improvement |
 
-1. **Logs but no eval discipline** → [Production traces journey](#1-production-traces-journey-customer-otel-traces)
-2. **Ratings but no closed loop** → [Feedback corpus journey](#2-feedback-corpus-journey-customer-feedback-loop)
-3. **Scenarios, judge, agent — full closed loop** → [Closed-loop journey](#3-closed-loop-journey-selfimprove-quickstart)
+All three paths return plain objects and run in your process.
+They call a remote service only when you pass a model client, exporter, or hosted endpoint.
 
-Each section: what the customer has, what they want, the code, what the report looks like.
+## 1. Analyze Existing Traces
 
----
-
-## 1. Production traces journey — `customer-otel-traces`
-
-**The customer:** an agentic GTM-as-a-service company. Multiple agent steps in prod (social media posting, image generation, translation). OTel observability piped to their collector. Doesn't run formal evals. CTO hand-rolled their tracing.
-
-**The frustration:** "Which step is unreliable? What's our cost-quality profile? Where do we fix next?" They have the data; they don't have the answer.
-
-**What they need from agent-eval:** day-1 analysis of their existing logs. No scenarios, no judges, no closed loop. Just turn the trace stream into a decision packet.
-
-### The code
+Use this path when the agent already emits OpenTelemetry spans and you do not want to run it again.
 
 ```ts
 import { analyzeRuns, fromOtelSpans } from '@tangle-network/agent-eval/contract'
 
-const runs = fromOtelSpans({ spans: yourOtelStream })
+const runs = fromOtelSpans({ spans: yourOtelSpans })
 const report = await analyzeRuns({ runs })
 
-// report.failureClusters → root causes
-// report.costQuality.pareto → cost-vs-quality scatter
-// report.composite → distribution
-// report.recommendations → top-3 actions
+console.log(report.composite)
+console.log(report.costQuality)
+console.log(report.recommendations)
 ```
 
-### What the report shows
+`fromOtelSpans()` groups spans by run ID and reads recorded scores, failures, model IDs, token counts, and costs.
+It does not infer values that are missing from the spans.
 
-```
-Runs analyzed:     40
-Composite mean:    0.721 (p50: 0.717, p95: 0.925, stddev: 0.210)
-Cost mean:         $0.103 (p95: $0.131)
-
-── Failures ──
-6 runs with status=ERROR or failureMode set:
-  tool.search  (3x)
-  agent.turn   (3x)
+Add model-based failure clustering only when you need it:
 
-── Cost-quality Pareto ──
-1 candidate(s) plotted; 1 on the frontier
-  otel-default: cost=$0.103 quality=0.721  (frontier)
+```ts
+const report = await analyzeRuns({
+  runs,
+  analyst,
+})
 
-── Recommendations ──
-[medium] expand-corpus — Mean composite 0.721 has room
+console.log(report.failureClusters)
 ```
 
-### Next steps for this customer
-
-1. Wire an `AnalystRegistry` to cluster the 6 failures by root cause via LLM analysis.
-2. Add `outcomeSignal` once they have downstream conversion / engagement / post-engagement data, and the report fits a reward model showing whether their score predicts the customer outcome.
-3. Once they identify a step worth optimizing (translation, say), graduate to journey #3 — wrap that step as an `agent(surface, scenario)` and call `defineAgentEval()`.
-
-**Runnable:** [`examples/customer-otel-traces/`](../examples/customer-otel-traces/)
-
----
-
-## 2. Feedback corpus journey — `customer-feedback-loop`
+Runnable example: [`examples/customer-otel-traces`](../examples/customer-otel-traces/)
 
-**The customer:** a research-validation team. A GitHub Action fires `claude -p` against the next claim, writes the research output to Obsidian. Three reviewers (Alice, Bob, Carol) tag results `#approved` or `#rejected`. Outputs feed a knowledge base. Knowledge feeds content. Content feeds engagement. The founder wants more engagement faster.
+## 2. Analyze Human Ratings
 
-**The frustration:** "We disagree on what's good. We don't know if our 'good' actually drives engagement. Reviewing every claim is slow."
-
-**What they need from agent-eval:** turn the approve/reject corpus into actionable signal:
-- Where do reviewers disagree? (triage list)
-- Can we synthesize each reviewer's taste into an LLM judge? (auto-grade)
-- Does the taste actually predict downstream engagement? (close the loop)
-
-### The code
+Use this path when multiple people score the same outputs in a database, spreadsheet, or review tool.
 
 ```ts
 import { analyzeRuns, fromFeedbackTable } from '@tangle-network/agent-eval/contract'
 
-// 1. Parse Obsidian #approved / #rejected tags into a flat table:
-const ratings = parseObsidianVault('./research-vault')
-// [{ runId: 'claim-1', rater: 'alice', rating: true }, ...]
+const ratings = [
+  { runId: 'answer-1', rater: 'alice', rating: true },
+  { runId: 'answer-1', rater: 'bob', rating: false },
+  { runId: 'answer-2', rater: 'alice', rating: true },
+  { runId: 'answer-2', rater: 'bob', rating: true },
+]
 
-// 2. Pipe through the adapter:
 const { runs, raterScores } = fromFeedbackTable({ ratings })
+const report = await analyzeRuns({ runs, raterScores })
 
-// 3. Analyze:
-const report = await analyzeRuns({
-  runs,
-  raterScores,
-  // Optional: close the loop with engagement data once you have it.
-  outcomeSignal: { metric: 'engagement_rate', valueByRunId: enrichedFromProd },
-})
-
-// report.interRater.disagreementCases → top 20 claims worth a meeting
-// report.outcomeCorrelation → does team taste predict engagement?
-// report.recommendations → action list
-```
-
-### What the report shows
-
+console.log(report.interRater?.kappa)
+console.log(report.interRater?.icc)
+console.log(report.interRater?.disagreementCases)
 ```
-Runs analyzed:     30
-Composite mean:    0.756 (approve rate ~76%)
-
-── Inter-rater agreement ──
-Raters:               3 (alice, bob, carol)
-Jointly rated runs:   30
-Pairwise pearson κ:
-  alice::bob     0.53
-  alice::carol   0.55
-  bob::carol     0.21
-Mean κ:               0.43
-
-── Top 5 disagreement cases ──
-  claim-1   range=1.00  ratings: alice=0, bob=0, carol=1
-  claim-7   range=1.00  ratings: alice=0, bob=1, carol=0
-  ...
-
-── Recommendations ──
-[high] recalibrate — Inter-rater agreement κ=0.43 is below 0.5
-  Raters disagree on what 'good' looks like. Refine the rubric or triage the disagreement cases.
-```
-
-### Next steps for this customer
-
-1. **Triage meeting on the disagreement cases.** Mean κ=0.43 means the rubric is ambiguous; clarify it on the cases that split.
-2. **Calibrate one LLM judge per reviewer.** Each reviewer's history is the gold signal — substrate primitive `calibrateJudge` against `raterScores` filtered to that reviewer.
-3. **Add engagement as `outcomeSignal`** once the content downstream is instrumented. The `outcomeCorrelation` section tells the team whether their taste predicts the founder's token-max goal — and if not, the linear reward model says how to retarget.
-4. **Graduate to journey #3** — wrap the research-generation Claude-P call as an `agent(surface, scenario)`, use the calibrated judges, run `evalKit.improve()` nightly. Open a PR against the GitHub Action when the holdout approval rate beats baseline.
-
-**Runnable:** [`examples/customer-feedback-loop/`](../examples/customer-feedback-loop/)
 
----
+Weighted kappa and ICC measure absolute agreement.
+Pearson and Spearman measure correlation and are reported separately because reviewers can correlate while using different score levels.
 
-## 3. Closed-loop journey — `selfimprove-quickstart`
+Review the largest disagreements before using those labels to calibrate a model judge.
+Test the judge on human ratings that were not used during calibration.
 
-**The customer:** a team with a scenario corpus, a judge, and an agent. Wants to improve the prompt under statistical confidence — propose better candidates, gate on holdout lift, ship the winner.
+Runnable example: [`examples/customer-feedback-loop`](../examples/customer-feedback-loop/)
 
-**The frustration:** "We can run an A/B by hand but we don't know if the improvement is real. We don't have time to run paired bootstrap by hand. We want a function that decides."
+## 3. Evaluate Or Improve A Runnable Agent
 
-**What they need from agent-eval:** one reusable eval definition — propose, score, gate, ship — with the full rigor packet on the way out.
-
-### The code
+Use this path when you can call the agent for a scenario and score the returned artifact.
 
 ```ts
 import { defineAgentEval } from '@tangle-network/agent-eval/contract'
 
 const evalKit = defineAgentEval({
   scenarios,
-  agent: async (surface, scenario) =>
-    await myAgent.run({ systemPrompt: (surface as { systemPrompt: string }).systemPrompt, scenario }),
+  agent: async (prompt, scenario) => yourAgent.run({ prompt: String(prompt), scenario }),
   judge: {
-    name: 'rubric',
-    dimensions: [{ key: 'clarity', weight: 1 }, { key: 'concision', weight: 1 }],
-    score: async ({ artifact }) => myJudgeFn(artifact),
+    name: 'task-quality',
+    dimensions: [
+      { key: 'correct', description: 'The answer is correct' },
+      { key: 'complete', description: 'The answer covers the whole request' },
+    ],
+    score: ({ artifact, scenario }) => scoreArtifact(artifact, scenario),
   },
-  baselineSurface: { kind: 'prompt', systemPrompt: 'You write marketing copy...' },
-  budget: { generations: 3, populationSize: 2 },
+  baselineSurface: currentPrompt,
 })
 
-const result = await evalKit.improve()
-
-result.gateDecision   // 'ship' | 'hold' | ...
-result.insight        // full decision packet
-```
-
-### What the report shows
-
+const baseline = await evalKit.evaluate()
+const candidate = await evalKit.evaluate({ surface: proposedPrompt })
 ```
-═══ selfImprove() decision packet ═══
 
-Gate decision:        ship
-Raw lift:             +0.361
+Call `.evaluate()` when you already have a candidate to compare.
+Call `.improve()` when you want the library to generate and test candidates:
 
-── Statistical lift (paired bootstrap) ──
-delta:    +0.359
-CI95:     [0.311, 0.408]
-pValue:   0.0013
-Cohen's d: 8.58
-MDE @ 80% power: 1.401
-required n at observed effect: 122
+```ts
+const result = await evalKit.improve({
+  llm: {
+    baseUrl: process.env.OPENAI_BASE_URL ?? 'https://api.openai.com/v1',
+    apiKey: process.env.OPENAI_API_KEY!,
+    model: 'gpt-4.1-mini',
+  },
+})
 
-── Recommendations ──
-[critical] ship — Ship — lift 0.359 (95% CI 0.311..0.408)
+console.log(result.winner.surface)
+console.log(result.lift)
+console.log(result.gateDecision)
 ```
 
-### Next steps for this customer
-
-1. **Ship the winner.** Either accept `result.winner.surface` programmatically and roll it out, or pass `autoOnPromote: 'pr'` + a GitHub repo to have selfImprove open a PR for you.
-2. **Wire `hostedTenant`** to ship the decision packet to a dashboard (the hosted Intelligence orchestrator, or your own implementation of the wire spec).
-3. **Add `canaryScenarios`** to guard against the holdout leaking into the candidate prompt.
-4. **Add `outcomeSignal`** in `analyzeRuns()` for any post-deploy reruns to verify the predicted lift actually shows up in real outcomes.
+The default candidate generator calls the configured model.
+The agent and judge may make their own calls depending on your implementation.
+Set generation, population, concurrency, and dollar limits through `budget`.
 
-**Runnable:** [`examples/selfimprove-quickstart/`](../examples/selfimprove-quickstart/)
+For production use, provide enough scenarios to keep candidate generation and final comparison disjoint.
+Use an explicit held-back scenario set when the split must remain stable across runs.
 
----
+Runnable example: [`examples/selfimprove-quickstart`](../examples/selfimprove-quickstart/)
 
-## How the three journeys compose
+## Moving Between Paths
 
-Journey #1 + #2 + #3 are **maturity stages**, not exclusive products. A team typically:
+The paths compose without changing data formats:
 
-1. Starts with **#1** (analyze production logs) to find what's broken.
-2. Adds **#2** (feedback corpus) once they have a sense of where to improve, to calibrate what "good" means.
-3. Graduates to **#3** (closed loop) once they have scenarios + judges, to automate the improvement.
+1. Convert traces or ratings into `RunRecord[]` and use `analyzeRuns()` to find recurring failures.
+2. Turn those failures into representative scenarios and deterministic checks where possible.
+3. Use `defineAgentEval()` to compare changes against the same scenarios.
+4. Use `.improve()` only after the evaluation reliably separates known-good from known-bad behavior.
 
-Same substrate, same `InsightReport` shape, no rip-and-replace between stages. The data you collect in #1 informs the scenarios you derive in #2 which feed the loop in #3.
+See [`concepts.md`](./concepts.md) for data types and [`examples/README.md`](../examples/README.md) for the full runnable index.

package/docs/design/loop-taxonomy.md

@@ -8,8 +8,7 @@ one meaning. Use these words and nothing else.
 Cross-links: [`concepts.md`](../concepts.md) (eval mental model),
 [`campaign-proposers.md`](../campaign-proposers.md) (proposer catalog),
 [`multi-shot-optimization.md`](../multi-shot-optimization.md) (GEPA), and
-[`auto-research-loop-end-to-end.md`](../auto-research-loop-end-to-end.md)
-(analyst / autoresearch).
+[`self-improvement-map.md`](../self-improvement-map.md) (supported improvement paths).
 
 ## Core Roles
 
@@ -18,12 +17,12 @@ Cross-links: [`concepts.md`](../concepts.md) (eval mental model),
 | **Execution driver** | The thing that decides or routes the next turn/action inside a sandbox or worker conversation. | Inner layer only |
 | **Surface proposer** | The thing that proposes the next prompt/config/code surface for the improvement loop to measure. | Outer layer only |
 | **Worker** | An agent harness instance (Claude Code, Codex, OpenCode, …) running inside a sandbox. Does the actual work; responds in chat. | Inner layer only |
-| **Sandbox** | A multi-harness VM. Hosts **1..N workers**, which can share a workspace. Not an agent — the substrate an agent runs in. | Inner layer only |
+| **Sandbox** | A multi-harness VM. Hosts **1..N workers**, which can share a workspace. Not an agent: the substrate an agent runs in. | Inner layer only |
 | **Measurement** | Runs the worker over a set of scenarios and judges the outputs into a scorecard with confidence intervals. This is `runCampaign`. | Outer layer |
 
 Two facts that trip people up:
 
-1. **A sandbox is not a worker.** One sandbox can hold ten workers — a driver
+1. **A sandbox is not a worker.** One sandbox can hold ten workers: a driver
    can coordinate CC + Codex + OpenCode siblings sharing one workspace, or a
    fleet spread across machines. `runLoop`'s placement encodes exactly this:
    `{ sibling, sandboxId }` = co-located workers; `{ fleet, fleetId,
@@ -38,21 +37,21 @@ Two facts that trip people up:
 There are two loops. The outer one improves the thing the inner one runs.
 
 ```
-runImprovementLoop                         OUTER loop — improve the agent over time
+runImprovementLoop                         OUTER loop: improve the agent over time
 │
 ├─ PROPOSER = SurfaceProposer              proposes a candidate SURFACE
 │           (evolutionary mutator |        (the worker's system prompt / tools / config)
-│            reflective analyst)            — NOT a conversation turn
+│            reflective analyst)           : NOT a conversation turn
 │
 └─ for each candidate surface:
      │
-     runCampaign                           a MEASUREMENT — scores ONE surface
+     runCampaign                           a MEASUREMENT: scores ONE surface
      │
      └─ for each scenario × rep:
           │
-          dispatch(scenario)               THE SEAM — topology-opaque, returns an artifact
+          dispatch(scenario)               THE SEAM: topology-opaque, returns an artifact
           │
-          └─ runLoop / runMultishot        INNER loop — one conversation
+          └─ runLoop / runMultishot        INNER loop: one conversation
                ├─ DRIVER  = persona / user / planner    chats with ↓
                └─ WORKERS = 1..N agent harnesses in 1..M sandboxes
                │
@@ -82,18 +81,18 @@ opinion about execution topology: the topology lives inside `dispatch`.
 - The worker is the agent in the sandbox. The driver talks to it. ✓
 - `runCampaign` is a **measurement**, not a worker. It *runs the worker* (via
   `dispatch`); the worker does not "run the eval".
-- The outer improvement loop has **no single worker** — its proposer proposes a
+- The outer improvement loop has **no single worker**: its proposer proposes a
   *surface*, and each surface is scored by a *measurement* that drives the
   inner workers.
 
-## The dataset flywheel — why every loop run matters
+## The dataset flywheel: why every loop run matters
 
 **Every loop run, regardless of why it ran, feeds the same dataset.** This is
 the through-line that ties measurement and improvement together.
 
 When `runCampaign` runs with a `labeledStore`, each cell captures
 `(scenario, artifact, judgeScore, source)` into the `LabeledScenarioStore`.
-The `source` discriminates *why* the run happened — but the captured tuple is
+The `source` discriminates *why* the run happened: but the captured tuple is
 identical in shape:
 
 | `captureSource` | The run that produced it |
@@ -109,7 +108,7 @@ production conversation, and an autoresearch loop all deposit the same
 `(input, output, reward)` tuples. The optimization proposer later samples from
 that corpus to evolve the surface. So:
 
-> Running *any* loop — even one whose purpose is not optimization — builds the
+> Running *any* loop: even one whose purpose is not optimization: builds the
 > dataset that optimization needs. The flywheel turns whether or not you are
 > currently optimizing.
 
@@ -147,7 +146,7 @@ interface SurfaceProposer<TFindings = unknown> {
 | Implementation | Strategy | How it proposes | Where it lives |
 |---|---|---|---|
 | `evolutionaryProposer` | Evolutionary (GEPA / AxGEPA) | Standalone `SurfaceProposer`. Mutates the current best surface into N candidates, blind to history beyond the current best. Optimizes against the dataset's rewards. | **agent-eval** (pure: dataset → surface, no sandbox) |
-| Runtime reflective proposer | Reflective | Cheap generator: drafts patches from the report and applies them into a worktree (shots=1, no sandbox). | **agent-runtime** — implements agent-eval's proposer contract |
+| Runtime reflective proposer | Reflective | Cheap generator: drafts patches from the report and applies them into a worktree (shots=1, no sandbox). | **agent-runtime**: implements agent-eval's proposer contract |
 | Runtime agentic proposer | Agentic | Full generator: runs a coding harness in the worktree (≤ `maxImprovementShots`) to edit in place. | **agent-runtime** |
 
 This resolves the prior duplication where `runImprovementLoop` (evolutionary,
@@ -158,7 +157,7 @@ same cost dial, not separate outer loops. The dependency direction permits this
 cleanly: agent-eval is the leaf and owns the proposer contract; agent-runtime
 imports agent-eval and implements it.
 
-## What "the surface" is — improvement tiers
+## What "the surface" is: improvement tiers
 
 `MutableSurface` is the thing the proposer changes. It has tiers, least → most
 invasive. `MutableSurface = string | CodeSurface` spans all of them: `string`
@@ -179,36 +178,36 @@ not represented by the candidate tree.
 
 The cost/capability distinction:
 
-- **`reflectiveGenerator`** updates the *signatures* — prompt + tool surface
+- **`reflectiveGenerator`** updates the *signatures*: prompt + tool surface
   (tiers 1–2). Cheap (drafts patches, no sandbox), reversible, measured
   directly against the dataset.
 - **`agenticGenerator`** updates the *code* (tier 4). A coding harness reads
   the repository + the report, edits in a worktree, iterates up to
-  `maxImprovementShots` — measured by re-running the inner loop against the
+  `maxImprovementShots`: measured by re-running the inner loop against the
   changed code.
 
 Both are implementations of the one proposer contract (propose → measure → gate
-→ PR). They differ only in *what* they edit and *how invasive* it is — and both
+→ PR). They differ only in *what* they edit and *how invasive* it is: and both
 consume the **same dataset** the flywheel builds.
 
 ## Vocabulary quick reference
 
-- **shot** — one conversational turn (driver says X, worker responds Y). Used
+- **shot**: one conversational turn (driver says X, worker responds Y). Used
   in `runMultishot`. Never used to mean a whole eval run.
-- **runMultishot** — many shots in one conversation; persona-driver ↔ one
+- **runMultishot**: many shots in one conversation; persona-driver ↔ one
   router-agent. agent-eval.
-- **runLoop** — driver ↔ workers in sandboxes; topology-agnostic execution.
+- **runLoop**: driver ↔ workers in sandboxes; topology-agnostic execution.
   agent-runtime.
-- **runCampaign** — a measurement: a surface scored over N scenarios × M reps.
+- **runCampaign**: a measurement: a surface scored over N scenarios × M reps.
   agent-eval. (A "campaign" = a coordinated batch of measurements.)
-- **runOptimization** — the improvement loop body: proposer suggests surfaces, each is measured, and only a candidate that beats the global incumbent is promoted. agent-eval.
-- **runImprovementLoop** — `runOptimization` + holdout re-score + release gate
+- **runOptimization**: the improvement loop body: proposer suggests surfaces, each is measured, and only a candidate that beats the global incumbent is promoted. agent-eval.
+- **runImprovementLoop**: `runOptimization` + holdout re-score + release gate
   + optional PR. agent-eval.
-- **runAnalystLoop** — reflective autoresearch: findings + knowledge updates +
+- **runAnalystLoop**: reflective autoresearch: findings + knowledge updates +
   improvement proposals. agent-runtime.
-- **SurfaceProposer** — the contract a surface proposer implements.
+- **SurfaceProposer**: the contract a surface proposer implements.
   `evolutionaryProposer` (agent-eval) is one; agent-runtime can provide
   reflective or agentic implementations.
-- **CandidateGenerator** — the byte-producing seam inside a runtime proposer;
+- **CandidateGenerator**: the byte-producing seam inside a runtime proposer;
   `reflectiveGenerator` (cheap, no sandbox) and `agenticGenerator` (coding
   harness in the worktree) are the two cost settings. agent-runtime.