@tangle-network/agent-eval 0.123.1 → 0.123.3

package/docs/customer-journeys.md

@@ -1,210 +1,135 @@
-# Customer journeys
+# Adoption Paths
 
-Three end-to-end journeys covering the surface of `@tangle-network/agent-eval`. Each one is a runnable example under `examples/` — clone the repo and `pnpm tsx examples/<journey>/index.ts` to see the actual output.
+Choose the path that matches the data and code you already have.
 
-The three journeys map to three customer-maturity stages:
+| Starting point | API | Result |
+|---|---|---|
+| Completed OpenTelemetry spans | `fromOtelSpans()` and `analyzeRuns()` | Failure, score, token, and cost summaries |
+| Human ratings | `fromFeedbackTable()` and `analyzeRuns()` | Reviewer agreement, largest disagreements, and score distributions |
+| A runnable agent, scenarios, and a judge | `defineAgentEval()` | Repeatable evaluation and optional prompt improvement |
 
-1. **Logs but no eval discipline** → [Production traces journey](#1-production-traces-journey-customer-otel-traces)
-2. **Ratings but no closed loop** → [Feedback corpus journey](#2-feedback-corpus-journey-customer-feedback-loop)
-3. **Scenarios, judge, agent — full closed loop** → [Closed-loop journey](#3-closed-loop-journey-selfimprove-quickstart)
+All three paths return plain objects and run in your process.
+They call a remote service only when you pass a model client, exporter, or hosted endpoint.
 
-Each section: what the customer has, what they want, the code, what the report looks like.
+## 1. Analyze Existing Traces
 
----
-
-## 1. Production traces journey — `customer-otel-traces`
-
-**The customer:** an agentic GTM-as-a-service company. Multiple agent steps in prod (social media posting, image generation, translation). OTel observability piped to their collector. Doesn't run formal evals. CTO hand-rolled their tracing.
-
-**The frustration:** "Which step is unreliable? What's our cost-quality profile? Where do we fix next?" They have the data; they don't have the answer.
-
-**What they need from agent-eval:** day-1 analysis of their existing logs. No scenarios, no judges, no closed loop. Just turn the trace stream into a decision packet.
-
-### The code
+Use this path when the agent already emits OpenTelemetry spans and you do not want to run it again.
 
 ```ts
 import { analyzeRuns, fromOtelSpans } from '@tangle-network/agent-eval/contract'
 
-const runs = fromOtelSpans({ spans: yourOtelStream })
+const runs = fromOtelSpans({ spans: yourOtelSpans })
 const report = await analyzeRuns({ runs })
 
-// report.failureClusters → root causes
-// report.costQuality.pareto → cost-vs-quality scatter
-// report.composite → distribution
-// report.recommendations → top-3 actions
+console.log(report.composite)
+console.log(report.costQuality)
+console.log(report.recommendations)
 ```
 
-### What the report shows
+`fromOtelSpans()` groups spans by run ID and reads recorded scores, failures, model IDs, token counts, and costs.
+It does not infer values that are missing from the spans.
 
-```
-Runs analyzed:     40
-Composite mean:    0.721 (p50: 0.717, p95: 0.925, stddev: 0.210)
-Cost mean:         $0.103 (p95: $0.131)
-
-── Failures ──
-6 runs with status=ERROR or failureMode set:
-  tool.search  (3x)
-  agent.turn   (3x)
+Add model-based failure clustering only when you need it:
 
-── Cost-quality Pareto ──
-1 candidate(s) plotted; 1 on the frontier
-  otel-default: cost=$0.103 quality=0.721  (frontier)
+```ts
+const report = await analyzeRuns({
+  runs,
+  analyst,
+})
 
-── Recommendations ──
-[medium] expand-corpus — Mean composite 0.721 has room
+console.log(report.failureClusters)
 ```
 
-### Next steps for this customer
-
-1. Wire an `AnalystRegistry` to cluster the 6 failures by root cause via LLM analysis.
-2. Add `outcomeSignal` once they have downstream conversion / engagement / post-engagement data, and the report fits a reward model showing whether their score predicts the customer outcome.
-3. Once they identify a step worth optimizing (translation, say), graduate to journey #3 — wrap that step as an `agent(surface, scenario)` and call `defineAgentEval()`.
-
-**Runnable:** [`examples/customer-otel-traces/`](../examples/customer-otel-traces/)
-
----
-
-## 2. Feedback corpus journey — `customer-feedback-loop`
+Runnable example: [`examples/customer-otel-traces`](../examples/customer-otel-traces/)
 
-**The customer:** a research-validation team. A GitHub Action fires `claude -p` against the next claim, writes the research output to Obsidian. Three reviewers (Alice, Bob, Carol) tag results `#approved` or `#rejected`. Outputs feed a knowledge base. Knowledge feeds content. Content feeds engagement. The founder wants more engagement faster.
+## 2. Analyze Human Ratings
 
-**The frustration:** "We disagree on what's good. We don't know if our 'good' actually drives engagement. Reviewing every claim is slow."
-
-**What they need from agent-eval:** turn the approve/reject corpus into actionable signal:
-- Where do reviewers disagree? (triage list)
-- Can we synthesize each reviewer's taste into an LLM judge? (auto-grade)
-- Does the taste actually predict downstream engagement? (close the loop)
-
-### The code
+Use this path when multiple people score the same outputs in a database, spreadsheet, or review tool.
 
 ```ts
 import { analyzeRuns, fromFeedbackTable } from '@tangle-network/agent-eval/contract'
 
-// 1. Parse Obsidian #approved / #rejected tags into a flat table:
-const ratings = parseObsidianVault('./research-vault')
-// [{ runId: 'claim-1', rater: 'alice', rating: true }, ...]
+const ratings = [
+  { runId: 'answer-1', rater: 'alice', rating: true },
+  { runId: 'answer-1', rater: 'bob', rating: false },
+  { runId: 'answer-2', rater: 'alice', rating: true },
+  { runId: 'answer-2', rater: 'bob', rating: true },
+]
 
-// 2. Pipe through the adapter:
 const { runs, raterScores } = fromFeedbackTable({ ratings })
+const report = await analyzeRuns({ runs, raterScores })
 
-// 3. Analyze:
-const report = await analyzeRuns({
-  runs,
-  raterScores,
-  // Optional: close the loop with engagement data once you have it.
-  outcomeSignal: { metric: 'engagement_rate', valueByRunId: enrichedFromProd },
-})
-
-// report.interRater.disagreementCases → top 20 claims worth a meeting
-// report.outcomeCorrelation → does team taste predict engagement?
-// report.recommendations → action list
-```
-
-### What the report shows
-
+console.log(report.interRater?.kappa)
+console.log(report.interRater?.icc)
+console.log(report.interRater?.disagreementCases)
 ```
-Runs analyzed:     30
-Composite mean:    0.756 (approve rate ~76%)
-
-── Inter-rater agreement ──
-Raters:               3 (alice, bob, carol)
-Jointly rated runs:   30
-Pairwise pearson κ:
-  alice::bob     0.53
-  alice::carol   0.55
-  bob::carol     0.21
-Mean κ:               0.43
-
-── Top 5 disagreement cases ──
-  claim-1   range=1.00  ratings: alice=0, bob=0, carol=1
-  claim-7   range=1.00  ratings: alice=0, bob=1, carol=0
-  ...
-
-── Recommendations ──
-[high] recalibrate — Inter-rater agreement κ=0.43 is below 0.5
-  Raters disagree on what 'good' looks like. Refine the rubric or triage the disagreement cases.
-```
-
-### Next steps for this customer
-
-1. **Triage meeting on the disagreement cases.** Mean κ=0.43 means the rubric is ambiguous; clarify it on the cases that split.
-2. **Calibrate one LLM judge per reviewer.** Each reviewer's history is the gold signal — substrate primitive `calibrateJudge` against `raterScores` filtered to that reviewer.
-3. **Add engagement as `outcomeSignal`** once the content downstream is instrumented. The `outcomeCorrelation` section tells the team whether their taste predicts the founder's token-max goal — and if not, the linear reward model says how to retarget.
-4. **Graduate to journey #3** — wrap the research-generation Claude-P call as an `agent(surface, scenario)`, use the calibrated judges, run `evalKit.improve()` nightly. Open a PR against the GitHub Action when the holdout approval rate beats baseline.
-
-**Runnable:** [`examples/customer-feedback-loop/`](../examples/customer-feedback-loop/)
 
----
+Weighted kappa and ICC measure absolute agreement.
+Pearson and Spearman measure correlation and are reported separately because reviewers can correlate while using different score levels.
 
-## 3. Closed-loop journey — `selfimprove-quickstart`
+Review the largest disagreements before using those labels to calibrate a model judge.
+Test the judge on human ratings that were not used during calibration.
 
-**The customer:** a team with a scenario corpus, a judge, and an agent. Wants to improve the prompt under statistical confidence — propose better candidates, gate on holdout lift, ship the winner.
+Runnable example: [`examples/customer-feedback-loop`](../examples/customer-feedback-loop/)
 
-**The frustration:** "We can run an A/B by hand but we don't know if the improvement is real. We don't have time to run paired bootstrap by hand. We want a function that decides."
+## 3. Evaluate Or Improve A Runnable Agent
 
-**What they need from agent-eval:** one reusable eval definition — propose, score, gate, ship — with the full rigor packet on the way out.
-
-### The code
+Use this path when you can call the agent for a scenario and score the returned artifact.
 
 ```ts
 import { defineAgentEval } from '@tangle-network/agent-eval/contract'
 
 const evalKit = defineAgentEval({
   scenarios,
-  agent: async (surface, scenario) =>
-    await myAgent.run({ systemPrompt: (surface as { systemPrompt: string }).systemPrompt, scenario }),
+  agent: async (prompt, scenario) => yourAgent.run({ prompt: String(prompt), scenario }),
   judge: {
-    name: 'rubric',
-    dimensions: [{ key: 'clarity', weight: 1 }, { key: 'concision', weight: 1 }],
-    score: async ({ artifact }) => myJudgeFn(artifact),
+    name: 'task-quality',
+    dimensions: [
+      { key: 'correct', description: 'The answer is correct' },
+      { key: 'complete', description: 'The answer covers the whole request' },
+    ],
+    score: ({ artifact, scenario }) => scoreArtifact(artifact, scenario),
   },
-  baselineSurface: { kind: 'prompt', systemPrompt: 'You write marketing copy...' },
-  budget: { generations: 3, populationSize: 2 },
+  baselineSurface: currentPrompt,
 })
 
-const result = await evalKit.improve()
-
-result.gateDecision   // 'ship' | 'hold' | ...
-result.insight        // full decision packet
-```
-
-### What the report shows
-
+const baseline = await evalKit.evaluate()
+const candidate = await evalKit.evaluate({ surface: proposedPrompt })
 ```
-═══ selfImprove() decision packet ═══
 
-Gate decision:        ship
-Raw lift:             +0.361
+Call `.evaluate()` when you already have a candidate to compare.
+Call `.improve()` when you want the library to generate and test candidates:
 
-── Statistical lift (paired bootstrap) ──
-delta:    +0.359
-CI95:     [0.311, 0.408]
-pValue:   0.0013
-Cohen's d: 8.58
-MDE @ 80% power: 1.401
-required n at observed effect: 122
+```ts
+const result = await evalKit.improve({
+  llm: {
+    baseUrl: process.env.OPENAI_BASE_URL ?? 'https://api.openai.com/v1',
+    apiKey: process.env.OPENAI_API_KEY!,
+    model: 'gpt-4.1-mini',
+  },
+})
 
-── Recommendations ──
-[critical] ship — Ship — lift 0.359 (95% CI 0.311..0.408)
+console.log(result.winner.surface)
+console.log(result.lift)
+console.log(result.gateDecision)
 ```
 
-### Next steps for this customer
-
-1. **Ship the winner.** Either accept `result.winner.surface` programmatically and roll it out, or pass `autoOnPromote: 'pr'` + a GitHub repo to have selfImprove open a PR for you.
-2. **Wire `hostedTenant`** to ship the decision packet to a dashboard (the hosted Intelligence orchestrator, or your own implementation of the wire spec).
-3. **Add `canaryScenarios`** to guard against the holdout leaking into the candidate prompt.
-4. **Add `outcomeSignal`** in `analyzeRuns()` for any post-deploy reruns to verify the predicted lift actually shows up in real outcomes.
+The default candidate generator calls the configured model.
+The agent and judge may make their own calls depending on your implementation.
+Set generation, population, concurrency, and dollar limits through `budget`.
 
-**Runnable:** [`examples/selfimprove-quickstart/`](../examples/selfimprove-quickstart/)
+For production use, provide enough scenarios to keep candidate generation and final comparison disjoint.
+Use an explicit held-back scenario set when the split must remain stable across runs.
 
----
+Runnable example: [`examples/selfimprove-quickstart`](../examples/selfimprove-quickstart/)
 
-## How the three journeys compose
+## Moving Between Paths
 
-Journey #1 + #2 + #3 are **maturity stages**, not exclusive products. A team typically:
+The paths compose without changing data formats:
 
-1. Starts with **#1** (analyze production logs) to find what's broken.
-2. Adds **#2** (feedback corpus) once they have a sense of where to improve, to calibrate what "good" means.
-3. Graduates to **#3** (closed loop) once they have scenarios + judges, to automate the improvement.
+1. Convert traces or ratings into `RunRecord[]` and use `analyzeRuns()` to find recurring failures.
+2. Turn those failures into representative scenarios and deterministic checks where possible.
+3. Use `defineAgentEval()` to compare changes against the same scenarios.
+4. Use `.improve()` only after the evaluation reliably separates known-good from known-bad behavior.
 
-Same substrate, same `InsightReport` shape, no rip-and-replace between stages. The data you collect in #1 informs the scenarios you derive in #2 which feed the loop in #3.
+See [`concepts.md`](./concepts.md) for data types and [`examples/README.md`](../examples/README.md) for the full runnable index.

package/docs/design/loop-taxonomy.md

@@ -6,10 +6,8 @@ layers, and the layers were getting conflated. Every role below has exactly
 one meaning. Use these words and nothing else.
 
 Cross-links: [`concepts.md`](../concepts.md) (eval mental model),
-[`campaign-proposers.md`](../campaign-proposers.md) (proposer catalog),
-[`multi-shot-optimization.md`](../multi-shot-optimization.md) (GEPA), and
-[`auto-research-loop-end-to-end.md`](../auto-research-loop-end-to-end.md)
-(analyst / autoresearch).
+[`campaign-proposers.md`](../campaign-proposers.md) (proposer catalog), and
+[`multi-shot-optimization.md`](../multi-shot-optimization.md) (GEPA).
 
 ## Core Roles
 
@@ -18,12 +16,12 @@ Cross-links: [`concepts.md`](../concepts.md) (eval mental model),
 | **Execution driver** | The thing that decides or routes the next turn/action inside a sandbox or worker conversation. | Inner layer only |
 | **Surface proposer** | The thing that proposes the next prompt/config/code surface for the improvement loop to measure. | Outer layer only |
 | **Worker** | An agent harness instance (Claude Code, Codex, OpenCode, …) running inside a sandbox. Does the actual work; responds in chat. | Inner layer only |
-| **Sandbox** | A multi-harness VM. Hosts **1..N workers**, which can share a workspace. Not an agent — the substrate an agent runs in. | Inner layer only |
+| **Sandbox** | A multi-harness VM. Hosts **1..N workers**, which can share a workspace. Not an agent: the substrate an agent runs in. | Inner layer only |
 | **Measurement** | Runs the worker over a set of scenarios and judges the outputs into a scorecard with confidence intervals. This is `runCampaign`. | Outer layer |
 
 Two facts that trip people up:
 
-1. **A sandbox is not a worker.** One sandbox can hold ten workers — a driver
+1. **A sandbox is not a worker.** One sandbox can hold ten workers: a driver
    can coordinate CC + Codex + OpenCode siblings sharing one workspace, or a
    fleet spread across machines. `runLoop`'s placement encodes exactly this:
    `{ sibling, sandboxId }` = co-located workers; `{ fleet, fleetId,
@@ -38,21 +36,21 @@ Two facts that trip people up:
 There are two loops. The outer one improves the thing the inner one runs.
 
 ```
-runImprovementLoop                         OUTER loop — improve the agent over time
+runImprovementLoop                         OUTER loop: improve the agent over time
 │
 ├─ PROPOSER = SurfaceProposer              proposes a candidate SURFACE
 │           (evolutionary mutator |        (the worker's system prompt / tools / config)
-│            reflective analyst)            — NOT a conversation turn
+│            reflective analyst)           : NOT a conversation turn
 │
 └─ for each candidate surface:
      │
-     runCampaign                           a MEASUREMENT — scores ONE surface
+     runCampaign                           a MEASUREMENT: scores ONE surface
      │
      └─ for each scenario × rep:
           │
-          dispatch(scenario)               THE SEAM — topology-opaque, returns an artifact
+          dispatch(scenario)               THE SEAM: topology-opaque, returns an artifact
           │
-          └─ runLoop / runMultishot        INNER loop — one conversation
+          └─ runLoop / runMultishot        INNER loop: one conversation
                ├─ DRIVER  = persona / user / planner    chats with ↓
                └─ WORKERS = 1..N agent harnesses in 1..M sandboxes
                │
@@ -82,18 +80,18 @@ opinion about execution topology: the topology lives inside `dispatch`.
 - The worker is the agent in the sandbox. The driver talks to it. ✓
 - `runCampaign` is a **measurement**, not a worker. It *runs the worker* (via
   `dispatch`); the worker does not "run the eval".
-- The outer improvement loop has **no single worker** — its proposer proposes a
+- The outer improvement loop has **no single worker**: its proposer proposes a
   *surface*, and each surface is scored by a *measurement* that drives the
   inner workers.
 
-## The dataset flywheel — why every loop run matters
+## The dataset flywheel: why every loop run matters
 
 **Every loop run, regardless of why it ran, feeds the same dataset.** This is
 the through-line that ties measurement and improvement together.
 
 When `runCampaign` runs with a `labeledStore`, each cell captures
 `(scenario, artifact, judgeScore, source)` into the `LabeledScenarioStore`.
-The `source` discriminates *why* the run happened — but the captured tuple is
+The `source` discriminates *why* the run happened: but the captured tuple is
 identical in shape:
 
 | `captureSource` | The run that produced it |
@@ -109,7 +107,7 @@ production conversation, and an autoresearch loop all deposit the same
 `(input, output, reward)` tuples. The optimization proposer later samples from
 that corpus to evolve the surface. So:
 
-> Running *any* loop — even one whose purpose is not optimization — builds the
+> Running *any* loop: even one whose purpose is not optimization: builds the
 > dataset that optimization needs. The flywheel turns whether or not you are
 > currently optimizing.
 
@@ -147,7 +145,7 @@ interface SurfaceProposer<TFindings = unknown> {
 | Implementation | Strategy | How it proposes | Where it lives |
 |---|---|---|---|
 | `evolutionaryProposer` | Evolutionary (GEPA / AxGEPA) | Standalone `SurfaceProposer`. Mutates the current best surface into N candidates, blind to history beyond the current best. Optimizes against the dataset's rewards. | **agent-eval** (pure: dataset → surface, no sandbox) |
-| Runtime reflective proposer | Reflective | Cheap generator: drafts patches from the report and applies them into a worktree (shots=1, no sandbox). | **agent-runtime** — implements agent-eval's proposer contract |
+| Runtime reflective proposer | Reflective | Cheap generator: drafts patches from the report and applies them into a worktree (shots=1, no sandbox). | **agent-runtime**: implements agent-eval's proposer contract |
 | Runtime agentic proposer | Agentic | Full generator: runs a coding harness in the worktree (≤ `maxImprovementShots`) to edit in place. | **agent-runtime** |
 
 This resolves the prior duplication where `runImprovementLoop` (evolutionary,
@@ -158,7 +156,7 @@ same cost dial, not separate outer loops. The dependency direction permits this
 cleanly: agent-eval is the leaf and owns the proposer contract; agent-runtime
 imports agent-eval and implements it.
 
-## What "the surface" is — improvement tiers
+## What "the surface" is: improvement tiers
 
 `MutableSurface` is the thing the proposer changes. It has tiers, least → most
 invasive. `MutableSurface = string | CodeSurface` spans all of them: `string`
@@ -179,36 +177,36 @@ not represented by the candidate tree.
 
 The cost/capability distinction:
 
-- **`reflectiveGenerator`** updates the *signatures* — prompt + tool surface
+- **`reflectiveGenerator`** updates the *signatures*: prompt + tool surface
   (tiers 1–2). Cheap (drafts patches, no sandbox), reversible, measured
   directly against the dataset.
 - **`agenticGenerator`** updates the *code* (tier 4). A coding harness reads
   the repository + the report, edits in a worktree, iterates up to
-  `maxImprovementShots` — measured by re-running the inner loop against the
+  `maxImprovementShots`: measured by re-running the inner loop against the
   changed code.
 
 Both are implementations of the one proposer contract (propose → measure → gate
-→ PR). They differ only in *what* they edit and *how invasive* it is — and both
+→ PR). They differ only in *what* they edit and *how invasive* it is: and both
 consume the **same dataset** the flywheel builds.
 
 ## Vocabulary quick reference
 
-- **shot** — one conversational turn (driver says X, worker responds Y). Used
+- **shot**: one conversational turn (driver says X, worker responds Y). Used
   in `runMultishot`. Never used to mean a whole eval run.
-- **runMultishot** — many shots in one conversation; persona-driver ↔ one
+- **runMultishot**: many shots in one conversation; persona-driver ↔ one
   router-agent. agent-eval.
-- **runLoop** — driver ↔ workers in sandboxes; topology-agnostic execution.
+- **runLoop**: driver ↔ workers in sandboxes; topology-agnostic execution.
   agent-runtime.
-- **runCampaign** — a measurement: a surface scored over N scenarios × M reps.
+- **runCampaign**: a measurement: a surface scored over N scenarios × M reps.
   agent-eval. (A "campaign" = a coordinated batch of measurements.)
-- **runOptimization** — the improvement loop body: proposer suggests surfaces, each is measured, and only a candidate that beats the global incumbent is promoted. agent-eval.
-- **runImprovementLoop** — `runOptimization` + holdout re-score + release gate
+- **runOptimization**: the improvement loop body: proposer suggests surfaces, each is measured, and only a candidate that beats the global incumbent is promoted. agent-eval.
+- **runImprovementLoop**: `runOptimization` + holdout re-score + release gate
   + optional PR. agent-eval.
-- **runAnalystLoop** — reflective autoresearch: findings + knowledge updates +
+- **runAnalystLoop**: reflective autoresearch: findings + knowledge updates +
   improvement proposals. agent-runtime.
-- **SurfaceProposer** — the contract a surface proposer implements.
+- **SurfaceProposer**: the contract a surface proposer implements.
   `evolutionaryProposer` (agent-eval) is one; agent-runtime can provide
   reflective or agentic implementations.
-- **CandidateGenerator** — the byte-producing seam inside a runtime proposer;
+- **CandidateGenerator**: the byte-producing seam inside a runtime proposer;
   `reflectiveGenerator` (cheap, no sandbox) and `agenticGenerator` (coding
   harness in the worktree) are the two cost settings. agent-runtime.

package/docs/design.md

@@ -0,0 +1,69 @@
+# Design rationale
+
+Why this package is shaped the way it is.
+This is background reading, not reference.
+The API itself is documented in [`concepts.md`](./concepts.md) and the [`README`](../README.md).
+
+## Composition with the Tangle agent stack
+
+`agent-eval` is one package in a larger stack.
+It sits at the bottom of the layering: consumers depend on it, it depends on none of them.
+
+```
+agent-runtime    Runs agents (chat turns, one-shot tasks, multi-attempt loops), captures every
+                 run as a trace, and exposes improve(), which composes agent-eval's improvement
+                 loop. Produces the RunRecords + traces agent-eval scores. Depends on agent-eval.
+
+agent-eval       selfImprove, analyzeRuns, runCampaign + surface proposers (GEPA proposer, …),
+   (this repo)   the gates (heldOutGate, defaultProductionGate, paretoSignificanceGate), the
+                 InsightReport, the RL bridge, the wire protocol. Depends on neither consumer.
+
+agent-knowledge  proposeKnowledgeWrites / applyKnowledgeWriteBlocks. agent-eval's analyst
+                 findings feed it; the knowledge gate consumes them. Depends on agent-eval.
+
+sandbox          Sandbox.create, streamPrompt. One execution environment the runtime's
+                 loops run on; agent-eval scores what comes back.
+```
+
+None of the sibling packages are required to use `agent-eval`; the library stands alone.
+The stack context matters only if you adopt more of it later.
+
+## The dependency rule
+
+This section is the public rationale.
+The enforceable maintainer rule lives in [`CLAUDE.md`](../CLAUDE.md#repo-layering--this-package-is-the-substrate).
+
+**`agent-eval` has zero upward dependencies on a consumer.**
+This is what keeps the package reusable outside our own stack: nothing in here imports from `agent-runtime`, `agent-knowledge`, or `sandbox`, whether at runtime, in development dependencies, or as type-only imports.
+
+The placement test for any shared type: *does this concept make sense without a running agent loop?*
+
+- Yes: it lives here. A judge score, a run record, a scenario, and a pass/fail verdict are all meaningful for a pile of logs with no agent attached.
+- No: it lives in the runtime layer. A validation context carrying an abort signal and a concrete sandbox session only exist mid-run.
+
+When in doubt, the type moves down into `agent-eval`: subtracting a dependency from a consumer is always cheaper than adding one here.
+Agent profile shape is the shared `@tangle-network/agent-interface` contract, so neither layer owns it.
+
+## Why "surface"
+
+The improvement loop needs one word for "the thing being changed", because it deliberately does not care what that thing is: a system prompt, a config object, a skill file, a set of few-shot examples.
+Proposers emit candidate surfaces, campaigns measure them, gates compare them against the baseline surface.
+Where a doc can say "prompt" concretely, it should; `surface` appears in API names where the generality is the point.
+
+## Why the report never invents signal
+
+Every section of the `analyzeRuns()` report is opt-in based on what the input data supports.
+If runs carry no judge scores, the judge section is empty rather than defaulted.
+If there is no baseline/candidate split, no lift is reported.
+Missing evidence is never scored as zero; a judge that throws is recorded as a failed cell, not silently folded into the average.
+The reasoning: a fabricated zero poisons every statistic downstream, and an eval library that quietly fabricates is worse than no eval at all.
+
+## Maintainer docs
+
+These files record operating conventions for maintainers and internal agents.
+They are not adoption reference:
+
+- [`building-doctrine.md`](./building-doctrine.md): conventions our agents follow when consuming this package (reachable model defaults, probe-before-debug, experiment integrity checklist)
+- [`design/loop-taxonomy.md`](./design/loop-taxonomy.md): the internal vocabulary for execution drivers, workers, measurements, and proposers
+- [`research-report-methodology.md`](./research-report-methodology.md): the evidence standard our own research reports are held to
+- [`.claude/skills/agent-eval/SKILL.md`](../.claude/skills/agent-eval/SKILL.md): directives for LLM agents writing integration code, encoding bug classes we have already shipped and fixed once

package/docs/distributed-driver.md

@@ -1,10 +1,10 @@
-# Distributed campaign execution — coordinator-on-A, workers-on-B
+# Distributed campaign execution: coordinator-on-A, workers-on-B
 
 The coordinator process (running `runCampaign` / `runImprovementLoop` /
 `gepaProposer`) and the worker (running your actual agent) **do not have to live in the
 same process, machine, region, or cloud.** `Dispatch` is just a
 function: scenario in, artifact out. Whatever returns the artifact is
-the worker — local, remote, sandboxed, or fanned out across a fleet.
+the worker: local, remote, sandboxed, or fanned out across a fleet.
 
 ## Why you'd want this
 
@@ -13,7 +13,7 @@ the worker — local, remote, sandboxed, or fanned out across a fleet.
 | **Coordinator on your VPC, workers on our sandbox fleet** | Coordinator holds secrets, training data, prompt corpus; workers stay stateless and scale horizontally |
 | **Multi-region campaigns** | Each cell runs in the region closest to its target API (latency, compliance, data residency) |
 | **Coordinator-as-a-service** | Long-running optimization process; reuses across many short-lived worker invocations |
-| **Heterogeneous workers** | One cell on a CPU container, another on a GPU box, another against a third-party API — same Dispatch shape, different placement |
+| **Heterogeneous workers** | One cell on a CPU container, another on a GPU box, another against a third-party API: same Dispatch shape, different placement |
 | **Budget-isolated workers** | Worker boxes get scoped, time-bounded credentials; coordinator never holds production keys |
 
 ## Two new pieces in 0.45.0
@@ -22,7 +22,7 @@ the worker — local, remote, sandboxed, or fanned out across a fleet.
 |---|---|
 | **`DispatchContext.placement?: string`** | Opaque placement key the substrate forwards to the Dispatch. |
 | **`RunCampaignOptions.cellPlacement?(input) → string \| undefined`** | Strategy function the substrate calls per cell to compute the placement key. |
-| **`@tangle-network/agent-eval/adapters/http`** | `httpDispatch` (client) + `runDispatchServer` (server) — wire shape for HTTP-based remote workers. |
+| **`@tangle-network/agent-eval/adapters/http`** | `httpDispatch` (client) + `runDispatchServer` (server): wire shape for HTTP-based remote workers. |
 
 Both ends of the wire are in the same package; no peer dep, no separate
 install. The substrate doesn't strategy-pick; you provide the
@@ -31,7 +31,7 @@ Dispatch reads it. Clean seam, no policy baked in.
 
 ## The three reference topologies
 
-### 1. In-process (the default — what you already have)
+### 1. In-process (the default: what you already have)
 
 ```ts
 await runCampaign({
@@ -74,7 +74,7 @@ import { runDispatchServer } from '@tangle-network/agent-eval/adapters/http'
 
 const handle = await runDispatchServer<MyScenario, MyArtifact>({
   dispatch: async (scenario, ctx) => {
-    // your agent — call OpenAI, LangChain, your sandbox, anything.
+    // your agent: call OpenAI, LangChain, your sandbox, anything.
     const artifact = await runMyAgent(scenario, ctx.signal)
     return artifact
   },
@@ -85,7 +85,7 @@ console.log(`worker listening on ${handle.port}`)
 ```
 
 Cancellation, retries on 5xx / 408 / 429, bounded timeouts, optional
-custom auth headers, optional `fetchImpl` override — all there.
+custom auth headers, optional `fetchImpl` override: all there.
 
 ### 3. Multi-region fan-out
 
@@ -121,7 +121,7 @@ await runCampaign({
 })
 ```
 
-`cellPlacement` is a pure function the substrate calls per cell — no
+`cellPlacement` is a pure function the substrate calls per cell: no
 state. Use whatever signal you want (tags, hash of scenario id,
 round-robin, region-affinity from a previous run, scheduling table).
 
@@ -134,7 +134,7 @@ round-robin, region-affinity from a previous run, scheduling table).
 | **Retries** | Idempotent retries on 5xx / 408 / 429 with exponential backoff + jitter. Coordinator aborts never retry. |
 | **Auth** | Bearer token on `Authorization`; pluggable via `auth: string \| () => string \| Promise<string>` for rotation/refresh. |
 | **Payload size** | Server enforces `maxBodyBytes` (default 10 MB). |
-| **Traces** | Both ends emit OTel — if both point at the same OTLP collector, you get a unified trace per cell. See `docs/adapters-observability.md`. |
+| **Traces** | Both ends emit OTel: if both point at the same OTLP collector, you get a unified trace per cell. See `docs/adapters-observability.md`. |
 | **Cost** | Worker's `ctx.cost.runPaidCall(...)` writes durable receipts in the worker process. Roll up those receipts server-side and attach them to worker telemetry; they are not forwarded to the coordinator automatically. |
 
 ## Running the reference example
@@ -142,10 +142,10 @@ round-robin, region-affinity from a previous run, scheduling table).
 See `examples/distributed-driver/`:
 
 ```sh
-# Terminal 1 — worker
+# Terminal 1: worker
 pnpm tsx examples/distributed-driver/worker.ts
 
-# Terminal 2 — coordinator
+# Terminal 2: coordinator
 WORKER_URL=http://localhost:8080/dispatch \
 WORKER_TOKEN=dev-token \
 pnpm tsx examples/distributed-driver/driver.ts
@@ -157,17 +157,17 @@ and using `cellPlacement` to fan across many of them.
 
 ## Known gaps + follow-ups
 
-- **Cost roll-up across the wire** — worker-side `ctx.cost` observations
+- **Cost roll-up across the wire**: worker-side `ctx.cost` observations
   stay on the worker. We need to forward them in the response body so
   `defaultProductionGate`'s `budgetUsd` ceiling reflects total spend, not
   coordinator-side spend. Tracked as a 0.45.x follow-up.
-- **Per-cell artifact streaming** — when the worker writes intermediate
+- **Per-cell artifact streaming**: when the worker writes intermediate
   artifacts via `ctx.artifacts.write`, those land on the worker's
   storage. For multi-worker campaigns you'll want a shared object store
   (S3/GCS) reachable from both sides; today consumers wire that as a
   `CampaignStorage` impl. A reference S3-backed storage is on the
   roadmap.
-- **gRPC / NATS / Temporal transports** — the wire is HTTP today by
+- **gRPC / NATS / Temporal transports**: the wire is HTTP today by
   default because everything speaks HTTP. Other transports can ship as
   additional adapters; the `Dispatch` interface itself is
   transport-agnostic.