@tangle-network/agent-eval 0.45.0 → 0.47.0

package/docs/design/phase-d-rfc.md

@@ -0,0 +1,125 @@
+# Phase D RFC — hosted-tier substrate
+
+Pinned scope decisions for the EXPAND tier. What we built, what we
+deliberately did NOT, and what's gated on Phase B evidence.
+
+---
+
+## What's in this version
+
+**Wire-format substrate (shipped):**
+
+1. `@tangle-network/agent-eval/hosted` — public client + types for shipping
+   eval-run events + trace spans to any orchestrator that speaks the wire
+   format.
+2. `docs/hosted-ingest-spec.md` — semver-committed wire spec
+   (`HostedWireVersion = "2026-05-26.v1"`).
+3. `examples/hosted-ingest-server/` — minimal hono-based reference
+   receiver (~200 LOC). Executable spec. Stays as the reference even
+   after the production orchestrator ships.
+4. `selfImprove({ hostedTenant })` opt-in — when set, the substrate
+   POSTs the final eval-run event to the configured endpoint. Failures
+   are logged but never fail the loop (LAND tier never blocks on
+   EXPAND-tier infra).
+
+**Production orchestrator (started):**
+
+5. HTTP ingest service in `@tangle-network/monorepo` accepting the wire
+   format. Lives under the orchestrator app. Tenant auth + isolation
+   + persistent storage + read endpoints. *Started this session — see
+   the @tangle-network/agent-dev-container PR. Not feature-complete:
+   tenant CRUD + adversarial isolation tests pending.*
+
+## What's deliberately deferred
+
+The wedge doc gates these on Phase B evidence — partner-validated
+signal about what the hosted product actually needs to do. Shipping
+them without that signal risks building the wrong thing.
+
+| Deferred until Phase B passes | Why |
+|---|---|
+| **Metered billing wire-up (Stripe + cost-ledger)** | The billable units (per-eval-run, per-ingested-MB, per-seat) depend on actual partner consumption patterns. Picking dimensions in a vacuum locks us into wrong pricing. |
+| **Multi-tenant dashboard UX** | Partners' first dashboard request defines the right default views. We have a stub list-runs page; the rest is post-signal. |
+| **Webhook callbacks per tenant** | The events partners want pushed (gate-decided, cost-threshold, regression-alert) are partner-shaped. Add them when a partner asks. |
+| **Cross-tenant aggregation / benchmarking** | This is the "Datadog for agents" tier — explicit roadmap, requires user volume we don't have. |
+| **Sandbox-cost roll-up into hosted billing** | Cross-product billing integration requires PLATFORM-tier partners. Out of scope until at least one. |
+| **Trace UI** | OTel-shape spans store fine. Visualization comes after partners ask. Phoenix / Jaeger / any OTLP-compatible viewer covers it in the interim. |
+| **Soc2 / compliance audit work** | Required for enterprise; not required for design partners. |
+
+## Architecture decisions locked
+
+These are committed and won't change without a major-version wire bump
+or a documented migration:
+
+1. **Wire format is JSON over HTTP**, not gRPC. Reasons: works in
+   browsers + edge + node + curl; OTel-compatible at the trace stream
+   level; lowest possible barrier to a self-hosted orchestrator.
+2. **Tenant auth is bearer-token + tenant-id header**, not OIDC /
+   service-account / mutual-TLS. Reasons: simplest thing that's
+   actually secure with proper key handling; defers complex IAM until
+   enterprise demand.
+3. **Idempotency via header, not transactional API**. Servers MUST
+   dedupe by `(tenantId, Idempotency-Key)` for 24h. Simpler than
+   making clients commit transactions.
+4. **Eval-runs and traces are SEPARATE streams** with pivot keys
+   (`tangle.runId` etc.) on spans. Reasons: traces can be best-effort
+   (lossy) without corrupting eval-run semantics; orchestrators can
+   prioritize eval-run durability without forcing trace durability.
+5. **Wire version is a date.v-N string**, not semver. Reasons: dates
+   communicate "when was this contract frozen"; v-N captures
+   incremental breaking changes between dates.
+
+## Open questions for Phase B to answer
+
+When the design-partner pairing happens, capture answers to these
+explicitly:
+
+1. **Surface confidentiality**: do partners want the verbatim surface
+   (system prompt) shipped, or just the hash? Today the wire format
+   has `surface?` as optional; partner default is what we ship.
+2. **Trace sampling**: at what cells-per-second do trace spans become
+   noise? What's the right default sampling rate?
+3. **Cost attribution granularity**: per cell? per generation? per
+   run? Per judge dimension? Partner needs determine what we surface
+   in billing reports.
+4. **Replay**: do partners want to re-run an old eval-run from the
+   stored data? That would require us to store more than the summary —
+   actual artifacts + prompts. Storage cost implication.
+5. **PII / sensitive scenarios**: how do partners want to handle
+   scenarios containing user data? Encryption-at-rest is table stakes;
+   redaction-at-ingest may be required for some.
+
+The partner pairing kit (`docs/phase-b-pairing-kit.md`) has discovery
+questions that probe these.
+
+## Non-goals (explicit)
+
+This RFC does NOT plan for:
+
+- Replacing Langfuse / Phoenix / Arize. We INGEST OTel; we don't
+  build a generic trace viewer. The dashboard is eval-run-shaped, not
+  trace-shaped.
+- Becoming a model gateway. Tangle Router exists; the hosted
+  orchestrator routes to Tangle Router by default but doesn't
+  duplicate its function.
+- Becoming an LLM-call CDN. Caching is the consumer's job (their
+  agent code, their HTTP client). We don't intercept LLM calls.
+- Building an "agents IDE." Substrate, not surface.
+
+## Migration path (post Phase B)
+
+When Phase B passes the gate, the production orchestrator finishes:
+
+1. Replace in-memory store with Postgres (tenant data) + S3 (large
+   artifacts) OR Cloudflare D1 + R2 (Workers-native).
+2. Wire metered events to Stripe + the cost-ledger.
+3. Tenant CRUD UI + onboarding flow.
+4. Multi-tenant dashboard MVP (list runs, drill into one, diff
+   generations, view shipped prompt).
+5. Adversarial tenant-isolation test battery in CI.
+6. Webhooks + observability for the orchestrator itself.
+
+Estimated effort post-Phase-B: ~1 week focused work for one engineer.
+This is fast precisely BECAUSE the wire format is locked and the
+reference receiver exists — the production server is a different
+implementation of the same contract.

package/docs/hosted-ingest-spec.md

@@ -0,0 +1,204 @@
+# Hosted-ingest wire spec — `2026-05-26.v1`
+
+The schema **every** orchestrator (ours, partners' self-hosted ones,
+any future open implementation) must accept. Frozen under semver:
+**new minors only add optional fields. Breaking changes mean a major
+bump and a new `HostedWireVersion` literal.**
+
+This is the contract that decouples the LAND-tier substrate
+(`@tangle-network/agent-eval`) from the EXPAND-tier hosted product. A
+foreign builder can:
+
+- Use our orchestrator at `https://orchestrator.tangle.tools/v1`.
+- Self-host the reference receiver from
+  `examples/hosted-ingest-server/`.
+- Implement their own orchestrator against this spec.
+
+All three are wire-compatible by definition.
+
+---
+
+## Transport
+
+Two endpoints, both `POST`, both JSON. Headers on every request:
+
+| Header | Value |
+|---|---|
+| `Authorization` | `Bearer <tenant-key>` (the orchestrator issues this) |
+| `Content-Type` | `application/json` |
+| `X-Tangle-Tenant-Id` | The tenant's stable id (the orchestrator's primary key for the tenant) |
+| `X-Tangle-Wire-Version` | `2026-05-26.v1` (this spec) |
+| `Idempotency-Key` (optional) | UUID; servers MUST treat repeated keys as dedup |
+
+Responses are JSON of shape `{ accepted: number, rejected: Array<{ index, reason }> }`. The
+server SHOULD return 202 (accepted, async) or 200 (accepted, synchronous);
+both are equivalent for the wire's purposes.
+
+### `POST /v1/ingest/eval-runs`
+
+Body: `IngestEvalRunsRequest = { wireVersion, events: EvalRunEvent[] }`.
+
+One ingest call per logical eval-run; generations stream in
+incrementally via repeated calls with the same `runId`. The
+orchestrator deduplicates by `(tenantId, runId, generation.index)`.
+
+### `POST /v1/ingest/traces`
+
+Body: `IngestTracesRequest = { wireVersion, spans: TraceSpanEvent[] }`.
+
+Standard OTLP-shaped spans with a few additional attributes
+(`tangle.runId`, `tangle.generation`, `tangle.cellId`,
+`tangle.scenarioId`) so the orchestrator can pivot between the
+eval-run stream and the underlying execution trace.
+
+---
+
+## `EvalRunEvent`
+
+```ts
+interface EvalRunEvent {
+  runId: string                      // stable; same id across all generations of one run
+  runDir: string                     // logical run directory (mem://... or filesystem path)
+  timestamp: string                  // ISO-8601
+  status:                            // lifecycle stage this event represents
+    | 'started'
+    | 'baseline-complete'
+    | 'generation-complete'
+    | 'gate-decided'
+    | 'finished'
+    | 'errored'
+  labels: Record<string, string>     // free-form (env, branch, model id, etc.)
+  baseline?: EvalRunGenerationSnapshot   // present when status >= baseline-complete
+  generations: EvalRunGenerationSnapshot[]
+  gateDecision?:                     // present when status >= gate-decided
+    | 'ship' | 'hold' | 'need_more_work' | 'model_ceiling' | 'arch_ceiling'
+  holdoutLift?: number               // winner-on-holdout - baseline-on-holdout
+  totalCostUsd: number
+  totalDurationMs: number
+  errorMessage?: string              // present when status === 'errored'
+}
+```
+
+## `EvalRunGenerationSnapshot`
+
+```ts
+interface EvalRunGenerationSnapshot {
+  index: number                      // 0 is baseline; 1..N are improvement generations
+  surfaceHash: string                // stable hash of the candidate surface (pivot key)
+  surface?: MutableSurface           // OMITTED to avoid PII when consumer prefers
+  cells: EvalRunCellScore[]
+  compositeMean: number
+  costUsd: number
+  durationMs: number
+}
+```
+
+## `EvalRunCellScore`
+
+```ts
+interface EvalRunCellScore {
+  scenarioId: string
+  rep: number                        // 0 for the default; > 0 when reps > 1
+  compositeMean: number              // composite across all judges + dimensions
+  dimensions: Record<                // outer key = judge name; inner = dimension name → score
+    string,
+    Record<string, number>
+  >
+  errorMessage?: string              // present when the dispatch threw
+}
+```
+
+## `TraceSpanEvent`
+
+```ts
+interface TraceSpanEvent {
+  // Standard OTel
+  traceId: string
+  spanId: string
+  parentSpanId?: string
+  name: string
+  startTimeUnixNano: number
+  endTimeUnixNano: number
+  attributes: Record<string, string | number | boolean>
+  events?: Array<{ timeUnixNano, name, attributes? }>
+  status?: { code: 'OK' | 'ERROR' | 'UNSET', message? }
+
+  // Tangle additions (all optional) for pivoting
+  'tangle.runId'?: string
+  'tangle.generation'?: number
+  'tangle.cellId'?: string
+  'tangle.scenarioId'?: string
+}
+```
+
+---
+
+## Server requirements
+
+Any orchestrator implementing this spec MUST:
+
+1. **Validate auth**: reject without `Authorization` header (401), with a
+   mismatched bearer token (401), or without a recognized `X-Tangle-Tenant-Id`
+   (404).
+2. **Validate wire version**: reject incompatible wire versions (400 with
+   a clear error message). The major component is the breaking-change axis.
+3. **Validate tenant isolation**: queries with `tenantId` X never return
+   data tagged with `tenantId` Y. Test this adversarially.
+4. **Honor idempotency**: when an `Idempotency-Key` matches a prior
+   request from the same tenant in the last 24h, return the same response
+   without double-processing.
+5. **Persist eval-runs durably**: at least the event + cell scores must
+   survive an orchestrator restart. Trace spans MAY be best-effort.
+6. **Provide read access**: GET endpoints for the tenant to list + fetch
+   their own runs. Wire format for reads is NOT part of this spec — each
+   orchestrator can pick its own (REST + JSON, gRPC, GraphQL).
+
+Servers SHOULD also:
+
+- Provide a webhook callback per tenant for `gate-decided` events.
+- Provide a billable-events emitter (Stripe meter / equivalent) per ingest
+  call so consumption can be metered.
+- Provide a dashboard or API to view + diff per-scenario lifts over time.
+
+---
+
+## Reference implementation
+
+`examples/hosted-ingest-server/` — a minimal hono-based receiver. ~200
+LOC. Validates auth, accepts ingest, stores in memory, exposes a
+read endpoint. Runs anywhere Node runs.
+
+```sh
+TENANT_KEY=dev-token TENANT_ID=acme pnpm tsx examples/hosted-ingest-server/server.ts
+```
+
+In another terminal:
+
+```sh
+HOSTED_ENDPOINT=http://localhost:8080 \
+HOSTED_TENANT_KEY=dev-token \
+HOSTED_TENANT_ID=acme \
+pnpm tsx examples/foreign-agent-quickstart/index.ts
+```
+
+The quickstart's eval-run gets POSTed to the reference receiver; the
+receiver's `GET /v1/runs` lists it back.
+
+---
+
+## Versioning
+
+`HostedWireVersion` is `"2026-05-26.v1"`.
+
+- Adding an optional field → no version change.
+- Adding a new endpoint or new event type → minor wire bump
+  (`2026-05-26.v2`).
+- Changing the shape of an existing field, removing a field, or
+  changing semantics of an existing field → major wire bump
+  (`2026-11-XX.v1`); a server may accept both versions during a
+  transition window.
+
+Servers MUST reject requests with `X-Tangle-Wire-Version` they don't
+support, with a 400 listing the versions they DO accept.
+
+The version string IS the spec id — pin against it.

package/docs/phase-b-pairing-kit.md

@@ -0,0 +1,188 @@
+# Phase-B partner pairing kit
+
+Everything we hand a design partner — the pitch, the discovery doc,
+the judge worksheet, the 4-hour pairing agenda, the success criteria.
+
+> This file is **partner-facing**. The internal driving runbook is in
+> [`phase-b-runbook.md`](./phase-b-runbook.md).
+
+---
+
+## The pitch (one-pager)
+
+You have a working agent. You don't have evals. You don't have a
+self-improvement loop. You don't know which prompt change actually
+made the agent better last week.
+
+We have all of that on a shelf — same engine our six internal product
+agents use in production. It's open source, free at the LAND tier, and
+sandbox-free if you don't want our sandbox.
+
+**The Phase-B offer:** in one 4-hour pairing, we wrap your agent
+behind our `Dispatch`, author your domain-specific judge with you,
+and run one real campaign + improvement loop on **your actual use
+case**. You walk away with:
+
+- A reproducible eval harness against scenarios you control.
+- A judge that scores your outputs on dimensions you defined.
+- One measurable lift on your real product, with a held-out gate.
+- Trace artifacts you own (locally on disk; nothing leaves your
+  network unless you point at our hosted tier).
+
+What we get: design-partner evidence the substrate works on a foreign
+agent we did not build. That validates the wedge for us. Nothing else
+changes hands.
+
+**Cost to you:** 4 hours of pairing + your LLM bill for the campaign
+run (typically $5-$50 depending on model + scenario count). No
+commitment, no contract, no exclusivity. We don't take your code, your
+data, or your secrets.
+
+---
+
+## Discovery questions (15 min, before the pairing)
+
+Send these to the partner ahead of the pairing so they walk in with
+their answers.
+
+### About the agent
+
+1. What does your agent **do** — one paragraph, end-user perspective?
+2. What's the **input** it accepts and the **output** it produces?
+   (Schemas help; English is fine.)
+3. What framework / stack? (LangChain / Mastra / OpenAI Agents SDK /
+   bespoke / something else.)
+4. Where does it run? (Local node / serverless / your sandbox /
+   browser / mobile / other.)
+5. What model(s) does it use today? Any model-routing layer
+   (OpenRouter, Portkey, your own)?
+
+### About quality
+
+6. How do you currently know your agent is good? (Eyeballing /
+   user feedback / metrics / nothing yet — all fine answers.)
+7. What does a **bad** output look like for you? Give 2-3 concrete
+   examples. Be specific.
+8. What does a **good** output look like? Same.
+9. Are there outputs that are *technically correct but feel wrong*?
+   What's the signal?
+10. How would a senior person on your team **score** an output, if
+    they had to give it a 1-10? Walk us through the rubric they'd
+    use, even informally.
+
+### About the loop
+
+11. If we could improve one thing about the agent in 4 hours, what
+    would move the needle the most for you?
+12. Are there *prompt* changes you've wanted to try but haven't had
+    the loop to validate?
+13. Anything you've explicitly tried that **didn't** work? (Saves us
+    suggesting it.)
+
+---
+
+## Judge-design worksheet (45 min into the pairing)
+
+The judge is the most under-discussed piece of an eval system. Most
+projects fail at the judge, not the agent.
+
+We start with a **strawman** — the 6 dimensions in our canonical
+marketing-quality judge:
+
+| Dim | What it measures |
+|---|---|
+| hook_strength | Opens with concrete user outcome, not category |
+| voice_match | Reads human-written; no AI slop |
+| cta_clarity | Next step unambiguous for the audience |
+| factual_grounding | Only claims things the brief supports |
+| surface_fit | Length + register correct for medium |
+| audience_specificity | Vocabulary the audience actually responds to |
+
+**Your job in this 45 min:** rip this apart. We expect:
+
+- **2-3 of these are wrong for you.** Replace them.
+- **2-3 dimensions are missing.** Add them. (E.g., "tone matches our
+  brand book" or "safety-critical claim has a citation" or "answer is
+  decisive — no hedging when the user wants a recommendation".)
+- **Weights are wrong.** For your use case some dims matter 5x more.
+
+The deliverable: a judge with 4-8 dimensions, each scored 0.0 - 1.0,
+each unambiguous enough that two independent humans would score the
+same artifact within 0.1.
+
+If a dimension is squishy, throw it out. A noisy judge poisons the
+loop.
+
+---
+
+## The 4-hour pairing agenda
+
+### Hour 1 — Discovery + Dispatch wiring
+
+| Time | What | Deliverable |
+|---|---|---|
+| 0:00 - 0:15 | Review discovery answers, align on scope | Shared doc with goals + constraints |
+| 0:15 - 0:45 | Wire `Dispatch` around their agent — typically 1 function | Working `Dispatch<TScenario, TArtifact>` |
+| 0:45 - 1:00 | Run 1-2 scenarios through `Dispatch` manually; see real artifacts | Confirmed wire shape |
+
+### Hour 2 — Judge calibration
+
+| Time | What | Deliverable |
+|---|---|---|
+| 1:00 - 1:45 | Walk through the strawman judge; redesign dimensions with the partner | Final `JudgeConfig` for their domain |
+| 1:45 - 2:00 | Calibrate judge against the 2 manual outputs from Hour 1 | Confirmed judge gives same scores a human would |
+
+### Hour 3 — First campaign + tuning
+
+| Time | What | Deliverable |
+|---|---|---|
+| 2:00 - 2:30 | Define 8-15 scenarios with the partner (or use ours as a template) | Scenario set with train + holdout split |
+| 2:30 - 3:00 | Run `runEval` for baseline; review per-scenario scores | Baseline score + identified failure modes |
+
+### Hour 4 — Improvement loop + go/no-go
+
+| Time | What | Deliverable |
+|---|---|---|
+| 3:00 - 3:30 | Configure `runImprovementLoop` with `gepaDriver` (3 generations, population 2) + `defaultProductionGate` | Improvement run completes |
+| 3:30 - 3:50 | Walk the partner through the gate decision + lift per scenario | Report artifact |
+| 3:50 - 4:00 | Capture: was the lift real? Would they ship the winner? Will they keep using the lib? | **Go/no-go signal for Phase D** |
+
+If we're tracking ahead at any hour, use the slack to deepen — add a
+red-team battery, swap the judge model, run more generations. If we're
+behind, cut the scenario set to 6 and ship.
+
+---
+
+## Success criteria — what counts as Phase B passed
+
+For us to greenlight Phase D (hosted orchestrator + metered billing),
+we need ALL of:
+
+1. **Real lift.** Held-out winner score > baseline by ≥ 0.05 composite
+   points (or the partner's chosen threshold). Not just train; held-out.
+2. **Partner-validated lift.** The partner reads the winner output on
+   3+ held-out scenarios and confirms it's actually better.
+3. **Integration time ≤ 1 day.** Discovery + wiring + judge took ≤ 4
+   hours for the pairing; partner could reach the same point solo in
+   ≤ 1 day from the quickstart doc.
+4. **Public commitment.** Partner agrees to a public reference (case
+   study / quote / logo) OR commits to running the LAND tier in their
+   own product within 2 weeks.
+
+3-of-4 = soft pass (revisit Phase D scope but proceed). 4-of-4 = hard
+pass (build Phase D). ≤ 2 = fail (back to substrate iteration).
+
+---
+
+## What we don't ask for
+
+- Your code. Wire `Dispatch` around your existing API; we never see the
+  source.
+- Your customer data. Use synthetic scenarios or anonymized real ones —
+  whichever you prefer.
+- Your model keys. You bring your own; if you want, route through Tangle
+  Router and we never see the prompts either.
+- Exclusivity, commitment, or contract. Walk away whenever.
+
+The point is to learn if the substrate works for someone we didn't
+build it for. That's it.

package/docs/phase-b-runbook.md

@@ -0,0 +1,176 @@
+# Phase-B runbook (internal)
+
+How we drive a design-partner pairing. Goes alongside
+[`phase-b-pairing-kit.md`](./phase-b-pairing-kit.md) (the partner-facing
+materials) — this file is for us.
+
+---
+
+## Before the pairing
+
+- **24-48h prior:** send discovery questions from
+  [`phase-b-pairing-kit.md`](./phase-b-pairing-kit.md). Don't run the
+  pairing without answers in hand. The pairing fails when we discover
+  the partner's quality bar live; we don't have time to interview AND
+  build in 4 hours.
+- **48h prior:** run the canonical demo (`pnpm tsx
+  examples/marketing-agent-canonical/index.ts`) end-to-end against the
+  partner's preferred model. Confirms the substrate + their LLM tier
+  compose. If it errors, fix the substrate before the pairing.
+- **24h prior:** mirror the partner's stack locally. If they're on
+  Cloudflare Workers, run a Worker. On LangChain, install `@langchain/*`.
+  Don't debug their tooling on the call.
+- **1h prior:** open the pairing kit, the agent-eval repo, the partner's
+  agent code/endpoint, a shared doc, and a screenshare ready.
+
+## During the pairing
+
+### Driving principles
+
+- **Talk less, ship more.** The partner is paying with their time and
+  attention; every minute we talk we aren't shipping their lift.
+- **They write the judge.** We start with our strawman so they have
+  something to react to, but the judge that ends up running is theirs.
+  This is the most-discussed seam — they should own it.
+- **No invented features.** Don't promise capabilities that don't exist
+  ("we have a hosted ingest for this") unless they actually exist.
+  Phase B is honesty's purest test.
+- **Capture verbatim.** Write down their exact words on what's broken /
+  what would change their mind. The wedge-gate evidence is qualitative
+  too.
+
+### When to escalate to Drew
+
+- Partner wants something Phase D would have (hosted dashboard, multi-
+  tenant, billing). **Escalate same day** — this is the GTM signal we're
+  hunting for; Drew should hear it directly.
+- Partner is the wrong fit (technical or business) and the pairing
+  would burn both sides' time. **Pause the pairing**, debrief with Drew,
+  reschedule with a better-fit partner.
+- Substrate breaks in a way that requires a published bump. **Pause
+  the pairing**, ship the fix in a focused PR, resume.
+
+### What to capture for the wedge gate
+
+Per [`docs/design/external-agent-wedge.md`](./design/external-agent-wedge.md),
+the gate decision hinges on Phase B evidence. We capture:
+
+1. **Quantitative lift** — held-out winner composite vs baseline, per
+   scenario + overall. Auto-generated in the report artifact by the
+   canonical demo (`.phase-b-runs/<ts>/phase-b-report.md`).
+2. **Qualitative partner-validation** — partner read 3+ winner outputs
+   and confirmed they're better. Capture as a 1-paragraph quote.
+3. **Integration friction** — minutes spent on each pairing phase. Were
+   any > 2x estimated? What broke?
+4. **Judge-design surprise** — which dimensions the partner added or
+   killed vs our strawman. Strong signal about what the substrate's
+   default judge templates are missing for adjacent domains.
+5. **Soft commitments** — would they reference us? Would they
+   self-serve from the quickstart doc? Would they pay for hosted?
+
+Capture into a single `phase-b-debrief.md` per partner. We don't
+publish these; they feed the next substrate iteration + the wedge
+go/no-go.
+
+---
+
+## Failure modes — what we do NOT do
+
+### "We'll just optimize on the train set"
+
+Hard no. The held-out gate is the entire point. A win that doesn't
+generalize is worse than no win — it's evidence that the substrate
+overfits, which is the failure mode the wedge tier rewards.
+
+If the holdout lift is < threshold but train looks great:
+
+1. Show the partner the gap. Explain what overfitting means here.
+2. Try raising `maxGenerations` to 5 (gives gepa more search budget).
+3. Try widening `populationSize` to 3 (more diverse mutations per gen).
+4. If still no lift on holdout: **report the result honestly**. A
+   negative finding is real evidence for us too — tells us this surface
+   isn't amenable to prompt-only mutation, and the partner needs Phase
+   C (code-tier optimization) or a different approach.
+
+### "The judge is too noisy"
+
+A judge whose two-run variance > 0.1 on the same artifact is broken.
+Fixes, in order:
+
+1. Lower temperature to 0.0 (the canonical judge uses 0.2, which is
+   already low).
+2. Use a stronger model than the agent (default: same model. Bump the
+   judge to GPT-5.5 / Claude Opus.)
+3. Add anchors to each dimension ("0.0 = X, 0.5 = Y, 1.0 = Z").
+4. If still noisy: collapse to fewer, simpler dimensions. 3 unambiguous
+   dimensions beat 6 squishy ones.
+
+### "We can't decide what the partner's judge should be"
+
+Then we don't have Phase B. The judge IS the partner's quality bar.
+If they can't articulate it in 45 minutes of pairing, we're in the
+wrong pairing — they need to do the interview-themselves work first.
+
+**Pause the pairing, send the discovery doc again, regroup in a week.**
+
+### "Their agent is slow / expensive"
+
+`maxConcurrency: 1` and reduce scenarios to 6. Cost scales linearly;
+time scales as `(scenarios × reps × generations × population) /
+concurrency`. Tune until the loop completes in ≤ 30 min.
+
+If the per-call cost is > $1, talk to Drew before the pairing — we
+might want to subsidize the partner's first run.
+
+### "They want to share their secrets through Tangle Router"
+
+Fine — `OPENAI_BASE_URL=https://router.tangle.tools/v1` works. Make
+sure they understand: every call routes through us; the prompts and
+responses are visible to whatever observability we have on the router.
+If they want zero data leaving their network, point at their own
+endpoint, not Tangle Router.
+
+---
+
+## After the pairing
+
+### Same day
+
+- Save the `phase-b-report.md` artifact + the partner's debrief notes
+  to `~/company/design-partners/<partner>/<date>/`.
+- Send the partner a thank-you with the winner artifact + the next-
+  steps doc. Whether or not we proceed to Phase D, leave them with
+  something concrete they can ship in their product.
+- Slack Drew the verdict against the [success criteria](./phase-b-pairing-kit.md#success-criteria--what-counts-as-phase-b-passed).
+
+### Within a week
+
+- If Phase B passed: open the Phase D RFC. Reuse the partner-validated
+  judge dimensions + scenarios as the spec for what the hosted tier
+  needs to support out of the box.
+- If Phase B failed: substrate iteration ticket(s). Specific gaps the
+  pairing surfaced (judge dim defaults, doc clarity, missing helper).
+- Either way: update the wedge doc (`docs/design/external-agent-wedge.md`)
+  with the partner-name redacted + the qualitative signal.
+
+### Within a month (regardless of go/no-go)
+
+- Followup with the partner. If they're still using the lib, capture a
+  metric. If they stopped, find out why. Both data points feed product.
+
+---
+
+## The canonical demo as a forcing function
+
+`examples/marketing-agent-canonical/` is the demo we open the pairing
+with. It does three things at once:
+
+1. **Proves the substrate works** — they see a real lift on a real-
+   feeling agent before we touch their code.
+2. **Sets the bar for the judge conversation** — they react to concrete
+   dimensions, not abstract questions.
+3. **Trains us** — running the canonical demo before the pairing
+   surfaces substrate bugs on the partner's preferred model BEFORE the
+   partner is watching. We hit those bugs first.
+
+Run the canonical demo before every Phase-B pairing. It's not optional.