@tangle-network/agent-eval 0.93.0 → 0.95.0

package/docs/phase-b-runbook.md

@@ -1,176 +0,0 @@
-# Phase-B runbook (internal)
-
-How we drive a design-partner pairing. Goes alongside
-[`phase-b-pairing-kit.md`](./phase-b-pairing-kit.md) (the partner-facing
-materials) — this file is for us.
-
----
-
-## Before the pairing
-
-- **24-48h prior:** send discovery questions from
-  [`phase-b-pairing-kit.md`](./phase-b-pairing-kit.md). Don't run the
-  pairing without answers in hand. The pairing fails when we discover
-  the partner's quality bar live; we don't have time to interview AND
-  build in 4 hours.
-- **48h prior:** run the canonical demo (`pnpm tsx
-  examples/marketing-agent-canonical/index.ts`) end-to-end against the
-  partner's preferred model. Confirms the substrate + their LLM tier
-  compose. If it errors, fix the substrate before the pairing.
-- **24h prior:** mirror the partner's stack locally. If they're on
-  Cloudflare Workers, run a Worker. On LangChain, install `@langchain/*`.
-  Don't debug their tooling on the call.
-- **1h prior:** open the pairing kit, the agent-eval repo, the partner's
-  agent code/endpoint, a shared doc, and a screenshare ready.
-
-## During the pairing
-
-### Driving principles
-
-- **Talk less, ship more.** The partner is paying with their time and
-  attention; every minute we talk we aren't shipping their lift.
-- **They write the judge.** We start with our strawman so they have
-  something to react to, but the judge that ends up running is theirs.
-  This is the most-discussed seam — they should own it.
-- **No invented features.** Don't promise capabilities that don't exist
-  ("we have a hosted ingest for this") unless they actually exist.
-  Phase B is honesty's purest test.
-- **Capture verbatim.** Write down their exact words on what's broken /
-  what would change their mind. The wedge-gate evidence is qualitative
-  too.
-
-### When to escalate to Drew
-
-- Partner wants something Phase D would have (hosted dashboard, multi-
-  tenant, billing). **Escalate same day** — this is the GTM signal we're
-  hunting for; Drew should hear it directly.
-- Partner is the wrong fit (technical or business) and the pairing
-  would burn both sides' time. **Pause the pairing**, debrief with Drew,
-  reschedule with a better-fit partner.
-- Substrate breaks in a way that requires a published bump. **Pause
-  the pairing**, ship the fix in a focused PR, resume.
-
-### What to capture for the wedge gate
-
-Per [`docs/design/external-agent-wedge.md`](./design/external-agent-wedge.md),
-the gate decision hinges on Phase B evidence. We capture:
-
-1. **Quantitative lift** — held-out winner composite vs baseline, per
-   scenario + overall. Auto-generated in the report artifact by the
-   canonical demo (`.phase-b-runs/<ts>/phase-b-report.md`).
-2. **Qualitative partner-validation** — partner read 3+ winner outputs
-   and confirmed they're better. Capture as a 1-paragraph quote.
-3. **Integration friction** — minutes spent on each pairing phase. Were
-   any > 2x estimated? What broke?
-4. **Judge-design surprise** — which dimensions the partner added or
-   killed vs our strawman. Strong signal about what the substrate's
-   default judge templates are missing for adjacent domains.
-5. **Soft commitments** — would they reference us? Would they
-   self-serve from the quickstart doc? Would they pay for hosted?
-
-Capture into a single `phase-b-debrief.md` per partner. We don't
-publish these; they feed the next substrate iteration + the wedge
-go/no-go.
-
----
-
-## Failure modes — what we do NOT do
-
-### "We'll just optimize on the train set"
-
-Hard no. The held-out gate is the entire point. A win that doesn't
-generalize is worse than no win — it's evidence that the substrate
-overfits, which is the failure mode the wedge tier rewards.
-
-If the holdout lift is < threshold but train looks great:
-
-1. Show the partner the gap. Explain what overfitting means here.
-2. Try raising `maxGenerations` to 5 (gives gepa more search budget).
-3. Try widening `populationSize` to 3 (more diverse mutations per gen).
-4. If still no lift on holdout: **report the result honestly**. A
-   negative finding is real evidence for us too — tells us this surface
-   isn't amenable to prompt-only mutation, and the partner needs Phase
-   C (code-tier optimization) or a different approach.
-
-### "The judge is too noisy"
-
-A judge whose two-run variance > 0.1 on the same artifact is broken.
-Fixes, in order:
-
-1. Lower temperature to 0.0 (the canonical judge uses 0.2, which is
-   already low).
-2. Use a stronger model than the agent (default: same model. Bump the
-   judge to GPT-5.5 / Claude Opus.)
-3. Add anchors to each dimension ("0.0 = X, 0.5 = Y, 1.0 = Z").
-4. If still noisy: collapse to fewer, simpler dimensions. 3 unambiguous
-   dimensions beat 6 squishy ones.
-
-### "We can't decide what the partner's judge should be"
-
-Then we don't have Phase B. The judge IS the partner's quality bar.
-If they can't articulate it in 45 minutes of pairing, we're in the
-wrong pairing — they need to do the interview-themselves work first.
-
-**Pause the pairing, send the discovery doc again, regroup in a week.**
-
-### "Their agent is slow / expensive"
-
-`maxConcurrency: 1` and reduce scenarios to 6. Cost scales linearly;
-time scales as `(scenarios × reps × generations × population) /
-concurrency`. Tune until the loop completes in ≤ 30 min.
-
-If the per-call cost is > $1, talk to Drew before the pairing — we
-might want to subsidize the partner's first run.
-
-### "They want to share their secrets through Tangle Router"
-
-Fine — `OPENAI_BASE_URL=https://router.tangle.tools/v1` works. Make
-sure they understand: every call routes through us; the prompts and
-responses are visible to whatever observability we have on the router.
-If they want zero data leaving their network, point at their own
-endpoint, not Tangle Router.
-
----
-
-## After the pairing
-
-### Same day
-
-- Save the `phase-b-report.md` artifact + the partner's debrief notes
-  to `~/company/design-partners/<partner>/<date>/`.
-- Send the partner a thank-you with the winner artifact + the next-
-  steps doc. Whether or not we proceed to Phase D, leave them with
-  something concrete they can ship in their product.
-- Slack Drew the verdict against the [success criteria](./phase-b-pairing-kit.md#success-criteria--what-counts-as-phase-b-passed).
-
-### Within a week
-
-- If Phase B passed: open the Phase D RFC. Reuse the partner-validated
-  judge dimensions + scenarios as the spec for what the hosted tier
-  needs to support out of the box.
-- If Phase B failed: substrate iteration ticket(s). Specific gaps the
-  pairing surfaced (judge dim defaults, doc clarity, missing helper).
-- Either way: update the wedge doc (`docs/design/external-agent-wedge.md`)
-  with the partner-name redacted + the qualitative signal.
-
-### Within a month (regardless of go/no-go)
-
-- Followup with the partner. If they're still using the lib, capture a
-  metric. If they stopped, find out why. Both data points feed product.
-
----
-
-## The canonical demo as a forcing function
-
-`examples/marketing-agent-canonical/` is the demo we open the pairing
-with. It does three things at once:
-
-1. **Proves the substrate works** — they see a real lift on a real-
-   feeling agent before we touch their code.
-2. **Sets the bar for the judge conversation** — they react to concrete
-   dimensions, not abstract questions.
-3. **Trains us** — running the canonical demo before the pairing
-   surfaces substrate bugs on the partner's preferred model BEFORE the
-   partner is watching. We hit those bugs first.
-
-Run the canonical demo before every Phase-B pairing. It's not optional.

package/docs/pilot/README.md

@@ -1,62 +0,0 @@
-# Pilot Kit — customer handoff materials
-
-What's here, in order of use:
-
-| File | For | When |
-|---|---|---|
-| [one-pager.md](./one-pager.md) | Customer's first read | Send as initial pitch — what they get, why it's different, what it looks like, what it costs. Now includes intake-paths matrix for non-Tangle customers (LangChain / LlamaIndex / Anthropic SDK / OpenAI Assistants / OpenRouter / vLLM / Ollama / custom). |
-| [integration-tangle-stack.md](./integration-tangle-stack.md) | Customer's engineer (Tangle-stack customers) | Send after one-pager when they want to see the code; full integration walkthrough for the canonical Tangle stack (sandbox + tcloud) |
-| [integration-foreign-stack.md](./integration-foreign-stack.md) | Customer's engineer (non-Tangle customers) | Send after one-pager when they're on OTel, LangChain, LlamaIndex, Anthropic SDK, OpenAI Assistants, OpenRouter, vLLM, Ollama, or custom. Covers every path. |
-| [sample-insight-report.json](./sample-insight-report.json) | Customer's team meeting | Concrete JSON they can show to demonstrate value pre-integration |
-| [customer-checklist.md](./customer-checklist.md) | Pre-onboarding-call | Send 48h before the call; ensures the 90min slot is productive. Provider-agnostic — works for any stack. |
-
-## How to use this kit
-
-**For a Tangle customer asking for it RIGHT NOW:**
-
-1. Reply with the one-pager (`one-pager.md`) inline + the sample InsightReport (`sample-insight-report.json`) attached. Their senior engineer reads this and decides if it's worth a call.
-2. If they say yes, send the integration guide (`integration-tangle-stack.md`) + the checklist (`customer-checklist.md`). Schedule a 90-minute onboarding call.
-3. On the call: walk through the integration, run a live `analyzeRuns()` against their existing sandbox sessions, render the deterministic packet, fire one small `selfImprove` cycle. By the end of the call they have a working pilot.
-
-**For Drew handling the conversation himself:**
-
-The whole kit is written in our voice (technical, direct, no marketing fluff). You can paste sections directly into Slack / email / a customer call. The one-pager is meant to read as YOUR pitch, not a generic SaaS handout.
-
-## What this kit assumes
-
-- Customer is on the Tangle stack (sandbox + tcloud) OR emits OTel traces
-- Customer has an agent with a clear system-prompt addendum we can optimize
-- Customer has at least 20 scenarios their agent handles
-- Customer is willing to set a `maxUsd` budget for closed-loop campaigns
-
-If any of those don't apply, the one-pager still works as a positioning piece. The integration doc gets adapted on the call.
-
-## Where this maps in the substrate
-
-- Substrate version: `@tangle-network/agent-eval@0.53.0` (npm), `agent-eval-rpc@0.53.0` (PyPI)
-- agent-runtime version: `@tangle-network/agent-runtime@0.29.0`
-- Key APIs: `fromTangleSandbox`, `fromOtelSpans`, `analyzeRuns`, `selfImprove`, `gepaDriver`, `defaultProductionGate`, `openAutoPr`
-- All ship today; no version-blocking dependencies
-
-## What this kit doesn't yet do
-
-- No `npx @tangle-network/intelligence demo` command shipped yet (queued #115 — extend existing `tangle-intel` CLI in ADC with customer-zero-touch subcommands `init` / `demo` / `report` / `improve`)
-- No `staging-intelligence.tangle.tools` live yet (queued #116 — matches existing `staging-{product}.tangle.tools` precedent like sandbox)
-- No live demo video (queued #117 — recorded against legal-agent canonical real data)
-- No screenshot dashboard (gated on Gate 2 task #109 — ADC intelligence frontend renders canonical InsightReport)
-- No published case study with named numbers (Gate 3 task #112 — after first pilot completes 4+ cycles)
-
-## Architectural decisions baked into this kit
-
-- **Customer-facing CLI is `@tangle-network/intelligence`** (binary `tangle-intel`), NOT `agent-eval`. `agent-eval` is the substrate package; `intelligence` is the customer product that wraps it. The CLI already exists at `services/intelligence/src/cli/` in agent-dev-container — we extend it with `init` / `demo` / `report` / `improve` subcommands per task #115.
-- **Hosted URL is `staging-intelligence.tangle.tools`** matching `staging-sandbox.tangle.tools` precedent. Production becomes `intelligence.tangle.tools` once Gate 2/3 close.
-- **`agent-eval` mentioned only when customer wants direct programmatic access** (not the default path). 90%+ of customers stay at the CLI + hosted dashboard layer.
-
-For the FIRST pilot conversation, the JSON sample is the dashboard substitute. After Gate 2 lands we replace it with live screenshots.
-
-## Update cadence
-
-This kit gets updated each time:
-- A substrate version ships that customers should know about
-- A real pilot completes and we have a case study to add
-- A customer gives feedback that re-shapes how we pitch

package/docs/pilot/customer-checklist.md

@@ -1,90 +0,0 @@
-# Pre-onboarding checklist — what to have ready
-
-Send this to the customer 48h before the onboarding call. If they show up to the call having done this, the 90-minute slot ends with a working pilot.
-
-## What we need from you before the call
-
-### Credentials
-
-- [ ] **LLM provider API key** — tcloud key, OpenRouter key, OpenAI key, Anthropic key, or any OpenAI-compat router endpoint
-- [ ] **GitHub token** with PR-write access to your agent repo (optional — required only if you want auto-PR promotion on green gate decisions)
-- [ ] **Sandbox session access** (Tangle stack customers only) — read access to the session IDs we'll analyze
-
-### Data
-
-- [ ] **Trace data** — ONE of:
-  - Tangle sandbox session IDs (we use `fromTangleSandbox`)
-  - OTel spans dumped as JSONL (we use `fromOtelSpans`)
-  - Multi-rater feedback table (CSV with runId / rater / score, we use `fromFeedbackTable`)
-  - LangChain / LlamaIndex / OpenAI Assistants trace export (we use the corresponding adapter)
-  - Custom trace format (we map it together on the call — usually 20 lines of glue)
-- [ ] **Scenarios** — 20-50 representative inputs your agent handles. Even YAML / JSON / TS array is fine; we'll convert to canonical `DatasetScenario[]` shape together
-- [ ] **The system prompt addendum** your agent uses today (or whichever text surface you want to optimize) — the closed loop edits this
-
-### Judge
-
-- [ ] **A judge function or rubric** — either:
-  - An existing function `(artifact) → { composite, dimensions }`
-  - A rubric describing what "good output" means (1-2 paragraphs is enough — we'll build the judge on the call)
-  - A set of "good" / "bad" labeled examples (we use these as anchors)
-
-### Constraints
-
-- [ ] **LLM cost budget for the closed loop** — default $25 per campaign. Tell us if you want a different ceiling
-- [ ] **Cadence** — how often should the loop run? Default: weekly. Some customers want daily; others want on-demand only
-- [ ] **Deployment gate preference** — do you want:
-  - Auto-PR on `ship-substrate` (we open the PR, your team reviews)
-  - Manual review only (we report; you decide)
-  - Auto-deploy on `ship-substrate` (only with explicit ack; not default)
-
-## Call agenda — 90 minutes
-
-| Time | Topic |
-|---|---|
-| 0:00 — 0:10 | Walk through your existing setup — what runs where, what scenarios exist, what success looks like for you |
-| 0:10 — 0:30 | Pick the right intake adapter; pull traces; run `analyzeRuns()` against last week's data — first decision packet rendered live |
-| 0:30 — 0:50 | Build the judge — either wrap your existing one or scaffold a new one from your rubric |
-| 0:50 — 1:10 | Fire one `selfImprove` cycle with a small budget ($5, single generation, 2 candidates) — watch the loop run end-to-end |
-| 1:10 — 1:25 | Wire the cron + auto-PR target; schedule first weekly run |
-| 1:25 — 1:30 | Confirm what we hand back to you between runs and what reaches you when |
-
-If something on the checklist isn't ready, we adapt — just send what you have. Worst case, we spend the first 30 minutes getting unblocked.
-
-## What you'll have at the end of the call
-
-- A working `analyzeRuns()` call against YOUR live trace data, returning a real `InsightReport`
-- A judge function (yours or scaffolded) wired to your agent's output shape
-- One completed `selfImprove` cycle with a real `gateDecision` + lift CI
-- A scheduled cron / GitHub Action that runs the loop weekly
-- Optional: an auto-PR target if you want green-gate proposals to land as draft PRs
-
-## After the call
-
-- Day 1-7: first weekly run fires; we monitor + jump in if anything breaks
-- Day 7: we send you a `selfImprove`-result summary + the corresponding `InsightReport`
-- Day 14-28: 3 more cycles complete; you have enough data to evaluate the pilot
-- Day 30: pilot review — what we found, what shipped, what's next
-
-## What we send back to you between runs
-
-- The full `InsightReport` JSON (you render it however you want, or use our hosted dashboard if it's available for your tier)
-- Slack / email digest of `regressedMetrics` + critical recommendations (opt-in)
-- Cost tally per campaign
-- Auto-PR links if green gate verdicts opened any
-
-## Common pre-call questions
-
-**Q: How small a corpus can we start with?**
-A: 15 scenarios works for the deterministic packet. 25+ is recommended for `selfImprove`'s held-out gate (the default `holdoutFraction: 0.3` reserves ~30% of scenarios for the gate).
-
-**Q: What if our judge isn't reliable yet?**
-A: Start with multi-rater intake — `fromFeedbackTable` produces inter-rater agreement (κ) so you can see exactly which scenarios humans disagree on. Iterate the judge until κ > 0.7, then go to closed loop.
-
-**Q: We don't use Tangle's sandbox — can we still pilot?**
-A: Yes. We have intake adapters for OTel, LangChain, LlamaIndex, Anthropic SDK, OpenAI Assistants, OpenRouter, multi-rater feedback tables, and custom trace formats. See `integration-foreign-stack.md`.
-
-**Q: We use OpenRouter — does the closed-loop driver work with our routing setup?**
-A: Yes. `gepaDriver` accepts any OpenAI-compatible endpoint via its `llm.baseUrl` option. Most customers run their selfImprove campaigns through OpenRouter or their existing provider — no migration required.
-
-**Q: What if the pilot fails — what do we get?**
-A: You get the deterministic `InsightReport` weekly regardless. Even if no `selfImprove` cycle ever ships a green gate verdict, you get the failure-cluster analysis, regressed-metric detection, and worst-runs surfacing. Those alone replace what most teams currently get from LangSmith / Braintrust / Phoenix scorecards.

package/docs/pilot/integration-foreign-stack.md

@@ -1,296 +0,0 @@
-# Integration — Tangle Intelligence on any stack (OpenRouter, OpenAI, Anthropic, LangChain, LlamaIndex, custom)
-
-Companion to `integration-tangle-stack.md`. This doc is the path for customers NOT on `@tangle-network/sandbox` + tcloud — you bring your own agent, your own LLM provider, your own trace format. We meet you where you are.
-
-## Zero-setup demo first
-
-```sh
-npx @tangle-network/intelligence demo
-```
-
-Synthetic agent + scenarios + selfImprove end-to-end with zero setup. Prints the `InsightReport` shape — same output you'll get against your real data. Hosted equivalent: **[staging-intelligence.tangle.tools](https://staging-intelligence.tangle.tools)**.
-
-## Decision tree — pick your starting point
-
-```
-What's your trace source?
-├── OTel-compatible (Datadog APM, Honeycomb, NewRelic, raw OTLP) → fromOtelSpans
-├── LangChain (LangSmith traces, LCEL run traces)               → fromLangChain      (#104, queued)
-├── LlamaIndex (callback traces)                                → fromLlamaIndex     (#104, queued)
-├── Anthropic SDK direct (Messages API call logs)               → fromAnthropicSDK   (#104, queued)
-├── OpenAI Assistants API (run + step events)                   → fromOpenAIAssistants (#104, queued)
-├── Multi-rater human approval/reject corpus                    → fromFeedbackTable
-├── Custom (your own logs / DB rows)                            → 20-line mapper to RunRecord
-└── @tangle-network/sandbox                                     → fromTangleSandbox (see Tangle-stack doc)
-
-What's your LLM provider for the closed loop?
-├── OpenAI                          → gepaDriver({ llm: { apiKey, baseUrl: 'https://api.openai.com/v1' } })
-├── Anthropic                       → gepaDriver({ llm: { apiKey, baseUrl: 'https://api.anthropic.com/v1' } })
-├── OpenRouter                      → gepaDriver({ llm: { apiKey, baseUrl: 'https://openrouter.ai/api/v1' } })
-├── Tangle tcloud                   → gepaDriver({ llm: { apiKey, baseUrl: 'https://router.tangle.tools/v1' } })
-├── Azure OpenAI                    → gepaDriver({ llm: { apiKey, baseUrl: 'https://<resource>.openai.azure.com/...' } })
-├── Bedrock / Vertex                → custom client wrapper (we help)
-└── Self-hosted (vLLM, Ollama, etc.) → OpenAI-compat endpoint works directly
-```
-
-## OTel → InsightReport (5 minutes)
-
-```ts
-import { fromOtelSpans } from '@tangle-network/agent-eval'
-import { analyzeRuns } from '@tangle-network/agent-eval/contract'
-
-// You probably already export OTel via OTLP. Pull a JSONL dump of spans
-// for your last week of agent activity.
-const spans = JSON.parse(fs.readFileSync('./agent-traces.jsonl', 'utf8').split('\n').filter(Boolean).map(JSON.parse))
-
-const runs = fromOtelSpans({ spans })  // RunRecord[]
-const report = await analyzeRuns({ runs })
-
-console.log(report.composite.mean)
-console.log(report.recommendations)
-```
-
-What `fromOtelSpans` expects in spans (it's flexible — it tries multiple attribute keys):
-- `trace_id` (groups spans into a run)
-- `attributes['llm.tokens.in' | 'llm.input_tokens']` (optional)
-- `attributes['llm.tokens.out' | 'llm.output_tokens']` (optional)
-- `attributes['tool.name']` (optional, for tool-failure-rate analysis)
-- `status.code: 'ERROR' | 'OK'` (for failure detection)
-- `start_unix_nano` + `end_unix_nano` (for duration)
-
-If your spans use different attribute keys, pass `{ attributeMap }` to override.
-
-## OpenRouter as your closed-loop LLM provider
-
-OpenRouter speaks OpenAI-compat. Plug it in directly:
-
-```ts
-import { selfImprove, gepaDriver } from '@tangle-network/agent-eval/contract'
-
-const result = await selfImprove({
-  scenarios: yourScenarios,
-  agent: yourAgent,             // your existing dispatch — any framework
-  judge: yourJudge,
-  baselineSurface: currentPrompt,
-  driver: gepaDriver({
-    llm: {
-      apiKey: process.env.OPENROUTER_KEY!,
-      baseUrl: 'https://openrouter.ai/api/v1',
-    },
-    model: 'anthropic/claude-sonnet-4.6',     // any OpenRouter-supported model
-    target: 'agent system prompt',
-  }),
-  budget: { generations: 3, populationSize: 4, holdoutFraction: 0.3, maxUsd: 25 },
-})
-```
-
-Same shape works for: OpenAI direct, Anthropic direct, Azure OpenAI, Vertex (with their OpenAI-compat layer), Bedrock (via LiteLLM proxy), self-hosted vLLM / Ollama / LMStudio.
-
-## LangChain customer — three minute integration
-
-While the dedicated `fromLangChain` adapter is queued (#104), the universal path:
-
-```ts
-// Step 1: configure LangSmith to also export OTel
-// (LangSmith → Project Settings → Trace exports → enable OTLP)
-
-// Step 2: ingest as OTel
-import { fromOtelSpans } from '@tangle-network/agent-eval'
-const runs = fromOtelSpans({ spans: yourLangSmithOtelDump })
-
-// Step 3: analyze
-import { analyzeRuns } from '@tangle-network/agent-eval/contract'
-const report = await analyzeRuns({ runs })
-```
-
-When `fromLangChain` lands, it'll be a one-liner:
-
-```ts
-// Coming in 0.55.0
-import { fromLangChain } from '@tangle-network/agent-eval/adapters/langchain'
-const runs = fromLangChain({ traces: yourLangSmithExport })
-```
-
-Same for LlamaIndex / OpenAI Assistants / Anthropic SDK — direct adapters queued in #104.
-
-## LlamaIndex customer
-
-LlamaIndex's callback manager emits OTel spans natively. Wire it once:
-
-```python
-# Python side — your LlamaIndex setup
-from llama_index.callbacks import OpenInferenceCallbackHandler
-
-callback_handler = OpenInferenceCallbackHandler()
-# This emits to your OTel exporter
-
-# Then on the agent-eval side (TypeScript or Python):
-from agent_eval_rpc import Client
-client = Client(base_url='https://api.tangle.tools/v1', api_key=YOUR_KEY)
-report = client.analyze_runs(spans=your_otel_spans)
-```
-
-The Python client (`agent-eval-rpc@0.53.0`) speaks the same wire protocol — no functional difference between TS and Python customers.
-
-## Anthropic SDK direct (no framework)
-
-If you're calling `@anthropic-ai/sdk` directly without an agent framework:
-
-```ts
-import Anthropic from '@anthropic-ai/sdk'
-import { fromOtelSpans } from '@tangle-network/agent-eval'
-
-// Step 1: wrap your Anthropic calls to emit OTel
-import { trace } from '@opentelemetry/api'
-const tracer = trace.getTracer('your-agent')
-
-async function callAnthropic(scenario: Scenario) {
-  return tracer.startActiveSpan('agent.turn', async (span) => {
-    const result = await anthropic.messages.create({...})
-    span.setAttribute('llm.input_tokens', result.usage.input_tokens)
-    span.setAttribute('llm.output_tokens', result.usage.output_tokens)
-    span.setAttribute('tangle.runId', scenario.id)
-    span.end()
-    return result
-  })
-}
-
-// Step 2: same pipeline
-const runs = fromOtelSpans({ spans: yourOtelExport })
-const report = await analyzeRuns({ runs })
-```
-
-20 lines of OTel wrapping; the rest is pure substrate.
-
-## OpenAI Assistants API
-
-The Assistants API emits `runs.steps` events natively. Map them to RunRecord:
-
-```ts
-// Custom mapper while fromOpenAIAssistants (queued #104) lands:
-function mapAssistantRunToRunRecord(threadId: string, runId: string): RunRecord {
-  const run = await openai.beta.threads.runs.retrieve(threadId, runId)
-  const steps = await openai.beta.threads.runs.steps.list(threadId, runId)
-
-  return {
-    runId: run.id,
-    experimentId: 'default',
-    candidateId: run.assistant_id,
-    seed: 0,
-    model: run.model,
-    promptHash: hashOf(run.instructions),
-    configHash: hashOf({ tools: run.tools, model: run.model }),
-    commitSha: process.env.GIT_SHA ?? 'unknown',
-    wallMs: (run.completed_at - run.created_at) * 1000,
-    costUsd: estimateCostFromUsage(run.usage),
-    tokenUsage: {
-      input: run.usage?.prompt_tokens ?? 0,
-      output: run.usage?.completion_tokens ?? 0,
-    },
-    outcome: {
-      holdoutScore: yourScoring(run),
-      raw: { stepCount: steps.data.length, status: run.status },
-    },
-    splitTag: 'holdout',
-  }
-}
-```
-
-Once the dedicated adapter ships in 0.55.0 this becomes one line.
-
-## Custom trace format
-
-Your logs / DB rows / proprietary schema → `RunRecord`:
-
-```ts
-import type { RunRecord } from '@tangle-network/agent-eval'
-
-function mapMyRowToRunRecord(row: MyAgentLog): RunRecord {
-  return {
-    runId: row.id,
-    experimentId: row.experiment_name ?? 'default',
-    candidateId: row.model_version,
-    seed: row.random_seed ?? 0,
-    model: row.model,
-    promptHash: row.prompt_hash,
-    configHash: row.config_hash,
-    commitSha: row.git_sha,
-    wallMs: row.duration_ms,
-    costUsd: row.cost,
-    tokenUsage: {
-      input: row.input_tokens,
-      output: row.output_tokens,
-    },
-    outcome: {
-      holdoutScore: row.score,
-      raw: row.raw_output,    // free-form bag for fields the substrate doesn't standardize
-    },
-    splitTag: row.is_holdout ? 'holdout' : 'search',
-  }
-}
-
-const runs = myLogs.map(mapMyRowToRunRecord)
-const report = await analyzeRuns({ runs })
-```
-
-That's the worst case. ~20 lines of mapping, and you're in.
-
-## Multi-rater human feedback (no LLM-as-judge yet)
-
-If you don't have an automated judge but you DO have human raters approving/rejecting agent outputs:
-
-```ts
-import { fromFeedbackTable } from '@tangle-network/agent-eval'
-import { analyzeRuns } from '@tangle-network/agent-eval/contract'
-
-// Your data shape:
-const ratings = [
-  { runId: 'r-001', rater: 'alice', score: 1 },   // approved
-  { runId: 'r-001', rater: 'bob', score: 0 },     // rejected — disagreement!
-  { runId: 'r-002', rater: 'alice', score: 1 },
-  { runId: 'r-002', rater: 'bob', score: 1 },
-  // ...
-]
-
-const { runs, raterScores } = fromFeedbackTable({ ratings })
-const report = await analyzeRuns({ runs, raterScores })
-
-// report.interRater.kappa → how much your raters agree
-// report.interRater.disagreementCases → which runs raters split on
-// → use these to iterate the rubric until kappa > 0.7
-// → then build an LLM-as-judge against that aligned rubric
-```
-
-This is the warm-up path for customers who don't have a judge yet.
-
-## Hosted vs self-hosted — what's the difference?
-
-| | Self-hosted | Hosted (Tangle Intelligence) |
-|---|---|---|
-| Cost | Your LLM bills + your compute | Same LLM bills + hosted-tier subscription |
-| Dashboard | You build it (or use the OSS examples) | Renders InsightReport out of the box |
-| Cron / scheduling | Your CI / cron / GitHub Action | Managed scheduler runs weekly |
-| Slack / email digest | You wire it | Included |
-| Multi-week trends | You persist | Persisted for you |
-| Decision packet generation | Local (free) | API call (same code; we run it) |
-| Closed-loop campaigns | Local (you pay LLM directly) | Pass-through pricing on LLM, plus per-campaign fee |
-| Auto-PR | Your GitHub token | Your GitHub token via OAuth |
-
-Both work end-to-end. Hosted tier is convenience; self-hosted is fine for engineering-heavy teams who want full control.
-
-## Common foreign-stack questions
-
-**Q: We use vLLM / Ollama / a custom self-hosted LLM. Does the closed-loop driver work?**
-A: Yes if your server speaks OpenAI-compat (most do). Pass `baseUrl: 'http://localhost:8000/v1'` (or wherever) and your dummy `apiKey`. We've shipped customers running selfImprove against local LMStudio + Ollama.
-
-**Q: We're a Python shop, not TypeScript. Does anything change?**
-A: `agent-eval-rpc@0.53.0` on PyPI speaks the same wire protocol. The Python client is a thin wrapper around the hosted endpoints — same `analyzeRuns()` / `selfImprove()` calls, same `InsightReport` shape, same `gateDecision` values.
-
-**Q: We have an extremely custom agent (not LLM-call-shaped). Can we still use this?**
-A: Yes. The substrate doesn't care what your agent IS — it only cares that you can express your runs as `RunRecord[]` and your judge as `(artifact) → JudgeScore`. RL-trained agents, multi-step plan-and-execute, browser-driving agents, code-generating agents — all map cleanly.
-
-**Q: What's the minimum cost to try it?**
-A: Free. `analyzeRuns()` is deterministic, runs locally, $0 LLM cost. You can ingest your last week of traces and get a real decision packet without spending a cent. The LLM-cost-incurring step is `selfImprove()` and you set the ceiling.
-
-**Q: We don't want to send traces to your hosted tier. Self-hosted only — works?**
-A: Yes. Every primitive in this doc runs locally. The package is MIT-licensed, no SaaS lock-in, no required network call.