@tangle-network/agent-eval 0.94.0 → 0.95.0

package/docs/design/self-improvement-protocol.md

@@ -1,223 +0,0 @@
-# Self-improvement protocol — the world-class architecture
-
-**Status:** Strategic design. The artifact that every roadmap entry maps to.
-**Date:** 2026-05-27.
-
-## Thesis
-
-**Self-improvement is a protocol, not a product.** We define the wire formats, surface abstractions, driver interface, gate interface, and insight format. We ship reference implementations. Customers plug in whatever framework, model, or runtime they already use — our infrastructure handles the rigorous middle (analysis, gating, version-safe deployment).
-
-No competitor ships this combination. LangSmith / Braintrust / Phoenix / LangFuse ship tracing. Hermes ships an agent. SkillOpt ships an academic optimizer. Anthropic's Claude Code ships skill-creation. **Nobody ships the protocol.**
-
-## The pipeline as a single abstract flow
-
-```
-┌──────────────────────────────────────────────────────────────────────┐
-│  WHATEVER YOU ALREADY USE                                            │
-│  LangChain · LlamaIndex · Anthropic SDK · OpenAI Assistants ·        │
-│  Hermes · Claude Code · Codex · agent-runtime · your own stack       │
-└─────────────────────────────────┬────────────────────────────────────┘
-                                  │ traces (any format)
-                                  ▼
-┌──────────────────────────────────────────────────────────────────────┐
-│  INGEST — universal trace adapters                                   │
-│  fromOtelSpans · fromFeedbackTable · fromLangChain · fromLlamaIndex ·│
-│  fromAnthropicSDK · fromOpenAISDK · fromHermesProfileLog · BYO       │
-│  → canonical RunRecord[]                                             │
-└─────────────────────────────────┬────────────────────────────────────┘
-                                  ▼
-┌──────────────────────────────────────────────────────────────────────┐
-│  ANALYZE — analyzeRuns({ runs, baselineRuns?, userFeedback? })       │
-│  paired-bootstrap CI · Pareto · failure clusters · prior-period      │
-│  delta · user-corrective-signal extraction · recommendations         │
-│  ← THE STATISTICAL EDGE NOBODY ELSE SHIPS                            │
-└─────────────────────────────────┬────────────────────────────────────┘
-                                  ▼
-┌──────────────────────────────────────────────────────────────────────┐
-│  IMPROVE — selfImprove() closed loop                                 │
-│  gepaDriver · evolutionaryDriver · BYO ImprovementDriver             │
-│  → ProfileDiff (versioned, hashed, content-addressable)              │
-└─────────────────────────────────┬────────────────────────────────────┘
-                                  ▼
-┌──────────────────────────────────────────────────────────────────────┐
-│  GATE — defaultProductionGate (paired-CI) · BYO gate                 │
-│  ship-substrate / ship-harness / merge / inconclusive                │
-│  ← STATISTICALLY STRICTER THAN ANY COMPETITOR                        │
-└─────────────────────────────────┬────────────────────────────────────┘
-                                  ▼
-┌──────────────────────────────────────────────────────────────────────┐
-│  DEPLOY — back into WHATEVER YOU ALREADY USE                         │
-│  agent-runtime · Hermes profile log · LangChain config · custom hook │
-└──────────────────────────────────────────────────────────────────────┘
-```
-
-## The integration promise
-
-Customers pick one of three integration shapes. All three work today (some are aspirational on adapter coverage). Every shape uses the same canonical types underneath.
-
-### Shape A — offline analysis only
-
-You have traces, you want a decision packet. Zero LLM cost. Zero closed loop.
-
-```typescript
-import { fromOtelSpans, analyzeRuns } from '@tangle-network/agent-eval'
-
-const runs = fromOtelSpans({ spans: mySpans })
-const report = await analyzeRuns({ runs })
-// → InsightReport with composite, recommendations, Pareto, ...
-```
-
-Use case: dashboards, weekly post-mortems, "did anything regress" checks. The intelligence-kernel ships this.
-
-### Shape B — closed loop, your runtime
-
-You have an agent, you want to improve it. We provide drivers + gate + insight. You decide when to deploy.
-
-```typescript
-import { selfImprove, gepaDriver } from '@tangle-network/agent-eval'
-
-const result = await selfImprove({
-  scenarios,
-  agent: yourAgent,           // any function (surface, scenario) → artifact
-  judge: yourJudge,           // any function (artifact) → JudgeScore
-  baselineSurface,
-  driver: gepaDriver({ llm, model, target }),
-  budget: { generations: 3, populationSize: 4, holdoutFraction: 0.3 },
-})
-// → SelfImproveResult { baselineHash, diff, winningHash, lift, gateDecision, insight }
-```
-
-Use case: every product agent we ship. Hermes-on-our-sandbox. Claude Code with skills. Anyone wanting "ship if statistically better, else hold."
-
-### Shape C — hosted, cross-language
-
-You stream traces from anywhere, get InsightReports + selfImprove orchestration. Bills usage-based.
-
-```sh
-# Stream traces
-curl https://api.tangle.tools/v1/ingest/otel \
-  -H "Authorization: Bearer ${TANGLE_KEY}" \
-  --data-binary @traces.jsonl
-
-# Get the decision packet
-curl https://api.tangle.tools/v1/insight/${runId}
-
-# Or run a closed-loop campaign
-curl https://api.tangle.tools/v1/improve \
-  -d '{"scenarios": ..., "baselineHash": "...", "budget": {...}}'
-```
-
-Use case: Python customers, Go customers, customers behind firewalls, customers who don't want to operate the substrate.
-
-## The five non-negotiables
-
-The protocol claim only holds if all five of these survive integration. Customers shouldn't have to compromise on any.
-
-1. **Universal ingest.** Any trace format → canonical RunRecord. Coverage: OTel ✓, multi-rater feedback ✓, LangChain ⏳, LlamaIndex ⏳, Anthropic SDK ⏳, OpenAI Assistants ⏳, Hermes profile log ⏳.
-2. **Statistical rigor.** Every claim falsifiable. Paired bootstrap CI on lift, Cohen's d on effect size, MDE-aware sample-size recommendations, p-values. **SkillOpt's gate is literal `cand > current`. Hermes has no gate. Ours has all of the above.** This is the moat.
-3. **Plug-in everything.** Driver, judge, gate, intake adapter, storage all swappable. Customer brings their LLM, their judge, their scenarios. We bring the rigor.
-4. **Version-safe deployment.** AgentProfile is content-addressable. Two writers (harness + substrate) can both mutate without lost-update. Gate verdicts are scoped to baseline hash, not absolute. Tracked as #98.
-5. **Cross-language wire format.** Python client at parity with TypeScript. Hosted ingest spec versioned. Customers in any language consume the same shape.
-
-## Where we are honest about gaps
-
-| Component | Status | Customer impact when missing |
-|---|---|---|
-| `fromOtelSpans` ingest adapter | ✓ shipped 0.50.0 | — |
-| `fromFeedbackTable` multi-rater intake | ✓ shipped 0.50.0 | — |
-| `analyzeRuns` decision packet | ✓ shipped 0.50.0 / 0.50.2 actionability | — |
-| `selfImprove` closed loop | ✓ shipped 0.50.0 | — |
-| Paired-bootstrap gate | ✓ shipped early; still our edge | — |
-| `gepaDriver` reflection (not full Pareto — task #101) | ⚠ partial | OK; customers don't need Pareto until plateau hit |
-| **Prior-period comparison** in `analyzeRuns` | ✗ MISSING | "Did my last change help?" — the #1 customer question — has no rigorous answer today |
-| **User-corrective-feedback signal extraction** | ✗ MISSING | Hermes' first-class skill signal. We have the trace data. We don't mine it. |
-| **`init` CLI** scaffolding canonical eval/ layout | ✗ MISSING | Every new consumer wires it by hand; the skill describes 80 lines they have to copy |
-| **Framework-specific intake adapters** (LangChain, LlamaIndex, Anthropic SDK, OpenAI Assistants) | ✗ MISSING | Customers using these frameworks can't ingest without writing custom adapter code |
-| **Profile versioning** (task #98) | ✗ MISSING | Offline/online drift; gate verdicts can be stale by the time they're applied |
-| **Composite driver** (optimize all surfaces against one gate) | ✗ MISSING | Customers can optimize prompts OR skills, not both jointly |
-| **Empirical proof drivers work** | ✗ MISSING | We've never published "we ran gepaDriver on real customer data, here's the lift CI" |
-| Hosted-tier production launch | ⚠ in scaffolding (intelligence-kernel) | Customers must self-host today |
-
-## The roadmap — what closes each gap
-
-Mapping every roadmap entry back to a concrete protocol gap.
-
-### 0.53.0 (this session-or-next) — answer "did my last change help?"
-
-- **`analyzeRuns({ runs, baselineRuns? })`** — when `baselineRuns` is provided, the report includes a `priorPeriodComparison?` block: per-metric delta with paired-bootstrap CI, MDE-aware significance judgment, "regressed metrics" surfaced in `recommendations`.
-- Built on top of existing `diffRuns()` primitive (already shipped 0.48.0).
-- 1 PR. Pure additive surface.
-- **Customer impact**: this is the conversion question for every prospect.
-
-### 0.54.0 — extract Hermes' missing signal
-
-- **`extractUserCorrections(runs)`** — new substrate primitive. Mines user messages in traces for corrective markers (regex pass + LLM classifier for nuance). Returns `UserCorrectionEvent[]` keyed by runId.
-- `analyzeRuns({ runs, userFeedback? })` includes a "common corrections" cluster in `recommendations`.
-- Bridge to Hermes-style signal without adopting Hermes' runtime.
-- **Customer impact**: distinctive — no competitor mines this signal.
-
-### 0.55.0 — framework-specific intake adapters
-
-- **`fromLangChain(traces)`**, **`fromLlamaIndex(traces)`**, **`fromAnthropicSDK(traces)`**, **`fromOpenAIAssistants(traces)`**.
-- Each maps the framework's native trace shape to RunRecord.
-- Top 4 frameworks = 80% of agent-builder market coverage.
-- **Customer impact**: removes "we don't support your framework" friction.
-
-### 0.56.0 — `init` CLI + worked examples
-
-- `pnpm dlx @tangle-network/agent-eval init` scaffolds the canonical `eval/scenarios.json` + 3 pnpm scripts + judges template + `.runs/` directory.
-- Adds 5+ end-to-end runnable examples covering Shapes A/B/C across the 4 framework adapters.
-- **Customer impact**: time-to-first-eval drops from 4 hours to 5 minutes.
-
-### 1.0.0 — profile versioning (#98) + composite driver
-
-- Content-addressable `AgentProfileVersion` + `ProfileDiff` + 3-way merge + 4-way `DriftGateDecision`.
-- `compositeDriver` — optimize all surfaces of one AgentProfile against one gate.
-- Hermes-on-sandbox forcing function validates the work before commit.
-- **Customer impact**: production-safe; the moat is locked.
-
-### 1.1.0 — empirical-proof publication
-
-- Pick one named customer or one synthetic-realistic corpus (legal-agent canonical).
-- Run gepaDriver end-to-end with real LLM cost.
-- Publish: "n=, lift=, CI=, p=, $cost=, vs no-driver baseline."
-- One blog post, one demo video, one runnable repro.
-- **Customer impact**: every other claim becomes credible because this one is verified.
-
-## Why this design is 100x
-
-Not a 10% improvement over LangSmith. A category change.
-
-| Capability | LangSmith / Braintrust / Phoenix | Hermes / Claude Code | Tangle (target) |
-|---|---|---|---|
-| Trace ingest | ✓ proprietary | ✓ own runtime | ✓ universal |
-| Decision packet | ⚠ scorecards (no CI) | ✗ | ✓ paired-bootstrap |
-| Closed loop | ✗ | ✓ heuristic | ✓ statistically rigorous |
-| Plug-in drivers | ✗ | ✗ | ✓ |
-| Profile versioning | ✗ | ✗ | ✓ (1.0.0) |
-| Composite multi-surface | ✗ | ✗ | ✓ (1.0.0) |
-| Cross-language | ✗ | ✗ | ✓ (Python at parity) |
-| Empirical-proof publication | ✗ | ✗ | ✓ (1.1.0) |
-
-Eight rows. Nobody else has eight. We can be the only one. The work is named, scoped, and queued.
-
-## What's NOT on the roadmap (and why)
-
-- **Building our own agent runtime.** Hermes / agent-runtime / Claude Code cover that. We are infrastructure, not a runtime.
-- **Single-vendor LLM.** Substrate stays model-agnostic.
-- **UI-first product.** API-first. UIs are downstream.
-- **LangChain replacement.** Wrong layer.
-- **"Self-improvement" without a held-out gate.** Hermes and SkillOpt both ship this; we explicitly refuse — every selfImprove() requires a holdout.
-
-## Decision log — what we committed to in 0.52.0 → 1.0.0
-
-1. **`skillOptDriver` removed; behavior in `gepaDriver({ constraints })`** — 0.52.0 ✓ shipped
-2. **Honest spec docs** — 0.52.0 ✓ shipped
-3. **Profile-versioning spec with symmetric-fork framing** — 0.52.0 ✓ shipped
-4. **No V2 names anywhere** — enforced
-5. **Forcing-function gate on profile-versioning work** — Hermes-on-sandbox experiment required before phases 1-5 commit
-6. **Single-PR-per-repo discipline** — enforced 0.52.0 onwards
-7. **Prior-period comparison as 0.53.0** — committed; the customer-conversion primitive
-8. **User-feedback extraction as 0.54.0** — committed; the Hermes-signal bridge
-9. **Framework intake adapters as 0.55.0** — committed; 80% market coverage
-10. **Empirical-proof publication as 1.1.0** — committed; the credibility lock

package/docs/design/self-improvement-roadmap.md

@@ -1,106 +0,0 @@
-# Self-Improvement Substrate — Closed-Loop Roadmap
-
-Single source of truth for the agent self-improvement program: the architecture decisions, the end-to-end loop, and a completable checklist. Companion to [`self-improvement-engine.md`](./self-improvement-engine.md) (the phase diagram) — this doc is the *execution* plan.
-
-## North star
-
-One system under test. **Prod chat == eval == benchmark == self-improvement candidate** all run through the same execution path — a trading-desk backtester you also deploy live. Every improvement is *measured* against that path before it ships, on **any model**, with the full evidence vector preserved (never a scalar collapse).
-
-## The closed loop (what we are completing)
-
-```
-   RUN ───────────► OBSERVE ─────────► DIAGNOSE ─────────► PROPOSE ─────────► EVALUATE ─────────► GATE ─────────► PROMOTE ──┐
- (ExecutionEnv:   (OTLP traces,     (EYES: competing    (HANDS: competing  (evidence VECTOR:   (pluggable,    (PR / config)  │
-  chat=eval=       RunRecords)       analysts/HALO →     drivers →          per-dim judges +    versioned,                    │
-  benchmark)                         AnalystFinding[])   candidates)        Pareto frontier +   benchmarkable                 │
-        ▲                                                                   counterfactual)     POLICIES)                     │
-        └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
-```
-
-Two architectural invariants, both load-bearing:
-
-### Invariant 1 — Unify the *contract*, pluralize the *strategies*. Never collapse the vector.
-
-The temptation ("merge the two loops under one gate") is wrong if it means one scalar criterion. The correct design, mirroring a quant desk:
-
-- **Evidence Bus** — a typed, high-dimensional record. Every judge dimension, every driver's proposal, every finding, every counterfactual verdict is preserved. Backed by what we already have: per-dimension `JudgeScore`, `computeParetoFrontier` (`pareto.ts`), the per-driver `DriverComparison` rows (`compare-drivers.ts`). **Nothing is averaged away.**
-- **Proposers** — gepa, skill-opt, evolutionary, analyst-findings, HALO, memory-curation, ACE, and *whatever ships next year* — all implement one `ImprovementDriver`/proposer contract and **compete** on the bus, each proposing different changes. `compareDrivers` already runs this head-to-head; make it first-class for findings-driven proposers too.
-- **Promotion Policies** — pluggable, versioned, and themselves benchmarkable. A policy is a function over the evidence *vector* (Pareto-dominance + per-dimension significance + safety gates), **not** a scalar threshold. Different loops may run different policies; policies compete and are A/B-backtested like strategies.
-
-The only thing that is truly *shared* is the **interface** (evidence schema + proposer contract + policy contract). New loops plug in forever. The one non-negotiable: a write path (the findings loop's auto-apply) must flow through *a* policy — today it ships on a bare confidence float with none. Plurality of policies, yes; zero policy on a git-write, no.
-
-### Invariant 2 — Everything is a versioned, benchmarkable surface. Including the signatures.
-
-Prompts are already optimizable surfaces (GEPA via `analyst-surface.ts`). The **signatures/schemas themselves** (Ax I/O contract, the `AnalystFinding` schema, the kind taxonomy) are currently hand-fixed and unsearched. Three layers, staged by cost:
-
-- **(a) Fix known-wrong structure now** — empirically caught, not speculative. The `subject` grammar rejected a valid finding; the kind taxonomy had no "successful-but-suboptimal" lens. → Track A.
-- **(b) Make signatures/schemas versioned + A/B-benchmarkable** through `compareDrivers` (e.g. `findings:json[]` vs `report, findings:json[]` becomes a measured choice). → Track A.
-- **(c) Search signature *structure* + let the analyst critique its own schema** (run the trace-analyst on its own runs; a "schema-fit" finding like "I dropped a row because subject grammar X" — we already have the first such data point). Program/architecture search, expensive, discrete. → **Track B research. Must not block shipping.**
-
----
-
-## Track A — Ship (weeks). In-substrate, proven mechanism, properly tested.
-
-### A0. Analyst → model-agnostic, HALO-parity  *(the live problem; ~2–3 wks to a measured result)*
-
-Root cause (probe-verified): our Ax-RLM **fuses** reasoning + JS-sandbox + a strict typed-array emission into every turn; a weak model resolves the fused contract by emitting nothing. HALO **decouples** exploration (free-form + native tool-calls, no schema) from structuring (a deferred sub-call). Fix = decouple + recover-don't-drop + deterministic reducers (the real any-model guarantee) + the right lens.
-
-- [x] **A0.1 — Recover the dropped finding (0→1, deterministic).** Widen cluster regex to admit `.`/`_` (exclude `:` — no prefix collision); fix `deriveQuestion` to pass a task directive, not the bare kind id. *Done — `finding-subject.ts:173`, `kind-factory.ts:224`; regression tests in `finding-subject.test.ts` (dotted-subject accepted + prefixed-subjects still route); 93 analyst tests green.*
-- [x] **A0.2 — Forgiving pre-parser** — `src/analyst/parse-tolerant.ts` (`stripCodeFences`/`coerceJson`/`coerceToFindingRows`), wired into `parseRawFinding` (coerce-then-retry before dropping). *Done — 3 tests incl. the arXiv:2605.02363 fence case.*
-- [x] **A0.3 — Deterministic behavioral reducers + analyst** (the any-model keystone). `src/trace-analyst/behavioral-metrics.ts` (`computeTraceMetrics` + signal detectors) + `src/analyst/behavioral-analyst.ts` (`behavioralAnalyst`, `cost.kind:'deterministic'`, + `deriveEfficiencyFindings`). *Done — went past "reducers feeding a prompt": the behavioral class is emitted DIRECTLY by a zero-LLM analyst, so it's any-model by construction (not per-model-measured). E2E on the real `530b157_1`: **4 findings exactly matching HALO's four** (13.1× input growth, 157→75 output decay, 7× single-tool, no self-verify), `status:ok`, deterministic cost. 7 unit tests + 94 suite green.*
-- [x] **A0.4 — Two-phase: free-form `report` + structuring pass.** Ax signature → `report:string, findings:json[]`; `structureFindings` wired into `kind-factory` as opt-in recovery on an empty harvest. **Round-trip live-validated on deepseek** (the `report` populates through Ax's responder; no regression — failure-mode still emits its finding; full suite 1720 green).
-- [ ] **A0.5 — New `efficiency` kind + broaden `failure-mode`.** Non-error-scoped, seeded with the `behavioral` block (HALO's 4 diagnoses); add the `span.attributes['llm.input_tokens']` projection hint. ~2d.
-- [x] **A0.6 — Fail-loud on empty harvest.** `kind-factory` surfaces a visible `info` finding (`outcome: extraction_failed`) carrying the actor's report when the harvest is empty + the report is substantive (≥200 chars); a short/no-report empty stays empty (no false-fire). **Live-validated on deepseek:** `improvement` went from a *silent 0* to a visible report-bearing finding.
-
-**Gen-2 also shipped (not in the original A0 list):** `buildDefaultAnalystRegistry` (`src/analyst/default-registry.ts`) — the missing "default suite" primitive: always registers the deterministic `behavioralAnalyst`, adds the agentic kinds when an `ai` is supplied. Carries the **any-model CI regression gate** (≥4 behavioral findings, no LLM). Consumers stop hand-wiring `new AnalystRegistry()`.
-- [ ] **A0.7 — Reflexion retry** in agent-runtime `runAnalystLoop` (the one piece that belongs in runtime, not substrate): Evaluator checks the harvest, on `extraction_failed` appends a critique and retries once. Add a **per-analysis call-count budget** (critique: the pipeline adds ≤4 model calls vs HALO's 1). ~1–2d.
-- [ ] **A0.8 — Terminal gate (reframed, honest).** Re-run the probe on `530b157_1` with deepseek-chat **and** moonshot-v1-128k via the `efficiency` kind; assert ≥4 findings, ≥3 overlapping HALO's four. **Prove with the structurer disabled** (reducer-seeded prose alone) so "any model" is *measured*, not assumed. Wire as a `compareDrivers`/`runProfileMatrix` CI invariant; GEPA-optimize the structurer + efficiency prompts vs goldens for the compounding lift number. ~2–3d.
-
-> Honest scorecard: A0.1 is a proven 0→**1** *correctness* finding (zero overlap with HALO's 4 behavioral findings). HALO-parity (≥4) rests on A0.3+A0.5 and is a **hypothesis to measure**, not yet a result.
-
-### A1. Evidence Bus + competing pluggable policies *(Invariant 1; ~2–3 wks)*
-
-- [x] **A1.1 — Light the dead EYES→HANDS wire.** SHIPPED (PR #161): `renderAnalystEvidence(findings, report)` (severity-ranked, capped, duck-typed) wired into both `gepaDriver` (reflection prompt) and `skillOptDriver` (`findingsNote` → `buildPatchPrompt`); a mutation now targets a NAMED diagnosed root cause, not just low-scoring trials. Loop closure: `runOptimization` gained `analyzeGeneration` — after each generation's candidates score, the producer re-diagnoses and the fresh findings REPLACE `ctx.findings` for the next generation's `propose()` (generic — no analyst import; consumer plugs its registry/HALO). 6 deterministic tests (each driver injects findings+report; producer feeds gen N→N+1; no-producer path stays static). *Remaining (live):* the `compareDrivers` findings-fed vs findings-blind lift CI — an LLM run, not deterministic-shippable.
-- [x] **A1.2 — Promotion-policy contract.** SHIPPED (PR #161): `buildEvidenceVector` (the bus) + `PromotionPolicy = (ev) => GateResult` + `paretoPolicy` (symmetric multi-objective — every axis is gain-source AND floor; ship iff candidate weakly dominates baseline at the confidence level; floor breach → hold, insufficient evidence → need_more_work, no-op → hold) + `paretoSignificanceGate` (bus + policy as a `Gate`, composes via `composeGate`, default loop unchanged). One `contributingGate` per axis — no scalar collapse. `policy?` override runs competing strategies over one bus. Reuses `pairHoldout`/`pairedBootstrap`/`detectScale`. 9 deterministic tests. Policy *registry* + auto-A/B-backtest deferred (needs a labeled promote/rollback corpus).
-- [ ] **A1.3 — Route the findings-loop write path through a policy.** agent-runtime `run-analyst-loop.ts` auto-apply flows through a substrate-exported policy instead of the bare confidence float. (Runtime calls down — no upward dep.)
-- [ ] **A1.4 — MAST-typed findings.** `analyst/types.ts` `area:string` → a MAST-aligned enum reconciled with `FailureClass` (drop multi-agent-only modes). Counterfactual verification (extend `counterfactual.ts` → `verifyFindingCausally`) on a *sampled/triaged* subset, not every finding (frontier LLMs attribute <10% zero-shot — budget it).
-
-### A2. More drivers *(fast wins; days each)*
-
-- [x] **A2.1 — ACE driver** SHIPPED (PR #161, `src/campaign/drivers/ace.ts`): delta-structured append-mostly playbook — appends provenance-tagged `- [gN] <lesson>` deltas, preserves existing verbatim (anti context-collapse, arXiv:2510.04618), idempotent on recurring findings, FIFO-evicts oldest on overflow (never merges), deterministic, fail-loud. Reuses the shared finding→text helpers (`_findings-text.ts`; memory.ts refactored onto them). 7 deterministic tests. *Remaining (live, same plateau as A1.1):* the `compareDrivers`-on-AppWorld CI'd head-to-head vs gepa/memory — needs the AppWorld env + an open model. Deferred the formal `playbook` MutableSurface kind (string surface suffices for the append-only mechanism).
-- [ ] **A2.2 — `majorityVoteReward`** (`rl/verifiable-reward.ts`): TTRL consensus pseudo-labels over best-of-N fanout, **verifiable subset only** (tax by-line, AppWorld — out of scope for rubric domains), through `heldoutSignificance` + the reward-hacking guard. ~2–3d.
-
-### A3. Execution-Environment seam *(the keystone; ~3–4 wks, staged, highest blast radius)*
-
-The one-SUT precondition. Specced at `agent-spine.md:58`, never built (zero code refs). `src/loops` and `src/conversation` share zero code today (loop kernel lacks the journal/idempotency/circuit-breaker the conversation kernel has).
-
-- [ ] **A3.1 — Define `ExecutionEnvironment`** (`agent-runtime/src/loops/execution-environment.ts`): `{ tools(), invoke(call), workspace, artifacts() }` with `WorkerEnv`/`SandboxEnv`/`DispatchEnv` impls. Additive; `runLoop` accepts an env, default preserves current path.
-- [ ] **A3.2 — Route eval + benchmark + self-improve through it** (lowest-risk paths first). *Gate:* one AppWorld run and one self-improve campaign through the **same** `DispatchEnv` → identical `RunRecord` shape + trace correlation.
-- [ ] **A3.3 — Lift the conversation kernel's distributed primitives** (`journal-sql.ts`/`turn-id.ts`/`call-policy.ts`) into the shared layer; migrate the conversation kernel **last**, behind a deletable shim once parity is proven.
-
----
-
-## Track B — Research (months, uncertain). Flagship bets; must NOT block Track A.
-
-- [ ] **B1 — Logprob-free causal step credit-assignment** (`rl/belief-delta.ts`): ΔP(success) via ablate-and-replay through the opaque `DispatchFn` (no logprobs — the regime everyone ships on; `process-reward.ts:29-33` concedes the gap). Publish a labeled attribution benchmark + leaderboard. *Risk:* replay non-determinism vs N; tool side-effects break replay. SOTA: AgenTracer (2509.03312).
-- [ ] **B2 — Self-distilled small models** (offline): an RLM trace-analyst + a causal-attribution model trained on our corpus exhaust (traces + outcome truth + causally-verified MAST-typed findings). Removes the frontier-root-LM dependency → kills 0-findings brittleness at the root. *Gated on:* B1 producing enough labels + a data-governance/redaction story. SOTA: AgenTracer-8B beats Gemini-2.5-Pro +18%.
-- [ ] **B3 — AZR self-play curriculum with a published anti-reward-hacking guarantee** (`drivers/proposer.ts`): a driver that *emits* scenarios, scored by learnability variance, gated by the reward-hacking detector + bootstrap-CI. The *guarantee* (non-collapse over cycles) is the research deliverable, not a mitigated risk. SOTA: Absolute Zero (2505.03335), TTRL.
-- [ ] **B4 — Signature-structure search + analyst-self-schema-critique** (Invariant 2, layer c): a meta-driver that proposes signature/schema variants and benchmarks them; a "schema-fit" analyst kind that critiques the finding schema. Architecture search — expensive, discrete.
-- [ ] **B5 — Trainable rubric reward models as product** (`rl/rubric-reward-export.ts`) + a Vertical Reward-Model leaderboard. Hypothesis: production-grounded GenRMs beat synthetic OOD. SOTA: GenRM (2410.12832).
-- [ ] **B6 — Verified Vertical RL-Environment Exchange** (`exportVerifiersEnvironment`): wrap `MultiLayerVerifier` as a PrimeIntellect-verifiers Environment so every product agent auto-publishes a verified vertical RL env. *Blocked on legal/compliance for regulated traces (see Out of Scope).*
-
----
-
-## Out of scope (now) — not an engineering sprint
-
-- Reselling/open-weighting on regulated tax/legal traces: PII/privilege/consent/membership-inference. Legal + compliance gated. B2/B6 must train only on a redacted, consented, gold-ladder-admitted slice; the de-identification pipeline for financial/legal artifacts is itself unsolved.
-
-## Research difficulty index
-
-| Easy (days) | Medium (1–4 wks) | Hard / research (months) |
-|---|---|---|
-| A0.1✓ A0.2 A0.3 A0.5 A0.6 A2.1 A2.2 | A0.4 A0.7 A0.8 A1.* A3.* B5 | B1 B2 B3 B4 B6(+legal) |
-
-## Verification discipline (every box)
-
-Real e2e (mock only at process boundaries), name the regression each test catches, extend the existing suite, assert exact shapes/CIs, adversarial inputs, fail-loud. The `530b157_1` probe is the analyst-track instrument throughout; `compareDrivers`/`runProfileMatrix` is the CI forcing-function for every driver and the any-model parity gate.

package/docs/design/substrate-gaps.md

@@ -1,118 +0,0 @@
-# Substrate gaps — design-partner readiness
-
-What's missing from `@tangle-network/agent-eval` substrate (this repo) and `~/code/agent-dev-container/products/intelligence/` (the orchestrator) to credibly hand to a first design partner.
-
-This doc is the engineering-side mirror of `~/company/gtm/experiments/2026-05-27/design-partner-readiness.md` — gtm tracks the partner-facing readiness, this tracks the code that backs each bar.
-
-## Current substrate state
-
-Shipped in v0.47:
-- `selfImprove({ scenarios, judges, dispatch, hostedTenant })` one-shot helper
-- `defaultProductionGate(deltaThreshold)` autonomous-ship gate
-- Wire format frozen at `HOSTED_WIRE_VERSION = '2026-05-26.v1'`
-- `/hosted/client.ts` — bearer auth, idempotency, bounded retries on 5xx/408/429
-- `examples/hosted-ingest-server/` — reference receiver implementing the spec
-- `docs/hosted-ingest-spec.md` — semver-locked wire spec
-- `docs/design/phase-d-rfc.md` — scope decisions + deferred items
-- `docs/quickstart-external.md` — foreign-agent quickstart
-- `docs/phase-b-pairing-kit.md` — partner discovery script
-- `adapters/langchain` + `adapters/http`
-
-## Engineering gaps keyed to first-partner readiness
-
-### Substrate gaps (this repo)
-
-**S1. TraceAI/OTel adapter (`adapters/traceai`).**
-Future AGI's `traceai` library is the strongest OTel-native instrumentation in the TS ecosystem. Partners using it should be able to wire its emitted spans into our hosted ingest via one config line. The adapter receives OTel spans, normalizes them to `TraceSpanEvent`, ensures the `tangle.runId` attribute is present, and forwards via the existing hosted client.
-
-Path: `src/adapters/traceai.ts`. Export from `tsup.config.ts`. Add to `docs/adapters-observability.md`.
-
-Estimate: 6-8h. Owner: claude. Priority: medium (Tier C in partner-readiness — defer until first partner asks, but pre-build the contract).
-
-**S2. Run-diff data primitive.**
-The orchestrator needs to render "v3 vs v4" comparisons. The substrate should expose a `diffRuns(runA: EvalRunEvent, runB: EvalRunEvent): RunDiff` helper that computes cell-by-cell judge-score deltas, artifact-text diff (using a stable diff algorithm), and lift summary. Without this, every consumer rebuilds the diff logic.
-
-Path: `src/contract/diff.ts`. Add to `/contract` entry.
-
-Estimate: 4-6h. Owner: claude. Priority: high (orchestrator's run-diff view depends on this).
-
-**S3. Sampling controls in hosted client.**
-The Phase D RFC flags trace sampling as an open question. Add `sampling: { traces: number /* 0-1 */ }` to `createHostedClient` options. Default 1.0. Document the cost implication. Reservoir-sample if over budget.
-
-Path: `src/hosted/client.ts`. Update `docs/hosted-ingest-spec.md` accordingly.
-
-Estimate: 2h. Owner: claude. Priority: medium (Tier B partner-readiness).
-
-**S4. Auto-instrumentation library (`@tangle-network/agent-eval/auto`).**
-LangSmith's `@traceable` decorator + auto-wrap of OpenAI/Anthropic SDKs is their highest-leverage adoption tool. Build the equivalent: a `traceable()` HOF that emits OTel spans with `tangle.runId` attribute and forwards via the hosted client. Optional auto-wrap of `OpenAI` / `Anthropic` SDK clients.
-
-This is the biggest unlock for non-LangChain TS partners. Defer until at least one partner asks — pre-shipping costs 16-20h and may be the wrong shape without partner signal.
-
-Path: `src/auto/index.ts` (new entry). Estimate: 16-20h. Owner: claude. Priority: low (Tier C — defer).
-
-**S5. Surface-confidentiality option in wire format.**
-RFC open question 1. Add `surfaceMode: 'verbatim' | 'hashed' | 'omitted'` to `selfImprove` config. When `'hashed'`, ship `surfaceHash` instead of `surface` on the eval-run event. When `'omitted'`, ship neither.
-
-This is partner-shaped — wait for the first conversation. But the wire format should accommodate it without a breaking version bump. Add the optional field to types now.
-
-Path: `src/hosted/types.ts` (add `surfaceHash?: string`). Estimate: 1h to land the type change; later partner work to wire selfImprove. Priority: low until asked.
-
-### Orchestrator gaps (`agent-dev-container/products/intelligence/`)
-
-These are the gtm-doc's Tier A items rephrased for engineering tracking.
-
-**O1. Adversarial tenant-isolation test suite.**
-`tests/auth.test.ts` exists. Need `tests/isolation.test.ts` covering: cross-tenant header mismatch, cross-tenant `/v1/runs/:id` reads, webhook tenant scoping, idempotency-key tenant scoping, raw-SQL cross-tenant query, JWT replay after revocation. Use `VITEST_INTEGRATION=1` with a real Postgres in CI.
-
-Estimate: 4-6h. Owner: claude. Priority: **critical** (blocker for any partner conversation).
-
-**O2. Web dashboard MVP.**
-List runs + run detail + login. See gtm doc A2 for shape. Pages: `/login`, `/runs`, `/runs/:id`, `/keys`. Use `intelligence-web` Vite scaffold; wire to `/v1/runs*` reads.
-
-Estimate: 12-16h. Owner: claude. Priority: **critical** (without UI, partner can't show anyone in their org).
-
-**O3. Free-tier plan limits enforcement.**
-`lib/plans.ts` defines limits. `routes/ingest.ts` does not enforce them. Add per-tenant counters (eval-runs/mo, trace-spans/day), check against plan, return 429 with clear message + reset time on exceed.
-
-Estimate: 3-4h. Owner: claude. Priority: medium (Tier B partner-readiness).
-
-**O4. Stale README sweep.**
-`api/README.md` lists T0-3..T0-8 as "next/pending" when 5 of them are shipped. Broken link to `../../../docs/intelligence-product-rfc.md` (gitignored path; should be `../RFC.md`).
-
-Estimate: 30min. Owner: claude. Priority: **must-fix-now** (5-min job, awful first impression for anyone reading the repo).
-
-**O5. Onboarding partner-facing doc.**
-Engineer-shaped `quickstart-external.md` exists in this repo. Partner-facing 10-minute walkthrough does not. Lives at `intelligence.tangle.tools/docs` once provisioned; for now, write at `products/intelligence/docs/partner-onboarding.md`.
-
-Estimate: 2-3h. Owner: claude. Priority: high (Tier A — needed for any partner call).
-
-## Recommended sequencing (engineering view)
-
-**Sprint 1 — partner-ready (≤ 1 week):**
-- O4 (README sweep) — ship today
-- O1 (isolation tests) — 4-6h
-- O2 (dashboard MVP) — 12-16h
-- O5 (partner onboarding doc) — 2-3h
-- S2 (run-diff primitive) — 4-6h (substrate side of O2 follow-up)
-
-Total: ~25-32h focused work. One engineer-week.
-
-**Sprint 2 — concurrent with first partner conversations:**
-- O3 (plan limits enforcement) — partner will hit it
-- S3 (trace sampling) — partner will ask about cost
-
-**Sprint 3 — after first partner ships to prod:**
-- S1 (TraceAI adapter)
-- Stripe billing wire-up in orchestrator
-
-**Holding for partner signal:**
-- S4 (auto-instrumentation library) — 16-20h speculative without ask
-- S5 wiring (surface confidentiality) — partner-shaped
-
-## Cross-references
-
-- `docs/design/phase-d-rfc.md` — substrate scope decisions (this doc operationalizes its "what's deferred until Phase B")
-- `docs/hosted-ingest-spec.md` — wire format spec (any change here is a wire-version bump)
-- `~/company/gtm/experiments/2026-05-27/design-partner-readiness.md` — partner-facing readiness, mirrors this
-- `~/company/gtm/products/tangle-intelligence.md` — product hub
-- `~/company/gtm/competitor-analysis/agent-improvement.md` — competitive frame

package/docs/phase-b-pairing-kit.md

@@ -1,188 +0,0 @@
-# Phase-B partner pairing kit
-
-Everything we hand a design partner — the pitch, the discovery doc,
-the judge worksheet, the 4-hour pairing agenda, the success criteria.
-
-> This file is **partner-facing**. The internal driving runbook is in
-> [`phase-b-runbook.md`](./phase-b-runbook.md).
-
----
-
-## The pitch (one-pager)
-
-You have a working agent. You don't have evals. You don't have a
-self-improvement loop. You don't know which prompt change actually
-made the agent better last week.
-
-We have all of that on a shelf — same engine our six internal product
-agents use in production. It's open source, free at the LAND tier, and
-sandbox-free if you don't want our sandbox.
-
-**The Phase-B offer:** in one 4-hour pairing, we wrap your agent
-behind our `Dispatch`, author your domain-specific judge with you,
-and run one real campaign + improvement loop on **your actual use
-case**. You walk away with:
-
-- A reproducible eval harness against scenarios you control.
-- A judge that scores your outputs on dimensions you defined.
-- One measurable lift on your real product, with a held-out gate.
-- Trace artifacts you own (locally on disk; nothing leaves your
-  network unless you point at our hosted tier).
-
-What we get: design-partner evidence the substrate works on a foreign
-agent we did not build. That validates the wedge for us. Nothing else
-changes hands.
-
-**Cost to you:** 4 hours of pairing + your LLM bill for the campaign
-run (typically $5-$50 depending on model + scenario count). No
-commitment, no contract, no exclusivity. We don't take your code, your
-data, or your secrets.
-
----
-
-## Discovery questions (15 min, before the pairing)
-
-Send these to the partner ahead of the pairing so they walk in with
-their answers.
-
-### About the agent
-
-1. What does your agent **do** — one paragraph, end-user perspective?
-2. What's the **input** it accepts and the **output** it produces?
-   (Schemas help; English is fine.)
-3. What framework / stack? (LangChain / Mastra / OpenAI Agents SDK /
-   bespoke / something else.)
-4. Where does it run? (Local node / serverless / your sandbox /
-   browser / mobile / other.)
-5. What model(s) does it use today? Any model-routing layer
-   (OpenRouter, Portkey, your own)?
-
-### About quality
-
-6. How do you currently know your agent is good? (Eyeballing /
-   user feedback / metrics / nothing yet — all fine answers.)
-7. What does a **bad** output look like for you? Give 2-3 concrete
-   examples. Be specific.
-8. What does a **good** output look like? Same.
-9. Are there outputs that are *technically correct but feel wrong*?
-   What's the signal?
-10. How would a senior person on your team **score** an output, if
-    they had to give it a 1-10? Walk us through the rubric they'd
-    use, even informally.
-
-### About the loop
-
-11. If we could improve one thing about the agent in 4 hours, what
-    would move the needle the most for you?
-12. Are there *prompt* changes you've wanted to try but haven't had
-    the loop to validate?
-13. Anything you've explicitly tried that **didn't** work? (Saves us
-    suggesting it.)
-
----
-
-## Judge-design worksheet (45 min into the pairing)
-
-The judge is the most under-discussed piece of an eval system. Most
-projects fail at the judge, not the agent.
-
-We start with a **strawman** — the 6 dimensions in our canonical
-marketing-quality judge:
-
-| Dim | What it measures |
-|---|---|
-| hook_strength | Opens with concrete user outcome, not category |
-| voice_match | Reads human-written; no AI slop |
-| cta_clarity | Next step unambiguous for the audience |
-| factual_grounding | Only claims things the brief supports |
-| surface_fit | Length + register correct for medium |
-| audience_specificity | Vocabulary the audience actually responds to |
-
-**Your job in this 45 min:** rip this apart. We expect:
-
-- **2-3 of these are wrong for you.** Replace them.
-- **2-3 dimensions are missing.** Add them. (E.g., "tone matches our
-  brand book" or "safety-critical claim has a citation" or "answer is
-  decisive — no hedging when the user wants a recommendation".)
-- **Weights are wrong.** For your use case some dims matter 5x more.
-
-The deliverable: a judge with 4-8 dimensions, each scored 0.0 - 1.0,
-each unambiguous enough that two independent humans would score the
-same artifact within 0.1.
-
-If a dimension is squishy, throw it out. A noisy judge poisons the
-loop.
-
----
-
-## The 4-hour pairing agenda
-
-### Hour 1 — Discovery + Dispatch wiring
-
-| Time | What | Deliverable |
-|---|---|---|
-| 0:00 - 0:15 | Review discovery answers, align on scope | Shared doc with goals + constraints |
-| 0:15 - 0:45 | Wire `Dispatch` around their agent — typically 1 function | Working `Dispatch<TScenario, TArtifact>` |
-| 0:45 - 1:00 | Run 1-2 scenarios through `Dispatch` manually; see real artifacts | Confirmed wire shape |
-
-### Hour 2 — Judge calibration
-
-| Time | What | Deliverable |
-|---|---|---|
-| 1:00 - 1:45 | Walk through the strawman judge; redesign dimensions with the partner | Final `JudgeConfig` for their domain |
-| 1:45 - 2:00 | Calibrate judge against the 2 manual outputs from Hour 1 | Confirmed judge gives same scores a human would |
-
-### Hour 3 — First campaign + tuning
-
-| Time | What | Deliverable |
-|---|---|---|
-| 2:00 - 2:30 | Define 8-15 scenarios with the partner (or use ours as a template) | Scenario set with train + holdout split |
-| 2:30 - 3:00 | Run `runEval` for baseline; review per-scenario scores | Baseline score + identified failure modes |
-
-### Hour 4 — Improvement loop + go/no-go
-
-| Time | What | Deliverable |
-|---|---|---|
-| 3:00 - 3:30 | Configure `runImprovementLoop` with `gepaDriver` (3 generations, population 2) + `defaultProductionGate` | Improvement run completes |
-| 3:30 - 3:50 | Walk the partner through the gate decision + lift per scenario | Report artifact |
-| 3:50 - 4:00 | Capture: was the lift real? Would they ship the winner? Will they keep using the lib? | **Go/no-go signal for Phase D** |
-
-If we're tracking ahead at any hour, use the slack to deepen — add a
-red-team battery, swap the judge model, run more generations. If we're
-behind, cut the scenario set to 6 and ship.
-
----
-
-## Success criteria — what counts as Phase B passed
-
-For us to greenlight Phase D (hosted orchestrator + metered billing),
-we need ALL of:
-
-1. **Real lift.** Held-out winner score > baseline by ≥ 0.05 composite
-   points (or the partner's chosen threshold). Not just train; held-out.
-2. **Partner-validated lift.** The partner reads the winner output on
-   3+ held-out scenarios and confirms it's actually better.
-3. **Integration time ≤ 1 day.** Discovery + wiring + judge took ≤ 4
-   hours for the pairing; partner could reach the same point solo in
-   ≤ 1 day from the quickstart doc.
-4. **Public commitment.** Partner agrees to a public reference (case
-   study / quote / logo) OR commits to running the LAND tier in their
-   own product within 2 weeks.
-
-3-of-4 = soft pass (revisit Phase D scope but proceed). 4-of-4 = hard
-pass (build Phase D). ≤ 2 = fail (back to substrate iteration).
-
----
-
-## What we don't ask for
-
-- Your code. Wire `Dispatch` around your existing API; we never see the
-  source.
-- Your customer data. Use synthetic scenarios or anonymized real ones —
-  whichever you prefer.
-- Your model keys. You bring your own; if you want, route through Tangle
-  Router and we never see the prompts either.
-- Exclusivity, commitment, or contract. Walk away whenever.
-
-The point is to learn if the substrate works for someone we didn't
-build it for. That's it.