@tangle-network/agent-eval 0.93.0 → 0.95.0

package/docs/design/primitives-integration-spec.md

@@ -1,393 +0,0 @@
-# Self-improvement primitives — integration spec
-
-**Audience:** an engineer (or agent) wiring a product onto the Tangle
-self-improvement stack. This is the authoritative "how to use the primitives"
-reference. It is exact: every signature, every seam, every forbidden pattern.
-
-**Packages (published):**
-- `@tangle-network/agent-eval@^0.40.3` — measurement + improvement loop +
-  worktree adapter + gates + dataset store. The leaf; depends on nothing
-  upstream. Import the loop surface from `@tangle-network/agent-eval/campaign`.
-- `@tangle-network/agent-runtime@^0.25.0` — the runtime-side improvement
-  driver (`improvementDriver`) + generators (`reflectiveGenerator`,
-  `agenticGenerator`). Import from `@tangle-network/agent-runtime/improvement`.
-
-Read [`loop-taxonomy.md`](./loop-taxonomy.md) (vocabulary) and
-[`self-improvement-engine.md`](./self-improvement-engine.md) (phases) first.
-This doc is the contract-level detail under them.
-
----
-
-## 0. The one-paragraph model
-
-A **measurement** (`runCampaign`) runs your agent (behind a `dispatch` seam)
-over `scenarios`, judges the outputs, and returns a scorecard with confidence
-intervals. An **improvement loop** (`runImprovementLoop`) drives an
-`ImprovementDriver` to propose candidate **surfaces** (a prompt string, or a
-`CodeSurface` = a git worktree of code edits), measures each on a **holdout**,
-runs a release **gate**, and opens a **PR** for the winner. Every run feeds a
-**dataset** (`LabeledScenarioStore`) — the same corpus the optimizer learns
-from. Three roles, fixed meaning: **driver** decides what's next; **worker** =
-the agent in a sandbox (invoked behind `dispatch`); **measurement** runs the
-worker and scores it.
-
----
-
-## 1. The seams you implement (everything else is substrate)
-
-You implement exactly three things. The substrate owns the rest.
-
-| Seam | Type | What it is |
-|---|---|---|
-| `dispatch` | `(scenario, ctx) => Promise<TArtifact>` | invoke YOUR agent on one scenario → the artifact judges score. Topology-opaque: one LLM call, or a driver↔workers-in-a-sandbox loop — substrate doesn't care. |
-| `judges` | `JudgeConfig<TArtifact, TScenario>[]` | score an artifact on named dimensions → composite. Your rubrics. |
-| `scenarios` | `Scenario[]` | the inputs (`{ id, kind, ... }`). Your eval set. |
-
-If you are also improving a surface, you additionally provide:
-
-| Seam | Type | What it is |
-|---|---|---|
-| `dispatchWithSurface` | `(surface, scenario, ctx) => Promise<TArtifact>` | like `dispatch`, but takes the candidate surface (prompt string or `CodeSurface`) — swap it into your agent before running. |
-| a **driver** | `ImprovementDriver` | how candidates are proposed (see §4). Use a shipped one; don't hand-roll. |
-| a **gate** | `Gate` | ship/hold decision (use `defaultProductionGate`). |
-
-**You never implement:** generation loops, population/top-K selection, seed
-propagation, manifest hashing, cell caching, bootstrap CIs, worktree git
-plumbing, PR-opening, or trace capture. Reimplementing any of these is the
-anti-pattern this whole stack exists to delete.
-
----
-
-## 2. `runCampaign` — the measurement primitive
-
-```ts
-import { runCampaign, type RunCampaignOptions } from '@tangle-network/agent-eval/campaign'
-
-const result = await runCampaign<MyScenario, MyArtifact>({
-  scenarios,                 // MyScenario[]
-  dispatch,                  // (scenario, ctx) => Promise<MyArtifact>
-  judges,                    // JudgeConfig<MyArtifact, MyScenario>[]  (optional)
-  runDir: '/abs/run/dir',    // REQUIRED — where artifacts + traces land
-  seed: 42,                  // default 42 — reproducibility
-  reps: 1,                   // per-scenario replicates; raise to 5+ for tight CIs
-  maxConcurrency: 2,         // parallel cells
-  costCeiling: 5.0,          // optional USD soft-abort
-  tracing: 'on',             // default on; 'off' refused by improvement loop w/ a driver
-  labeledStore: store,       // optional capture (see §8); 'off' to disable
-  captureSource: 'eval-run', // provenance for captured rows
-})
-```
-
-Returns `CampaignResult<TArtifact, TScenario>`:
-```ts
-{
-  manifestHash: string        // sha256(scenarios, judges, dispatch ref, seed, reps) — run identity
-  seed: number
-  startedAt, endedAt, durationMs
-  cells: CampaignCellResult[] // one per scenario×rep: { cellId, scenarioId, rep, artifact, judgeScores, costUsd, cached, error? }
-  aggregates: {
-    byJudge:    Record<string, JudgeAggregate>     // { mean, stdev, ci95:[lo,hi], n } — bootstrap CIs
-    byScenario: Record<string, ScenarioAggregate>
-    totalCostUsd, cellsExecuted, cellsSkipped, cellsCached, cellsFailed
-  }
-  runDir, artifactsByPath, scenarios
-}
-```
-
-**Rules:**
-- `dispatch` must be a *named* function (`dispatch.name` feeds the manifest hash
-  — anonymous arrows weaken reproducibility identity).
-- Inspect `cell.error` before trusting `cell.artifact`. Cells fail-soft
-  individually (one bad scenario doesn't kill the run) but the error is
-  recorded, never swallowed.
-- Re-running the same `runDir` with `resumable: true` (default) skips cached
-  cells by `(manifestHash, scenarioId, rep)`.
-
-`runEval(opts)` is a thin alias for the scorecard-only case (no improvement).
-
----
-
-## 3. `JudgeConfig`, `Scenario` — the domain types you own
-
-```ts
-interface Scenario { id: string; kind: string; /* + your fields */ }
-
-interface JudgeConfig<TArtifact, TScenario = Scenario> {
-  name: string
-  dimensions: { key: string; weight?: number }[]
-  appliesTo?: (scenario: TScenario) => boolean   // scope a judge to some scenarios
-  score(args: { artifact: TArtifact; scenario: TScenario; signal: AbortSignal })
-    : Promise<JudgeScore> | JudgeScore
-}
-interface JudgeScore { composite: number; dimensions: Record<string, number>; notes: string }
-```
-
-Judges are where your rubric lives. They MUST fail loud: if the judge LLM call
-fails, throw — do not return a `composite: 0` (a fake zero is indistinguishable
-from a real zero and silently corrupts every aggregate downstream).
-
----
-
-## 4. The improvement loop — `runImprovementLoop`
-
-```ts
-import {
-  runImprovementLoop, defaultProductionGate, evolutionaryDriver,
-} from '@tangle-network/agent-eval/campaign'
-
-const result = await runImprovementLoop({
-  // --- measurement config (same as runCampaign, minus dispatch) ---
-  scenarios: trainScenarios,
-  judges,
-  runDir,
-  // --- surface improvement ---
-  baselineSurface,                 // string | CodeSurface — current best
-  dispatchWithSurface,             // (surface, scenario, ctx) => artifact
-  driver,                          // ImprovementDriver — see §5/§6
-  populationSize: 4,               // BREADTH: candidates per generation
-  maxGenerations: 3,
-  promoteTopK: 2,
-  maxImprovementShots: 3,          // DEPTH: forwarded to the driver's propose()
-  // --- gated promotion ---
-  holdoutScenarios,                // NEVER in the training pool — gate scores on these
-  gate: defaultProductionGate({ holdoutScenarios, deltaThreshold: 0.02 }),
-  autoOnPromote: 'pr',             // 'pr' | 'none'  (NO 'config' in v0.40 — throws)
-  ghOwner: 'tangle-network',
-  ghRepo: 'gtm-agent',             // required when autoOnPromote: 'pr'
-})
-// → { winnerSurface, winnerSurfaceHash, generations, baselineOnHoldout,
-//     winnerOnHoldout, gateResult, prResult? }
-```
-
-`runOptimization(opts)` is the loop body without the gate/holdout/PR (use it
-when you want candidates + a winner but will gate yourself).
-
-**Hard refusals (by design — these throw):**
-- `autoOnPromote: 'config'` → deferred to a later pass (live self-mutation
-  needs the full safety stack). Use `'pr'` or `'none'`.
-- `tracing: 'off'` while a `driver` is wired → an improvement loop that doesn't
-  feed the dataset is unattributable.
-- `autoOnPromote: 'pr'` without `ghOwner`/`ghRepo`.
-
----
-
-## 5. `ImprovementDriver` + `ProposeContext` — the contract
-
-```ts
-interface ImprovementDriver<TFindings = unknown> {
-  kind: string
-  propose(ctx: ProposeContext<TFindings>): Promise<MutableSurface[]>   // PLAN
-  decide?(args: { history: GenerationRecord[] }): { stop: boolean; reason?: string }
-}
-
-interface ProposeContext<TFindings = unknown> {
-  currentSurface: MutableSurface
-  history: GenerationRecord[]     // prior generations + scores
-  findings: TFindings[]
-  populationSize: number          // how many candidates to return
-  generation: number
-  signal: AbortSignal
-  report?: unknown                // Phase-2 research report (analyst findings + diff)
-  dataset?: LabeledScenarioStore  // handle to all captured data
-  maxImprovementShots?: number    // DEPTH knob
-}
-
-type MutableSurface = string | CodeSurface
-interface CodeSurface { kind: 'code'; worktreeRef: string; baseRef?: string; summary?: string }
-```
-
-`propose()` returns candidates; it does NOT measure (the loop measures). For a
-code-tier driver, `propose()` may itself be agentic (spawn a harness, write a
-worktree) — that's the recursion. Pick a shipped driver:
-
----
-
-## 6. The shipped drivers (use these; don't hand-roll)
-
-### `evolutionaryDriver` (agent-eval) — prompt mutation, no sandbox
-```ts
-import { evolutionaryDriver } from '@tangle-network/agent-eval/campaign'
-
-const driver = evolutionaryDriver({
-  mutator: {                       // YOUR Mutator (the only domain bit)
-    kind: 'reflection',
-    async mutate({ currentSurface, populationSize, findings, signal }) {
-      // return N prompt-string variants of currentSurface
-      return [...]
-    },
-  },
-})
-```
-Use when the surface is a **prompt string** and you have a mutation strategy
-(reflection, GEPA, AxGEPA). Cheap, deterministic-friendly.
-
-### `improvementDriver` + generators (agent-runtime) — one driver, a cost dial
-```ts
-import {
-  improvementDriver, reflectiveGenerator, agenticGenerator,
-} from '@tangle-network/agent-runtime/improvement'
-import { gitWorktreeAdapter } from '@tangle-network/agent-eval/campaign'
-
-const worktree = gitWorktreeAdapter({ repoRoot: '/abs/repo' })
-
-// cheap, no sandbox: drafts patches from findings, applies them
-const cheap = improvementDriver({
-  worktree,
-  generator: reflectiveGenerator({ improvementAdapter }), // wraps proposeFromFindings
-  baseRef: 'main',
-})
-
-// full agentic: a real coding harness edits the worktree, retries up to maxShots
-const deep = improvementDriver({
-  worktree,
-  generator: agenticGenerator({ harness: 'claude' }),     // claude | codex | opencode
-  baseRef: 'main',
-})
-```
-One driver; the generator is the cost dial. Both emit `CodeSurface`s the loop
-measures + gates. `agenticGenerator.generate()` runs the harness with
-`cwd = worktree`, trusts the **git diff** (not harness stdout) to decide
-"applied", and retries up to `maxImprovementShots` on a clean tree.
-
----
-
-## 7. Gates — `defaultProductionGate`, `composeGate`, `heldOutGate`
-
-```ts
-import { defaultProductionGate, composeGate, heldOutGate } from '@tangle-network/agent-eval/campaign'
-
-// opinionated default: heldout-delta + budget + red-team + reward-hacking + canary
-const gate = defaultProductionGate({
-  holdoutScenarios,
-  deltaThreshold: 0.02,      // winner must beat baseline by this on holdout
-  budgetUsd: 5,              // optional cost ceiling
-  redTeamBattery: [...],     // optional adversarial probes
-})
-
-// compose your own: ALL must ship, else the worst verdict wins
-const custom = composeGate(heldOutGate({ scenarios: holdoutScenarios, deltaThreshold: 0.02 }), myDomainGate)
-```
-
-`Gate.decide(ctx) → GateResult` with a 5-valued verdict:
-`GateDecision = 'ship' | 'hold' | 'need_more_work' | 'model_ceiling' | 'arch_ceiling'`.
-`composeGate` returns `ship` only if all sub-gates ship; otherwise the
-precedence is `arch_ceiling > model_ceiling > hold > need_more_work`. Use the
-non-ship verdicts to route: `need_more_work` → more data, `model_ceiling` →
-try a stronger model, `arch_ceiling` → the surface can't fix it.
-
-`openAutoPr({ result, gate, promotedDiff, ghOwner, ghRepo })` opens the PR —
-**refuses unless `gate.decision === 'ship'`**, dry-runs without a GH token.
-
----
-
-## 8. The dataset flywheel — `FsLabeledScenarioStore`
-
-```ts
-import { FsLabeledScenarioStore } from '@tangle-network/agent-eval/campaign'
-
-const store = new FsLabeledScenarioStore({ root: '/abs/dataset', maxWritesPerMinutePerBucket: 60 })
-// pass to runCampaign({ labeledStore: store, captureSource: 'production-trace' })
-```
-Every campaign cell captures `(scenario, artifact, judgeScore, source)`. This
-corpus IS the optimizer's training set. Discipline enforced at the store:
-- **provenance required** on every write (source / sourceVersionHash /
-  capturedAt / redactionStatus).
-- **temporal split**: `sample()` requires explicit `split` + `capturedBefore`.
-- **`production-trace` is excluded from the train split by default** (no
-  contamination of the holdout it's judged against).
-
----
-
-## 9. The migration recipe (what to DELETE / KEEP / REWIRE)
-
-For a product that already has eval + prompt-evolution wrappers:
-
-**DELETE (orchestration the substrate now owns):**
-- generation/population/top-K loops, trial-matrix construction, frontier
-  tracking, seed plumbing, manifest hashing, cell caching, scorecard
-  aggregation, CI math, PR-opening scaffolding, worktree git commands.
-- any local `runProductionLoop` / `runPromptEvolution` / `runAnalystLoop`
-  wrapper whose body is a loop over generations × candidates × reps.
-
-**KEEP (domain logic — it does not move):**
-- scenarios (your eval inputs) → become `scenarios`.
-- judges/rubrics/dimension weights → become `judges`.
-- the agent-invocation function → becomes `dispatch` / `dispatchWithSurface`.
-- the mutation strategy (reflection prompt) → becomes a `Mutator` or a
-  generator's `buildPrompt`.
-- domain gates (e.g. anti-fabrication) → compose with `defaultProductionGate`.
-
-**REWIRE:**
-- `buildHoldoutRunner()` → `dispatchWithSurface`.
-- `buildScorer()` → `judges`.
-- `buildMutator()` → `evolutionaryDriver({ mutator })`.
-- `runProductionLoop(...)` → `runImprovementLoop(...)`.
-- `runPromptEvolution(...)` → `runImprovementLoop` (surface = prompt string).
-- `runAnalystLoop(...)` improvement step → `improvementDriver` + a generator;
-  its findings-ledger + knowledge-graph writes stay.
-
-Net for a typical consumer: ~2,400 LOC of orchestration deleted, ~800 LOC
-rewired into the three seams.
-
----
-
-## 10. Forbidden anti-patterns (a review will reject these)
-
-1. **No silent fallbacks.** No `catch { return null }`, no `?? 0` on a judge
-   composite, no returning `false`/empty on an error you can't interpret.
-   External-boundary calls return typed outcomes or throw. A git/LLM/subprocess
-   failure is a *throw*, never a fold-into-a-default.
-2. **Don't reimplement the loop.** If you write a `for (gen of generations)`
-   that mutates + scores + selects, you've rebuilt the substrate. Stop; call
-   `runImprovementLoop`.
-3. **Don't conflate train and holdout.** Holdout scenarios never enter the
-   training pool. The gate scores on holdout only.
-4. **Don't trust harness stdout.** For code edits, the git diff is the truth,
-   not what the agent says it did.
-5. **Account for every worktree.** A created worktree is finalized into a
-   surface or discarded — never leaked, even on throw (the shipped
-   `improvementDriver` already guarantees this; preserve it if you extend).
-6. **Don't auto-deploy.** Promotion opens a PR (`autoOnPromote: 'pr'`). Live
-   self-mutation (`'config'`) is deferred behind the full safety stack.
-7. **Tracing stays on when improving.** The loop refuses `tracing: 'off'` with
-   a driver wired — the dataset must be fed.
-8. **Name your `dispatch`.** Anonymous dispatch weakens the manifest-hash
-   reproducibility identity.
-
----
-
-## 11. Minimal end-to-end skeleton
-
-```ts
-import {
-  runImprovementLoop, defaultProductionGate, evolutionaryDriver,
-  FsLabeledScenarioStore,
-} from '@tangle-network/agent-eval/campaign'
-
-const store = new FsLabeledScenarioStore({ root: '.dataset' })
-
-async function dispatchWithSurface(surface: string, scenario: MyScenario) {
-  return runMyAgent({ systemPrompt: surface, input: scenario }) // → MyArtifact
-}
-
-const judges = [{
-  name: 'quality',
-  dimensions: [{ key: 'grounding' }, { key: 'actionability' }],
-  async score({ artifact, scenario }) { /* → JudgeScore, throw on failure */ },
-}]
-
-const result = await runImprovementLoop<MyScenario, MyArtifact>({
-  scenarios: train, holdoutScenarios: holdout, judges,
-  baselineSurface: CURRENT_PROMPT,
-  dispatchWithSurface,
-  driver: evolutionaryDriver({ mutator: myReflectionMutator }),
-  populationSize: 4, maxGenerations: 3, promoteTopK: 2,
-  gate: defaultProductionGate({ holdoutScenarios: holdout, deltaThreshold: 0.02 }),
-  autoOnPromote: 'pr', ghOwner: 'tangle-network', ghRepo: 'my-agent',
-  runDir: '.runs/improve', labeledStore: store, captureSource: 'eval-run',
-})
-
-if (result.gateResult.decision === 'ship') console.log('PR:', result.prResult?.prUrl)
-```
-
-That is the whole integration. Everything not in this skeleton is substrate.

package/docs/design/product-self-improvement-loop.md

@@ -1,146 +0,0 @@
-# The product self-improvement loop — the finish-line target
-
-This is the **end state** every Tangle product agent (gtm, legal, tax, creative,
-agent-builder, blueprint, physim) converges to. It is the target the consumer
-migrations build toward — not a 1:1 port of whatever eval/improvement code a
-product has today.
-
-**Thesis.** A product agent is *one closed, automated self-improvement loop*
-that makes the agent measurably better over time while humans only approve
-PRs. A product should NOT have a "production loop" *and* a pile of `eval/*`
-CLIs *and* bespoke optimization orchestration. It has **one** loop, composed
-from the substrate. Everything else is deleted.
-
-Primitives reference: [`primitives-integration-spec.md`](./primitives-integration-spec.md).
-Engine internals: [`self-improvement-engine.md`](./self-improvement-engine.md).
-
----
-
-## The loop (7 steps, exact substrate composition)
-
-```
-1. SAMPLE the eval matrix
-   scenarios = cartesian(
-     profileVariants,        // the surface(s) under test: baseline + candidates
-     productScenarios,       // the hard product tasks (gtm: attribution honesty, …)
-     personas,               // simulated users / drivers
-   )  ∪  productionFailures  // real failures pulled from the LabeledScenarioStore
-                             // (the flywheel: prod traces become eval scenarios)
-
-2. MEASURE — runCampaign
-   dispatch(scenario) = runMultishot({           // the multi-turn challenging flow
-     persona: scenario.persona,                  //   driver = simulated user
-     profile: productAgentProfile(surface),      //   worker = the agent under test
-     shape:   scenario.flow,                     //   the real task, many turns
-     tools:   productTools,                      //   real tools, real side-effects
-   }) → transcript artifact
-   judges = product ensemble (domain dimensions) → scorecard + bootstrap CIs
-   labeledStore: capture EVERY cell (scenario, artifact, score, source) → the dataset
-
-3. ANALYZE — trace analysts (runAnalystLoop / AnalystRegistry)
-   read the campaign traces → a research report (failure modes, why, where).
-   This REPLACES bespoke "failure clustering": the analyst is the richer,
-   LLM-driven version of "what should we improve and why".
-
-4. IMPROVE — runImprovementLoop( improvementDriver + agenticGenerator )
-   driver.propose({ report, dataset, … }) → candidate surfaces.
-   The agentic generator runs a coding harness in a worktree, reading the
-   report + the codebase, making REAL product changes — prompt, tools, AND
-   code — not just an addendum string. Each candidate is measured on a
-   HELD-OUT slice of the matrix.
-
-5. GATE — defaultProductionGate (+ domain gates, composed)
-   heldout-delta + budget + red-team + reward-hacking + canary, plus any
-   product-specific gate (e.g. anti-fabrication) and an overfit-gap check.
-   Verdict ∈ ship | hold | need_more_work | model_ceiling | arch_ceiling.
-
-6. PROMOTE — openAutoPr
-   the winning worktree → a PR against the product repo. Human approves → ships.
-   (autoOnPromote: 'pr'. Live self-mutation is deferred behind the full safety
-   stack.)
-
-7. LOOP
-   the shipped, improved agent runs in production → emits traces → the dataset
-   grows → back to (1). The loop is scheduled (cron) and/or triggered when the
-   analyst report crosses a severity threshold. Autonomous between PR approvals.
-```
-
-**One entry point, no new abstractions.** A product exposes a single
-`run<Product>ImprovementCycle()` that *composes* the substrate primitives
-above. It does NOT define `runFooPromptEvolution`, `FooOptimizer`,
-`FooProductionLoop`, etc. The substrate carries every name; the product only
-wires its domain pieces into the seams.
-
----
-
-## What each product OWNS vs DELETES vs COMPOSES
-
-**OWNS (domain — stays, this is the product's value):**
-- `productScenarios` — the hard tasks the agent must handle.
-- `personas` — the simulated users that drive the multi-shot flows.
-- `judges` / rubrics / dimension weights — how "good" is defined.
-- `productTools` — the real tools the agent uses.
-- deterministic checks (anti-slop, format, forbidden-claim) — fast pre-judges.
-- domain gates (e.g. anti-fabrication) — composed into the gate.
-
-**DELETES (orchestration the substrate now owns):**
-- every `for (gen of generations)` mutate→score→select loop.
-- bespoke prompt-evolution / production-loop / analyst-loop wrappers.
-- trial-matrix construction, frontier tracking, seed plumbing, manifest
-  hashing, cell caching, scorecard aggregation, CI math.
-- PR-opening scaffolding, worktree git plumbing.
-- parallel `eval/*` CLIs that each re-implement a slice of the above.
-
-**COMPOSES (the substrate, in the one cycle):**
-- `runCampaign` (matrix measurement) · `runMultishot` (the dispatch flow) ·
-  `FsLabeledScenarioStore` (dataset) · analysts (report) ·
-  `runImprovementLoop` + `improvementDriver` + `agenticGenerator` (improve) ·
-  `defaultProductionGate` + `composeGate` (gate) · `openAutoPr` (promote).
-
----
-
-## Definition of done (a product is "at the finish line" when)
-
-1. **One cycle, one entry.** A single `run<Product>ImprovementCycle()` composes
-   the substrate; the old eval/improvement systems are deleted, not coexisting.
-2. **Matrix eval is real.** `dispatch` runs genuine multi-shot persona↔agent
-   flows with real tools — not single-turn projections, not stubbed workers
-   (non-zero token usage is asserted).
-3. **The dataset is fed.** Every cell captures to `LabeledScenarioStore` with
-   correct provenance; production failures flow back in as scenarios.
-4. **Improvement is code-real.** The agentic generator produces worktree
-   changes (prompt/tools/code), measured on holdout — not just addendum-string
-   mutation.
-5. **The gate is honest.** Composed `defaultProductionGate` + domain gates +
-   overfit-gap; fails closed; holdout never overlaps train.
-6. **Promotion is a PR.** `openAutoPr` opens it; a human approves; nothing
-   auto-deploys.
-7. **It's scheduled + triggered.** Runs on cadence and/or when the analyst
-   report crosses severity; autonomous between approvals.
-8. **Tests + a real proof run.** Contract tests assert the wiring; one real
-   end-to-end cycle produces a scorecard and (on a shipping gate) a PR.
-
-Anything short of this is mid-migration, not done.
-
----
-
-## gtm-agent — the worked instantiation (first reference build)
-
-| Loop step | gtm wiring |
-|---|---|
-| SAMPLE | profile variants of `OPERATOR_CEO_SYSTEM_PROMPT` + addendum; `GTM_LOOP_HOLDOUT_SCENARIOS` + `eval/business-owner/personas.json`; production failures from the trace store |
-| MEASURE | `dispatch` = `runMultishot(persona ↔ gtm-agent via runChatThroughRuntime, real tools)`; judges = the 3-model ensemble (`attribution_honesty`, `proposal_grounding`) + canonical 12-dim |
-| ANALYZE | trace analysts over the campaign traces → report (supersedes `FailureClusterConfig` clustering) |
-| IMPROVE | `improvementDriver` + `agenticGenerator` (claude harness) edits prompt/tools/code in a worktree, fed the report |
-| GATE | `composeGate(defaultProductionGate, antiFabricationGate, overfitGapGate)` |
-| PROMOTE | `openAutoPr` → PR against `tangle-network/gtm-agent` |
-
-**Deleted:** `eval/run-prompt-evolution.ts`, `eval/analyst-loop.ts`,
-`eval/optimization-campaign.ts`, `scripts/evals/*`, the orchestration body of
-`production-loop/index.ts` and `eval/canonical.ts`.
-**Kept:** scenarios, personas, judges, tools, deterministic checks, the
-`composeProductionLoopSystemPrompt` wiring.
-**Result:** one `runGtmImprovementCycle()`; ~3–4k LOC of scattered orchestration
-gone, replaced by a substrate composition.
-
-This gtm build is the reference the other six products copy.

package/docs/design/self-improvement-engine.md

@@ -1,140 +0,0 @@
-# The self-improvement engine
-
-How the pieces compose into a closed loop that improves an agent over time.
-This builds on [`loop-taxonomy.md`](./loop-taxonomy.md) (the role vocabulary)
-— read that first. Here we describe the *engine*: the phases, the data flow,
-and where each existing primitive plugs in.
-
-## The closed loop, by phase
-
-```
-PHASE 1 — RUN
-  driver ↔ workers (sandbox) over scenarios
-  → traces emitted → TraceStore + LabeledScenarioStore (the dataset)
-  Every run feeds the dataset regardless of why it ran (see the flywheel
-  section in loop-taxonomy.md). This is the only source of improvement signal.
-
-PHASE 2 — ANALYZE                                  ← the research report is born here
-  trace analysts run over the accumulated traces
-  (today: runAnalystLoop steps 2–4 in agent-runtime)
-    - run the analyst registry over traces → findings
-    - persist findings to the ledger
-    - diff the new findings vs the baseline → research report
-  Output: a research report = { findings, diff } grounded in real traces.
-
-PHASE 3 — PROPOSE
-  ImprovementDriver.propose(input) → MutableSurface[]
-  input carries:
-    - currentSurface      the current best surface (prompt string or CodeSurface)
-    - history             prior generations + their scores
-    - report              the Phase-2 research report (findings + diff)
-    - traces              all traces (read access) — "all the data"
-    - dataset             the LabeledScenarioStore handle
-    - populationSize      BREADTH: how many candidate surfaces to return
-    - maxImprovementShots DEPTH: how many runLoop iterations each candidate
-                          generation may take (1..MAX_IMPROVEMENT_SHOTS)
-  For the code-tier (autoresearch) driver, propose() runs a FULL sandbox
-  runLoop: a driver↔worker(s) loop that reads report+traces+codebase and
-  produces the improvement as commits in ONE worktree per candidate.
-  Output: CodeSurface{ worktreeRef }[] (or string[] for prompt-tier).
-
-PHASE 4 — MEASURE
-  each candidate → runCampaign on the holdout set
-  (checks out the candidate's worktree, runs the worker against the changed
-   code/prompt, judges, scores). The measurement is driver-agnostic.
-
-PHASE 5 — GATE + PROMOTE
-  defaultProductionGate(winner vs baseline on holdout) → ship | hold | …
-  on ship → open a PR from the winning worktree (one worktree = one PR).
-
-  ↺ loop back to PHASE 1 with the promoted surface as the new baseline.
-```
-
-The improvement loop body (`runOptimization`) owns Phases 3–4; the gated
-shell (`runImprovementLoop`) adds Phase 5. Phases 1–2 are upstream — the
-run that produces traces, and the analysts that turn traces into a report.
-
-## `propose()` — the plan step, recursively agentic
-
-`propose()` does NOT run the worker and does NOT measure. It returns N
-candidate surfaces to measure next.
-
-There is ONE driver, not several. agent-runtime ships a single
-`improvementDriver` that implements this contract and owns the candidate
-lifecycle (worktree create → generate → finalize/discard, × `populationSize`);
-it delegates the only thing that varies — *how* a candidate change is produced
-— to a pluggable `CandidateGenerator`. The generators span a cost spectrum:
-
-| Generator | mechanism | sandbox? | output |
-|---|---|---|---|
-| `evolutionaryDriver` (agent-eval) | mutate current surface text into N variants | no | `string[]` |
-| `reflectiveGenerator` (agent-runtime) | LLM drafts patches from the report → applies them | LLM call | `CodeSurface` |
-| `agenticGenerator` (agent-runtime) | a real coding harness in the worktree (≤ `maxImprovementShots`) reads report+codebase → edits in place | **yes** | `CodeSurface` |
-
-`evolutionaryDriver` is a standalone `ImprovementDriver` (pure, agent-eval).
-The reflective + agentic paths are two *generators* of the one
-`improvementDriver` — the same operation at two settings of the cost dial, not
-two separate drivers.
-
-The recursion: generating *one* candidate (the agentic generator) is itself a
-harness-agent editing a worktree, nested inside the *measurement* of that
-candidate (Phase 4), nested inside the improvement loop. "A loop whose step
-contains a loop."
-
-Two knobs, not one:
-- **`populationSize`** — breadth: how many candidates `propose()` returns.
-- **`maxImprovementShots`** — depth: how many runLoop iterations the
-  generating agent gets per candidate (N=1 → single-shot; N>1 → it can
-  iterate on its own change before handing it back to be measured).
-
-## Package boundaries (respecting the leaf direction)
-
-agent-eval is the leaf (imports nothing upstream). agent-runtime imports it.
-So:
-
-| Piece | Package | Why |
-|---|---|---|
-| `ImprovementDriver` contract | agent-eval | the shared interface; everyone implements it |
-| widened `propose()` input (report/traces/dataset) | agent-eval | part of the contract |
-| `evolutionaryDriver` | agent-eval | pure: dataset → surface, no sandbox |
-| **VCS-pluggable worktree adapter** | agent-eval | pure git/FS, no sandbox; produces `CodeSurface` |
-| `runOptimization` / `runImprovementLoop` | agent-eval | driver-agnostic loop body + gated shell |
-| `defaultProductionGate` | agent-eval | measurement-side safety |
-| **`improvementDriver`** + `reflectiveGenerator` / `agenticGenerator` | agent-runtime | needs the coding-harness runner + worktree on a real FS |
-| trace analysts / `runAnalystLoop` (Phase 2) | agent-runtime | runs agents to analyze |
-
-## The worktree adapter (VCS-pluggable)
-
-One improvement = one worktree, PR-like (multiple commits allowed). The
-adapter abstracts the VCS so the driver code is VCS-agnostic:
-
-```ts
-interface WorktreeAdapter {
-  create(opts: { baseRef: string; label: string }): Promise<Worktree>
-  // ... agent commits into worktree.path ...
-  finalize(wt: Worktree, summary: string): Promise<CodeSurface>  // → { kind:'code', worktreeRef, baseRef, summary }
-  discard(wt: Worktree): Promise<void>
-}
-```
-
-- **git** impl ships first (`git worktree add` / branch / commit).
-- **jj** ([jj-vcs](https://github.com/jj-vcs/jj)) is a candidate second impl —
-  not built now; the interface exists so it can slot in without touching
-  driver code.
-
-The measurement (Phase 4) consumes a `CodeSurface` by checking out
-`worktreeRef` before running the worker; on promotion (Phase 5) the worktree
-becomes the PR branch.
-
-## Build sequence
-
-1. **agent-eval 0.40.2** ✅: widen `propose()` input (additive optional
-   `report` / `dataset` / `maxImprovementShots`); VCS-pluggable worktree
-   adapter with a git impl.
-2. **agent-runtime 0.25.0** ✅: `improvementDriver` (the one driver) +
-   `reflectiveGenerator` (shots=1, no sandbox) + `agenticGenerator` (coding
-   harness in the worktree, ≤ `maxImprovementShots`), all fed the Phase-2
-   report. Default tracing was descoped — the flywheel's production capture is
-   already served by agent-runtime's OTEL export + `runCampaign`'s labeledStore.
-3. Wire one consumer end-to-end (Phase 4 of the broader rollout), prove it,
-   then fan out.