@tangle-network/agent-eval 0.123.1 → 0.123.3

package/docs/improvement-glossary.md

@@ -1,204 +0,0 @@
-# Improvement glossary + proposer chooser + composition
-
-> **In plain terms:** this is the dictionary for the *improvement* half of the stack — the words that show up when you optimize an agent (proposer, surface, candidate, generation, holdout, lift, gate…) rather than when you *run* one (driver, worker, iteration — those live in [`agent-runtime/docs/glossary.md`](../../agent-runtime/docs/glossary.md)).
-> Read this once and you can read any improvement result, pick a proposer, and wire two of them together.
-
-**Who this is for.**
-A **novice** (never seen the repo) should be able to read a `CampaignResult`, a `ProposeContext`, and a proposer chooser table without opening the source.
-An **expert** who knows DSPy/GEPA should be able to map their existing mental model onto our names in about a minute — every core term below carries a *"if you know DSPy/GEPA"* line.
-If code and this file disagree, the code wins — fix this file the same turn (the anti-staleness law).
-
-Neighbors, so this page does not duplicate them: [`concepts.md`](./concepts.md) (eval mental model), [`self-improvement-map.md`](./self-improvement-map.md) (one loop / four roles / proposer catalog), [`campaign-proposers.md`](./campaign-proposers.md) (proposer ELI5), [`eval-surface-map.md`](./eval-surface-map.md) (which `run*` primitive), [`design/loop-taxonomy.md`](./design/loop-taxonomy.md) (execution vs proposer layering).
-
-## The 60-second mental model
-
-One loop improves one **surface**.
-Every generation: a **proposer** reads what failed and emits **candidate** surfaces; a **campaign** measures each candidate by running the agent over **scenarios × reps** and **judging** every run; the loop ranks candidates and re-scores the best on a **holdout**; a **gate** decides whether that beat the **baseline** for real; if yes, it is **promoted**.
-
-```text
-baseline surface
-   └─► PROPOSER: read findings ─► N candidate surfaces          (one GENERATION)
-          └─► CAMPAIGN: run agent on scenarios × reps ─► JUDGE each ─► composite
-                 └─► rank candidates ─► re-score best on HOLDOUT
-                        └─► GATE: did it beat baseline (significance)?  ship / hold
-   repeat for maxGenerations
-```
-
-## Glossary — one plain sentence each
-
-Grounded to `agent-eval/src/campaign/types.ts` unless noted.
-
-| Term | Plain sentence | If you know DSPy/GEPA |
-|---|---|---|
-| **surface** | The one thing being changed this run — a prompt string, a JSON config string, or a finalized code candidate (`MutableSurface = string \| CodeSurface`, `types.ts`). | The optimized artifact: a signature/predictor's instruction text, or the module config. |
-| **proposer** | The strategy that, given the current surface + what failed, proposes the next batch of candidate surfaces to measure — it does **not** run the agent or score anything (`SurfaceProposer.propose`, `types.ts:286`). | The optimizer / teleprompter (MIPRO, BootstrapFewShot, GEPA's reflective proposer). |
-| **candidate** | One proposed surface plus its human `label` and `rationale`, ready to be measured (`ProposedCandidate`, `types.ts:166`). | One trial instruction/program the optimizer wants to evaluate. |
-| **generation** | One round of *propose → measure → rank → promote*; the loop runs up to `maxGenerations` of them (`GenerationRecord`, `types.ts:549`). | One GEPA iteration / optimization step. |
-| **populationSize** | BREADTH — how many candidate surfaces the proposer returns *this* generation (`ProposeContext.populationSize`, `types.ts:237`). Paired with `maxGenerations` (DEPTH) as the search budget. | Beam width / number of minibatch candidates per step. |
-| **population budget** | Informal name for the pair `{ populationSize, maxGenerations }` — the total candidates the search may evaluate (breadth × depth). | Optimizer trial budget. |
-| **campaign** | One measurement: run a dispatch over scenarios × seeds × reps, judge each output, aggregate → `CampaignResult` (`runCampaign`; a "campaign" = a coordinated batch of measurements). | One evaluation pass of a candidate over a valset. |
-| **cell** | The atomic measurement unit — exactly one `(scenario, rep)` execution producing one artifact + its judge scores + its cost (`CampaignCellResult`, `types.ts:509`; `DispatchContext.cellId/rep`). | One (example, seed) evaluation datapoint. |
-| **scenario** | One input case with a stable `id` + `kind` (consumers attach their payload: persona, task, requirement) (`Scenario`, `types.ts:23`). | One dataset example / `dspy.Example`. |
-| **rep** | The repetition index — the same scenario run more than once so noise/variance is measurable rather than mistaken for signal (`ctx.rep`). | Repeated sampling of the same example (temperature/seed variance). |
-| **judge** | A scorer: given an artifact, return dimensions + a single `composite` + free-form `notes`; it throws on failure rather than silently scoring 0 (`JudgeConfig` → `JudgeScore`, `types.ts:94/116`). | The metric function, but pluggable (LLM-judge, deterministic checks, or an ensemble). |
-| **composite** | The single 0..1 number that combines all judge dimensions — the number you rank and gate on (`JudgeScore.composite`, `types.ts:116`). | The scalar metric value. |
-| **findings** | The failure analysis handed to the proposer so it edits from evidence, not guesses — worst cells + judge reasons, or a trace-analyst's clusters (`ProposeContext.findings`). | GEPA's reflective feedback / the "textual gradient". |
-| **baseline** | The starting surface (the current prompt/config) every candidate is measured against (`baselineSurface`). | The unoptimized program you compare lift against. |
-| **holdout** | A separate scenario split the winner is re-scored on and that the proposer is *never* allowed to see — a compile-time firewall (`ProposeContext.judgeScores: never`, `types.ts:265`) keeps held-out verdicts out of proposal so the optimizer can't game the acceptance axis. | The held-out valset/testset — but here it is *write-only* to the optimizer. |
-| **lift** | Winner minus baseline `composite` on the holdout — the actual improvement, reported with a bootstrap confidence interval (`ImproveResult.lift`, `ProposerScore.lift`). | Δ metric between optimized and baseline program. |
-| **MDE** | Minimum Detectable Effect — the smallest lift your budget could statistically distinguish from noise, computed up front from baseline cells; a structurally-hopeless budget warns before you spend (surfaced as `result.power` / "power preflight", `agent-optimization-map.md`). Not spelled "MDE" in code yet — this is the standard name for it. | Power analysis on the eval set; the reason a tiny valset can't certify a small gain. |
-| **Pareto frontier** | The set of surfaces that are non-dominated across the per-scenario score vectors — a candidate worse on the mean but uniquely best on one hard scenario survives, so its lesson is not discarded (`ParetoParent`, `types.ts:198`; GEPA, arXiv:2507.19457). | Exactly GEPA's Pareto candidate pool — same paper, same idea. |
-| **gate** | The promotion decision: does the winner beat baseline on the holdout with significance, returning one of five verdicts `ship / hold / need_more_work / model_ceiling / arch_ceiling` (`Gate`, `GateDecision`, `types.ts:315/340`). | The accept/reject rule on the held-out valset, plus a significance test and ceiling diagnosis. |
-| **promotion** | What happens on a `ship` verdict — the winning surface is written back into the profile field it came from (`GenerationRecord.promoted`; `applyWinnerToProfile`, `improve.ts`). | Committing the optimized program as the new default. |
-
-## Which proposer — the chooser
-
-Every optimizer is a factory `xProposer(opts): SurfaceProposer`, all exported from `@tangle-network/agent-eval/campaign`.
-Start at the top row and only move down when the row's *"reach for it when"* matches your failure mode.
-
-| Proposer factory | Reach for it when | Surface it edits | Wired to the paved path? |
-|---|---|---|---|
-| `gepaProposer` | You want the strong default: reflective full-surface prompt rewrites, grounded in findings, keeping a Pareto frontier of complementary winners. | prompt string | **Yes — `improve({ surface: 'prompt' })` default.** Proven live. |
-| `skillOptProposer` | You are editing a structured `SKILL.md`/runbook and want small anchored add/delete/replace patches that preserve earlier rules. | skill/prompt string | **Yes — `improve({ surface: 'skills' })` default.** Not yet proven live. |
-| `parameterSweepProposer` | The likely fix is a config knob, not words — `retrieval.k`, `temperature`, `max_tokens`. You give it candidate patches; it applies them to a JSON surface. | JSON config string | Yes, but you supply the candidate list. |
-| `fapoProposer` | You want *evidence to decide when to escalate*: try prompt edits first, move to parameters, then to structural code — one scoped change per cycle, only escalating when the cheaper level is exhausted. | whatever its level proposers return | Exported; you wire the level proposers. |
-| `compositeProposer` | You want several proposers to share one candidate-generation budget in the same round. It allocates the population by declared weights, preserves member provenance, deduplicates surfaces, and isolates a member failure unless every member fails. | whatever its member proposers return | Exported; you wire the member proposers. |
-| `aceProposer` | You are accumulating hard-won lessons into a playbook and must **never** summarize an old lesson away (append-only, provenance-tagged). | playbook string | Exported. |
-| `memoryCurationProposer` | Same as ACE but you want a compact, deduped, re-ranked memory instead of append-only growth. | memory string | Exported. |
-| `evolutionaryProposer` | You want blind population search (mutate → measure → select) with no reflection over findings — a cheap control or a baseline to beat. | any string | Exported. |
-| `traceAnalystProposer` | Bench-only: race our trace-analysis evidence engine head-to-head inside `compareProposers`. | prompt string | Bench-only. |
-| `haloProposer` | Bench-only: race the external `halo-engine` analysis against ours. | prompt string | Bench-only, external. |
-
-Default path: `gepaProposer` for prompts; add `parameterSweepProposer` when a config knob is the suspect; wrap levels in `fapoProposer` when the loop should decide *when* to escalate, or use `compositeProposer` when multiple proposer families must split one fixed population budget.
-
-## Composing proposers — four distinct shapes
-
-Choose the shape that matches the experiment:
-
-1. **Portfolio** — `compositeProposer` splits one generation's population across member proposers by fixed weights and returns one provenance-labelled pool.
-2. **Escalate** — `fapoProposer` wraps prompt + parameter + structural levels into one proposer and spends on the cheapest level until evidence says to escalate.
-3. **Race** — `compareProposers` gives proposers separate loops, then re-scores their winners on one holdout and returns per-proposer lift intervals plus pairwise results.
-4. **Plug in** — hand any proposer to `runImprovementLoop({ proposer })`, or use `improve({ surface, generator })` in `@tangle-network/agent-runtime`.
-
-### 2 + 4 — compose by escalation, then run the improvement loop
-
-```ts
-import {
-  fapoProposer,
-  gepaProposer,
-  parameterSweepProposer,
-  defaultProductionGate,
-  runImprovementLoop,
-} from '@tangle-network/agent-eval/campaign'
-
-const llm = { baseUrl: process.env.TANGLE_BASE_URL, apiKey: process.env.TANGLE_API_KEY }
-const model = 'deepseek-v4-flash'
-
-// Compose by escalation: prompt edits first (GEPA), then a config knob only
-// when prompt-level search plateaus. This is distinct from a peer portfolio.
-const proposer = fapoProposer({
-  scope: { allowedLevels: ['prompt', 'parameter'] },   // no structural/code tier here
-  promptProposer: gepaProposer({ llm, model, target: 'agent system prompt' }),
-  parameterProposer: parameterSweepProposer({
-    candidates: [
-      {
-        label: 'raise-retrieval-k',
-        rationale: 'retrieval-miss findings suggest the search budget is too low',
-        changes: [{ path: 'retrieval.k', value: 10 }],
-      },
-    ],
-  }),
-})
-
-// dispatchWithSurface scores ONE surface on ONE scenario — this is the topology-
-// opaque seam (one LLM call, one worker, or a whole fleet — the loop can't tell).
-const dispatchWithSurface = async (surface, scenario, ctx) =>
-  runYourAgent({ surface, scenario, signal: ctx.signal })   // returns the artifact to judge
-
-const result = await runImprovementLoop({
-  scenarios: trainScenarios,          // proposer trains on these
-  holdoutScenarios,                   // winner is re-scored here; proposer never sees them
-  baselineSurface: currentPromptString,
-  dispatchWithSurface,
-  judges,                             // JudgeConfig[]
-  proposer,
-  gate: defaultProductionGate({ holdoutScenarios, deltaThreshold: 0 }),
-  populationSize: 2,                  // BREADTH per generation
-  maxGenerations: 6,                  // DEPTH
-  autoOnPromote: 'none',              // 'pr' to open a PR on ship
-  runDir: '/tmp/improve-run',         // a REAL path makes the run durable
-})
-
-result.winnerSurface   // the promoted surface
-result.gateDecision    // 'ship' | 'hold' | 'need_more_work' | 'model_ceiling' | 'arch_ceiling'
-```
-
-Same thing, one line, when you have an `AgentProfile` (facade in `@tangle-network/agent-runtime`):
-
-```ts
-import { improve } from '@tangle-network/agent-runtime'
-
-// surface 'prompt'/'skills' pick their proposer automatically; pass `generator`
-// to plug the composed FAPO proposer above into the gated loop instead.
-const out = await improve(profile, findings, {
-  surface: 'prompt',
-  generator: proposer,               // any SurfaceProposer, incl. the composed one
-  scenarios, judge, agent, runDir: '/tmp/improve-run',
-})
-if (out.shipped) deploy(out.profile)  // out.lift is the held-out winner − baseline
-```
-
-### 3 — race proposers head-to-head for a lift CI
-
-```ts
-import {
-  compareProposers,
-  gepaParetoEntry,
-  fapoEscalationEntry,
-} from '@tangle-network/agent-eval/campaign'
-
-const config = {
-  baselineSurface: currentPromptString,
-  trainScenarios,
-  holdoutScenarios,
-  dispatchWithSurface,
-  judges,
-  llm, model,
-  target: 'agent system prompt',
-  runDir: '/tmp/compare-run',
-}
-
-const comparison = await compareProposers({
-  proposers: [
-    gepaParetoEntry(config),                        // GEPA + Pareto frontier
-    fapoEscalationEntry({                            // FAPO escalation policy
-      ...config,
-      parameterCandidates: [
-        { label: 'raise-retrieval-k', rationale: 'retrieval misses', changes: [{ path: 'retrieval.k', value: 10 }] },
-      ],
-    }),
-  ],
-  baselineSurface: currentPromptString,
-  holdoutScenarios,
-  dispatchWithSurface,
-  judges,
-  runDir: '/tmp/compare-run',
-})
-
-comparison.best          // highest-lift proposer
-comparison.scores        // per-proposer { lift, liftCi:{low,high}, cost } — low>0 ⇒ real gain
-comparison.pairwise      // best vs each other, paired-bootstrap: 'a' | 'b' | 'tie'
-```
-
-Every entrant is re-scored on the **same** holdout with the **same** judges, so the comparison never trusts how a proposer measured itself — the only variable is proposal quality.
-
-## Common traps (they cost the most)
-
-- Do not put eval logic inside a proposer — scoring lives in `dispatch` + `judges`, proposing lives in the proposer.
-- Do not let a proposer read held-out judge scores — `ProposeContext` makes that a compile error on purpose; a proposer that games the acceptance axis is an oracle, not an optimizer.
-- Do not read `lift` without `result.power`/MDE — a "+4" on a valset too small to detect +4 is noise wearing a number.
-- Do not confuse `compositeProposer` with `fapoProposer`: the former allocates one fixed population across peers, while the latter escalates through ordered levels from cheaper to more structural changes.
-
-### neutralizationGate — the placebo / content-causality control
-
-Standard gates prove a candidate *beat baseline*. `neutralizationGate` proves the candidate's **content** caused the lift, not the extra prompt bytes: it blanks the candidate's added content to byte-length-matched filler, holds everything else fixed, and requires the lift to vanish. A fully-neutralized candidate that still scores is decorative and is rejected. Exports: `neutralizationGate` (`src/campaign/gates/neutralization-gate.ts`), `neutralizeText` (`src/campaign/neutralize.ts`). Since 0.107.0. Pair it with `heldOutGate` in any loop that promotes authored artifacts (prompts, tool docs, knowledge) so a lift that is really just added prompt size cannot be shipped.

package/docs/self-improvement-map.md

@@ -1,115 +0,0 @@
-# The self-improvement map
-
-One loop. Four roles. A proposer catalog of pluggable strategies. A bench rig that
-proves the loop produces real lift. Nothing here is duplicated — it is one engine
-pointed at different surfaces. This map exists because the surface count makes it
-*look* like many competing systems when it is one.
-
-Neighbors: [`concepts.md`](./concepts.md) (the mental model), [`trace-analysis.md`](./trace-analysis.md)
-(the evidence engine), [`distributed-driver.md`](./distributed-driver.md) (running the loop across cells).
-
-## The one loop
-
-`runImprovementLoop()` (wrapped by `selfImprove()` for the one-call surface). Every
-product imports the same function. Each generation it does four things:
-
-```
-   run AGENT on SCENARIOS ──► JUDGE scores each run
-                                      │
-                          PROPOSER reads the failures and
-                          proposes better SURFACE versions
-                                      │
-                  GATE: did a candidate beat the parent on a
-                  HELD-OUT split, for real (significance test)?
-                        │ yes → promote      │ no → discard
-                                      │
-                              repeat N generations
-```
-
-## The four roles — keep them separate and the confusion clears
-
-| Role | What it is | Plain meaning |
-|---|---|---|
-| **Surface** | a *string* — an agent directive, a `SKILL.md`, a playbook, a memory, a judge rubric | **what** gets improved |
-| **Proposer** | a `SurfaceProposer` (the catalog below) | **how** candidate surfaces are proposed |
-| **Gate** | held-out split + significance (`paretoSignificanceGate` / `heldOutGate` / `defaultProductionGate`) | **did it actually get better**, vs noise |
-| **Judge** | scores a run | **how good** any version is |
-
-## The proposer catalog (one loop, multiple strategies)
-
-The package intentionally exposes named proposer factories instead of a hidden
-auto-selector. The split that matters: **production proposers** mutate a live
-surface; **bench-only proposers** exist solely to be raced inside
-`compareProposers`.
-
-| Proposer factory | Surface | Strategy | Role | Notes |
-|---|---|---|---|---|
-| `gepaProposer` | prompt | reflective full-surface rewrite + Pareto frontier | **production default** | consumes trace-analysis findings — see below |
-| `fapoProposer` | prompt/config/code | reviewed escalation policy over prompt → parameter → structural proposers | production, benchmark | encodes FAPO's scope + reviewer + prompt-first escalation rules; structural generator is injected |
-| `parameterSweepProposer` | config | JSON config patch/sweep | production, benchmark | middle FAPO level for parameter/config edits such as `retrieval.k`, `temperature`, `max_tokens` |
-| `skillOptProposer` | skill-doc | anchored add/delete/replace patch | production | preserves earlier rules; edit budget = "textual learning rate" |
-| `aceProposer` | playbook | append-only, provenance-tagged | production | accumulate hard-won lessons, never summarize away |
-| `memoryCurationProposer` | memory | dedup + rank + graft | production | compact alternative to `ace` |
-| `evolutionaryProposer` | any | population mutate → measure → select | production | blind search; no reflection over findings |
-| `traceAnalystProposer` | prompt | analysis → one LLM edit | **bench-only** | our evidence engine, wrapped as a proposer |
-| `haloProposer` | prompt | analysis → one LLM edit | **bench-only**, external | wraps `pip install halo-engine` (Inference.net) |
-
-Default choice: start with `gepaProposer` for prompt surfaces, add
-`parameterSweepProposer` when config knobs are the likely failure mode, and wrap
-them with `fapoProposer` when evidence should decide when to escalate.
-
-## Trace analysis — what it is and the three places it is used
-
-"Trace analysis" is the **evidence layer**: it turns raw OTLP traces into "here is
-exactly *why* the agent failed" (failure clusters → findings). The engine is
-`analyzeRuns()` + the analyst registry (`src/contract/analyze-runs.ts`). It is used in
-three places — this is the answer to "if GEPA does its own thing, what is trace
-analysis *for*?":
-
-1. **Ships to customers** — `analyzeRuns()` → `InsightReport`, the Intelligence product.
-2. **Feeds the proposer** — `gepaProposer` calls `renderAnalystEvidence(ctx.findings,
-   ctx.report)` (`src/campaign/proposers/gepa.ts`). GEPA's rewrites are grounded
-   in the diagnosis instead of guessing blind. Trace analysis
-   **is** on the GEPA side.
-3. **Races HALO** — wrapped as `traceAnalystProposer` so our analysis competes
-   head-to-head with the external SOTA inside `compareProposers`.
-
-## Where HALO fits (and why it feels "removed")
-
-`haloProposer` is alive (`src/campaign/proposers/halo.ts`, exported from the campaign
-barrel) but it is **never in the product loop**. It shells out to an *external* engine
-(`halo-engine`) — so the analysis genuinely lives outside this repo; we only wrap it.
-
-Its only job is the **bake-off**. HALO's real opponent is **not** `gepaProposer` — it is
-`traceAnalystProposer`. `compareProposers` holds the apply step identical (same
-`APPLY_SYSTEM`, same `traces.jsonl`, same held-out scoring) so the only variable is
-**analysis quality: HALO vs ours.** A measuring stick, like a benchmark baseline.
-
-## `gepa-refine.ts` is the loop on a test bench, not a second loop
-
-`agent-runtime/bench/src/gepa-refine.ts` runs **this same loop** against a *public
-benchmark* (AppWorld, CAD, …) instead of product data. Why a separate rig:
-
-- On product traces, "+4 lift" can be model noise or a judge flattering itself — no
-  ground truth.
-- On a benchmark the score is **objective and ungameable** (AppWorld runs the agent's
-  code against its own unit tests). If a GEPA-optimized directive beats a deliberately
-  weak baseline **on held-out tasks it never trained on**, with a CI excluding zero,
-  that is a *certified* proof the loop produces real lift.
-
-Products run the loop to get **better**. `gepa-refine` runs the identical loop to
-**prove the loop works at all**.
-
-## Where the "mess" feeling actually comes from
-
-The code is well-factored; the confusion is narrative:
-
-- **Surface sprawl reads as chaos.** Several proposers with overlapping shapes *look* like
-  competing loops. They are exported factories for one loop; this map makes that
-  explicit.
-- **The real gap is the missing proof, not the design.** The loop kept being proved on
-  benchmarks too easy to show value: when a capable model ceilings an extraction task,
-  **0 findings fire** and the whole trace-analysis→optimizer apparatus is inert. It
-  earns its keep only on **hard agentic tasks** — which is why the AppWorld REPL run
-  (multi-turn, real tool execution, unbounded turns) is the one that can finally
-  separate the evidence-grounded optimizer from baseline.