@tangle-network/agent-eval 0.94.0 → 0.95.0

package/docs/research/research-roadmap.md

@@ -1,205 +0,0 @@
-# Research Roadmap — Agent Self-Improvement as a Research Field
-
-**Status:** Living. Separate from the product roadmap. Updated when a thesis formalizes, an experiment runs, or a draft posts.
-**Tracking:** task #107.
-**Audience:** Dario, Yann, Ilya, Sam, lab researchers, peer reviewers.
-**Posture:** Honest about what we have, sharp about what we'd need.
-
-## One-sentence pitch
-
-**Agent self-improvement is missing its statistical foundation, its formal model of two-writer state, and its standard benchmark. We claim all three.**
-
-## The three publishable theses
-
-### Thesis 1 — Branch-benchmark consensus for safe offline+online self-improvement
-
-**Setup.** Two writers concurrently mutate an agent's behavior surface: an in-sandbox harness (per-turn, online) and an offline substrate (batch, statistically-gated). Both produce divergent versions from a common ancestor. Existing literature handles online RL (single writer = policy) and offline RL (no in-runtime writes) — nobody has formalized the *combined* regime where both writers coexist.
-
-**Claim to prove.** Given common ancestor `P_anc`, harness branch `P_h`, substrate branch `P_s`, scenarios `S`, and judge `J`, a branch-benchmark consensus procedure produces a winner `P_w` with regret bounded by `max(R(P_h), R(P_s)) − ε` with probability `≥ 1−δ` under explicit assumptions about judge calibration + scenario coverage.
-
-**Why it's publishable.**
-- Genuinely novel regime — the combined offline+online assumption set is uncharted.
-- Maps to a real customer pain point (Hermes-on-our-sandbox drift).
-- Tractable proof structure: paired-bootstrap + Hoeffding + union bound across surface dimensions.
-- Empirical validation: instrument Hermes-on-our-sandbox, measure consensus-vs-naïve-merge regret.
-
-**Estimated effort.** ~3 months focused. Maps to product task #98 (profile-versioning architecture).
-
-**Venue.** NeurIPS or ICLR main track. Or workshop at NeurIPS Foundation Models for Decision Making.
-
----
-
-### Thesis 2 — Natural-language corrective feedback as a learnable gradient
-
-**Setup.** RL provides scalar reward `r ∈ ℝ`. Natural-language feedback ("stop doing X", "you always Y", "this is too verbose") carries strictly more information per bit but no formal model says how to combine it with scalar reward to update policy or skill state. Hermes uses corrective feedback heuristically; GEPA paper claims language is a richer learning medium but doesn't formalize this specific signal. The gap is wide open.
-
-**Claim to prove.**
-1. Information-theoretic: corrective utterances carry `H(c) > H(r)` bits of policy-relevant information under realistic distributions of user satisfaction.
-2. Algorithmic: an extraction+integration procedure exists that improves sample efficiency over scalar-only RL by `k×` (target k ≥ 5) on the proposed benchmark.
-3. Empirical: validation on multi-turn agent tasks with explicit user-corrective channels.
-
-**Why it's publishable.**
-- Connects to GEPA paper's central claim, makes it falsifiable for the corrective sub-class.
-- Maps to product task #103 (`extractUserCorrections`).
-- Distinctive — no observability or RL competitor formalizes corrective feedback as a gradient.
-
-**Estimated effort.** ~4 months. Information-theoretic framing is delicate.
-
-**Venue.** ICLR or main-track NeurIPS. Possibly EMNLP for the NLP angle.
-
----
-
-### Thesis 3 — Sample-efficient self-improvement under a paired-bootstrap gate
-
-**Setup.** GEPA paper claims ~35× fewer rollouts than GRPO. They use a binary improvement check. Our substrate uses paired-bootstrap CI + Cohen's d + MDE (strictly stricter gate). The trade-off between gate-strictness and rollout efficiency is unmodeled.
-
-**Claim to prove.** Given a paired-bootstrap gate with significance level α and minimum detectable effect δ, `selfImprove` requires `O((σ/δ)² · log(1/α))` rollouts to detect a true ε-improvement with power `1-β`. Tight constants. Compare empirically to GRPO and to GEPA's simple-improvement gate on identical benchmarks.
-
-**Why it's publishable.**
-- Closes a gap GEPA left open (their efficiency claim has no power analysis).
-- Maps to product task #101 (real GEPA Pareto + sample-size theory).
-- Provides a tool — power calculator for the field's self-improvement runs.
-
-**Estimated effort.** ~2 months — the cleanest of the three. Mostly classical sample-size theory + careful experiments.
-
-**Venue.** ICML or AISTATS. The statistical framing fits both.
-
-## The fourth thesis (long-horizon, highest prestige)
-
-### Thesis 4 — A standardized benchmark for self-improvement
-
-**Setup.** No standard benchmark exists for "did self-improvement help, robustly, across distribution shift?" GAIA + SWE-Bench + AgentBench measure agent capability; nothing measures self-improvement quality. The field is publishing self-improvement results on disparate ad-hoc setups; nobody compares.
-
-**Claim to ship.** A benchmark with:
-- 100+ scenarios spanning distinct distribution shifts (intra-domain, cross-domain, adversarial corruption)
-- Held-out test split with strict contamination guards
-- Reference baselines (no-driver / random / scalar-only-RL / GEPA / our substrate)
-- Standard scorecard: lift CI, sample efficiency, distribution-shift robustness, cost
-- Public leaderboard
-
-**Why it matters.** Whoever owns the benchmark owns the measuring stick. ImageNet for vision, GLUE for NLP, GAIA for agent capability — the gap for *self-improvement quality* is open.
-
-**Estimated effort.** 6 months. Real scenario authoring, contamination engineering, community outreach for leaderboard adoption.
-
-**Venue.** Datasets + Benchmarks track at NeurIPS. Or workshop debut → main-track followup.
-
-## 12 open research questions, ranked by signal-to-noise
-
-Each is a falsifiable claim or unanswered formal question. Each maps to publishable work.
-
-1. **Information content of corrective feedback.** What's the empirical mutual information `I(correction; preferred_policy)` across realistic agent deployments? Is it consistently `> H(scalar_reward)`?
-
-2. **Convergence of branch-benchmark consensus.** Under what assumptions on judge calibration does the symmetric-fork merge protocol converge to a global optimum vs a local one?
-
-3. **The cost of statistical strictness.** How much does a paired-bootstrap gate cost in rollouts vs a literal `>` gate (SkillOpt's choice), as a function of true effect size? Where's the crossover where strictness costs more than it saves?
-
-4. **Cross-surface attribution.** When `compositeDriver` ships a winner where N surfaces changed, which surface's change drove the lift? Shapley estimators on agent-profile surfaces — tractable? Required sample size?
-
-5. **Sample-efficient evaluation under distribution shift.** Given a held-out test slice and a known shift class (intra-domain / cross-domain / adversarial), how few held-out scenarios are needed to detect lift with target power? Is it a function of shift magnitude?
-
-6. **Diminishing returns of recursive self-improvement.** A substrate that optimizes its own SKILL.md against held-out tasks — does it converge or drift? At what point do recursive self-edits become net-negative on a true holdout? Map the loss landscape.
-
-7. **Skill semantic-duplicate detection.** Substrate's `summarize-pr` vs harness's `pr-summarizer`. What's the right embedding + threshold? Is human review for borderline cases unavoidable or can it be automated?
-
-8. **Reward-hacking under self-improvement.** When the optimizer can mutate the judge prompt (the recursive surface), what's the formal condition under which it learns to game the judge instead of solving the task? Connect to Goodhart + AIRP.
-
-9. **Cost-quality Pareto across drivers.** What's the empirical Pareto frontier when you trade off `gepaDriver` (high $/gen) vs `evolutionaryDriver` ($0/gen) vs heuristic mutations? Is it task-dependent or universal?
-
-10. **Online-offline merge regret.** When harness branch and substrate branch are merged, what's the regret of the merged policy vs the better-of-two? Bounded? Worst-case adversarial?
-
-11. **Universal trace ingest tax.** Cross-framework adapter coverage (LangChain / LlamaIndex / Anthropic / OpenAI) — how much signal loss is forced by the lowest-common-denominator RunRecord shape? Quantify in terms of recoverable lift CI.
-
-12. **Foundation-model-as-judge calibration drift.** When the judge LLM updates (Claude → Claude+1), what's the variance in judge scores on a fixed corpus? Is held-out gate validity preserved across judge versions? Empirical study, longitudinal.
-
-## The processes (how we actually do this)
-
-**Cadence.**
-- Daily: product work continues (Track A). Research is a separate 30%-time block.
-- Weekly: research log + open-questions revision. One paper-quality paragraph per week.
-- Monthly: experiment milestone — either proof attempt or empirical-run results.
-- Quarterly: paper-draft milestone.
-
-**Artifacts.**
-- `docs/research/<thesis>/notes.md` — running research log, hypothesis, current status.
-- `docs/research/<thesis>/experiments.md` — every run + numbers + analysis.
-- `docs/research/<thesis>/paper-draft.md` — building toward arXiv submission.
-- `docs/research/belief-state-agent-eval-roadmap.md` — belief-state / adaptive-control research tracker and 24-month gate plan.
-- `.evolve/research/<thesis>/` — code + data + figures, version-controlled.
-
-**Quality bar.**
-- Every claim falsifiable. Every number has CI, p, and sample size.
-- Every experiment reproducible — script + seed + commit hash + data hash.
-- Every figure has an underlying CSV the reviewer can download.
-- Every theorem has a proof in the doc, not just a citation.
-
-**Review cadence.**
-- Internal critique pass before any external sharing — find every weak spot.
-- External review at 80%-draft: one peer in the field, one peer outside.
-- ArXiv submission as the gating event for public claim.
-
-## What we explicitly will NOT do
-
-- **Will not pretend product-grade engineering is research.** Architecture docs are not papers. Strategic framing is not contribution.
-- **Will not chase trendy directions (RLHF variants, constitutional AI, scaling laws) where we have no edge.** Our edges are specific: two-writer state, corrective feedback as gradient, statistical strictness. Stay in lane.
-- **Will not publish empirical results without proper baselines.** "Our substrate produces N% lift on dataset X" is meaningless without no-driver/random/GEPA/SkillOpt baselines on identical infrastructure.
-- **Will not optimize for citation count over insight.** One paper that changes how the field thinks > five papers that move a benchmark by 2 points.
-
-## Where we are right now
-
-**Track A (product) status as of 2026-05-27:**
-- agent-eval shipped 0.47 → 0.53 in one session
-- Six consumers on substrate 0.50+
-- Honest spec docs landed
-- Product roadmap 0.53 → 1.1 mapped
-
-**Track B (research) status as of 2026-05-27:**
-- This doc exists
-- Zero experiments run on published benchmarks
-- Zero papers drafted
-- Three theses identified, none formalized
-- Twelve open questions enumerated, none answered
-
-We are at Track-B day 0. Honesty matters.
-
-## Deliverables — 12-month plan
-
-**Q3 2026 — proof of life.**
-- Run our drivers against AgentBench / SWE-Bench Verified / GAIA. Report numbers with CI.
-- Pick one named partner customer who'd validate Thesis 1 with us on their real deployment.
-
-**Q4 2026 — Thesis 3 paper draft.**
-- Sample-efficient self-improvement is the cleanest claim — fastest to publish, sharpens our gate's edge.
-- Target: arXiv pre-print + AISTATS submission.
-
-**Q1 2027 — Thesis 1 paper draft.**
-- Branch-benchmark consensus — the deepest claim, the one that needs forcing-function data from a Hermes-on-sandbox deployment.
-- Target: NeurIPS / ICLR submission.
-
-**Q2 2027 — Thesis 4 benchmark public release.**
-- The benchmark + leaderboard is the highest-prestige play.
-- Target: Datasets + Benchmarks track at NeurIPS 2027.
-
-**Q3 2027 — Thesis 2 paper draft.**
-- Corrective feedback as gradient — slowest to ripen, hardest to formalize.
-- Target: ICLR submission.
-
-## How a lab lead would react to this doc
-
-If you printed this and slid it across Dario's desk:
-
-**The good.** Specific named theses with falsifiable claims. Honest about gap from product to research. Three publishable directions in clear scope. Twelve open questions readable as a research-program statement.
-
-**The hostile-reviewer attack.** "Show me one number on one published benchmark from your existing infrastructure. You have a substrate with a paired-bootstrap gate that's never been compared to anything." That is correct. Q3 2026 deliverable is the answer.
-
-**The deepest question they'd ask.** "Why does this matter for AGI / safety / capability? Why work on this instead of pretraining / alignment / interpretability?" Honest answer: agents that self-improve in production are a near-term reality. The work to make that *safe* and *measurable* is path-dependent on whether the field formalizes it or accepts ad-hoc product implementations. Our pitch is "be the lab that formalized it before it became a 1000-org engineering mess." That's a defensible answer if backed by the published work.
-
-## The one-sentence inspirational version per audience
-
-- **For Dario:** "We're building the statistical foundation that turns 'agents that self-improve' from a marketing slogan into a measurable claim with calibrated error bars."
-- **For Yann:** "Self-improvement is offline-RL with two writers — and nobody has formalized the consensus regime. We will."
-- **For Ilya:** "What's the simplest formalism under which self-improving agents converge to a global optimum vs a local one? Branch-benchmark consensus is our hypothesis."
-- **For Sam:** "We are going to ship the substrate that lets every customer's agent self-improve safely, then publish the science that proves it works. The product builds the data; the data writes the papers; the papers create the moat."
-
-## The harshest honest sentence
-
-If we don't run a published benchmark by Q3 2026, this entire doc is fan-fiction. Build the empirical infrastructure first, formalize after, publish last.

package/docs/specs/driver-honest-spec.md

@@ -1,251 +0,0 @@
-# Driver Honest Spec — what each driver IS, what each methodology actually is, where we deviate
-
-**Status:** Living document. Updated when we learn the truth from primary sources.
-**Date:** 2026-05-27
-
-This document exists because the project shipped two drivers with methodology names attached (`gepaDriver`, `skillOptDriver`) without the methodology specs being precisely encoded anywhere in the repo. That created an integrity gap. This doc closes it.
-
-Every claim in this doc is sourced from a primary reference (paper, code, or directly verifiable from our source). Marketing language is forbidden. If something is not implemented we say so.
-
----
-
-## Part 1 — GEPA (the paper)
-
-**Source**: Agrawal et al., *"GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning"*, arXiv:2507.19457, July 2025.
-
-### What GEPA actually does
-
-Outer loop (verbatim from abstract): "samples trajectories (e.g., reasoning, tool calls, and tool outputs) and reflects on them in natural language to diagnose problems, propose and test prompt updates, and combine complementary lessons from the **Pareto frontier of its own attempts**."
-
-Named primitives in the paper:
-- **GEPA** (Genetic-Pareto) — the overall optimizer
-- **Pareto frontier** — non-dominated candidate set retained across iterations
-- **Prompt updates** — mutations proposed by reflection
-- **Rollouts** — trajectory samples
-
-### What gepaDriver in our substrate ACTUALLY does
-
-Source: `src/campaign/drivers/gepa.ts` (132 lines)
-
-- Single LLM call per `propose()` invocation
-- Input: prior generation's **single best candidate by composite score** + that candidate's top/bottom scenarios + 3 weakest dimensions (`buildEvidence`)
-- Output: N proposals, each a full document rewrite
-- Dedup by exact text equality
-
-### Deviations from the GEPA paper
-
-| GEPA paper | Our `gepaDriver` |
-|---|---|
-| **Pareto frontier** of candidates | **Single "best by composite"** — no Pareto set, no non-dominated tracking |
-| **Combine complementary lessons** from frontier | Each generation reflects on ONE prior candidate; no combination |
-| Multi-objective optimization | Single-objective (composite score) |
-| Genetic operators (mutation, crossover) | Reflection only — no crossover |
-| Sample efficiency claim (35× fewer rollouts than GRPO) | Unmeasured against any baseline |
-
-**Honest assessment**: our `gepaDriver` is a **reflective full-rewrite driver**, not GEPA. It captures GEPA's *reflection* primitive but not its *Pareto* mechanism. The name oversells. A faithful renaming would be `reflectiveRewriteDriver`. A faithful implementation would add a Pareto candidate pool + combine step.
-
----
-
-## Part 2 — SkillOpt (the paper + code)
-
-**Source**:
-- README: https://github.com/microsoft/SkillOpt
-- Source: `/tmp/SkillOpt/skillopt/` (cloned 2026-05-27)
-- Key files: `engine/trainer.py`, `optimizer/clip.py` (rank_and_select), `optimizer/update_modes.py`, `evaluation/gate.py`, `types.py`
-
-### What SkillOpt actually does
-
-**6-stage per-step pipeline** (verbatim from `trainer.py:516` and adjacent):
-
-1. **Rollout** — `adapter.rollout(train_env, current_skill, ...)` collects trajectories on a batch.
-2. **Reflect** — `adapter.reflect()` analyses trajectories and emits **structured patches** (NOT full rewrites in patch mode). Failure trials → failure patches; success trials → success patches.
-3. **Aggregate** — `merge_patches(current_skill, all_failure_patches, all_success_patches, batch_size=merge_bs)` — hierarchically merges patches across accumulated batches.
-4. **Select** — `rank_and_select(current_skill, merged_patch, max_edits=edit_budget)` — if edit pool > budget, calls an optimizer LLM to **rank edits by importance** and keep top-L. Budget is "analogous to gradient clipping" (their words).
-5. **Update** — apply patch in one of 3 modes:
-   - **`patch`** — deterministic diff apply via `apply_patch_with_report()`; ops are `append | insert_after | replace | delete`
-   - **`rewrite_from_suggestions`** — LLM regenerates full skill from suggestions
-   - **`full_rewrite_minibatch`** — reflection directly emits complete candidate skills; select picks the best
-6. **Evaluate & Gate** — runs candidate on selection set, calls `evaluate_gate(cand_hard, current_score, best_score)`. Returns `accept_new_best | accept | reject` from a **literal `cand_hard > current_score`** comparison (`evaluation/gate.py:38`). No statistical test.
-
-Plus epoch-level stages:
-- **Slow update** — `run_slow_update()` builds longitudinal pairs across epochs.
-- **Meta skill** — `run_meta_skill()` produces optimizer-side memory of patterns across adjacent epochs.
-
-### Canonical patch shape (from `types.py:22-45`)
-
-```python
-EditOp = Literal["append", "insert_after", "replace", "delete"]
-
-@dataclass
-class Edit:
-    op: EditOp
-    content: str
-    target: str  # for replace/delete/insert_after
-    support_count: int | None  # how many trials voted for this edit
-    source_type: Literal["failure", "success"] | None
-    merge_level: int | None
-
-@dataclass
-class Patch:
-    edits: list[Edit]
-    reasoning: str
-    ranking_details: dict | None
-```
-
-### What `skillOptDriver` v0.51.0 in our substrate ACTUALLY does
-
-Source: `src/campaign/drivers/skillopt.ts` (current as of 0.51.0)
-
-- Single LLM call per `propose()` returning N full document rewrites
-- Post-parse rejection on: (a) any H2 header dropped, (b) sentence-edit count > editBudget × 2
-- Substantively equivalent to `gepaDriver` + 2 validation constraints
-
-### Deviations from SkillOpt
-
-| SkillOpt actual | Our 0.51.0 `skillOptDriver` |
-|---|---|
-| 6-stage pipeline (rollout → reflect → aggregate → select → update → gate) | Single LLM call → N rewrites |
-| **Patch-based edits** (`{op, target, content, support_count, source_type}`) | Full document rewrites only |
-| `merge_patches()` hierarchical merge across batches | No aggregation; each `propose()` is independent |
-| `rank_and_select(max_edits=edit_budget)` LLM-ranking of edits | All candidates that pass validation are returned |
-| 3 update modes (`patch`, `rewrite_from_suggestions`, `full_rewrite_minibatch`) | Only `full_rewrite_minibatch`-equivalent |
-| `evaluate_gate()` with `accept_new_best/accept/reject` codes | Substrate's outer gate decides ship/hold/inspect; driver doesn't see fine-grained accept signal |
-| Longitudinal `slow_update` across epochs | Not implemented |
-| `meta_skill` optimizer-side memory | Not implemented |
-| Selection-set cache (`sel_cache`) for repeated candidate hashes | Not implemented |
-| Edit-budget LR scheduler (constant / linear / cosine / autonomous) | Single fixed `editBudget` |
-| Mini-batch accumulation (`steps_per_epoch`, `merge_batch_size`) | Not implemented |
-| `decide_autonomous_learning_rate()` | Not implemented |
-| `longitudinal_pair_policy` (mixed / changed / unchanged) | Not implemented |
-
-**Honest assessment**: 13 substantive deviations. `skillOptDriver` 0.51.0 is **not** SkillOpt. It is `gepaDriver` with two post-validation constraints (section preservation, sentence-edit count). The methodology name oversells the implementation.
-
-### One thing where we are STRICTER than SkillOpt
-
-**The gate.** SkillOpt: literal `cand_hard > current_score` (`evaluation/gate.py:38`). Our substrate: paired bootstrap + 95% CI + Cohen's d + MDE + p-value (`defaultProductionGate`). When the lift CI straddles zero, our gate returns `hold` / `inspect`. SkillOpt would accept any improvement at all, even single-sample noise.
-
-This is real differentiation we have not been crediting ourselves for.
-
----
-
-## Part 3 — Hermes Agent's "self-improvement"
-
-**Source**: `/tmp/hermes-agent/` (cloned 2026-05-27)
-- `agent/curator.py` (the actual loop)
-- `agent/skill_commands.py`
-- `agent/skill_utils.py`
-
-### What Hermes actually does
-
-From `curator.py` line 1: "Curator — background skill maintenance orchestrator. The curator is an auxiliary-model task that periodically reviews agent-created skills and maintains the collection."
-
-Trigger: idle-driven, with default `DEFAULT_INTERVAL_HOURS = 24 * 7` (7 days). When the agent has been idle for `DEFAULT_MIN_IDLE_HOURS = 2` and the last curator run was > 7 days ago, `maybe_run_curator()` spawns a forked AIAgent.
-
-What the curator does:
-- "Auto-transition lifecycle states based on derived skill activity timestamps"
-- "Spawn a background review agent that can **pin / archive / consolidate / patch** agent-created skills via `skill_manage`"
-- "Persist curator state (last_run_at, paused, etc.) in `.curator_state`"
-
-Strict invariants:
-- Only touches agent-created skills
-- "Never auto-deletes — only archives"
-- Pinned skills bypass auto-transitions
-- Uses the auxiliary client (separate from main session)
-
-### Hermes' actual gate
-
-**There is none.** The curator is an LLM editor making editorial decisions. There is no:
-- Held-out validation
-- Performance comparison between old and new skill versions
-- Statistical test
-- Rejection-on-regression mechanism
-
-Skills are refined by an LLM looking at usage patterns; the refinement is accepted because the LLM proposed it.
-
-### Honest assessment
-
-Hermes has a **skill curation system**, not a self-improvement loop. The README's claim "the only agent with a built-in learning loop" is generous — it's a 7-day-cron LLM librarian. There's no measurable guarantee that today's curated skill collection performs better than yesterday's.
-
-Compare:
-| Component | Hermes | SkillOpt | Tangle |
-|---|---|---|---|
-| Validation gate | None | `>` | Paired bootstrap CI |
-| Patch-level edits | No (LLM rewrites whole skill) | Yes | No (full rewrite only) |
-| Skill ranking / selection | No | Yes | No |
-| Sample efficiency claim | None | 35× vs GRPO | None |
-| Frequency | 7-day cron | Per training step | Per `selfImprove()` call |
-
-Where Tangle WINS: the gate. Where SkillOpt WINS: the pipeline sophistication. Where Hermes WINS: the deployment story (multi-platform, multi-tool-backend).
-
----
-
-## Part 4 — What we should actually do
-
-### Phase A — rename to honest names (0.51.1, this session)
-
-The current `skillOptDriver` and `gepaDriver` names overclaim. Options:
-
-1. **Rename both:**
-   - `gepaDriver` → `reflectiveRewriteDriver` (drops the "Pareto" implication)
-   - `skillOptDriver` → `constrainedReflectiveDriver` (drops the SkillOpt-methodology implication)
-   - Reserve `gepaDriver` + `skillOptDriver` for faithful implementations
-2. **Keep `gepaDriver` name** (it's our most-used driver; renaming is disruptive); rename `skillOptDriver`.
-3. **Keep both names; add `@experimental` + a "differs from paper" docstring section.** Cheapest. Truthful enough.
-
-Recommendation: **option 3 plus a frontmatter "deviations from paper" section** in each driver source file. Empirically test before renaming.
-
-### Phase B — build the honest empirical harness (0.51.1, this session)
-
-`tests/driver-empirical.bench.ts` — for each driver:
-- Same scenarios (5 synthetic + 5 real legal-agent scenarios)
-- Same judge
-- Same `baselineSurface`
-- Same `budget` (1 gen, 3 candidates, holdout 0.3)
-- Report: lift mean, lift CI95, p-value, rollouts spent, $$ spent
-
-Drivers in the matrix:
-- `gepaDriver` (current full-rewrite reflection)
-- `skillOptDriver` (current 0.51.0 full-rewrite + constraints)
-- Future: real `skillOptDriverV2` with patch mode
-
-This is the **falsifiable test** of whether our drivers' methodology claims are worth the names.
-
-### Phase C — implement SkillOpt patch mode for real (0.52.0)
-
-Build `skillOptDriverV2` with:
-1. **`Edit` type matching SkillOpt's**: `{op: 'append'|'insert_after'|'replace'|'delete', content, target?, support_count?, source_type?}`
-2. **Reflect step emits patches**, not full rewrites
-3. **`mergePatches()`** — LLM-driven hierarchical merge of failure + success patches
-4. **`rankAndSelect()`** — LLM-driven ranking when edit pool > budget
-5. **Deterministic `applyPatch()`** — string ops, no LLM
-6. **Keep our gate** (paired bootstrap CI). Don't downgrade to SkillOpt's `>` — that's our edge.
-
-Estimated scope: 400-600 lines + tests.
-
-### Phase D — implement GEPA's Pareto frontier (0.53.0)
-
-Build `gepaDriverV2` with:
-1. **Candidate pool** retained across generations (non-dominated)
-2. **Multi-objective evaluation** (composite + cost + length + diversity)
-3. **Combine step** — LLM combines lessons from non-dominated candidates
-4. Keep reflection.
-5. Sample-efficiency target: match the paper's ~35× claim on a benchmark we choose.
-
-Estimated scope: 500-800 lines + tests.
-
----
-
-## Source pointers (audit trail)
-
-- GEPA paper: https://arxiv.org/abs/2507.19457
-- SkillOpt repo: https://github.com/microsoft/SkillOpt (cloned at `/tmp/SkillOpt/` 2026-05-27)
-- Hermes repo: https://github.com/NousResearch/hermes-agent (cloned at `/tmp/hermes-agent/` 2026-05-27)
-- Our gepaDriver: `src/campaign/drivers/gepa.ts`
-- Our skillOptDriver: `src/campaign/drivers/skillopt.ts`
-- Our gate: `src/campaign/gates/default-production-gate.ts`
-- Our reflection primitive: `src/reflective-mutation.ts`
-
-Update this doc when:
-- We discover new behavior in any of the upstream methods (via reading their code, not their READMEs)
-- We ship a driver that closes one of the named gaps
-- We run the empirical harness and have real numbers to add

package/docs/specs/hermes-self-improvement-audit.md

@@ -1,93 +0,0 @@
-# Hermes self-improvement — corrected audit
-
-**Status:** Active. This corrects an earlier underestimate where I claimed Hermes only had the 7-day curator. Drew pushed back; he was right.
-**Source:** github.com/NousResearch/hermes-agent cloned 2026-05-27 at /tmp/hermes-agent.
-
-## The corrected picture
-
-Hermes has **two** self-improvement mechanisms, not one. Per their own source comments: "background self-improvement review fork" (`tools/skill_provenance.py:5`).
-
-### Mechanism 1 — per-turn background review (the actual learning loop I missed)
-
-**File:** `agent/background_review.py` (593 lines)
-
-**Trigger.** `spawn_background_review_thread()` runs after every turn (`AIAgent.run_conversation`). Forks a daemon thread that:
-1. Snapshots the conversation history
-2. Boots a forked `AIAgent` inheriting the parent's runtime (model, provider, base_url, credentials, cached system prompt — exact same auth for prompt-cache reuse)
-3. Feeds the fork one of three review prompts:
-   - `_MEMORY_REVIEW_PROMPT` — should we save anything about the user?
-   - `_SKILL_REVIEW_PROMPT` — should we update the skill library?
-   - `_COMBINED_REVIEW_PROMPT` — both
-4. The fork executes with a tool whitelist (memory + skill management only)
-5. Writes go straight to `~/.hermes/skills/` and the memory store
-6. Provenance tag: `_memory_write_origin = "background_review"`
-
-**Critical signal source.** The skill-review prompt explicitly looks for **user-feedback signal during the conversation**:
-
-> "User corrected your style, tone, format, legibility, or verbosity. **Frustration signals** like 'stop doing X', 'this is too verbose', 'don't format like this', 'why are you explaining', 'just give me the answer', 'you always do Y and I hate it', or an explicit 'remember this' are FIRST-CLASS skill signals, not just memory signals."
-
-> "Be ACTIVE — most sessions produce at least one skill update, even if small. A pass that does nothing is a missed learning opportunity, not a neutral outcome."
-
-This is **qualitative LLM-judges-LLM optimization driven by real user-corrective feedback**. The validation gate is the forked agent's own judgment.
-
-**No held-out validation.** No A/B between skill versions. No regression rejection. No statistical test. The agent decides "save this" or "don't" and writes immediately.
-
-### Mechanism 2 — 7-day curator (housekeeping, not learning)
-
-**File:** `agent/curator.py`. As I described earlier — periodic LLM editorial pass over agent-created skills, pin/archive/consolidate/patch. **Only touches skills that the per-turn loop created.** Doesn't refine via measurement; refines via LLM editorial judgment.
-
-### Storage
-
-- `~/.hermes/skills/<name>/SKILL.md` + `references/` directory per skill (their own documented invariant)
-- `~/.hermes/skills/.usage.json` — sidecar telemetry per skill (usage counts, lifecycle states `active → stale → archived → pinned`)
-- Lifecycle states drive curator decisions but never the per-turn review
-
-## Corrected competitive matrix
-
-| Component | Hermes | SkillOpt | Tangle |
-|---|---|---|---|
-| Trigger | **Per-turn fork** + 7-day curator | Per training step | Per `selfImprove()` invocation |
-| Signal source | **User corrective feedback during chat** + agent retrospection | Judge scores on held-out batches | Judge scores + held-out + multi-rater |
-| Patch granularity | Tool-call level (skill_manage create/edit/patch) | Structured `Edit` ops with `support_count` | Full document rewrite (today) |
-| Validation gate | **None** — forked agent's own judgment | Literal `cand_hard > current_score` | **Paired bootstrap + CI + Cohen's d + MDE** |
-| Rejection-on-regression | No | Yes (gate returns `reject`) | Yes (gate returns `hold` / `inspect`) |
-| Cross-batch aggregation | No | Yes (`merge_patches`) | No |
-| Edit ranking under budget | No | Yes (`rank_and_select`) | No |
-| Longitudinal memory | Usage telemetry only | Yes (`run_slow_update`, `run_meta_skill`) | No |
-| Statistical rigor | None | None | **Highest** |
-| User-feedback signal | **Yes — first-class** | No (offline only) | No (offline only) |
-
-## What we beat them on — what they beat us on
-
-**Tangle wins:** the gate. Paired bootstrap CI + Cohen's d + MDE is statistically stricter than both. We refuse to ship on noise; both Hermes and SkillOpt accept improvements that could be noise.
-
-**Hermes wins:** the signal. They use real user-corrective feedback ("you always do Y and I hate it") as a first-class gradient. We use judge scores; they use both judge scores AND user-language feedback. Their loop fires **per turn**, ours fires **per offline campaign**.
-
-**SkillOpt wins:** the pipeline. Structured patches, hierarchical merge, edit ranking under budget, multiple update modes, longitudinal slow-update, meta-skill memory. Our pipeline is full-rewrite-then-validate; theirs is patch-with-multi-trial-evidence.
-
-## The real architectural insight from this audit
-
-Hermes' per-turn loop is **online**. Our `selfImprove()` is **offline batch**. When Hermes runs on top of our sandbox, **the harness will mutate skills underneath us continuously**. By the time our offline eval finishes, the baseline we measured against may be 50 generations behind production.
-
-That's the gap task **#98 — Profile-versioning architecture** exists to close.
-
-## What we should actually do differently
-
-1. **Stop dismissing Hermes' loop.** It's real, it uses signal we don't, and it's been deployed at scale. Their methodology paper would be: "user-corrective-feedback-driven self-improvement with LLM-judges-LLM acceptance and usage-telemetry-driven housekeeping." We should treat this as a real prior, not marketing.
-
-2. **Add user-feedback signal as a substrate primitive.** Today our `RunRecord.outcome` carries judge scores and raw artifact data. It doesn't carry **in-conversation corrective signals** ("user said 'stop doing X' at turn 7"). If we want to fuse our statistical gate with Hermes' signal source, we need a `RunRecord.userFeedback?: UserCorrectionEvent[]` field.
-
-3. **Recognize the offline/online divide is structural.** Hermes is online. Our substrate is offline. The bridge is the profile-versioning architecture (task #98) — let the harness do per-turn online updates, let the substrate do batch offline eval against versioned snapshots, then merge/rebase via a real diff protocol.
-
-4. **Do the per-turn signal extraction NOW (cheap).** Even without versioning, we could parse traces for user-corrective markers (regex on user messages: "stop", "don't", "I hate", "always Y", "just give me", "this is too X") and emit them as a new `RunRecord` field. That captures Hermes' signal source as additive substrate evidence.
-
-## Source pointers (audit trail)
-
-- `agent/background_review.py:1-30` (header docstring naming the loop)
-- `agent/background_review.py:_MEMORY_REVIEW_PROMPT`, `_SKILL_REVIEW_PROMPT`, `_COMBINED_REVIEW_PROMPT` (the actual prompts)
-- `agent/background_review.py:_run_review_in_thread` (the fork worker)
-- `agent/background_review.py:spawn_background_review_thread` (the entry)
-- `tools/skill_provenance.py:1-15` (docstring: "background self-improvement review fork" — Hermes' own term for their loop)
-- `tools/skill_usage.py:1-25` (telemetry + lifecycle)
-- `agent/curator.py` (7-day housekeeping)
-- `skills/autonomous-ai-agents/hermes-agent/SKILL.md` (45KB CLI/architecture reference)