@tangle-network/agent-eval 0.93.0 → 0.95.0

package/docs/specs/profile-versioning.md

@@ -1,291 +0,0 @@
-# Profile versioning — closing the offline/online drift gap
-
-**Status:** Architecture design. Greenfield, replace existing primitives in place. No V2 suffix.
-**Owner:** spans agent-eval + agent-runtime + agent-knowledge + sandbox SDK.
-**Tracking:** task #98.
-**Date:** 2026-05-27.
-
-## Architecture in one diagram — symmetric fork
-
-Neither writer is privileged. Both branches are first-class. When they reconverge, the substrate's job is to BENCHMARK the branches and propose what to keep — not to be the authority.
-
-```
-            AgentProfile lineage
-              ╱           ╲
-             ╱             ╲
-       harness branch   substrate branch
-        (per-turn writes)   (selfImprove diff)
-             ╲             ╱
-              ╲           ╱
-             DIVERGENCE EVENT
-                     │
-                     ▼
-            benchmark both branches
-            against the same held-out
-                     │
-            ┌────────┼────────┐
-            ▼        ▼        ▼
-       ship-harness ship-substrate merge
-                     │
-                     ▼
-              inconclusive → expand
-              corpus / human review
-```
-
-The substrate becomes a peer, not an owner. The gate verdict names *which* branch won, not just "ship."
-
-## What we are fixing
-
-Two writers, same state, no coordination:
-
-- **Harness writer** — Hermes-style per-turn `spawn_background_review_thread`, agent-runtime's runLoop, any future in-sandbox self-modification. Online, continuous, fires every turn.
-- **Substrate writer** — `selfImprove()` running offline against a frozen snapshot, producing a winner with held-out gate confidence. Batch, fires per campaign.
-
-Failure modes today:
-
-1. **Lost update.** Substrate ships a winner. Harness's per-turn updates since baseline evaporate.
-2. **Stale eval.** Substrate's lift CI is `winner vs P₀`. Production is at `P_h`. The CI says nothing about `winner vs P_h`.
-3. **Gate becomes a lie.** `gateDecision: ship` against `P₀` looks legitimate. Consumer ships. Regresses against `P_h`. Detection fails because metrics moved too.
-
-## The minimum design
-
-Single concept, single operation, content-addressable.
-
-### `AgentProfile` is a versioned, content-addressable object
-
-```typescript
-// src/profile/types.ts
-
-export interface AgentProfileVersion {
-  /** Content-hash of the materialised profile state. */
-  hash: string
-  /** Parent in the lineage, null for the genesis profile. */
-  parentHash: string | null
-  /** Who wrote this version. */
-  source: 'harness' | 'substrate' | 'human'
-  /** When. */
-  timestamp: number
-  /** Human-readable label, optional. */
-  label?: string
-}
-
-export type ProfileDiff =
-  | { kind: 'patch'; edits: ProfileEdit[] }
-  | { kind: 'replace'; content: MutableSurface }
-
-export interface ProfileEdit {
-  /** Which surface inside the profile this edit targets. */
-  surface: 'systemPrompt' | 'skill' | 'tool' | 'mcp' | 'subagent' | 'modelByRole'
-  /** Surface-scoped identifier — skillName, toolName, mcpId, subagentId, role. */
-  surfaceId?: string
-  op: 'append' | 'insert_after' | 'replace' | 'delete'
-  target?: string
-  content: string
-  /** Support count from multi-trial evidence. */
-  supportCount?: number
-  /** Source classification for the merge/rank stage. */
-  sourceType?: 'failure' | 'success'
-}
-```
-
-That's the whole substrate type surface. Two types. No interface explosion.
-
-### `RunRecord` carries the version it was captured at
-
-Replace the existing `commitSha` / `promptHash` / `configHash` triple with a single canonical hash. Greenfield, no compat shim:
-
-```typescript
-// src/run-record.ts — IN-PLACE replacement
-export interface RunRecord {
-  // ... existing fields ...
-  /** Content-hash of the AgentProfileVersion that produced this run. */
-  agentProfileHash: string
-}
-```
-
-`commitSha`, `promptHash`, `configHash` become *inputs* to `hashProfile()`, not separate fields.
-
-### `selfImprove()` returns a diff, and the gate becomes 4-way
-
-Replace the current return shape. Greenfield, in place:
-
-```typescript
-// src/contract/self-improve.ts — IN-PLACE replacement
-export interface SelfImproveResult {
-  /** What we measured against. */
-  baselineHash: string
-  /** What we recommend applying. */
-  diff: ProfileDiff
-  /** Hash of `applyDiff(baseline, diff)` — verifiable by consumer. */
-  winningHash: string
-  /** Statistical evidence — paired bootstrap CI vs baseline. */
-  lift: LiftInsight
-  /** Substrate verdict — see DriftGateDecision below. */
-  gateDecision: DriftGateDecision
-  insight: InsightReport
-}
-
-export type DriftGateDecision =
-  | { kind: 'ship-substrate'; reason: string; vs?: 'baseline' | 'harness-live' }
-  | { kind: 'ship-harness'; reason: string }
-  | { kind: 'merge'; mergedDiff: ProfileDiff; reason: string }
-  | { kind: 'inconclusive'; reason: string }
-```
-
-When the substrate runs WITHOUT `driftPolicy: benchmark-branches`, only `ship-substrate` / `inconclusive` (or the equivalent `hold` framing) are possible. When `benchmark-branches` is on, all four kinds may surface.
-
-The substrate is now explicit: *"this diff is statistically valid against `baselineHash`. Whether to apply it to your live state is your call — and we'll tell you what we found when we compared branches."*
-
-### The opt-in drift policy
-
-```typescript
-selfImprove({
-  // ... existing
-  driftPolicy?:
-    | { kind: 'ignore' }                                   // default — assume single-writer
-    | { kind: 'reject-on-drift' }                          // cheap safety mode
-    | { kind: 'benchmark-branches'; benchmarkBudget: { generations, populationSize } }
-})
-```
-
-- **`ignore`** is the default. Same as today. Zero overhead for consumers whose sandbox harness doesn't self-modify.
-- **`reject-on-drift`** is the cheap safety mode. Substrate notices `currentHash != baselineHash` at apply time and refuses to ship. Tells the consumer "your profile drifted; re-run selfImprove against current state."
-- **`benchmark-branches`** is the full thing — only used when the harness DOES self-modify (Hermes per-turn, Claude Code with skill creation, Codex with user-prompted skill edits, agent-builder RL bridge, any future autonomous improvement loop). Costs an extra mini-campaign. Returns the 4-way `DriftGateDecision`.
-
-### Generalises past Hermes
-
-Any in-sandbox profile mutation appends to the same profile log, regardless of trigger:
-
-- Hermes-style autonomous (per-turn `background_review` fork)
-- Claude/Codex user-prompted ("hey, create a skill for X")
-- agent-runtime's runLoop self-modifying its prompt addendum
-- RL-style policy parameter updates
-- Manual user edits via `skill_manage` commands
-
-The substrate doesn't care WHY the harness wrote. It just sees: live profile is at hash X, my baseline was Y. Same merge protocol applies.
-
-### Conflict resolution — the four cases
-
-For the `benchmark-branches` policy, the substrate handles four cases:
-
-1. **No conflict.** Edits target different surfaces (substrate edited `systemPrompt`, harness wrote a new `skill/X.md`). Auto-merge into a combined candidate, benchmark merged vs each branch.
-
-2. **Orthogonal edits to the same surface.** Both touched `systemPrompt` but different H2 sections (subsumed by `GepaDriverConstraints.preserveSections`). Auto-merge by union of edits, benchmark.
-
-3. **Semantic duplication.** Substrate proposed a new skill `summarize-pr`; harness already created `pr-summarizer` (similar purpose, different file). Substrate runs a similarity-detection step: embed both, threshold cosine similarity, surface as a "duplicate-likely" finding. Resolution: head-to-head benchmark with both → keep the winner → archive the loser.
-
-4. **Direct same-region conflict.** Both edited the same paragraph. Three resolution paths the substrate offers:
-   - **Head-to-head**: run both branches, pick the winner.
-   - **LLM-mediated merge**: prompt an LLM with both candidate edits + the held-out failure trials, ask for a synthesis that addresses both. Benchmark the synthesis.
-   - **Human review**: surface the diff with `requires-resolution: true` and stop.
-
-### Sandbox-side merge protocol
-
-```typescript
-// agent-runtime exports:
-export async function getCurrentProfileVersion(): Promise<AgentProfileVersion>
-export async function applyDiff(diff: ProfileDiff): Promise<ApplyResult>
-
-export type ApplyResult =
-  | { ok: true; newHash: string }
-  | { ok: false; reason: 'conflict'; ancestor: string; ours: string; theirs: string }
-  | { ok: false; reason: 'stale-baseline'; expected: string; actual: string }
-```
-
-Sandbox keeps an append-only profile log at `~/.tangle/profile-log.jsonl`. Every harness write appends an entry. Every substrate-proposed apply appends or returns conflict.
-
-### The merge algorithm (3-way, surface-scoped)
-
-When substrate proposes `diff(baselineHash → winningHash)` but live state is at `currentHash != baselineHash`:
-
-1. **Walk the lineage** — find common ancestor of `baselineHash` and `currentHash`. If `baselineHash` IS an ancestor of `currentHash`, we have a clean rebase target.
-2. **Per-surface 3-way merge** — for each `ProfileEdit` in the diff:
-   - If the targeted surface (skillName, toolName, etc.) hasn't been touched in `currentHash` lineage since `baselineHash` → apply.
-   - If touched but the textual edit is on a different region → apply (no conflict).
-   - If touched on the same region → return `conflict` with ancestor/ours/theirs for the human or substrate to resolve.
-3. **Re-eval recommendation** — if non-trivial conflicts, recommend `selfImprove()` re-run against `currentHash` rather than blind merge.
-
-The consumer chooses: rebase + re-eval (statistically clean), force merge (skip re-eval, ship-at-own-risk), or reject (substrate's proposal is too stale).
-
-## How this changes the substrate flow
-
-```
-Today:
-  ingest_baseline_P0 → eval → winner W → consumer ships W (regardless of drift)
-
-Tomorrow:
-  ingest_baseline_hashed → eval → {baselineHash, diff, winningHash, lift, gate}
-                                  ↓
-  sandbox.applyDiff(diff) → ok | conflict | stale-baseline
-                          ↓
-  if stale-baseline:    substrate re-eval against currentHash
-  if conflict:          substrate proposes targeted resolution OR human reviews
-  if ok:                profile log gets a new entry, substrate notified
-```
-
-## What changes per package
-
-| Package | Files | Change |
-|---|---|---|
-| **agent-eval** | `src/profile/types.ts` (new) | `AgentProfileVersion`, `ProfileDiff`, `ProfileEdit` |
-| | `src/profile/hash.ts` (new) | `hashProfile()` — content-hash of the materialised state |
-| | `src/profile/diff.ts` (new) | `diffProfiles(a, b)`, `applyDiff(profile, diff)`, `threeWayMerge(ancestor, ours, theirs)` |
-| | `src/run-record.ts` | REPLACE `commitSha`/`promptHash`/`configHash` triple with `agentProfileHash` (greenfield) |
-| | `src/contract/self-improve.ts` | REPLACE `SelfImproveResult` to return `{baselineHash, diff, winningHash, lift, gateDecision, insight}` |
-| | `src/contract/analyze-runs.ts` | Add `agentProfileLineage` section to `InsightReport` — what versions ran, drift detected |
-| **agent-runtime** | `src/profile/log.ts` (new) | Append-only `~/.tangle/profile-log.jsonl`. `appendVersion()`, `readLineage()`, `findCommonAncestor()` |
-| | `src/profile/api.ts` (new) | `getCurrentProfileVersion()`, `applyDiff()` |
-| | `src/loops/run-loop.ts` | Every harness-side write to skills/memory/prompt-addendum appends to profile log |
-| **agent-knowledge** | `src/skills/version.ts` (new) | Skills become independently versioned objects; profile references them by `skillSetHash` |
-| **sandbox** | `src/agent-profile.ts` | Expose `getCurrentProfileVersion()` over the SDK |
-
-## What the gate semantics become
-
-`defaultProductionGate` today: "is the candidate statistically better than the baseline?"
-
-`defaultProductionGate` tomorrow: same question, scoped to the baseline. The consumer (sandbox / human / hosted-tier) decides whether to apply, given the answer + the current live state.
-
-We do NOT downgrade our paired-bootstrap CI. That's our edge over SkillOpt and Hermes. We just stop pretending the ship verdict is a deployment decision — it's a measurement.
-
-## The forcing function (task C from the audit)
-
-Before we commit weeks to this implementation, set up the empirical case:
-
-1. Run Hermes on top of our sandbox.
-2. Hermes' per-turn loop mutates skills.
-3. Run `selfImprove()` against the baseline at sandbox boot.
-4. Observe `gateDecision: ship` produce a winner that, when applied to the now-drifted live state, regresses.
-5. Capture the actual lift CI gap between `winner vs baseline` and `winner vs live`.
-
-If that gap is small (< MDE), profile-versioning is over-engineering. If it's large, this work is critical. We should know the number, not the intuition.
-
-## Phasing
-
-### Phase 0 — forcing function (1 week)
-Hermes-on-sandbox drift experiment. Real numbers on the gap. Either proves this work is needed or kills it.
-
-### Phase 1 — types + hashing (3 days)
-`AgentProfileVersion`, `ProfileDiff`, `ProfileEdit`. `hashProfile()`. `diffProfiles()`. `applyDiff()`. Pure functions, fully tested, no integration yet.
-
-### Phase 2 — substrate-side rewire (5 days)
-Replace `RunRecord` triple with `agentProfileHash`. Replace `SelfImproveResult` shape. Update `analyzeRuns` to detect lineage drift. Update tests + all 6 consumer products.
-
-### Phase 3 — sandbox + runtime (1 week)
-Profile log primitive in agent-runtime. `getCurrentProfileVersion()` + `applyDiff()` API. Sandbox SDK surface. Three-way merge for surface-scoped edits.
-
-### Phase 4 — agent-knowledge skill versioning (3 days)
-Skills become independently versioned. `skillSetHash` referenced from profile.
-
-### Phase 5 — Hermes adapter (3 days)
-Bridge: Hermes' `~/.hermes/skills/` write events → our profile log via a runtime hook.
-
-Total: ~3 weeks of focused work. Phase 0 in this session if Drew greenlights.
-
-## Source pointers
-
-- Task: #98
-- Related audit: `docs/specs/hermes-self-improvement-audit.md`
-- Related spec: `docs/specs/driver-honest-spec.md`
-- Current pre-versioning `RunRecord`: `src/run-record.ts`
-- Current pre-versioning `SelfImproveResult`: `src/contract/self-improve.ts`
-- Current gate: `src/campaign/gates/default-production-gate.ts`

package/docs/three-package-architecture.md

@@ -1,168 +0,0 @@
-# Three-package architecture: agent-eval × agent-knowledge × agent-runtime
-
-The Tangle agent stack splits responsibilities across three TypeScript
-packages with explicit, narrow contracts. This doc is the reference for how
-they fit together — what each owns, what each consumes from the others, and
-the canonical data shapes that move between them.
-
-## Why three packages
-
-Each one has a single, defensible job. Combining them was a real temptation
-(less version drift, fewer registries) and we said no on purpose:
-
-- **`@tangle-network/agent-eval`** owns measurement, optimization, and the
-  RL bridge. It has no opinion about *what* the agent does or *how* it runs;
-  it has strong opinions about whether the answer is good and how to make it
-  better.
-- **`@tangle-network/agent-knowledge`** owns the data side: source-grounded
-  knowledge graphs, source citations, eval-gated knowledge growth, knowledge
-  readiness scoring. It is domain-agnostic — legal, tax, coding, research
-  workflows define their own policies on top of it.
-- **`@tangle-network/agent-runtime`** owns the *execution* side: the task
-  lifecycle, knowledge-readiness gating, control-loop orchestration,
-  streaming session kernels. It does not own domain policy, models, tools,
-  or UI; it standardizes the lifecycle and delegates domain behavior to
-  adapters.
-
-Each package can be reasoned about independently. Each can be replaced
-without rewriting the others.
-
-## The data interchange — `RunRecord`, `Scenario`, `KnowledgeBundle`
-
-These three types travel between the packages and tie the architecture
-together.
-
-### `RunRecord` (owned by agent-eval)
-
-Every measurable thing — a campaign cell, an optimization trial, a
-production rollout, a deployment outcome — projects to a `RunRecord`. It
-carries identity (`runId`, `experimentId`, `candidateId`, `seed`,
-`scenarioId`), provenance (`commitSha`, `model`, `promptHash`, `configHash`),
-cost (`costUsd`, `tokenUsage`), and the outcome (per-split scores +
-free-form `raw` metric bag).
-
-agent-knowledge consumes `RunRecord[]` for release reporting and
-optimization analysis. agent-runtime exposes hooks for projecting its own
-task results into `RunRecord` shape. Every consumer of agent-eval's
-campaign / RL primitives produces `RunRecord[]`.
-
-### `Scenario` (currently each owner defines its own)
-
-agent-eval's `runEvalCampaign` takes
-`{ scenarioId: string; tags?: Record<string,string> }`. agent-knowledge
-defines richer scenario types for knowledge-base optimization. agent-runtime
-takes `TaskSpec` which is one task at a time, not a scenario set.
-
-This is a known minor friction; not load-bearing yet. When it becomes one,
-`Scenario` will get promoted to a shared interface.
-
-### `KnowledgeBundle` (owned by agent-knowledge)
-
-agent-knowledge produces `KnowledgeBundle` (a versioned graph of source
-citations + generated content) and `KnowledgeReadinessReport` (gap
-analysis). agent-eval's `KnowledgeRequirement` / `KnowledgeBundle` types
-are imported from agent-eval into agent-knowledge — agent-knowledge
-**adapts** its richer types to agent-eval's wire types, not the other way
-around. The wire types are the contract; the rich types are agent-knowledge's
-internal model.
-
-## Dependency direction
-
-```
-                        ┌────────────────────┐
-                        │  agent-runtime     │
-                        │  (executor)        │
-                        └─────────┬──────────┘
-                                  │
-                                  ▼ imports
-                        ┌────────────────────┐
-                        │  agent-eval        │
-                        │  (measurement)     │
-                        └────────────────────┘
-                                  ▲
-                                  │ imports
-                        ┌─────────┴──────────┐
-                        │  agent-knowledge   │
-                        │  (data side)       │
-                        └────────────────────┘
-```
-
-**Both** agent-runtime and agent-knowledge import agent-eval. agent-eval
-imports neither. This is deliberate: agent-eval is the leaf — its API is
-the bottleneck, so its surface stays narrow and stable.
-
-## What each package contributes to the auto-research loop
-
-```
-       ┌────────────────────┐           ┌────────────────────┐
-       │  agent-knowledge   │   ────►   │   agent-eval       │
-       │                    │           │                    │
-       │ - scenario sets    │           │ - runEvalCampaign  │
-       │ - knowledge bundle │           │ - capture integrity│
-       │ - readiness gates  │           │ - researchReport   │
-       │ - source citations │           │ - replayCampaign   │
-       │                    │           │ - sequential       │
-       │ produces:          │           │ - RL bridge        │
-       │   KnowledgeBundle  │           │ - preferences      │
-       │   Scenario         │           │ - off-policy       │
-       └────────────────────┘           │ - tournament       │
-                                        │                    │
-                                        │ produces:          │
-                                        │   RunRecord[]      │
-                                        │   PreferenceTriple │
-                                        │   etc.             │
-                                        └─────────▲──────────┘
-                                                  │
-       ┌────────────────────┐                     │
-       │   agent-runtime    │   ──────────────────┘
-       │                    │
-       │ - runAgentTask     │
-       │ - runAgentControl  │
-       │ - readiness gating │
-       │ - SSE / sessions   │
-       │                    │
-       │ produces:          │
-       │   ControlRunResult │
-       │   SSE events       │
-       └────────────────────┘
-```
-
-agent-knowledge brings the *what* (scenarios, knowledge, source data).
-agent-runtime brings the *how to run it once* (task lifecycle, control
-loop). agent-eval brings the *measurement and improvement* (campaign,
-report, RL bridge).
-
-## Cross-package contracts
-
-| From → To | Type | What it carries |
-|---|---|---|
-| agent-knowledge → agent-eval | `RunRecord` | (consumed via `runImprovementLoop` for knowledge-base optimization) |
-| agent-knowledge → agent-eval | `KnowledgeReadinessReport`, `KnowledgeBundle`, `KnowledgeRequirement` | (re-exported from agent-eval; agent-knowledge populates) |
-| agent-knowledge → agent-eval | `ControlRuntimeConfig<KnowledgeBaseCandidate>` | (knowledge research adapter) |
-| agent-runtime → agent-eval | `runAgentControlLoop`, `scoreKnowledgeReadiness`, `blockingKnowledgeEval` | (consumed; agent-runtime calls these in its task lifecycle) |
-| agent-runtime → agent-eval | `RunRecord`, `TraceStore`, `ControlRunResult`, `ControlStep` | (re-exported types; agent-runtime adapters projects into these) |
-| agent-eval ↘ neither package | (no upstream imports) | |
-
-## Known gaps in the contracts
-
-1. **Shared `Scenario` interface.** Each package has its own scenario
-   shape. agent-eval will promote a minimal `Scenario` to shared use when
-   the second consumer needs it.
-2. **agent-knowledge and agent-runtime pin older agent-eval minors.**
-   Until both bump to current, `RunRecord`'s `scenarioId` field won't be
-   populated by their existing run records and `RawProviderSink`
-   integration is per-consumer rather than automatic.
-3. **No first-class trace-analyst hook in agent-runtime.** agent-runtime's
-   `runAgentTask` emits traces but doesn't auto-execute the trace analyst
-   on completion the way `runEvalCampaign` does. A `onRunComplete` hook
-   on agent-runtime would close this.
-
-## Versioning policy
-
-Each package versions independently. The minor-version axis carries
-breaking changes; agent-eval's minor versions are tied to major
-methodological shifts.
-
-When agent-eval ships a minor, agent-knowledge and agent-runtime get a
-follow-up PR to consume the new surface. The follow-up is tracked as a
-deliberate change, not a passive caret pickup.