@tangle-network/agent-eval 0.191.0 → 0.193.0

package/docs/search-ledger.md

@@ -54,10 +54,13 @@ The same code runs in the producer's journal, the kernel, and the Intelligence v
 - **Graph:** a node exists before an edge or cell names it, and a cell's node already has an edge. A parent registered before its child and already in the tree, so the graph is acyclic. A parent in another search appears only on a `derive` edge that matches `derivedFrom`.
 - **Re-proposal:** identical content is a second edge into the existing node, counted as a re-proposal.
 - **Splits:** a task belongs to one split and one unit. With `heldOutUnits`, no test unit appears in train or selection. Stages match splits: `claim` cells run on test, `screen` and `rung` on selection.
-- **Seal:** only the root (the first registered node) and nodes decided `finalist` run test cells.
+- **Seal:** only the root (the first registered node) and nodes decided `finalist` run test cells. The root is never a finalist, at most 3 nodes are, and none is added once a claim cell exists.
+- **Claim start:** once a node is decided `finalist`, the search only claims: no operation, node, edge or non-claim cell follows.
+- **Parents:** a node decided `invalid` (by admission or by the divergence rule) never appears as a parent on a later edge.
 - **Attempts:** attempts count from 1 without gaps. A `passed` or `failed` outcome is final; only a retryable `errored` outcome admits another attempt.
 - **Budget:** at every `cell-allocated` and `operation-started`, committed spend plus open reservations plus the unspent claim reserve plus the new reservation stays within `maxUsd`. Spend above a reservation is recorded as overspend, never refused. An unknown cost counts as its proven floor.
-- **Completion:** `search-closed` needs every allocated cell settled or cancelled, every started operation recorded, and every node with an edge and a terminal decision. A `ship` claim needs a promoted finalist and held-out test units; any other claim keeps the root.
+- **Completion:** `search-closed` needs every allocated cell settled or cancelled, every started operation recorded, and every node with an edge and a terminal decision.
+- **Claim:** the claim names every node decided `finalist`, estimates each against the root on test, and tests each at `1 - (1 - confidence) / k` for its k finalists. A `ship` claim needs a promoted finalist, held-out test units, a pinned judge, and every test unit scored by the root and the shipped finalist; any other claim keeps the root.
 
 A ledger written under another schema tag, such as the retired `tangle.search-ledger.v1` candidate-slot format, is refused with the tag named.
 It is never translated.
@@ -114,6 +117,28 @@ Its variance is the search's pooled between-unit variance of those improvements
 The root sits at exactly 0, and the pooled variance stays null until some node shares 2 units with the root.
 Posterior numbers steer spend and claim nothing; a claim comes from the sealed test split.
 
+## The agent's view
+
+`renderSearchSummary(state, { split, limit? })` turns a `SearchStateView` into compact text: the leading nodes against the root with their estimates, the most recently discarded nodes with the measurement that discarded them, and a log of recent proposals — as AIDE's journal summary does.
+`agent-eval search show <ledger>` verifies a ledger and prints this for a terminal, on the search's own ranking split (`selection` when the search declares one, else `train`); there is no local HTML renderer.
+
+```ts
+import { renderSearchSummary, searchProposerView } from '@tangle-network/agent-eval/campaign'
+
+renderSearchSummary(state, { split: 'train' })
+// search s1 — vb/coder — maximize composite
+//   12 nodes · 34 settled cells (2 open) · $1.42 known
+//   status: open
+//
+// Leading nodes (vs root, train split):
+//   node_… (advanced rung 2): Δ=+0.0821 [0.0340, 0.1290] dashed, sign p=0.0156 (8 units)
+// ...
+```
+
+`ProposeContext.train` (a `SearchProposerView`) and `ProposeContext.summary` (this text, always rendered on `split: 'train'`) are how `runOptimization`'s proposal step hands a proposer the search so far.
+`searchProposerView(state)` has no parameter that can select another split: its `scoredCells`/`unitScores` always read `'train'`, so a proposer built on it cannot reach the sealed selection or test split even by mistake.
+`ProposeContext.parents` carries every parent the policy chose, primary first, with its artifact — `currentSurface` is `parents[0].artifact`; a `merge` proposal needs every parent, and the built-in `incumbent`/`crowdedFrontierParent` policies always choose one.
+
 ## Record a search
 
 `SearchRecorder` writes each fact the moment it exists.
@@ -134,13 +159,108 @@ const cellId = await recorder.allocateCell({ nodeId: root.nodeId, taskId, split:
 await recorder.settleCell({ cellId, outcome, accounting, identity, runRecord })
 ```
 
-`runOptimization({ searchLedger: { ledger, identity } })` and `selfImprove` record through the recorder and return the receipt on `searchHistory`.
+`runOptimization` and `selfImprove` run on the kernel below and return the receipt on `searchHistory`.
 The loop's scenarios are its proposer's feedback, so they are the train split; its promotions are budget decisions and it makes no claim.
 
 `gepaOptimizationMethod({ searchLedger: { identity } })` records GEPA's search when it finishes.
 `importGepaPopulation` turns the population into nodes and `correlated` edges and reports collapsed duplicates.
 `importExternalEvaluations` turns every callback evaluation into an `external` cell; a candidate GEPA evaluated but kept out of its population gets an `unknown` edge and is decided `pruned`.
 
+## Run a search: the kernel
+
+`runSearch` is the one loop every optimizer runs on.
+It separates three decisions: where to expand (a `SearchPolicy`), where to spend rollouts (a `SearchAllocator`), and what to claim (the claim step on the sealed test split).
+There is no generation barrier.
+A lane that frees up takes the next allocated cell: claim cells first, then rung and root cells, then screens, then train cells.
+The policy proposes when no cell waits, fewer than twice the lanes' capacity run, and the cap admits one proposal and its expected screens.
+
+```ts
+import { incumbent, runSearch, SearchRecorder, uniform } from '@tangle-network/agent-eval/campaign'
+
+const policy = incumbent({ patience: 3 })
+const allocation = uniform({ reps: 1 })
+const recorder = await SearchRecorder.open({ ledger }, {
+  ...opening,
+  policy: { expansion: policy.name, allocation: allocation.name, seed },
+})
+const { state, leader, reason, claim } = await runSearch({
+  recorder, root, codec, policy, allocation, proposer, executor, maxExpansions: 10,
+})
+```
+
+The ports:
+
+- **Executor:** `lanes()` declares pools of slots, each `hard` (it enforces `cellUsd` per cell) or `estimate` (it cannot); `place(cell)` picks a cell's lane; `run(work)` runs one attempt and returns its outcome, accounting and identity; `adopt(work)` returns an attempt an earlier process finished, or null.
+  An environment fault is an `errored` outcome; a rejection stops the search.
+- **Proposer:** `propose({ parents, operator, leader })` returns children with a label, a rationale and optional typed `attribution`, plus the operation's accounting.
+  Its output is stored as a `proposal` blob on `operation-recorded` before any child is registered.
+- **Codec:** `node(recorder, artifact)` content-addresses an artifact, `diff` stores the parent-to-child diff, and `load` reads a node's artifact back.
+- **Policy:** `expand(view)` returns parents and an operator, or null to wait; `leader(view)` names the node the search keeps.
+  The view holds the policy split (selection, or train when a search has none) and no test cell.
+  `incumbent({ patience })` is the hill climb: it expands the leader once every earlier child is screened, and a node that scored every unit of its screen leads when it beats the leader on the units they share.
+  `crowdedFrontierParent({ seed })` draws the parent from the Pareto frontier by a seeded crowded tournament and keeps the incumbent's leader rule.
+- **Allocator:** `plan(state, nodeId)` lists the cells a node needs now; the kernel allocates the ones the ledger lacks.
+  `uniform({ reps })` gives the root every train and selection task as `root` cells, and every other node every train task (`train`) and selection task (`screen`).
+
+**Budget.**
+Every reservation passes the ledger's admission rule: committed spend plus open reservations plus the unspent claim reserve plus the new hold stays within `maxUsd`.
+The kernel checks `state.budget.headroomUsd` first and prices an expansion as one proposal plus `childrenPerProposal` screens.
+A hard lane holds its maximum; an estimate lane holds 1.5 times the p99 of its settled cells once 20 settled, else its prior.
+Spend above a hold is recorded as overspend, never refused.
+Expansion stops at `maxNodes`, `maxExpansions`, the deadline, after `patience` expansions without a new leader, when the cap cannot admit one more expansion, or when the proposer stops.
+Allocated cells still run; at the deadline the ones not yet started are cancelled.
+A search without a test split closes with no claim: the leader is decided `selected` when it has a scored cell, and every other undecided node `rejected` with its estimate against the leader.
+A search with a test split closes with its claim (below).
+
+**Divergence.**
+When a node's cells finish, the kernel compares it with its primary parent.
+If its train mean rose on the train units they share while its selection interval against the parent (6 or more units, not indeterminate) lies wholly on the worse side, the node is decided `invalid` with rule `divergence` and the interval as its basis.
+Train is what the proposer reads, so a gain there that selection contradicts is the signature of fitting the feedback instead of the task.
+An invalid node is never a parent and never a finalist.
+
+**Resume.**
+The ledger is the only checkpoint.
+Running the kernel again on an open ledger replays it and continues.
+An operation that started without a result is recorded `failed` with an unknown cost and a floor of 0, and the next proposal runs under a new operation id.
+A recorded proposal whose children were not all registered is finished from its stored output.
+A cell the ledger shows unsettled is offered to `executor.adopt` before its first dispatch, so an attempt that finished before the restart is recorded once and not run again.
+Aborting the `signal` pauses the search: in-flight cells are aborted, a scored result that still arrives is recorded, an attempt that ends in an error while the search stops stays unsettled (the error may be the interruption), and the ledger stays open.
+One kernel runs a ledger file at a time on a host: a second one is refused by a pid lock beside the ledger, and a killed holder's lock is reclaimed.
+A closed ledger returns its result; more work on a closed search is a new search.
+
+`scripts/search-sim.ts` runs the real kernel, ledger, policies and claim over a seeded synthetic objective, proposer and executor.
+`kill-resume` SIGKILLs it at random ledger positions and compares the resumed search with an uninterrupted one.
+`claims --searches 200 --null` runs 200 searches in which no node differs from the root and reports how often the claim ships, with Wilson and Clopper-Pearson intervals; `--plant-gain X` plants a real gain to measure how often the claim finds it.
+
+## The claim
+
+A search that declares a test split ends in its claim, made once, on test data no node was selected on.
+`runSearch` refuses to start such a search without a selection split, a claim `minimumEffect`, and a pinned judge.
+Under a cap it also refuses a `reservedClaimUsd` below `searchClaimReserveUsd({ testTasks, reps, cellUsd })`, the hold for the root and 3 finalists on every test task, or a cap that cannot also cover the root and one screening round.
+Size the reserve at the hold the lane will take: an estimate lane holds 1.5 times the p99 of its settled cells once 20 settled.
+
+When expansion stops, the kernel:
+
+1. **Fixes the design.** `planSearchClaim` ranks the finalists: at most 3 non-root nodes, not invalid, that scored every selection unit and beat the root's mean on them, best selection mean first.
+   It fixes the estimator (binary when every unit is one task at one repeat and every pre-test score is 0 or one value, else continuous).
+   It checks power: `pairedPromotionPower` simulates the claim's own `decidePairedPromotion` call at a true improvement of `minimumEffect`, with the search's pooled between-unit selection variance, on the test units, at the Bonferroni confidence.
+   It keeps the largest k whose power reaches 0.8 and whose test cells the unspent claim reserve and headroom cover.
+   The plan is stored as the `claim-plan` blob on the `claim` operation before any test cell exists.
+2. **Seals the family.** Each planned finalist is decided `finalist` with its selection estimate against the root.
+   From here the search only claims.
+3. **Runs the test together.** The root's and the finalists' test cells are allocated task by task and dispatched before anything else, so drift over a long search cannot bias the pairs. They run past the deadline, on the reserve held since the start.
+4. **Decides.** `decideSearchClaim` tests each finalist against the root with `decidePairedPromotion` on per-unit test means at confidence `1 - 0.05 / k`, requiring every test unit.
+   The promoted finalist with the largest improvement is `selected` and the claim is `ship`; otherwise the claim is `hold` and the root is kept.
+   Every other open node is `rejected` with its selection estimate against the root.
+
+When the power check fails (a continuous claim needs at least 20 test units to be decided at all) or the reserve cannot cover one finalist, the claim closes as `test-cannot-resolve` and spends nothing on test cells.
+The claim records its rule (`SEARCH_CLAIM_RULE`, whose revision digests every parameter), the family-wise confidence, the power check, each finalist's test estimate and deciding interval, and its reason.
+A verifier recomputes it with `decideSearchClaim(state, plan)` from the stored plan and the test cells.
+A judge change is a changed `search-opened` header, which `SearchRecorder.open` refuses; it starts a derived search instead of mixing verdicts.
+
+`compareOptimizationMethods` keeps its own held-out comparison for black-box methods such as GEPA, which return one winner and never see the test split.
+A method's search ledger closes before the comparison starts, so the comparison cannot add claim cells to it.
+
 ## Ship a search
 
 A hosted store (Intelligence, or the reference receiver in `examples/hosted-ingest-server/`) receives a search through the [hosted ingest wire](./hosted-ingest-spec.md).
@@ -185,7 +305,13 @@ Those claims need the sealed test split, the claim's power check, and held-out e
 - `src/campaign/search-ledger.ts`: schemas, canonical ordering, the codec, and `FileSearchLedger`.
 - `src/campaign/search-state.ts`: `SearchState`, the invariants and read model, and the id functions.
 - `src/campaign/estimate-node.ts`: `estimateNode`, `estimateNodeFromCells` and `searchPosterior`.
+- `src/campaign/search-summary.ts`: `renderSearchSummary` and `searchProposerView`.
 - `src/campaign/search-ledger-recording.ts`: `SearchRecorder` and the surface helpers.
+- `src/campaign/search-kernel.ts`: `runSearch`, the executor, proposer and codec ports, `searchPolicyView` and `searchDivergence`.
+- `src/campaign/search-claim.ts`: `planSearchClaim`, `decideSearchClaim` and `searchClaimReserveUsd`.
+- `src/campaign/search-policy.ts`: `SearchPolicy`, `incumbent` and `crowdedFrontierParent`.
+- `src/campaign/allocation.ts`: `SearchAllocator` and `uniform`.
+- `src/campaign/presets/run-optimization.ts`: `runOptimization` as a search on the kernel.
 - `src/campaign/gepa-search-import.ts`: the GEPA population and evaluation importers.
 - `src/campaign/search-history-receipt.ts`: receipts and admission.
 - `src/ledger-core/`: hashing, locking, durable appends, chain verification, and the trusted-head pin.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tangle-network/agent-eval",
-  "version": "0.191.0",
+  "version": "0.193.0",
   "description": "Evaluate and improve AI agents from runs, traces, judges, and feedback. Compare candidates, cluster failures, measure lift, and gate releases.",
   "homepage": "https://github.com/tangle-network/agent-eval#readme",
   "repository": {