@tangle-network/agent-eval 0.189.0 → 0.190.1

package/docs/redaction.md

@@ -24,9 +24,11 @@ The report lists the JSON Pointer, category and detector of each change, never t
 | `share` | Also phone, IP and postal-address fields; pseudonymizes user, account, tenant, session and request ids | 64 KiB |
 | `strict` | Also prompt, completion, message, tool payload and other raw-content fields, and `data:` media | 4 KiB |
 
-A string that holds a credential is replaced whole with `[REDACTED:<detector>]`.
+A string that holds a credential shape is replaced whole with `[REDACTED:<detector>]`.
 Personal data inside prose is replaced in place.
 Pass `knownSecrets` to remove exact values in any form: as written, base64, base64url or URL-encoded.
+Each occurrence is cut out where it stands, so an error message that contains the key keeps the rest of its text.
+A whole-string base64 payload that holds a known secret at any byte offset is replaced whole.
 
 ## Field names
 
@@ -34,6 +36,10 @@ Pass `knownSecrets` to remove exact values in any form: as written, base64, base
 `apiKey`, `x-api-key`, `client.secret`, `refreshToken` and `OPENAI_API_KEY` are credentials.
 Token counts and model limits are not: `inputTokens`, `outputTokens`, `max_tokens`, `token_count`, `gen_ai.usage.input_tokens` and `next_page_token` survive.
 `author`, `auth_type`, `secret_name` and `content_type` also survive, because they name or describe a value instead of holding it.
+
+agent-interface's `looksLikeCredential` stays the candidate schemas' refusal check, and the core does not call it.
+On 629,707 strings from real sessions and VerticalBench runs, it flagged 280 strings the core passes, and all 280 were `Bearer ${token}`-style code or placeholders.
+The core flagged 39 credentials that `looksLikeCredential` passes.
 A number under a credential name survives, because it is a limit or a count.
 
 ## Share-safety verdict

package/docs/search-ledger.md

@@ -0,0 +1,174 @@
+# Search ledger
+
+A selected prompt, profile, or patch is not a record of the search that produced it.
+The search ledger is that record: a hash-chained event log of one search's nodes, edges, and cells, where every decision carries the evidence that justified it.
+It is the search's only lineage record and its only resume checkpoint.
+
+```text
+search-ledger.jsonl          canonical JSONL, SHA-256 hash chain, trusted-head pin
+  ├── blobs/<sha256>.json    surfaces, profiles, rationales, diffs, RunRecords
+  ├── SearchState            the one projection: invariants and the read model
+  └── SearchHistoryReceipt   bounded proof envelope for compareOptimizationMethods
+```
+
+## Terms
+
+- **Node:** a content-addressed artifact, for example an AgentProfile or a prompt.
+  `nodeId` is `searchNodeId(searchId, artifactDigest)`, so identical content in one search is one node.
+- **Edge:** the proposal that derived a node from its parents.
+  It records the operator, the proposer, the redacted rationale, and one parent-to-child diff per parent.
+  `edgeId` is `searchEdgeId(searchId, childNodeId, proposalKey)`, where the proposal key names the proposal within the search, so a resumed search recognizes its own edges.
+- **Cell:** one node on one task in one split at one repeat.
+  `cellId` is `searchCellId(searchId, nodeId, taskId, split, rep)`.
+  Each attempt at a cell has the run id `cellId:attempt`, the id its RunRecord carries.
+- **Unit:** the claim's independent unit, usually the task's source.
+  Repeats and sibling tasks of one unit average inside it before any statistic.
+- **Splits:** `train` is the proposer's feedback, `selection` is private to the policy and allocator, and `test` is sealed for the claim.
+
+## Events
+
+| Event | Records |
+|---|---|
+| `search-opened` | Subject, process, artifact kind, objective with judge and claim, the three splits with their tasks and units, policy, budget, containment, derivation, identity. First and once. |
+| `operation-started`, `operation-recorded` | A non-cell spend, such as one proposer call, with its reservation and then its outcome and accounting. |
+| `node-registered` | A node, its artifact digest and declared surfaces. |
+| `edge-recorded` | Parents (primary first), operator (`seed`, `draft`, `improve`, `debug`, `merge`, `derive`), attribution, proposer, selection rule, rationale, diffs, label. |
+| `cell-allocated` | A planned cell with its stage (`root`, `train`, `screen`, `rung`, `claim`, `external`), lane, and reservation. |
+| `cell-settled` | One attempt: outcome, accounting, time, placement, identity, surface evidence, trace reference, and optionally its RunRecord blob. |
+| `cell-cancelled` | A cell that will not run: `pruned`, `budget`, `deadline`, or `aborted`. |
+| `node-decided` | `advanced {rung}`, `pruned`, `invalid`, `finalist`, `selected`, or `rejected`, with its rule, reason, and the estimate it used. May repeat; the latest wins. |
+| `search-closed` | The stop reason and the claim: power, finalists, the selected node, and `ship`, `hold`, or `test-cannot-resolve`. Last and once. |
+
+Edge attribution says how the parents are known and is never inferred from timing or order:
+
+- `explicit`: the proposer that created the child emitted the edge.
+- `correlated`: an importer joined an optimizer's own parent record by content digest, for example GEPA's `parentIndices`.
+- `unknown`: no parent record exists, and the edge names no parent.
+
+## Invariants
+
+`SearchState` applies every entry and refuses one that breaks an invariant.
+The same code runs in the producer's journal, the kernel, and the Intelligence verifier.
+
+- **Order:** `search-opened` comes first; nothing follows `search-closed`. More work on a closed search is a derived search.
+- **Graph:** a node exists before an edge or cell names it, and a cell's node already has an edge. A parent registered before its child and already in the tree, so the graph is acyclic. A parent in another search appears only on a `derive` edge that matches `derivedFrom`.
+- **Re-proposal:** identical content is a second edge into the existing node, counted as a re-proposal.
+- **Splits:** a task belongs to one split and one unit. With `heldOutUnits`, no test unit appears in train or selection. Stages match splits: `claim` cells run on test, `screen` and `rung` on selection.
+- **Seal:** only the root (the first registered node) and nodes decided `finalist` run test cells.
+- **Attempts:** attempts count from 1 without gaps. A `passed` or `failed` outcome is final; only a retryable `errored` outcome admits another attempt.
+- **Budget:** at every `cell-allocated` and `operation-started`, committed spend plus open reservations plus the unspent claim reserve plus the new reservation stays within `maxUsd`. Spend above a reservation is recorded as overspend, never refused. An unknown cost counts as its proven floor.
+- **Completion:** `search-closed` needs every allocated cell settled or cancelled, every started operation recorded, and every node with an edge and a terminal decision. A `ship` claim needs a promoted finalist and held-out test units; any other claim keeps the root.
+
+A ledger written under another schema tag, such as the retired `tangle.search-ledger.v1` candidate-slot format, is refused with the tag named.
+It is never translated.
+
+## Reading the state
+
+`ledger.state()` and every append return a `SearchStateView`.
+Its `header`, `head`, `audit`, `completion`, and `closed` are plain values.
+Its node, edge, cell, unit-score, and lineage reads go to the live indexes, so taking a view costs the same at any search size.
+A view is valid until the ledger applies its next entry; a later read throws instead of mixing two ledger positions.
+
+```ts
+const state = await ledger.state()
+state.audit.cells // { allocated, settled, cancelled, open }
+state.scoredCells(nodeId, 'selection') // [{ cellId, unitId, attempt, score }]
+state.unitScores(nodeId, 'selection') // [{ unitId, sum, count, mean }]
+state.lineage(record.search) // { depth, ordinal, rep, containingRunId } for mintRolloutRows
+```
+
+A unit's mean sums its cells in cellId order, so it depends on the set of cells and not on the order they settled in.
+
+## Estimates
+
+`estimateNode(state, nodeId, { against, split })` is the one statistic of a node.
+It averages each node's scored cells inside their units, pairs the units both nodes scored, and runs `pairedDeltaTest` on the per-unit deltas.
+Unscored cells (errored, cancelled or in flight) are absent, never zero.
+
+| Pairs | `method` | Reported |
+|---|---|---|
+| 0 or 1 | `none` | nothing; the view shows "unknown (1 unit)" |
+| 2 to 5 | `insufficient` | `delta` only |
+| 6 to 19 | `descriptive` | `delta`, the bootstrap `interval` as spread, and `exactSignP` |
+| 20 or more | `bootstrap` | `delta` and the decision-grade `interval` |
+
+The thresholds are the library's own: `minimumPairsForPairedDeltaTest(0.95)` and `BOOTSTRAP_GATE_MIN_N`.
+`units` counts the units the node scored; `pairs` counts the units both nodes scored.
+`delta` is node minus `against` in the metric's units, so an improvement on a `minimize` objective is negative.
+`exactSignP` is one-sided toward improvement in the objective's direction.
+When every paired delta is equal, the estimate is `indeterminate`: its interval would have zero width, so it carries neither an interval nor a p-value.
+
+`cellSetDigest` digests the contrast and exactly the cells read: id, unit, attempt and score, in cellId order.
+Its first 32 bits seed the bootstrap, and `estimator` is `SEARCH_ESTIMATOR`, whose revision digests every parameter of the computation.
+Equal cells therefore give equal bits in any process and in any row order.
+A verifier that reads cells from its own store calls `estimateNodeFromCells` with them; `searchCellSetDigest` tells it whether a node's cells changed.
+
+```ts
+const estimate = estimateNode(state, childId, { against: parentId, split: 'selection' })
+await recorder.decideNode({ nodeId: childId, decision, basis: estimate, rule, reason })
+```
+
+`searchPosterior(state, { split })` gives every node a normal posterior on its improvement over the root, for sampling parents.
+Its mean is the node's mean per-unit improvement over the root, oriented so that larger is better in either direction.
+Its variance is the search's pooled between-unit variance of those improvements divided by the node's shared units, so a node measured on one unit has a wide posterior, not none.
+The root sits at exactly 0, and the pooled variance stays null until some node shares 2 units with the root.
+Posterior numbers steer spend and claim nothing; a claim comes from the sealed test split.
+
+## Record a search
+
+`SearchRecorder` writes each fact the moment it exists.
+Ids are deterministic and every write checks the ledger's state first, so a resumed search continues its ledger instead of conflicting with its own history.
+Rationales and labels are redacted with the `share` profile before they are hashed; surfaces are stored as they ran.
+
+```ts
+import { openSearchLedger, SearchRecorder, surfaceNode } from '@tangle-network/agent-eval/campaign'
+
+const ledger = openSearchLedger({ path: `${runDir}/search-ledger.jsonl`, searchId })
+const recorder = await SearchRecorder.open({ ledger }, opening)
+const root = await recorder.registerNode(surfaceNode(recorder, baseline))
+await recorder.recordEdge({
+  childNodeId: root.nodeId, parents: [], operator: 'seed', attribution: 'explicit',
+  proposer: null, proposalKey: 'baseline', rationale: { unknown: 'the starting surface' }, diffs: [],
+})
+const cellId = await recorder.allocateCell({ nodeId: root.nodeId, taskId, split: 'selection', rep: 0, stage: 'root' })
+await recorder.settleCell({ cellId, outcome, accounting, identity, runRecord })
+```
+
+`runOptimization({ searchLedger: { ledger, identity } })` and `selfImprove` record through the recorder and return the receipt on `searchHistory`.
+The loop's scenarios are its proposer's feedback, so they are the train split; its promotions are budget decisions and it makes no claim.
+
+`gepaOptimizationMethod({ searchLedger: { identity } })` records GEPA's search when it finishes.
+`importGepaPopulation` turns the population into nodes and `correlated` edges and reports collapsed duplicates.
+`importExternalEvaluations` turns every callback evaluation into an `external` cell; a candidate GEPA evaluated but kept out of its population gets an `unknown` edge and is decided `pruned`.
+
+## Receipts and `require-complete`
+
+`createSearchHistoryReceipt({ producerId, runId, ledger })` reads the ledger's bytes, replays them, and binds the byte digest, the audit digest, and a bounded summary.
+A receipt is complete exactly when the ledger holds `search-closed`, which the completion invariant admits only when nothing is outstanding.
+
+```ts
+const comparison = await compareOptimizationMethods({
+  // ...methods, partitions, dispatch, judges, runDir
+  searchHistoryPolicy: 'require-complete',
+  searchHistoryVerification: 'ledger',
+  storage,
+})
+```
+
+Under `require-complete`, a missing, malformed, producer-mismatched, or open search refuses the final comparison before its first test dispatch.
+`searchHistoryVerification: 'ledger'` also resolves the receipt's URI through `storage.read`, checks the bytes' SHA-256 and length, and replays them.
+`verifySearchHistoryReceipt` checks the envelope alone; `assertSearchHistoryMatchesState` checks it against a replayed state.
+
+A receipt does not prove that the optimizer searched well, that a node is correct or safe, or that a winner generalizes.
+Those claims need the sealed test split, the claim's power check, and held-out evaluation.
+
+## Files
+
+- `src/campaign/search-ledger-types.ts`: the event and audit types.
+- `src/campaign/search-ledger.ts`: schemas, canonical ordering, the codec, and `FileSearchLedger`.
+- `src/campaign/search-state.ts`: `SearchState`, the invariants and read model, and the id functions.
+- `src/campaign/estimate-node.ts`: `estimateNode`, `estimateNodeFromCells` and `searchPosterior`.
+- `src/campaign/search-ledger-recording.ts`: `SearchRecorder` and the surface helpers.
+- `src/campaign/gepa-search-import.ts`: the GEPA population and evaluation importers.
+- `src/campaign/search-history-receipt.ts`: receipts and admission.
+- `src/ledger-core/`: hashing, locking, durable appends, chain verification, and the trusted-head pin.

package/docs/trace-contracts.md

@@ -4,7 +4,7 @@ A trace contract states which path an agent run must take, and checks the record
 Two runs can give the same answer while one of them calls a forbidden tool; an output check passes both, and a contract fails the wrong one.
 
 The checker is deterministic and calls no model.
-It reads any span array that has `name`, `kind` or span-kind attributes, timestamps, and attributes: agent-eval `TraceStore` spans, otel-bridge spans, or OTLP rows that `traces check` maps.
+It reads any span array that has `name`, `kind` or span-kind attributes, timestamps, and attributes: agent-eval `TraceStore` spans or OTLP rows that `traces check` maps.
 
 ## Write a contract
 
@@ -27,7 +27,7 @@ Write the contract as JSON and compile it with `compileTraceContractSpec`:
 
 | Key | Checks |
 |---|---|
-| `run.requireCompleted` | The run reached a terminal status and recorded an end time. |
+| `run.requireCompleted` | The run status is `completed` and its end time is recorded. A failed or aborted run fails. |
 | `run.allowedStatuses` | The run status is one of `running`, `completed`, `failed`, `aborted`. |
 | `run.maxDurationMs` | End minus start. An unknown duration fails. |
 | `tools.required` | Each tool is called at least once. |
@@ -36,7 +36,9 @@ Write the contract as JSON and compile it with `compileTraceContractSpec`:
 | `tools.maxCalls`, `tools.maxCallsPerTool` | Ceilings on tool calls, overall and per tool. |
 | `tools.requiredOrder`, `tools.orderMode` | Each tool is called, and each comes before the next. |
 | `tools.arguments` | An RFC 6901 JSON Pointer check (`exists`, `equals`, `oneOf`, `type`) on tool-call arguments. A call without captured arguments fails. |
-| `llm.maxCalls`, `llm.maxTotalTokens` | Ceilings on model calls and on input plus output tokens. An unrecorded token count fails. |
+| `llm.maxCalls`, `llm.maxTotalTokens` | Ceilings on model calls and on input plus output tokens. Input includes cache-read and cache-write tokens when the span records them, per the OTel GenAI definition — a cached call's prompt-token count alone omits most of what it spent. An unrecorded token count fails. |
+| `tools.enforced` | The trace records the tools the harness offered the model (`gen_ai.tool.definitions`), every call is one of them, and every offered tool is in `tools.allowed`. A trace that records no offered tools fails. |
+| `retries.reads`, `retries.writes` | A tool called again with the same arguments repeats its side effect. A read may repeat. A write may repeat only when every call carries the same idempotency key at `idempotencyKey`. Any other tool may not repeat. |
 | `llm.allowedModels` | Every model call names a listed model. A call without a model fails. |
 | `scope.root` | Check only the one span that matches this predicate and its descendants: one sub-agent or one search-tree node. Zero or several matches make every rule an error. |
 | `alternatives.anyOf` | Named paths, each with its own `run`, `tools`, `llm` and `rules`. At least one must pass completely, as well as the base checks. Use it for a legitimate shortcut such as a cache hit. |
@@ -50,6 +52,45 @@ An unknown key or an unknown status is an error that names the closest known wor
 `lintTraceContractSpec` reports contradictions (a tool that is both required and forbidden) and likely mistakes (an order that uses the default `start-order` mode, which passes overlapping calls).
 `explainTraceContract` states each compiled rule in one line.
 
+## Check what the harness enforced
+
+A tool list in a prompt or a CLI flag is a declaration, not proof of enforcement.
+`tools.enforced` compares three sets: the tools the contract allows, the tools the harness offered the model, and the tools the run called.
+`traces` records the offered set from the `system` `init` record of `claude -p --output-format stream-json --verbose` output.
+Any OTel GenAI producer that sets `gen_ai.tool.definitions` works the same way.
+
+In a recorded run, `claude -p --tools Read,Grep` also offered eight claude.ai connector tools, so the contract below failed.
+The same run with `--strict-mcp-config` offered only `Read` and `Grep`, and passed.
+
+```json
+{
+  "name": "refund-desk-enforced",
+  "tools": { "required": ["Read"], "allowed": ["Read", "Grep"], "enforced": true }
+}
+```
+
+## Check that a retry cannot apply twice
+
+A write that times out may still have applied.
+Calling it again with the same arguments applies it twice unless the target deduplicates on an idempotency key.
+
+```json
+{
+  "name": "payments",
+  "retries": {
+    "reads": ["lookup_order"],
+    "writes": [{ "tool": "charge_card", "idempotencyKey": "/idempotency_key" }]
+  }
+}
+```
+
+The check groups calls by tool and arguments, without the idempotency key.
+A group of more than one call passes for a read, and for a write only when every call carries the same key.
+A repeated call of an unlisted tool fails, because its side effect is unknown.
+A call without captured arguments fails when its tool is called more than once, because the checker cannot tell whether it repeats.
+Grouping matches arguments exactly: a retry that re-encodes a value (`42.5` vs `"42.50"`) is not recognized as the same call.
+Independent of grouping, a declared write whose call itself errored (including a timeout) and carries no idempotency key always fails — its outcome at the target is unknown, so nothing proves a retry, however it was encoded, would be safe.
+
 ## Rule semantics
 
 Every rule reports `pass`, `fail` or `error` in `ContractVerdict.ruleExecutions`.
@@ -68,6 +109,8 @@ A required successor that never ran fails `precedes`, a span without the timesta
 | `atMost(p, max)`, `tokensAtMost(p, max)` | Call and token ceilings. |
 | `run({ ... })` | The run status and duration. |
 | `argument(p, { pointer, check, occurrence })` | A JSON Pointer check on tool arguments. |
+| `toolsOffered(declared)` | The offered tools are recorded, every call is one of them, and every offered tool is declared. |
+| `retrySafe({ reads, writes })` | No call repeats a side effect that is not proven safe to repeat. |
 
 Ordering has three modes.
 `start-order` (the default) needs an `a` that started before each `b`.
@@ -87,6 +130,8 @@ const result = checkTraceContracts(await store.spans({ runId }), [contract])
 
 ## Gate CI on a contract
 
+**Unreleased.** `traces check` lives on `feat/traces-check` in `/traces` (PR #128), proven against this package but not yet mergeable: it needs the traces/agent-runtime redaction-core migration first, so a `/traces` build on this package's contract API does not crash at load. Until then, call `evaluateTraceContract` / `checkTraceContracts` directly, as shown above.
+
 `traces check` runs a contract over a recorded trace and exits with a code a CI step can read:
 
 ```sh

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tangle-network/agent-eval",
-  "version": "0.189.0",
+  "version": "0.190.1",
   "description": "Evaluate and improve AI agents from runs, traces, judges, and feedback. Compare candidates, cluster failures, measure lift, and gate releases.",
   "homepage": "https://github.com/tangle-network/agent-eval#readme",
   "repository": {