@tangle-network/agent-eval 0.144.5 → 0.144.7

package/docs/experiment.md

@@ -0,0 +1,104 @@
+# The experiment subpath
+
+`@tangle-network/agent-eval/experiment` turns an experiment's registration into the object that runs it.
+
+## The covenant
+
+1. **The registered rule is the executed rule.**
+   Every rule — row admission, subset selection, estimand, interval, decision table, validity gate, halt, budget, matched budget, reissue — is a typed data node, never a closure or prose.
+   `sealExperiment` canonicalizes and hashes the whole tree into one digest.
+   Every interpreter takes only a sealed node plus evidence records.
+   No execution surface has a parameter for alpha, threshold, metric, or stopping rule, so registered-vs-ran drift is unrepresentable rather than checked.
+2. **Refusals live inside artifacts.**
+   An inadequate cluster count, a mismatched arm budget, a non-monotone funnel stage, a non-total decision table — each produces a typed verdict object (or a typed error), never a warning sentence beside a number.
+3. **A change is a re-seal.**
+   `amendExperiment` verifies the current seal, validates the new spec, and appends a `{at, reason, blind[], digest}` entry.
+   The digest history is the audit trail; changing what is decided without a new digest is impossible.
+
+## The objects
+
+| object | entry point | what it closes |
+| --- | --- | --- |
+| Registered-rule AST | `src/experiment/ast.ts` (14 node families) | prose rules; every registered condition compiles to data the seal covers |
+| Define / seal / execute | `defineExperiment`, `sealExperiment`, `amendExperiment`, `openSealedExperiment` | hand-written PREREG.md files; the runner executes the sealed rule itself |
+| Cluster-aware power | `clusteredPower`, `assertDesignAdequate` | "4 clusters cannot certify any effect size, including 1.0" — learned by running the experiment, now refused before a dollar is spent |
+| Denominator chain | `buildFunnel`, `executeAdmissionRule`, `composeFunnels`, `renderFunnelTable` | hand-assembled `20 → 15 → 14 → admitted` chains, formatted differently each run |
+| Matched budgets | `verifyMatchedBudgets`, `assertMatchedBudgets` | "realized tokens must agree within 5%" verified by hand |
+
+### Sealing and execution
+
+```ts
+import {
+  openSealedExperiment,
+  sealExperiment,
+} from '@tangle-network/agent-eval/experiment'
+
+const sealed = await sealExperiment(spec)      // canonicalize + sha256 over the whole tree
+const registered = await openSealedExperiment(sealed) // verifies the digest first
+
+const admission = registered.admit(rows)       // funnel + survivors, from the sealed rule
+const gate = registered.gate('power-floor', { kind: 'power-floor', curve })
+const halt = registered.halt([gate])           // refuse-spend fires before any contrast
+const outcome = registered.decide(quantities)  // the sealed table; non-total tables throw
+```
+
+`openSealedExperiment` is the only execution surface.
+A rule that is not in the sealed spec cannot run; a rule that is cannot run differently.
+
+### Cluster-aware power refusal
+
+Two floors, one simulation:
+
+- **Closed form, zero spend.** With `C` independent clusters, the exact whole-cluster sign-flip test can never produce a two-sided p below `2^(1-C)`.
+  Four clusters give 0.125 and three give 0.25 — both above alpha 0.05, so those designs are refused at any effect size.
+  Six clusters is the smallest certifiable count at 0.05.
+- **Seeded simulation.** Per-row paired contrasts are drawn under a registered effect model (base win/loss rates, optional noisy clusters), each trial takes a whole-cluster percentile bootstrap, and power is the fraction of trials whose interval excludes zero.
+
+The refusal is a verdict inside the returned artifact (`result.refusal`), with `assertDesignAdequate` as the throwing form.
+The `power-floor` validity gate consumes a power curve as evidence and fails when the curve tops out under the registered target.
+
+### The funnel
+
+`buildFunnel` refuses a stage that gains rows, named exclusions that do not sum, and partitions that overdraw their source stage.
+`executeAdmissionRule` runs a sealed admission rule over rows and returns the funnel, the survivors, and the partition rows in one object — the chain and the rows can never disagree.
+Partitions carry `pooling: 'never'`: a secondary set is reported beside the primary chain and cannot be pooled into it.
+The object is its own JSON render; `renderFunnelTable` prints the text table with the reconciliation line (`input = surviving + excluded`).
+
+### Matched budgets
+
+`verifyMatchedBudgets` compares realized per-arm tokens under the registered tolerance and returns a verdict whose `refusal` field carries `onFail: 'refuse-contrast'` when arms diverge.
+A contrast between arms that spent differently is not a contrast; the refusal is the artifact that says so.
+
+## Acceptance: the three preregistrations
+
+The module's acceptance suite (`tests/experiment/preregistration-acceptance.test.ts`) re-derives the week's three hand-written preregistrations as sealed specs and reproduces each recorded decision by executing the sealed rules against the recorded evidence:
+
+- **killtest-20260810** — all four validity gates fail on the recorded evidence (the rep-4 oracle flip, the 2-row population drift, the zero-call control, the 0.692 power ceiling) and the halt rule refuses the spend, matching the recorded `$0.00, contrast never run`.
+  The obligation node routes a positive interval without the registered control to `blocked-pending-registered-control`, never to `thesis-survives`.
+- **freelunch-20260810** — the admission funnel reproduces the recorded `48 > 43 > 35 > 35 > 32` chain with the 3-row secondary partition; the uniform-pass budget reproduces the recorded uniform n=2; the amendment-6 ledger under the same sealed rule refuses pass 2 — the registered-vs-ran divergence the seal makes unrepresentable; the report-only decision reproduces `3/64` and `2/32`.
+- **tbench-20260808 milestone 2** — the round-robin selection reproduces the recorded 20-row subset in pick order; the m3 subset filters the SEALED m2 draw (16 rows); the decision table on the recorded interval reproduces `not-certified-at-this-n`.
+
+## What is composed, not duplicated
+
+The statistical machinery underneath is re-exported from its existing homes; this subpath adds registration and refusal, not estimator forks.
+
+| family | home |
+| --- | --- |
+| `pairedBootstrap`, `mcnemar`/`mcnemarPower`/`mcnemarRequiredN`, `pairedRiskDifference*`, `holm`, `benjaminiHochberg`, `eProcess`, `wilson`, `mulberry32`, sample-size helpers | `src/statistics.ts` |
+| `clusteredPairedBinary` (cluster bootstrap + sign flip) | `src/clustered-paired-binary.ts` |
+| `pairedEvalueSequence` (anytime-valid) | `src/sequential.ts` |
+| `powerPreflight` (variance-based MDE refusal) | `src/campaign/gates/power-preflight.ts` |
+| `sequentialPairedGate`, `sequentialDecide` (manifest-bound) | `src/campaign/gates/sequential.ts` |
+| `heldoutSignificance`, `pairHoldout` | `src/campaign/gates/statistical-heldout.ts` |
+| `paretoSignificanceGate`, `buildEvidenceVector` | `src/campaign/gates/promotion-policy.ts` |
+| `pairArms`, `comparePairedArms`, `pairRunRecords` | `src/paired-arms.ts` |
+| `canonicalize`, `hashJson`, `signManifest`, `verifyManifest`, `HypothesisManifest` | `src/pre-registration.ts` |
+| `ExperimentTracker` (run ledger with KEEP/ITERATE/NOISE/REGRESSION) | `src/experiment-tracker.ts` |
+
+`HypothesisManifest` stays as the lightweight single-metric registration; `sealExperiment` is the full-design registration.
+The trace-repair admission machinery (`buildDenominatorChain`, oracle determinism, control policy) keeps its repair vocabulary in `./trace-repair`; this module is the general form new experiments should register against.
+
+## Where this sits
+
+This is Wave 2 of the [charter](./charter.md): the experiment subpath, built after the kill test that re-derived the three preregistrations as decision-rule objects (verdict: extended — ten node families beyond the seed AST, no opaque node, no rule dropped).
+Wave 3 wires these objects to the live-sandbox seam; the improvement receipt (Wave 4) serializes a sealed experiment's digest, gates, and refusal outcomes into one attested file.

package/docs/prime-analyst.md

@@ -5,6 +5,7 @@ It is the third scored arm of `agent-eval analyst-benchmark`, beside the recursi
 It speaks the CodeTraceBench failure-block contract only; `--analyst prime` with `--dataset agentrx` is rejected.
 
 Implementation: `src/analyst/benchmark-runner-prime.ts` (`createPrimeBenchmarkRunner`) binds the CodeTraceBench block grammar to the shared protocol in `src/analyst/prime-protocol.ts` and `src/analyst/prime-bridge-transport.ts`.
+The arm is declared as `primeCodeTraceAnalystDefinition()` and the creator is a thin shell over it; see the Analyst Definitions section in [trace-analysis.md](./trace-analysis.md).
 Wiring: `--analyst prime` in `src/analyst/benchmark-command.ts`.
 
 ## What the runner does

package/docs/trace-analysis.md

@@ -215,6 +215,32 @@ Unknown, transformed, or fabricated identifiers are rejected.
 An empty findings array means no submitted claim passed the evidence rules.
 It does not prove the run was correct.
 
+## Analyst Definitions
+
+`AnalystDefinition` (`src/analyst/definition.ts`, exported from `@tangle-network/agent-eval/analyst`) is the declarative unit behind an analyst arm.
+One definition value declares everything the arm can say to a model: the question, the task text, the `ReplyContract` row grammar, the `EvidenceProjection` (`inline` | `chunked` | `repl-variable` | `agent-tools`), a profile fragment (pinned model and reasoning-effort hints), the budget, and the repair-turn count.
+`bindAnalyst(definition, transports)` compiles a definition plus a transport binding (`prime-bridge` or `model-owner`) into a runnable `AnalystBenchmarkRunner`.
+
+The three benchmark arms are expressed this way:
+
+| Arm | Definition builder | Projection | Repair turns |
+|---|---|---|---|
+| `direct` | `publicDirectAnalystDefinition(dataset, args)` | `chunked` (descending per-attribute byte caps) | 0 |
+| `dspy-rlm` | `publicRlmAnalystDefinition(dataset, args)` | `repl-variable` (store bound as an engine REPL variable) | 1 (engine-internal typed repair) |
+| `prime` | `primeCodeTraceAnalystDefinition(args)` | `inline` (serialized JSON with a capped refetch) | 0 or 1 |
+
+`createPublicBenchmarkDirectRunner`, `createPublicBenchmarkRlmRunner`, and `createPrimeBenchmarkRunner` are thin shells over those builders, so no consumer changes.
+The parity suite (`src/analyst/definition-parity.test.ts`) runs each compiled definition and its entry point over the same fixture rows with a fake transport and asserts byte-identical request bodies plus equal protocol digests.
+A mismatch fails CI: expression loss between the declarative layer and the executing arm is caught by construction.
+Do not loosen those assertions; report the construct that cannot be expressed and extend the definition slots instead.
+
+`analystDefinitionProtocolSha256(definition)` digests the definition's protocol content; for an inline definition it equals the digest the prime arm records (`primeAnalystProtocolSha256()`).
+`analystDefinitionAsymmetries(definitions)` compares arms on equal terms: it refuses a set whose definitions declare unequal repair turns (a retry is a second sample) and renders the declared differences — projection mode, reasoning effort, budget — beside each arm.
+A definition a strategy cannot compile fails loud with `AnalystExpressivenessError` naming the construct.
+
+`AnalystContext.probe` (`ExecutionProbe`) is the optional live-execution port: a runtime that owns a sandbox or checkout fills it so an analyst can run a bounded command against the run's produced state and read a typed outcome.
+This package defines only the port; an absent probe means the analyst works from recorded evidence.
+
 ## Measure Analyst Quality
 
 Measure the analyst on labeled traces before using its findings for automated changes.

package/docs/trace-repair-admission.md

@@ -0,0 +1,194 @@
+# TB-Repair admission
+
+Admission decides which corpus rows enter a campaign.
+It runs before any analyst reads a row, and it publishes the denominator it produced.
+
+`Delta-repair = P(tests pass | intervention) - P(tests pass | no-fix control)` is an average over a set of rows.
+An analyst that declines the rows it cannot solve shrinks that set and raises its own score.
+So the set is fixed first, by checks no analyst can influence, and every row that leaves is counted with the reason it left.
+
+Source: [`src/trace-repair/`](../src/trace-repair/).
+The policy the controls run under is in [trace-repair-continuation.md](./trace-repair-continuation.md).
+
+## The five conditions
+
+A row is admitted only when all five hold.
+
+| # | condition | what it removes |
+| --- | --- | --- |
+| 0 | the task's own suite returns one verdict on byte-identical state | tasks whose grader is not a function of the state, where every condition below is a draw |
+| 1 | the recorded prefix replays with divergence at or below 10% | rows whose recording does not reproduce, so no claim about step `k` can be executed |
+| 2 | the task's held-out tests fail on the recorded end state | rows that did not actually fail |
+| 3 | the no-fix control fails every rollout | rows the continuation policy rescues with no intervention at all |
+| 4 | the no-op control fails every rollout | rows an inert action plus continuation rescues, which is a flaky task or a lucky policy |
+
+Condition 0 runs first and costs nothing at run time: it reads the task's certification, which was measured once by [`certify-task-oracle.sh`](../benchmarks/trace-repair/tools/README.md) and checked in as [`task-oracles.json`](../benchmarks/trace-repair/task-oracles.json).
+Conditions 1 to 4 all read the same suite, so a suite that answers differently about identical bytes makes each of them a coin flip rather than a measurement.
+A task with no certification is excluded as `task-oracle-uncertified`, which is a different fact from `task-oracle-nondeterministic`: one says the check has not run, the other says it ran and the task failed it.
+
+Conditions 3 and 4 are the ones that keep `Delta-repair` honest.
+Without them a row that any continuation would have passed counts as a repair, and the number measures the continuation policy rather than the analyst.
+
+## The control has to be able to rescue
+
+Conditions 3 and 4 ask whether a row is rescued by continuing from the recorded end state.
+That question has an answer only under a control that can act.
+
+A control rollout changes the graded state by executing commands, and it executes only what a model call asks for.
+At a step budget of zero it executes nothing, so the container it grades holds the same bytes condition 2 already graded as failing.
+A control pass under such a policy is not a rescue — it is the task's own grader answering differently about one state.
+Condition 3 then cannot fire for the reason it exists, and every row walks through it.
+
+So the control is a declared, hashed parameter rather than a default, and the criteria say which reading applies:
+
+| `controlScreening` | requires | a control pass means |
+| --- | --- | --- |
+| `enforced` (default) | `stepBudget >= 1` | `no-fix-control-passed` — the row is repairable by continuing alone |
+| `declared-inert` | `stepBudget == 0` | `control-passed-on-identical-state` — the task's grader disagreed with itself |
+
+Either pairing the other way round raises `UncalibratedControlError` at the call, before a verdict is produced for any row.
+A screening control that cannot act, and a control declared inert that can, are both configuration faults rather than properties of a row.
+
+`defineControlPolicy` hashes the whole declaration — id, step budget, scaffold, model, command timeout — so a step budget cannot move under an unchanged label, and a policy that calls no model must record `model: null`.
+
+Every admission decision carries an `AdmissionScreeningRecord`: the control policy and its digest, the screening mode, the task name, and the task's measured oracle flip rate.
+It is on the rejected decisions too, so a reader of an artifact can tell which control screened a row without opening the runner's source.
+`AdmissionRowVerdict` carries the same three fields for rows the executing pre-pass excluded before any control ran.
+
+Divergence is `divergences / prefixExecuted`, and the threshold admits a row sitting exactly on it.
+A replay that executed fewer steps than the recording holds is excluded as `prefix-replay-truncated` before that ratio is read, because a truncated run computes divergence over the steps it did reach and a short replay would look perfect.
+
+## The population split
+
+The corpus assay measured the final recorded return code of every admitted row.
+
+| stratum | final return code | assay share | can one command repair it |
+| --- | --- | --- | --- |
+| `clean-exit` | `0` | 87.13% | yes — the agent stopped believing it was done and the tests disagree |
+| `command-error` | `> 0` | 12.65% | yes |
+| `signal-kill` | `< 0` | 0.22% | no — the command was killed at a timeout |
+
+Rows are stratified before anything samples them, and every row carries its stratum, admitted or not.
+`admitStrata` defaults to `clean-exit` and `command-error`; the excluded signal kills appear in the chain as `stratum-not-admitted` rather than disappearing.
+
+`AdmissionReport.strata` holds admitted ids per stratum and there is no pooled list.
+Sampling reads one stratum at a time; pooling an addressable population with an unaddressable one takes an explicit concatenation that a reader can see.
+
+## The denominator chain
+
+A benchmark whose denominator is not auditable is not a benchmark.
+Every run emits the funnel as JSON and renders it as a table: the reason, the rows that reached that stage, the rows it removed, and the rows that survived.
+
+| stage | exclusion reason | entering | excluded | remaining |
+| --- | --- | --- | --- | --- |
+| 1 | `no-recorded-commands` | 5 | 1 | 4 |
+| 2 | `unparseable-final-returncode` | 4 | 0 | 4 |
+| 3 | `stratum-not-admitted` | 4 | 1 | 3 |
+| 4 | `task-oracle-uncertified` | 3 | 0 | 3 |
+| 5 | `task-oracle-nondeterministic` | 3 | 0 | 3 |
+| 6 | `prefix-replay-error` | 3 | 0 | 3 |
+| 7 | `prefix-replay-empty` | 3 | 0 | 3 |
+| 8 | `prefix-replay-truncated` | 3 | 0 | 3 |
+| 9 | `prefix-divergence-above-threshold` | 3 | 1 | 2 |
+| … | … | … | … | … |
+
+`assertChainReconciles` runs on every build of the artifact and throws unless `input = admitted + sum(excluded)`, the last stage's `remaining` equals `admitted`, and the per-stratum inputs plus the unstratified rows cover the overall input.
+
+The stage order is also the order of the checks, and it is cost-ordered.
+Everything decidable from the recording runs before the first container, and the six control rollouts run last.
+A stratum a campaign does not admit costs nothing at all.
+
+## A boundary failure is never a verdict
+
+Four exclusion reasons exist only because an external call failed: `prefix-replay-error`, `end-state-oracle-error`, `no-fix-control-error`, and `no-op-control-error`.
+
+An errored control rollout is not counted as a failed one.
+Counting it as a failure would admit a row nobody verified, which is the same corruption as a silent zero.
+The row leaves with the error reason and the message from the boundary, and it stays visible in the chain.
+
+## Analyst independence, in code
+
+Two guards, both mechanical.
+
+`assertAnalystIndependent(rows)` rejects any row carrying a key outside the closed list `rowId, taskName, recordedModel, recordedCommands, finalReturncode`.
+The check is a closed key list rather than a list of known analyst field names, because the failure to catch is "some new analyst output leaked into the gate".
+
+`assertDenominatorIntact({ report, strata, sampled, scored })` rejects the three ways a denominator moves after the fact:
+
+- a sampled row that was never admitted,
+- a scored row that was never sampled,
+- a sampled row that was never scored.
+
+The third is the one an analyst can cause alone.
+A row it declines still needs an outcome — `no-decisive-failure` is an answer — so a missing outcome raises `denominator shrank by N row(s)` instead of quietly reducing `n`.
+
+## What this changes about numbers already produced
+
+Runs made before the control was declared and the task oracles were certified are not rewritten here.
+What they measured is stated instead, so a reader can price them.
+
+Two facts are measured, not inferred.
+
+`largest-eigenval` is graded in part by a wall-clock assertion (`assert dt < ref_dt`, `tests/test_outputs.py:111`), the only one across the four tasks the milestones sampled.
+Certification re-graded each task's published image 16 times with nothing written between the runs — 8 on the untouched image and 8 after the reference solution, half of each under CPU contention — at the same image digests the milestones ran against.
+
+| task | replicates | units that flipped | worst per-unit flip | verdict |
+| --- | --- | --- | --- | --- |
+| `password-recovery` | 16 | 0 | 0 % | `CERTIFIED` |
+| `sanitize-git-repo` | 16 | 0 | 0 % | `CERTIFIED` |
+| `count-dataset-tokens` | 16 | 0 | 0 % | `CERTIFIED` |
+| `largest-eigenval` | 16 | 8 | 37.5 % | `NONDETERMINISTIC_ORACLE` |
+
+All eight flipped units are `test_speedup[size]` parameters on the untouched image, where `/app/eigen.py` holds the reference implementation the assertion compares against.
+`test_speedup[10]`, `[3]` and `[6]` each returned 5 passes and 3 fails over 8 gradings of identical bytes.
+The whole-suite reward never moved on that state — 0 of 8 — which is why the conjunction has to be counted term by term to see any of this.
+An earlier run of the same certification put the worst unit at 50 %; a flip rate that is itself unstable is what a coin flip looks like.
+
+No row from `largest-eigenval` can be admitted while that certification stands.
+
+Both milestone runs screened their controls with a policy pinned to zero model calls, so conditions 3 and 4 could never fire for the reason they exist.
+Every control pass those runs recorded fell on `largest-eigenval`: 3 of 3 across 68 row-evaluations, and 0 of 61 on the three tasks whose graders certify stable.
+One row, `largest-eigenval__4GTN8MQ`, was excluded by milestone 1 on a control pass of 1/3 and admitted by milestone 2 on a control pass of 0/3 — the same row and the same control, with opposite verdicts.
+
+The consequence differs by run.
+
+| run | rows | rows from the timing-graded task | what moves |
+| --- | --- | --- | --- |
+| milestone 1 | 20 evaluated, 17 admitted | 2 evaluated, **0 admitted** | the two exclusions were labelled `no-fix-control-passed`, which claimed the row was repairable by continuing alone; nothing continued. The headline is computed on 17 rows that contain none of them |
+| milestone 2 | 48 evaluated, 43 admitted | 5 evaluated, **4 admitted** | four admitted rows come from a task whose grader is not a function of the state; its admitted set is contaminated and its numbers are conditional on that |
+
+Milestone 1's headline separation survives excluding the timing-graded task, because it never included it: oracle-fix `+0.353` over 17 rows and inert-probe `0.000` over the same 17 rows are unchanged when `largest-eigenval` is dropped.
+What does not survive is the reason recorded for its two exclusions, and any reading of that run as evidence that the no-fix control screened anything.
+
+## Running it
+
+```ts
+import {
+  admissionArtifact,
+  definePinnedContinuationPolicy,
+  renderAdmissionReport,
+  runAdmission,
+} from '@tangle-network/agent-eval/../src/trace-repair'
+
+const policy = definePinnedContinuationPolicy({ model: 'pinned/model-id', seed: 20260808 })
+
+const report = await runAdmission({
+  rows, // corpus rows, recording fields only
+  policy,
+  replayer, // replays the prefix in the task's pinned image
+  oracle, // runs the task's held-out tests on the recorded end state
+  controls, // restores the state, applies the arm, continues, then grades
+  taskOracles: parseTaskOracleRegistry(JSON.parse(readFileSync(taskOraclesPath, 'utf8'))),
+  config: { concurrency: 8 },
+})
+
+const artifact = admissionArtifact(report)
+writeFileSync('admission.json', JSON.stringify(artifact, null, 2))
+writeFileSync('ADMISSION.md', renderAdmissionReport(artifact, { rowLimit: 50 }))
+```
+
+The three boundaries are injected, and each carries an `id` that lands in the artifact.
+A report produced against fakes names those fakes, so it cannot be read as one produced against real containers.
+
+The no-op injection step is drawn from the policy seed, the row id, and the rollout index, so it is reproducible from the artifact.
+Every rollout the controls return is checked against the arm, row, and index that was requested, and both arms are checked for a shared policy digest and paired seeds before a row is admitted.

package/docs/trace-repair-analyst-arms.md

@@ -0,0 +1,121 @@
+# The repair analyst arms
+
+The [repair grader](./trace-repair-grader.md) executes an answer.
+This page is about who produces the answer, and about what has to be equal between them before a difference between two arms means anything.
+
+An **arm** is one way of executing the repair question: a single chat completion, an agent harness, a DSPy program.
+The question, the reply grammar, the action budget and the bounded repair turn belong to the comparison and not to the arm.
+
+## Nothing is certified on this task
+
+No arm here runs a certified prompt, and each one says so in its own declaration.
+
+The one GEPA-certified analyst artifact this repository holds — `oht2-coverage-instructions.txt` — was earned on the CodeTraceBench incorrect-step task.
+That contract asks for blocks of recorded steps that were wrong, carrying `first_step`, `last_step`, `consequence_step` and `escape_status`.
+The repair contract asks a different question: one step `k`, and one executable action of the same type and budget the recorded scaffold itself took, which must make the task's held-out suite pass.
+The certified text cannot answer that, and re-authoring it for the repair contract would void the certification, because the benchmark it was earned on has been retired.
+
+So every arm is authored fresh for the repair task, and every arm is uncertified.
+`repairArmAsymmetries` refuses a set where some arms carry a certification and others do not: an optimisation applies to every arm or to none.
+
+## What is equal by construction
+
+| Property | Where it is enforced |
+|---|---|
+| One question, one task policy | `repair-prompt.ts` holds them; each arm composes its prompt from those shared constants and declares its own reply grammar as `promptContract` |
+| One action budget (4096 bytes, one top-level statement, one heredoc) | `askRepairArm` measures it; `gradeRepairRow` refuses a violation |
+| One bounded repair turn on a malformed reply | declared per arm as `repairTurns`; `repairArmAsymmetries` refuses a set that disagrees |
+| No arm sees a grading field | `RepairArmRequest` carries only a `BlindedTrajectoryPrefix`; the admitted row is not in the type, so a grading field is unreachable rather than merely unread |
+| A row an arm could not answer is a typed failure | `RepairArmReply` has no shape for a silent null |
+| A k the recording does not hold is a declined reply | both arms drop the row with its reason; the same model mistake lands in the same funnel cell on every execution path |
+
+The budget is **recorded** at answer time and **enforced** at grade time.
+One authority refuses; measuring early is what lets a report say what an arm spent its bytes on without paying for a rollout to find out.
+
+Prompt identity is recorded at two grains.
+`repairQuestionSha256` digests what every arm shares: the question, the task policy, the budget the caps are read from.
+Each answer additionally stamps `repairArmPromptSha256`, which folds in the arm's own declared contract text — so the chat arms, which ask the identical composed question, share one digest, and the DSPy arm, whose typed SUBMIT grammar is a materially different question, stamps another.
+A contract change, including a `DSPY_REPAIR_SIGNATURE` version bump, changes the per-arm digest.
+
+## What is allowed to differ, and is recorded
+
+`repairArmAsymmetries` renders the differences beside the result rather than leaving a reader to infer them from two runners' source.
+
+| Arm | Executes | Affordances |
+|---|---|---|
+| `bare-framing` | one chat completion, no harness | inline trajectory |
+| `prime` | the prime agent harness through a bridge | inline trajectory, agent loop |
+| `dspy-rlm` | a DSPy RLM program with a typed repair signature | inline trajectory, code interpreter, agent loop |
+
+The DSPy arm reads the trajectory with code inside its own environment; the completion arms read it as prompt text.
+That is a real advantage and it is declared, not hidden.
+
+## The DSPy arm
+
+`createDspyRepairArm` binds the DSPy RLM engine to the repair contract.
+The engine is the incumbent one, unchanged. What is new is a typed signature authored for this task:
+
+```
+question, analyst_instructions, trajectory, taskStatement  →  answer, repairs: list[RepairProposal]
+```
+
+`RepairProposal` is a pydantic model in `dspy_rlm_bridge.py`: `k`, `failure_claim`, `intervention_kind`, `action`.
+Its action cap mirrors the scaffold budget the grader enforces, so the typed field cannot refuse an action the grader would have accepted.
+The instructions name the task token `tb-repair-typed-`, which is how the bridge selects this signature; the same mechanism selects the CodeTraceBench signature.
+
+The trajectory arrives as `taskInputs` on the engine request and is bound as a variable in the program's environment.
+It is not fetched from a trace store: a Terminal-Bench trajectory is not in one, and a bridge that silently answered without it would be answering about material the caller never delivered — so a repair analysis with no `taskInputs.trajectory` stops the run.
+
+Typed rows return under `runtime.repair`, not under `findings`.
+The engine-neutral finding schema caps `recommended_action` at 2000 characters while the scaffold action budget is 4096 bytes, so routing a repair through it would hand this arm a smaller action than every other arm gets — an affordance asymmetry hidden inside a schema.
+The measured comparison run makes that concrete: chat-completion answers ran to 2705, 3413, 3987 and 4733 bytes.
+
+## Omni is still blocked, for a different reason than before
+
+GEPA's Omni recipe was deferred on 2026-08-03 because no selection metric existed that a candidate could not game.
+The repair grader discharges that: the metric is whether the held-out suite passes after the proposed action executes, which no prose can satisfy.
+
+It stays blocked, on two grounds that the grader does not touch.
+
+**It is an optimiser, not an arm.**
+Running it produces certified instruction text for one arm.
+The comparison rule above — an optimisation applies to every arm or to none — is now enforced in code, so an Omni-tuned arm beside three untuned ones is a set `repairArmAsymmetries` refuses.
+
+**There is no leakage-free split to train on.**
+The corpus records 48 rows from exactly four Terminal-Bench-2 tasks, and admission passes 43: `sanitize-git-repo` 16, `count-dataset-tokens` 12, `password-recovery` 11, `largest-eigenval` 4.
+The 20 pre-registered measurement rows draw from all four.
+The binding constraint on the existing result is the four task clusters, not the 20 rows, so any training split shares clusters with the measurement set at exactly the level that binds.
+
+The usable set is smaller still.
+`largest-eigenval` grades speedup with a wall-clock assertion and its suite is certified nondeterministic — 8 of its 27 assertion units flip on byte-identical state, worst-unit flip rate 0.375 — so the oracle-determinism gate refuses every one of its rows, and 16 of the 20 pre-registered rows survive, in three clusters.
+All five `password-recovery` rows among those 16 score zero on the oracle-fix ceiling arm because the reference solution is bash and the scaffold runs dash, which makes their ceiling unmeasured rather than zero.
+That leaves 11 measurement rows in two clusters with a measured ceiling.
+
+Omni unblocks when the corpus carries enough independent task clusters to hold out a measurement set that shares none with the training split — not when the grader improves.
+
+## Wiring
+
+```ts
+import {
+  askRepairArm,
+  createCompletionRepairArm,
+  createDspyRepairArm,
+  repairArmAsymmetries,
+  repairArmResponse,
+  gradeRepairRow,
+} from '@tangle-network/agent-eval/trace-repair'
+
+const arms = [bareFraming, prime, dspy]
+// Refuses unequal budgets, unequal repair turns, or a partly certified set.
+const asymmetries = repairArmAsymmetries(arms)
+
+for (const arm of arms) {
+  const answer = await askRepairArm({ arm, row })
+  const response = repairArmResponse(answer)
+  if (response === null) continue // the arm failed; the row still counts in the denominator
+  const result = await gradeRepairRow({ row, response, sessions, oracle, continuation })
+}
+```
+
+Publish `asymmetries` with the result.
+A comparison that reports a difference without reporting what still differs between the arms is not reporting the difference it measured.

package/docs/trace-repair-continuation.md

@@ -0,0 +1,107 @@
+# TB-Repair continuation policy
+
+The continuation policy answers one question: after an analyst names a failing step `k` and proposes a single action, what happens if the agent keeps working from there?
+
+`Delta-repair = P(tests pass | intervention) - P(tests pass | no-fix control)` is the headline number of TB-Repair.
+It only measures the intervention when all three arms run forward under the same policy.
+This module makes that symmetry structural rather than promised.
+
+Source: [`src/trace-repair/`](../src/trace-repair/).
+The pre-pass that decides which rows the arms run on is in [trace-repair-admission.md](./trace-repair-admission.md).
+
+## What runs
+
+| element | value | why it is fixed |
+| --- | --- | --- |
+| scaffold | mini-swe-agent | The corpus recorded it, so a continuation stays in the same distribution as the prefix. |
+| step budget | 20 model calls | Bounds a rollout without a wall-clock limit, which would end rollouts at different points. |
+| temperature | 0 | With a fixed seed, the same prefix draws the same continuation. |
+| command timeout | 30 s | The recorded runs used the scaffold's own 30-second limit. A longer limit lets the continuation finish commands the recorded agent could not. |
+| network | `none` | A container with a network can install what the recorded run could not. |
+| format-error cap | 3 consecutive turns | Ends a rollout that has stopped producing actions. |
+| model and seed | chosen per campaign | No default stands in for either. |
+
+`definePinnedContinuationPolicy({ model, seed })` freezes a policy and validates it.
+`continuationPolicyDigest(policy)` hashes the policy together with the scaffold text it renders, so an edited template produces a different digest and cannot be pooled with earlier rollouts.
+
+The step budget is what lets this policy serve as a control at all.
+A rollout changes the graded state by executing commands, and it executes only what a model call asks for, so at a budget of zero a control arm grades the bytes the end-state check already graded as failing.
+`runAdmission` checks that with `assertControlCalibrated` before it opens a container; the reading of a control pass is in [trace-repair-admission.md](./trace-repair-admission.md).
+
+## Why the arms cannot drift apart
+
+Three arms share one code path, and the arm is a label on the record:
+
+- **intervention** — the analyst's action substituted at step `k`.
+- **no-fix control** — step `k` replayed unchanged.
+- **no-op control** — step `k` replaced by an action that changes nothing.
+
+The arms differ only in the container state handed to the runner, which is the treatment under test.
+Three guards keep the policy itself identical:
+
+1. `continuationSeed(policySeed, rowId, rolloutIndex)` cannot read the arm, so paired rollouts across arms draw the same seed.
+2. Every rollout records the `policyDigest` it ran under.
+3. `assertArmSymmetry(rollouts)` rejects a set whose rollouts disagree on that digest or on a paired seed.
+
+## What a rollout records
+
+Every rollout carries the assistant message, the parsed action, the observation, the return code, the timeout flag, per-call latency, the served model id, token usage, and cost.
+
+Two record rules keep a partial measurement from reading as a complete one:
+
+- `usage.callsWithUsage` below `usage.calls` clears `usage.captured`. Calls that reported nothing are never filled with zeros.
+- One unpriced call makes the whole rollout `{ kind: 'uncaptured', usd: null }`. Summing the priced calls would report less than what was spent.
+
+`rolloutRecordedSteps(rollout)` projects the continuation into the `{ src, msg, tools, obs }` steps the trajectory corpus stores, so the replay layer reads a continuation with the reader it already has.
+`rolloutDigest(rollout)` hashes the deterministic content — actions, observations, seeds, exit status, usage — and excludes wall-clock fields, which vary between identical runs.
+
+## Exit statuses
+
+| status | meaning |
+| --- | --- |
+| `submitted` | A command echoed the sentinel and exited 0. The submission text is recorded. |
+| `step-budget-exhausted` | The rollout used all 20 calls without submitting. |
+| `repeated-format-error` | Three consecutive turns held no single bash block. |
+| `model-error` | The provider call failed. The rollout is recorded with `terminalError`, not dropped. |
+| `environment-error` | A container call failed. The step keeps the action that hit it. |
+
+## Running one
+
+```ts
+import {
+  createDockerContinuationEnvironment,
+  definePinnedContinuationPolicy,
+  nodeProcessRunner,
+  runContinuation,
+} from '@tangle-network/agent-eval/../src/trace-repair'
+
+const policy = definePinnedContinuationPolicy({ model: 'pinned/model-id', seed: 20260808 })
+
+const rollouts = await runContinuation({
+  policy,
+  arm: 'intervention',
+  rowId: 'break-filter-js-from-html:7',
+  prefix, // messages through step k, rebuilt by the replay layer
+  rollouts: 3,
+  model,
+  environments: {
+    id: 'docker',
+    async create({ arm, rolloutIndex }) {
+      return createDockerContinuationEnvironment({
+        containerRef: await restoreState({ arm, rolloutIndex }),
+        cwd: '/app',
+        runProcess: nodeProcessRunner,
+        removeOnDispose: true,
+      })
+    },
+  },
+})
+```
+
+The runner calls `describe()` before the first model call and refuses any container whose network mode is not `none`.
+The environment reports the mode the daemon holds, not the mode the caller asked for.
+
+The prefix must start with a system message then a user message, and must end on a user message.
+A prefix ending on an assistant turn means the replay left an action unanswered, and the runner rejects it.
+
+Each rollout gets its own environment, because a rollout mutates the container it runs in.