@tangle-network/agent-eval 0.145.13 → 0.145.15

package/docs/campaign-proposers.md

@@ -223,6 +223,10 @@ uv sync --frozen --group gepa-release
 uv sync --frozen --group gepa-source
 ```
 
+The published package supports the standard `gepa` engine.
+The composed recipes below — `sequential`, `adaptive-sequential`, `best-of`, `vote`, and `omni` — need the tested official source revision.
+Move that revision only after both the release and the source compatibility tests pass.
+
 ## Configure GEPA
 
 `gepaOptimizationMethod()` accepts text surfaces and component surfaces.

package/docs/concepts.md

@@ -13,12 +13,38 @@ Use the lower-level functions when you need direct control over execution, stora
 | Function | When to call it | What you give it | What you get back |
 |---|---|---|---|
 | **`defineAgentEval()`** | You have scenarios, an agent, a judge, and a baseline surface, and you want one object you can score or improve. | scenarios, agent, judge, baseline surface | `{ evaluate(), improve() }` where `evaluate()` returns a campaign result and `improve()` returns a report |
-| **`selfImprove()`** | You want candidate generation, scoring, and a release decision in one call. | scenarios, agent, judge, baseline surface | report, ship/hold decision, winner surface |
+| **`selfImprove()`** | You want candidate generation, scoring, and a release decision in one call. | scenarios, agent, judge, baseline surface | report, winner surface, and a `gateDecision` (see below) |
 | **`loadEvalFixtureScenarios()`** | You want agents to add evals as folders with `PROMPT.md`, checks, and starter files. | `evals/<name>/PROMPT.md + EVAL.ts + package.json` | `Scenario[]` that runs through `runCampaign`; pair with `planEvalFixtureRun()` before spending tokens |
 | **`analyzeRuns()`** | You have existing runs and do not need to invoke an agent. | `RunRecord[]` and options | `InsightReport` |
 | **Intake adapters** (`fromFeedbackTable`, `fromOtelSpans`) | Your data isn't already in `RunRecord` shape: it's in Obsidian, Sheets, an OTel collector, etc. | source-specific input | `RunRecord[]` ready to pipe into `analyzeRuns()` |
+| **`sealExperiment()` / `openSealedExperiment()`** | The result must convince a reader who does not trust you, so the rules must be fixed before the data arrives. | arms, admission funnel, estimand, interval, decision table | a hashed rule tree plus executors that can run no other rule ([`experiment.md`](./experiment.md)) |
+| **`runEquivalenceCheck()`** | The work has no held-out test suite, so no answer key exists to grade against. | a claim, two blind arms, an injected checker | a certification naming who vouched and how it can fail ([`verification-strategies.md`](./verification-strategies.md)) |
+| **`AnalystRegistry.runExact()`** | A batch of runs failed and you need cited findings, with the caller owning every execution choice. | recorded evidence, a declared analyst list | findings with evidence references, an execution plan, and a receipt ([`trace-analysis.md`](./trace-analysis.md)) |
 
 See [`customer-journeys.md`](./customer-journeys.md) for runnable paths from existing logs, human ratings, and a callable agent.
+The [README front-door table](../README.md#which-front-door) lists every callable entry point with a runnable example.
+
+### The five release decisions
+
+`selfImprove()` and every gate return a `GateDecision`, not a two-way ship/hold flag.
+Folding the last three into `hold` throws away the action each one names.
+
+| Decision | What it means | What to do next |
+|---|---|---|
+| `ship` | Every gate passed on sufficient evidence. | Release the candidate. |
+| `hold` | A gate failed on sufficient evidence. | Reject this candidate. |
+| `need_more_work` | A gate could not decide: the evidence was missing, or the paired sample was too small to claim significance. | Gather more runs, then gate again. |
+| `model_ceiling` | Reserved for a caller-supplied gate that attributes the limit to the model. | Handle it; no gate in this package emits it. |
+| `arch_ceiling` | Reserved for a caller-supplied gate that attributes the limit to the architecture. | Handle it; no gate in this package emits it. |
+
+The last two are part of the taxonomy and of the composition order, but no built-in gate returns them today.
+Handle all five anyway: a caller's own gate may return either, and the type will not let you ignore them.
+
+`need_more_work` is not a quiet `hold`.
+"Gather more evidence" and "reject this candidate" are different actions, and folding the first into the second abandons a real gain that was only underpowered.
+
+When gates are composed, `ship` requires every gate to ship.
+Otherwise the strongest hold wins, in this order: `arch_ceiling`, `model_ceiling`, `hold`, `need_more_work`.
 
 `analyzeRuns()` and the high-level contract return the same `InsightReport` shape.
 It contains score distributions, paired lift intervals, judge agreement, cost, failure clusters, contamination checks, outcome correlation, and recommendations.
@@ -92,6 +118,48 @@ that can seed memory, replay scenarios, and optimization.
 | **Composite score** | A 0..1 number combining all dimensions. The single number you gate on. |
 | **Rubric version** | A stable hash of the rubric. Scores from different rubric versions are not comparable. |
 
+### Running an evaluation
+
+| Term | Plain English |
+|---|---|
+| **Case** (`Scenario`) | One task the agent must do. The unit every score is per. |
+| **Surface** | The value being changed: a prompt, a skill, or a serialized configuration. |
+| **Dispatch** | The function that runs your agent on one case and returns the artifact. |
+| **Campaign** | One complete pass of every case, executed, scored, and cached under a run directory. |
+| **Cell** | One (case × replicate) of a campaign. Cells are cached, so a rerun skips the ones that finished. |
+| **Receipt** | The record of what one paid call actually cost, in dollars and tokens. Absent when nothing measured it. |
+| **Cost ledger** | The spend account receipts are written to. A capped ledger refuses a call that would exceed the cap. |
+| **Provenance** | Where a number came from: the package version, the source revision, the run identity, the exact attempt. |
+| **`RunRecord`** | The analysis-time projection of one run: who ran, on what, with which seed, at what cost, and what it scored. |
+
+### Improving a surface
+
+| Term | Plain English |
+|---|---|
+| **Optimizer** | Any procedure that writes candidate surfaces and picks one. |
+| **GEPA** | An open-source optimizer that mutates text using reflection over failures. It searches; this package executes and scores. |
+| **SkillOpt** | Microsoft's skill optimizer. Same division of labour. |
+| **Engine** | One named search procedure inside GEPA. |
+| **Recipe** | How several engines are composed: in order, adaptively, best-of, or by vote. |
+| **Train cases** | Evidence the optimizer reads to write candidates. |
+| **Selection cases** | Evidence the optimizer reads to choose among its candidates. |
+| **Final cases** | Held back from the optimizer entirely. They produce the reported lift. |
+
+The three-way split is the reason a reported lift means anything.
+An optimizer that saw the final cases can score well on them without the agent getting better.
+
+### Proving a result
+
+| Term | Plain English |
+|---|---|
+| **Experiment** | The rules — arms, funnel, estimand, interval, decision — written as data before the data arrives. |
+| **Seal** | A hash of that whole rule tree. The execution surface accepts no rule outside it. |
+| **Estimand** | The exact quantity being measured, for example the paired difference in pass rate. |
+| **Funnel** | The denominator chain: how many rows entered, what each stage removed, and how many remain. |
+| **Verification strategy** | One of ten ways to certify a result, each with a documented way it can certify a wrong one. |
+| **Certification** | Who vouched for a verdict, with what checker version, and what the checker did not check. |
+| **Analyst** | A function that reads recorded evidence and returns findings that cite it. |
+
 ## The feedback trajectory loop
 
 Normal review activity can provide labels without a separate labeling interface:
@@ -154,19 +222,21 @@ When you have a multi-step pipeline (install → typecheck → build → lint
 
 ```ts
 const verifier = new MultiLayerVerifier([
-  installLayer,      // runs `pnpm install`
-  typecheckLayer,    // runs `tsc --noEmit`, depends on install
-  buildLayer,        // runs `pnpm build`, depends on typecheck
-  semanticLayer,     // LLM judge, weight 3, depends on build
+  installLayer, // runs `pnpm install`
+  typecheckLayer, // runs `tsc --noEmit`, depends on install
+  buildLayer, // runs `pnpm build`, depends on typecheck
+  semanticLayer, // LLM judge, weight 3, depends on build
 ])
 
-const report = await verifier.run({ env: { runner, workdir, ... } })
-report.allPass        // boolean: every layer passed
-report.taskScore      // complete task score, or undefined
-report.blendedScore   // diagnostic weighted aggregate, possibly partial
-report.layers         // per-layer status, findings, duration
+const report = await verifier.run({ env })
+report.allPass // boolean: every layer passed
+report.taskScore // complete task score, or undefined
+report.blendedScore // diagnostic weighted aggregate, possibly partial
+report.layers // per-layer status, findings, duration
 ```
 
+`env` carries the sandbox driver, the working directory, and the harness commands each layer runs.
+
 Use `taskScore` when creating task labels or training data.
 An errored, timed-out, skipped, or incomplete scoring panel leaves `taskScore` undefined.
 Use `blendedScore` only to inspect the measurements that did complete.
@@ -184,9 +254,31 @@ Two questions to answer before trusting any LLM judge:
 1. **Does it agree with humans?** `calibrateJudge(golden, candidate)` reports Pearson, MAE, integer-rounded κ, and worst-N miscalibrations vs a human golden set.
 2. **Does it agree with itself / other judges?** `continuousAgreement(scores)` and `calibrateJudgeContinuous(golden, candidate)` report κ_w + ICC(2,1) + Pearson + Spearman with bootstrap 95% CIs on the raw [0,1] scores.
 
-Why two κ flavours: the original `calibrateJudge` rounds scores to ints before computing κ. For fine-grained judges that loses information: 0.78 vs 0.81 both round to "1" and look perfectly agreed. Use `calibrateJudgeContinuous` (or `continuousAgreement` for N≥2 raters) when scores are continuous. ICC(2,1) catches systematic bias that Pearson misses: if judge B scores 2× judge A, Pearson stays ≈ 1 while ICC drops: that's the signal.
+Each statistic answers a different question:
+
+| Statistic | What it answers | What it misses |
+|---|---|---|
+| Pearson | Do the two raters move together? | Constant offset and constant scaling |
+| Spearman | Do they rank the same way? | The size of any gap |
+| MAE (mean absolute error) | How far apart are they, on average? | Whether the gap is systematic |
+| κ (Cohen's kappa) | Do they agree more than chance? | Everything below the rounding step |
+| ICC(2,1) | Do they agree in absolute value, not just in shape? | — |
+
+Use two flavours of κ for one reason.
+`calibrateJudge` rounds each score to an integer first.
+For a fine-grained judge that throws information away: 0.78 and 0.81 both round to 1 and look perfectly agreed.
+Use `calibrateJudgeContinuous`, or `continuousAgreement` for two or more raters, when the scores are continuous.
+
+ICC(2,1) catches a bias Pearson cannot see.
+If judge B always scores twice judge A, the two move together perfectly and Pearson stays near 1, while ICC drops.
+That drop is the signal.
+
+Every reported interval is a bootstrap 95 % interval: the statistic is recomputed on many resamples of the data, and the middle 95 % of those values is the interval.
 
-Bias probes (`positionalBias`, `verbosityBias`, `selfPreference`) cover the orthogonal failure modes: position-dependent scoring, length-correlated scoring, and judge-prefers-its-own-family.
+Three bias probes cover three separate failure modes.
+`positionalBias` finds a judge that scores by position.
+`verbosityBias` finds one that scores by length.
+`selfPreference` finds one that prefers output from its own model family.
 
 ## Trace Model
 
@@ -222,3 +314,8 @@ release decision.
 - **Building a code-generator eval?** → Start with `BuilderSession`, `SandboxHarness`, and `MultiLayerVerifier`.
 - **Multi-layer verifier?** → Use [control-runtime.md](./control-runtime.md) and `MultiLayerVerifier` for ordered gates with dependencies.
 - **Adding a new judge or rubric?** → `src/wire/rubrics.ts` for the cross-language path; `src/anti-slop.ts` and `src/judges.ts` for the in-process path.
+- **Registering an experiment before the data arrives?** Read [experiment.md](./experiment.md) for the rule AST, the seal, the funnel, and the refusals.
+- **Certifying a result with no answer key?** Read [verification-strategies.md](./verification-strategies.md) for the ten-member family and the blind two-arm protocol.
+- **Reading a verdict someone else produced?** Read [verdicts.md](./verdicts.md) for what `certification` carries and what an absent one means.
+- **Grading a finding by executing its repair?** Read [trace-repair-grader.md](./trace-repair-grader.md), and [trajectory-replay.md](./trajectory-replay.md) for re-executing a recorded failure.
+- **Wondering why this package exists at all?** Read [charter.md](./charter.md) for the four end-states it is built against.

package/docs/feature-guide.md

@@ -34,7 +34,7 @@ Evaluation measures whether the result met its requirements, whether another att
 | “I need train/dev/test/holdout examples.” | `Dataset` plus feedback trajectory conversion | Stable splits and contamination control. |
 | “Which optimization procedure wins?” | `compareOptimizationMethods` | Runs complete methods on shared train and selection cases, then compares them on separate final cases. |
 | “Improve a multi-turn agent with candidates from my runtime.” | `runImprovementLoop` | Evaluates caller-generated candidates and applies a separate release rule. |
-| “Improve prompts, then code if prompts plateau.” | `runPromptEvolution`, composite mutator, code mutator | Bounded evolution with telemetry and lineage. |
+| “Improve prompts, then code if prompts plateau.” | `gepaOptimizationMethod` or `externalTextOptimizationMethod` for the prompt; agent-runtime's worktree path for the code | Text search stays here; executable code changes belong to the runtime. |
 | “Find why a regression happened.” | bisector, traces, run records | Narrows changes and preserves evidence. |
 | “Expose evals to another language.” | Wire protocol and Python client | HTTP/RPC boundary for non-TypeScript apps. |
 

package/docs/trace-analysis.md

@@ -241,6 +241,32 @@ A definition a strategy cannot compile fails loud with `AnalystExpressivenessErr
 `AnalystContext.probe` (`ExecutionProbe`) is the optional live-execution port: a runtime that owns a sandbox or checkout fills it so an analyst can run a bounded command against the run's produced state and read a typed outcome.
 This package defines only the port; an absent probe means the analyst works from recorded evidence.
 
+## Exact Runs
+
+Call `AnalystRegistry.runExact()` when the caller, and not the registry, must own every execution choice.
+The runnable minimum is [`examples/custom-trace-analyst`](../examples/custom-trace-analyst/).
+
+The `analystIds` array is the execution order, and exact runs are serial.
+A caller that needs recursive or concurrent scheduling composes exact runs through its own runtime rather than adding a second scheduler here.
+Every option must be present, and `null` disables a channel on purpose, so a missing budget can never be read as an unlimited one.
+
+An analyst reaches an exact run only when it declares `executionConfig`: canonical JSON for every behavior knob that `version` does not already bind.
+The receipt stores a digest of it, so two runs that behaved differently cannot look identical.
+`defineCustomAnalyst()` returns an exact-capable analyst when that field is present, and the built-in analysts already declare it.
+
+Every other live component admitted to an exact run — the cost ledger, a registry hook, a registry chat client — carries an `ExactExecutionComponentIdentity`: a non-secret `id`, a `version`, and a canonical `config` object.
+`snapshotExactExecutionComponentIdentity()` reduces each one to an `ExactExecutionComponentSnapshot`, which keeps `id` and `version` and replaces `config` with `config_digest`.
+That is how a receipt names what ran without ever storing a credential.
+
+Three rules govern what a finished exact run may claim.
+
+1. Lifecycle hooks receive frozen snapshots. A hook observes the planned context; it cannot rewrite it.
+2. Persisted results store configuration digests, never raw configuration. The plan records the exact equal or weighted allocation for every routed analyst, and archival validates each summary against that same plan.
+3. Every receipt says whether it is `complete` or `failed`. A complete receipt must cover the whole plan. A failed receipt may cover only the prefix that ran.
+
+Any failure after an exact run starts rejects with `ExactAnalystRunExecutionError`.
+Its immutable failed receipt keeps the summaries, findings, usage, and cost that were already valid, so a late failure does not erase what was measured before it.
+
 ## Measure Analyst Quality
 
 Measure the analyst on labeled traces before using its findings for automated changes.

package/docs/trace-repair-continuation.md

@@ -15,7 +15,7 @@ The pre-pass that decides which rows the arms run on is in [trace-repair-admissi
 | --- | --- | --- |
 | scaffold | mini-swe-agent | The corpus recorded it, so a continuation stays in the same distribution as the prefix. |
 | step budget | 20 model calls | Bounds a rollout without a wall-clock limit, which would end rollouts at different points. |
-| temperature | 0 | With a fixed seed, the same prefix draws the same continuation. |
+| temperature | 0 | Removes the sampler as a source of variation the policy controls. It does not make a continuation repeat: measured against the z.ai seat, 19 of 20 replies to one identical prompt were distinct on `glm-5.3` and 8 of 20 on `glm-4.7`. A paired design must carry that variation as a threat to validity — see [trace-repair-gated-stop.md](./trace-repair-gated-stop.md). |
 | command timeout | 30 s | The recorded runs used the scaffold's own 30-second limit. A longer limit lets the continuation finish commands the recorded agent could not. |
 | network | `none` | A container with a network can install what the recorded run could not. |
 | format-error cap | 3 consecutive turns | Ends a rollout that has stopped producing actions. |

package/docs/trace-repair-free-lunch.md

@@ -0,0 +1,119 @@
+# What unconditional continuation rescues
+
+TB-Repair's admission condition 3 asks whether a row is rescued by continuing from the recorded end state with no intervention.
+Both milestone runs answered it under a control pinned to **zero model calls**.
+A rollout that makes no model call executes no command, so the container the control graded held the same bytes the end-state check had already graded as failing: on the 32 rows measured here that control returned **0 passes in 96 rollouts**, the only answer it could return.
+
+This is the same question asked with a budget.
+
+Source: [`scripts/tb-repair-freelunch.ts`](../scripts/tb-repair-freelunch.ts).
+The control contract is in [trace-repair-admission.md](./trace-repair-admission.md); the policy is in [trace-repair-continuation.md](./trace-repair-continuation.md).
+
+## The answer
+
+**3 of 64 rollouts, 4.7 %.** Two of 32 rows were rescued at least once, 6.2 %.
+
+The 64 rollouts are not 64 independent draws.
+A seed-derivation defect (threat 8) made the second pass repeat the first pass's seed, and 14 of the 32 second-pass rollouts are byte-identical action repeats of their first-pass rollout.
+Over the 50 distinct rollouts the rate is **3 of 50, 6.0 %**.
+All three rescues come from pairs whose two rollouts differ, so no rescue is a repeat counted twice.
+
+| interval, 95 % | rescue rate |
+| --- | --- |
+| task-clustered bootstrap (3 clusters, 10 000 resamples, seed 7) | 0.0 % – 10.0 % |
+| row-clustered bootstrap (32 clusters) | 0.0 % – 12.5 % |
+| exact Clopper-Pearson on rollouts | 1.0 % – 13.1 % |
+
+Three clusters cannot carry a stable interval; the row-clustered and exact intervals are reported beside it for that reason.
+
+**The rate is not zero, and it is not noise.** One row, `count-dataset-tokens__HL3ZzrX`, was rescued in **both** of its rollouts on a suite that returned the same verdict in all 16 certification replicates.
+
+## What a rescue looked like
+
+`count-dataset-tokens__HL3ZzrX`, 13 steps, submitted. The recorded agent had computed the right answer and written it in a form the grader rejected. The continuation recomputed the count offline, then found and removed the formatting defect:
+
+```
+10. cat /app/answer.txt
+11. printf '%s' "$(cat /app/answer.txt)" > /app/answer.txt && cat /app/answer.txt | xxd | head -5
+12. printf '%s' "$(cat /app/answer.txt)" > /app/answer.txt && od -c /app/answer.txt
+13. echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT
+```
+
+`password-recovery__oDL7kv9`, 19 steps, submitted, rescued in 1 of 2 rollouts. This one is not a formatting fix — it is the task being solved: the continuation searched the disk image, hexdumped it, carved an embedded archive, and wrote the recovered password.
+
+## By task, and the asymmetry that explains it
+
+| task | rows | rollouts | passes | rate | recorded steps that used the network |
+| --- | --- | --- | --- | --- | --- |
+| `count-dataset-tokens` | 10 | 20 | 2 | 10.0 % | 24.3 % |
+| `password-recovery` | 6 | 12 | 1 | 8.3 % | 0 % |
+| `sanitize-git-repo` | 16 | 32 | 0 | 0 % | 0 % |
+
+The pinned policy disables the network the recorded agents had. Measured on the continuations' own actions, `count-dataset-tokens` rollouts spent **30.2 % of their steps reaching for a network that was not there** (114 of 377), against 0.8 % for `sanitize-git-repo` and 0 % for `password-recovery`.
+`password-recovery` is the clean sub-population: its recordings never used the network, so its 8.3 % is unconfounded by the policy.
+
+## By exit status
+
+| exit status | rollouts | passes | rate |
+| --- | --- | --- | --- |
+| `submitted` | 15 | 3 | 20.0 % |
+| `step-budget-exhausted` | 46 | 0 | 0 % |
+| `model-error` | 3 | 0 | 0 % |
+
+Every rescue came from a rollout that decided it was finished. No rollout that burned all 20 steps ever passed.
+The three `model-error` rollouts were ended by provider 503s after four retries; they are recorded and graded, never dropped, so the rate is a lower bound by at most those three.
+
+## What it cost
+
+64 rollouts, priced from the token counts the provider reported for this run's own calls at the router's published `glm-5.2` rate.
+
+| quantity | min | median | p90 | max | sum |
+| --- | --- | --- | --- | --- | --- |
+| prompt tokens | 15 146 | 170 939 | 260 088 | 321 288 | 10 734 764 |
+| completion tokens | 151 | 6 663 | 14 961 | 23 866 | 546 346 |
+| continuation steps | 1 | 20 | 20 | 20 | 1 143 |
+| cost, USD | 0.0138 | 0.1506 | 0.2281 | 0.2821 | **9.4806** |
+
+**$0.1481 per rollout, $0.2963 per row at n = 2.** A paired study can budget-match against those two numbers directly.
+
+Cost is attributed per call, not from the router's account counter: `GET /v1/credits` covers the whole key, and 18 other processes on this host were calling it during the run. The counter's delta over the two passes was $8.60, which is neither this run's cost nor an upper bound on it once other traffic is in both directions.
+
+## What the number means
+
+A **high** rate would have meant unconditional continuation captures most of the available headroom, killing the gated-stop thesis. It did not.
+
+A **low but non-zero** rate means the headroom exists and a gate could claim it — which licenses a paired study without proving it. That is where this lands, with two qualifications that matter more than the point estimate:
+
+- The rescues are **not free**. Each cost $0.148 and up to 20 model calls. A gate that fires on every failed row pays that on every row.
+- **Condition 3 is now calibrated.** It has a real screen rate to compare against: 4.7 % of rollouts and 6.2 % of rows, against the 0 % a zero-step control was structurally obliged to report.
+
+## Threats
+
+1. **Reconstructed assistant messages.** The corpus stores each recorded assistant turn as an elided placeholder, so the continuation inherits the bash block without the reasoning that produced it.
+2. **Network asymmetry.** The recorded agents had internet; the pinned policy does not. On `count-dataset-tokens` that consumed 30.2 % of continuation steps, so the overall rate is a lower bound for a networked continuation.
+3. **Three clusters.** A task-clustered interval over three tasks is coarse by construction.
+4. **One model, `glm-5.2` at temperature 0.** Not a statement about continuation in general.
+5. **n = 2.** Within-row variance is measured on two draws. One row rescued twice, one rescued once of two.
+6. **Wall-time is not clean.** For part of the first pass the measurement seat was not held, because a killed sibling wrapper's exit trap removed a lock it no longer owned. Verdicts and token counts are unaffected; latency and throughput are not clean.
+7. **Snapshot boundary.** State moves between container generations as a committed image, so a process the recording left running does not survive.
+8. **The two passes shared one seed.** Each pass called `runContinuation` with `rollouts: 1`, and the seed derived from that call's internal index, which is always 0. Shifting `ROLLOUT_BASE` therefore never reached the seed, and both passes sent the provider the index-0 seed on an identical prompt. Measured on the records: 14 of 32 row pairs are byte-identical action sequences, and 31 of 32 share their first action. The runner now forwards the pass's base index through `rolloutBase`, so a future pass draws its own seed.
+
+## Reproducing
+
+```bash
+# containers only, no model calls, no seat needed
+npx tsx scripts/tb-repair-freelunch.ts --stop-points-only
+
+# one uniform pass over every row, under the measurement seat
+TBR_FL_ROLLOUTS=1 TBR_FL_ROLLOUT_BASE=0 TBR_FL_OUT=freelunch-pass1.json \
+  npx tsx scripts/tb-repair-freelunch.ts
+```
+
+`--plan` prints the denominator chain and the selected rows without opening a container.
+The pre-registration, its amendments, and the raw per-rollout records are in `~/bench-cache/freelunch-20260810/`.
+
+## The raw records stay local
+
+`freelunch.json` holds every continuation's actions and observations, which are container state. GitHub push protection refused an earlier commit of it because a container carried a **Hugging Face user access token** in its cached credentials, at `free-lunch-n2.json:1883`.
+
+Raw per-rollout records are therefore kept out of the repository. What is committed is the runner, this report, and the numbers derived from the records. Anyone re-running the campaign should treat the artifact directory as credential-bearing.

package/docs/trace-repair-gated-stop.md

@@ -0,0 +1,114 @@
+# Gated stop against blind continuation
+
+The study asks one question.
+At one matched total token budget, does an agent that may stop only after an executable held-out check passes finish more rows than the same agent spending the identical budget on unconditional continuation?
+
+The mechanism under test is budget allocation, not detection.
+A row that clears the check early returns its unspent budget to a pool.
+The pool pays for extra steps on rows that have not cleared.
+Detection quality is not the claim: on this corpus the recorded done-signal fires on 51.69 % of failed runs and 31.14 % of successes, which is 62.5 % precision as a success predictor.
+
+Design and runner: [`benchmarks/trace-repair/gated-stop-ab/design.json`](../benchmarks/trace-repair/gated-stop-ab/design.json) and [`scripts/tb-gated-stop-ab.ts`](../scripts/tb-gated-stop-ab.ts).
+The registration primitives are described in [experiment.md](./experiment.md).
+
+## The two arms
+
+| arm | role | stop rule | graded at |
+| --- | --- | --- | --- |
+| `blind-continue` | control | spend the allotment unconditionally | best intermediate state |
+| `gated-continue` | treatment | stop when the held-out check passes | best intermediate state |
+
+Both arms replay the same recorded prefix, run the same scaffold, and are graded by the same injected suite after every step.
+The check that gates the treatment arm is the check that scores both arms.
+
+That symmetry forces a control the report must carry.
+The gate is the outcome, so the treatment arm cannot lose a success it already reached, while the control arm can regress out of one.
+The control arm is therefore graded twice, at its final state and at its best intermediate state, and both contrasts are reported.
+When the two disagree, the harsher contrast is the headline.
+
+## Choosing the draw
+
+The admitted set is the ceiling, not the draw.
+`settlingDraw` returns the smallest draw that clears the registered power target at the settling effect of 0.10.
+Spending more rows than the design needs is as undisciplined as spending fewer.
+
+The search runs at 15 000 trials and the registered gate runs at 3 000.
+Near the floor the two estimates straddle the target, because the standard error at 3 000 trials is about 0.007 and adjacent draws differ by less.
+A draw must clear the target under both before the search accepts it.
+
+## The identity gate runs before the spend
+
+`confirm` evaluates the registered `servedModel` gate before it grades a row.
+The gate compares the pinned model id against the id the seat reports.
+Its registered action on failure is `abort`.
+
+A contrast measured on a substituted model belongs to an experiment nobody registered.
+The gate therefore ends the run at zero spend and writes `confirm-refusal.json`, which records the pinned id, the served id, the rows graded, and the dollars spent.
+A refusal is a verdict object, not prose beside one.
+
+## Measured seat behaviour
+
+These facts were measured against the z.ai coding seat and they bound what the study can claim.
+
+| fact | measurement | n |
+| --- | --- | --- |
+| `glm-5.2`, `glm-5.1` and `glm-5` are all answered by `glm-5.3` | served id differs from requested id | 3 ids |
+| `glm-4.7` and `glm-4.6` are answered by themselves | served id equals requested id | 2 ids |
+| `glm-5.3` is not deterministic at temperature 0 | 19 of 20 replies distinct | 20 |
+| `glm-4.7` is not deterministic at temperature 0 | 8 of 20 replies distinct | 20 |
+| the scaffold's first turn draws more than one action block | 6 of 20 replies, both models | 20 per model |
+| the same rate inside a run, behind a replayed prefix | 4 of 24 steps | 24 |
+
+Temperature 0 does not give a repeated continuation on this provider.
+A paired design that assumes it must treat run-to-run variation as a threat to validity and report it.
+
+## Label quality on `qemu-startup`
+
+One row, `qemu-startup__PnXK6EH::ord0`, is recorded at reward 0 and passes its own held-out suite on the replayed end state.
+The end-state screen measures exactly that condition, so the registered funnel excludes the row at the `recorded-end-state-fails-its-own-suite` stage.
+The row is absent from the draw.
+
+The disagreement rate is 1 of 18 screened rows on `qemu-startup`, against 0 of 285 on every other task.
+The concentration is the finding, not the single row.
+`qemu-startup` rows still enter the study, because the screen tests each row against the condition that would disqualify it and the remaining rows pass that test.
+A task whose labels disagree with its own oracle at 5.6 % cannot carry a study on its own, and this design does not ask it to: it contributes 8 of 151 rows inside a task-clustered bootstrap that resamples whole tasks.
+
+## Running the confirmatory arm
+
+```bash
+node --import tsx scripts/tb-gated-stop-ab.ts design    # re-seal; prints both digests
+node --import tsx scripts/tb-gated-stop-ab.ts confirm   # identity gate, then both arms
+```
+
+`design` writes to the work directory. Copy the result over `benchmarks/trace-repair/gated-stop-ab/design.json` to move the checked-in seal forward; the next `design` reads that file to report the digest it replaces.
+
+`confirm` runs the control arm at the seat's concurrency limit and the treatment arm one row at a time.
+The treatment arm is serial because its pool is sequential state: a row draws the budget that earlier rows returned, so processing order decides allocation.
+Running those rows concurrently would change the allocation the seal registered, which makes the scheduling part of the experiment rather than a detail of it.
+
+The control arm therefore finishes in about a third of the wall time of the treatment arm on the same draw.
+
+Every finished row is written to `confirm-runs.json` before the next row starts.
+A re-invocation reads that file, skips rows already held, and rebuilds the pool from the entitlement and spend of each held treatment row.
+An interrupted run costs the row in flight and nothing before it.
+
+## Reading the result
+
+`confirm-report.json` carries the digest that produced it, both contrasts, the matched-budget verdict and the served model ids observed across every step.
+
+The matched-budget rule is a refusal object, not an assertion.
+The treatment arm can only spend returned budget on rows that come after the row that returned it, so budget freed by the last rows has nowhere to go.
+When that trailing shortfall pushes the arms more than 5 % apart, the registered decision table returns `contrast-refused-unmatched-budget` and the contrast is not read.
+
+## The measured result
+
+The confirmatory run completed both arms over the sealed draw: 302 rows graded, 151 per arm.
+
+The corrected primary contrast, best intermediate state in both arms, is **+0.0596** with a 95 % cluster-bootstrap interval of **[-0.0061, +0.1210]**.
+The interval includes zero, so the registered decision table reads `no-effect-resolved-at-this-n`.
+The study does not certify a gated-stop advantage at this draw.
+
+Two degradations bound what this run can claim.
+
+- 15 gated-arm rows were degraded by provider 429 rate-limit responses, and the treatment arm carries all of them; the contrast above is the corrected value after the tail audit accounted for them.
+- The registered estimand pipeline as first shipped crashed at report time, because the evidence rows carried a boolean where the paired-mean-diff estimand requires a number; the contrast above was recomputed from the persisted `confirm-runs.json` after the defect was fixed. The runner now writes the verdict as 1 or 0.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tangle-network/agent-eval",
-  "version": "0.145.13",
+  "version": "0.145.15",
   "description": "Evaluate and improve AI agents from runs, traces, judges, and feedback. Compare candidates, cluster failures, measure lift, and gate releases.",
   "homepage": "https://github.com/tangle-network/agent-eval#readme",
   "repository": {
@@ -180,8 +180,8 @@
   "dependencies": {
     "@asteasolutions/zod-to-openapi": "^9.1.0",
     "@hono/node-server": "^2.0.12",
-    "@tangle-network/agent-core": "0.8.0",
-    "@tangle-network/agent-interface": "0.52.0",
+    "@tangle-network/agent-core": "0.9.0",
+    "@tangle-network/agent-interface": "0.53.0",
     "@tangle-network/agent-trace-contract": "^1.0.2",
     "hono": "^4.12.32",
     "linear-sum-assignment": "1.0.9",