@tangle-network/agent-eval 0.144.6 → 0.144.7

package/docs/trace-repair-continuation.md

@@ -0,0 +1,107 @@
+# TB-Repair continuation policy
+
+The continuation policy answers one question: after an analyst names a failing step `k` and proposes a single action, what happens if the agent keeps working from there?
+
+`Delta-repair = P(tests pass | intervention) - P(tests pass | no-fix control)` is the headline number of TB-Repair.
+It only measures the intervention when all three arms run forward under the same policy.
+This module makes that symmetry structural rather than promised.
+
+Source: [`src/trace-repair/`](../src/trace-repair/).
+The pre-pass that decides which rows the arms run on is in [trace-repair-admission.md](./trace-repair-admission.md).
+
+## What runs
+
+| element | value | why it is fixed |
+| --- | --- | --- |
+| scaffold | mini-swe-agent | The corpus recorded it, so a continuation stays in the same distribution as the prefix. |
+| step budget | 20 model calls | Bounds a rollout without a wall-clock limit, which would end rollouts at different points. |
+| temperature | 0 | With a fixed seed, the same prefix draws the same continuation. |
+| command timeout | 30 s | The recorded runs used the scaffold's own 30-second limit. A longer limit lets the continuation finish commands the recorded agent could not. |
+| network | `none` | A container with a network can install what the recorded run could not. |
+| format-error cap | 3 consecutive turns | Ends a rollout that has stopped producing actions. |
+| model and seed | chosen per campaign | No default stands in for either. |
+
+`definePinnedContinuationPolicy({ model, seed })` freezes a policy and validates it.
+`continuationPolicyDigest(policy)` hashes the policy together with the scaffold text it renders, so an edited template produces a different digest and cannot be pooled with earlier rollouts.
+
+The step budget is what lets this policy serve as a control at all.
+A rollout changes the graded state by executing commands, and it executes only what a model call asks for, so at a budget of zero a control arm grades the bytes the end-state check already graded as failing.
+`runAdmission` checks that with `assertControlCalibrated` before it opens a container; the reading of a control pass is in [trace-repair-admission.md](./trace-repair-admission.md).
+
+## Why the arms cannot drift apart
+
+Three arms share one code path, and the arm is a label on the record:
+
+- **intervention** — the analyst's action substituted at step `k`.
+- **no-fix control** — step `k` replayed unchanged.
+- **no-op control** — step `k` replaced by an action that changes nothing.
+
+The arms differ only in the container state handed to the runner, which is the treatment under test.
+Three guards keep the policy itself identical:
+
+1. `continuationSeed(policySeed, rowId, rolloutIndex)` cannot read the arm, so paired rollouts across arms draw the same seed.
+2. Every rollout records the `policyDigest` it ran under.
+3. `assertArmSymmetry(rollouts)` rejects a set whose rollouts disagree on that digest or on a paired seed.
+
+## What a rollout records
+
+Every rollout carries the assistant message, the parsed action, the observation, the return code, the timeout flag, per-call latency, the served model id, token usage, and cost.
+
+Two record rules keep a partial measurement from reading as a complete one:
+
+- `usage.callsWithUsage` below `usage.calls` clears `usage.captured`. Calls that reported nothing are never filled with zeros.
+- One unpriced call makes the whole rollout `{ kind: 'uncaptured', usd: null }`. Summing the priced calls would report less than what was spent.
+
+`rolloutRecordedSteps(rollout)` projects the continuation into the `{ src, msg, tools, obs }` steps the trajectory corpus stores, so the replay layer reads a continuation with the reader it already has.
+`rolloutDigest(rollout)` hashes the deterministic content — actions, observations, seeds, exit status, usage — and excludes wall-clock fields, which vary between identical runs.
+
+## Exit statuses
+
+| status | meaning |
+| --- | --- |
+| `submitted` | A command echoed the sentinel and exited 0. The submission text is recorded. |
+| `step-budget-exhausted` | The rollout used all 20 calls without submitting. |
+| `repeated-format-error` | Three consecutive turns held no single bash block. |
+| `model-error` | The provider call failed. The rollout is recorded with `terminalError`, not dropped. |
+| `environment-error` | A container call failed. The step keeps the action that hit it. |
+
+## Running one
+
+```ts
+import {
+  createDockerContinuationEnvironment,
+  definePinnedContinuationPolicy,
+  nodeProcessRunner,
+  runContinuation,
+} from '@tangle-network/agent-eval/../src/trace-repair'
+
+const policy = definePinnedContinuationPolicy({ model: 'pinned/model-id', seed: 20260808 })
+
+const rollouts = await runContinuation({
+  policy,
+  arm: 'intervention',
+  rowId: 'break-filter-js-from-html:7',
+  prefix, // messages through step k, rebuilt by the replay layer
+  rollouts: 3,
+  model,
+  environments: {
+    id: 'docker',
+    async create({ arm, rolloutIndex }) {
+      return createDockerContinuationEnvironment({
+        containerRef: await restoreState({ arm, rolloutIndex }),
+        cwd: '/app',
+        runProcess: nodeProcessRunner,
+        removeOnDispose: true,
+      })
+    },
+  },
+})
+```
+
+The runner calls `describe()` before the first model call and refuses any container whose network mode is not `none`.
+The environment reports the mode the daemon holds, not the mode the caller asked for.
+
+The prefix must start with a system message then a user message, and must end on a user message.
+A prefix ending on an assistant turn means the replay left an action unanswered, and the runner rejects it.
+
+Each rollout gets its own environment, because a rollout mutates the container it runs in.

package/docs/trace-repair-grader.md

@@ -0,0 +1,163 @@
+# The repair grader
+
+`@tangle-network/agent-eval/trace-repair` grades one claim about a recorded failure by executing the repair it proposes.
+
+An analyst reads a blinded trajectory prefix and answers with exactly one finding — a step `k`, what went wrong there, and one action to run instead — or with the literal `no-decisive-failure`.
+The grader turns that answer into a measured difference.
+
+[The analyst arms](./trace-repair-analyst-arms.md) page covers who produces the answer, and what has to be equal between two analysts before their difference means anything.
+
+## The headline
+
+```
+Delta-repair = P(tests pass | intervention) − P(tests pass | no-fix control)
+```
+
+Paired per row, then bootstrapped over rows.
+Every admitted row stays in the denominator, including the rows an analyst declined and the answers the grader rejected.
+Those rows contribute a paired difference of exactly zero, because with no intervention to run their arm is their control arm.
+
+## The five tiers
+
+| Tier | Question | Pays |
+|---|---|---|
+| t0 parsed | Is this one answer, inside the action budget? | — |
+| t1 reproduced | Does the recorded state at `k` come back under replay? | **nothing** |
+| t2 executes | Does the intervention run at `k`? | `executes` |
+| t3 local flip | Does the held-out suite pass immediately after it? | `localFlip` |
+| t4 repair flip | Does the suite pass after the pinned policy continues? | `repairRate` |
+
+t0, t1 and t2 are nested gates.
+t3 and t4 are two measurements of the same intervention, and neither contains the other: an intervention that fixes the task outright flips locally, and one that unblocks an agent with work still to do flips only after the continuation.
+
+**t1 pays nothing, structurally.**
+`RepairCredit` has three numeric terms and no reproduction term, so there is no field a reproduced step could pay into.
+Naming the first step with a nonzero exit code earns exactly what naming a step that does not reproduce earns.
+
+Reproduction is not "a nonzero exit came back".
+Most admitted rows end on a clean exit, so the gate asks whether the recorded state at `k` reproduces, whatever that state was, including a returncode of zero.
+A step that recorded no observation has nothing to reproduce; the gate passes vacuously, opens no container, and records `basis: 'no-recorded-observation'`.
+
+## The action budget
+
+The intervention is one action from the action space the scaffold had.
+
+| Rule | Default | Why |
+|---|---|---|
+| One top-level statement | `maxStatements: 1` | A command list joined by `&&`, `\|\|` or a pipe is one statement. Two statements separated by a newline or `;` are two actions. |
+| One authored file | `maxHeredocs: 1` | An `edit` writes file content inline through one heredoc. |
+| 4 KB | `maxBytes: 4096` | The scaffold's per-action budget. |
+
+Heredoc bodies, comments and compound blocks (`if`, `for`, `while`, `until`, `case`, `{ … }`) are inside a statement, never separators.
+An action the scanner cannot resolve stays inside one oversized statement and is rejected on bytes, so an unparseable answer fails closed.
+
+The declared kind must match the action: an answer that calls a file rewrite a shell command is rejected rather than silently corrected.
+
+## Admission
+
+Five checks, none of which reads a finding, a `k`, or a label.
+Four are anchored at the recorded end state, so the same evidence admits a row whatever an analyst later blames; the fifth is about the task rather than the row.
+
+| Check | Criterion |
+|---|---|
+| oracle determinism | the task's own suite returned one verdict per assertion on byte-identical state |
+| prefix fidelity | at most 10 % of replayed steps diverge from their recorded returncode |
+| end state fails | the held-out suite fails on the recorded end state |
+| no-fix control | 3 of 3 continuations from the end state fail |
+| no-op control | 3 of 3 continuations from the end state, after an action that changes nothing, fail |
+
+The two controls only screen under a control that can act.
+A control with a step budget of zero executes no command, so it grades the bytes the end-state check already read as failing, and a pass there is the task's grader disagreeing with itself.
+`admitRow` refuses that pairing at the call rather than admitting every row through a check that cannot fire; see [trace-repair-admission.md](./trace-repair-admission.md).
+
+`admitRow` is pure: a campaign runner executes the checks against real containers and hands it the measured evidence.
+Splitting it that way keeps every admission re-derivable from the recorded numbers without re-running a container.
+
+An admitted row carries a phantom brand that only `admitRow` can produce.
+`blindTrajectory` and `gradeRepairRow` both require one, so there is no signature anywhere that shows an unadmitted row to an analyst or grades an answer about one.
+
+## What defeats the degenerate answers
+
+`DEGENERATE_STRATEGIES` names each one in code, with the mechanism that removes its reward and where that mechanism lives.
+A gate refuses the answer before a rollout is spent; a measurement lets it run and measures it at its control.
+
+| Strategy | Defeated by |
+|---|---|
+| point at any nonzero-exit step | gate — reproduction pays nothing |
+| propose the recorded command again | gate — compared against the recorded action at `k` |
+| propose a no-op | gate for the literal ones, measurement for a semantic one |
+| submit instead of repair | gate — the sentinel is rejected |
+| touch the test suite | gate — the oracle injects the suite from outside and verifies the bytes it reads back |
+| buy a bigger action | gate — statements, heredocs and bytes |
+| decline every hard row | measurement — a declined row keeps its cell and its place in the denominator |
+| repair somewhere other than `k` | measurement — the intervention runs at the named `k` only |
+
+The suite is the load-bearing one.
+`injectedTestOracle` purges the suite root, uploads the held-out suite from outside the session, reads the bytes back from inside and hashes them, and refuses to grade when the read-back digest is not the uploaded digest.
+A container that silently drops the upload raises `TestSuiteTamperedError` rather than returning a result.
+
+Ordering carries the same weight as the upload.
+A repair rollout replays the prefix, runs the intervention, hands the container to the continuation policy, and only then calls the oracle.
+The continuing agent therefore works in a container where the suite does not exist yet, and cannot read the thing that will grade it.
+
+Every arm is graded against the suite the row was admitted against.
+A grade produced from a different suite digest raises rather than being compared, because two arms that answered different questions are not a difference.
+
+## Wiring
+
+Three injected ports, one job each.
+
+```ts
+import {
+  admitRow,
+  blindTrajectory,
+  deltaRepair,
+  gradeRepairRow,
+  injectedTestOracle,
+  parseAnalystResponse,
+  renderDeltaRepairReport,
+} from '@tangle-network/agent-eval/trace-repair'
+
+const outcome = admitRow(evidence)
+if (!outcome.admitted) return
+
+const prompt = blindTrajectory(outcome.row)
+const answer = parseAnalystResponse(await analyst(prompt))
+if (!answer.succeeded) return
+
+const graded = await gradeRepairRow({
+  row: outcome.row,
+  response: answer.value,
+  sessions,       // a fresh container at the trajectory's own image
+  oracle,         // injectedTestOracle({ files, command, purge })
+  continuation,   // the pinned policy, run forward from a prepared session
+})
+
+const report = deltaRepair(rows)
+console.log(renderDeltaRepairReport(report))
+```
+
+The image must be the published one the trajectory was recorded against.
+A locally rebuilt image drifts from the recording through unpinned apt and pip installs, and every number measured on it is about a different environment.
+
+## Honest limits
+
+The arms are matched on policy, budget and suite, and not on position.
+The no-fix control continues from the recorded end state; the intervention arm continues from `k` and has to redo the work the recording did after `k` inside the same step budget.
+That asymmetry biases against the intervention, and it travels with the number as the `control-position-asymmetry` threat.
+
+Every admitted row has a control rate of zero by admission, so Delta-repair equals the intervention rate on an admitted corpus.
+The estimate is conditional on that admission and says nothing about rows the control can already repair.
+
+Under `controlScreening: 'declared-inert'` that zero is weaker still: the control made no model call, so a control rate of zero restates the end-state check instead of measuring what continuing alone can repair.
+Rows screened that way carry the `control-cannot-rescue` threat into the report, so the caveat travels with the number.
+
+Oracle determinism is certified per task, not per row, and it is certified at the two states certification can construct: the published image and that image after the reference solution ran.
+Those are anchors, and a suite can be steady at an anchor while flipping near its threshold.
+Per-assertion counting is what makes the anchor informative — a suite whose per-parameter timing assertions flip shows it there even when the whole-suite reward does not — but the certification is still a measurement at two states and not a proof about every state a campaign will grade.
+
+A command-level repair cannot address a run the harness killed at a timeout.
+Split that class out before sampling; the grader measures actions, not wall clock.
+
+The interval carries `gateEligible`, which is false below the pair count where a percentile bootstrap holds its nominal error rate.
+Below it the interval is descriptive spread and a promotion must not turn on it.

package/docs/trajectory-replay.md

@@ -0,0 +1,110 @@
+# Trajectory replay
+
+`@tangle-network/agent-eval/trajectory-replay` re-executes a recorded shell trajectory and produces a verdict about what really happened.
+
+The primitive is one question: replay steps `1..k-1` of a recording inside the image it ran in, execute step `k`, and does the recorded failure come back?
+That question needs a recording, an image, and a way to run a command.
+It does not need a running agent loop, so it is substrate, not runtime.
+
+## The one thing you must supply
+
+The package never depends on a sandbox client, a container runtime, or a model provider.
+Every entry point takes the execution boundary as an argument.
+
+```ts
+import {
+  type ReplayExecBackend,
+  replayVerify,
+} from '@tangle-network/agent-eval/trajectory-replay'
+
+const backend: ReplayExecBackend = {
+  async open() {
+    const box = await myPlatform.create({ image })
+    return {
+      async exec(command, timeoutMs) {
+        const r = await box.exec(command, { timeoutMs })
+        return { exitCode: r.exitCode, stdout: r.stdout, stderr: r.stderr }
+      },
+      async close() {
+        await box.delete()
+      },
+    }
+  },
+}
+
+const verdict = await replayVerify({
+  stepsPath: 'normalized/<traj>/steps.json',
+  image: 'example/img:tag',
+  at: 37,
+  cwd: '/repo',
+  out: 'out/proof',
+  backend,
+})
+```
+
+`open()` must return a FRESH environment every call.
+Each arm replays the prefix in its own session, so a reused session would prove the corrected step against debris from the previous arm.
+
+Entry points that resolve images themselves take a `ReplayExecBackendFactory` instead: `(image) => ReplayExecBackend`.
+Those are `runReplayBatch`, `replayVerifyFinding`, and `verifyFindings`.
+
+## Verdict shape
+
+`replayVerify` runs up to two arms and writes `replay-verdict.json` plus `report.md` into `out`.
+
+| Field | Meaning |
+|---|---|
+| `armA.failureSignatureMatch` | The recorded returncode came back, and the recorded error substring appeared. |
+| `armB.failureVanished` | A corrected command exited 0 and the error substring was gone. |
+| `prefixDivergences` | Prefix steps that did not confirm the recording, each with its `kind`. |
+| `prefixDivergencePct` | Divergent steps over executed steps. This is the number an admission pre-pass gates on. |
+| `prefixWithinTolerance` | `prefixDivergencePct` is at most `PREFIX_DIVERGENCE_TOLERANCE_PCT` (10). |
+| `signatureBasis` | `returncode+output-substring`, or `returncode-only` when the recording carries no error line. |
+
+## Prefix fidelity
+
+A verdict is only as good as the state the prefix rebuilt.
+Agreement therefore requires positive evidence: a prefix step counts as confirmed only when the recording carries a returncode AND the replayed exit equals it.
+
+Every other step is a divergence of one of two kinds.
+
+| Kind | Meaning |
+|---|---|
+| `returncode-mismatch` | The recording carries a returncode and the replayed exit differs. |
+| `unknown-expectation` | The recording carries no returncode, so the replay was never checked. |
+
+An `unknown-expectation` step is never agreement.
+Counting it as agreement lets a replay that fails on every step report a perfect prefix, which makes every verdict built on that prefix meaningless.
+
+`runReplayBatch` reports the same split per case and across the corpus, under `headline.prefixFidelity`.
+Its `replayed` predicate requires `prefixWithinTolerance`, so an unconfirmed prefix is never admitted.
+`verifyFindings` applies the same rule: a proof whose prefix fell outside the tolerance is `divergent`, whatever its arms did.
+
+Prefix divergence is reported, never hidden.
+A high divergence rate is a finding about replay fidelity, not a harness error.
+
+## Layers
+
+| Module | Role |
+|---|---|
+| `steps` | The recorded step and its `<returncode>` / `<output>` grammar. |
+| `exec` | The execution boundary and mini-SWE `/bin/sh` command wrapping. |
+| `verify` | One case: prefix replay, arm A, optional arm B, `ReplayVerdict`. |
+| `corpus` | Labeled corpora to replayable cases, with a reason for every exclusion. |
+| `image-preparer` | Derive a replay-ready image from a recorded one. |
+| `fix` / `fix-loop` | Generate and iterate arm-B corrections through an injected chat caller. |
+| `batch` | Every replayable case to replayability and fix-flip rates. |
+| `wire` / `findings` | One analyst finding to an executed, receipted proof. |
+
+## Honest limits
+
+Only trajectories that record their image are replayable.
+A task whose environment needs external compose peers cannot be replayed this way.
+
+A gold label on the submit step is never a replay target.
+A submit decision has no executable failure to reproduce, so those cases are excluded and counted.
+
+`dockerImagePreparer` shells out to `docker`.
+Pass `preparer: null` on a corpus source when the images are already replay-ready.
+
+Verified batch verdicts become RL rows through `src/rl/verified-findings-dataset.ts`; see [verified-labels-flywheel](./verified-labels-flywheel.md).

package/docs/verification-strategies.md

@@ -0,0 +1,103 @@
+# Verification strategies: certifying without an answer key
+
+This document covers the verification-strategy family (`src/verification-strategy.ts`), verdict epistemics (`src/verdict.ts`), and the blind statement-equivalence protocol (`src/equivalence-check.ts`).
+It is the Wave 5 surface of the charter (`docs/charter.md`): an unsolved problem has no held-out test suite by definition, so the held-out suite must be one member of a strategy family, not the family itself.
+
+The package ships the taxonomy, the record types, and the refusals.
+It ships no checker implementation.
+A checker is an injected executable boundary that returns a typed outcome, so a consumer binds its own Lean toolchain, invariant harness, replication runner, or judge.
+Instruments, never methods.
+
+## The family
+
+Every member has a documented failure mode: the specific way it can certify a wrong result.
+A consumer that reads a certification must weigh the member's failure mode, not treat "certified" as one bit.
+The registry `VERIFICATION_STRATEGIES` carries each profile at runtime, so a reader can surface the failure mode without this document at hand.
+
+| Member | Determinism | Certifies | Failure mode |
+| --- | --- | --- | --- |
+| `compile` | deterministic | typecheck / build / lint passed | code that compiles is not code that is correct |
+| `test` | deterministic | suite pass-rate against an answer key | certifies nothing outside suite coverage; a stubbed integration reports green |
+| `schema` | deterministic | structured output validates | shape is not meaning; a well-formed wrong answer passes |
+| `sandbox` | deterministic | sandbox execution exit code | one bit compresses the run; a faked success exits 0 |
+| `judge` | probabilistic | LLM judge score | drifts across model versions; Goodhart-gameable by the graded policy |
+| `composite` | inherited | weighted blend of members | scalar collapse hides which member carried the score |
+| `proof-kernel` | deterministic | a proof assistant's kernel accepted a formal proof | the formalization gap: the kernel never certifies that the formal statement matches the informal claim |
+| `invariant` | deterministic | invariant / metamorphic properties held | weak invariants pass everything; a set uncalibrated by seeded bugs is a rubber stamp |
+| `replication` | deterministic (given pins) | independent re-execution reproduced the result | re-runs the method, so it never catches an error the method itself carries |
+| `agreement` | probabilistic | independently-derived results agree | the shared blind spot: derivers with common corpora or priors agree for the same wrong reason |
+
+Three calibration rules follow from the failure modes.
+
+1. A `proof-kernel` certificate is incomplete until the statement-equivalence obligation below is discharged; the kernel proves theorems about statements, never about intentions.
+2. An `invariant` set earns weight only through seeded-bug calibration: demonstrate a mutation the set provably rejects before its pass carries any.
+3. An `agreement` certificate is only as strong as its blindness provenance; the two-arm protocol below is the shape that makes the provenance checkable.
+
+## The port shape
+
+Execution binds through one port, `StrategyChecker<Input, Result>`:
+
+- `strategy` — which family member the checker discharges obligations for;
+- `identity` — exact checker identity (`CheckerIdentity`): name, version, and content pins, e.g. `{ name: 'lean4', version: '4.33.0', pins: { mathlib: 'db584cd6d46c' } }`;
+- `determinism` — the class the checker claims for its outcomes;
+- `check(input)` — returns `CheckerOutcome<Result>`: `{ succeeded: true, value }` or `{ succeeded: false, error }` with the full error text.
+
+A checker that ran but could not decide returns `succeeded: false` with the reason.
+`succeeded: true` is reserved for a discharged check.
+
+## Verdict epistemics
+
+`DefaultVerdict` carries an optional `certification: VerdictCertification`:
+
+- `strategy` — the family member that certified the verdict;
+- `checker` — the exact `CheckerIdentity`;
+- `assumptions` — every step the certificate rests on that the checker did not verify; an empty array is an explicit, auditable claim of none;
+- `evidenceDigest` — digest of the evidence artifact.
+
+A kernel-checked verdict and a judge-scored verdict can carry the same `{ valid: true, score: 1 }`.
+A consumer that cares reads `certification.strategy` and tells them apart.
+A consumer that does not read the field sees the exact verdict it always saw; the field is additive and every pre-existing verdict shape remains valid.
+
+The reward-source union `VerifiableRewardSource` (`src/rl/verifiable-reward.ts`) is the same family, so an RL consumer and a certification reader mean the same thing by `proof-kernel`.
+
+## The statement-equivalence protocol
+
+The proof-kernel failure mode gets its own discharge protocol, `defineEquivalenceCheck` (`src/equivalence-check.ts`): the blind two-arm design as a typed primitive.
+
+Two arms derive the formal statement independently, blind to each other and to the outcome.
+A bound checker then discharges the obligation that the two statements are equivalent:
+
+- `proved` — the statements match; the formalization gap is closed for this claim;
+- `refuted-with-separating-witness` — the statements provably differ, with the witness in hand; **a mismatch is a successful outcome** — it is the formalization gap made visible;
+- `unresolved` — the obligation was not discharged; the record keeps the full reason.
+
+The refusals are the design, and each one throws `EquivalenceProtocolError` with a machine-readable `code`:
+
+- an arm that saw the other arm's statement (`arm-saw-other`) or the outcome (`arm-saw-outcome`) invalidates the check — nothing was independently derived;
+- a design with any arm count other than 2 (`arm-count`) or with `blind: false` (`not-blind`) is a different protocol, not a parameter choice;
+- a refutation without its separating witness (`witness-missing`), a proof carrying one (`witness-on-proved`), an unresolved obligation without its reason (`reason-missing`), and a verdict without its evidence digest (`evidence-missing`) are all refused;
+- a checker whose declared strategy differs from the spec's (`checker-strategy-mismatch`) is refused before it runs.
+
+Arm refusals fire before the checker executes: an invalid check must not spend.
+
+## Worked example: the BCWW (4.6) pilot
+
+The pilot this shape reproduces was assembled by hand for the refutation of the BCWW (4.6) inequality, before the substrate carried these types.
+Artifacts: `~/bench-cache/bcww-formalization/` (`REPORT.md` is the check-phase report; `PRIORITY.md` records the search-coverage limits).
+
+- Arm A (`armA/Statement.lean`) derived the Lean statement from the paper's LaTeX source (arXiv:1507.05650, display `eqn:cd2`).
+- Arm B (`armB/Statement.lean`) derived it from the campaign's artifacts: discovery-lab KB pages plus the standalone verifier source.
+- Both arms were blind to each other and committed before the counterexample was read.
+- The checker was the Lean kernel: toolchain `leanprover/lean4:v4.33.0`, Mathlib pin `db584cd6d46c` (`check/Check.lean`, no `sorry`).
+- Obligation: **proved** — `statement_equivalence` gives the iff at the level of the committed propositions, and the stronger `lhs46_entVec` shows the paper's linear form and the campaign's functional are equal as functions.
+- The five-atom counterexample was then kernel-checked against both statements and violates both, with the exact value `(9*log2(3) - 14)/5` bits.
+- The certification's assumption list is not empty: the diagonal embedding (classical distribution → quantum state) is argued in prose, not formalized.
+  That line is what an honest `VerdictCertification.assumptions` exists to carry.
+
+In this package's terms: `defineEquivalenceCheck({ source: 'proof-kernel', artifact: 'arXiv:1507.05650 inequality (4.6) + five-atom counterexample record', arms: 2, blind: true })`, two `EquivalenceArm` records with `blindness: { toOtherArms: true, toOutcome: true }`, and an `EquivalenceObligation` with `status: 'proved'`, the lean4 `CheckerIdentity`, and the evidence digest of the kernel run.
+
+## What this package refuses to own
+
+- No Lean (or any checker) implementation ships here; the port is the boundary and the consumer binds its own kernel.
+- No strategy selection: choosing proof-kernel over agreement for a task is a method decision, and methods live above the substrate (discovery's covenant).
+- No certification laundering: a record missing its checker identity, evidence digest, blindness provenance, witness, or reason does not get a weaker record — it gets a thrown `EquivalenceProtocolError`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tangle-network/agent-eval",
-  "version": "0.144.6",
+  "version": "0.144.7",
   "description": "Evaluate and improve AI agents from runs, traces, judges, and feedback. Compare candidates, cluster failures, measure lift, and gate releases.",
   "homepage": "https://github.com/tangle-network/agent-eval#readme",
   "repository": {
@@ -134,6 +134,21 @@
       "import": "./dist/hosted/index.js",
       "default": "./dist/hosted/index.js"
     },
+    "./trace-repair": {
+      "types": "./dist/trace-repair/index.d.ts",
+      "import": "./dist/trace-repair/index.js",
+      "default": "./dist/trace-repair/index.js"
+    },
+    "./experiment": {
+      "types": "./dist/experiment/index.d.ts",
+      "import": "./dist/experiment/index.js",
+      "default": "./dist/experiment/index.js"
+    },
+    "./trajectory-replay": {
+      "types": "./dist/trajectory-replay/index.d.ts",
+      "import": "./dist/trajectory-replay/index.js",
+      "default": "./dist/trajectory-replay/index.js"
+    },
     "./openapi.json": {
       "default": "./dist/openapi.json"
     }
@@ -158,12 +173,14 @@
     "test:watch": "vitest",
     "typecheck": "tsc --noEmit",
     "typecheck:examples": "tsc -p tsconfig.examples.json",
+    "typecheck:scripts": "tsc -p tsconfig.script.json",
     "lint": "biome check src",
     "format": "biome format --write src",
     "check:skill": "node scripts/check-skill.mjs",
+    "check:model-ids": "node scripts/check-model-id-requests.mjs",
     "check:analyst-benchmark": "node scripts/check-analyst-benchmark-implementation.mjs",
     "openapi": "node dist/cli.js openapi --out dist/openapi.json",
-    "verify:package": "pnpm check:analyst-benchmark && pnpm run check:skill && publint && attw --pack --profile esm-only . && node scripts/verify-package-exports.mjs"
+    "verify:package": "pnpm check:analyst-benchmark && pnpm run check:skill && pnpm run check:model-ids && publint && attw --pack --profile esm-only . && node scripts/verify-package-exports.mjs"
   },
   "dependencies": {
     "@asteasolutions/zod-to-openapi": "^9.1.0",