@tangle-network/agent-bench 0.3.7 → 0.3.8

package/src/swe-arena/fixtures/factory/agent-eval-309/manifest.json

@@ -0,0 +1,29 @@
+{
+  "id": "factory.agent-eval.309",
+  "repo": "tangle-network/agent-eval",
+  "repo_local_mirror": "/home/drew/code/agent-eval",
+  "base_commit": "5fe8d0c83a7e38f2893b2d7a0f6cd62dd5430521",
+  "judge_ref": "9b0a4f82257e07ac96d90d4af24364bc964ed178",
+  "spec_md": "spec.md",
+  "judge_tests": [
+    "src/paired-arms.test.ts",
+    "src/capability-headroom.test.ts"
+  ],
+  "excluded_tests": [],
+  "worker_visible_paths_note": "worker gets the base tree only (git archive of base_commit, no refs); judge tests overlaid at judge time",
+  "command_image": "node:24-bookworm@sha256:5711a0d445a1af54af9589066c646df387d1831a608226f4cd694fc59e745059",
+  "setup_cmds": [
+    "pnpm install"
+  ],
+  "judge_cmds": [
+    "npx vitest run src/paired-arms.test.ts src/capability-headroom.test.ts"
+  ],
+  "resolved_criterion": "all 30 judge tests pass; partial score = passed/30",
+  "timeout_s": 600,
+  "runtime": "node>=20, pnpm 11, vitest 3 (repo devDependency)",
+  "calibration": {
+    "gold": "PASS 30/30 (twice, deterministic)",
+    "base": "FAIL 2/2 test files (unresolvable imports: modules do not exist at base)",
+    "receipts": "calibration.md"
+  }
+}

package/src/swe-arena/fixtures/factory/agent-eval-309/spec.md

@@ -0,0 +1,64 @@
+# Add matched-pair arm comparison and a capability-headroom gate to the statistics layer
+
+## Background
+
+agent-eval ships a statistics module (`src/statistics.ts`) with paired estimators — `mcnemar`, paired risk difference, paired bootstrap, `wilcoxonSignedRank` — but every consumer that runs a two-arm A/B over run records hand-rolls the same two things on top of it:
+
+1. Matching baseline rows to treatment rows into pairs before feeding the paired estimators.
+2. A calibrate-before-measure gate that checks the benchmark can even see the capability being measured (if the capability-absent baseline already passes a task, that task carries no signal).
+
+Lift both into the library as new modules. Compose the existing statistics exports; do not add new math. No changes to any existing export.
+
+## Deliverable 1 — matched-pair arm comparison (`src/paired-arms.ts`)
+
+Input rows are arm-labeled records the caller projects from its own run data:
+
+- `PairedArmRow`: `{ pairKey: string; repKey?: string; arm: string; pass?: boolean; metrics?: Record<string, number> }`.
+
+### `pairArms(rows, { baselineArm, treatmentArm })`
+
+Builds matched pairs. Hard requirements:
+
+- **Pairing keys on row identity, never on outcome content.** A pair is an exact `(pairKey, repKey)` match across the two arms. Outcome-keyed matching is forbidden — it deflates discordant counts and manufactures fake lifts on null A/Bs.
+- Single-rep tasks (one row per arm for a `pairKey`) pair directly without `repKey`.
+- If any `pairKey` has multiple reps in either arm, **every** row of that `pairKey` must carry `repKey`; otherwise throw a validation error whose message names the offending pairKey and says a repKey is missing (e.g. matches `pairKey 't1' has multiple reps ... missing repKey`).
+- Throw on a duplicate `repKey` within one `(pairKey, arm)` group; message names the repKey, pairKey, and arm (matches `duplicate repKey 'r1' for pairKey 't1' in arm 'off'`).
+- Throw on an arm name with no rows; message lists the arms actually present (matches `no rows for arm 'onn' ... arms present: off, on`). Throw when baseline and treatment name the same arm (message contains `cannot be compared to itself`).
+- Rows without a counterpart are **reported, never dropped**: result carries `unpairedBaseline` and `unpairedTreatment` arrays holding the leftover rows.
+- Result shape: `{ pairs, unpairedBaseline, unpairedTreatment }`; each pair carries `pairKey`, a `repIndex` (0-based per-pair ordinal within its pairKey), and the full `baseline` / `treatment` rows.
+- Deterministic under input reordering: `pairArms(shuffled)` deep-equals `pairArms(original)`.
+
+### `comparePairedArms(rows, options)`
+
+Runs the full comparison over the pairs:
+
+- **Correctness**: derive discordant counts (baseline-pass/treatment-fail and the reverse) from `pass` and report exact McNemar (`pValue`) plus the paired risk difference under a `correctness` result section, with `nPairs`, `nUnpairedBaseline`, `nUnpairedTreatment` visible.
+- **Efficiency**: for every metric name present, per-pair deltas (treatment − baseline) summarized with the paired bootstrap and Wilcoxon signed-rank, reported as `metricDeltas` (one entry per metric).
+- A metric with zero complete pairs reports `null` for its bootstrap CI and Wilcoxon — the bootstrap's all-zero sentinel must never read as a measured tight null.
+- Bootstrap must be seeded/deterministic so repeated runs agree.
+
+## Deliverable 2 — capability-headroom gate (`src/capability-headroom.ts`)
+
+### `capabilityHeadroom(rows, options)`
+
+Given per-task baseline outcomes for the capability-absent arm, report which tasks have headroom (baseline fails → task can show the capability) with fail-closed semantics:
+
+- Outcomes are per-rep and may be pass / fail / unknown; **unknown never counts as headroom** and never counts as a known rep.
+- Reject an unrecognized baseline outcome value loudly (message names the bad value, e.g. `unrecognized baselineOutcome 'passed'`); reject an empty baseline (`no baseline rows`).
+- Per-task output includes `nKnown`; a `summary` includes `tasksWithGap` and `repsUnknown` so thin evidence stays visible.
+- Numeric guards validate loudly: minimum reps must be an integer ≥ 1 (message contains `integer ≥ 1`); where two of something are required the message says `need ≥ 2`.
+- Result exposes `tasks` (per-task rows) and `summary`.
+
+### `assertCapabilityHeadroom(rows, options)`
+
+Throws an actionable error when the benchmark cannot see the capability it claims to measure (no headroom); otherwise returns the headroom report.
+
+## Wiring
+
+- Export both modules' public symbols from `src/index.ts`.
+- Match repo conventions: strict TS, colocated `*.test.ts` allowed, biome-clean.
+
+## Acceptance
+
+- `pnpm typecheck` and `pnpm exec biome check` clean.
+- Hidden acceptance tests import `{ pairArms, comparePairedArms, type PairedArmRow }` from `src/paired-arms` and `{ capabilityHeadroom, assertCapabilityHeadroom }` from `src/capability-headroom`, plus `mcnemar` from `src/statistics`, and exercise every behavior above, including: identity-based multi-rep pairing on a both-reps-discordant scenario, leftover reporting, all the fail-loud validation messages, null CI on zero-pair metrics, determinism under reordering, and fail-closed unknown handling in the headroom gate.

package/src/swe-arena/fixtures/factory/agent-runtime-232/calibration.md

@@ -0,0 +1,48 @@
+# Calibration — factory.agent-runtime.232
+
+Run 2026-07-23, node v24.13.0, pnpm 11.15.1, local mirror `/home/drew/code/agent-runtime` (read-only; work in throwaway clones).
+
+## Gold run — base + real PR impl (tests excluded) must PASS
+
+```
+git clone /home/drew/code/agent-runtime work/ar-232-gold
+git -C work/ar-232-gold checkout ce6194a8e4e8468fb47d7a040f9df7366c0dda9b
+git -C /home/drew/code/agent-runtime diff ce6194a8 690a8fd0 -- ':(exclude)tests/' > ar232-impl.patch
+git -C work/ar-232-gold apply ar232-impl.patch
+mkdir -p work/ar-232-gold/tests/mcp
+git -C /home/drew/code/agent-runtime show 690a8fd0:tests/mcp/delegation-store.test.ts   > work/ar-232-gold/tests/mcp/delegation-store.test.ts
+git -C /home/drew/code/agent-runtime show 690a8fd0:tests/mcp/task-queue-durable.test.ts > work/ar-232-gold/tests/mcp/task-queue-durable.test.ts
+cd work/ar-232-gold && pnpm install
+npx vitest run tests/mcp/delegation-store.test.ts tests/mcp/task-queue-durable.test.ts
+```
+
+Output:
+
+```
+ ✓ tests/mcp/delegation-store.test.ts (10 tests) 12ms
+ ✓ tests/mcp/task-queue-durable.test.ts (11 tests) 82ms
+ Test Files  2 passed (2)
+      Tests  21 passed (21)
+```
+
+Re-run (flakiness check): `PASS (21) FAIL (0)`.
+
+## Base run — judge tests on bare base must FAIL
+
+Same clone/checkout of `ce6194a8`, overlay only the two test files, `pnpm install`, same vitest command.
+
+Output:
+
+```
+ Test Files  2 failed (2)
+      Tests  no tests
+```
+
+Both files fail at collection: `src/mcp/delegation-store` does not exist and `task-queue.ts` lacks the durable exports (`DelegationResumeDriver`, store option). Producing them is the task.
+
+## Judge-set notes
+
+- Test bodies use temporary-directory round trips and fake in-process drivers. The judge runs in the pinned factory container with network disabled and no operator environment. One test uses deterministic fake timers (`vi`). **Excluded: none.**
+- The judge asserts **typed error classes** (`DelegationPersistenceError`, `DelegationStateCorruptError`) and one error `kind` string (`'DriverRestartError'`), not message prose — clean behavioral contract; all named in `spec.md`.
+- `DelegationRecord`, `DelegationTaskQueue`, `hashIdempotencyInput`, and `DelegateCodeArgs` pre-exist at base; the judge extends them rather than asserting incidental internals.
+- The PR also touched `README.md` — excluded from the impl patch definition of "the feature" for grading purposes (docs churn, no behavioral content). It applies cleanly either way since the impl patch used `':(exclude)tests/'` only.

package/src/swe-arena/fixtures/factory/agent-runtime-232/manifest.json

@@ -0,0 +1,29 @@
+{
+  "id": "factory.agent-runtime.232",
+  "repo": "tangle-network/agent-runtime",
+  "repo_local_mirror": "/home/drew/code/agent-runtime",
+  "base_commit": "ce6194a8e4e8468fb47d7a040f9df7366c0dda9b",
+  "judge_ref": "690a8fd0e0ed5eec3a2b1c19031717a087aabd5d",
+  "spec_md": "spec.md",
+  "judge_tests": [
+    "tests/mcp/delegation-store.test.ts",
+    "tests/mcp/task-queue-durable.test.ts"
+  ],
+  "excluded_tests": [],
+  "worker_visible_paths_note": "worker gets the base tree only; judge tests overlaid at judge time",
+  "command_image": "node:24-bookworm@sha256:5711a0d445a1af54af9589066c646df387d1831a608226f4cd694fc59e745059",
+  "setup_cmds": [
+    "pnpm install"
+  ],
+  "judge_cmds": [
+    "npx vitest run tests/mcp/delegation-store.test.ts tests/mcp/task-queue-durable.test.ts"
+  ],
+  "resolved_criterion": "all 21 judge tests pass; partial score = passed/21",
+  "timeout_s": 600,
+  "runtime": "node>=20, pnpm 11, vitest (repo devDependency)",
+  "calibration": {
+    "gold": "PASS 21/21 (twice, deterministic)",
+    "base": "FAIL 2/2 test files (src/mcp/delegation-store does not exist; task-queue lacks durable exports)",
+    "receipts": "calibration.md"
+  }
+}

package/src/swe-arena/fixtures/factory/agent-runtime-232/spec.md

@@ -0,0 +1,48 @@
+# Durable delegation queue — persistence port, file store, restart-honest rehydration, resume seam
+
+## Background
+
+The MCP delegation server keeps its task registry (`DelegationTaskQueue` in `src/mcp/task-queue.ts`, records typed as `DelegationRecord`) purely in memory: when the MCP process restarts, delegation status/history vanish and an idempotently re-submitted task re-runs. `task-queue.ts` already names a Phase-2 follow-up: make the registry survive the process. Sandbox filesystems are snapshot-persisted by products, so a file-backed store makes delegation state genuinely span conversations.
+
+Build that follow-up. Additive only: zero behavior change for existing consumers who configure nothing; existing tests must stay untouched and green.
+
+## Deliverable 1 — persistence port + stores (new module `src/mcp/delegation-store.ts`)
+
+Public surface (also re-exported from `src/mcp/index.ts`):
+
+- `DelegationStore` — async port with the operations the queue needs: load all persisted records, upsert one record, remove one record, and resolve/persist idempotency-key → taskId mappings.
+- `InMemoryDelegationStore` — the default; semantics identical to today. Round-trips records, isolates stored state from caller mutation (no shared references), supports removal and idempotency resolution.
+- `FileDelegationStore` — single-file JSON snapshot store:
+  - A missing file is an empty store, not an error.
+  - Writes are atomic (write tmp file, then rename); concurrent upserts serialize into one parseable snapshot; no tmp litter left behind.
+  - Writing before an initial load is a programming error → typed `DelegationPersistenceError`.
+  - Upserts and removals persist across store instances (new instance over the same path sees them).
+  - Corrupt state (unparseable JSON, or valid JSON with the wrong shape) **refuses to load** with a typed `DelegationStateCorruptError` — never silently starts fresh.
+  - Explicit opt-in recovery (`recoverCorrupt` behavior, wired to env `AGENT_RUNTIME_DELEGATION_STATE_RECOVER=1` in the bin): archives the corrupt file (kept on disk under a recognizable archived name) and starts empty.
+- Typed errors `DelegationPersistenceError` and `DelegationStateCorruptError` exported from the same module.
+
+## Deliverable 2 — durable mode in `DelegationTaskQueue` (`src/mcp/task-queue.ts`)
+
+Queue options grow a `store?: DelegationStore` (default in-memory, unchanged behavior) plus:
+
+- **Restore/rehydration** (a `restore()` step a fresh queue instance runs over the store):
+  - Terminal records (completed/failed/cancelled) become queryable again: status and history visible to the fresh instance.
+  - The idempotency index rebuilds: re-submitting a previously-seen idempotency key returns the prior taskId and terminal state **without re-running** the delegate (`hashIdempotencyInput` unchanged).
+  - Records that were in-flight when the previous driver died must not pretend to be running: settle them as failed with a truthful error whose `error.kind` is `'DriverRestartError'` — **unless** the record carries a `detachedSessionRef` (see resume seam).
+  - `restore()` rejects with `DelegationStateCorruptError` over a corrupt state file.
+- **Resume seam**: a `DelegationResumeDriver` interface (a `tick`-style driver the queue polls to re-drive a detached session to completion, designed to map 1:1 onto the sandbox SDK's turn-drive result). Behavior:
+  - A restored in-flight record with a `detachedSessionRef` and a configured resume delegate is resumed through the driver (the driver sees the ref) and settles with the driver's outcome.
+  - With a `detachedSessionRef` but **no** resume delegate configured, the record settles failed truthfully (no resurrection).
+  - A driver tick that throws settles the record as failed.
+  - `cancel()` aborts an in-progress resume loop.
+- **Retention**: a `maxTerminalRecords` cap evicts the oldest terminal records beyond the cap, both in memory and in the persisted file. A non-positive cap is rejected loudly at construction.
+- **Fail loud on persistence loss**: after a store write failure the queue refuses further submissions (surfacing `DelegationPersistenceError`) rather than degrading to memory-only.
+
+## Deliverable 3 — wiring (`src/mcp/bin.ts`)
+
+- Opt-in via env `AGENT_RUNTIME_DELEGATION_STATE_FILE=<path>` → the bin constructs the queue over a `FileDelegationStore` at that path; `AGENT_RUNTIME_DELEGATION_STATE_RECOVER=1` enables corrupt-state recovery. Nothing set → in-memory, exactly today's behavior.
+
+## Acceptance
+
+- `pnpm typecheck`, lint, and the full existing test suite stay green.
+- Hidden acceptance tests import `{ DelegationPersistenceError, DelegationStateCorruptError, FileDelegationStore, InMemoryDelegationStore, type DelegationStore }` from `src/mcp/delegation-store` and `{ DelegationTaskQueue, hashIdempotencyInput, type DelegationResumeDriver }` plus `type DelegationRecord` from `src/mcp/task-queue`, and exercise every behavior above with real tmpdir files (atomicity, cross-instance persistence, corrupt-state refusal and recovery, restart-honest settlement, detached resume, dedupe-across-restart, eviction, write-failure lockout).

package/src/swe-arena/fixtures/factory/loops-28/calibration.md

@@ -0,0 +1,47 @@
+# Calibration — factory.loops.28
+
+Run 2026-07-23, node v24.13.0, pnpm 11.15.1, local mirror `/home/drew/code/loops` (read-only; work in throwaway clones).
+
+## Gold run — base + real PR impl (tests excluded) must PASS
+
+```
+git clone /home/drew/code/loops work/lp-28-gold
+git -C work/lp-28-gold checkout 3ca55b9eb24adae9d5c8383dc785bd610e1db5b0
+git -C /home/drew/code/loops diff 3ca55b9e 7805270f -- ':(exclude)tests/' > lp28-impl.patch
+git -C work/lp-28-gold apply lp28-impl.patch
+for t in best-effort worker-clone worker-evidence; do
+  git -C /home/drew/code/loops show 7805270f:tests/$t.test.ts > work/lp-28-gold/tests/$t.test.ts
+done
+cd work/lp-28-gold && pnpm install
+npx vitest run tests/best-effort.test.ts tests/worker-clone.test.ts tests/worker-evidence.test.ts
+```
+
+Output:
+
+```
+ Test Files  3 passed (3)
+      Tests  20 passed (20)
+```
+
+Re-run (flakiness check): `PASS (20) FAIL (0)`.
+
+## Base run — judge tests on bare base must FAIL
+
+Same clone/checkout of `3ca55b9e`, overlay only the three test files, `pnpm install`, same vitest command.
+
+Output:
+
+```
+ Test Files  3 failed (3)
+      Tests  no tests
+```
+
+All three fail at collection: `src/best-effort.ts`, `src/worker-clone.ts`, `src/worker-evidence.ts` do not exist at base.
+
+## Judge-set notes
+
+- Tests spawn `git` in temporary directories with identity passed inline, and use symlink/chmod, so they are POSIX-only. The judge runs in the pinned factory container with network disabled and no operator environment. **Excluded: none.**
+- `tests/worker-evidence.test.ts` imports the bound constants (`EVIDENCE_MAX_CHARS` etc.) and asserts **relative to them** (`length ≤ EVIDENCE_MAX_CHARS`, tail contains `'v'.repeat(VERIFY_TAIL_CHARS)`) — exact values are the builder's choice; the names and semantics are contract and are in `spec.md`. Not trimmed: self-referencing bounds have behavioral content (bounding + tail retention).
+- One pinned prose contract: the best-effort delivery commit subject must contain `best-effort delivery from <label>` and `no worker passed the verify gate` — promoted into `spec.md` as acceptance criteria.
+- Judge imports `@tangle-network/agent-runtime/loops` (`gitWorkspace`, `runInWorkspace`) — a published npm dependency already in the base lockfile, not sibling unmerged work.
+- Scope note: the merged PR also tuned worker budgets and prompts (`extensions/pi/`, `src/top-model.ts`); the judge does not cover those, so this instance grades the three library modules only. The spec describes the modules as the deliverable.

package/src/swe-arena/fixtures/factory/loops-28/manifest.json

@@ -0,0 +1,30 @@
+{
+  "id": "factory.loops.28",
+  "repo": "drewstone/loops",
+  "repo_local_mirror": "/home/drew/code/loops",
+  "base_commit": "3ca55b9eb24adae9d5c8383dc785bd610e1db5b0",
+  "judge_ref": "7805270fda9f82869028b197e9424292ec611245",
+  "spec_md": "spec.md",
+  "judge_tests": [
+    "tests/best-effort.test.ts",
+    "tests/worker-clone.test.ts",
+    "tests/worker-evidence.test.ts"
+  ],
+  "excluded_tests": [],
+  "worker_visible_paths_note": "worker gets the base tree only; judge tests overlaid at judge time",
+  "command_image": "node:24-bookworm@sha256:5711a0d445a1af54af9589066c646df387d1831a608226f4cd694fc59e745059",
+  "setup_cmds": [
+    "pnpm install"
+  ],
+  "judge_cmds": [
+    "npx vitest run tests/best-effort.test.ts tests/worker-clone.test.ts tests/worker-evidence.test.ts"
+  ],
+  "resolved_criterion": "all 20 judge tests pass; partial score = passed/20",
+  "timeout_s": 600,
+  "runtime": "node>=20, pnpm 11, vitest 4 (repo devDependency); requires local git binary; POSIX (symlink/chmod)",
+  "calibration": {
+    "gold": "PASS 20/20 (twice, deterministic)",
+    "base": "FAIL 3/3 test files (src/best-effort.ts, src/worker-clone.ts, src/worker-evidence.ts do not exist at base)",
+    "receipts": "calibration.md"
+  }
+}

package/src/swe-arena/fixtures/factory/loops-28/spec.md

@@ -0,0 +1,50 @@
+# Supervisor delivery robustness — best-effort patch delivery, artifact-inheriting worker clones, evidence-rich settlement
+
+## Background
+
+The loops supervisor fans a coding task out to workers, each in a fresh `git clone` of the shared workspace, and settles each worker with a verify verdict. A paired SWE-bench round exposed three delivery defects, all reproduced from run artifacts:
+
+1. **All-or-nothing delivery.** A worker whose patch applied but whose verify failed delivered *nothing* — several losses were literal 0-byte deliveries of real work.
+2. **`git clone` drops untracked compiled artifacts.** Fresh worker clones were missing untracked/ignored build outputs present in the workspace (the compiled-tree shape), producing wholesale import crashes in workers.
+3. **Evidence starvation at settlement.** The supervisor rated workers on a one-line verdict; failed patches, the verify tail, and the worker's own closing note never survived to the brain, so "close — refine it" and "dead end" looked identical.
+
+Build three library modules fixing these, with the event-dir conventions below. (The supervisor loop wiring in `extensions/pi/` consumes them; the modules themselves are the deliverable.)
+
+Worker event-dir convention used throughout: each worker `<label>` writes its patch to `<eventDir>/workers/<label>.patch` and appends lifecycle events to `<eventDir>/workers/<label>.ndjson` — a `finished` event carries `{ kind: 'finished', passed, testPassed, typecheckPassed, at }` (ISO timestamp).
+
+## Deliverable 1 — best-effort delivery (`src/best-effort.ts`)
+
+- `rankBestEffortCandidates(candidates)` — orders failed-verify candidates `{ label, patch, testPassed, typecheckPassed, finishedAt }` by: verify progression first (typecheck passed beats not), then larger patch, then more recent `finishedAt`, then label. Deterministic: ranking a shuffled copy yields the identical order.
+- `deliverBestEffortPatch(eventDir, workspace)` — when no worker passed the gate, scan the event dir for worker patches and deliver the best applicable one into the workspace:
+  - Skip blank/whitespace-only patches; if only those exist, deliver nothing and leave the workspace untouched (return `undefined`).
+  - Try candidates in ranked order; a patch that does not apply is skipped and the next one is tried.
+  - A delivered patch is **applied and committed** in the workspace; the commit subject marks it as a best-effort delivery, names the worker (`best-effort delivery from <label>`), and states that no worker passed the verify gate.
+  - Returns `{ worker, patchBytes }` for the delivered candidate.
+
+## Deliverable 2 — artifact-inheriting worker clones (`src/worker-clone.ts`)
+
+- `copyUntrackedIntoClone(sourceWorkspace, cloneDir)` — copies files present in the source workspace but absent from a fresh clone: untracked files AND git-ignored build outputs. Returns stats including `copied` (count). Requirements:
+  - Preserve the executable bit; rewrite symlinks whose targets point inside the source workspace to point inside the clone.
+  - Skip loop-infrastructure directories and nested git repos.
+  - A payload above a sanity bound warns but still copies (never silently truncates the workspace).
+- `withUntrackedArtifacts(...)` — wraps worker-clone materialization so a worker clone contains both the tracked files and the untracked artifacts, while a valid delivery (worker pushing back to the shared ref) **never commits the copied artifacts** back to the shared repository.
+
+## Deliverable 3 — evidence-rich settlement (`src/worker-evidence.ts`)
+
+Exported bounds (constants, also exported): `EVIDENCE_MAX_CHARS`, `VERIFY_TAIL_CHARS`, `NOTE_MAX_CHARS` (order-of-magnitude guidance: a few thousand / ~1k / a few hundred chars).
+
+- `composeWorkerEvidence(input)` — one bounded evidence block for the brain from a worker's patch, verify log, and note:
+  - Always ≤ `EVIDENCE_MAX_CHARS`, even when every input is oversized.
+  - Keeps the **tail** of the verify log (at least the last `VERIFY_TAIL_CHARS` chars) — the failing assertion lives at the END of the log.
+  - Labels an empty patch explicitly; surfaces the worker note; a passing verify is reported without inventing a failure section.
+- `settledWorkerOut(input: { passed, patch, evidence })` — what settlement hands the brain:
+  - Passing worker with real edits → its patch (the deliverable).
+  - Failing worker → its evidence block, never the raw patch.
+  - Passing NO-EDIT worker (e.g. a post-delivery reviewer) → its evidence, not an empty patch.
+  - No evidence at all → fall back to the patch; nothing at all → empty string.
+- `closingWorkerNote(stdout, stderr)` — the worker's closing self-report: keep the **tail** of stdout (≤ `NOTE_MAX_CHARS`) so a final verdict line survives; fall back to stderr when stdout is blank; `undefined` when both are.
+
+## Acceptance
+
+- Strict TS, repo lint clean, existing tests untouched and green.
+- Hidden acceptance tests import `{ deliverBestEffortPatch, rankBestEffortCandidates }` from `src/best-effort.js`, `{ copyUntrackedIntoClone, withUntrackedArtifacts }` from `src/worker-clone.js`, and `{ closingWorkerNote, composeWorkerEvidence, settledWorkerOut, EVIDENCE_MAX_CHARS, NOTE_MAX_CHARS, VERIFY_TAIL_CHARS }` from `src/worker-evidence.js`, driving real local git repos in tmpdirs through every behavior above.

package/src/swe-arena/fixtures/run-report/README.md

@@ -0,0 +1,43 @@
+# run-report evidence
+
+Reports over completed runs, committed as the measured record behind the supervisor-behavior
+claims. Both source run directories were read READ-ONLY (`--report-dir` wrote the reports
+here, never into the run).
+
+The reader is `@tangle-network/agent-eval/supervisor-run` — a supervision tree is a rollout
+trace with one more dimension, so it lives in the trace-analysis layer next to single-rollout
+analysis. `src/swe-arena/run-report.mts` is only the CLI over it.
+
+These files were captured before that move, so they are the pre-move record, not current
+library output. Every metric key and value below was reproduced from the same run bytes with
+zero diffs when the reader moved (tangle-network/agent-eval#417); what a re-run today would
+change is the envelope and the metric set, never these numbers:
+
+- `schema` is `swe-arena/run-report@1` / `swe-arena/run-report-rollup@1` here; the library
+  now emits `tangle.supervisor-run@1` / `tangle.supervisor-run-rollup@1`.
+- the per-run envelope names the source `cellDir`; the library names it `runRef`, because the
+  input contract is a reader over bytes rather than a directory.
+- `economics.brainTruncations` did not exist at capture time and is absent here.
+
+Re-capturing them means re-reading the source run directories, which are not part of this
+repo — so they stay as committed evidence rather than being regenerated in place.
+
+| File | Source run | Command |
+|---|---|---|
+| `gen3-rollup.{json,md}` | gen-3 arena, 51 cells (`…/scratchpad/hh/gen3`) | `tsx src/swe-arena/run-report.mts --round <gen3> --report-dir <out>` |
+| `factory-agent-eval-309-FSUP0.{json,md}` | factory gen-0 live, cell `factory.agent-eval.309` rep 0 (`/tmp/factory-gen0-live`) | `tsx src/swe-arena/run-report.mts <cellDir> --patch <p> --ledger <ledger.jsonl> --report-dir <out>` |
+
+Headline facts these files carry:
+
+- **gen-3: 0 steers across all 51 cells.** 42 cells measured a real `0`; 9 cells report
+  `unavailable` (no `workers/` directory — the supervisor never started a worker there).
+  161 workers spawned, 35 accepted, mean 3.1 waves per cell, mean worker utilization 0.47,
+  mean idle share 52.6%, $3.26 brain spend, 17/51 judged resolved.
+- **factory.agent-eval.309 (FSUP0, rep 0): 0 steers, 4 waves `[2,1,1,1]`, 5 workers,
+  utilization 0.663, idle 15.6min of a 32.2min run (48.3%).** 3 accepted / 1 rejected /
+  1 empty-pass; every respawn was preceded by settled evidence (3/3); delegation depth 1;
+  judge 13/30 (score 0.4333, unresolved).
+
+Read together: the supervisor is a dispatcher with a retry loop — spawn → wait → read
+settled evidence → respawn — not a lead steering contributors mid-task. Half the wall clock
+has no worker running at all.

package/src/swe-arena/fixtures/run-report/factory-agent-eval-309-FSUP0.json

@@ -0,0 +1,173 @@
+{
+ "schema": "swe-arena/run-report@1",
+ "cellDir": "/tmp/factory-gen0-live/rep-0/runs/factory.agent-eval.309/FSUP0",
+ "instanceId": "factory.agent-eval.309",
+ "arm": "FSUP0",
+ "supervisorId": "sup-1-8f75ee",
+ "generatedAt": "2026-07-24T00:32:17.691Z",
+ "orchestration": {
+  "workersSpawned": 5,
+  "workersSettled": 5,
+  "workersCancelled": 0,
+  "steers": 0,
+  "steersDelivered": 0,
+  "steersByWorker": [
+   {
+    "worker": "w-0",
+    "queued": 0,
+    "delivered": 0
+   },
+   {
+    "worker": "w-1",
+    "queued": 0,
+    "delivered": 0
+   },
+   {
+    "worker": "w-2",
+    "queued": 0,
+    "delivered": 0
+   },
+   {
+    "worker": "w-3",
+    "queued": 0,
+    "delivered": 0
+   },
+   {
+    "worker": "w-4",
+    "queued": 0,
+    "delivered": 0
+   }
+  ],
+  "driverSteerCalls": 0,
+  "waves": 4,
+  "waveSizes": [
+   2,
+   1,
+   1,
+   1
+  ],
+  "maxConcurrency": 2,
+  "respawns": 3,
+  "repeatedLabels": [],
+  "delegationDepth": 1,
+  "timeToFirstSpawnMs": 831016,
+  "supervisorWallMs": 1934608,
+  "idleMs": 934563,
+  "idlePct": 48.3,
+  "workerUtilization": 0.663
+ },
+ "decision": {
+  "settledByStatus": {
+   "done": 5
+  },
+  "settledVerdicts": {},
+  "accepted": 3,
+  "rejected": 1,
+  "emptyPass": 1,
+  "observeThenRespawn": 3,
+  "respawnWithoutEvidence": 0,
+  "reviewActions": 0,
+  "workerEvidenceBytes": 12153
+ },
+ "economics": {
+  "brain": {
+   "tokensIn": 228169,
+   "tokensOut": 19398,
+   "usd": 0.179577,
+   "source": "journal metered events (n=17)"
+  },
+  "workers": {
+   "tokensIn": 286763,
+   "tokensOut": 69794,
+   "usd": 0,
+   "source": "journal settled spend + opencode sessions (n=5)"
+  },
+  "totalUsd": 0.179577,
+  "totalUsdSource": "state.json result.spentUsd — brain-priced only; worker CLI inference is unpriced (see worker token counts)",
+  "costPerAcceptedPatchUsd": 0.059859,
+  "workerWallMsDistribution": {
+   "n": 5,
+   "min": 70445,
+   "p50": 235872,
+   "p90": 556457,
+   "max": 556457,
+   "sum": 1274117
+  },
+  "perWorker": [
+   {
+    "worker": "w-0",
+    "wallMs": 556457,
+    "tokensIn": 0,
+    "tokensOut": 0,
+    "usd": 0,
+    "patchBytes": 21183,
+    "passed": false
+   },
+   {
+    "worker": "w-1",
+    "wallMs": 279898,
+    "tokensIn": 0,
+    "tokensOut": 0,
+    "usd": 0,
+    "patchBytes": 14573,
+    "passed": true
+   },
+   {
+    "worker": "w-2",
+    "wallMs": 70445,
+    "tokensIn": 0,
+    "tokensOut": 0,
+    "usd": 0,
+    "patchBytes": 0,
+    "passed": true
+   },
+   {
+    "worker": "w-3",
+    "wallMs": 235872,
+    "tokensIn": 0,
+    "tokensOut": 0,
+    "usd": 0,
+    "patchBytes": 21325,
+    "passed": true
+   },
+   {
+    "worker": "w-4",
+    "wallMs": 131445,
+    "tokensIn": 0,
+    "tokensOut": 0,
+    "usd": 0,
+    "patchBytes": 1016,
+    "passed": true
+   }
+  ]
+ },
+ "outcome": {
+  "supStatus": "completed",
+  "supVerdict": "delivered",
+  "delivered": true,
+  "judgeResolved": false,
+  "judgeScore": 0.4333,
+  "judgePassed": 13,
+  "judgeTotal": 30,
+  "verifyPass": {
+   "unavailable": "result.json absent or has no verify_pass"
+  },
+  "verifyRc": {
+   "unavailable": "result.json absent or has no verify_rc"
+  },
+  "patch": {
+   "files": 5,
+   "linesAdded": 939,
+   "linesRemoved": 0,
+   "testFilesTouched": [
+    "src/capability-headroom.test.ts",
+    "src/paired-arms.test.ts"
+   ]
+  },
+  "judgeSource": "/tmp/factory-gen0-live/ledger.jsonl (ledger row)"
+ },
+ "gaps": [
+  "verifyPass: result.json absent or has no verify_pass"
+ ],
+ "traceCommand": "npx --yes @tangle-network/traces@latest analyze --harness opencode --cwd <worker-clone-cwd>"
+}