@tangle-network/agent-bench 0.1.0 → 0.3.5

package/CHANGELOG.md

@@ -0,0 +1,17 @@
+# Changelog
+
+## 0.3.5
+
+- Build the workspace runtime before source type checks so the published-package verification command works from a clean checkout; the packed consumer still installs `@tangle-network/agent-runtime@0.97.0` from npm.
+
+## 0.3.4
+
+- Publish the current benchmark suite against `@tangle-network/agent-runtime@0.97.0` and align its evaluation dependency with runtime and knowledge at `@tangle-network/agent-eval@0.122.8`.
+
+## 0.3.3
+
+- Consume `@tangle-network/agent-runtime@0.94.9` so large candidate task-outcome and isolated-memory archives are validated before persistence, then recorded as artifact references instead of oversized embedded payloads.
+
+## 0.3.2
+
+- Run large Pier workspace identity checks from a hash-verified evaluator file, avoiding Linux command-size limits while preserving exact path, type, mode, length, hardlink, and content checks.

package/HARNESS.md

@@ -0,0 +1,302 @@
+# bench harness — START HERE (the map, so you don't re-read 15 files)
+
+If you're an agent picking this up: read this page, then run `pnpm help` + `pnpm gate` —
+do NOT re-derive the harness from source. This map is SHORT on purpose; if it disagrees
+with the code, the code wins — fix this page in the same turn (the anti-rediscovery law).
+Verified against source 2026-07-18 · agent-eval pinned `0.122.8`. The CANONICAL surface is now
+the published optimization suite (`@tangle-network/agent-runtime/loops`): `Environment` +
+`Strategy`/`defineStrategy` + `runBenchmark` — see the section below FIRST. The recursive
+diverse-vs-blind gate runs through the keystone (`gate-cli.mts` → `runGate`);
+the offline selector replay (`corpus-replay.mts` / `corpus-report.mts`) gates the legacy corpora.
+
+## What this harness answers
+**The success criterion is Gate B** (docs/learning-flywheel.md, docs/architecture.md §2): across
+repeated runs on a persistent, checkable task family, the deployed policy's verifier-graded
+**multi-objective** score (correct · fast · secure · cheap, each its own deployable checker)
+improves **run-over-run** at matched per-run compute, surviving a frozen-policy control, significant
+at adequate n. That across-run slope is RSI. **The harness has NOT yet run Gate B** — see the durable
+gap below.
+
+What the harness measures **today is Gate A** (docs/roadmap-rsi.md — the inner GO/NO-GO for the
+within-run adaptive-driver layer): **does any non-blind topology beat blind compute at EQUAL COMPUTE
+(Σ rollouts × turns — `k` counts rollouts, each may be multi-turn/stateful), under a DEPLOYABLE
+(non-oracle) selector, at significant n?** Gate A is a **narrow diagnostic** — the cost-justification
+for parallel/adaptive topology, **NOT** the product verdict. A failed Gate A deletes within-run
+steering only; it never touches the corpus+policy product (Gate B). The invariant is equal-COMPUTE,
+not equal-k-on-stateless-samples.
+
+**Terminology (one word, used consistently).** A **rollout** (≡ a "shot") is ONE agent running an
+`AgentProfile` to completion — a full, possibly **multi-turn / stateful** trajectory. `k` counts
+*rollouts*; **turns live *inside* a rollout**, never as separate shots. A single **stateless
+completion** (`maxTurns=0`, `harness: null`, one model call, no persistent workspace) is the
+*degenerate* rollout — fine as a selector **lower bound**, never the canonical unit. The HumanEval
+probe (`bench/src/humaneval-gate.mts`) uses exactly that degenerate shape — it calls the router
+directly and does **not** route through `AgentProfile` / the sandbox / the keystone — so its numbers
+are the **no-self-correction lower bound** on the selector, distinct from the rollout-based keystone
+gate above. Bridge it to the product by running the same arms with real rollouts (an `AgentProfile`
+through `runLoop`), dialing `maxTurns`.
+
+Two things to keep straight: today's judges grade a single
+*correctness* scalar (the multi-objective vector is the open contract, architecture.md §6), and every
+number below is single-objective + within-run — read them as Gate-A diagnostics, not Gate-B results.
+- Within-run STEER (verify-and-revise family) **LOSES** (rung-0, n=40: blind 37.5% →
+  random@3 60.0% → refineGepa@3 45.0%; the earlier +20pp was confounded compute).
+- On the COMMITTED finsearch corpus, the self-consistency selector also **loses**:
+  selector@k − random@k = **−8.2pp** (n=51). So "pick the consensus among k identical-ish
+  attempts" does not beat a random draw here.
+- **VERIFIER-GROUNDED selection is the one selector that wins** — but ONLY where a domain has
+  WITHIN-TASK graded variance. Proven POSITIVE on HumanEval (deployable-checker, binary, n=50:
+  verifier−sc = +12.0pp CI[+4,+22]). The continuous-reward generalization (`selector.ts`
+  `summarizeVerifierSelector`, `corpus-replay --selector=verifier`) ranks k attempts by their
+  stored deployable-checker `score` and reports selector(=best-of-k) vs random(=mean-of-k) with a
+  paired bootstrap CI. **aec-bench is structurally DEAD for it** (n=12 gpt-4.1, **0/12 random +
+  1/12 diverse tasks have any within-task score spread**): closed-form engineering calcs score
+  deterministically w.r.t. sampling — across-task difficulty (33% resolve) but ~0 within-task
+  selection headroom. The gpt-5 null (oracle 2.5%) was a worker artifact (no JSON emission);
+  gpt-4.1 fixes the band (mean 36.1%) but the selector gate stays flat. **commit0 is the right
+  Layer-1 domain** (different impls pass different test subsets → real within-task spread).
+- **UNTESTED (still Gate A):** parallel **DIVERSE strategies** (different reasoning paths,
+  `directives.ts` → `DIVERSE_STRATEGY_LENSES` / `composeStrategies`) @k vs blind sample(n=k). A
+  distinct family from what rung-0 falsified — the open *within-run* question, and what runProgram's
+  `parallel` is built to deploy.
+- **UNBUILT (Gate B):** the across-run policy-improvement curve on a multi-objective task stream.
+  No harness runs it yet; it is the durable next step, not a corpus-replay over the existing
+  single-objective records.
+
+## The run archive (where results LIVE)
+
+Every run's full artifact is committed under `agent-lab/runs/<date>/` — self-describing JSON
+(models + config + per-task cells + gate verdicts with CIs), portable to any repo without
+this codebase. `agent-lab/runs/RUNS.md` is the index mapping artifacts → verdicts → the
+findings gist. **Set `OUT=runs/<date>/<name>.json` (in agent-lab) on every run — never `/tmp`**
+(a reboot erases ramdisk; ~20 runs nearly died there once).
+
+## Data flow (the whole experiment in one line)
+`rollout (worker → answer) → adapter.judge (valid?) → CORPUS RunRecord (k attempts, output+valid each) → corpus-replay --selector (pick WITHOUT the judge) → corpus-report CI → gate verdict`
+The expensive part (rollouts) produces a **reusable corpus**; selection + stats are free
+and offline (zero new rollouts, zero judge calls).
+
+## THE CANONICAL SUITE (2026-06-10) — the published path; start here
+
+The optimization layer ships from the package; bench scripts compose it. A domain = an
+`Environment` (5 hooks); a strategy = how budget is spent to beat its check; `runBenchmark`
+returns per-strategy means + the per-task LOSSES table + the (score,$) Pareto frontier.
+Promotion is the package gate (`promotionGate` — seeded paired bootstrap, evidence floor,
+two modes: **superiority** and **non-inferiority** = score CI low > −tolerance AND cost
+savings CI low > 0, the "same quality, cheaper" gate; verdicts carry paired Δlatency).
+Authoring is `authorStrategy` (named `fallbackModel` retry). **Funnel-alignment law**: the
+search-side champion tie-band must be no stricter than the gate's tolerance (under
+`OBJECTIVE=cost` it defaults to it). Endurance envs on the evolve runner: `CHECKPOINT=path`
+(phase ledger + resume — a killed run re-pays ONE phase), `GYM_RECREATE='docker …'`
+(recreate the container at phase boundaries — the wedge killer). Observability:
+`createWaterfallCollector` (every spawn billed+timed) + `anytimeReport` (TTT / shots-to-
+target / COCO ERT / hill-climb AUC per satisficing target). Models policy: cheap router
+models only (defaults `deepseek-v4-pro`/`deepseek-v4-flash`; compressor = flash with
+`gpt-4o-mini` fallback) — never CC models; every verdict banner + artifact is
+self-describing (models + config).
+
+| entry point | what it answers | one-liner |
+|---|---|---|
+| **the research lines** | the flywheel/evolution runs, σ×κ factor grid, steering hypercube, model matrix, E3 certified memory, depth-vs-breadth, corpus A/Bs — **moved to [tangle-network/agent-lab](https://github.com/tangle-network/agent-lab) (private)** with the EOPS/math domains and the run archive | `~/code/agent-lab` — map in its README |
+| `src/commit0-env-run.mts` | the HARD domain (implement whole libraries vs their test suites) through `runBenchmark` | `IDS=commit-0/wcwidth BUDGET=3 INNER_TURNS=10 tsx src/commit0-env-run.mts` |
+| `src/examples/strategy-demo.mts` | the 3-layer API demo (gym-free) | `WORKER_MODEL=gpt-4o-mini tsx src/examples/strategy-demo.mts` |
+| `src/examples/math-demo.mts` | any-domain proof: math via `createVerifierEnvironment` (the tax/legal/gtm answer-shape) | `BUDGET=3 tsx src/examples/math-demo.mts` |
+
+`run-benchmarks-cli.mts` can run a bounded per-task refinement loop with
+`LOOP_ATTEMPTS=N`: attempt 1 answers the original task; later attempts receive previous artifacts
+plus redacted checker feedback; the loop stops early on pass. This is for testing whether agents can
+use the benchmark's own feedback to solve the task, not for leaking gold answers into prompts.
+For local subscription-backed workers, set `BACKEND=bridge`, `BRIDGE_URL`, `BRIDGE_BEARER`, and pass
+the full bridge model id in `CELLS`, e.g. `CELLS=opencode/deepseek/deepseek-v4-pro`.
+
+EOPS standup (one container): `docker run -d --rm --name eops -p 8006:8005
+shivakrishnareddyma225/enterpriseops-gym-mcp-itsm:latest` + `EOPS_GYM_DBS_DIR=<unzipped
+gym_dbs.zip from github.com/ServiceNow/EnterpriseOps-Gym>`; restart it FRESH per big run
+(it wedges under load); `EOPS_SPLIT=csm|hr|…` selects other domains (their gym containers
+not yet sourced). **Parallel lanes:** tasks carry the dataset's literal gym URL
+(`http://localhost:8006`); `EOPS_GYM_URL=http://localhost:8007` rebases every server URL,
+so N concurrent runs use N containers (`-p 8007:8005`, `-p 8008:8005`, …) instead of
+serializing on one wedge-prone gym. Bring-up check: `agent-lab/domains/lane-probe.mts`. Cross-cutting laws baked into the suite: keep-best checkpoint scoring
+(final-state scoring is biased −6–8pp), equal compute via the conserved pool, the analyst
+is firewalled (trace-only), costs are real (router usage → `{usd, ms, tokens}`).
+
+### The QUEUED runs for the test fleet (wired, one command each, unrun)
+1. **Relevance-primed corpus A/B** — `PRIME_MODE=relevance K_FACTS=2 N=16 HOLDOUT=4` (the read-side design that survived the naive-priming negative).
+2. ~~Strategy tournament at power~~ — RAN (n=24, budget 4, ×3 configs): HOLD verdicts; the cost-frontier finding ×3 + the funnel-alignment law came out of these. Live ledger: `.evolve/current.json` + the findings gist.
+3. **Commit0 at real budget** — `BUDGET=3 INNER_TURNS=12 N=3` sample-vs-refine on the hard domain.
+4. **Cross-domain replication** — blocked on sourcing the csm/hr gym containers (`EOPS_SPLIT` is wired).
+
+## Commands (the standalone tools — each its own `main`)
+the gate + measurement tools:
+  corpus-replay.mts  --selector: selector@k vs random@k vs oracle@k over a corpus (THE offline gate)
+  corpus-report.mts  paired-bootstrap CI + Benjamini-Hochberg over corpora
+  gate-cli.mts  the recursive diverse-vs-blind gate through `runGate` (Supervisor)
+  run-benchmarks-cli.mts  runBenchmarks: any subset of the ADAPTERS registry × model/harness cells, one combined ranked report (#420)
+  commit0-env-run.mts  the HARD domain through `runBenchmark` (the optimization suite)
+  terminal-compare.ts  Terminal-Bench compare (own main)
+  pnpm verify:pier  zero-model failure/pass Pier controls through a separate verifier
+unit tests (the only fully-green, cred-free runnable surface besides offline replay):
+  node --test --import tsx src/{selector,refine-loop}.test.mts
+  tsx src/gate.test.mts   # offline plumbing test (no creds)
+
+## Run the GATE — today, zero creds (it already runs)
+```
+cd bench
+pnpm gate                                              # = corpus-replay.mts corpus/finsearch.jsonl --selector
+tsx src/corpus-replay.mts corpus/finsearch.jsonl --selector --condition=refine   # other arms
+tsx src/corpus-replay.mts <corpus.jsonl> --selector=verifier   # GRADED domains: rank k by deployable-checker score
+pnpm gate-report                                       # paired-bootstrap CI + BH-FDR
+```
+`--selector=verifier` is for corpora whose attempts carry a continuous `score` (commit0
+pytest pass-rate / aec verify.py partial credit) and where text doesn't cluster: it ranks by
+the deployable checker (argmax score) and reports selector vs random with a paired bootstrap CI.
+It needs WITHIN-TASK score spread to move — flat on aec (closed-form), live on commit0 (code).
+The committed `corpus/finsearch.jsonl` (152 records: random@3 / refineHand@3 / refineGepa@3)
+makes the gate replayable with no rollouts. To gate the DIVERSE arm you generate a
+diverse-strategy corpus (k different `composeStrategies` prefixes per instance) by running
+`gate-cli.mts` with the distinct-directive arms — the blind (identical-children) arm is the
+control on the same run.
+
+## Run the DIVERSE-vs-blind gate THROUGH the keystone (the recursive runtime, live)
+```
+cd bench
+export TANGLE_API_KEY=…                                 # router + the deployable judge
+BENCH=enterpriseops-gym EOPS_FIXTURES=1 N=20 K=4 pnpm gate-cli
+```
+`gate-cli.mts` → `runGate` (`src/gate.ts`): a `Persona` + the generic
+`fanout` combinator over the budget-conserving `Supervisor`. Blind = K identical children, diverse
+= K distinct strategy directives — equal-k by construction (conserved pool), proven by
+`equalKOnCost`. The DEPLOYABLE selector is the benchmark's OWN `adapter.judge` (each child solves
+via the router, is graded by the runnable checker, and that `BenchScore` is the child's verdict
+`defaultSelectWinner` ranks on — selector ≠ oracle/LLM-judge). Pick a deployable-checker bench
+(enterpriseops-gym / swe-bench / terminal-bench), NOT finsearchcomp (LLM-judge → not deployable).
+Offline plumbing test (no creds): `tsx src/gate.test.mts`. The gate runs through the SAME recursive
+atom every personified loop uses.
+
+## "Supervisor" (iterate/decompose) vs blind — through the PUBLISHED suite
+The supervisor-vs-blind gate is NOT a bespoke harness: it is `runBenchmark([sample, refine, …])`
+over an Environment. blind = `sample` (best-of-k); "supervisor" = `refine`/`sampleThenRefine`
+(depth: attempt→firewalled-analyst-steer→retry — *"a multi-agent team is just a Strategy whose driver
+spawns several agents"*). Equal compute by the substrate's CONSERVED budget; the deployable check is
+the Environment's `score`; the can't-fake-the-check firewall is built in. Run it on the HARD real
+domain via `commit0-env-run.mts` (above) or the toy `strategy-demo.mts` (offline). The LLM
+agent-driver (an LLM that itself decides spawns via the coordination MCP) is the SEPARATE product
+path — `atom-mcp-e2e.mts` / `atom-commit0.mts` — not a strategy. Evolve any strategy on a frozen
+holdout with `runStrategyEvolution`.
+
+## Generate a fresh corpus + gate it
+The rollout generators now live with their domains: the recursive gate
+(`gate-cli.mts`) and the optimization-suite env runs (`commit0-env-run.mts`,
+`research-gate.mts` for the off-sandbox RAG baseline) each append corpus `RunRecord`s. Gate any
+written corpus offline with the selector:
+```
+tsx src/corpus-replay.mts <corpus.jsonl> --selector
+```
+(hotpotqa is cheap + deterministic-judge but near-ceiling/weak-signal; simpleqa similar;
+finsearchcomp is the strong-signal domain but needs the sandbox/local-web worker.)
+
+## Optimize the strategy/prompt (so the gate tests BEST-effort, not strawman)
+Strategy-space search is the package's `runStrategyEvolution` (the optimization suite); the diverse
+lenses (`directives.ts`) layer on top of the shared base directive consumed by the gate arms.
+
+## Workers (the rollout substrate)
+The gate solves each child via the router and grades it with the benchmark's own
+deployable `adapter.judge`; `research-gate.mts` is the off-sandbox retrieve→answer baseline
+(`SEARCH=<provider>` selects the web-search arm). The steer text lives in `directives.ts`, NOT in the
+worker (the worker is substrate). A strategy is a prompt PREFIX; the judge is unchanged.
+
+## Adapters (benchmarks/) — honest state (the code wins over this line; verified 2026-06-04)
+The code-benches share `benchmarks/_harness.ts` (stage artifact → run the bench's OWN evaluator
+in a `.venv`/Docker subprocess → parse its JSON report → `{resolved,score}`). No per-adapter
+copy of the process/venv/Docker/temp/report plumbing; commit0+appworld also share its
+stdin-piping runner (`runVenvScriptStdin`).
+Published-package consumers set `AGENT_BENCH_PYTHON` to the absolute path of an interpreter containing the benchmark dependencies.
+If unset, source checkouts keep using `bench/.venv/bin/python`.
+SWE-bench callers that need a hard evaluation deadline construct `createSweBenchAdapter({ timeoutMs })`; evaluator errors and incomplete or ambiguous reports throw instead of scoring zero.
+Callers that must inspect the exact per-task image after scoring pass `cacheLevel: 'instance'`; the default remains `'env'`.
+Callers retain the complete official evaluator tree plus raw process output with `captureEvaluatorArtifacts: ({ taskId, attemptSequence }) => ({ destination })`; the returned score (or `StagedJudgeError`) carries a receipt with every file hash and a whole-tree hash.
+- **Real, runnable with ZERO extra deps:** finsearchcomp (GitHub dataset + fixtures + LLM judge — the gate bench), hotpotqa + simpleqa + frames (HF/web QA + F1/LLM judge; `*_FIXTURES=1` offline), **ragbench**, **crag**, **nomiracl**, **open-rag-bench**, **t2-ragbench** (SOTA RAG/knowledge benchmarks with committed fixtures and deterministic answer/relevance judges; live mode reads explicit `*_DATA_FILE` JSON/JSONL exports), **aec-bench** (real GitHub task tree + fixtures; judge = the task's own `tests/verify.py` over python3 stdlib — **deterministic, graded per-field partial credit, no Docker, no LLM** → the candidate non-oracle correctable-middle-band bench for the open gate).
+- **Real code, needs an external harness/tools to run (fail loud with the exact install/Docker fix; never a fabricated score):** swe-bench + terminal-bench (`bench/.venv` + Docker), **commit0** (ISOLATED `bench/.venv-commit0` via `python3 -m venv bench/.venv-commit0 && bench/.venv-commit0/bin/pip install commit0 datasets` — its deps conflict with the shared `.venv`; override dir with `COMMIT0_VENV` — plus Docker; judge = official pytest harness, graded (passed+xfail)/total; the rollout prompt stages in-box (clones `commit-0/<repo>` @ `base_commit`, emits `git diff`); `COMMIT0_FIXTURES=1` for offline listing), **programbench** (`pip install programbench` + Docker on linux/amd64 + HF blobs; judge = official cleanroom eval, graded passed/total; `PROGRAMBENCH_FIXTURES=1` offline), **appworld** (`pip install appworld` + `appworld install` + `appworld download data`; judge = AppWorld's own `world.evaluate()`, graded passes/num_tests — NO committed fixture: task data exists only after `download data`, so loadTasks fails loud rather than fabricate a task), **dabstep** (`DABSTEP_DIR=/path/to/EnvCommons/DABStep` with the released `dataset.csv`, `splits/*.txt`, `files/*`, and `grade.py`; judge delegates to official `grade.py`; `DABSTEP_FIXTURES=1` only tests adapter plumbing and does not fabricate benchmark scores), **webarena-verified** (`WEBARENA_VERIFIED_DIR=/path/to/webarena-verified`; judge delegates to official `eval-tasks` over a run output directory), **tau2-bench** (`TAU2_BENCH_DIR=/path/to/tau2-bench`; judge recomputes tau2 trajectory rewards), **tau3-banking** (`TAU3_BENCH_DIR=/path/to/tau2-bench`; default domain `banking_knowledge`; judge recomputes tau trajectory rewards through the upstream tau3 package), **agentbench** DBBench subset (`AGENTBENCH_DIR=/path/to/AgentBench`; exact-match deterministic label judge), **bfcl** deterministic function-call subset (`BFCL_DIR=/path/to/gorilla/berkeley-function-call-leaderboard`; loads official BFCL JSONL + `possible_answer`; score = structured call/argument match, not the full BFCL leaderboard evaluator), **toollm** API-selection subset (`TOOLBENCH_DIR=/path/to/ToolBench`; score = recall of ToolBench `relevant APIs` labels, resolved only when the worker emits the requested structured JSON call list; official ToolEval pass rate remains LLM-judged/stochastic), **finresearchbench** (`FINRESEARCHBENCH_DATA_FILE=/path/to/export.jsonl`; rows must carry official `judge_system_prompt` + `judge_prompt_template`; no self-authored live judge), mind2web, cad-design + cadbench + cadgenbench (openscad/blender/build123d).
+- **goldArtifact:** aec-bench returns the task's real `golden_pass.md` (verify-judge works fully offline). commit0 / programbench / appworld return `undefined` — the oracle is a git ref / stripped source / engine-bundled solution, not a portable string; judge correctness is proven by a real solve through the harness, not a synthetic gold (documented + fail-loud, not a fake).
+- **Absent (not built):** swe-gym, swe-bench-multimodal, and the rest of the survey set.
+Every unbuilt/scaffold adapter fails LOUD (throws with the integration step) rather than faking a score — no silent zeros in any corpus. Offline fixture tests: `benchmarks/{aec-bench,commit0,programbench,appworld,rag-benchmarks}.test.mts` (`tsx --test`).
+
+## Pier candidate bridge
+
+`pier_agents.tangle_candidate:TangleCandidateAgent` is the reusable Pier custom-agent path for frozen Tangle candidates.
+`executePreparedPierCandidate()` is the only public entry point; its private staging step writes the runtime's execution-plan and materialization-receipt bytes verbatim.
+The Python bridge rechecks those bytes and their signed task, candidate, profile, repository, instruction, and workspace identities before launch.
+It rejects a raw candidate bundle, never projects an `AgentProfile`, and leaves task isolation, patch transfer, verification, retries, and result storage to Pier.
+
+The adapter fails the trial when any prepared identity drifts, when the task checkout or immutable OCI image differs from the signed identity, or when the candidate exits nonzero or exceeds its signed deadline.
+Candidate code runs as an unprivileged numeric user, while evaluator inputs and timeout evidence remain root-owned.
+It never accepts candidate-authored token, cost, or trace receipts; `executePreparedPierCandidate()` uses the runtime's atomic execution path and reconciles the protected `TraceStore` with the model-gateway ledger before returning a gradable receipt.
+The prepared object contains no credentials.
+The runtime passes model and trace bindings only to the trusted executor request.
+The Pier launcher inherits their values through its protected process environment and passes only `${NAME}` references on the command line, so credentials never enter prepared bytes, CLI arguments, or job files.
+The executor builds fresh task, candidate, and profile trees from the request's exact verified file bytes; prepared staging directories are never launch authority.
+Isolated memory and knowledge-bearing candidates currently fail closed until Pier has executor-owned mount and after-state capture.
+The signed wall deadline is a hard stop: the runtime aborts, Pier kills the process tree, and the executor acknowledges process and container death.
+The signed tool-step count is a post-run validity check over protected traces, not a pre-tool stop; generic black-box Pier processes cannot honestly prevent step N+1.
+The executable zero-model fixture is `fixtures/pier-agent/`; run it against the R360 Pier checkout with `PIER_REPO=/path/to/pier pnpm verify:pier`.
+That command runs a no-change candidate that must score 0/1, proves a fresh evaluator process can kill a persisted child and remove its real Docker container, and runs a known-good candidate that must score 1/1.
+It then checks that each official result and exact task patch is bound into its own runtime receipt with zero model usage.
+
+For a real frozen candidate, use `FilePierCandidateTrialController`, append `agentArgs` and `attemptArgs`, and pass each executor-only `evaluatorEnv` entry through Pier's evaluator-owned environment mechanism.
+The launch callback's second argument carries the signed runtime `request`, protected `traceStore`, cancellation `signal`, and absolute `deadlineAtMs`; production launchers must use that context rather than reconstructing it.
+The controller sends secrets to its supervisor over a pipe, while its durable files contain only process and Docker-project identities:
+The supervisor never inherits `process.env`; non-default Docker connections use a stable `dockerConnection.id` plus the exact environment injected into both Pier and cleanup.
+Fresh recovery workers must reconstruct that same named connection; `terminate-pier-trial.mts` selects only the comma-delimited variables named by `PIER_DOCKER_ENV_NAMES` when `PIER_DOCKER_CONNECTION_ID` is set.
+`jobName` must be unique per prepared execution; the controller atomically reserves that job directory so recovery can remove only containers owned by that execution.
+
+```ts
+const controller = new FilePierCandidateTrialController({
+  directory: '/var/lib/tangle/pier-control',
+  launch: (staged, { request }) => {
+    const jobName = request.executionId
+    return {
+      command: 'uv',
+      args: ['run', 'pier', 'run', ...staged.agentArgs, ...staged.attemptArgs],
+      cwd: pierCheckout,
+      env: { ...evaluatorEnvironment, ...staged.evaluatorEnv },
+      jobsDirectory,
+      jobName,
+      readResult: () => readOfficialPierResult(jobsDirectory, jobName),
+    }
+  },
+})
+
+const result = await executePreparedPierCandidate({
+  prepared,
+  directory: '/sealed/candidate',
+  pierVersion: '0.3.0',
+  traceStore,
+  claimStore,
+  outputArtifacts,
+  grader,
+  controller,
+})
+```
+
+The controller's result resolves only after normal process/container cleanup.
+On a signed deadline, external abort, or recovery by another evaluator process, the adapter waits for `terminateAndWait()` to acknowledge both process exit and container removal before it returns control to the runtime.
+
+One prepared execution always maps to one Pier attempt (`--n-attempts 1 --max-retries 0`).
+Production callers pass a long-lived `FileAgentCandidateExecutionClaimStore`; an in-memory claim store is test-only and cannot prevent a second process from replaying the same attempt.
+Any allowed pre-model infrastructure retry is a new prepared execution with its own counted attempt identity.
+
+## Is it runnable RIGHT NOW? (verify the map, don't trust it blindly)
+```
+ls src/*.mts src/*.ts          # the real tool list (each its own main — source of truth)
+tsx src/gate.test.mts # offline plumbing test (no creds)
+```
+Creds: the router/sandbox paths read `ROUTER_KEY`/`SANDBOX_KEY` (+ `ROUTER_BASE`/`SANDBOX_BASE_URL`)
+from the environment. Source them from the operator's private secret store (documented in the
+global agent config, NOT here — this repo is public) into the run process; never print them.
+NOT needed for the offline selector gate, the hotpotqa/swe-bench deterministic judges, or
+RESEARCH=1 local-opencode rollouts — if unset, those paths are cred-blocked, not code-blocked.
+
+## Durable next step (so this stops drifting)
+The surviving tools are standalone `.mts` mains (no `run.ts` registry). Next: a manifest test that
+asserts every committed tool + package.json script is named on this page, so the map can't silently
+drift from the code again.

package/README.md

@@ -1,6 +1,6 @@
 # agent-runtime-bench
 
-Private experiment workspace nested in agent-runtime; decoupled from its build/lint/release (the package builds `src/`, lints `src tests examples` — `bench/` is none of those).
+Published as `@tangle-network/agent-bench`, with independent CI and release checks for its TypeScript and Python surfaces.
 
 **Read [`bench/HARNESS.md`](./HARNESS.md) FIRST.** It is the one maintained map: the commands, the `rollout → corpus → selector → CI → gate` data flow, the canonical-suite table, the wired/needs-creds/scaffolded matrix, and the gate one-liners — kept verified against source.
 
@@ -13,3 +13,28 @@ pnpm install                                              # tsx + link parent
 ```
 
 The judge needs only Docker; workers need a model key (Tangle router `TANGLE_API_KEY`, or a direct provider).
+
+Retain every official per-test log and report before the temporary evaluator directory is removed:
+
+```ts
+const adapter = createSweBenchAdapter({
+  captureEvaluatorArtifacts: ({ taskId, attemptSequence }) => ({
+    destination: path.join(runDirectory, taskId, String(attemptSequence)),
+  }),
+})
+const score = await adapter.judge(task, patch)
+console.log(score.judgeArtifacts?.manifestPath)
+```
+
+Each destination contains the untouched evaluator tree under `evaluator/`, raw `stdout.bin` and `stderr.bin` under `process/`, and `receipt.json` with per-file SHA-256 values plus a whole-tree SHA-256.
+The destination must be unique and absent; an existing path fails loud instead of overwriting evidence.
+Failed evaluators throw `StagedJudgeError` with the same `judgeArtifacts` receipt after retaining partial logs.
+
+## Pier custom candidates
+
+The package executes a branded `PreparedAgentCandidateExecution` from `@tangle-network/agent-runtime` through one atomic API and ships `pier_agents.tangle_candidate:TangleCandidateAgent` as its thin Pier transport.
+The executor recreates every input from runtime-verified file bytes and reveals model credentials only inside the claimed execution callback.
+Pier owns the task container and verifier; protected model usage and traces stay in `@tangle-network/agent-eval` and are finalized by the shared runtime.
+`FilePierCandidateTrialController` atomically reserves a unique Pier job, then persists the supervisor PID, process-session identity, and that job's exact Docker projects so a fresh evaluator process can stop and remove an abandoned trial.
+Run `PIER_REPO=/path/to/pier pnpm verify:pier` for the zero-model failure/pass and fresh-process recovery proof, and see `HARNESS.md` for the exact invocation and failure contract.
+From an installed npm package, expose the shipped Python module with `export PYTHONPATH="$(npm root)/@tangle-network/agent-bench${PYTHONPATH:+:$PYTHONPATH}"` before invoking Pier.

package/fixtures/aec-bench.json

@@ -0,0 +1,18 @@
+[
+  {
+    "id": "electrical/catenary-sag",
+    "discipline": "electrical",
+    "instruction": "You are a senior electrical engineer specializing in overhead transmission lines.\n\n## Problem\n\nCalculate the conductor sag and total conductor length for a single level span\nusing catenary equations. The conductor hangs between two towers at equal height.\n\n## Given\n\n| Parameter | Value | Unit |\n|-----------|-------|------|\n| Span length | 250 | m |\n| Conductor unit weight | 7.58 | N/m |\n| Horizontal tension | 64500 | N |\n\n## Required\n\nCalculate the following:\n- Catenary constant (m)\n- Maximum sag at mid-span (m)\n- Total conductor length between attachment points (m)\n\n## Applicable Standards\n\n- IEC 60826 \u2014 Design criteria of overhead transmission lines (for reference)\n- AS/NZS 7000 \u2014 Overhead line design (for reference)\n\nExact clause citation is optional.\n\n## Constraints\n\n- No internet access is available. Work from engineering knowledge only.\n- Use catenary (not parabolic) equations for this calculation.\n\n## Output Format\n\nShow your step-by-step working in Markdown, including the formulas you use and\nintermediate calculations. At the end of your solution, include a JSON block\nwith your final answers in exactly this format:\n\n```json\n{\n  \"catenary_constant_m\": <numeric_value>,\n  \"sag_m\": <numeric_value>,\n  \"conductor_length_m\": <numeric_value>\n}\n```\n\nWrite your complete solution to `/workspace/output.md`.\n",
+    "task_toml": "version = \"1.0\"\n\n[metadata]\ndifficulty = \"easy\"\ncategory = \"reasoning\"\ntags = [\"electrical\", \"transmission-lines\", \"deterministic\", \"IEC-60826\"]\n\n[agent]\ntimeout_sec = 600.0\n\n[verifier]\ntimeout_sec = 120.0\n\n[environment]\nextensions = []\nbuild_timeout_sec = 600.0\ncpus = 1\nmemory_mb = 2048\nstorage_mb = 5120\nallow_internet = true\n",
+    "verify_py": "# ABOUTME: Verifier for catenary-sag task.\n# ABOUTME: Computes ground truth from catenary formulas and scores agent output.\n\nimport argparse\nimport json\nimport math\nimport re\nfrom pathlib import Path\n\nDEFAULT_OUTPUT_FILE = Path(\"/workspace/output.md\")\nDEFAULT_REWARD_FILE = Path(\"/logs/verifier/reward.json\")\n\n\ndef write_reward(reward: float, details: dict[str, float], path: Path) -> None:\n    \"\"\"Write reward JSON for Harbor, plus per-field details to a sibling file.\"\"\"\n    path.parent.mkdir(parents=True, exist_ok=True)\n    path.write_text(json.dumps({\"reward\": round(reward, 2)}))\n    details_path = path.parent / \"details.json\"\n    details_path.write_text(json.dumps(details))\n\n\ndef compute_ground_truth() -> dict[str, float]:\n    \"\"\"Compute expected answers from catenary formulas.\"\"\"\n    H = 64500.0  # horizontal tension (N)\n    w = 7.58  # conductor unit weight (N/m)\n    L = 250.0  # span length (m)\n\n    c = H / w\n    sag = c * (math.cosh(L / (2 * c)) - 1)\n    length = 2 * c * math.sinh(L / (2 * c))\n\n    return {\n        \"catenary_constant_m\": c,\n        \"sag_m\": sag,\n        \"conductor_length_m\": length,\n    }\n\n\ndef extract_json_block(text: str) -> dict | None:\n    \"\"\"Extract the last fenced JSON block from Markdown text.\"\"\"\n    pattern = r\"```json\\s*\\n(.*?)\\n\\s*```\"\n    matches = re.findall(pattern, text, re.DOTALL)\n    if not matches:\n        return None\n    try:\n        return json.loads(matches[-1])\n    except json.JSONDecodeError:\n        return None\n\n\ndef score_field(expected: float, actual_val, rel_tol: float) -> float:\n    \"\"\"Score a single field: 1.0 if within tolerance, 0.0 otherwise.\"\"\"\n    if actual_val is None:\n        return 0.0\n    try:\n        actual_float = float(actual_val)\n    except (TypeError, ValueError):\n        return 0.0\n    if math.isclose(actual_float, expected, rel_tol=rel_tol):\n        return 1.0\n    return 0.0\n\n\ndef score_answers(\n    expected: dict[str, float], actual: dict | None\n) -> tuple[float, dict[str, float]]:\n    \"\"\"Score each field and return overall reward + per-field details.\"\"\"\n    tolerances = {\n        \"catenary_constant_m\": 0.02,\n        \"sag_m\": 0.02,\n        \"conductor_length_m\": 0.02,\n    }\n\n    if not actual:\n        details = {k: 0.0 for k in expected}\n        return 0.0, details\n\n    details = {}\n    for key, exp_val in expected.items():\n        act_val = actual.get(key)\n        tol = tolerances.get(key, 0.02)\n        details[key] = score_field(exp_val, act_val, tol)\n\n    total = len(expected)\n    reward = sum(details.values()) / total if total > 0 else 0.0\n    return round(reward, 2), details\n\n\ndef main() -> None:\n    parser = argparse.ArgumentParser(description=\"Verify catenary-sag output\")\n    parser.add_argument(\"--input\", type=Path, default=DEFAULT_OUTPUT_FILE)\n    parser.add_argument(\"--output\", type=Path, default=DEFAULT_REWARD_FILE)\n    args = parser.parse_args()\n\n    try:\n        if not args.input.exists() or args.input.stat().st_size == 0:\n            details = {k: 0.0 for k in compute_ground_truth()}\n            write_reward(0.0, details, args.output)\n            return\n\n        text = args.input.read_text()\n        expected = compute_ground_truth()\n        actual = extract_json_block(text)\n        reward, details = score_answers(expected, actual)\n        write_reward(reward, details, args.output)\n    except Exception:\n        details = {k: 0.0 for k in compute_ground_truth()}\n        write_reward(0.0, details, args.output)\n\n\nif __name__ == \"__main__\":\n    main()\n",
+    "golden_pass_md": "# Catenary Sag Calculation\n\n## Given\n- Span length: 250 m\n- Conductor unit weight: 7.58 N/m\n- Horizontal tension: 64500 N\n\n## Step 1: Catenary Constant\n\nThe catenary constant is defined as:\nc = H / w = 64500 / 7.58 = 8509.2348 m\n\n## Step 2: Maximum Sag\n\nUsing the catenary equation:\nsag = c * (cosh(L / (2c)) - 1)\nsag = 8509.2348 * (cosh(250 / 17018.4696) - 1)\nsag = 8509.2348 * (cosh(0.01469) - 1)\nsag = 8509.2348 * 0.000107895\nsag = 0.9181 m\n\n## Step 3: Conductor Length\n\nlength = 2c * sinh(L / (2c))\nlength = 2 * 8509.2348 * sinh(0.01469)\nlength = 17018.4696 * 0.014691\nlength = 250.0090 m\n\n```json\n{\n  \"catenary_constant_m\": 8509.2348,\n  \"sag_m\": 0.9181,\n  \"conductor_length_m\": 250.009\n}\n```\n"
+  },
+  {
+    "id": "electrical/voltage-drop",
+    "discipline": "electrical",
+    "instruction": "You are a senior electrical engineer specializing in building services.\n\n## Problem\n\nCalculate the voltage drop for a three-phase cable circuit using the impedance\nmethod, and determine whether it complies with the maximum allowable voltage\ndrop limit.\n\n## Given\n\n| Parameter | Value | Unit |\n|-----------|-------|------|\n| Circuit type | Three-phase | - |\n| Load current | 45 | A |\n| Cable length (one way) | 80 | m |\n| Cable resistance (R) | 0.524 | ohm/km |\n| Cable reactance (X) | 0.08 | ohm/km |\n| Power factor (cos phi) | 0.85 | - |\n| System voltage (line-to-line) | 400 | V |\n| Maximum allowable voltage drop | 5 | % |\n\n## Required\n\nCalculate the following:\n- Voltage drop (V)\n- Voltage drop as a percentage of system voltage (%)\n- Compliance with the 5% limit (1 if compliant, 0 if not)\n\n## Applicable Standards\n\n- AS/NZS 3008.1 \u2014 Electrical installations: selection of cables (for reference)\n- AS/NZS 3000 \u2014 Wiring rules (for reference)\n\nExact clause citation is optional.\n\n## Constraints\n\n- No internet access is available. Work from engineering knowledge only.\n- Use the impedance method (not simplified resistance-only method).\n\n## Output Format\n\nShow your step-by-step working in Markdown, including the formulas you use and\nintermediate calculations. At the end of your solution, include a JSON block\nwith your final answers in exactly this format:\n\n```json\n{\n  \"voltage_drop_v\": <numeric_value>,\n  \"voltage_drop_pct\": <numeric_value>,\n  \"compliance\": <1_or_0>\n}\n```\n\nWrite your complete solution to `/workspace/output.md`.\n",
+    "task_toml": "version = \"1.0\"\n\n[metadata]\ndifficulty = \"easy\"\ncategory = \"reasoning\"\ntags = [\"electrical\", \"buildings-electrical\", \"deterministic\", \"AS-NZS-3008\"]\n\n[agent]\ntimeout_sec = 600.0\n\n[verifier]\ntimeout_sec = 120.0\n\n[environment]\nextensions = []\nbuild_timeout_sec = 600.0\ncpus = 1\nmemory_mb = 2048\nstorage_mb = 5120\nallow_internet = true\n",
+    "verify_py": "# ABOUTME: Verifier for voltage-drop task.\n# ABOUTME: Computes ground truth from impedance method and scores agent output.\n\nimport argparse\nimport json\nimport math\nimport re\nfrom pathlib import Path\n\nDEFAULT_OUTPUT_FILE = Path(\"/workspace/output.md\")\nDEFAULT_REWARD_FILE = Path(\"/logs/verifier/reward.json\")\n\n\ndef write_reward(reward: float, details: dict[str, float], path: Path) -> None:\n    \"\"\"Write reward JSON for Harbor, plus per-field details to a sibling file.\"\"\"\n    path.parent.mkdir(parents=True, exist_ok=True)\n    path.write_text(json.dumps({\"reward\": round(reward, 2)}))\n    details_path = path.parent / \"details.json\"\n    details_path.write_text(json.dumps(details))\n\n\ndef compute_ground_truth() -> dict[str, float]:\n    \"\"\"Compute expected answers from three-phase voltage drop formula.\"\"\"\n    current = 45.0  # load current (A)\n    L = 80.0  # cable length (m)\n    R = 0.524  # cable resistance (ohm/km)\n    X = 0.08  # cable reactance (ohm/km)\n    pf = 0.85  # power factor\n    V = 400.0  # system voltage (V)\n\n    sin_phi = math.sin(math.acos(pf))\n    Vd = math.sqrt(3) * current * L * (R * pf + X * sin_phi) / 1000\n    Vd_pct = Vd / V * 100\n    compliance = 1.0 if Vd_pct <= 5.0 else 0.0\n\n    return {\n        \"voltage_drop_v\": Vd,\n        \"voltage_drop_pct\": Vd_pct,\n        \"compliance\": compliance,\n    }\n\n\ndef extract_json_block(text: str) -> dict | None:\n    \"\"\"Extract the last fenced JSON block from Markdown text.\"\"\"\n    pattern = r\"```json\\s*\\n(.*?)\\n\\s*```\"\n    matches = re.findall(pattern, text, re.DOTALL)\n    if not matches:\n        return None\n    try:\n        return json.loads(matches[-1])\n    except json.JSONDecodeError:\n        return None\n\n\ndef score_field(\n    expected: float, actual_val, rel_tol: float, exact: bool = False\n) -> float:\n    \"\"\"Score a single field: 1.0 if within tolerance, 0.0 otherwise.\"\"\"\n    if actual_val is None:\n        return 0.0\n    try:\n        actual_float = float(actual_val)\n    except (TypeError, ValueError):\n        return 0.0\n    if exact:\n        return 1.0 if actual_float == expected else 0.0\n    if math.isclose(actual_float, expected, rel_tol=rel_tol):\n        return 1.0\n    return 0.0\n\n\ndef score_answers(\n    expected: dict[str, float], actual: dict | None\n) -> tuple[float, dict[str, float]]:\n    \"\"\"Score each field and return overall reward + per-field details.\"\"\"\n    tolerances = {\n        \"voltage_drop_v\": {\"rel_tol\": 0.03, \"exact\": False},\n        \"voltage_drop_pct\": {\"rel_tol\": 0.03, \"exact\": False},\n        \"compliance\": {\"rel_tol\": 0.0, \"exact\": True},\n    }\n\n    if not actual:\n        details = {k: 0.0 for k in expected}\n        return 0.0, details\n\n    details = {}\n    for key, exp_val in expected.items():\n        act_val = actual.get(key)\n        cfg = tolerances.get(key, {\"rel_tol\": 0.03, \"exact\": False})\n        details[key] = score_field(exp_val, act_val, cfg[\"rel_tol\"], cfg[\"exact\"])\n\n    total = len(expected)\n    reward = sum(details.values()) / total if total > 0 else 0.0\n    return round(reward, 2), details\n\n\ndef main() -> None:\n    parser = argparse.ArgumentParser(description=\"Verify voltage-drop output\")\n    parser.add_argument(\"--input\", type=Path, default=DEFAULT_OUTPUT_FILE)\n    parser.add_argument(\"--output\", type=Path, default=DEFAULT_REWARD_FILE)\n    args = parser.parse_args()\n\n    try:\n        if not args.input.exists() or args.input.stat().st_size == 0:\n            details = {k: 0.0 for k in compute_ground_truth()}\n            write_reward(0.0, details, args.output)\n            return\n\n        text = args.input.read_text()\n        expected = compute_ground_truth()\n        actual = extract_json_block(text)\n        reward, details = score_answers(expected, actual)\n        write_reward(reward, details, args.output)\n    except Exception:\n        details = {k: 0.0 for k in compute_ground_truth()}\n        write_reward(0.0, details, args.output)\n\n\nif __name__ == \"__main__\":\n    main()\n",
+    "golden_pass_md": "# Voltage Drop Calculation\n\n## Given\n- Circuit type: Three-phase\n- Load current: 45 A\n- Cable length: 80 m (one way)\n- Cable resistance: 0.524 ohm/km\n- Cable reactance: 0.08 ohm/km\n- Power factor: 0.85\n- System voltage: 400 V\n\n## Step 1: Calculate sin(phi)\ncos(phi) = 0.85\nsin(phi) = sin(acos(0.85)) = 0.5268\n\n## Step 2: Voltage Drop\nUsing the three-phase impedance method:\nVd = sqrt(3) x I x L x (R x cos(phi) + X x sin(phi)) / 1000\nVd = 1.7321 x 45 x 80 x (0.524 x 0.85 + 0.08 x 0.5268) / 1000\nVd = 1.7321 x 45 x 80 x (0.4454 + 0.0421) / 1000\nVd = 1.7321 x 45 x 80 x 0.4876 / 1000\nVd = 3.0400 V\n\n## Step 3: Percentage\nVd% = 3.0400 / 400 x 100 = 0.7600%\n\n## Step 4: Compliance\n0.76% is less than 5% limit, so the cable complies.\n\n```json\n{\n  \"voltage_drop_v\": 3.04,\n  \"voltage_drop_pct\": 0.76,\n  \"compliance\": 1\n}\n```\n"
+  }
+]

package/fixtures/agentbench-dbbench.json

@@ -0,0 +1,22 @@
+[
+  {
+    "description": "What are the Notes when the Method is decision?",
+    "label": ["Women +60kg Bronze"],
+    "table": {
+      "table_name": "Jiu-Jitsu Championships Results",
+      "table_info": {
+        "columns": [
+          { "name": "Result", "type": "TEXT" },
+          { "name": "Opponent", "type": "TEXT" },
+          { "name": "Method", "type": "TEXT" },
+          { "name": "Event", "type": "TEXT" },
+          { "name": "Notes", "type": "TEXT" }
+        ],
+        "rows": [
+          ["Win", "Rosangela Conceicao", "Decision", "2009 ADCC Submission Wrestling World Championships", "Women +60kg Bronze"],
+          ["Loss", "Penny Thomas", "Points (-2 x -1)", "2009 ADCC Submission Wrestling World Championships", "Women +60kg Semifinal"]
+        ]
+      }
+    }
+  }
+]

package/fixtures/bfcl.json

@@ -0,0 +1,45 @@
+{
+  "category": "BFCL_fixture_simple_python",
+  "rows": [
+    {
+      "id": "simple_python_fixture_0",
+      "question": [
+        [
+          {
+            "role": "user",
+            "content": "Find the area of a triangle with a base of 10 units and height of 5 units."
+          }
+        ]
+      ],
+      "function": [
+        {
+          "name": "calculate_triangle_area",
+          "description": "Calculate the area of a triangle given its base and height.",
+          "parameters": {
+            "type": "dict",
+            "properties": {
+              "base": { "type": "integer" },
+              "height": { "type": "integer" },
+              "unit": { "type": "string" }
+            },
+            "required": ["base", "height"]
+          }
+        }
+      ]
+    }
+  ],
+  "answers": [
+    {
+      "id": "simple_python_fixture_0",
+      "ground_truth": [
+        {
+          "calculate_triangle_area": {
+            "base": [10],
+            "height": [5],
+            "unit": ["units", ""]
+          }
+        }
+      ]
+    }
+  ]
+}

package/fixtures/commit0.json

@@ -0,0 +1,72 @@
+[
+  {
+    "instance_id": "commit-0/wcwidth",
+    "repo": "commit-0/wcwidth",
+    "original_repo": "jquast/wcwidth",
+    "base_commit": "0d0054189bdb0fc7b9de0456fc3cee2b67a6ceba",
+    "reference_commit": "36a625179ed2675287fe6b61c2ad319406449e60",
+    "setup": {
+      "install": "pip install -e .",
+      "packages": null,
+      "pip_packages": null,
+      "pre_install": null,
+      "python": "3.10",
+      "specification": "https://wcwidth.readthedocs.io/en/stable/"
+    },
+    "test": {
+      "test_cmd": "pytest",
+      "test_dir": "tests/"
+    },
+    "src_dir": "wcwidth/"
+  },
+  {
+    "instance_id": "commit-0/deprecated",
+    "repo": "commit-0/deprecated",
+    "original_repo": "laurent-laporte-pro/deprecated",
+    "base_commit": "b7e2114c046abb489e4e23ab9f829778b076650d",
+    "reference_commit": "378636d8a1e3d62d5894329c57f44ba0030c2dbd",
+    "setup": {
+      "install": "pip install -e .[dev]",
+      "packages": null,
+      "pip_packages": null,
+      "pre_install": null,
+      "python": "3.10",
+      "specification": "https://deprecated.readthedocs.io/en/latest/"
+    },
+    "test": {
+      "test_cmd": "pytest",
+      "test_dir": "tests/"
+    },
+    "src_dir": "deprecated/"
+  },
+  {
+    "instance_id": "commit-0/tinydb",
+    "repo": "commit-0/tinydb",
+    "original_repo": "msiemens/tinydb",
+    "base_commit": "ed761a72c8c1e1cb24ca4dbcc089f35c5264d357",
+    "reference_commit": "429b27a513f0ad301632379e55667dd99865fb16",
+    "setup": {
+      "install": "pip install -e .",
+      "packages": null,
+      "pip_packages": [
+        "pytest>=7.2.0,<8.0.0",
+        "pytest-pycodestyle>=2.3.1,<3.0.0",
+        "pytest-cov>=4.0.0,<5.0.0",
+        "pycodestyle>=2.10.0,<3.0.0",
+        "sphinx>=7.0.0,<8.0.0",
+        "coveralls>=3.3.1,<4.0.0",
+        "pyyaml>=6.0,<7.0",
+        "types-PyYAML>=6.0.0,<7.0.0",
+        "typing-extensions>=4.4.0,<5.0.0"
+      ],
+      "pre_install": null,
+      "python": "3.10",
+      "specification": "https://tinydb.readthedocs.io/_/downloads/en/v4.8.0/pdf/"
+    },
+    "test": {
+      "test_cmd": "pytest",
+      "test_dir": "tests/"
+    },
+    "src_dir": "tinydb/"
+  }
+]

package/fixtures/crag.json

@@ -0,0 +1,10 @@
+[
+  {
+    "id": "crag-fixture-0",
+    "domain": "finance",
+    "question_type": "simple",
+    "static_or_dynamic": "slow-changing",
+    "query": "What was Contoso's reported revenue in 2025?",
+    "answer": ["$12.4 billion", "12.4 billion dollars"]
+  }
+]

package/fixtures/dabstep.json

@@ -0,0 +1,22 @@
+[
+  {
+    "task_id": 1,
+    "instructions": "Using the payment files, answer this calibration task with the exact integer 42.",
+    "all_golds_by_task": [
+      {
+        "kind": "number",
+        "value": 42.0
+      }
+    ]
+  },
+  {
+    "task_id": 2,
+    "instructions": "Using the payment files, answer this calibration task with the card scheme nexpay.",
+    "all_golds_by_task": [
+      {
+        "kind": "scheme",
+        "value": "nexpay"
+      }
+    ]
+  }
+]