@tangle-network/agent-bench 0.7.1 → 0.8.1

package/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 0.8.1
+
+- Consume Eval 0.144.8 and Knowledge 7.1.3 so Bench uses the same published dependency family as Runtime 0.131.6.
+
+## 0.8.0
+
+- Consume Runtime 0.131.0, Eval 0.144.6, Interface 0.46.1, Knowledge 7.1.2, and Sandbox 0.19.4 as one compatible dependency set.
+
+## 0.7.2
+
+- Consume Runtime 0.129.0, Eval 0.144.4, Interface 0.43.1, Knowledge 7.0.11, and Sandbox 0.19.1 so benchmark model calls use the exact-profile execution boundary and the released optimizer callback contract without loading duplicate agent contracts.
+
 ## 0.7.1
 
 - Consume Runtime 0.126.0 with Eval 0.143.0 and Knowledge 7.0.8, so campaign cost remains observed, estimated, or explicitly uncaptured across the complete benchmark dependency tree.

package/HARNESS.md

@@ -34,7 +34,7 @@ probe (`bench/src/humaneval-gate.mts`) uses exactly that degenerate shape — it
 directly and does **not** route through `AgentProfile` / the sandbox / the keystone — so its numbers
 are the **no-self-correction lower bound** on the selector, distinct from the rollout-based keystone
 gate above. Bridge it to the product by running the same arms with real rollouts (an `AgentProfile`
-through `runLoop`), dialing `maxTurns`.
+through `runAgentRounds`), dialing `maxTurns`.
 
 Two things to keep straight: today's judges grade a single
 *correctness* scalar (the multi-objective vector is the open contract, architecture.md §6), and every
@@ -259,7 +259,7 @@ SWE-bench callers that need a hard evaluation deadline construct `createSweBench
 Callers that must inspect the exact per-task image after scoring pass `cacheLevel: 'instance'`; the default remains `'env'`.
 Callers retain the complete official evaluator tree plus raw process output with `captureEvaluatorArtifacts: ({ taskId, attemptSequence }) => ({ destination })`; the returned score (or `StagedJudgeError`) carries a receipt with every file hash and a whole-tree hash.
 - **Real, runnable with ZERO extra deps:** finsearchcomp (GitHub dataset + fixtures + LLM judge — the gate bench), hotpotqa + simpleqa + frames (HF/web QA + F1/LLM judge; `*_FIXTURES=1` offline), **ragbench**, **crag**, **nomiracl**, **open-rag-bench**, **t2-ragbench** (SOTA RAG/knowledge benchmarks with committed fixtures and deterministic answer/relevance judges; live mode reads explicit `*_DATA_FILE` JSON/JSONL exports), **aec-bench** (real GitHub task tree + fixtures; judge = the task's own `tests/verify.py` over python3 stdlib — **deterministic, graded per-field partial credit, no Docker, no LLM** → the candidate non-oracle correctable-middle-band bench for the open gate).
-- **Real code, needs an external harness/tools to run (fail loud with the exact install/Docker fix; never a fabricated score):** swe-bench + terminal-bench (`bench/.venv` + Docker), **commit0** (ISOLATED `bench/.venv-commit0` via `python3 -m venv bench/.venv-commit0 && bench/.venv-commit0/bin/pip install commit0 datasets` — its deps conflict with the shared `.venv`; override dir with `COMMIT0_VENV` — plus Docker; judge = official pytest harness, graded (passed+xfail)/total; the rollout prompt stages in-box (clones `commit-0/<repo>` @ `base_commit`, emits `git diff`); `COMMIT0_FIXTURES=1` for offline listing), **programbench** (`pip install programbench` + Docker on linux/amd64 + HF blobs; judge = official cleanroom eval, graded passed/total; `PROGRAMBENCH_FIXTURES=1` offline), **appworld** (`pip install appworld` + `appworld install` + `appworld download data`; judge = AppWorld's own `world.evaluate()`, graded passes/num_tests — NO committed fixture: task data exists only after `download data`, so loadTasks fails loud rather than fabricate a task), **dabstep** (`DABSTEP_DIR=/path/to/EnvCommons/DABStep` with the released `dataset.csv`, `splits/*.txt`, `files/*`, and `grade.py`; judge delegates to official `grade.py`; `DABSTEP_FIXTURES=1` only tests adapter plumbing and does not fabricate benchmark scores), **webarena-verified** (`WEBARENA_VERIFIED_DIR=/path/to/webarena-verified`; judge delegates to official `eval-tasks` over a run output directory), **tau2-bench** (`TAU2_BENCH_DIR=/path/to/tau2-bench`; judge recomputes tau2 trajectory rewards), **tau3-banking** (`TAU3_BENCH_DIR=/path/to/tau2-bench`; default domain `banking_knowledge`; judge recomputes tau trajectory rewards through the upstream tau3 package), **agentbench** DBBench subset (`AGENTBENCH_DIR=/path/to/AgentBench`; exact-match deterministic label judge), **bfcl** deterministic function-call subset (`BFCL_DIR=/path/to/gorilla/berkeley-function-call-leaderboard`; loads official BFCL JSONL + `possible_answer`; score = structured call/argument match, not the full BFCL leaderboard evaluator), **toollm** API-selection subset (`TOOLBENCH_DIR=/path/to/ToolBench`; score = recall of ToolBench `relevant APIs` labels, resolved only when the worker emits the requested structured JSON call list; official ToolEval pass rate remains LLM-judged/stochastic), **finresearchbench** (`FINRESEARCHBENCH_DATA_FILE=/path/to/export.jsonl`; rows must carry official `judge_system_prompt` + `judge_prompt_template`; no self-authored live judge), mind2web, cad-design + cadbench + cadgenbench (openscad/blender/build123d).
+- **Real code, needs an external harness/tools to run (fail loud with the exact install/Docker fix; never a fabricated score):** swe-bench + terminal-bench (`bench/.venv` + Docker), **commit0** (ISOLATED `bench/.venv-commit0` via `python3 -m venv bench/.venv-commit0 && bench/.venv-commit0/bin/pip install commit0 datasets` — its deps conflict with the shared `.venv`; override dir with `COMMIT0_VENV` — plus Docker; judge = official pytest harness, graded (passed+xfail)/total; the rollout prompt stages in-box (clones `commit-0/<repo>` @ `base_commit`, emits `git diff`); `COMMIT0_FIXTURES=1` for offline listing), **programbench** (`pip install programbench` + Docker on linux/amd64 + HF blobs; judge = official cleanroom eval, graded passed/total; `PROGRAMBENCH_FIXTURES=1` offline), **appworld** (`pip install appworld` + `appworld install` + `appworld download data`; judge = AppWorld's own `world.evaluate()`, graded passes/num_tests — NO committed fixture: task data exists only after `download data`, so loadTasks fails loud rather than fabricate a task), **dabstep** (`DABSTEP_DIR=/path/to/EnvCommons/DABStep` with the released `dataset.csv`, `splits/*.txt`, `files/*`, and `grade.py`; judge delegates to official `grade.py`; `DABSTEP_FIXTURES=1` only tests adapter plumbing and does not fabricate benchmark scores), **webarena-verified** (`WEBARENA_VERIFIED_DIR=/path/to/webarena-verified`; judge delegates to official `eval-tasks` over a run output directory), **tau2-bench** (`TAU2_BENCH_DIR=/path/to/tau2-bench`; judge recomputes tau2 trajectory rewards), **tau3-banking** (`TAU3_BENCH_DIR=/path/to/tau2-bench`; default domain `banking_knowledge`; judge recomputes tau trajectory rewards through the upstream tau3 package), **agentbench** DBBench subset (`AGENTBENCH_DIR=/path/to/AgentBench`; exact-match deterministic label judge), **bfcl** deterministic function-call subset (`BFCL_DIR=/path/to/gorilla/berkeley-function-call-leaderboard`; loads official BFCL JSONL + `possible_answer`; score = structured call/argument match, not the full BFCL leaderboard evaluator), **toollm** API-selection subset (`TOOLBENCH_DIR=/path/to/ToolBench`; score = recall of ToolBench `relevant APIs` labels, resolved only when the worker emits the requested structured JSON call list; official ToolEval pass rate remains LLM-judged/stochastic), **finresearchbench** (`FINRESEARCHBENCH_DATA_FILE=/path/to/export.jsonl`; rows must carry official `judge_system_prompt` + `judge_prompt_template`; no self-authored live judge), mind2web, cad-design + cadbench + cadgenbench (openscad/blender/build123d), **mcad** (openscad + xvfb-run; 10 dimensioned mechanical parts adopted from the MIT text-to-cad benchmark set; judge = OpenSCAD compile then pure-TS mesh assertions — bbox, closed-form volume bands, body count, and ray-parity point-in-solid probes at the spec's pinned hole coordinates; all 10 tasks carry a verified gold, and the must-reject direction is tested: one missing bore of four fails on its named probe), **mcad-cq** (the SAME 10 parts, tasks and spec assertions, but the worker writes a Python **CadQuery** script that must export BOTH `part.step` and an ASCII `part.stl` — which closes the format deviation `mcad-tasks.ts` documents, since upstream asks for STEP and OpenSCAD cannot emit it; the geometry engine is imported from `mcad-bench.ts`, not copied, and STEP delivery is one EXTRA scored check `stepEmitted` = `part.step` starts with `ISO-10303-21`; 120 s deadline per script, all 10 tasks carry a verified CadQuery gold at score 1.0, and four must-rejects are tested — one bore deleted, no STL, a binary STL, and the STEP export dropped. Needs an ISOLATED `bench/.venv-cadquery`: `cd bench && uv venv --python 3.12 .venv-cadquery && uv pip install --python .venv-cadquery/bin/python 'cadquery==2.4.0' 'numpy<2'` — both pins are load-bearing, cadquery-ocp 7.7.2 has no cp313 wheel and cadquery 2.4.0's pinned nptyping 2.0.1 reads the numpy-1.x-only `np.bool8`; override with the absolute path `MCAD_CQ_PYTHON`. The judge RUNS the worker's Python on the host, same trust level as `mcad` running arbitrary OpenSCAD).
 - **goldArtifact:** aec-bench returns the task's real `golden_pass.md` (verify-judge works fully offline). commit0 / programbench / appworld return `undefined` — the oracle is a git ref / stripped source / engine-bundled solution, not a portable string; judge correctness is proven by a real solve through the harness, not a synthetic gold (documented + fail-loud, not a fake).
 - **Absent (not built):** swe-gym, swe-bench-multimodal, and the rest of the survey set.
 Every unbuilt/scaffold adapter fails LOUD (throws with the integration step) rather than faking a score — no silent zeros in any corpus. Offline fixture tests: `benchmarks/{aec-bench,commit0,programbench,appworld,rag-benchmarks}.test.mts` (`tsx --test`).

package/dist/adapters.js

@@ -2,9 +2,9 @@ import { createAecBenchAdapter } from "./benchmarks/aec-bench.js";
 import { createAgentBenchAdapter } from "./benchmarks/agentbench.js";
 import { createAppWorldAdapter, createAppWorldReactAdapter } from "./benchmarks/appworld.js";
 import { createBfclAdapter } from "./benchmarks/bfcl.js";
-import { t as createCadBenchAdapter } from "./cadbench-BrpwOU6A.js";
+import { t as createCadBenchAdapter } from "./cadbench-BLSyxR1N.js";
 import { createCadDesignAdapter } from "./benchmarks/cad-design.js";
-import { t as createCadGenBenchAdapter } from "./cadgenbench-DF7hYHdl.js";
+import { t as createCadGenBenchAdapter } from "./cadgenbench-x2OFkf8y.js";
 import { createCommit0Adapter } from "./benchmarks/commit0.js";
 import { createCragAdapter } from "./benchmarks/crag.js";
 import { createDabstepAdapter } from "./benchmarks/dabstep.js";
@@ -14,6 +14,8 @@ import { createFinsearchcompAdapter } from "./benchmarks/finsearchcomp.js";
 import { createFramesAdapter } from "./benchmarks/frames.js";
 import { createHotpotqaAdapter } from "./benchmarks/hotpotqa.js";
 import { createHumanEvalAdapter } from "./benchmarks/humaneval.js";
+import { createMcadBenchAdapter } from "./benchmarks/mcad-bench.js";
+import { createMcadCqAdapter } from "./benchmarks/mcad-cq-bench.js";
 import { createMind2WebAdapter } from "./benchmarks/mind2web.js";
 import { createNoMiraclAdapter } from "./benchmarks/nomiracl.js";
 import { createOpenRagBenchAdapter } from "./benchmarks/open-rag-bench.js";
@@ -52,6 +54,8 @@ const ADAPTERS = {
 	"appworld-react": createAppWorldReactAdapter,
 	"enterpriseops-gym": createEnterpriseOpsGymAdapter,
 	"cad-design": createCadDesignAdapter,
+	mcad: createMcadBenchAdapter,
+	"mcad-cq": createMcadCqAdapter,
 	cadbench: createCadBenchAdapter,
 	cadgenbench: createCadGenBenchAdapter,
 	frames: createFramesAdapter,

package/dist/adapters.js.map

@@ -1 +1 @@
-{"version":3,"file":"adapters.js","names":[],"sources":["../src/adapters.ts"],"sourcesContent":["/**\n * The benchmark registry — the single source of truth for every wired benchmark.\n * One key per benchmark; the value is its `BenchmarkAdapter` factory. `runBenchmarks`\n * (the unifier) maps over this; `run.ts`, `rsi.ts`, and `corpus-replay.mts` all read it\n * here rather than each keeping their own copy.\n */\n\nimport { createAecBenchAdapter } from './benchmarks/aec-bench'\nimport { createAgentBenchAdapter } from './benchmarks/agentbench'\nimport { createAppWorldAdapter, createAppWorldReactAdapter } from './benchmarks/appworld'\nimport { createBfclAdapter } from './benchmarks/bfcl'\nimport { createCadBenchAdapter } from './benchmarks/cadbench'\nimport { createCadDesignAdapter } from './benchmarks/cad-design'\nimport { createCadGenBenchAdapter } from './benchmarks/cadgenbench'\nimport { createCommit0Adapter } from './benchmarks/commit0'\nimport { createCragAdapter } from './benchmarks/crag'\nimport { createDabstepAdapter } from './benchmarks/dabstep'\nimport { createEnterpriseOpsGymAdapter } from './benchmarks/enterpriseops-gym'\nimport { createFinResearchBenchAdapter } from './benchmarks/finresearchbench'\nimport { createFinsearchcompAdapter } from './benchmarks/finsearchcomp'\nimport { createFramesAdapter } from './benchmarks/frames'\nimport { createHotpotqaAdapter } from './benchmarks/hotpotqa'\nimport { createHumanEvalAdapter } from './benchmarks/humaneval'\nimport { createMind2WebAdapter } from './benchmarks/mind2web'\nimport { createNoMiraclAdapter } from './benchmarks/nomiracl'\nimport { createOpenRagBenchAdapter } from './benchmarks/open-rag-bench'\nimport { createProgrambenchAdapter } from './benchmarks/programbench'\nimport { createRagBenchAdapter } from './benchmarks/ragbench'\nimport { createSimpleQaAdapter } from './benchmarks/simpleqa'\nimport { createSweBenchAdapter } from './benchmarks/swe-bench'\nimport { createT2RagBenchAdapter } from './benchmarks/t2-ragbench'\nimport { createTau2BenchAdapter } from './benchmarks/tau2-bench'\nimport { createTau3BankingAdapter } from './benchmarks/tau3-banking'\nimport { createTerminalBenchAdapter } from './benchmarks/terminal-bench'\nimport { createToolLlmAdapter } from './benchmarks/toollm'\nimport { createTrataHedgeAdapter } from './benchmarks/trata-hedge'\nimport { createWebArenaVerifiedAdapter } from './benchmarks/webarena-verified'\nimport type { BenchmarkAdapter } from './benchmarks/types'\n\nexport const ADAPTERS: Record<string, () => BenchmarkAdapter> = {\n  'swe-bench': createSweBenchAdapter,\n  'terminal-bench': createTerminalBenchAdapter,\n  // Code-benches sharing ./benchmarks/_harness (stage → external evaluator → report).\n  // loadTasks fetches the REAL dataset (committed fixtures fallback offline); judge\n  // delegates to the benchmark's own harness and fails loud when it/Docker is absent.\n  'aec-bench': createAecBenchAdapter,\n  commit0: createCommit0Adapter,\n  dabstep: createDabstepAdapter,\n  programbench: createProgrambenchAdapter,\n  'webarena-verified': createWebArenaVerifiedAdapter,\n  'tau2-bench': createTau2BenchAdapter,\n  'tau3-banking': createTau3BankingAdapter,\n  agentbench: createAgentBenchAdapter,\n  bfcl: createBfclAdapter,\n  toollm: createToolLlmAdapter,\n  appworld: createAppWorldAdapter,\n  // AppWorld's native interactive protocol — the worker is the in-engine ReAct\n  // episode (execution feedback every turn), the mode published baselines use.\n  'appworld-react': createAppWorldReactAdapter,\n  'enterpriseops-gym': createEnterpriseOpsGymAdapter,\n  'cad-design': createCadDesignAdapter,\n  cadbench: createCadBenchAdapter,\n  cadgenbench: createCadGenBenchAdapter,\n  frames: createFramesAdapter,\n  ragbench: createRagBenchAdapter,\n  crag: createCragAdapter,\n  nomiracl: createNoMiraclAdapter,\n  'open-rag-bench': createOpenRagBenchAdapter,\n  't2-ragbench': createT2RagBenchAdapter,\n  finresearchbench: createFinResearchBenchAdapter,\n  finsearchcomp: createFinsearchcompAdapter,\n  simpleqa: createSimpleQaAdapter,\n  hotpotqa: createHotpotqaAdapter,\n  // Deployable-checker code domain: worker = router completion, judge = Docker test\n  // run (--network=none). The steering A/B counterpart to humaneval-gate.mts (selection).\n  humaneval: createHumanEvalAdapter,\n  mind2web: createMind2WebAdapter,\n  'trata-hedge': createTrataHedgeAdapter,\n}\n\n/** Resolve a benchmark key to its adapter, failing loud with the known keys. */\nexport function resolveAdapter(key: string): BenchmarkAdapter {\n  const make = ADAPTERS[key]\n  if (!make) throw new Error(`unknown benchmark ${JSON.stringify(key)} (have: ${Object.keys(ADAPTERS).join(', ')})`)\n  return make()\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAuCA,MAAa,WAAmD;CAC9D,aAAa;CACb,kBAAkB;CAIlB,aAAa;CACb,SAAS;CACT,SAAS;CACT,cAAc;CACd,qBAAqB;CACrB,cAAc;CACd,gBAAgB;CAChB,YAAY;CACZ,MAAM;CACN,QAAQ;CACR,UAAU;CAGV,kBAAkB;CAClB,qBAAqB;CACrB,cAAc;CACd,UAAU;CACV,aAAa;CACb,QAAQ;CACR,UAAU;CACV,MAAM;CACN,UAAU;CACV,kBAAkB;CAClB,eAAe;CACf,kBAAkB;CAClB,eAAe;CACf,UAAU;CACV,UAAU;CAGV,WAAW;CACX,UAAU;CACV,eAAe;AACjB;;AAGA,SAAgB,eAAe,KAA+B;CAC5D,MAAM,OAAO,SAAS;CACtB,IAAI,CAAC,MAAM,MAAM,IAAI,MAAM,qBAAqB,KAAK,UAAU,GAAG,EAAE,UAAU,OAAO,KAAK,QAAQ,CAAC,CAAC,KAAK,IAAI,EAAE,EAAE;CACjH,OAAO,KAAK;AACd"}
+{"version":3,"file":"adapters.js","names":[],"sources":["../src/adapters.ts"],"sourcesContent":["/**\n * The benchmark registry — the single source of truth for every wired benchmark.\n * One key per benchmark; the value is its `BenchmarkAdapter` factory. `runBenchmarks`\n * (the unifier) maps over this; `run.ts`, `rsi.ts`, and `corpus-replay.mts` all read it\n * here rather than each keeping their own copy.\n */\n\nimport { createAecBenchAdapter } from './benchmarks/aec-bench'\nimport { createAgentBenchAdapter } from './benchmarks/agentbench'\nimport { createAppWorldAdapter, createAppWorldReactAdapter } from './benchmarks/appworld'\nimport { createBfclAdapter } from './benchmarks/bfcl'\nimport { createCadBenchAdapter } from './benchmarks/cadbench'\nimport { createCadDesignAdapter } from './benchmarks/cad-design'\nimport { createCadGenBenchAdapter } from './benchmarks/cadgenbench'\nimport { createCommit0Adapter } from './benchmarks/commit0'\nimport { createCragAdapter } from './benchmarks/crag'\nimport { createDabstepAdapter } from './benchmarks/dabstep'\nimport { createEnterpriseOpsGymAdapter } from './benchmarks/enterpriseops-gym'\nimport { createFinResearchBenchAdapter } from './benchmarks/finresearchbench'\nimport { createFinsearchcompAdapter } from './benchmarks/finsearchcomp'\nimport { createFramesAdapter } from './benchmarks/frames'\nimport { createHotpotqaAdapter } from './benchmarks/hotpotqa'\nimport { createHumanEvalAdapter } from './benchmarks/humaneval'\nimport { createMcadBenchAdapter } from './benchmarks/mcad-bench'\nimport { createMcadCqAdapter } from './benchmarks/mcad-cq-bench'\nimport { createMind2WebAdapter } from './benchmarks/mind2web'\nimport { createNoMiraclAdapter } from './benchmarks/nomiracl'\nimport { createOpenRagBenchAdapter } from './benchmarks/open-rag-bench'\nimport { createProgrambenchAdapter } from './benchmarks/programbench'\nimport { createRagBenchAdapter } from './benchmarks/ragbench'\nimport { createSimpleQaAdapter } from './benchmarks/simpleqa'\nimport { createSweBenchAdapter } from './benchmarks/swe-bench'\nimport { createT2RagBenchAdapter } from './benchmarks/t2-ragbench'\nimport { createTau2BenchAdapter } from './benchmarks/tau2-bench'\nimport { createTau3BankingAdapter } from './benchmarks/tau3-banking'\nimport { createTerminalBenchAdapter } from './benchmarks/terminal-bench'\nimport { createToolLlmAdapter } from './benchmarks/toollm'\nimport { createTrataHedgeAdapter } from './benchmarks/trata-hedge'\nimport { createWebArenaVerifiedAdapter } from './benchmarks/webarena-verified'\nimport type { BenchmarkAdapter } from './benchmarks/types'\n\nexport const ADAPTERS: Record<string, () => BenchmarkAdapter> = {\n  'swe-bench': createSweBenchAdapter,\n  'terminal-bench': createTerminalBenchAdapter,\n  // Code-benches sharing ./benchmarks/_harness (stage → external evaluator → report).\n  // loadTasks fetches the REAL dataset (committed fixtures fallback offline); judge\n  // delegates to the benchmark's own harness and fails loud when it/Docker is absent.\n  'aec-bench': createAecBenchAdapter,\n  commit0: createCommit0Adapter,\n  dabstep: createDabstepAdapter,\n  programbench: createProgrambenchAdapter,\n  'webarena-verified': createWebArenaVerifiedAdapter,\n  'tau2-bench': createTau2BenchAdapter,\n  'tau3-banking': createTau3BankingAdapter,\n  agentbench: createAgentBenchAdapter,\n  bfcl: createBfclAdapter,\n  toollm: createToolLlmAdapter,\n  appworld: createAppWorldAdapter,\n  // AppWorld's native interactive protocol — the worker is the in-engine ReAct\n  // episode (execution feedback every turn), the mode published baselines use.\n  'appworld-react': createAppWorldReactAdapter,\n  'enterpriseops-gym': createEnterpriseOpsGymAdapter,\n  'cad-design': createCadDesignAdapter,\n  // Dimensioned mechanical parts: same OpenSCAD+xvfb judge deps as cad-design, but\n  // the spec is metric (bbox / volume / body count / point-in-solid hole probes)\n  // rather than qualitative, and every task carries a verified gold.\n  mcad: createMcadBenchAdapter,\n  // The same 10 parts and the same spec assertions, but the worker writes a Python\n  // CadQuery script that must export STEP as well as the ASCII STL — which closes\n  // the format deviation `mcad-tasks.ts` documents (upstream asks for STEP;\n  // OpenSCAD cannot emit it). STEP delivery is one extra SCORED check.\n  'mcad-cq': createMcadCqAdapter,\n  cadbench: createCadBenchAdapter,\n  cadgenbench: createCadGenBenchAdapter,\n  frames: createFramesAdapter,\n  ragbench: createRagBenchAdapter,\n  crag: createCragAdapter,\n  nomiracl: createNoMiraclAdapter,\n  'open-rag-bench': createOpenRagBenchAdapter,\n  't2-ragbench': createT2RagBenchAdapter,\n  finresearchbench: createFinResearchBenchAdapter,\n  finsearchcomp: createFinsearchcompAdapter,\n  simpleqa: createSimpleQaAdapter,\n  hotpotqa: createHotpotqaAdapter,\n  // Deployable-checker code domain: worker = router completion, judge = Docker test\n  // run (--network=none). The steering A/B counterpart to humaneval-gate.mts (selection).\n  humaneval: createHumanEvalAdapter,\n  mind2web: createMind2WebAdapter,\n  'trata-hedge': createTrataHedgeAdapter,\n}\n\n/** Resolve a benchmark key to its adapter, failing loud with the known keys. */\nexport function resolveAdapter(key: string): BenchmarkAdapter {\n  const make = ADAPTERS[key]\n  if (!make) throw new Error(`unknown benchmark ${JSON.stringify(key)} (have: ${Object.keys(ADAPTERS).join(', ')})`)\n  return make()\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAyCA,MAAa,WAAmD;CAC9D,aAAa;CACb,kBAAkB;CAIlB,aAAa;CACb,SAAS;CACT,SAAS;CACT,cAAc;CACd,qBAAqB;CACrB,cAAc;CACd,gBAAgB;CAChB,YAAY;CACZ,MAAM;CACN,QAAQ;CACR,UAAU;CAGV,kBAAkB;CAClB,qBAAqB;CACrB,cAAc;CAId,MAAM;CAKN,WAAW;CACX,UAAU;CACV,aAAa;CACb,QAAQ;CACR,UAAU;CACV,MAAM;CACN,UAAU;CACV,kBAAkB;CAClB,eAAe;CACf,kBAAkB;CAClB,eAAe;CACf,UAAU;CACV,UAAU;CAGV,WAAW;CACX,UAAU;CACV,eAAe;AACjB;;AAGA,SAAgB,eAAe,KAA+B;CAC5D,MAAM,OAAO,SAAS;CACtB,IAAI,CAAC,MAAM,MAAM,IAAI,MAAM,qBAAqB,KAAK,UAAU,GAAG,EAAE,UAAU,OAAO,KAAK,QAAQ,CAAC,CAAC,KAAK,IAAI,EAAE,EAAE;CACjH,OAAO,KAAK;AACd"}

package/dist/benchmarks/appworld.d.ts

@@ -4,14 +4,73 @@ import { OutputAdapter } from "@tangle-network/agent-runtime/kernel";
 /** Worker solution code = the last fenced ```python block, else the raw text. */
 declare const appworldSolutionOutput: OutputAdapter<string>;
 declare function createAppWorldAdapter(): BenchmarkAdapter;
-/** SandboxClient whose leaf is OUR routerToolLoop driving a persistent world session. */
+/**
+ * AppWorld in its NATIVE protocol, run by OUR runtime: the worker is
+ * Runtime's profile-bound `router-tools` executor with one tool —
+ * `execute_python` — bound to a persistent AppWorld world session. The driver's
+ * `session` subcommand is a dumb world shim (stdin JSONL: execute → output,
+ * evaluate → verdict); every inference turn, the metering, and the typed
+ * toolTrace the analyst steers on belong to the runtime, so runtime
+ * improvements are what this benchmark measures.
+ *
+ * The one-shot codegen adapter above plays a strictly harder game (no execution
+ * feedback — the first wrong API call kills the whole program at judge time),
+ * which flatlines the score against ANY steering; this mode is what the
+ * benchmark's published baselines use, where behavior can move sub-tests.
+ *
+ * Protocol: the round task string is `@appworld-react <taskId> <split>` on
+ * line 1; everything after line 1 is the steer (an analyst correction, a push
+ * directive) appended to the system prompt — so the existing arms steer this
+ * worker without modification. The artifact is the episode evaluation JSON
+ * (AppWorld's evaluator ran in-world); judge() parses it, never re-executes.
+ */
+interface ReactResult {
+  success?: boolean;
+  passes?: number;
+  fails?: number;
+  num_tests?: number;
+  failure_names?: string[];
+  turns?: number;
+  input_tokens?: number;
+  output_tokens?: number;
+  cost_usd?: number;
+  transcript?: string;
+}
+interface ReactRuntimeUsage {
+  input: number;
+  output: number;
+  costUsd?: number;
+  tokensKnown?: boolean;
+  usdKnown?: boolean;
+}
+/** Preserve a completed scientific/task result even when one accounting dimension is incomplete.
+ * Unknown usage fields stay absent; later comparison/reporting can refuse a cost claim without
+ * discarding the episode's task evidence. */
+declare function appworldReactResultWithUsage(verdict: ReactResult, usage: ReactRuntimeUsage, turns: number | undefined, transcript: string): ReactResult;
+/** Emit only usage the Runtime actually knows. Catalog estimates never become observed dollars. */
+declare function appworldReactUsageEvent(result: ReactResult, model: string): {
+  type: 'llm_call';
+  data: Record<string, unknown>;
+} | undefined;
+/** One persistent world session: line-JSONL request/response over the driver. */
+declare function withWorldSession<T>(taskId: string, split: string, signal: AbortSignal, fn: (call: (cmd: Record<string, unknown>) => Promise<Record<string, unknown>>, instruction: string) => Promise<T>): Promise<T>;
+type AppWorldWorldSession = typeof withWorldSession;
+type AppWorldComplete = (body: Record<string, unknown>, request?: {
+  readonly headers: Readonly<Record<string, string>>;
+  readonly signal?: AbortSignal;
+}) => Promise<unknown>;
+/** SandboxClient whose leaf is Runtime's profile-bound Router executor driving a world session. */
 declare function appworldToolLoopClient(cfg: {
   model: string;
   routerBaseUrl: string;
   routerKey: string;
   maxTurns?: number;
+  /** Offline-test seam; production always uses the Python AppWorld session above. */
+  runWorldSession?: AppWorldWorldSession;
+  /** Offline-test seam; production uses Runtime's Router HTTP transport. */
+  complete?: AppWorldComplete;
 }): unknown;
 declare function createAppWorldReactAdapter(): BenchmarkAdapter;
 //#endregion
-export { appworldSolutionOutput, appworldToolLoopClient, createAppWorldAdapter, createAppWorldReactAdapter };
+export { ReactResult, appworldReactResultWithUsage, appworldReactUsageEvent, appworldSolutionOutput, appworldToolLoopClient, createAppWorldAdapter, createAppWorldReactAdapter };
 //# sourceMappingURL=appworld.d.ts.map

package/dist/benchmarks/appworld.js

@@ -2,7 +2,7 @@ import { benchRoot, preflightVenvImports, runVenvScriptStdin, venvPython } from
 import { join } from "node:path";
 import { spawn } from "node:child_process";
 import { createInterface } from "node:readline";
-import { routerToolLoop } from "@tangle-network/agent-runtime/kernel";
+import { collectAgentTurn, createExecutor, streamAgentTurn } from "@tangle-network/agent-runtime/kernel";
 //#region src/benchmarks/appworld.ts
 /**
 * AppWorld adapter (StonyBrookNLP/appworld). Worker artifact = the agent's
@@ -128,6 +128,38 @@ function createAppWorldAdapter() {
 		}
 	};
 }
+/** Preserve a completed scientific/task result even when one accounting dimension is incomplete.
+* Unknown usage fields stay absent; later comparison/reporting can refuse a cost claim without
+* discarding the episode's task evidence. */
+function appworldReactResultWithUsage(verdict, usage, turns, transcript) {
+	return {
+		...verdict,
+		...turns !== void 0 ? { turns } : {},
+		...usage.tokensKnown === false ? {} : {
+			input_tokens: usage.input,
+			output_tokens: usage.output
+		},
+		...usage.usdKnown === false || usage.costUsd === void 0 ? {} : { cost_usd: usage.costUsd },
+		transcript
+	};
+}
+/** Emit only usage the Runtime actually knows. Catalog estimates never become observed dollars. */
+function appworldReactUsageEvent(result, model) {
+	const hasTokens = typeof result.input_tokens === "number" && typeof result.output_tokens === "number";
+	const hasCost = typeof result.cost_usd === "number";
+	if (!hasTokens && !hasCost) return void 0;
+	return {
+		type: "llm_call",
+		data: {
+			model,
+			...hasTokens ? {
+				tokensIn: result.input_tokens,
+				tokensOut: result.output_tokens
+			} : {},
+			...hasCost ? { costUsd: result.cost_usd } : {}
+		}
+	};
+}
 const REACT_HEADER = /^@appworld-react (\S+) (\S+)\n?/;
 const SESSION_SYSTEM = [
 	"You are completing a task in AppWorld, a simulated multi-app environment.",
@@ -153,7 +185,8 @@ const EXECUTE_TOOL = {
 	}
 };
 /** One persistent world session: line-JSONL request/response over the driver. */
-async function withWorldSession(taskId, split, fn) {
+async function withWorldSession(taskId, split, signal, fn) {
+	signal.throwIfAborted();
 	const child = spawn(venvPython, [
 		DRIVER,
 		"session",
@@ -162,6 +195,10 @@ async function withWorldSession(taskId, split, fn) {
 		"--split",
 		split
 	], { cwd: benchRoot });
+	const stopChild = () => {
+		if (!child.killed) child.kill("SIGTERM");
+	};
+	signal.addEventListener("abort", stopChild, { once: true });
 	const rl = createInterface({ input: child.stdout });
 	const pending = [];
 	const backlog = [];
@@ -175,19 +212,34 @@ async function withWorldSession(taskId, split, fn) {
 		stderr += c.toString("utf8");
 	});
 	const nextLine = (timeoutMs) => new Promise((resolve, reject) => {
+		if (signal.aborted) {
+			reject(signal.reason);
+			return;
+		}
 		const fromBacklog = backlog.shift();
 		if (fromBacklog !== void 0) return resolve(fromBacklog);
 		const t = setTimeout(() => reject(/* @__PURE__ */ new Error(`appworld session: no response in ${timeoutMs}ms; stderr: ${stderr.slice(-400)}`)), timeoutMs);
 		const onExit = (code) => {
 			clearTimeout(t);
-			reject(/* @__PURE__ */ new Error(`appworld session exited (${code}); stderr: ${stderr.slice(-400)}`));
+			signal.removeEventListener("abort", onAbort);
+			reject(signal.aborted ? signal.reason : /* @__PURE__ */ new Error(`appworld session exited (${code}); stderr: ${stderr.slice(-400)}`));
+		};
+		const onAbort = () => {
+			clearTimeout(t);
+			child.removeListener("exit", onExit);
+			const index = pending.indexOf(onLine);
+			if (index >= 0) pending.splice(index, 1);
+			reject(signal.reason);
 		};
-		pending.push((l) => {
+		const onLine = (line) => {
 			clearTimeout(t);
 			child.removeListener("exit", onExit);
-			resolve(l);
-		});
+			signal.removeEventListener("abort", onAbort);
+			resolve(line);
+		};
+		pending.push(onLine);
 		child.once("exit", onExit);
+		signal.addEventListener("abort", onAbort, { once: true });
 	});
 	try {
 		const ready = JSON.parse(await nextLine(12e4));
@@ -200,55 +252,74 @@ async function withWorldSession(taskId, split, fn) {
 		};
 		return await fn(call, ready.instruction ?? "");
 	} finally {
+		signal.removeEventListener("abort", stopChild);
 		child.stdin.end();
-		child.kill("SIGTERM");
+		stopChild();
 	}
 }
-/** SandboxClient whose leaf is OUR routerToolLoop driving a persistent world session. */
+/** SandboxClient whose leaf is Runtime's profile-bound Router executor driving a world session. */
 function appworldToolLoopClient(cfg) {
 	const maxTurns = cfg.maxTurns ?? Number(process.env.REACT_MAX_TURNS ?? 40);
+	const runWorldSession = cfg.runWorldSession ?? withWorldSession;
 	let seq = 0;
 	return { async create() {
 		return {
 			id: `appworld-toolloop-${seq++}`,
-			async *streamPrompt(prompt) {
+			async *streamPrompt(prompt, promptOpts) {
+				const signal = promptOpts?.signal ?? new AbortController().signal;
+				signal.throwIfAborted();
 				const m = prompt.match(REACT_HEADER);
 				if (!m) throw new Error(`appworld-react leaf: prompt missing '@appworld-react <taskId> <split>' header — got: ${prompt.slice(0, 120)}`);
 				const [, taskId, split] = m;
 				const directive = prompt.replace(REACT_HEADER, "").trim();
-				const out = await withWorldSession(taskId, split, async (call, instruction) => {
+				const out = await runWorldSession(taskId, split, signal, async (call, instruction) => {
 					const system = directive ? `${SESSION_SYSTEM}\n\n${directive}` : SESSION_SYSTEM;
-					const loop = await routerToolLoop({
-						routerBaseUrl: cfg.routerBaseUrl,
-						routerKey: cfg.routerKey,
-						model: cfg.model
-					}, system, `Task: ${instruction}`, [EXECUTE_TOOL], async (name, args) => {
-						if (name !== "execute_python") return `error: unknown tool ${name}`;
-						const res = await call({
-							op: "execute",
-							code: String(args.code ?? "")
-						});
-						const done = res.task_completed === true;
-						return `${String(res.output ?? "")}${done ? "\n\n[TASK MARKED COMPLETE — reply with a final summary and do not call the tool again]" : ""}`;
-					}, { maxTurns });
-					const verdict = await call({ op: "evaluate" });
-					const transcript = loop.toolTrace.slice(-3).map((t) => `CODE:\n${t.args.slice(0, 600)}\nOUTPUT:\n${t.result.slice(0, 600)}`).join("\n---\n").slice(0, 1600);
-					return {
-						...verdict,
-						turns: loop.turns,
-						input_tokens: loop.usage.input,
-						output_tokens: loop.usage.output,
-						transcript
+					const transcriptSteps = [];
+					const profile = {
+						name: "appworld-react-worker",
+						harness: "cli-base",
+						model: {
+							provider: "tangle-router",
+							default: cfg.model,
+							metadata: { maxTurns }
+						},
+						prompt: { systemPrompt: system },
+						tools: { execute_python: true }
 					};
+					const loop = await collectAgentTurn(streamAgentTurn({
+						kind: "executor",
+						factory: createExecutor({
+							backend: "router-tools",
+							routerBaseUrl: cfg.routerBaseUrl,
+							routerKey: cfg.routerKey,
+							...cfg.complete ? { complete: cfg.complete } : {},
+							tools: [EXECUTE_TOOL],
+							executeToolCall: async (name, args) => {
+								if (name !== "execute_python") return `error: unknown tool ${name}`;
+								const res = await call({
+									op: "execute",
+									code: String(args.code ?? "")
+								});
+								const done = res.task_completed === true;
+								const result = `${String(res.output ?? "")}${done ? "\n\n[TASK MARKED COMPLETE — reply with a final summary and do not call the tool again]" : ""}`;
+								transcriptSteps.push({
+									args: JSON.stringify(args),
+									result
+								});
+								return result;
+							}
+						}),
+						profile
+					}, `Task: ${instruction}`, { signal }));
+					if (loop.status !== "completed") throw new Error(loop.error?.message ?? `AppWorld turn ended with ${loop.status}`);
+					const verdict = await call({ op: "evaluate" });
+					const transcript = transcriptSteps.slice(-3).map((t) => `CODE:\n${t.args.slice(0, 600)}\nOUTPUT:\n${t.result.slice(0, 600)}`).join("\n---\n").slice(0, 1600);
+					const finalEvent = loop.events.at(-1);
+					const resultMetadata = finalEvent?.type === "final" && finalEvent.metadata?.result ? finalEvent.metadata.result : void 0;
+					return appworldReactResultWithUsage(verdict, loop.usage, resultMetadata?.spent?.iterations, transcript);
 				});
-				if (out.input_tokens || out.output_tokens) yield {
-					type: "llm_call",
-					data: {
-						tokensIn: out.input_tokens ?? 0,
-						tokensOut: out.output_tokens ?? 0,
-						model: cfg.model
-					}
-				};
+				const usageEvent = appworldReactUsageEvent(out, cfg.model);
+				if (usageEvent) yield usageEvent;
 				yield {
 					type: "result",
 					data: { finalText: JSON.stringify(out) }
@@ -314,6 +385,6 @@ function createAppWorldReactAdapter() {
 	};
 }
 //#endregion
-export { appworldSolutionOutput, appworldToolLoopClient, createAppWorldAdapter, createAppWorldReactAdapter };
+export { appworldReactResultWithUsage, appworldReactUsageEvent, appworldSolutionOutput, appworldToolLoopClient, createAppWorldAdapter, createAppWorldReactAdapter };
 
 //# sourceMappingURL=appworld.js.map

package/dist/benchmarks/appworld.js.map

@@ -1 +1 @@
-{"version":3,"file":"appworld.js","names":[],"sources":["../../src/benchmarks/appworld.ts"],"sourcesContent":["/**\n * AppWorld adapter (StonyBrookNLP/appworld). Worker artifact = the agent's\n * Python solution that calls the simulated apps' APIs (the same `apis.<app>.<fn>`\n * surface AppWorld exposes inside `world.execute(...)`), ending in\n * `apis.supervisor.complete_task()`. Judge = AppWorld's OWN programmatic\n * evaluator: a driver runs the solution in a fresh `AppWorld(task_id=...)` world,\n * then `world.evaluate().to_dict()` reports `success` (binary TGC), `num_tests`\n * (per-requirement total) and the `passes`/`failures` lists. Score =\n * passes / num_tests — GRADED; resolved = success. Fully deterministic — no LLM judge.\n *\n * loadTasks enumerates the real task suite via `load_task_ids(split)`\n * (train|dev|test_normal|test_challenge); the prompt = `world.task.instruction`.\n * The OutputAdapter is stream-only, so the worker emits its solution as a fenced\n * ```python block which the driver executes.\n *\n * Requires for a live run: the bench `.venv` with `appworld` installed + the\n * unpacked engine + downloaded data (`appworld install` ; `appworld download\n * data`). preflight + loadTasks + judge all fail loud with the exact step when the\n * engine/data is absent — never a fabricated score.\n */\n\nimport { spawn } from 'node:child_process'\nimport { join } from 'node:path'\nimport { createInterface } from 'node:readline'\nimport { type OutputAdapter, routerToolLoop, type ToolSpec } from '@tangle-network/agent-runtime/kernel'\nimport { benchRoot, preflightVenvImports, runVenvScriptStdin, venvPython } from './_harness'\nimport type { BenchmarkAdapter, BenchScore, BenchTask, LoadOptions } from './types'\n\nconst DRIVER = join(benchRoot, 'scripts', 'appworld_driver.py')\n\n/** AppWorld splits; only the test splits ship evaluation-only (no setup/solution). */\nconst DEFAULT_SPLIT = 'test_normal'\n\ninterface AppWorldMeta {\n  taskId: string\n  split: string\n}\n\n/** Worker solution code = the last fenced ```python block, else the raw text. */\nexport const appworldSolutionOutput: OutputAdapter<string> = {\n  parse(events) {\n    let text = ''\n    for (const ev of events) {\n      const d = (ev as { data?: Record<string, unknown> })?.data\n      const t = d?.finalText ?? d?.text ?? d?.result\n      if (typeof t === 'string' && t.length > 0) text = t\n    }\n    const fences = [...text.matchAll(/```(?:python|py)?\\s*\\n([\\s\\S]*?)```/g)]\n    return (fences.at(-1)?.[1] ?? text).trim()\n  },\n}\n\nconst WORKER_CONTRACT = [\n  '',\n  'Solve this by writing Python that calls the available app APIs (the `apis.<app>.<function>(...)` surface). You may inspect API docs with `apis.api_docs.show_api_descriptions(app_name=...)` and `apis.api_docs.show_api_doc(app_name=..., api_name=...)`.',\n  'Authenticate where needed via the supervisor-provided credentials, perform every step the task requires, and FINISH with `apis.supervisor.complete_task()`.',\n  'Emit your COMPLETE solution as the LAST thing in your reply, in a single fenced ```python block. Nothing after the closing fence.',\n].join('\\n')\n\nfunction readMeta(task: BenchTask): AppWorldMeta {\n  const md = task.metadata\n  if (!md || typeof md.taskId !== 'string') {\n    throw new Error(`appworld task ${task.id} missing metadata.taskId — loadTasks did not populate it`)\n  }\n  return md as unknown as AppWorldMeta\n}\n\n/**\n * Run the appworld engine driver with a subcommand; JSON on the LAST stdout line.\n * The solution code (evaluate) is piped to stdin via the shared stdin-aware runner —\n * execFile's `input` option is not honored async and hangs the driver's\n * sys.stdin.read() forever. `load` ignores stdin, so an empty pipe is harmless.\n */\nasync function driver(args: string[], input = ''): Promise<unknown> {\n  let stdout: string\n  try {\n    stdout = await runVenvScriptStdin(DRIVER, args, input, { cwd: benchRoot })\n  } catch (err) {\n    const e = err as { message?: string }\n    throw new Error(`appworld driver failed (${args.join(' ')}): ${(e.message || String(err)).slice(0, 1500)}`)\n  }\n  const last = stdout.trim().split('\\n').at(-1) ?? '{}'\n  const parsed = JSON.parse(last) as { error?: string }\n  if (parsed.error) throw new Error(`appworld driver error: ${parsed.error}`)\n  return parsed\n}\n\nexport function createAppWorldAdapter(): BenchmarkAdapter {\n  return {\n    name: 'appworld',\n    output: appworldSolutionOutput,\n\n    async preflight() {\n      await preflightVenvImports({\n        modules: ['appworld'],\n        requireDocker: false,\n        fix:\n          'Fix: bench/.venv/bin/pip install appworld ; ' +\n          'bench/.venv/bin/appworld install ; bench/.venv/bin/appworld download data ' +\n          '(unpacks the engine + downloads the simulated-app data/tasks). ' +\n          'Set APPWORLD_ROOT to the data root if not the default.',\n      })\n    },\n\n    async loadTasks(opts: LoadOptions = {}): Promise<BenchTask[]> {\n      const split = opts.split ?? DEFAULT_SPLIT\n      const out = (await driver([\n        'load',\n        '--split', split,\n        ...(opts.limit !== undefined ? ['--limit', String(opts.limit)] : []),\n        ...(opts.ids ? ['--ids', opts.ids.join(',')] : []),\n      ])) as { tasks?: Array<{ task_id: string; instruction: string }> }\n      const tasks = out.tasks ?? []\n      if (tasks.length === 0) {\n        throw new Error(`appworld loadTasks returned no tasks for split=${split} ${JSON.stringify(opts)}`)\n      }\n      return tasks.map(\n        (t): BenchTask => ({\n          id: t.task_id,\n          split,\n          prompt: t.instruction + WORKER_CONTRACT,\n          metadata: { taskId: t.task_id, split } as unknown as Record<string, unknown>,\n        }),\n      )\n    },\n\n    async goldArtifact() {\n      // Reference solution code ships only for train/dev, and only inside the\n      // engine's decrypted `.bundle` (it is not a portable string this adapter can\n      // emit across splits). The test splits are evaluation-only. So verify-judge\n      // here requires a real solve on a train/dev task through the live engine\n      // rather than a synthetic gold — returning a fabricated artifact would be a\n      // fake. Returns undefined.\n      return undefined\n    },\n\n    async judge(task: BenchTask, artifact: string): Promise<BenchScore> {\n      const meta = readMeta(task)\n      const out = (await driver(['evaluate', '--task-id', meta.taskId, '--split', meta.split], artifact)) as {\n        success?: boolean\n        passes?: number\n        fails?: number\n        num_tests?: number\n        failure_names?: string[]\n      }\n      const passes = out.passes ?? 0\n      const fails = out.fails ?? 0\n      // num_tests is the evaluator's authoritative per-requirement count; prefer it\n      // over passes+fails (which can disagree if a requirement neither passed nor\n      // failed). Never default the total to a phantom denominator.\n      const total = out.num_tests ?? passes + fails\n      const score = total > 0 ? passes / total : 0\n      // failure_names = WHICH sub-tests failed — the evidence a trace analyst\n      // steers on. Carried in `detail` so it reaches the verdict's `notes`.\n      const failures = Array.isArray(out.failure_names) ? out.failure_names : []\n      return {\n        resolved: out.success === true,\n        score,\n        detail: JSON.stringify({\n          taskId: meta.taskId,\n          success: out.success,\n          passes,\n          fails,\n          total,\n          ...(failures.length ? { failures } : {}),\n        }),\n      }\n    },\n  }\n}\n\n/**\n * AppWorld in its NATIVE protocol, run by OUR runtime: the worker is\n * `routerToolLoop` (the runtime's off-box agentic tool loop) with one tool —\n * `execute_python` — bound to a persistent AppWorld world session. The driver's\n * `session` subcommand is a dumb world shim (stdin JSONL: execute → output,\n * evaluate → verdict); every inference turn, the metering, and the typed\n * toolTrace the analyst steers on belong to the runtime, so runtime\n * improvements are what this benchmark measures.\n *\n * The one-shot codegen adapter above plays a strictly harder game (no execution\n * feedback — the first wrong API call kills the whole program at judge time),\n * which flatlines the score against ANY steering; this mode is what the\n * benchmark's published baselines use, where behavior can move sub-tests.\n *\n * Protocol: the round task string is `@appworld-react <taskId> <split>` on\n * line 1; everything after line 1 is the steer (an analyst correction, a push\n * directive) appended to the system prompt — so the existing arms steer this\n * worker without modification. The artifact is the episode evaluation JSON\n * (AppWorld's evaluator ran in-world); judge() parses it, never re-executes.\n */\n\ninterface ReactResult {\n  success?: boolean\n  passes?: number\n  fails?: number\n  num_tests?: number\n  failure_names?: string[]\n  turns?: number\n  input_tokens?: number\n  output_tokens?: number\n  transcript?: string\n}\n\nconst REACT_HEADER = /^@appworld-react (\\S+) (\\S+)\\n?/\n\nconst SESSION_SYSTEM = [\n  'You are completing a task in AppWorld, a simulated multi-app environment.',\n  'Use the execute_python tool to run Python that calls the app APIs (the `apis.<app>.<function>(...)` surface).',\n  'Inspect API docs with `apis.api_docs.show_api_descriptions(app_name=...)` and `apis.api_docs.show_api_doc(app_name=..., api_name=...)`.',\n  'Authenticate where needed via the supervisor-provided credentials (`apis.supervisor.show_account_passwords()`).',\n  'Work incrementally: small snippets, read each output, correct course.',\n  'When every step of the task is done, run `apis.supervisor.complete_task()` and then reply WITHOUT calling the tool again.',\n].join('\\n')\n\nconst EXECUTE_TOOL: ToolSpec = {\n  type: 'function',\n  function: {\n    name: 'execute_python',\n    description:\n      'Execute a Python snippet in the persistent AppWorld world. State persists across calls. Returns the execution output (API results or errors).',\n    parameters: {\n      type: 'object',\n      properties: { code: { type: 'string', description: 'Python code calling apis.<app>.<fn>(...)' } },\n      required: ['code'],\n    },\n  },\n}\n\n/** One persistent world session: line-JSONL request/response over the driver. */\nasync function withWorldSession<T>(\n  taskId: string,\n  split: string,\n  fn: (call: (cmd: Record<string, unknown>) => Promise<Record<string, unknown>>, instruction: string) => Promise<T>,\n): Promise<T> {\n  const child = spawn(venvPython, [DRIVER, 'session', '--task-id', taskId, '--split', split], {\n    cwd: benchRoot,\n  })\n  const rl = createInterface({ input: child.stdout })\n  const pending: Array<(line: string) => void> = []\n  const backlog: string[] = []\n  rl.on('line', (l) => {\n    const next = pending.shift()\n    if (next) next(l)\n    else backlog.push(l)\n  })\n  let stderr = ''\n  child.stderr.on('data', (c: Buffer) => {\n    stderr += c.toString('utf8')\n  })\n  const nextLine = (timeoutMs: number): Promise<string> =>\n    new Promise((resolve, reject) => {\n      const fromBacklog = backlog.shift()\n      if (fromBacklog !== undefined) return resolve(fromBacklog)\n      const t = setTimeout(\n        () => reject(new Error(`appworld session: no response in ${timeoutMs}ms; stderr: ${stderr.slice(-400)}`)),\n        timeoutMs,\n      )\n      // One exit listener per await leaks (25-turn episodes blow the listener\n      // cap) — remove it on the resolve path.\n      const onExit = (code: number | null): void => {\n        clearTimeout(t)\n        reject(new Error(`appworld session exited (${code}); stderr: ${stderr.slice(-400)}`))\n      }\n      pending.push((l) => {\n        clearTimeout(t)\n        child.removeListener('exit', onExit)\n        resolve(l)\n      })\n      child.once('exit', onExit)\n    })\n  try {\n    const ready = JSON.parse(await nextLine(120_000)) as { ready?: boolean; instruction?: string; error?: string }\n    if (!ready.ready) throw new Error(`appworld session failed to start: ${ready.error ?? 'no ready line'}`)\n    const call = async (cmd: Record<string, unknown>): Promise<Record<string, unknown>> => {\n      child.stdin.write(`${JSON.stringify(cmd)}\\n`)\n      const res = JSON.parse(await nextLine(180_000)) as Record<string, unknown>\n      if (typeof res.error === 'string') throw new Error(`appworld session op failed: ${res.error}`)\n      return res\n    }\n    return await fn(call, ready.instruction ?? '')\n  } finally {\n    child.stdin.end()\n    child.kill('SIGTERM')\n  }\n}\n\n/** SandboxClient whose leaf is OUR routerToolLoop driving a persistent world session. */\nexport function appworldToolLoopClient(cfg: {\n  model: string\n  routerBaseUrl: string\n  routerKey: string\n  maxTurns?: number\n}): unknown {\n  const maxTurns = cfg.maxTurns ?? Number(process.env.REACT_MAX_TURNS ?? 40)\n  let seq = 0\n  return {\n    async create() {\n      const id = `appworld-toolloop-${seq++}`\n      return {\n        id,\n        async *streamPrompt(prompt: string) {\n          const m = prompt.match(REACT_HEADER)\n          if (!m) {\n            throw new Error(\n              `appworld-react leaf: prompt missing '@appworld-react <taskId> <split>' header — got: ${prompt.slice(0, 120)}`,\n            )\n          }\n          const [, taskId, split] = m\n          const directive = prompt.replace(REACT_HEADER, '').trim()\n          const out = await withWorldSession(taskId as string, split as string, async (call, instruction) => {\n            const system = directive ? `${SESSION_SYSTEM}\\n\\n${directive}` : SESSION_SYSTEM\n            const loop = await routerToolLoop(\n              { routerBaseUrl: cfg.routerBaseUrl, routerKey: cfg.routerKey, model: cfg.model },\n              system,\n              `Task: ${instruction}`,\n              [EXECUTE_TOOL],\n              async (name, args) => {\n                if (name !== 'execute_python') return `error: unknown tool ${name}`\n                const res = await call({ op: 'execute', code: String(args.code ?? '') })\n                const done = res.task_completed === true\n                return `${String(res.output ?? '')}${done ? '\\n\\n[TASK MARKED COMPLETE — reply with a final summary and do not call the tool again]' : ''}`\n              },\n              { maxTurns },\n            )\n            const verdict = (await call({ op: 'evaluate' })) as unknown as ReactResult\n            const transcript = loop.toolTrace\n              .slice(-3)\n              .map((t) => `CODE:\\n${t.args.slice(0, 600)}\\nOUTPUT:\\n${t.result.slice(0, 600)}`)\n              .join('\\n---\\n')\n              .slice(0, 1600)\n            return {\n              ...verdict,\n              turns: loop.turns,\n              input_tokens: loop.usage.input,\n              output_tokens: loop.usage.output,\n              transcript,\n            } satisfies ReactResult\n          })\n          // Real usage from the episode — flat llm_call so the kernel meters it.\n          if (out.input_tokens || out.output_tokens) {\n            yield {\n              type: 'llm_call',\n              data: { tokensIn: out.input_tokens ?? 0, tokensOut: out.output_tokens ?? 0, model: cfg.model },\n            }\n          }\n          yield { type: 'result', data: { finalText: JSON.stringify(out) } }\n        },\n        async delete() {},\n      }\n    },\n  }\n}\n\n/** Artifact = the episode's evaluation JSON, verbatim (no fence extraction). */\nconst reactEpisodeOutput: OutputAdapter<string> = {\n  parse(events) {\n    let text = ''\n    for (const ev of events) {\n      const d = (ev as { data?: Record<string, unknown> })?.data\n      const t = d?.finalText\n      if (typeof t === 'string' && t.length > 0) text = t\n    }\n    return text\n  },\n}\n\nexport function createAppWorldReactAdapter(): BenchmarkAdapter {\n  const base = createAppWorldAdapter()\n  return {\n    name: 'appworld-react',\n    output: reactEpisodeOutput,\n    preflight: () => base.preflight(),\n\n    async loadTasks(opts: LoadOptions = {}): Promise<BenchTask[]> {\n      const tasks = await base.loadTasks(opts)\n      return tasks.map((t) => {\n        const meta = readMeta(t)\n        return {\n          ...t,\n          // Header carries task identity to the leaf; the body (empty at round 0)\n          // is the directive slot the arms append their steer into.\n          prompt: `@appworld-react ${meta.taskId} ${meta.split}\\n`,\n        }\n      })\n    },\n\n    goldArtifact: () => Promise.resolve(undefined),\n\n    async judge(task: BenchTask, artifact: string): Promise<BenchScore> {\n      const meta = readMeta(task)\n      let out: ReactResult\n      try {\n        out = JSON.parse(artifact) as ReactResult\n      } catch {\n        throw new Error(\n          `appworld-react judge: artifact is not the episode's evaluation JSON (task ${meta.taskId}): ${artifact.slice(0, 200)}`,\n        )\n      }\n      if (typeof out.success !== 'boolean' || typeof out.num_tests !== 'number') {\n        throw new Error(\n          `appworld-react judge: episode JSON missing success/num_tests (task ${meta.taskId}): ${artifact.slice(0, 200)}`,\n        )\n      }\n      const passes = out.passes ?? 0\n      const total = out.num_tests\n      const failures = Array.isArray(out.failure_names) ? out.failure_names : []\n      return {\n        resolved: out.success === true,\n        score: total > 0 ? passes / total : 0,\n        detail: JSON.stringify({\n          taskId: meta.taskId,\n          success: out.success,\n          passes,\n          fails: out.fails ?? 0,\n          total,\n          turns: out.turns,\n          ...(failures.length ? { failures } : {}),\n          ...(out.transcript ? { transcriptTail: out.transcript.slice(-800) } : {}),\n        }),\n      }\n    },\n\n    leafClient: (c) => appworldToolLoopClient(c),\n  }\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;AA4BA,MAAM,SAAS,KAAK,WAAW,WAAW,oBAAoB;;AAG9D,MAAM,gBAAgB;;AAQtB,MAAa,yBAAgD,EAC3D,MAAM,QAAQ;CACZ,IAAI,OAAO;CACX,KAAK,MAAM,MAAM,QAAQ;EACvB,MAAM,IAAK,IAA2C;EACtD,MAAM,IAAI,GAAG,aAAa,GAAG,QAAQ,GAAG;EACxC,IAAI,OAAO,MAAM,YAAY,EAAE,SAAS,GAAG,OAAO;CACpD;CAEA,QAAQ,CADQ,GAAG,KAAK,SAAS,sCAAsC,CAC1D,CAAC,CAAC,GAAG,EAAE,CAAC,GAAG,MAAM,KAAA,CAAM,KAAK;AAC3C,EACF;AAEA,MAAM,kBAAkB;CACtB;CACA;CACA;CACA;AACF,CAAC,CAAC,KAAK,IAAI;AAEX,SAAS,SAAS,MAA+B;CAC/C,MAAM,KAAK,KAAK;CAChB,IAAI,CAAC,MAAM,OAAO,GAAG,WAAW,UAC9B,MAAM,IAAI,MAAM,iBAAiB,KAAK,GAAG,yDAAyD;CAEpG,OAAO;AACT;;;;;;;AAQA,eAAe,OAAO,MAAgB,QAAQ,IAAsB;CAClE,IAAI;CACJ,IAAI;EACF,SAAS,MAAM,mBAAmB,QAAQ,MAAM,OAAO,EAAE,KAAK,UAAU,CAAC;CAC3E,SAAS,KAAK;EACZ,MAAM,IAAI;EACV,MAAM,IAAI,MAAM,2BAA2B,KAAK,KAAK,GAAG,EAAE,MAAM,EAAE,WAAW,OAAO,GAAG,EAAA,CAAG,MAAM,GAAG,IAAI,GAAG;CAC5G;CACA,MAAM,OAAO,OAAO,KAAK,CAAC,CAAC,MAAM,IAAI,CAAC,CAAC,GAAG,EAAE,KAAK;CACjD,MAAM,SAAS,KAAK,MAAM,IAAI;CAC9B,IAAI,OAAO,OAAO,MAAM,IAAI,MAAM,0BAA0B,OAAO,OAAO;CAC1E,OAAO;AACT;AAEA,SAAgB,wBAA0C;CACxD,OAAO;EACL,MAAM;EACN,QAAQ;EAER,MAAM,YAAY;GAChB,MAAM,qBAAqB;IACzB,SAAS,CAAC,UAAU;IACpB,eAAe;IACf,KACE;GAIJ,CAAC;EACH;EAEA,MAAM,UAAU,OAAoB,CAAC,GAAyB;GAC5D,MAAM,QAAQ,KAAK,SAAS;GAO5B,MAAM,SAAQ,MANK,OAAO;IACxB;IACA;IAAW;IACX,GAAI,KAAK,UAAU,KAAA,IAAY,CAAC,WAAW,OAAO,KAAK,KAAK,CAAC,IAAI,CAAC;IAClE,GAAI,KAAK,MAAM,CAAC,SAAS,KAAK,IAAI,KAAK,GAAG,CAAC,IAAI,CAAC;GAClD,CAAC,EAAA,CACiB,SAAS,CAAC;GAC5B,IAAI,MAAM,WAAW,GACnB,MAAM,IAAI,MAAM,kDAAkD,MAAM,GAAG,KAAK,UAAU,IAAI,GAAG;GAEnG,OAAO,MAAM,KACV,OAAkB;IACjB,IAAI,EAAE;IACN;IACA,QAAQ,EAAE,cAAc;IACxB,UAAU;KAAE,QAAQ,EAAE;KAAS;IAAM;GACvC,EACF;EACF;EAEA,MAAM,eAAe,CAQrB;EAEA,MAAM,MAAM,MAAiB,UAAuC;GAClE,MAAM,OAAO,SAAS,IAAI;GAC1B,MAAM,MAAO,MAAM,OAAO;IAAC;IAAY;IAAa,KAAK;IAAQ;IAAW,KAAK;GAAK,GAAG,QAAQ;GAOjG,MAAM,SAAS,IAAI,UAAU;GAC7B,MAAM,QAAQ,IAAI,SAAS;GAI3B,MAAM,QAAQ,IAAI,aAAa,SAAS;GACxC,MAAM,QAAQ,QAAQ,IAAI,SAAS,QAAQ;GAG3C,MAAM,WAAW,MAAM,QAAQ,IAAI,aAAa,IAAI,IAAI,gBAAgB,CAAC;GACzE,OAAO;IACL,UAAU,IAAI,YAAY;IAC1B;IACA,QAAQ,KAAK,UAAU;KACrB,QAAQ,KAAK;KACb,SAAS,IAAI;KACb;KACA;KACA;KACA,GAAI,SAAS,SAAS,EAAE,SAAS,IAAI,CAAC;IACxC,CAAC;GACH;EACF;CACF;AACF;AAmCA,MAAM,eAAe;AAErB,MAAM,iBAAiB;CACrB;CACA;CACA;CACA;CACA;CACA;AACF,CAAC,CAAC,KAAK,IAAI;AAEX,MAAM,eAAyB;CAC7B,MAAM;CACN,UAAU;EACR,MAAM;EACN,aACE;EACF,YAAY;GACV,MAAM;GACN,YAAY,EAAE,MAAM;IAAE,MAAM;IAAU,aAAa;GAA2C,EAAE;GAChG,UAAU,CAAC,MAAM;EACnB;CACF;AACF;;AAGA,eAAe,iBACb,QACA,OACA,IACY;CACZ,MAAM,QAAQ,MAAM,YAAY;EAAC;EAAQ;EAAW;EAAa;EAAQ;EAAW;CAAK,GAAG,EAC1F,KAAK,UACP,CAAC;CACD,MAAM,KAAK,gBAAgB,EAAE,OAAO,MAAM,OAAO,CAAC;CAClD,MAAM,UAAyC,CAAC;CAChD,MAAM,UAAoB,CAAC;CAC3B,GAAG,GAAG,SAAS,MAAM;EACnB,MAAM,OAAO,QAAQ,MAAM;EAC3B,IAAI,MAAM,KAAK,CAAC;OACX,QAAQ,KAAK,CAAC;CACrB,CAAC;CACD,IAAI,SAAS;CACb,MAAM,OAAO,GAAG,SAAS,MAAc;EACrC,UAAU,EAAE,SAAS,MAAM;CAC7B,CAAC;CACD,MAAM,YAAY,cAChB,IAAI,SAAS,SAAS,WAAW;EAC/B,MAAM,cAAc,QAAQ,MAAM;EAClC,IAAI,gBAAgB,KAAA,GAAW,OAAO,QAAQ,WAAW;EACzD,MAAM,IAAI,iBACF,uBAAO,IAAI,MAAM,oCAAoC,UAAU,cAAc,OAAO,MAAM,IAAI,GAAG,CAAC,GACxG,SACF;EAGA,MAAM,UAAU,SAA8B;GAC5C,aAAa,CAAC;GACd,uBAAO,IAAI,MAAM,4BAA4B,KAAK,aAAa,OAAO,MAAM,IAAI,GAAG,CAAC;EACtF;EACA,QAAQ,MAAM,MAAM;GAClB,aAAa,CAAC;GACd,MAAM,eAAe,QAAQ,MAAM;GACnC,QAAQ,CAAC;EACX,CAAC;EACD,MAAM,KAAK,QAAQ,MAAM;CAC3B,CAAC;CACH,IAAI;EACF,MAAM,QAAQ,KAAK,MAAM,MAAM,SAAS,IAAO,CAAC;EAChD,IAAI,CAAC,MAAM,OAAO,MAAM,IAAI,MAAM,qCAAqC,MAAM,SAAS,iBAAiB;EACvG,MAAM,OAAO,OAAO,QAAmE;GACrF,MAAM,MAAM,MAAM,GAAG,KAAK,UAAU,GAAG,EAAE,GAAG;GAC5C,MAAM,MAAM,KAAK,MAAM,MAAM,SAAS,IAAO,CAAC;GAC9C,IAAI,OAAO,IAAI,UAAU,UAAU,MAAM,IAAI,MAAM,+BAA+B,IAAI,OAAO;GAC7F,OAAO;EACT;EACA,OAAO,MAAM,GAAG,MAAM,MAAM,eAAe,EAAE;CAC/C,UAAU;EACR,MAAM,MAAM,IAAI;EAChB,MAAM,KAAK,SAAS;CACtB;AACF;;AAGA,SAAgB,uBAAuB,KAK3B;CACV,MAAM,WAAW,IAAI,YAAY,OAAO,QAAQ,IAAI,mBAAmB,EAAE;CACzE,IAAI,MAAM;CACV,OAAO,EACL,MAAM,SAAS;EAEb,OAAO;GACL,IAAA,qBAF8B;GAG9B,OAAO,aAAa,QAAgB;IAClC,MAAM,IAAI,OAAO,MAAM,YAAY;IACnC,IAAI,CAAC,GACH,MAAM,IAAI,MACR,wFAAwF,OAAO,MAAM,GAAG,GAAG,GAC7G;IAEF,MAAM,GAAG,QAAQ,SAAS;IAC1B,MAAM,YAAY,OAAO,QAAQ,cAAc,EAAE,CAAC,CAAC,KAAK;IACxD,MAAM,MAAM,MAAM,iBAAiB,QAAkB,OAAiB,OAAO,MAAM,gBAAgB;KACjG,MAAM,SAAS,YAAY,GAAG,eAAe,MAAM,cAAc;KACjE,MAAM,OAAO,MAAM,eACjB;MAAE,eAAe,IAAI;MAAe,WAAW,IAAI;MAAW,OAAO,IAAI;KAAM,GAC/E,QACA,SAAS,eACT,CAAC,YAAY,GACb,OAAO,MAAM,SAAS;MACpB,IAAI,SAAS,kBAAkB,OAAO,uBAAuB;MAC7D,MAAM,MAAM,MAAM,KAAK;OAAE,IAAI;OAAW,MAAM,OAAO,KAAK,QAAQ,EAAE;MAAE,CAAC;MACvE,MAAM,OAAO,IAAI,mBAAmB;MACpC,OAAO,GAAG,OAAO,IAAI,UAAU,EAAE,IAAI,OAAO,2FAA2F;KACzI,GACA,EAAE,SAAS,CACb;KACA,MAAM,UAAW,MAAM,KAAK,EAAE,IAAI,WAAW,CAAC;KAC9C,MAAM,aAAa,KAAK,UACrB,MAAM,EAAE,CAAC,CACT,KAAK,MAAM,UAAU,EAAE,KAAK,MAAM,GAAG,GAAG,EAAE,aAAa,EAAE,OAAO,MAAM,GAAG,GAAG,GAAG,CAAC,CAChF,KAAK,SAAS,CAAC,CACf,MAAM,GAAG,IAAI;KAChB,OAAO;MACL,GAAG;MACH,OAAO,KAAK;MACZ,cAAc,KAAK,MAAM;MACzB,eAAe,KAAK,MAAM;MAC1B;KACF;IACF,CAAC;IAED,IAAI,IAAI,gBAAgB,IAAI,eAC1B,MAAM;KACJ,MAAM;KACN,MAAM;MAAE,UAAU,IAAI,gBAAgB;MAAG,WAAW,IAAI,iBAAiB;MAAG,OAAO,IAAI;KAAM;IAC/F;IAEF,MAAM;KAAE,MAAM;KAAU,MAAM,EAAE,WAAW,KAAK,UAAU,GAAG,EAAE;IAAE;GACnE;GACA,MAAM,SAAS,CAAC;EAClB;CACF,EACF;AACF;;AAGA,MAAM,qBAA4C,EAChD,MAAM,QAAQ;CACZ,IAAI,OAAO;CACX,KAAK,MAAM,MAAM,QAAQ;EAEvB,MAAM,KADK,IAA2C,KAAA,EACzC;EACb,IAAI,OAAO,MAAM,YAAY,EAAE,SAAS,GAAG,OAAO;CACpD;CACA,OAAO;AACT,EACF;AAEA,SAAgB,6BAA+C;CAC7D,MAAM,OAAO,sBAAsB;CACnC,OAAO;EACL,MAAM;EACN,QAAQ;EACR,iBAAiB,KAAK,UAAU;EAEhC,MAAM,UAAU,OAAoB,CAAC,GAAyB;GAE5D,QAAO,MADa,KAAK,UAAU,IAAI,EAAA,CAC1B,KAAK,MAAM;IACtB,MAAM,OAAO,SAAS,CAAC;IACvB,OAAO;KACL,GAAG;KAGH,QAAQ,mBAAmB,KAAK,OAAO,GAAG,KAAK,MAAM;IACvD;GACF,CAAC;EACH;EAEA,oBAAoB,QAAQ,QAAQ,KAAA,CAAS;EAE7C,MAAM,MAAM,MAAiB,UAAuC;GAClE,MAAM,OAAO,SAAS,IAAI;GAC1B,IAAI;GACJ,IAAI;IACF,MAAM,KAAK,MAAM,QAAQ;GAC3B,QAAQ;IACN,MAAM,IAAI,MACR,6EAA6E,KAAK,OAAO,KAAK,SAAS,MAAM,GAAG,GAAG,GACrH;GACF;GACA,IAAI,OAAO,IAAI,YAAY,aAAa,OAAO,IAAI,cAAc,UAC/D,MAAM,IAAI,MACR,sEAAsE,KAAK,OAAO,KAAK,SAAS,MAAM,GAAG,GAAG,GAC9G;GAEF,MAAM,SAAS,IAAI,UAAU;GAC7B,MAAM,QAAQ,IAAI;GAClB,MAAM,WAAW,MAAM,QAAQ,IAAI,aAAa,IAAI,IAAI,gBAAgB,CAAC;GACzE,OAAO;IACL,UAAU,IAAI,YAAY;IAC1B,OAAO,QAAQ,IAAI,SAAS,QAAQ;IACpC,QAAQ,KAAK,UAAU;KACrB,QAAQ,KAAK;KACb,SAAS,IAAI;KACb;KACA,OAAO,IAAI,SAAS;KACpB;KACA,OAAO,IAAI;KACX,GAAI,SAAS,SAAS,EAAE,SAAS,IAAI,CAAC;KACtC,GAAI,IAAI,aAAa,EAAE,gBAAgB,IAAI,WAAW,MAAM,IAAI,EAAE,IAAI,CAAC;IACzE,CAAC;GACH;EACF;EAEA,aAAa,MAAM,uBAAuB,CAAC;CAC7C;AACF"}
+{"version":3,"file":"appworld.js","names":[],"sources":["../../src/benchmarks/appworld.ts"],"sourcesContent":["/**\n * AppWorld adapter (StonyBrookNLP/appworld). Worker artifact = the agent's\n * Python solution that calls the simulated apps' APIs (the same `apis.<app>.<fn>`\n * surface AppWorld exposes inside `world.execute(...)`), ending in\n * `apis.supervisor.complete_task()`. Judge = AppWorld's OWN programmatic\n * evaluator: a driver runs the solution in a fresh `AppWorld(task_id=...)` world,\n * then `world.evaluate().to_dict()` reports `success` (binary TGC), `num_tests`\n * (per-requirement total) and the `passes`/`failures` lists. Score =\n * passes / num_tests — GRADED; resolved = success. Fully deterministic — no LLM judge.\n *\n * loadTasks enumerates the real task suite via `load_task_ids(split)`\n * (train|dev|test_normal|test_challenge); the prompt = `world.task.instruction`.\n * The OutputAdapter is stream-only, so the worker emits its solution as a fenced\n * ```python block which the driver executes.\n *\n * Requires for a live run: the bench `.venv` with `appworld` installed + the\n * unpacked engine + downloaded data (`appworld install` ; `appworld download\n * data`). preflight + loadTasks + judge all fail loud with the exact step when the\n * engine/data is absent — never a fabricated score.\n */\n\nimport { spawn } from 'node:child_process'\nimport { join } from 'node:path'\nimport { createInterface } from 'node:readline'\nimport {\n  collectAgentTurn,\n  createExecutor,\n  type OutputAdapter,\n  streamAgentTurn,\n  type ToolSpec,\n} from '@tangle-network/agent-runtime/kernel'\nimport { benchRoot, preflightVenvImports, runVenvScriptStdin, venvPython } from './_harness'\nimport type { BenchmarkAdapter, BenchScore, BenchTask, LoadOptions } from './types'\n\nconst DRIVER = join(benchRoot, 'scripts', 'appworld_driver.py')\n\n/** AppWorld splits; only the test splits ship evaluation-only (no setup/solution). */\nconst DEFAULT_SPLIT = 'test_normal'\n\ninterface AppWorldMeta {\n  taskId: string\n  split: string\n}\n\n/** Worker solution code = the last fenced ```python block, else the raw text. */\nexport const appworldSolutionOutput: OutputAdapter<string> = {\n  parse(events) {\n    let text = ''\n    for (const ev of events) {\n      const d = (ev as { data?: Record<string, unknown> })?.data\n      const t = d?.finalText ?? d?.text ?? d?.result\n      if (typeof t === 'string' && t.length > 0) text = t\n    }\n    const fences = [...text.matchAll(/```(?:python|py)?\\s*\\n([\\s\\S]*?)```/g)]\n    return (fences.at(-1)?.[1] ?? text).trim()\n  },\n}\n\nconst WORKER_CONTRACT = [\n  '',\n  'Solve this by writing Python that calls the available app APIs (the `apis.<app>.<function>(...)` surface). You may inspect API docs with `apis.api_docs.show_api_descriptions(app_name=...)` and `apis.api_docs.show_api_doc(app_name=..., api_name=...)`.',\n  'Authenticate where needed via the supervisor-provided credentials, perform every step the task requires, and FINISH with `apis.supervisor.complete_task()`.',\n  'Emit your COMPLETE solution as the LAST thing in your reply, in a single fenced ```python block. Nothing after the closing fence.',\n].join('\\n')\n\nfunction readMeta(task: BenchTask): AppWorldMeta {\n  const md = task.metadata\n  if (!md || typeof md.taskId !== 'string') {\n    throw new Error(`appworld task ${task.id} missing metadata.taskId — loadTasks did not populate it`)\n  }\n  return md as unknown as AppWorldMeta\n}\n\n/**\n * Run the appworld engine driver with a subcommand; JSON on the LAST stdout line.\n * The solution code (evaluate) is piped to stdin via the shared stdin-aware runner —\n * execFile's `input` option is not honored async and hangs the driver's\n * sys.stdin.read() forever. `load` ignores stdin, so an empty pipe is harmless.\n */\nasync function driver(args: string[], input = ''): Promise<unknown> {\n  let stdout: string\n  try {\n    stdout = await runVenvScriptStdin(DRIVER, args, input, { cwd: benchRoot })\n  } catch (err) {\n    const e = err as { message?: string }\n    throw new Error(`appworld driver failed (${args.join(' ')}): ${(e.message || String(err)).slice(0, 1500)}`)\n  }\n  const last = stdout.trim().split('\\n').at(-1) ?? '{}'\n  const parsed = JSON.parse(last) as { error?: string }\n  if (parsed.error) throw new Error(`appworld driver error: ${parsed.error}`)\n  return parsed\n}\n\nexport function createAppWorldAdapter(): BenchmarkAdapter {\n  return {\n    name: 'appworld',\n    output: appworldSolutionOutput,\n\n    async preflight() {\n      await preflightVenvImports({\n        modules: ['appworld'],\n        requireDocker: false,\n        fix:\n          'Fix: bench/.venv/bin/pip install appworld ; ' +\n          'bench/.venv/bin/appworld install ; bench/.venv/bin/appworld download data ' +\n          '(unpacks the engine + downloads the simulated-app data/tasks). ' +\n          'Set APPWORLD_ROOT to the data root if not the default.',\n      })\n    },\n\n    async loadTasks(opts: LoadOptions = {}): Promise<BenchTask[]> {\n      const split = opts.split ?? DEFAULT_SPLIT\n      const out = (await driver([\n        'load',\n        '--split', split,\n        ...(opts.limit !== undefined ? ['--limit', String(opts.limit)] : []),\n        ...(opts.ids ? ['--ids', opts.ids.join(',')] : []),\n      ])) as { tasks?: Array<{ task_id: string; instruction: string }> }\n      const tasks = out.tasks ?? []\n      if (tasks.length === 0) {\n        throw new Error(`appworld loadTasks returned no tasks for split=${split} ${JSON.stringify(opts)}`)\n      }\n      return tasks.map(\n        (t): BenchTask => ({\n          id: t.task_id,\n          split,\n          prompt: t.instruction + WORKER_CONTRACT,\n          metadata: { taskId: t.task_id, split } as unknown as Record<string, unknown>,\n        }),\n      )\n    },\n\n    async goldArtifact() {\n      // Reference solution code ships only for train/dev, and only inside the\n      // engine's decrypted `.bundle` (it is not a portable string this adapter can\n      // emit across splits). The test splits are evaluation-only. So verify-judge\n      // here requires a real solve on a train/dev task through the live engine\n      // rather than a synthetic gold — returning a fabricated artifact would be a\n      // fake. Returns undefined.\n      return undefined\n    },\n\n    async judge(task: BenchTask, artifact: string): Promise<BenchScore> {\n      const meta = readMeta(task)\n      const out = (await driver(['evaluate', '--task-id', meta.taskId, '--split', meta.split], artifact)) as {\n        success?: boolean\n        passes?: number\n        fails?: number\n        num_tests?: number\n        failure_names?: string[]\n      }\n      const passes = out.passes ?? 0\n      const fails = out.fails ?? 0\n      // num_tests is the evaluator's authoritative per-requirement count; prefer it\n      // over passes+fails (which can disagree if a requirement neither passed nor\n      // failed). Never default the total to a phantom denominator.\n      const total = out.num_tests ?? passes + fails\n      const score = total > 0 ? passes / total : 0\n      // failure_names = WHICH sub-tests failed — the evidence a trace analyst\n      // steers on. Carried in `detail` so it reaches the verdict's `notes`.\n      const failures = Array.isArray(out.failure_names) ? out.failure_names : []\n      return {\n        resolved: out.success === true,\n        score,\n        detail: JSON.stringify({\n          taskId: meta.taskId,\n          success: out.success,\n          passes,\n          fails,\n          total,\n          ...(failures.length ? { failures } : {}),\n        }),\n      }\n    },\n  }\n}\n\n/**\n * AppWorld in its NATIVE protocol, run by OUR runtime: the worker is\n * Runtime's profile-bound `router-tools` executor with one tool —\n * `execute_python` — bound to a persistent AppWorld world session. The driver's\n * `session` subcommand is a dumb world shim (stdin JSONL: execute → output,\n * evaluate → verdict); every inference turn, the metering, and the typed\n * toolTrace the analyst steers on belong to the runtime, so runtime\n * improvements are what this benchmark measures.\n *\n * The one-shot codegen adapter above plays a strictly harder game (no execution\n * feedback — the first wrong API call kills the whole program at judge time),\n * which flatlines the score against ANY steering; this mode is what the\n * benchmark's published baselines use, where behavior can move sub-tests.\n *\n * Protocol: the round task string is `@appworld-react <taskId> <split>` on\n * line 1; everything after line 1 is the steer (an analyst correction, a push\n * directive) appended to the system prompt — so the existing arms steer this\n * worker without modification. The artifact is the episode evaluation JSON\n * (AppWorld's evaluator ran in-world); judge() parses it, never re-executes.\n */\n\nexport interface ReactResult {\n  success?: boolean\n  passes?: number\n  fails?: number\n  num_tests?: number\n  failure_names?: string[]\n  turns?: number\n  input_tokens?: number\n  output_tokens?: number\n  cost_usd?: number\n  transcript?: string\n}\n\ninterface ReactRuntimeUsage {\n  input: number\n  output: number\n  costUsd?: number\n  tokensKnown?: boolean\n  usdKnown?: boolean\n}\n\n/** Preserve a completed scientific/task result even when one accounting dimension is incomplete.\n * Unknown usage fields stay absent; later comparison/reporting can refuse a cost claim without\n * discarding the episode's task evidence. */\nexport function appworldReactResultWithUsage(\n  verdict: ReactResult,\n  usage: ReactRuntimeUsage,\n  turns: number | undefined,\n  transcript: string,\n): ReactResult {\n  return {\n    ...verdict,\n    ...(turns !== undefined ? { turns } : {}),\n    ...(usage.tokensKnown === false\n      ? {}\n      : { input_tokens: usage.input, output_tokens: usage.output }),\n    ...(usage.usdKnown === false || usage.costUsd === undefined\n      ? {}\n      : { cost_usd: usage.costUsd }),\n    transcript,\n  }\n}\n\n/** Emit only usage the Runtime actually knows. Catalog estimates never become observed dollars. */\nexport function appworldReactUsageEvent(\n  result: ReactResult,\n  model: string,\n): { type: 'llm_call'; data: Record<string, unknown> } | undefined {\n  const hasTokens =\n    typeof result.input_tokens === 'number' && typeof result.output_tokens === 'number'\n  const hasCost = typeof result.cost_usd === 'number'\n  if (!hasTokens && !hasCost) return undefined\n  return {\n    type: 'llm_call',\n    data: {\n      model,\n      ...(hasTokens\n        ? { tokensIn: result.input_tokens, tokensOut: result.output_tokens }\n        : {}),\n      ...(hasCost ? { costUsd: result.cost_usd } : {}),\n    },\n  }\n}\n\nconst REACT_HEADER = /^@appworld-react (\\S+) (\\S+)\\n?/\n\nconst SESSION_SYSTEM = [\n  'You are completing a task in AppWorld, a simulated multi-app environment.',\n  'Use the execute_python tool to run Python that calls the app APIs (the `apis.<app>.<function>(...)` surface).',\n  'Inspect API docs with `apis.api_docs.show_api_descriptions(app_name=...)` and `apis.api_docs.show_api_doc(app_name=..., api_name=...)`.',\n  'Authenticate where needed via the supervisor-provided credentials (`apis.supervisor.show_account_passwords()`).',\n  'Work incrementally: small snippets, read each output, correct course.',\n  'When every step of the task is done, run `apis.supervisor.complete_task()` and then reply WITHOUT calling the tool again.',\n].join('\\n')\n\nconst EXECUTE_TOOL: ToolSpec = {\n  type: 'function',\n  function: {\n    name: 'execute_python',\n    description:\n      'Execute a Python snippet in the persistent AppWorld world. State persists across calls. Returns the execution output (API results or errors).',\n    parameters: {\n      type: 'object',\n      properties: { code: { type: 'string', description: 'Python code calling apis.<app>.<fn>(...)' } },\n      required: ['code'],\n    },\n  },\n}\n\n/** One persistent world session: line-JSONL request/response over the driver. */\nasync function withWorldSession<T>(\n  taskId: string,\n  split: string,\n  signal: AbortSignal,\n  fn: (call: (cmd: Record<string, unknown>) => Promise<Record<string, unknown>>, instruction: string) => Promise<T>,\n): Promise<T> {\n  signal.throwIfAborted()\n  const child = spawn(venvPython, [DRIVER, 'session', '--task-id', taskId, '--split', split], {\n    cwd: benchRoot,\n  })\n  const stopChild = (): void => {\n    if (!child.killed) child.kill('SIGTERM')\n  }\n  signal.addEventListener('abort', stopChild, { once: true })\n  const rl = createInterface({ input: child.stdout })\n  const pending: Array<(line: string) => void> = []\n  const backlog: string[] = []\n  rl.on('line', (l) => {\n    const next = pending.shift()\n    if (next) next(l)\n    else backlog.push(l)\n  })\n  let stderr = ''\n  child.stderr.on('data', (c: Buffer) => {\n    stderr += c.toString('utf8')\n  })\n  const nextLine = (timeoutMs: number): Promise<string> =>\n    new Promise((resolve, reject) => {\n      if (signal.aborted) {\n        reject(signal.reason)\n        return\n      }\n      const fromBacklog = backlog.shift()\n      if (fromBacklog !== undefined) return resolve(fromBacklog)\n      const t = setTimeout(\n        () => reject(new Error(`appworld session: no response in ${timeoutMs}ms; stderr: ${stderr.slice(-400)}`)),\n        timeoutMs,\n      )\n      // One exit listener per await leaks (25-turn episodes blow the listener\n      // cap) — remove it on the resolve path.\n      const onExit = (code: number | null): void => {\n        clearTimeout(t)\n        signal.removeEventListener('abort', onAbort)\n        reject(\n          signal.aborted\n            ? signal.reason\n            : new Error(`appworld session exited (${code}); stderr: ${stderr.slice(-400)}`),\n        )\n      }\n      const onAbort = (): void => {\n        clearTimeout(t)\n        child.removeListener('exit', onExit)\n        const index = pending.indexOf(onLine)\n        if (index >= 0) pending.splice(index, 1)\n        reject(signal.reason)\n      }\n      const onLine = (line: string): void => {\n        clearTimeout(t)\n        child.removeListener('exit', onExit)\n        signal.removeEventListener('abort', onAbort)\n        resolve(line)\n      }\n      pending.push(onLine)\n      child.once('exit', onExit)\n      signal.addEventListener('abort', onAbort, { once: true })\n    })\n  try {\n    const ready = JSON.parse(await nextLine(120_000)) as { ready?: boolean; instruction?: string; error?: string }\n    if (!ready.ready) throw new Error(`appworld session failed to start: ${ready.error ?? 'no ready line'}`)\n    const call = async (cmd: Record<string, unknown>): Promise<Record<string, unknown>> => {\n      child.stdin.write(`${JSON.stringify(cmd)}\\n`)\n      const res = JSON.parse(await nextLine(180_000)) as Record<string, unknown>\n      if (typeof res.error === 'string') throw new Error(`appworld session op failed: ${res.error}`)\n      return res\n    }\n    return await fn(call, ready.instruction ?? '')\n  } finally {\n    signal.removeEventListener('abort', stopChild)\n    child.stdin.end()\n    stopChild()\n  }\n}\n\ntype AppWorldWorldSession = typeof withWorldSession\ntype AppWorldComplete = (\n  body: Record<string, unknown>,\n  request?: {\n    readonly headers: Readonly<Record<string, string>>\n    readonly signal?: AbortSignal\n  },\n) => Promise<unknown>\n\n/** SandboxClient whose leaf is Runtime's profile-bound Router executor driving a world session. */\nexport function appworldToolLoopClient(cfg: {\n  model: string\n  routerBaseUrl: string\n  routerKey: string\n  maxTurns?: number\n  /** Offline-test seam; production always uses the Python AppWorld session above. */\n  runWorldSession?: AppWorldWorldSession\n  /** Offline-test seam; production uses Runtime's Router HTTP transport. */\n  complete?: AppWorldComplete\n}): unknown {\n  const maxTurns = cfg.maxTurns ?? Number(process.env.REACT_MAX_TURNS ?? 40)\n  const runWorldSession = cfg.runWorldSession ?? withWorldSession\n  let seq = 0\n  return {\n    async create() {\n      const id = `appworld-toolloop-${seq++}`\n      return {\n        id,\n        async *streamPrompt(prompt: string, promptOpts?: { signal?: AbortSignal }) {\n          const signal = promptOpts?.signal ?? new AbortController().signal\n          signal.throwIfAborted()\n          const m = prompt.match(REACT_HEADER)\n          if (!m) {\n            throw new Error(\n              `appworld-react leaf: prompt missing '@appworld-react <taskId> <split>' header — got: ${prompt.slice(0, 120)}`,\n            )\n          }\n          const [, taskId, split] = m\n          const directive = prompt.replace(REACT_HEADER, '').trim()\n          const out = await runWorldSession(taskId as string, split as string, signal, async (call, instruction) => {\n            const system = directive ? `${SESSION_SYSTEM}\\n\\n${directive}` : SESSION_SYSTEM\n            const transcriptSteps: Array<{ args: string; result: string }> = []\n            const profile = {\n              name: 'appworld-react-worker',\n              harness: 'cli-base' as const,\n              model: {\n                provider: 'tangle-router',\n                default: cfg.model,\n                metadata: { maxTurns },\n              },\n              prompt: { systemPrompt: system },\n              tools: { execute_python: true },\n            }\n            const factory = createExecutor({\n              backend: 'router-tools',\n              routerBaseUrl: cfg.routerBaseUrl,\n              routerKey: cfg.routerKey,\n              ...(cfg.complete ? { complete: cfg.complete } : {}),\n              tools: [EXECUTE_TOOL],\n              executeToolCall: async (name, args) => {\n                if (name !== 'execute_python') return `error: unknown tool ${name}`\n                const res = await call({ op: 'execute', code: String(args.code ?? '') })\n                const done = res.task_completed === true\n                const result = `${String(res.output ?? '')}${done ? '\\n\\n[TASK MARKED COMPLETE — reply with a final summary and do not call the tool again]' : ''}`\n                transcriptSteps.push({ args: JSON.stringify(args), result })\n                return result\n              },\n            })\n            const loop = await collectAgentTurn(\n              streamAgentTurn(\n                { kind: 'executor', factory, profile },\n                `Task: ${instruction}`,\n                { signal },\n              ),\n            )\n            if (loop.status !== 'completed') {\n              throw new Error(loop.error?.message ?? `AppWorld turn ended with ${loop.status}`)\n            }\n            const verdict = (await call({ op: 'evaluate' })) as unknown as ReactResult\n            const transcript = transcriptSteps\n              .slice(-3)\n              .map((t) => `CODE:\\n${t.args.slice(0, 600)}\\nOUTPUT:\\n${t.result.slice(0, 600)}`)\n              .join('\\n---\\n')\n              .slice(0, 1600)\n            const finalEvent = loop.events.at(-1)\n            const resultMetadata =\n              finalEvent?.type === 'final' && finalEvent.metadata?.result\n                ? (finalEvent.metadata.result as { spent?: { iterations?: number } })\n                : undefined\n            return appworldReactResultWithUsage(\n              verdict,\n              loop.usage,\n              resultMetadata?.spent?.iterations,\n              transcript,\n            )\n          })\n          const usageEvent = appworldReactUsageEvent(out, cfg.model)\n          if (usageEvent) yield usageEvent\n          yield { type: 'result', data: { finalText: JSON.stringify(out) } }\n        },\n        async delete() {},\n      }\n    },\n  }\n}\n\n/** Artifact = the episode's evaluation JSON, verbatim (no fence extraction). */\nconst reactEpisodeOutput: OutputAdapter<string> = {\n  parse(events) {\n    let text = ''\n    for (const ev of events) {\n      const d = (ev as { data?: Record<string, unknown> })?.data\n      const t = d?.finalText\n      if (typeof t === 'string' && t.length > 0) text = t\n    }\n    return text\n  },\n}\n\nexport function createAppWorldReactAdapter(): BenchmarkAdapter {\n  const base = createAppWorldAdapter()\n  return {\n    name: 'appworld-react',\n    output: reactEpisodeOutput,\n    preflight: () => base.preflight(),\n\n    async loadTasks(opts: LoadOptions = {}): Promise<BenchTask[]> {\n      const tasks = await base.loadTasks(opts)\n      return tasks.map((t) => {\n        const meta = readMeta(t)\n        return {\n          ...t,\n          // Header carries task identity to the leaf; the body (empty at round 0)\n          // is the directive slot the arms append their steer into.\n          prompt: `@appworld-react ${meta.taskId} ${meta.split}\\n`,\n        }\n      })\n    },\n\n    goldArtifact: () => Promise.resolve(undefined),\n\n    async judge(task: BenchTask, artifact: string): Promise<BenchScore> {\n      const meta = readMeta(task)\n      let out: ReactResult\n      try {\n        out = JSON.parse(artifact) as ReactResult\n      } catch {\n        throw new Error(\n          `appworld-react judge: artifact is not the episode's evaluation JSON (task ${meta.taskId}): ${artifact.slice(0, 200)}`,\n        )\n      }\n      if (typeof out.success !== 'boolean' || typeof out.num_tests !== 'number') {\n        throw new Error(\n          `appworld-react judge: episode JSON missing success/num_tests (task ${meta.taskId}): ${artifact.slice(0, 200)}`,\n        )\n      }\n      const passes = out.passes ?? 0\n      const total = out.num_tests\n      const failures = Array.isArray(out.failure_names) ? out.failure_names : []\n      return {\n        resolved: out.success === true,\n        score: total > 0 ? passes / total : 0,\n        detail: JSON.stringify({\n          taskId: meta.taskId,\n          success: out.success,\n          passes,\n          fails: out.fails ?? 0,\n          total,\n          turns: out.turns,\n          ...(failures.length ? { failures } : {}),\n          ...(out.transcript ? { transcriptTail: out.transcript.slice(-800) } : {}),\n        }),\n      }\n    },\n\n    leafClient: (c) => appworldToolLoopClient(c),\n  }\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;;;;;;;;;AAkCA,MAAM,SAAS,KAAK,WAAW,WAAW,oBAAoB;;AAG9D,MAAM,gBAAgB;;AAQtB,MAAa,yBAAgD,EAC3D,MAAM,QAAQ;CACZ,IAAI,OAAO;CACX,KAAK,MAAM,MAAM,QAAQ;EACvB,MAAM,IAAK,IAA2C;EACtD,MAAM,IAAI,GAAG,aAAa,GAAG,QAAQ,GAAG;EACxC,IAAI,OAAO,MAAM,YAAY,EAAE,SAAS,GAAG,OAAO;CACpD;CAEA,QAAQ,CADQ,GAAG,KAAK,SAAS,sCAAsC,CAC1D,CAAC,CAAC,GAAG,EAAE,CAAC,GAAG,MAAM,KAAA,CAAM,KAAK;AAC3C,EACF;AAEA,MAAM,kBAAkB;CACtB;CACA;CACA;CACA;AACF,CAAC,CAAC,KAAK,IAAI;AAEX,SAAS,SAAS,MAA+B;CAC/C,MAAM,KAAK,KAAK;CAChB,IAAI,CAAC,MAAM,OAAO,GAAG,WAAW,UAC9B,MAAM,IAAI,MAAM,iBAAiB,KAAK,GAAG,yDAAyD;CAEpG,OAAO;AACT;;;;;;;AAQA,eAAe,OAAO,MAAgB,QAAQ,IAAsB;CAClE,IAAI;CACJ,IAAI;EACF,SAAS,MAAM,mBAAmB,QAAQ,MAAM,OAAO,EAAE,KAAK,UAAU,CAAC;CAC3E,SAAS,KAAK;EACZ,MAAM,IAAI;EACV,MAAM,IAAI,MAAM,2BAA2B,KAAK,KAAK,GAAG,EAAE,MAAM,EAAE,WAAW,OAAO,GAAG,EAAA,CAAG,MAAM,GAAG,IAAI,GAAG;CAC5G;CACA,MAAM,OAAO,OAAO,KAAK,CAAC,CAAC,MAAM,IAAI,CAAC,CAAC,GAAG,EAAE,KAAK;CACjD,MAAM,SAAS,KAAK,MAAM,IAAI;CAC9B,IAAI,OAAO,OAAO,MAAM,IAAI,MAAM,0BAA0B,OAAO,OAAO;CAC1E,OAAO;AACT;AAEA,SAAgB,wBAA0C;CACxD,OAAO;EACL,MAAM;EACN,QAAQ;EAER,MAAM,YAAY;GAChB,MAAM,qBAAqB;IACzB,SAAS,CAAC,UAAU;IACpB,eAAe;IACf,KACE;GAIJ,CAAC;EACH;EAEA,MAAM,UAAU,OAAoB,CAAC,GAAyB;GAC5D,MAAM,QAAQ,KAAK,SAAS;GAO5B,MAAM,SAAQ,MANK,OAAO;IACxB;IACA;IAAW;IACX,GAAI,KAAK,UAAU,KAAA,IAAY,CAAC,WAAW,OAAO,KAAK,KAAK,CAAC,IAAI,CAAC;IAClE,GAAI,KAAK,MAAM,CAAC,SAAS,KAAK,IAAI,KAAK,GAAG,CAAC,IAAI,CAAC;GAClD,CAAC,EAAA,CACiB,SAAS,CAAC;GAC5B,IAAI,MAAM,WAAW,GACnB,MAAM,IAAI,MAAM,kDAAkD,MAAM,GAAG,KAAK,UAAU,IAAI,GAAG;GAEnG,OAAO,MAAM,KACV,OAAkB;IACjB,IAAI,EAAE;IACN;IACA,QAAQ,EAAE,cAAc;IACxB,UAAU;KAAE,QAAQ,EAAE;KAAS;IAAM;GACvC,EACF;EACF;EAEA,MAAM,eAAe,CAQrB;EAEA,MAAM,MAAM,MAAiB,UAAuC;GAClE,MAAM,OAAO,SAAS,IAAI;GAC1B,MAAM,MAAO,MAAM,OAAO;IAAC;IAAY;IAAa,KAAK;IAAQ;IAAW,KAAK;GAAK,GAAG,QAAQ;GAOjG,MAAM,SAAS,IAAI,UAAU;GAC7B,MAAM,QAAQ,IAAI,SAAS;GAI3B,MAAM,QAAQ,IAAI,aAAa,SAAS;GACxC,MAAM,QAAQ,QAAQ,IAAI,SAAS,QAAQ;GAG3C,MAAM,WAAW,MAAM,QAAQ,IAAI,aAAa,IAAI,IAAI,gBAAgB,CAAC;GACzE,OAAO;IACL,UAAU,IAAI,YAAY;IAC1B;IACA,QAAQ,KAAK,UAAU;KACrB,QAAQ,KAAK;KACb,SAAS,IAAI;KACb;KACA;KACA;KACA,GAAI,SAAS,SAAS,EAAE,SAAS,IAAI,CAAC;IACxC,CAAC;GACH;EACF;CACF;AACF;;;;AA+CA,SAAgB,6BACd,SACA,OACA,OACA,YACa;CACb,OAAO;EACL,GAAG;EACH,GAAI,UAAU,KAAA,IAAY,EAAE,MAAM,IAAI,CAAC;EACvC,GAAI,MAAM,gBAAgB,QACtB,CAAC,IACD;GAAE,cAAc,MAAM;GAAO,eAAe,MAAM;EAAO;EAC7D,GAAI,MAAM,aAAa,SAAS,MAAM,YAAY,KAAA,IAC9C,CAAC,IACD,EAAE,UAAU,MAAM,QAAQ;EAC9B;CACF;AACF;;AAGA,SAAgB,wBACd,QACA,OACiE;CACjE,MAAM,YACJ,OAAO,OAAO,iBAAiB,YAAY,OAAO,OAAO,kBAAkB;CAC7E,MAAM,UAAU,OAAO,OAAO,aAAa;CAC3C,IAAI,CAAC,aAAa,CAAC,SAAS,OAAO,KAAA;CACnC,OAAO;EACL,MAAM;EACN,MAAM;GACJ;GACA,GAAI,YACA;IAAE,UAAU,OAAO;IAAc,WAAW,OAAO;GAAc,IACjE,CAAC;GACL,GAAI,UAAU,EAAE,SAAS,OAAO,SAAS,IAAI,CAAC;EAChD;CACF;AACF;AAEA,MAAM,eAAe;AAErB,MAAM,iBAAiB;CACrB;CACA;CACA;CACA;CACA;CACA;AACF,CAAC,CAAC,KAAK,IAAI;AAEX,MAAM,eAAyB;CAC7B,MAAM;CACN,UAAU;EACR,MAAM;EACN,aACE;EACF,YAAY;GACV,MAAM;GACN,YAAY,EAAE,MAAM;IAAE,MAAM;IAAU,aAAa;GAA2C,EAAE;GAChG,UAAU,CAAC,MAAM;EACnB;CACF;AACF;;AAGA,eAAe,iBACb,QACA,OACA,QACA,IACY;CACZ,OAAO,eAAe;CACtB,MAAM,QAAQ,MAAM,YAAY;EAAC;EAAQ;EAAW;EAAa;EAAQ;EAAW;CAAK,GAAG,EAC1F,KAAK,UACP,CAAC;CACD,MAAM,kBAAwB;EAC5B,IAAI,CAAC,MAAM,QAAQ,MAAM,KAAK,SAAS;CACzC;CACA,OAAO,iBAAiB,SAAS,WAAW,EAAE,MAAM,KAAK,CAAC;CAC1D,MAAM,KAAK,gBAAgB,EAAE,OAAO,MAAM,OAAO,CAAC;CAClD,MAAM,UAAyC,CAAC;CAChD,MAAM,UAAoB,CAAC;CAC3B,GAAG,GAAG,SAAS,MAAM;EACnB,MAAM,OAAO,QAAQ,MAAM;EAC3B,IAAI,MAAM,KAAK,CAAC;OACX,QAAQ,KAAK,CAAC;CACrB,CAAC;CACD,IAAI,SAAS;CACb,MAAM,OAAO,GAAG,SAAS,MAAc;EACrC,UAAU,EAAE,SAAS,MAAM;CAC7B,CAAC;CACD,MAAM,YAAY,cAChB,IAAI,SAAS,SAAS,WAAW;EAC/B,IAAI,OAAO,SAAS;GAClB,OAAO,OAAO,MAAM;GACpB;EACF;EACA,MAAM,cAAc,QAAQ,MAAM;EAClC,IAAI,gBAAgB,KAAA,GAAW,OAAO,QAAQ,WAAW;EACzD,MAAM,IAAI,iBACF,uBAAO,IAAI,MAAM,oCAAoC,UAAU,cAAc,OAAO,MAAM,IAAI,GAAG,CAAC,GACxG,SACF;EAGA,MAAM,UAAU,SAA8B;GAC5C,aAAa,CAAC;GACd,OAAO,oBAAoB,SAAS,OAAO;GAC3C,OACE,OAAO,UACH,OAAO,yBACP,IAAI,MAAM,4BAA4B,KAAK,aAAa,OAAO,MAAM,IAAI,GAAG,CAClF;EACF;EACA,MAAM,gBAAsB;GAC1B,aAAa,CAAC;GACd,MAAM,eAAe,QAAQ,MAAM;GACnC,MAAM,QAAQ,QAAQ,QAAQ,MAAM;GACpC,IAAI,SAAS,GAAG,QAAQ,OAAO,OAAO,CAAC;GACvC,OAAO,OAAO,MAAM;EACtB;EACA,MAAM,UAAU,SAAuB;GACrC,aAAa,CAAC;GACd,MAAM,eAAe,QAAQ,MAAM;GACnC,OAAO,oBAAoB,SAAS,OAAO;GAC3C,QAAQ,IAAI;EACd;EACA,QAAQ,KAAK,MAAM;EACnB,MAAM,KAAK,QAAQ,MAAM;EACzB,OAAO,iBAAiB,SAAS,SAAS,EAAE,MAAM,KAAK,CAAC;CAC1D,CAAC;CACH,IAAI;EACF,MAAM,QAAQ,KAAK,MAAM,MAAM,SAAS,IAAO,CAAC;EAChD,IAAI,CAAC,MAAM,OAAO,MAAM,IAAI,MAAM,qCAAqC,MAAM,SAAS,iBAAiB;EACvG,MAAM,OAAO,OAAO,QAAmE;GACrF,MAAM,MAAM,MAAM,GAAG,KAAK,UAAU,GAAG,EAAE,GAAG;GAC5C,MAAM,MAAM,KAAK,MAAM,MAAM,SAAS,IAAO,CAAC;GAC9C,IAAI,OAAO,IAAI,UAAU,UAAU,MAAM,IAAI,MAAM,+BAA+B,IAAI,OAAO;GAC7F,OAAO;EACT;EACA,OAAO,MAAM,GAAG,MAAM,MAAM,eAAe,EAAE;CAC/C,UAAU;EACR,OAAO,oBAAoB,SAAS,SAAS;EAC7C,MAAM,MAAM,IAAI;EAChB,UAAU;CACZ;AACF;;AAYA,SAAgB,uBAAuB,KAS3B;CACV,MAAM,WAAW,IAAI,YAAY,OAAO,QAAQ,IAAI,mBAAmB,EAAE;CACzE,MAAM,kBAAkB,IAAI,mBAAmB;CAC/C,IAAI,MAAM;CACV,OAAO,EACL,MAAM,SAAS;EAEb,OAAO;GACL,IAAA,qBAF8B;GAG9B,OAAO,aAAa,QAAgB,YAAuC;IACzE,MAAM,SAAS,YAAY,UAAU,IAAI,gBAAgB,CAAC,CAAC;IAC3D,OAAO,eAAe;IACtB,MAAM,IAAI,OAAO,MAAM,YAAY;IACnC,IAAI,CAAC,GACH,MAAM,IAAI,MACR,wFAAwF,OAAO,MAAM,GAAG,GAAG,GAC7G;IAEF,MAAM,GAAG,QAAQ,SAAS;IAC1B,MAAM,YAAY,OAAO,QAAQ,cAAc,EAAE,CAAC,CAAC,KAAK;IACxD,MAAM,MAAM,MAAM,gBAAgB,QAAkB,OAAiB,QAAQ,OAAO,MAAM,gBAAgB;KACxG,MAAM,SAAS,YAAY,GAAG,eAAe,MAAM,cAAc;KACjE,MAAM,kBAA2D,CAAC;KAClE,MAAM,UAAU;MACd,MAAM;MACN,SAAS;MACT,OAAO;OACL,UAAU;OACV,SAAS,IAAI;OACb,UAAU,EAAE,SAAS;MACvB;MACA,QAAQ,EAAE,cAAc,OAAO;MAC/B,OAAO,EAAE,gBAAgB,KAAK;KAChC;KAgBA,MAAM,OAAO,MAAM,iBACjB,gBACE;MAAE,MAAM;MAAY,SAjBR,eAAe;OAC7B,SAAS;OACT,eAAe,IAAI;OACnB,WAAW,IAAI;OACf,GAAI,IAAI,WAAW,EAAE,UAAU,IAAI,SAAS,IAAI,CAAC;OACjD,OAAO,CAAC,YAAY;OACpB,iBAAiB,OAAO,MAAM,SAAS;QACrC,IAAI,SAAS,kBAAkB,OAAO,uBAAuB;QAC7D,MAAM,MAAM,MAAM,KAAK;SAAE,IAAI;SAAW,MAAM,OAAO,KAAK,QAAQ,EAAE;QAAE,CAAC;QACvE,MAAM,OAAO,IAAI,mBAAmB;QACpC,MAAM,SAAS,GAAG,OAAO,IAAI,UAAU,EAAE,IAAI,OAAO,2FAA2F;QAC/I,gBAAgB,KAAK;SAAE,MAAM,KAAK,UAAU,IAAI;SAAG;QAAO,CAAC;QAC3D,OAAO;OACT;MACF,CAG8B;MAAG;KAAQ,GACrC,SAAS,eACT,EAAE,OAAO,CACX,CACF;KACA,IAAI,KAAK,WAAW,aAClB,MAAM,IAAI,MAAM,KAAK,OAAO,WAAW,4BAA4B,KAAK,QAAQ;KAElF,MAAM,UAAW,MAAM,KAAK,EAAE,IAAI,WAAW,CAAC;KAC9C,MAAM,aAAa,gBAChB,MAAM,EAAE,CAAC,CACT,KAAK,MAAM,UAAU,EAAE,KAAK,MAAM,GAAG,GAAG,EAAE,aAAa,EAAE,OAAO,MAAM,GAAG,GAAG,GAAG,CAAC,CAChF,KAAK,SAAS,CAAC,CACf,MAAM,GAAG,IAAI;KAChB,MAAM,aAAa,KAAK,OAAO,GAAG,EAAE;KACpC,MAAM,iBACJ,YAAY,SAAS,WAAW,WAAW,UAAU,SAChD,WAAW,SAAS,SACrB,KAAA;KACN,OAAO,6BACL,SACA,KAAK,OACL,gBAAgB,OAAO,YACvB,UACF;IACF,CAAC;IACD,MAAM,aAAa,wBAAwB,KAAK,IAAI,KAAK;IACzD,IAAI,YAAY,MAAM;IACtB,MAAM;KAAE,MAAM;KAAU,MAAM,EAAE,WAAW,KAAK,UAAU,GAAG,EAAE;IAAE;GACnE;GACA,MAAM,SAAS,CAAC;EAClB;CACF,EACF;AACF;;AAGA,MAAM,qBAA4C,EAChD,MAAM,QAAQ;CACZ,IAAI,OAAO;CACX,KAAK,MAAM,MAAM,QAAQ;EAEvB,MAAM,KADK,IAA2C,KAAA,EACzC;EACb,IAAI,OAAO,MAAM,YAAY,EAAE,SAAS,GAAG,OAAO;CACpD;CACA,OAAO;AACT,EACF;AAEA,SAAgB,6BAA+C;CAC7D,MAAM,OAAO,sBAAsB;CACnC,OAAO;EACL,MAAM;EACN,QAAQ;EACR,iBAAiB,KAAK,UAAU;EAEhC,MAAM,UAAU,OAAoB,CAAC,GAAyB;GAE5D,QAAO,MADa,KAAK,UAAU,IAAI,EAAA,CAC1B,KAAK,MAAM;IACtB,MAAM,OAAO,SAAS,CAAC;IACvB,OAAO;KACL,GAAG;KAGH,QAAQ,mBAAmB,KAAK,OAAO,GAAG,KAAK,MAAM;IACvD;GACF,CAAC;EACH;EAEA,oBAAoB,QAAQ,QAAQ,KAAA,CAAS;EAE7C,MAAM,MAAM,MAAiB,UAAuC;GAClE,MAAM,OAAO,SAAS,IAAI;GAC1B,IAAI;GACJ,IAAI;IACF,MAAM,KAAK,MAAM,QAAQ;GAC3B,QAAQ;IACN,MAAM,IAAI,MACR,6EAA6E,KAAK,OAAO,KAAK,SAAS,MAAM,GAAG,GAAG,GACrH;GACF;GACA,IAAI,OAAO,IAAI,YAAY,aAAa,OAAO,IAAI,cAAc,UAC/D,MAAM,IAAI,MACR,sEAAsE,KAAK,OAAO,KAAK,SAAS,MAAM,GAAG,GAAG,GAC9G;GAEF,MAAM,SAAS,IAAI,UAAU;GAC7B,MAAM,QAAQ,IAAI;GAClB,MAAM,WAAW,MAAM,QAAQ,IAAI,aAAa,IAAI,IAAI,gBAAgB,CAAC;GACzE,OAAO;IACL,UAAU,IAAI,YAAY;IAC1B,OAAO,QAAQ,IAAI,SAAS,QAAQ;IACpC,QAAQ,KAAK,UAAU;KACrB,QAAQ,KAAK;KACb,SAAS,IAAI;KACb;KACA,OAAO,IAAI,SAAS;KACpB;KACA,OAAO,IAAI;KACX,GAAI,SAAS,SAAS,EAAE,SAAS,IAAI,CAAC;KACtC,GAAI,IAAI,aAAa,EAAE,gBAAgB,IAAI,WAAW,MAAM,IAAI,EAAE,IAAI,CAAC;IACzE,CAAC;GACH;EACF;EAEA,aAAa,MAAM,uBAAuB,CAAC;CAC7C;AACF"}

package/dist/benchmarks/cadbench.js

@@ -1,2 +1,2 @@
-import { t as createCadBenchAdapter } from "../cadbench-BrpwOU6A.js";
+import { t as createCadBenchAdapter } from "../cadbench-BLSyxR1N.js";
 export { createCadBenchAdapter };

package/dist/benchmarks/cadgenbench.js

@@ -1,2 +1,2 @@
-import { t as createCadGenBenchAdapter } from "../cadgenbench-DF7hYHdl.js";
+import { t as createCadGenBenchAdapter } from "../cadgenbench-x2OFkf8y.js";
 export { createCadGenBenchAdapter };

package/dist/benchmarks/finresearchbench.js

@@ -1,4 +1,5 @@
 import { benchRoot } from "./_harness.js";
+import { t as runBenchRouterTurn } from "../router-turn-C2wMiDoo.js";
 import { readFile, stat } from "node:fs/promises";
 import { join } from "node:path";
 //#region src/benchmarks/finresearchbench.ts
@@ -122,28 +123,21 @@ function parseJudgeScore(content) {
 async function runOfficialJudge(meta, response) {
 	if (!meta.judgeSystemPrompt) throw new Error(`FinResearchBench task ${meta.id} missing judge_system_prompt`);
 	const router = routerConfig();
-	const res = await fetch(`${router.baseUrl}/chat/completions`, {
-		method: "POST",
-		headers: {
-			"content-type": "application/json",
-			authorization: `Bearer ${router.key}`
-		},
-		body: JSON.stringify({
-			model: router.model,
-			temperature: 0,
-			messages: [{
-				role: "system",
-				content: meta.judgeSystemPrompt
-			}, {
-				role: "user",
-				content: fillTemplate(meta, response)
-			}]
-		})
-	});
-	if (!res.ok) throw new Error(`FinResearchBench judge HTTP ${res.status}: ${(await res.text()).slice(0, 300)}`);
-	const body = await res.json();
-	const content = body.choices?.[0]?.message?.content;
-	if (typeof content !== "string") throw new Error(`FinResearchBench judge returned no message content: ${JSON.stringify(body).slice(0, 300)}`);
+	const content = (await runBenchRouterTurn({
+		routerBaseUrl: router.baseUrl,
+		routerKey: router.key,
+		profile: {
+			name: "finresearchbench-judge",
+			harness: "cli-base",
+			model: {
+				provider: "tangle-router",
+				default: router.model,
+				metadata: { temperature: 0 }
+			},
+			prompt: { systemPrompt: meta.judgeSystemPrompt }
+		}
+	}, fillTemplate(meta, response))).finalText;
+	if (!content) throw new Error("FinResearchBench judge returned no message content");
 	const { score, raw } = parseJudgeScore(content);
 	return {
 		resolved: score >= Number(process.env.FINRESEARCHBENCH_PASS_THRESHOLD ?? .8),