@tangle-network/agent-bench 0.8.19 → 0.8.22

package/CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+## 0.8.22
+
+### Eval 0.163.2 and Knowledge 10.7.0 cohort
+
+The Eval catalog range becomes `>=0.163.2 <0.164.0` and the Knowledge catalog range `^10.7.0`, matching the Runtime cohort. Bench resolves both through the catalog, so its published manifest states the new ranges. Nothing in bench changes behavior.
+
+A consumer that pins Eval or Knowledge itself must move them with this package.
+
+## 0.8.21
+
+### Interface 1.4.0 cohort
+
+The Agent Interface dependency range becomes `^1.4.0`, matching the Runtime cohort. Interface 1.4.0 adds the environment creation receipt and the provider-native child-task stream event; nothing in bench changes behavior.
+
+## 0.8.20
+
+### Official adapter evidence for tau, DABStep, and FinResearchBench
+
+Official tau2/tau3 task loads stamp the upstream checkout commit and the installed `tau2` distribution version into task metadata, and the trajectory judge records both in its score detail.
+A dirty checkout, a non-git bench directory, a missing `tau2` distribution, or a checkout that moved between load and judge now fails loud.
+FinResearchBench records the judge turn's exact Runtime usage on `BenchScore.judgeUsage`, so an unproven judge cost stays unknown instead of reading as zero spend.
+DABStep accepts an absolute `DABSTEP_DATASET_CSV` override for the released task rows, and its preflight error states where those rows come from.
+
 ## 0.8.19
 
 - Align the Bench package with Runtime 0.143.0, Interface 1.3.0, and Sandbox 0.31.0.

package/README.md

@@ -63,6 +63,35 @@ Each destination contains the untouched evaluator tree under `evaluator/`, raw `
 The destination must be unique and absent; an existing path fails loud instead of overwriting evidence.
 Failed evaluators throw `StagedJudgeError` with the same `judgeArtifacts` receipt after retaining partial logs.
 
+## Official-data adapters: tau2/tau3, DABStep, FinResearchBench
+
+Fixture mode (`TAU2_FIXTURES=1`, `TAU3_FIXTURES=1`, `DABSTEP_FIXTURES=1`, `FINRESEARCHBENCH_FIXTURES=1`) proves adapter plumbing only.
+A fixture result is never a benchmark score.
+A benchmark score requires the official data below plus the benchmark's own judge, and nothing else counts.
+
+| Adapter | Official data | Judge |
+|---|---|---|
+| `tau2-bench` / `tau3-banking` | `TAU2_BENCH_DIR` / `TAU3_BENCH_DIR` = clean git checkout of [sierra-research/tau2-bench](https://github.com/sierra-research/tau2-bench); domain via `TAU2_DOMAIN` / `TAU3_DOMAIN` | tau2's own reward recomputation over a tau `results.json` trajectory |
+| `dabstep` | `DABSTEP_DIR` = checkout of [EnvCommons/DABStep](https://github.com/EnvCommons/DABStep) (`grade.py`, `splits/`, `files/`); released `dataset.csv` beside it or via `DABSTEP_DATASET_CSV` | official `grade.py`, deterministic, no LLM |
+| `finresearchbench` | `FINRESEARCHBENCH_DATA_FILE` = JSON/JSONL export whose rows carry `judge_system_prompt` and `judge_prompt_template` | official logic-tree model judge over the Tangle router (`TANGLE_API_KEY`) |
+
+**tau2/tau3.**
+Run `uv sync --extra knowledge` inside the checkout and set `AGENT_BENCH_PYTHON=<checkout>/.venv/bin/python3` so the loader and judge run in the interpreter that holds the `tau2` distribution.
+Every official load stamps `upstreamCommit` (checkout HEAD) and `upstreamVersion` (installed `tau2` distribution) into task metadata, and fails loud on a dirty or non-git checkout or a missing distribution.
+The judge re-resolves the pin, refuses a checkout or interpreter that moved after load, and writes both values into its score detail.
+A trajectory score needs a tau `results.json` produced by the paid tau simulation (agent plus user simulator); the adapter only recomputes the official reward from that artifact.
+
+**DABStep.**
+The git checkout does not ship `dataset.csv`.
+The row file (`task_id,question,guidelines,all_golds_by_task`) is distributed with the OpenReward DABStep environment, which mounts it at `/orwd_data/dataset.csv`.
+The public [adyen/DABstep](https://huggingface.co/datasets/adyen/DABstep) release carries the task rows without golds (`data/tasks/all.jsonl`; answers withheld for the leaderboard) and a 10-task dev split with reference answers (`data/tasks/dev.jsonl`).
+Point `DABSTEP_DATASET_CSV` at an absolute row-file path when the checkout and the rows live apart.
+
+**FinResearchBench.**
+The judge is a model call, so its score records the exact judge turn usage on `BenchScore.judgeUsage` (tokens, cost, model).
+`tokensKnown: false` and `usdKnown: false` survive verbatim; an unknown judge cost never reads as zero spend.
+`FINRESEARCHBENCH_JUDGE_MODEL` (or `JUDGE_MODEL`) selects the judge; `FINRESEARCHBENCH_PASS_THRESHOLD` moves the resolve bar from 0.8.
+
 ## Pier custom candidates
 
 The package executes a branded `PreparedAgentCandidateExecution` from `@tangle-network/agent-runtime` through one atomic API and ships `pier_agents.tangle_candidate:TangleCandidateAgent` as its thin Pier transport.

package/dist/benchmarks/dabstep.js

@@ -1,22 +1,44 @@
 import { benchRoot, runVenvPython, runVenvScriptStdin } from "./_harness.js";
 import { access, readFile, stat } from "node:fs/promises";
-import { join } from "node:path";
+import { isAbsolute, join } from "node:path";
 //#region src/benchmarks/dabstep.ts
 /**
 * DABStep adapter (EnvCommons/DABStep) — data-analysis questions over synthetic
 * payment files. Worker artifact = final answer text. Judge = the official
 * DABStep `grade.py` normalization/matching function. No LLM judge.
 *
-* Live tasks require `DABSTEP_DIR` pointing at an official DABStep checkout that
-* includes `dataset.csv`, `splits/*.txt`, `files/*`, and `grade.py`. The adapter
-* exposes `metadata.resourceRoot` so runners can mount the benchmark files into
-* AgentProfile.resources.files; it does not paste the dataset into prompt text.
+* Live tasks require `DABSTEP_DIR` pointing at an official DABStep checkout
+* (https://github.com/EnvCommons/DABStep) for `splits/*.txt`, `files/*`, and
+* `grade.py`, plus the released `dataset.csv` task rows — beside the checkout
+* or anywhere via `DABSTEP_DATASET_CSV`. The checkout does not ship
+* `dataset.csv`; see the preflight error text for where the rows come from.
+* The adapter exposes `metadata.resourceRoot` so runners can mount the
+* benchmark files into AgentProfile.resources.files; it does not paste the
+* dataset into prompt text.
 */
 const FIXTURES = join(benchRoot, "fixtures", "dabstep.json");
 const DEFAULT_SPLIT = "easy";
 const dabstepDir = () => process.env.DABSTEP_DIR;
 const gradeFile = (dir) => join(dir, "grade.py");
 const resourceRoot = (dir) => join(dir, "files");
+/**
+* Where the released task rows come from. The EnvCommons/DABStep git checkout
+* carries grade.py, splits/, and files/ but not dataset.csv: the row file
+* (task_id,question,guidelines,all_golds_by_task) ships with the OpenReward
+* DABStep environment, mounted at /orwd_data/dataset.csv on that platform.
+* The public https://huggingface.co/datasets/adyen/DABstep release holds the
+* same task rows without golds (data/tasks/all.jsonl; answers withheld for the
+* leaderboard) and a 10-task dev split with reference answers
+* (data/tasks/dev.jsonl).
+*/
+const datasetAcquisitionHint = "The DABStep checkout ships grade.py, splits/, and files/ but not dataset.csv. dataset.csv (task_id,question,guidelines,all_golds_by_task) is distributed with the OpenReward DABStep environment (mounted at /orwd_data/dataset.csv on that platform); the public https://huggingface.co/datasets/adyen/DABstep release carries the task rows without golds (data/tasks/all.jsonl) and a 10-task dev split with reference answers (data/tasks/dev.jsonl). Place dataset.csv beside the checkout, or point DABSTEP_DATASET_CSV at it.";
+/** The released row file: `<dir>/dataset.csv`, or the `DABSTEP_DATASET_CSV` override so checkout and export can live apart. */
+function datasetFile(dir) {
+	const override = process.env.DABSTEP_DATASET_CSV;
+	if (override === void 0) return join(dir, "dataset.csv");
+	if (!isAbsolute(override)) throw new Error("DABSTEP_DATASET_CSV must be an absolute path");
+	return override;
+}
 async function assertFile(path, label) {
 	try {
 		await access(path);
@@ -25,7 +47,12 @@ async function assertFile(path, label) {
 	}
 }
 async function assertOfficialFiles(dir, split) {
-	await assertFile(join(dir, "dataset.csv"), "released dataset.csv");
+	const dataset = datasetFile(dir);
+	try {
+		await access(dataset);
+	} catch (err) {
+		throw new Error(`DABStep: missing released dataset.csv at ${dataset} (${err instanceof Error ? err.message : err}). ${datasetAcquisitionHint}`);
+	}
 	await assertFile(join(dir, "splits", `${split}.txt`), `${split} split file`);
 	await assertFile(gradeFile(dir), "official grade.py");
 	const files = resourceRoot(dir);
@@ -91,10 +118,10 @@ root = Path(sys.argv[1])
 split = sys.argv[2]
 limit = None if sys.argv[3] == "" else int(sys.argv[3])
 ids = set(json.loads(sys.argv[4]))
-dataset = root / "dataset.csv"
+dataset = Path(sys.argv[5])
 split_file = root / "splits" / f"{split}.txt"
 if not dataset.exists():
-    raise SystemExit(f"missing official DABStep dataset.csv at {dataset}")
+    raise SystemExit(f"missing official DABStep dataset.csv at {dataset}; the checkout does not ship it — see the adapter preflight error for acquisition")
 if not split_file.exists():
     raise SystemExit(f"missing official DABStep split file at {split_file}")
 split_ids = {int(line.strip()) for line in split_file.read_text().splitlines() if line.strip()}
@@ -120,7 +147,8 @@ print(json.dumps(out))
 		dir,
 		split,
 		opts.limit === void 0 ? "" : String(opts.limit),
-		JSON.stringify(opts.ids ?? [])
+		JSON.stringify(opts.ids ?? []),
+		datasetFile(dir)
 	]);
 	return selectRows(JSON.parse(stdout), opts, split, dir);
 }
@@ -132,7 +160,7 @@ function createDabstepAdapter() {
 		async preflight() {
 			if (fixturesMode) return;
 			const dir = dabstepDir();
-			if (!dir) throw new Error("DABSTEP_DIR is required. Fix: clone https://github.com/EnvCommons/DABStep, add the released dataset.csv under that checkout, then set DABSTEP_DIR=/path/to/DABStep.");
+			if (!dir) throw new Error("DABSTEP_DIR is required. Fix: clone https://github.com/EnvCommons/DABStep and set DABSTEP_DIR=/path/to/DABStep. The DABStep checkout ships grade.py, splits/, and files/ but not dataset.csv. dataset.csv (task_id,question,guidelines,all_golds_by_task) is distributed with the OpenReward DABStep environment (mounted at /orwd_data/dataset.csv on that platform); the public https://huggingface.co/datasets/adyen/DABstep release carries the task rows without golds (data/tasks/all.jsonl) and a 10-task dev split with reference answers (data/tasks/dev.jsonl). Place dataset.csv beside the checkout, or point DABSTEP_DATASET_CSV at it.");
 			await assertOfficialFiles(dir, DEFAULT_SPLIT);
 			await loadOfficialTasks(dir, { limit: 1 }, DEFAULT_SPLIT);
 		},

package/dist/benchmarks/dabstep.js.map

@@ -1 +1 @@
-{"version":3,"file":"dabstep.js","names":[],"sources":["../../src/benchmarks/dabstep.ts"],"sourcesContent":["/**\n * DABStep adapter (EnvCommons/DABStep) — data-analysis questions over synthetic\n * payment files. Worker artifact = final answer text. Judge = the official\n * DABStep `grade.py` normalization/matching function. No LLM judge.\n *\n * Live tasks require `DABSTEP_DIR` pointing at an official DABStep checkout that\n * includes `dataset.csv`, `splits/*.txt`, `files/*`, and `grade.py`. The adapter\n * exposes `metadata.resourceRoot` so runners can mount the benchmark files into\n * AgentProfile.resources.files; it does not paste the dataset into prompt text.\n */\n\nimport { join } from 'node:path'\nimport { access, readFile, stat } from 'node:fs/promises'\nimport type { OutputAdapter } from '@tangle-network/agent-runtime/kernel'\nimport { benchRoot, runVenvPython, runVenvScriptStdin } from './_harness'\nimport type { BenchmarkAdapter, BenchScore, BenchTask, LoadOptions } from './types'\n\nconst FIXTURES = join(benchRoot, 'fixtures', 'dabstep.json')\nconst DEFAULT_SPLIT = 'easy'\n\ninterface DabstepFixtureRow {\n  task_id: number\n  instructions: string\n  all_golds_by_task: Array<Record<string, unknown>>\n}\n\ninterface DabstepMeta {\n  taskId: number\n  split: string\n  golds: Array<Record<string, unknown>>\n  resourceRoot?: string\n}\n\nconst dabstepDir = (): string | undefined => process.env.DABSTEP_DIR\nconst gradeFile = (dir: string): string => join(dir, 'grade.py')\nconst resourceRoot = (dir: string): string => join(dir, 'files')\n\nasync function assertFile(path: string, label: string): Promise<void> {\n  try {\n    await access(path)\n  } catch (err) {\n    throw new Error(`DABStep: missing ${label} at ${path} (${err instanceof Error ? err.message : err})`)\n  }\n}\n\nasync function assertOfficialFiles(dir: string, split: string): Promise<void> {\n  await assertFile(join(dir, 'dataset.csv'), 'released dataset.csv')\n  await assertFile(join(dir, 'splits', `${split}.txt`), `${split} split file`)\n  await assertFile(gradeFile(dir), 'official grade.py')\n  const files = resourceRoot(dir)\n  try {\n    const s = await stat(files)\n    if (!s.isDirectory()) throw new Error('not a directory')\n  } catch (err) {\n    throw new Error(`DABStep: missing benchmark files directory at ${files} (${err instanceof Error ? err.message : err})`)\n  }\n}\n\nexport const dabstepAnswerOutput: OutputAdapter<string> = {\n  parse(events) {\n    let text = ''\n    for (const ev of events) {\n      const d = (ev as { data?: Record<string, unknown> })?.data\n      const t = d?.finalText ?? d?.text ?? d?.result\n      if (typeof t === 'string' && t.length > 0) text = t\n    }\n    const fences = [...text.matchAll(/```(?:text|answer)?\\s*\\n([\\s\\S]*?)```/g)]\n    return (fences.at(-1)?.[1] ?? text).trim()\n  },\n}\n\nfunction rowToTask(row: DabstepFixtureRow, split: string, dir?: string): BenchTask {\n  const meta: DabstepMeta = {\n    taskId: row.task_id,\n    split,\n    golds: row.all_golds_by_task,\n    ...(dir ? { resourceRoot: resourceRoot(dir) } : {}),\n  }\n  return {\n    id: String(row.task_id),\n    split,\n    prompt: [\n      'Solve this DABStep data-analysis task using the mounted payment files.',\n      'Use code or shell commands as needed, then return only the final answer.',\n      '',\n      row.instructions,\n    ].join('\\n'),\n    metadata: meta as unknown as Record<string, unknown>,\n  }\n}\n\nfunction readMeta(task: BenchTask): DabstepMeta {\n  const md = task.metadata\n  if (!md || typeof md.taskId !== 'number' || !Array.isArray(md.golds)) {\n    throw new Error(`dabstep task ${task.id} missing metadata — loadTasks did not populate it`)\n  }\n  return md as unknown as DabstepMeta\n}\n\nfunction selectRows(rows: DabstepFixtureRow[], opts: LoadOptions, split: string, dir?: string): BenchTask[] {\n  let tasks = rows.map((row) => rowToTask(row, split, dir))\n  if (opts.ids) {\n    const want = new Set(opts.ids)\n    tasks = tasks.filter((task) => want.has(task.id))\n  } else if (opts.limit !== undefined) {\n    tasks = tasks.slice(0, opts.limit)\n  }\n  if (tasks.length === 0) throw new Error(`DABStep: no tasks matched ${JSON.stringify(opts)} for split=${split}`)\n  return tasks\n}\n\nasync function loadFixtures(opts: LoadOptions, split: string): Promise<BenchTask[]> {\n  const rows = JSON.parse(await readFile(FIXTURES, 'utf8')) as DabstepFixtureRow[]\n  console.warn(`[dabstep] DABSTEP_FIXTURES=1 — loading ${rows.length} adapter fixtures from ${FIXTURES}`)\n  return selectRows(rows, opts, split)\n}\n\nasync function loadOfficialTasks(dir: string, opts: LoadOptions, split: string): Promise<BenchTask[]> {\n  const script = `\nimport ast, csv, json, sys\nfrom pathlib import Path\n\nroot = Path(sys.argv[1])\nsplit = sys.argv[2]\nlimit = None if sys.argv[3] == \"\" else int(sys.argv[3])\nids = set(json.loads(sys.argv[4]))\ndataset = root / \"dataset.csv\"\nsplit_file = root / \"splits\" / f\"{split}.txt\"\nif not dataset.exists():\n    raise SystemExit(f\"missing official DABStep dataset.csv at {dataset}\")\nif not split_file.exists():\n    raise SystemExit(f\"missing official DABStep split file at {split_file}\")\nsplit_ids = {int(line.strip()) for line in split_file.read_text().splitlines() if line.strip()}\nout = []\nwith dataset.open(newline=\"\") as f:\n    for row in csv.DictReader(f):\n        task_id = int(row[\"task_id\"])\n        if task_id not in split_ids:\n            continue\n        if ids and str(task_id) not in ids:\n            continue\n        out.append({\n            \"task_id\": task_id,\n            \"instructions\": f\"{row['question']}\\\\n{row['guidelines']}\",\n            \"all_golds_by_task\": ast.literal_eval(str(row[\"all_golds_by_task\"])),\n        })\n        if limit is not None and len(out) >= limit:\n            break\nif not out:\n    raise SystemExit(f\"no DABStep rows matched split={split} ids={sorted(ids)} limit={limit}\")\nprint(json.dumps(out))\n`\n  const stdout = await runVenvPython(script, [dir, split, opts.limit === undefined ? '' : String(opts.limit), JSON.stringify(opts.ids ?? [])])\n  return selectRows(JSON.parse(stdout) as DabstepFixtureRow[], opts, split, dir)\n}\n\nexport function createDabstepAdapter(): BenchmarkAdapter {\n  const fixturesMode = process.env.DABSTEP_FIXTURES === '1'\n\n  return {\n    name: 'dabstep',\n    output: dabstepAnswerOutput,\n\n    async preflight() {\n      if (fixturesMode) return\n      const dir = dabstepDir()\n      if (!dir) {\n        throw new Error(\n          'DABSTEP_DIR is required. Fix: clone https://github.com/EnvCommons/DABStep, add the released dataset.csv under that checkout, then set DABSTEP_DIR=/path/to/DABStep.',\n        )\n      }\n      await assertOfficialFiles(dir, DEFAULT_SPLIT)\n      await loadOfficialTasks(dir, { limit: 1 }, DEFAULT_SPLIT)\n    },\n\n    async loadTasks(opts: LoadOptions = {}) {\n      const split = opts.split ?? DEFAULT_SPLIT\n      if (fixturesMode) return loadFixtures(opts, split)\n      const dir = dabstepDir()\n      if (!dir) throw new Error('DABSTEP_DIR is required to load official DABStep tasks')\n      return loadOfficialTasks(dir, opts, split)\n    },\n\n    async goldArtifact(task: BenchTask) {\n      const meta = readMeta(task)\n      const first = meta.golds[0]\n      if (!first) return undefined\n      const value = first.value\n      return value === undefined ? undefined : String(value)\n    },\n\n    async judge(task: BenchTask, artifact: string): Promise<BenchScore> {\n      const meta = readMeta(task)\n      const dir = dabstepDir()\n      if (!dir) throw new Error('DABSTEP_DIR is required to judge DABStep tasks with the official grade.py')\n      const stdout = await runVenvScriptStdin(\n        join(benchRoot, 'scripts', 'dabstep_judge.py'),\n        ['--grade-file', gradeFile(dir)],\n        JSON.stringify({ prediction: artifact, golds: meta.golds }),\n        { cwd: benchRoot },\n      )\n      const report = JSON.parse(stdout.trim().split('\\n').at(-1) ?? '{}') as { correct?: boolean; score?: number; error?: string }\n      if (report.error) throw new Error(`DABStep judge error for ${task.id}: ${report.error}`)\n      const score = typeof report.score === 'number' ? report.score : 0\n      return {\n        resolved: report.correct === true,\n        score,\n        detail: JSON.stringify({ taskId: meta.taskId, split: meta.split, correct: report.correct }),\n      }\n    },\n  }\n}\n"],"mappings":";;;;;;;;;;;;;;AAiBA,MAAM,WAAW,KAAK,WAAW,YAAY,cAAc;AAC3D,MAAM,gBAAgB;AAetB,MAAM,mBAAuC,QAAQ,IAAI;AACzD,MAAM,aAAa,QAAwB,KAAK,KAAK,UAAU;AAC/D,MAAM,gBAAgB,QAAwB,KAAK,KAAK,OAAO;AAE/D,eAAe,WAAW,MAAc,OAA8B;CACpE,IAAI;EACF,MAAM,OAAO,IAAI;CACnB,SAAS,KAAK;EACZ,MAAM,IAAI,MAAM,oBAAoB,MAAM,MAAM,KAAK,IAAI,eAAe,QAAQ,IAAI,UAAU,IAAI,EAAE;CACtG;AACF;AAEA,eAAe,oBAAoB,KAAa,OAA8B;CAC5E,MAAM,WAAW,KAAK,KAAK,aAAa,GAAG,sBAAsB;CACjE,MAAM,WAAW,KAAK,KAAK,UAAU,GAAG,MAAM,KAAK,GAAG,GAAG,MAAM,YAAY;CAC3E,MAAM,WAAW,UAAU,GAAG,GAAG,mBAAmB;CACpD,MAAM,QAAQ,aAAa,GAAG;CAC9B,IAAI;EAEF,IAAI,EAAC,MADW,KAAK,KAAK,EAAA,CACnB,YAAY,GAAG,MAAM,IAAI,MAAM,iBAAiB;CACzD,SAAS,KAAK;EACZ,MAAM,IAAI,MAAM,iDAAiD,MAAM,IAAI,eAAe,QAAQ,IAAI,UAAU,IAAI,EAAE;CACxH;AACF;AAEA,MAAa,sBAA6C,EACxD,MAAM,QAAQ;CACZ,IAAI,OAAO;CACX,KAAK,MAAM,MAAM,QAAQ;EACvB,MAAM,IAAK,IAA2C;EACtD,MAAM,IAAI,GAAG,aAAa,GAAG,QAAQ,GAAG;EACxC,IAAI,OAAO,MAAM,YAAY,EAAE,SAAS,GAAG,OAAO;CACpD;CAEA,QAAQ,CADQ,GAAG,KAAK,SAAS,wCAAwC,CAC5D,CAAC,CAAC,GAAG,EAAE,CAAC,GAAG,MAAM,KAAA,CAAM,KAAK;AAC3C,EACF;AAEA,SAAS,UAAU,KAAwB,OAAe,KAAyB;CACjF,MAAM,OAAoB;EACxB,QAAQ,IAAI;EACZ;EACA,OAAO,IAAI;EACX,GAAI,MAAM,EAAE,cAAc,aAAa,GAAG,EAAE,IAAI,CAAC;CACnD;CACA,OAAO;EACL,IAAI,OAAO,IAAI,OAAO;EACtB;EACA,QAAQ;GACN;GACA;GACA;GACA,IAAI;EACN,CAAC,CAAC,KAAK,IAAI;EACX,UAAU;CACZ;AACF;AAEA,SAAS,SAAS,MAA8B;CAC9C,MAAM,KAAK,KAAK;CAChB,IAAI,CAAC,MAAM,OAAO,GAAG,WAAW,YAAY,CAAC,MAAM,QAAQ,GAAG,KAAK,GACjE,MAAM,IAAI,MAAM,gBAAgB,KAAK,GAAG,kDAAkD;CAE5F,OAAO;AACT;AAEA,SAAS,WAAW,MAA2B,MAAmB,OAAe,KAA2B;CAC1G,IAAI,QAAQ,KAAK,KAAK,QAAQ,UAAU,KAAK,OAAO,GAAG,CAAC;CACxD,IAAI,KAAK,KAAK;EACZ,MAAM,OAAO,IAAI,IAAI,KAAK,GAAG;EAC7B,QAAQ,MAAM,QAAQ,SAAS,KAAK,IAAI,KAAK,EAAE,CAAC;CAClD,OAAO,IAAI,KAAK,UAAU,KAAA,GACxB,QAAQ,MAAM,MAAM,GAAG,KAAK,KAAK;CAEnC,IAAI,MAAM,WAAW,GAAG,MAAM,IAAI,MAAM,6BAA6B,KAAK,UAAU,IAAI,EAAE,aAAa,OAAO;CAC9G,OAAO;AACT;AAEA,eAAe,aAAa,MAAmB,OAAqC;CAClF,MAAM,OAAO,KAAK,MAAM,MAAM,SAAS,UAAU,MAAM,CAAC;CACxD,QAAQ,KAAK,0CAA0C,KAAK,OAAO,yBAAyB,UAAU;CACtG,OAAO,WAAW,MAAM,MAAM,KAAK;AACrC;AAEA,eAAe,kBAAkB,KAAa,MAAmB,OAAqC;CAmCpG,MAAM,SAAS,MAAM,cAAc;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAAQ;EAAC;EAAK;EAAO,KAAK,UAAU,KAAA,IAAY,KAAK,OAAO,KAAK,KAAK;EAAG,KAAK,UAAU,KAAK,OAAO,CAAC,CAAC;CAAC,CAAC;CAC3I,OAAO,WAAW,KAAK,MAAM,MAAM,GAA0B,MAAM,OAAO,GAAG;AAC/E;AAEA,SAAgB,uBAAyC;CACvD,MAAM,eAAe,QAAQ,IAAI,qBAAqB;CAEtD,OAAO;EACL,MAAM;EACN,QAAQ;EAER,MAAM,YAAY;GAChB,IAAI,cAAc;GAClB,MAAM,MAAM,WAAW;GACvB,IAAI,CAAC,KACH,MAAM,IAAI,MACR,qKACF;GAEF,MAAM,oBAAoB,KAAK,aAAa;GAC5C,MAAM,kBAAkB,KAAK,EAAE,OAAO,EAAE,GAAG,aAAa;EAC1D;EAEA,MAAM,UAAU,OAAoB,CAAC,GAAG;GACtC,MAAM,QAAQ,KAAK,SAAS;GAC5B,IAAI,cAAc,OAAO,aAAa,MAAM,KAAK;GACjD,MAAM,MAAM,WAAW;GACvB,IAAI,CAAC,KAAK,MAAM,IAAI,MAAM,wDAAwD;GAClF,OAAO,kBAAkB,KAAK,MAAM,KAAK;EAC3C;EAEA,MAAM,aAAa,MAAiB;GAElC,MAAM,QADO,SAAS,IACL,CAAC,CAAC,MAAM;GACzB,IAAI,CAAC,OAAO,OAAO,KAAA;GACnB,MAAM,QAAQ,MAAM;GACpB,OAAO,UAAU,KAAA,IAAY,KAAA,IAAY,OAAO,KAAK;EACvD;EAEA,MAAM,MAAM,MAAiB,UAAuC;GAClE,MAAM,OAAO,SAAS,IAAI;GAC1B,MAAM,MAAM,WAAW;GACvB,IAAI,CAAC,KAAK,MAAM,IAAI,MAAM,2EAA2E;GACrG,MAAM,SAAS,MAAM,mBACnB,KAAK,WAAW,WAAW,kBAAkB,GAC7C,CAAC,gBAAgB,UAAU,GAAG,CAAC,GAC/B,KAAK,UAAU;IAAE,YAAY;IAAU,OAAO,KAAK;GAAM,CAAC,GAC1D,EAAE,KAAK,UAAU,CACnB;GACA,MAAM,SAAS,KAAK,MAAM,OAAO,KAAK,CAAC,CAAC,MAAM,IAAI,CAAC,CAAC,GAAG,EAAE,KAAK,IAAI;GAClE,IAAI,OAAO,OAAO,MAAM,IAAI,MAAM,2BAA2B,KAAK,GAAG,IAAI,OAAO,OAAO;GACvF,MAAM,QAAQ,OAAO,OAAO,UAAU,WAAW,OAAO,QAAQ;GAChE,OAAO;IACL,UAAU,OAAO,YAAY;IAC7B;IACA,QAAQ,KAAK,UAAU;KAAE,QAAQ,KAAK;KAAQ,OAAO,KAAK;KAAO,SAAS,OAAO;IAAQ,CAAC;GAC5F;EACF;CACF;AACF"}
+{"version":3,"file":"dabstep.js","names":[],"sources":["../../src/benchmarks/dabstep.ts"],"sourcesContent":["/**\n * DABStep adapter (EnvCommons/DABStep) — data-analysis questions over synthetic\n * payment files. Worker artifact = final answer text. Judge = the official\n * DABStep `grade.py` normalization/matching function. No LLM judge.\n *\n * Live tasks require `DABSTEP_DIR` pointing at an official DABStep checkout\n * (https://github.com/EnvCommons/DABStep) for `splits/*.txt`, `files/*`, and\n * `grade.py`, plus the released `dataset.csv` task rows — beside the checkout\n * or anywhere via `DABSTEP_DATASET_CSV`. The checkout does not ship\n * `dataset.csv`; see the preflight error text for where the rows come from.\n * The adapter exposes `metadata.resourceRoot` so runners can mount the\n * benchmark files into AgentProfile.resources.files; it does not paste the\n * dataset into prompt text.\n */\n\nimport { isAbsolute, join } from 'node:path'\nimport { access, readFile, stat } from 'node:fs/promises'\nimport type { OutputAdapter } from '@tangle-network/agent-runtime/kernel'\nimport { benchRoot, runVenvPython, runVenvScriptStdin } from './_harness'\nimport type { BenchmarkAdapter, BenchScore, BenchTask, LoadOptions } from './types'\n\nconst FIXTURES = join(benchRoot, 'fixtures', 'dabstep.json')\nconst DEFAULT_SPLIT = 'easy'\n\ninterface DabstepFixtureRow {\n  task_id: number\n  instructions: string\n  all_golds_by_task: Array<Record<string, unknown>>\n}\n\ninterface DabstepMeta {\n  taskId: number\n  split: string\n  golds: Array<Record<string, unknown>>\n  resourceRoot?: string\n}\n\nconst dabstepDir = (): string | undefined => process.env.DABSTEP_DIR\nconst gradeFile = (dir: string): string => join(dir, 'grade.py')\nconst resourceRoot = (dir: string): string => join(dir, 'files')\n\n/**\n * Where the released task rows come from. The EnvCommons/DABStep git checkout\n * carries grade.py, splits/, and files/ but not dataset.csv: the row file\n * (task_id,question,guidelines,all_golds_by_task) ships with the OpenReward\n * DABStep environment, mounted at /orwd_data/dataset.csv on that platform.\n * The public https://huggingface.co/datasets/adyen/DABstep release holds the\n * same task rows without golds (data/tasks/all.jsonl; answers withheld for the\n * leaderboard) and a 10-task dev split with reference answers\n * (data/tasks/dev.jsonl).\n */\nconst datasetAcquisitionHint =\n  'The DABStep checkout ships grade.py, splits/, and files/ but not dataset.csv. ' +\n  'dataset.csv (task_id,question,guidelines,all_golds_by_task) is distributed with the OpenReward DABStep environment ' +\n  '(mounted at /orwd_data/dataset.csv on that platform); the public https://huggingface.co/datasets/adyen/DABstep release ' +\n  'carries the task rows without golds (data/tasks/all.jsonl) and a 10-task dev split with reference answers (data/tasks/dev.jsonl). ' +\n  'Place dataset.csv beside the checkout, or point DABSTEP_DATASET_CSV at it.'\n\n/** The released row file: `<dir>/dataset.csv`, or the `DABSTEP_DATASET_CSV` override so checkout and export can live apart. */\nfunction datasetFile(dir: string): string {\n  const override = process.env.DABSTEP_DATASET_CSV\n  if (override === undefined) return join(dir, 'dataset.csv')\n  if (!isAbsolute(override)) throw new Error('DABSTEP_DATASET_CSV must be an absolute path')\n  return override\n}\n\nasync function assertFile(path: string, label: string): Promise<void> {\n  try {\n    await access(path)\n  } catch (err) {\n    throw new Error(`DABStep: missing ${label} at ${path} (${err instanceof Error ? err.message : err})`)\n  }\n}\n\nasync function assertOfficialFiles(dir: string, split: string): Promise<void> {\n  const dataset = datasetFile(dir)\n  try {\n    await access(dataset)\n  } catch (err) {\n    throw new Error(\n      `DABStep: missing released dataset.csv at ${dataset} (${err instanceof Error ? err.message : err}). ${datasetAcquisitionHint}`,\n    )\n  }\n  await assertFile(join(dir, 'splits', `${split}.txt`), `${split} split file`)\n  await assertFile(gradeFile(dir), 'official grade.py')\n  const files = resourceRoot(dir)\n  try {\n    const s = await stat(files)\n    if (!s.isDirectory()) throw new Error('not a directory')\n  } catch (err) {\n    throw new Error(`DABStep: missing benchmark files directory at ${files} (${err instanceof Error ? err.message : err})`)\n  }\n}\n\nexport const dabstepAnswerOutput: OutputAdapter<string> = {\n  parse(events) {\n    let text = ''\n    for (const ev of events) {\n      const d = (ev as { data?: Record<string, unknown> })?.data\n      const t = d?.finalText ?? d?.text ?? d?.result\n      if (typeof t === 'string' && t.length > 0) text = t\n    }\n    const fences = [...text.matchAll(/```(?:text|answer)?\\s*\\n([\\s\\S]*?)```/g)]\n    return (fences.at(-1)?.[1] ?? text).trim()\n  },\n}\n\nfunction rowToTask(row: DabstepFixtureRow, split: string, dir?: string): BenchTask {\n  const meta: DabstepMeta = {\n    taskId: row.task_id,\n    split,\n    golds: row.all_golds_by_task,\n    ...(dir ? { resourceRoot: resourceRoot(dir) } : {}),\n  }\n  return {\n    id: String(row.task_id),\n    split,\n    prompt: [\n      'Solve this DABStep data-analysis task using the mounted payment files.',\n      'Use code or shell commands as needed, then return only the final answer.',\n      '',\n      row.instructions,\n    ].join('\\n'),\n    metadata: meta as unknown as Record<string, unknown>,\n  }\n}\n\nfunction readMeta(task: BenchTask): DabstepMeta {\n  const md = task.metadata\n  if (!md || typeof md.taskId !== 'number' || !Array.isArray(md.golds)) {\n    throw new Error(`dabstep task ${task.id} missing metadata — loadTasks did not populate it`)\n  }\n  return md as unknown as DabstepMeta\n}\n\nfunction selectRows(rows: DabstepFixtureRow[], opts: LoadOptions, split: string, dir?: string): BenchTask[] {\n  let tasks = rows.map((row) => rowToTask(row, split, dir))\n  if (opts.ids) {\n    const want = new Set(opts.ids)\n    tasks = tasks.filter((task) => want.has(task.id))\n  } else if (opts.limit !== undefined) {\n    tasks = tasks.slice(0, opts.limit)\n  }\n  if (tasks.length === 0) throw new Error(`DABStep: no tasks matched ${JSON.stringify(opts)} for split=${split}`)\n  return tasks\n}\n\nasync function loadFixtures(opts: LoadOptions, split: string): Promise<BenchTask[]> {\n  const rows = JSON.parse(await readFile(FIXTURES, 'utf8')) as DabstepFixtureRow[]\n  console.warn(`[dabstep] DABSTEP_FIXTURES=1 — loading ${rows.length} adapter fixtures from ${FIXTURES}`)\n  return selectRows(rows, opts, split)\n}\n\nasync function loadOfficialTasks(dir: string, opts: LoadOptions, split: string): Promise<BenchTask[]> {\n  const script = `\nimport ast, csv, json, sys\nfrom pathlib import Path\n\nroot = Path(sys.argv[1])\nsplit = sys.argv[2]\nlimit = None if sys.argv[3] == \"\" else int(sys.argv[3])\nids = set(json.loads(sys.argv[4]))\ndataset = Path(sys.argv[5])\nsplit_file = root / \"splits\" / f\"{split}.txt\"\nif not dataset.exists():\n    raise SystemExit(f\"missing official DABStep dataset.csv at {dataset}; the checkout does not ship it — see the adapter preflight error for acquisition\")\nif not split_file.exists():\n    raise SystemExit(f\"missing official DABStep split file at {split_file}\")\nsplit_ids = {int(line.strip()) for line in split_file.read_text().splitlines() if line.strip()}\nout = []\nwith dataset.open(newline=\"\") as f:\n    for row in csv.DictReader(f):\n        task_id = int(row[\"task_id\"])\n        if task_id not in split_ids:\n            continue\n        if ids and str(task_id) not in ids:\n            continue\n        out.append({\n            \"task_id\": task_id,\n            \"instructions\": f\"{row['question']}\\\\n{row['guidelines']}\",\n            \"all_golds_by_task\": ast.literal_eval(str(row[\"all_golds_by_task\"])),\n        })\n        if limit is not None and len(out) >= limit:\n            break\nif not out:\n    raise SystemExit(f\"no DABStep rows matched split={split} ids={sorted(ids)} limit={limit}\")\nprint(json.dumps(out))\n`\n  const stdout = await runVenvPython(script, [\n    dir,\n    split,\n    opts.limit === undefined ? '' : String(opts.limit),\n    JSON.stringify(opts.ids ?? []),\n    datasetFile(dir),\n  ])\n  return selectRows(JSON.parse(stdout) as DabstepFixtureRow[], opts, split, dir)\n}\n\nexport function createDabstepAdapter(): BenchmarkAdapter {\n  const fixturesMode = process.env.DABSTEP_FIXTURES === '1'\n\n  return {\n    name: 'dabstep',\n    output: dabstepAnswerOutput,\n\n    async preflight() {\n      if (fixturesMode) return\n      const dir = dabstepDir()\n      if (!dir) {\n        throw new Error(\n          'DABSTEP_DIR is required. Fix: clone https://github.com/EnvCommons/DABStep and set DABSTEP_DIR=/path/to/DABStep. ' +\n            datasetAcquisitionHint,\n        )\n      }\n      await assertOfficialFiles(dir, DEFAULT_SPLIT)\n      await loadOfficialTasks(dir, { limit: 1 }, DEFAULT_SPLIT)\n    },\n\n    async loadTasks(opts: LoadOptions = {}) {\n      const split = opts.split ?? DEFAULT_SPLIT\n      if (fixturesMode) return loadFixtures(opts, split)\n      const dir = dabstepDir()\n      if (!dir) throw new Error('DABSTEP_DIR is required to load official DABStep tasks')\n      return loadOfficialTasks(dir, opts, split)\n    },\n\n    async goldArtifact(task: BenchTask) {\n      const meta = readMeta(task)\n      const first = meta.golds[0]\n      if (!first) return undefined\n      const value = first.value\n      return value === undefined ? undefined : String(value)\n    },\n\n    async judge(task: BenchTask, artifact: string): Promise<BenchScore> {\n      const meta = readMeta(task)\n      const dir = dabstepDir()\n      if (!dir) throw new Error('DABSTEP_DIR is required to judge DABStep tasks with the official grade.py')\n      const stdout = await runVenvScriptStdin(\n        join(benchRoot, 'scripts', 'dabstep_judge.py'),\n        ['--grade-file', gradeFile(dir)],\n        JSON.stringify({ prediction: artifact, golds: meta.golds }),\n        { cwd: benchRoot },\n      )\n      const report = JSON.parse(stdout.trim().split('\\n').at(-1) ?? '{}') as { correct?: boolean; score?: number; error?: string }\n      if (report.error) throw new Error(`DABStep judge error for ${task.id}: ${report.error}`)\n      const score = typeof report.score === 'number' ? report.score : 0\n      return {\n        resolved: report.correct === true,\n        score,\n        detail: JSON.stringify({ taskId: meta.taskId, split: meta.split, correct: report.correct }),\n      }\n    },\n  }\n}\n"],"mappings":";;;;;;;;;;;;;;;;;;AAqBA,MAAM,WAAW,KAAK,WAAW,YAAY,cAAc;AAC3D,MAAM,gBAAgB;AAetB,MAAM,mBAAuC,QAAQ,IAAI;AACzD,MAAM,aAAa,QAAwB,KAAK,KAAK,UAAU;AAC/D,MAAM,gBAAgB,QAAwB,KAAK,KAAK,OAAO;;;;;;;;;;;AAY/D,MAAM,yBACJ;;AAOF,SAAS,YAAY,KAAqB;CACxC,MAAM,WAAW,QAAQ,IAAI;CAC7B,IAAI,aAAa,KAAA,GAAW,OAAO,KAAK,KAAK,aAAa;CAC1D,IAAI,CAAC,WAAW,QAAQ,GAAG,MAAM,IAAI,MAAM,8CAA8C;CACzF,OAAO;AACT;AAEA,eAAe,WAAW,MAAc,OAA8B;CACpE,IAAI;EACF,MAAM,OAAO,IAAI;CACnB,SAAS,KAAK;EACZ,MAAM,IAAI,MAAM,oBAAoB,MAAM,MAAM,KAAK,IAAI,eAAe,QAAQ,IAAI,UAAU,IAAI,EAAE;CACtG;AACF;AAEA,eAAe,oBAAoB,KAAa,OAA8B;CAC5E,MAAM,UAAU,YAAY,GAAG;CAC/B,IAAI;EACF,MAAM,OAAO,OAAO;CACtB,SAAS,KAAK;EACZ,MAAM,IAAI,MACR,4CAA4C,QAAQ,IAAI,eAAe,QAAQ,IAAI,UAAU,IAAI,KAAK,wBACxG;CACF;CACA,MAAM,WAAW,KAAK,KAAK,UAAU,GAAG,MAAM,KAAK,GAAG,GAAG,MAAM,YAAY;CAC3E,MAAM,WAAW,UAAU,GAAG,GAAG,mBAAmB;CACpD,MAAM,QAAQ,aAAa,GAAG;CAC9B,IAAI;EAEF,IAAI,EAAC,MADW,KAAK,KAAK,EAAA,CACnB,YAAY,GAAG,MAAM,IAAI,MAAM,iBAAiB;CACzD,SAAS,KAAK;EACZ,MAAM,IAAI,MAAM,iDAAiD,MAAM,IAAI,eAAe,QAAQ,IAAI,UAAU,IAAI,EAAE;CACxH;AACF;AAEA,MAAa,sBAA6C,EACxD,MAAM,QAAQ;CACZ,IAAI,OAAO;CACX,KAAK,MAAM,MAAM,QAAQ;EACvB,MAAM,IAAK,IAA2C;EACtD,MAAM,IAAI,GAAG,aAAa,GAAG,QAAQ,GAAG;EACxC,IAAI,OAAO,MAAM,YAAY,EAAE,SAAS,GAAG,OAAO;CACpD;CAEA,QAAQ,CADQ,GAAG,KAAK,SAAS,wCAAwC,CAC5D,CAAC,CAAC,GAAG,EAAE,CAAC,GAAG,MAAM,KAAA,CAAM,KAAK;AAC3C,EACF;AAEA,SAAS,UAAU,KAAwB,OAAe,KAAyB;CACjF,MAAM,OAAoB;EACxB,QAAQ,IAAI;EACZ;EACA,OAAO,IAAI;EACX,GAAI,MAAM,EAAE,cAAc,aAAa,GAAG,EAAE,IAAI,CAAC;CACnD;CACA,OAAO;EACL,IAAI,OAAO,IAAI,OAAO;EACtB;EACA,QAAQ;GACN;GACA;GACA;GACA,IAAI;EACN,CAAC,CAAC,KAAK,IAAI;EACX,UAAU;CACZ;AACF;AAEA,SAAS,SAAS,MAA8B;CAC9C,MAAM,KAAK,KAAK;CAChB,IAAI,CAAC,MAAM,OAAO,GAAG,WAAW,YAAY,CAAC,MAAM,QAAQ,GAAG,KAAK,GACjE,MAAM,IAAI,MAAM,gBAAgB,KAAK,GAAG,kDAAkD;CAE5F,OAAO;AACT;AAEA,SAAS,WAAW,MAA2B,MAAmB,OAAe,KAA2B;CAC1G,IAAI,QAAQ,KAAK,KAAK,QAAQ,UAAU,KAAK,OAAO,GAAG,CAAC;CACxD,IAAI,KAAK,KAAK;EACZ,MAAM,OAAO,IAAI,IAAI,KAAK,GAAG;EAC7B,QAAQ,MAAM,QAAQ,SAAS,KAAK,IAAI,KAAK,EAAE,CAAC;CAClD,OAAO,IAAI,KAAK,UAAU,KAAA,GACxB,QAAQ,MAAM,MAAM,GAAG,KAAK,KAAK;CAEnC,IAAI,MAAM,WAAW,GAAG,MAAM,IAAI,MAAM,6BAA6B,KAAK,UAAU,IAAI,EAAE,aAAa,OAAO;CAC9G,OAAO;AACT;AAEA,eAAe,aAAa,MAAmB,OAAqC;CAClF,MAAM,OAAO,KAAK,MAAM,MAAM,SAAS,UAAU,MAAM,CAAC;CACxD,QAAQ,KAAK,0CAA0C,KAAK,OAAO,yBAAyB,UAAU;CACtG,OAAO,WAAW,MAAM,MAAM,KAAK;AACrC;AAEA,eAAe,kBAAkB,KAAa,MAAmB,OAAqC;CAmCpG,MAAM,SAAS,MAAM,cAAc;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAAQ;EACzC;EACA;EACA,KAAK,UAAU,KAAA,IAAY,KAAK,OAAO,KAAK,KAAK;EACjD,KAAK,UAAU,KAAK,OAAO,CAAC,CAAC;EAC7B,YAAY,GAAG;CACjB,CAAC;CACD,OAAO,WAAW,KAAK,MAAM,MAAM,GAA0B,MAAM,OAAO,GAAG;AAC/E;AAEA,SAAgB,uBAAyC;CACvD,MAAM,eAAe,QAAQ,IAAI,qBAAqB;CAEtD,OAAO;EACL,MAAM;EACN,QAAQ;EAER,MAAM,YAAY;GAChB,IAAI,cAAc;GAClB,MAAM,MAAM,WAAW;GACvB,IAAI,CAAC,KACH,MAAM,IAAI,MACR,snBAEF;GAEF,MAAM,oBAAoB,KAAK,aAAa;GAC5C,MAAM,kBAAkB,KAAK,EAAE,OAAO,EAAE,GAAG,aAAa;EAC1D;EAEA,MAAM,UAAU,OAAoB,CAAC,GAAG;GACtC,MAAM,QAAQ,KAAK,SAAS;GAC5B,IAAI,cAAc,OAAO,aAAa,MAAM,KAAK;GACjD,MAAM,MAAM,WAAW;GACvB,IAAI,CAAC,KAAK,MAAM,IAAI,MAAM,wDAAwD;GAClF,OAAO,kBAAkB,KAAK,MAAM,KAAK;EAC3C;EAEA,MAAM,aAAa,MAAiB;GAElC,MAAM,QADO,SAAS,IACL,CAAC,CAAC,MAAM;GACzB,IAAI,CAAC,OAAO,OAAO,KAAA;GACnB,MAAM,QAAQ,MAAM;GACpB,OAAO,UAAU,KAAA,IAAY,KAAA,IAAY,OAAO,KAAK;EACvD;EAEA,MAAM,MAAM,MAAiB,UAAuC;GAClE,MAAM,OAAO,SAAS,IAAI;GAC1B,MAAM,MAAM,WAAW;GACvB,IAAI,CAAC,KAAK,MAAM,IAAI,MAAM,2EAA2E;GACrG,MAAM,SAAS,MAAM,mBACnB,KAAK,WAAW,WAAW,kBAAkB,GAC7C,CAAC,gBAAgB,UAAU,GAAG,CAAC,GAC/B,KAAK,UAAU;IAAE,YAAY;IAAU,OAAO,KAAK;GAAM,CAAC,GAC1D,EAAE,KAAK,UAAU,CACnB;GACA,MAAM,SAAS,KAAK,MAAM,OAAO,KAAK,CAAC,CAAC,MAAM,IAAI,CAAC,CAAC,GAAG,EAAE,KAAK,IAAI;GAClE,IAAI,OAAO,OAAO,MAAM,IAAI,MAAM,2BAA2B,KAAK,GAAG,IAAI,OAAO,OAAO;GACvF,MAAM,QAAQ,OAAO,OAAO,UAAU,WAAW,OAAO,QAAQ;GAChE,OAAO;IACL,UAAU,OAAO,YAAY;IAC7B;IACA,QAAQ,KAAK,UAAU;KAAE,QAAQ,KAAK;KAAQ,OAAO,KAAK;KAAO,SAAS,OAAO;IAAQ,CAAC;GAC5F;EACF;CACF;AACF"}

package/dist/benchmarks/finresearchbench.d.ts

@@ -1,6 +1,17 @@
-import { BenchmarkAdapter } from "./types.js";
+import { BenchScore, BenchTask, BenchmarkAdapter } from "./types.js";
+import { AgentTurnUsage } from "@tangle-network/agent-runtime/kernel";
 //#region src/benchmarks/finresearchbench.d.ts
+/**
+ * Map one drained judge turn to a BenchScore. The judge turn's exact Runtime
+ * usage record lands on `judgeUsage` verbatim, so an unproven token count or
+ * dollar amount stays flagged (`tokensKnown`/`usdKnown` false) instead of
+ * reading as a known zero cost.
+ */
+declare function scoreOfficialJudgeTurn(task: BenchTask, turn: Readonly<{
+  finalText?: string;
+  usage: AgentTurnUsage;
+}>, requestedJudgeModel: string): BenchScore;
 declare function createFinResearchBenchAdapter(): BenchmarkAdapter;
 //#endregion
-export { createFinResearchBenchAdapter };
+export { createFinResearchBenchAdapter, scoreOfficialJudgeTurn };
 //# sourceMappingURL=finresearchbench.d.ts.map

package/dist/benchmarks/finresearchbench.js

@@ -120,10 +120,33 @@ function parseJudgeScore(content) {
 	}
 	throw new Error(`FinResearchBench judge produced no parseable JSON score: ${content.slice(0, 400)}`);
 }
-async function runOfficialJudge(meta, response) {
+/**
+* Map one drained judge turn to a BenchScore. The judge turn's exact Runtime
+* usage record lands on `judgeUsage` verbatim, so an unproven token count or
+* dollar amount stays flagged (`tokensKnown`/`usdKnown` false) instead of
+* reading as a known zero cost.
+*/
+function scoreOfficialJudgeTurn(task, turn, requestedJudgeModel) {
+	const meta = readMeta(task);
+	const content = turn.finalText;
+	if (!content) throw new Error("FinResearchBench judge returned no message content");
+	const { score, raw } = parseJudgeScore(content);
+	return {
+		resolved: score >= Number(process.env.FINRESEARCHBENCH_PASS_THRESHOLD ?? .8),
+		score,
+		detail: JSON.stringify({
+			scoring: meta.scoring,
+			category: meta.category,
+			judgeModel: requestedJudgeModel,
+			raw
+		}),
+		judgeUsage: turn.usage
+	};
+}
+async function runOfficialJudge(task, meta, response) {
 	if (!meta.judgeSystemPrompt) throw new Error(`FinResearchBench task ${meta.id} missing judge_system_prompt`);
 	const router = routerConfig();
-	const content = (await runBenchRouterTurn({
+	return scoreOfficialJudgeTurn(task, await runBenchRouterTurn({
 		routerBaseUrl: router.baseUrl,
 		routerKey: router.key,
 		profile: {
@@ -136,19 +159,7 @@ async function runOfficialJudge(meta, response) {
 			},
 			prompt: { systemPrompt: meta.judgeSystemPrompt }
 		}
-	}, fillTemplate(meta, response))).finalText;
-	if (!content) throw new Error("FinResearchBench judge returned no message content");
-	const { score, raw } = parseJudgeScore(content);
-	return {
-		resolved: score >= Number(process.env.FINRESEARCHBENCH_PASS_THRESHOLD ?? .8),
-		score,
-		detail: JSON.stringify({
-			scoring: meta.scoring,
-			category: meta.category,
-			judgeModel: router.model,
-			raw
-		})
-	};
+	}, fillTemplate(meta, response)), router.model);
 }
 function normalizeText(value) {
 	return value.toLowerCase().replace(/\s+/g, " ").trim();
@@ -203,11 +214,11 @@ function createFinResearchBenchAdapter() {
 					reason: "empty answer"
 				})
 			};
-			return runOfficialJudge(meta, artifact);
+			return runOfficialJudge(task, meta, artifact);
 		}
 	};
 }
 //#endregion
-export { createFinResearchBenchAdapter };
+export { createFinResearchBenchAdapter, scoreOfficialJudgeTurn };
 
 //# sourceMappingURL=finresearchbench.js.map

package/dist/benchmarks/finresearchbench.js.map

@@ -1 +1 @@
-{"version":3,"file":"finresearchbench.js","names":[],"sources":["../../src/benchmarks/finresearchbench.ts"],"sourcesContent":["/**\n * FinResearchBench-compatible adapter.\n *\n * The paper defines a logic-tree Agent-as-a-Judge benchmark for financial\n * research reports, but there is no stable public scorer package wired here.\n * Live mode therefore requires a local data export whose rows carry the official\n * judge prompt/template/logic tree. The adapter refuses to invent a judge.\n */\n\nimport { readFile, stat } from 'node:fs/promises'\nimport { join } from 'node:path'\nimport { runBenchRouterTurn } from '../router-turn'\nimport { benchRoot } from './_harness'\nimport type { BenchmarkAdapter, BenchScore, BenchTask, LoadOptions } from './types'\n\nconst FIXTURES = join(benchRoot, 'fixtures', 'finresearchbench.json')\n\ninterface FinResearchRecord {\n  id: string\n  question: string\n  category?: string\n  reference_answer?: string\n  reference_report?: string\n  logic_tree?: unknown\n  rubric?: unknown\n  judge_system_prompt?: string\n  judge_prompt_template?: string\n}\n\ninterface FinResearchMeta {\n  id: string\n  category: string\n  question: string\n  referenceAnswer: string\n  referenceReport: string\n  logicTree: unknown\n  rubric: unknown\n  judgeSystemPrompt?: string\n  judgePromptTemplate?: string\n  scoring: 'official-logic-tree-judge' | 'fixture-exact-reference'\n}\n\nconst dataFile = (): string | undefined => process.env.FINRESEARCHBENCH_DATA_FILE\n\nfunction routerConfig(): { baseUrl: string; key: string; model: string } {\n  const key = process.env.TANGLE_API_KEY\n  if (!key) throw new Error('TANGLE_API_KEY is required for FinResearchBench live LLM judging')\n  return {\n    baseUrl: process.env.ROUTER_BASE ?? 'https://router.tangle.tools/v1',\n    key,\n    model: process.env.FINRESEARCHBENCH_JUDGE_MODEL ?? process.env.JUDGE_MODEL ?? 'deepseek-v4-flash',\n  }\n}\n\nasync function assertReadable(path: string, label: string): Promise<void> {\n  try {\n    await stat(path)\n  } catch (err) {\n    throw new Error(`FinResearchBench: missing ${label} at ${path} (${err instanceof Error ? err.message : err})`)\n  }\n}\n\nfunction readRecords(raw: string): FinResearchRecord[] {\n  const trimmed = raw.trim()\n  if (trimmed.startsWith('[')) return JSON.parse(trimmed) as FinResearchRecord[]\n  return trimmed\n    .split(/\\r?\\n/)\n    .map((line) => line.trim())\n    .filter((line) => line.length > 0)\n    .map((line) => JSON.parse(line) as FinResearchRecord)\n}\n\nfunction assertLiveJudgeFields(records: readonly FinResearchRecord[], source: string): void {\n  const missing = records\n    .filter((row) => !row.judge_system_prompt || !row.judge_prompt_template)\n    .map((row) => row.id)\n  if (missing.length > 0) {\n    throw new Error(\n      `FinResearchBench live rows from ${source} missing official judge prompts for ${missing.length}/${records.length} row(s): ${missing.slice(0, 5).join(', ')}. ` +\n        'Use a benchmark export with judge_system_prompt and judge_prompt_template, or do not score this benchmark live.',\n    )\n  }\n}\n\nfunction rowToTask(row: FinResearchRecord, fixturesMode: boolean): BenchTask {\n  const referenceAnswer = row.reference_answer ?? ''\n  const referenceReport = row.reference_report ?? referenceAnswer\n  const meta: FinResearchMeta = {\n    id: row.id,\n    category: row.category ?? 'unknown',\n    question: row.question,\n    referenceAnswer,\n    referenceReport,\n    logicTree: row.logic_tree ?? null,\n    rubric: row.rubric ?? null,\n    judgeSystemPrompt: row.judge_system_prompt,\n    judgePromptTemplate: row.judge_prompt_template,\n    scoring: fixturesMode ? 'fixture-exact-reference' : 'official-logic-tree-judge',\n  }\n  return {\n    id: row.id,\n    split: row.category,\n    prompt: [\n      'Complete this FinResearchBench financial research task.',\n      'Produce a decision-grade research answer with explicit reasoning, evidence, and final conclusion.',\n      '',\n      row.question,\n    ].join('\\n'),\n    metadata: meta as unknown as Record<string, unknown>,\n  }\n}\n\nfunction readMeta(task: BenchTask): FinResearchMeta {\n  const md = task.metadata\n  if (!md || typeof md.question !== 'string') {\n    throw new Error(`FinResearchBench task ${task.id} missing metadata — loadTasks did not populate it`)\n  }\n  return md as unknown as FinResearchMeta\n}\n\nfunction selectRows(rows: FinResearchRecord[], opts: LoadOptions, fixturesMode: boolean): BenchTask[] {\n  let tasks = rows.map((row) => rowToTask(row, fixturesMode))\n  if (opts.split) tasks = tasks.filter((task) => task.split === opts.split)\n  if (opts.ids) {\n    const want = new Set(opts.ids)\n    tasks = tasks.filter((task) => want.has(task.id))\n  } else if (opts.limit !== undefined) {\n    tasks = tasks.slice(0, opts.limit)\n  }\n  if (tasks.length === 0) throw new Error(`FinResearchBench: no tasks matched ${JSON.stringify(opts)}`)\n  return tasks\n}\n\nasync function loadFixtures(opts: LoadOptions): Promise<BenchTask[]> {\n  const records = readRecords(await readFile(FIXTURES, 'utf8'))\n  console.warn(`[finresearchbench] FINRESEARCHBENCH_FIXTURES=1 — loading ${records.length} adapter fixtures`)\n  return selectRows(records, opts, true)\n}\n\nasync function loadOfficialTasks(path: string, opts: LoadOptions): Promise<BenchTask[]> {\n  const records = readRecords(await readFile(path, 'utf8'))\n  assertLiveJudgeFields(records, path)\n  return selectRows(records, opts, false)\n}\n\nfunction fillTemplate(meta: FinResearchMeta, response: string): string {\n  const template = meta.judgePromptTemplate\n  if (!template) throw new Error(`FinResearchBench task ${meta.id} missing judge_prompt_template`)\n  return template\n    .replaceAll('{question}', meta.question)\n    .replaceAll('{response}', response)\n    .replaceAll('{reference_answer}', meta.referenceAnswer)\n    .replaceAll('{reference_report}', meta.referenceReport)\n    .replaceAll('{logic_tree}', JSON.stringify(meta.logicTree, null, 2))\n    .replaceAll('{rubric}', JSON.stringify(meta.rubric, null, 2))\n}\n\nfunction parseJudgeScore(content: string): { score: number; raw: unknown } {\n  const candidates: string[] = []\n  for (const m of content.matchAll(/```(?:json)?\\s*([\\s\\S]*?)```/g)) candidates.push(m[1].trim())\n  for (const m of content.matchAll(/\\{[\\s\\S]*?\\}/g)) candidates.push(m[0])\n  candidates.push(content.trim())\n  for (const candidate of candidates) {\n    let parsed: Record<string, unknown>\n    try {\n      parsed = JSON.parse(candidate) as Record<string, unknown>\n    } catch {\n      continue\n    }\n    const raw = parsed.score ?? parsed.overall_score ?? parsed.total_score ?? parsed.answer_score\n    const n = typeof raw === 'number' ? raw : typeof raw === 'string' ? Number(raw) : NaN\n    if (!Number.isFinite(n)) continue\n    if (n < 0) throw new Error(`FinResearchBench judge score is negative: ${JSON.stringify(parsed)}`)\n    const score = n <= 1 ? n : n <= 10 ? n / 10 : n / 100\n    if (score > 1) throw new Error(`FinResearchBench judge score outside supported range: ${JSON.stringify(parsed)}`)\n    return { score, raw: parsed }\n  }\n  throw new Error(`FinResearchBench judge produced no parseable JSON score: ${content.slice(0, 400)}`)\n}\n\nasync function runOfficialJudge(meta: FinResearchMeta, response: string): Promise<BenchScore> {\n  if (!meta.judgeSystemPrompt) throw new Error(`FinResearchBench task ${meta.id} missing judge_system_prompt`)\n  const router = routerConfig()\n  const turn = await runBenchRouterTurn(\n    {\n      routerBaseUrl: router.baseUrl,\n      routerKey: router.key,\n      profile: {\n        name: 'finresearchbench-judge',\n        harness: 'cli-base',\n        model: {\n          provider: 'tangle-router',\n          default: router.model,\n          metadata: { temperature: 0 },\n        },\n        prompt: { systemPrompt: meta.judgeSystemPrompt },\n      },\n    },\n    fillTemplate(meta, response),\n  )\n  const content = turn.finalText\n  if (!content) throw new Error('FinResearchBench judge returned no message content')\n  const { score, raw } = parseJudgeScore(content)\n  return {\n    resolved: score >= Number(process.env.FINRESEARCHBENCH_PASS_THRESHOLD ?? 0.8),\n    score,\n    detail: JSON.stringify({ scoring: meta.scoring, category: meta.category, judgeModel: router.model, raw }),\n  }\n}\n\nfunction normalizeText(value: string): string {\n  return value.toLowerCase().replace(/\\s+/g, ' ').trim()\n}\n\nfunction scoreFixture(meta: FinResearchMeta, artifact: string): BenchScore {\n  const answer = normalizeText(meta.referenceAnswer || meta.referenceReport)\n  const response = normalizeText(artifact)\n  const score = answer.length > 0 && response.includes(answer) ? 1 : 0\n  return {\n    resolved: score === 1,\n    score,\n    detail: JSON.stringify({ scoring: meta.scoring, category: meta.category }),\n  }\n}\n\nexport function createFinResearchBenchAdapter(): BenchmarkAdapter {\n  const fixturesMode = process.env.FINRESEARCHBENCH_FIXTURES === '1'\n\n  return {\n    name: 'finresearchbench',\n\n    async preflight() {\n      if (fixturesMode) {\n        await assertReadable(FIXTURES, 'fixture file')\n        return\n      }\n      const file = dataFile()\n      if (!file) {\n        throw new Error(\n          'FINRESEARCHBENCH_DATA_FILE is required. Fix: export the official FinResearchBench rows as JSON/JSONL with judge_system_prompt and judge_prompt_template fields, then set FINRESEARCHBENCH_DATA_FILE=/path/to/export.jsonl.',\n        )\n      }\n      routerConfig()\n      await assertReadable(file, 'data file')\n      await loadOfficialTasks(file, { limit: 1 })\n    },\n\n    async loadTasks(opts: LoadOptions = {}) {\n      if (fixturesMode) return loadFixtures(opts)\n      const file = dataFile()\n      if (!file) throw new Error('FINRESEARCHBENCH_DATA_FILE is required to load FinResearchBench rows')\n      return loadOfficialTasks(file, opts)\n    },\n\n    async goldArtifact(task: BenchTask) {\n      const meta = readMeta(task)\n      return meta.referenceReport || meta.referenceAnswer || undefined\n    },\n\n    async judge(task: BenchTask, artifact: string): Promise<BenchScore> {\n      const meta = readMeta(task)\n      if (fixturesMode) return scoreFixture(meta, artifact)\n      if (artifact.trim().length === 0) {\n        return {\n          resolved: false,\n          score: 0,\n          detail: JSON.stringify({ scoring: meta.scoring, category: meta.category, reason: 'empty answer' }),\n        }\n      }\n      return runOfficialJudge(meta, artifact)\n    },\n  }\n}\n"],"mappings":";;;;;;;;;;;;;AAeA,MAAM,WAAW,KAAK,WAAW,YAAY,uBAAuB;AA2BpE,MAAM,iBAAqC,QAAQ,IAAI;AAEvD,SAAS,eAAgE;CACvE,MAAM,MAAM,QAAQ,IAAI;CACxB,IAAI,CAAC,KAAK,MAAM,IAAI,MAAM,kEAAkE;CAC5F,OAAO;EACL,SAAS,QAAQ,IAAI,eAAe;EACpC;EACA,OAAO,QAAQ,IAAI,gCAAgC,QAAQ,IAAI,eAAe;CAChF;AACF;AAEA,eAAe,eAAe,MAAc,OAA8B;CACxE,IAAI;EACF,MAAM,KAAK,IAAI;CACjB,SAAS,KAAK;EACZ,MAAM,IAAI,MAAM,6BAA6B,MAAM,MAAM,KAAK,IAAI,eAAe,QAAQ,IAAI,UAAU,IAAI,EAAE;CAC/G;AACF;AAEA,SAAS,YAAY,KAAkC;CACrD,MAAM,UAAU,IAAI,KAAK;CACzB,IAAI,QAAQ,WAAW,GAAG,GAAG,OAAO,KAAK,MAAM,OAAO;CACtD,OAAO,QACJ,MAAM,OAAO,CAAC,CACd,KAAK,SAAS,KAAK,KAAK,CAAC,CAAC,CAC1B,QAAQ,SAAS,KAAK,SAAS,CAAC,CAAC,CACjC,KAAK,SAAS,KAAK,MAAM,IAAI,CAAsB;AACxD;AAEA,SAAS,sBAAsB,SAAuC,QAAsB;CAC1F,MAAM,UAAU,QACb,QAAQ,QAAQ,CAAC,IAAI,uBAAuB,CAAC,IAAI,qBAAqB,CAAC,CACvE,KAAK,QAAQ,IAAI,EAAE;CACtB,IAAI,QAAQ,SAAS,GACnB,MAAM,IAAI,MACR,mCAAmC,OAAO,sCAAsC,QAAQ,OAAO,GAAG,QAAQ,OAAO,WAAW,QAAQ,MAAM,GAAG,CAAC,CAAC,CAAC,KAAK,IAAI,EAAE,kHAE7J;AAEJ;AAEA,SAAS,UAAU,KAAwB,cAAkC;CAC3E,MAAM,kBAAkB,IAAI,oBAAoB;CAChD,MAAM,kBAAkB,IAAI,oBAAoB;CAChD,MAAM,OAAwB;EAC5B,IAAI,IAAI;EACR,UAAU,IAAI,YAAY;EAC1B,UAAU,IAAI;EACd;EACA;EACA,WAAW,IAAI,cAAc;EAC7B,QAAQ,IAAI,UAAU;EACtB,mBAAmB,IAAI;EACvB,qBAAqB,IAAI;EACzB,SAAS,eAAe,4BAA4B;CACtD;CACA,OAAO;EACL,IAAI,IAAI;EACR,OAAO,IAAI;EACX,QAAQ;GACN;GACA;GACA;GACA,IAAI;EACN,CAAC,CAAC,KAAK,IAAI;EACX,UAAU;CACZ;AACF;AAEA,SAAS,SAAS,MAAkC;CAClD,MAAM,KAAK,KAAK;CAChB,IAAI,CAAC,MAAM,OAAO,GAAG,aAAa,UAChC,MAAM,IAAI,MAAM,yBAAyB,KAAK,GAAG,kDAAkD;CAErG,OAAO;AACT;AAEA,SAAS,WAAW,MAA2B,MAAmB,cAAoC;CACpG,IAAI,QAAQ,KAAK,KAAK,QAAQ,UAAU,KAAK,YAAY,CAAC;CAC1D,IAAI,KAAK,OAAO,QAAQ,MAAM,QAAQ,SAAS,KAAK,UAAU,KAAK,KAAK;CACxE,IAAI,KAAK,KAAK;EACZ,MAAM,OAAO,IAAI,IAAI,KAAK,GAAG;EAC7B,QAAQ,MAAM,QAAQ,SAAS,KAAK,IAAI,KAAK,EAAE,CAAC;CAClD,OAAO,IAAI,KAAK,UAAU,KAAA,GACxB,QAAQ,MAAM,MAAM,GAAG,KAAK,KAAK;CAEnC,IAAI,MAAM,WAAW,GAAG,MAAM,IAAI,MAAM,sCAAsC,KAAK,UAAU,IAAI,GAAG;CACpG,OAAO;AACT;AAEA,eAAe,aAAa,MAAyC;CACnE,MAAM,UAAU,YAAY,MAAM,SAAS,UAAU,MAAM,CAAC;CAC5D,QAAQ,KAAK,4DAA4D,QAAQ,OAAO,kBAAkB;CAC1G,OAAO,WAAW,SAAS,MAAM,IAAI;AACvC;AAEA,eAAe,kBAAkB,MAAc,MAAyC;CACtF,MAAM,UAAU,YAAY,MAAM,SAAS,MAAM,MAAM,CAAC;CACxD,sBAAsB,SAAS,IAAI;CACnC,OAAO,WAAW,SAAS,MAAM,KAAK;AACxC;AAEA,SAAS,aAAa,MAAuB,UAA0B;CACrE,MAAM,WAAW,KAAK;CACtB,IAAI,CAAC,UAAU,MAAM,IAAI,MAAM,yBAAyB,KAAK,GAAG,+BAA+B;CAC/F,OAAO,SACJ,WAAW,cAAc,KAAK,QAAQ,CAAC,CACvC,WAAW,cAAc,QAAQ,CAAC,CAClC,WAAW,sBAAsB,KAAK,eAAe,CAAC,CACtD,WAAW,sBAAsB,KAAK,eAAe,CAAC,CACtD,WAAW,gBAAgB,KAAK,UAAU,KAAK,WAAW,MAAM,CAAC,CAAC,CAAC,CACnE,WAAW,YAAY,KAAK,UAAU,KAAK,QAAQ,MAAM,CAAC,CAAC;AAChE;AAEA,SAAS,gBAAgB,SAAkD;CACzE,MAAM,aAAuB,CAAC;CAC9B,KAAK,MAAM,KAAK,QAAQ,SAAS,+BAA+B,GAAG,WAAW,KAAK,EAAE,EAAE,CAAC,KAAK,CAAC;CAC9F,KAAK,MAAM,KAAK,QAAQ,SAAS,eAAe,GAAG,WAAW,KAAK,EAAE,EAAE;CACvE,WAAW,KAAK,QAAQ,KAAK,CAAC;CAC9B,KAAK,MAAM,aAAa,YAAY;EAClC,IAAI;EACJ,IAAI;GACF,SAAS,KAAK,MAAM,SAAS;EAC/B,QAAQ;GACN;EACF;EACA,MAAM,MAAM,OAAO,SAAS,OAAO,iBAAiB,OAAO,eAAe,OAAO;EACjF,MAAM,IAAI,OAAO,QAAQ,WAAW,MAAM,OAAO,QAAQ,WAAW,OAAO,GAAG,IAAI;EAClF,IAAI,CAAC,OAAO,SAAS,CAAC,GAAG;EACzB,IAAI,IAAI,GAAG,MAAM,IAAI,MAAM,6CAA6C,KAAK,UAAU,MAAM,GAAG;EAChG,MAAM,QAAQ,KAAK,IAAI,IAAI,KAAK,KAAK,IAAI,KAAK,IAAI;EAClD,IAAI,QAAQ,GAAG,MAAM,IAAI,MAAM,yDAAyD,KAAK,UAAU,MAAM,GAAG;EAChH,OAAO;GAAE;GAAO,KAAK;EAAO;CAC9B;CACA,MAAM,IAAI,MAAM,4DAA4D,QAAQ,MAAM,GAAG,GAAG,GAAG;AACrG;AAEA,eAAe,iBAAiB,MAAuB,UAAuC;CAC5F,IAAI,CAAC,KAAK,mBAAmB,MAAM,IAAI,MAAM,yBAAyB,KAAK,GAAG,6BAA6B;CAC3G,MAAM,SAAS,aAAa;CAkB5B,MAAM,WAAU,MAjBG,mBACjB;EACE,eAAe,OAAO;EACtB,WAAW,OAAO;EAClB,SAAS;GACP,MAAM;GACN,SAAS;GACT,OAAO;IACL,UAAU;IACV,SAAS,OAAO;IAChB,UAAU,EAAE,aAAa,EAAE;GAC7B;GACA,QAAQ,EAAE,cAAc,KAAK,kBAAkB;EACjD;CACF,GACA,aAAa,MAAM,QAAQ,CAC7B,EAAA,CACqB;CACrB,IAAI,CAAC,SAAS,MAAM,IAAI,MAAM,oDAAoD;CAClF,MAAM,EAAE,OAAO,QAAQ,gBAAgB,OAAO;CAC9C,OAAO;EACL,UAAU,SAAS,OAAO,QAAQ,IAAI,mCAAmC,EAAG;EAC5E;EACA,QAAQ,KAAK,UAAU;GAAE,SAAS,KAAK;GAAS,UAAU,KAAK;GAAU,YAAY,OAAO;GAAO;EAAI,CAAC;CAC1G;AACF;AAEA,SAAS,cAAc,OAAuB;CAC5C,OAAO,MAAM,YAAY,CAAC,CAAC,QAAQ,QAAQ,GAAG,CAAC,CAAC,KAAK;AACvD;AAEA,SAAS,aAAa,MAAuB,UAA8B;CACzE,MAAM,SAAS,cAAc,KAAK,mBAAmB,KAAK,eAAe;CACzE,MAAM,WAAW,cAAc,QAAQ;CACvC,MAAM,QAAQ,OAAO,SAAS,KAAK,SAAS,SAAS,MAAM,IAAI,IAAI;CACnE,OAAO;EACL,UAAU,UAAU;EACpB;EACA,QAAQ,KAAK,UAAU;GAAE,SAAS,KAAK;GAAS,UAAU,KAAK;EAAS,CAAC;CAC3E;AACF;AAEA,SAAgB,gCAAkD;CAChE,MAAM,eAAe,QAAQ,IAAI,8BAA8B;CAE/D,OAAO;EACL,MAAM;EAEN,MAAM,YAAY;GAChB,IAAI,cAAc;IAChB,MAAM,eAAe,UAAU,cAAc;IAC7C;GACF;GACA,MAAM,OAAO,SAAS;GACtB,IAAI,CAAC,MACH,MAAM,IAAI,MACR,4NACF;GAEF,aAAa;GACb,MAAM,eAAe,MAAM,WAAW;GACtC,MAAM,kBAAkB,MAAM,EAAE,OAAO,EAAE,CAAC;EAC5C;EAEA,MAAM,UAAU,OAAoB,CAAC,GAAG;GACtC,IAAI,cAAc,OAAO,aAAa,IAAI;GAC1C,MAAM,OAAO,SAAS;GACtB,IAAI,CAAC,MAAM,MAAM,IAAI,MAAM,sEAAsE;GACjG,OAAO,kBAAkB,MAAM,IAAI;EACrC;EAEA,MAAM,aAAa,MAAiB;GAClC,MAAM,OAAO,SAAS,IAAI;GAC1B,OAAO,KAAK,mBAAmB,KAAK,mBAAmB,KAAA;EACzD;EAEA,MAAM,MAAM,MAAiB,UAAuC;GAClE,MAAM,OAAO,SAAS,IAAI;GAC1B,IAAI,cAAc,OAAO,aAAa,MAAM,QAAQ;GACpD,IAAI,SAAS,KAAK,CAAC,CAAC,WAAW,GAC7B,OAAO;IACL,UAAU;IACV,OAAO;IACP,QAAQ,KAAK,UAAU;KAAE,SAAS,KAAK;KAAS,UAAU,KAAK;KAAU,QAAQ;IAAe,CAAC;GACnG;GAEF,OAAO,iBAAiB,MAAM,QAAQ;EACxC;CACF;AACF"}
+{"version":3,"file":"finresearchbench.js","names":[],"sources":["../../src/benchmarks/finresearchbench.ts"],"sourcesContent":["/**\n * FinResearchBench-compatible adapter.\n *\n * The paper defines a logic-tree Agent-as-a-Judge benchmark for financial\n * research reports, but there is no stable public scorer package wired here.\n * Live mode therefore requires a local data export whose rows carry the official\n * judge prompt/template/logic tree. The adapter refuses to invent a judge.\n */\n\nimport { readFile, stat } from 'node:fs/promises'\nimport { join } from 'node:path'\nimport type { AgentTurnUsage } from '@tangle-network/agent-runtime/kernel'\nimport { runBenchRouterTurn } from '../router-turn'\nimport { benchRoot } from './_harness'\nimport type { BenchmarkAdapter, BenchScore, BenchTask, LoadOptions } from './types'\n\nconst FIXTURES = join(benchRoot, 'fixtures', 'finresearchbench.json')\n\ninterface FinResearchRecord {\n  id: string\n  question: string\n  category?: string\n  reference_answer?: string\n  reference_report?: string\n  logic_tree?: unknown\n  rubric?: unknown\n  judge_system_prompt?: string\n  judge_prompt_template?: string\n}\n\ninterface FinResearchMeta {\n  id: string\n  category: string\n  question: string\n  referenceAnswer: string\n  referenceReport: string\n  logicTree: unknown\n  rubric: unknown\n  judgeSystemPrompt?: string\n  judgePromptTemplate?: string\n  scoring: 'official-logic-tree-judge' | 'fixture-exact-reference'\n}\n\nconst dataFile = (): string | undefined => process.env.FINRESEARCHBENCH_DATA_FILE\n\nfunction routerConfig(): { baseUrl: string; key: string; model: string } {\n  const key = process.env.TANGLE_API_KEY\n  if (!key) throw new Error('TANGLE_API_KEY is required for FinResearchBench live LLM judging')\n  return {\n    baseUrl: process.env.ROUTER_BASE ?? 'https://router.tangle.tools/v1',\n    key,\n    model: process.env.FINRESEARCHBENCH_JUDGE_MODEL ?? process.env.JUDGE_MODEL ?? 'deepseek-v4-flash',\n  }\n}\n\nasync function assertReadable(path: string, label: string): Promise<void> {\n  try {\n    await stat(path)\n  } catch (err) {\n    throw new Error(`FinResearchBench: missing ${label} at ${path} (${err instanceof Error ? err.message : err})`)\n  }\n}\n\nfunction readRecords(raw: string): FinResearchRecord[] {\n  const trimmed = raw.trim()\n  if (trimmed.startsWith('[')) return JSON.parse(trimmed) as FinResearchRecord[]\n  return trimmed\n    .split(/\\r?\\n/)\n    .map((line) => line.trim())\n    .filter((line) => line.length > 0)\n    .map((line) => JSON.parse(line) as FinResearchRecord)\n}\n\nfunction assertLiveJudgeFields(records: readonly FinResearchRecord[], source: string): void {\n  const missing = records\n    .filter((row) => !row.judge_system_prompt || !row.judge_prompt_template)\n    .map((row) => row.id)\n  if (missing.length > 0) {\n    throw new Error(\n      `FinResearchBench live rows from ${source} missing official judge prompts for ${missing.length}/${records.length} row(s): ${missing.slice(0, 5).join(', ')}. ` +\n        'Use a benchmark export with judge_system_prompt and judge_prompt_template, or do not score this benchmark live.',\n    )\n  }\n}\n\nfunction rowToTask(row: FinResearchRecord, fixturesMode: boolean): BenchTask {\n  const referenceAnswer = row.reference_answer ?? ''\n  const referenceReport = row.reference_report ?? referenceAnswer\n  const meta: FinResearchMeta = {\n    id: row.id,\n    category: row.category ?? 'unknown',\n    question: row.question,\n    referenceAnswer,\n    referenceReport,\n    logicTree: row.logic_tree ?? null,\n    rubric: row.rubric ?? null,\n    judgeSystemPrompt: row.judge_system_prompt,\n    judgePromptTemplate: row.judge_prompt_template,\n    scoring: fixturesMode ? 'fixture-exact-reference' : 'official-logic-tree-judge',\n  }\n  return {\n    id: row.id,\n    split: row.category,\n    prompt: [\n      'Complete this FinResearchBench financial research task.',\n      'Produce a decision-grade research answer with explicit reasoning, evidence, and final conclusion.',\n      '',\n      row.question,\n    ].join('\\n'),\n    metadata: meta as unknown as Record<string, unknown>,\n  }\n}\n\nfunction readMeta(task: BenchTask): FinResearchMeta {\n  const md = task.metadata\n  if (!md || typeof md.question !== 'string') {\n    throw new Error(`FinResearchBench task ${task.id} missing metadata — loadTasks did not populate it`)\n  }\n  return md as unknown as FinResearchMeta\n}\n\nfunction selectRows(rows: FinResearchRecord[], opts: LoadOptions, fixturesMode: boolean): BenchTask[] {\n  let tasks = rows.map((row) => rowToTask(row, fixturesMode))\n  if (opts.split) tasks = tasks.filter((task) => task.split === opts.split)\n  if (opts.ids) {\n    const want = new Set(opts.ids)\n    tasks = tasks.filter((task) => want.has(task.id))\n  } else if (opts.limit !== undefined) {\n    tasks = tasks.slice(0, opts.limit)\n  }\n  if (tasks.length === 0) throw new Error(`FinResearchBench: no tasks matched ${JSON.stringify(opts)}`)\n  return tasks\n}\n\nasync function loadFixtures(opts: LoadOptions): Promise<BenchTask[]> {\n  const records = readRecords(await readFile(FIXTURES, 'utf8'))\n  console.warn(`[finresearchbench] FINRESEARCHBENCH_FIXTURES=1 — loading ${records.length} adapter fixtures`)\n  return selectRows(records, opts, true)\n}\n\nasync function loadOfficialTasks(path: string, opts: LoadOptions): Promise<BenchTask[]> {\n  const records = readRecords(await readFile(path, 'utf8'))\n  assertLiveJudgeFields(records, path)\n  return selectRows(records, opts, false)\n}\n\nfunction fillTemplate(meta: FinResearchMeta, response: string): string {\n  const template = meta.judgePromptTemplate\n  if (!template) throw new Error(`FinResearchBench task ${meta.id} missing judge_prompt_template`)\n  return template\n    .replaceAll('{question}', meta.question)\n    .replaceAll('{response}', response)\n    .replaceAll('{reference_answer}', meta.referenceAnswer)\n    .replaceAll('{reference_report}', meta.referenceReport)\n    .replaceAll('{logic_tree}', JSON.stringify(meta.logicTree, null, 2))\n    .replaceAll('{rubric}', JSON.stringify(meta.rubric, null, 2))\n}\n\nfunction parseJudgeScore(content: string): { score: number; raw: unknown } {\n  const candidates: string[] = []\n  for (const m of content.matchAll(/```(?:json)?\\s*([\\s\\S]*?)```/g)) candidates.push(m[1].trim())\n  for (const m of content.matchAll(/\\{[\\s\\S]*?\\}/g)) candidates.push(m[0])\n  candidates.push(content.trim())\n  for (const candidate of candidates) {\n    let parsed: Record<string, unknown>\n    try {\n      parsed = JSON.parse(candidate) as Record<string, unknown>\n    } catch {\n      continue\n    }\n    const raw = parsed.score ?? parsed.overall_score ?? parsed.total_score ?? parsed.answer_score\n    const n = typeof raw === 'number' ? raw : typeof raw === 'string' ? Number(raw) : NaN\n    if (!Number.isFinite(n)) continue\n    if (n < 0) throw new Error(`FinResearchBench judge score is negative: ${JSON.stringify(parsed)}`)\n    const score = n <= 1 ? n : n <= 10 ? n / 10 : n / 100\n    if (score > 1) throw new Error(`FinResearchBench judge score outside supported range: ${JSON.stringify(parsed)}`)\n    return { score, raw: parsed }\n  }\n  throw new Error(`FinResearchBench judge produced no parseable JSON score: ${content.slice(0, 400)}`)\n}\n\n/**\n * Map one drained judge turn to a BenchScore. The judge turn's exact Runtime\n * usage record lands on `judgeUsage` verbatim, so an unproven token count or\n * dollar amount stays flagged (`tokensKnown`/`usdKnown` false) instead of\n * reading as a known zero cost.\n */\nexport function scoreOfficialJudgeTurn(\n  task: BenchTask,\n  turn: Readonly<{ finalText?: string; usage: AgentTurnUsage }>,\n  requestedJudgeModel: string,\n): BenchScore {\n  const meta = readMeta(task)\n  const content = turn.finalText\n  if (!content) throw new Error('FinResearchBench judge returned no message content')\n  const { score, raw } = parseJudgeScore(content)\n  return {\n    resolved: score >= Number(process.env.FINRESEARCHBENCH_PASS_THRESHOLD ?? 0.8),\n    score,\n    detail: JSON.stringify({ scoring: meta.scoring, category: meta.category, judgeModel: requestedJudgeModel, raw }),\n    judgeUsage: turn.usage,\n  }\n}\n\nasync function runOfficialJudge(task: BenchTask, meta: FinResearchMeta, response: string): Promise<BenchScore> {\n  if (!meta.judgeSystemPrompt) throw new Error(`FinResearchBench task ${meta.id} missing judge_system_prompt`)\n  const router = routerConfig()\n  const turn = await runBenchRouterTurn(\n    {\n      routerBaseUrl: router.baseUrl,\n      routerKey: router.key,\n      profile: {\n        name: 'finresearchbench-judge',\n        harness: 'cli-base',\n        model: {\n          provider: 'tangle-router',\n          default: router.model,\n          metadata: { temperature: 0 },\n        },\n        prompt: { systemPrompt: meta.judgeSystemPrompt },\n      },\n    },\n    fillTemplate(meta, response),\n  )\n  return scoreOfficialJudgeTurn(task, turn, router.model)\n}\n\nfunction normalizeText(value: string): string {\n  return value.toLowerCase().replace(/\\s+/g, ' ').trim()\n}\n\nfunction scoreFixture(meta: FinResearchMeta, artifact: string): BenchScore {\n  const answer = normalizeText(meta.referenceAnswer || meta.referenceReport)\n  const response = normalizeText(artifact)\n  const score = answer.length > 0 && response.includes(answer) ? 1 : 0\n  return {\n    resolved: score === 1,\n    score,\n    detail: JSON.stringify({ scoring: meta.scoring, category: meta.category }),\n  }\n}\n\nexport function createFinResearchBenchAdapter(): BenchmarkAdapter {\n  const fixturesMode = process.env.FINRESEARCHBENCH_FIXTURES === '1'\n\n  return {\n    name: 'finresearchbench',\n\n    async preflight() {\n      if (fixturesMode) {\n        await assertReadable(FIXTURES, 'fixture file')\n        return\n      }\n      const file = dataFile()\n      if (!file) {\n        throw new Error(\n          'FINRESEARCHBENCH_DATA_FILE is required. Fix: export the official FinResearchBench rows as JSON/JSONL with judge_system_prompt and judge_prompt_template fields, then set FINRESEARCHBENCH_DATA_FILE=/path/to/export.jsonl.',\n        )\n      }\n      routerConfig()\n      await assertReadable(file, 'data file')\n      await loadOfficialTasks(file, { limit: 1 })\n    },\n\n    async loadTasks(opts: LoadOptions = {}) {\n      if (fixturesMode) return loadFixtures(opts)\n      const file = dataFile()\n      if (!file) throw new Error('FINRESEARCHBENCH_DATA_FILE is required to load FinResearchBench rows')\n      return loadOfficialTasks(file, opts)\n    },\n\n    async goldArtifact(task: BenchTask) {\n      const meta = readMeta(task)\n      return meta.referenceReport || meta.referenceAnswer || undefined\n    },\n\n    async judge(task: BenchTask, artifact: string): Promise<BenchScore> {\n      const meta = readMeta(task)\n      if (fixturesMode) return scoreFixture(meta, artifact)\n      if (artifact.trim().length === 0) {\n        return {\n          resolved: false,\n          score: 0,\n          detail: JSON.stringify({ scoring: meta.scoring, category: meta.category, reason: 'empty answer' }),\n        }\n      }\n      return runOfficialJudge(task, meta, artifact)\n    },\n  }\n}\n"],"mappings":";;;;;;;;;;;;;AAgBA,MAAM,WAAW,KAAK,WAAW,YAAY,uBAAuB;AA2BpE,MAAM,iBAAqC,QAAQ,IAAI;AAEvD,SAAS,eAAgE;CACvE,MAAM,MAAM,QAAQ,IAAI;CACxB,IAAI,CAAC,KAAK,MAAM,IAAI,MAAM,kEAAkE;CAC5F,OAAO;EACL,SAAS,QAAQ,IAAI,eAAe;EACpC;EACA,OAAO,QAAQ,IAAI,gCAAgC,QAAQ,IAAI,eAAe;CAChF;AACF;AAEA,eAAe,eAAe,MAAc,OAA8B;CACxE,IAAI;EACF,MAAM,KAAK,IAAI;CACjB,SAAS,KAAK;EACZ,MAAM,IAAI,MAAM,6BAA6B,MAAM,MAAM,KAAK,IAAI,eAAe,QAAQ,IAAI,UAAU,IAAI,EAAE;CAC/G;AACF;AAEA,SAAS,YAAY,KAAkC;CACrD,MAAM,UAAU,IAAI,KAAK;CACzB,IAAI,QAAQ,WAAW,GAAG,GAAG,OAAO,KAAK,MAAM,OAAO;CACtD,OAAO,QACJ,MAAM,OAAO,CAAC,CACd,KAAK,SAAS,KAAK,KAAK,CAAC,CAAC,CAC1B,QAAQ,SAAS,KAAK,SAAS,CAAC,CAAC,CACjC,KAAK,SAAS,KAAK,MAAM,IAAI,CAAsB;AACxD;AAEA,SAAS,sBAAsB,SAAuC,QAAsB;CAC1F,MAAM,UAAU,QACb,QAAQ,QAAQ,CAAC,IAAI,uBAAuB,CAAC,IAAI,qBAAqB,CAAC,CACvE,KAAK,QAAQ,IAAI,EAAE;CACtB,IAAI,QAAQ,SAAS,GACnB,MAAM,IAAI,MACR,mCAAmC,OAAO,sCAAsC,QAAQ,OAAO,GAAG,QAAQ,OAAO,WAAW,QAAQ,MAAM,GAAG,CAAC,CAAC,CAAC,KAAK,IAAI,EAAE,kHAE7J;AAEJ;AAEA,SAAS,UAAU,KAAwB,cAAkC;CAC3E,MAAM,kBAAkB,IAAI,oBAAoB;CAChD,MAAM,kBAAkB,IAAI,oBAAoB;CAChD,MAAM,OAAwB;EAC5B,IAAI,IAAI;EACR,UAAU,IAAI,YAAY;EAC1B,UAAU,IAAI;EACd;EACA;EACA,WAAW,IAAI,cAAc;EAC7B,QAAQ,IAAI,UAAU;EACtB,mBAAmB,IAAI;EACvB,qBAAqB,IAAI;EACzB,SAAS,eAAe,4BAA4B;CACtD;CACA,OAAO;EACL,IAAI,IAAI;EACR,OAAO,IAAI;EACX,QAAQ;GACN;GACA;GACA;GACA,IAAI;EACN,CAAC,CAAC,KAAK,IAAI;EACX,UAAU;CACZ;AACF;AAEA,SAAS,SAAS,MAAkC;CAClD,MAAM,KAAK,KAAK;CAChB,IAAI,CAAC,MAAM,OAAO,GAAG,aAAa,UAChC,MAAM,IAAI,MAAM,yBAAyB,KAAK,GAAG,kDAAkD;CAErG,OAAO;AACT;AAEA,SAAS,WAAW,MAA2B,MAAmB,cAAoC;CACpG,IAAI,QAAQ,KAAK,KAAK,QAAQ,UAAU,KAAK,YAAY,CAAC;CAC1D,IAAI,KAAK,OAAO,QAAQ,MAAM,QAAQ,SAAS,KAAK,UAAU,KAAK,KAAK;CACxE,IAAI,KAAK,KAAK;EACZ,MAAM,OAAO,IAAI,IAAI,KAAK,GAAG;EAC7B,QAAQ,MAAM,QAAQ,SAAS,KAAK,IAAI,KAAK,EAAE,CAAC;CAClD,OAAO,IAAI,KAAK,UAAU,KAAA,GACxB,QAAQ,MAAM,MAAM,GAAG,KAAK,KAAK;CAEnC,IAAI,MAAM,WAAW,GAAG,MAAM,IAAI,MAAM,sCAAsC,KAAK,UAAU,IAAI,GAAG;CACpG,OAAO;AACT;AAEA,eAAe,aAAa,MAAyC;CACnE,MAAM,UAAU,YAAY,MAAM,SAAS,UAAU,MAAM,CAAC;CAC5D,QAAQ,KAAK,4DAA4D,QAAQ,OAAO,kBAAkB;CAC1G,OAAO,WAAW,SAAS,MAAM,IAAI;AACvC;AAEA,eAAe,kBAAkB,MAAc,MAAyC;CACtF,MAAM,UAAU,YAAY,MAAM,SAAS,MAAM,MAAM,CAAC;CACxD,sBAAsB,SAAS,IAAI;CACnC,OAAO,WAAW,SAAS,MAAM,KAAK;AACxC;AAEA,SAAS,aAAa,MAAuB,UAA0B;CACrE,MAAM,WAAW,KAAK;CACtB,IAAI,CAAC,UAAU,MAAM,IAAI,MAAM,yBAAyB,KAAK,GAAG,+BAA+B;CAC/F,OAAO,SACJ,WAAW,cAAc,KAAK,QAAQ,CAAC,CACvC,WAAW,cAAc,QAAQ,CAAC,CAClC,WAAW,sBAAsB,KAAK,eAAe,CAAC,CACtD,WAAW,sBAAsB,KAAK,eAAe,CAAC,CACtD,WAAW,gBAAgB,KAAK,UAAU,KAAK,WAAW,MAAM,CAAC,CAAC,CAAC,CACnE,WAAW,YAAY,KAAK,UAAU,KAAK,QAAQ,MAAM,CAAC,CAAC;AAChE;AAEA,SAAS,gBAAgB,SAAkD;CACzE,MAAM,aAAuB,CAAC;CAC9B,KAAK,MAAM,KAAK,QAAQ,SAAS,+BAA+B,GAAG,WAAW,KAAK,EAAE,EAAE,CAAC,KAAK,CAAC;CAC9F,KAAK,MAAM,KAAK,QAAQ,SAAS,eAAe,GAAG,WAAW,KAAK,EAAE,EAAE;CACvE,WAAW,KAAK,QAAQ,KAAK,CAAC;CAC9B,KAAK,MAAM,aAAa,YAAY;EAClC,IAAI;EACJ,IAAI;GACF,SAAS,KAAK,MAAM,SAAS;EAC/B,QAAQ;GACN;EACF;EACA,MAAM,MAAM,OAAO,SAAS,OAAO,iBAAiB,OAAO,eAAe,OAAO;EACjF,MAAM,IAAI,OAAO,QAAQ,WAAW,MAAM,OAAO,QAAQ,WAAW,OAAO,GAAG,IAAI;EAClF,IAAI,CAAC,OAAO,SAAS,CAAC,GAAG;EACzB,IAAI,IAAI,GAAG,MAAM,IAAI,MAAM,6CAA6C,KAAK,UAAU,MAAM,GAAG;EAChG,MAAM,QAAQ,KAAK,IAAI,IAAI,KAAK,KAAK,IAAI,KAAK,IAAI;EAClD,IAAI,QAAQ,GAAG,MAAM,IAAI,MAAM,yDAAyD,KAAK,UAAU,MAAM,GAAG;EAChH,OAAO;GAAE;GAAO,KAAK;EAAO;CAC9B;CACA,MAAM,IAAI,MAAM,4DAA4D,QAAQ,MAAM,GAAG,GAAG,GAAG;AACrG;;;;;;;AAQA,SAAgB,uBACd,MACA,MACA,qBACY;CACZ,MAAM,OAAO,SAAS,IAAI;CAC1B,MAAM,UAAU,KAAK;CACrB,IAAI,CAAC,SAAS,MAAM,IAAI,MAAM,oDAAoD;CAClF,MAAM,EAAE,OAAO,QAAQ,gBAAgB,OAAO;CAC9C,OAAO;EACL,UAAU,SAAS,OAAO,QAAQ,IAAI,mCAAmC,EAAG;EAC5E;EACA,QAAQ,KAAK,UAAU;GAAE,SAAS,KAAK;GAAS,UAAU,KAAK;GAAU,YAAY;GAAqB;EAAI,CAAC;EAC/G,YAAY,KAAK;CACnB;AACF;AAEA,eAAe,iBAAiB,MAAiB,MAAuB,UAAuC;CAC7G,IAAI,CAAC,KAAK,mBAAmB,MAAM,IAAI,MAAM,yBAAyB,KAAK,GAAG,6BAA6B;CAC3G,MAAM,SAAS,aAAa;CAkB5B,OAAO,uBAAuB,MAAM,MAjBjB,mBACjB;EACE,eAAe,OAAO;EACtB,WAAW,OAAO;EAClB,SAAS;GACP,MAAM;GACN,SAAS;GACT,OAAO;IACL,UAAU;IACV,SAAS,OAAO;IAChB,UAAU,EAAE,aAAa,EAAE;GAC7B;GACA,QAAQ,EAAE,cAAc,KAAK,kBAAkB;EACjD;CACF,GACA,aAAa,MAAM,QAAQ,CAC7B,GAC0C,OAAO,KAAK;AACxD;AAEA,SAAS,cAAc,OAAuB;CAC5C,OAAO,MAAM,YAAY,CAAC,CAAC,QAAQ,QAAQ,GAAG,CAAC,CAAC,KAAK;AACvD;AAEA,SAAS,aAAa,MAAuB,UAA8B;CACzE,MAAM,SAAS,cAAc,KAAK,mBAAmB,KAAK,eAAe;CACzE,MAAM,WAAW,cAAc,QAAQ;CACvC,MAAM,QAAQ,OAAO,SAAS,KAAK,SAAS,SAAS,MAAM,IAAI,IAAI;CACnE,OAAO;EACL,UAAU,UAAU;EACpB;EACA,QAAQ,KAAK,UAAU;GAAE,SAAS,KAAK;GAAS,UAAU,KAAK;EAAS,CAAC;CAC3E;AACF;AAEA,SAAgB,gCAAkD;CAChE,MAAM,eAAe,QAAQ,IAAI,8BAA8B;CAE/D,OAAO;EACL,MAAM;EAEN,MAAM,YAAY;GAChB,IAAI,cAAc;IAChB,MAAM,eAAe,UAAU,cAAc;IAC7C;GACF;GACA,MAAM,OAAO,SAAS;GACtB,IAAI,CAAC,MACH,MAAM,IAAI,MACR,4NACF;GAEF,aAAa;GACb,MAAM,eAAe,MAAM,WAAW;GACtC,MAAM,kBAAkB,MAAM,EAAE,OAAO,EAAE,CAAC;EAC5C;EAEA,MAAM,UAAU,OAAoB,CAAC,GAAG;GACtC,IAAI,cAAc,OAAO,aAAa,IAAI;GAC1C,MAAM,OAAO,SAAS;GACtB,IAAI,CAAC,MAAM,MAAM,IAAI,MAAM,sEAAsE;GACjG,OAAO,kBAAkB,MAAM,IAAI;EACrC;EAEA,MAAM,aAAa,MAAiB;GAClC,MAAM,OAAO,SAAS,IAAI;GAC1B,OAAO,KAAK,mBAAmB,KAAK,mBAAmB,KAAA;EACzD;EAEA,MAAM,MAAM,MAAiB,UAAuC;GAClE,MAAM,OAAO,SAAS,IAAI;GAC1B,IAAI,cAAc,OAAO,aAAa,MAAM,QAAQ;GACpD,IAAI,SAAS,KAAK,CAAC,CAAC,WAAW,GAC7B,OAAO;IACL,UAAU;IACV,OAAO;IACP,QAAQ,KAAK,UAAU;KAAE,SAAS,KAAK;KAAS,UAAU,KAAK;KAAU,QAAQ;IAAe,CAAC;GACnG;GAEF,OAAO,iBAAiB,MAAM,MAAM,QAAQ;EAC9C;CACF;AACF"}

package/dist/benchmarks/tau-bench-shared.d.ts

@@ -11,8 +11,22 @@ interface TauBenchConfig {
   installHint: string;
   taskIntro: string;
 }
+/** The exact upstream identity a tau task or score is bound to. */
+interface TauUpstreamPin {
+  upstreamCommit: string;
+  upstreamVersion: string;
+}
+/**
+ * Refuse to score a task against a checkout that moved after the task was
+ * loaded: the reward recomputation would run under a different upstream
+ * identity than the one stamped on the task.
+ */
+declare function assertTauUpstreamPinUnchanged(benchName: string, taskId: string, recorded: Readonly<{
+  upstreamCommit?: string;
+  upstreamVersion?: string;
+}>, live: TauUpstreamPin): void;
 declare const tauResultsOutput: OutputAdapter<string>;
 declare function createTauBenchAdapter(config: TauBenchConfig): BenchmarkAdapter;
 //#endregion
-export { TauBenchConfig, createTauBenchAdapter, tauResultsOutput };
+export { TauBenchConfig, TauUpstreamPin, assertTauUpstreamPinUnchanged, createTauBenchAdapter, tauResultsOutput };
 //# sourceMappingURL=tau-bench-shared.d.ts.map

package/dist/benchmarks/tau-bench-shared.js

@@ -1,6 +1,8 @@
 import { runVenvPython } from "./_harness.js";
 import { readFile, stat } from "node:fs/promises";
 import { resolve } from "node:path";
+import { execFile } from "node:child_process";
+import { promisify } from "node:util";
 //#region src/benchmarks/tau-bench-shared.ts
 /**
 * Shared tau-bench adapter spine.
@@ -10,6 +12,59 @@ import { resolve } from "node:path";
 * for task loading and reward recomputation so the domain/version adapters only
 * choose env names, default domain, and fixture file.
 */
+const execFileAsync = promisify(execFile);
+/**
+* Resolve the upstream checkout's git identity. Throws when the directory is
+* not a git checkout or holds uncommitted changes, because a commit over dirty
+* state does not identify the code that produced the tasks or the score.
+*/
+async function resolveUpstreamCommit(root, benchName) {
+	let head;
+	try {
+		const { stdout } = await execFileAsync("git", [
+			"-C",
+			root,
+			"rev-parse",
+			"HEAD"
+		]);
+		head = stdout.trim();
+	} catch (err) {
+		throw new Error(`${benchName}: cannot resolve the upstream commit of ${root} (${err instanceof Error ? err.message : err}). The bench dir must be a git checkout of https://github.com/sierra-research/tau2-bench.`);
+	}
+	const { stdout: status } = await execFileAsync("git", [
+		"-C",
+		root,
+		"status",
+		"--porcelain"
+	]);
+	if (status.trim().length > 0) {
+		const dirty = status.trim().split("\n").slice(0, 5).join(", ");
+		throw new Error(`${benchName}: upstream checkout ${root} has uncommitted changes (${dirty}). Commit or stash them so the pinned commit identifies the benchmark code.`);
+	}
+	return head;
+}
+/**
+* Refuse to score a task against a checkout that moved after the task was
+* loaded: the reward recomputation would run under a different upstream
+* identity than the one stamped on the task.
+*/
+function assertTauUpstreamPinUnchanged(benchName, taskId, recorded, live) {
+	if (recorded.upstreamCommit !== void 0 && recorded.upstreamCommit !== live.upstreamCommit) throw new Error(`${benchName} task ${taskId} was loaded from upstream commit ${recorded.upstreamCommit} but the checkout is now at ${live.upstreamCommit}; reload tasks before judging.`);
+	if (recorded.upstreamVersion !== void 0 && recorded.upstreamVersion !== live.upstreamVersion) throw new Error(`${benchName} task ${taskId} was loaded with tau2 ${recorded.upstreamVersion} but the interpreter now holds tau2 ${live.upstreamVersion}; reload tasks before judging.`);
+}
+const upstreamVersionSnippet = `
+import sys
+sys.dont_write_bytecode = True
+import importlib.metadata
+try:
+    upstream_version = importlib.metadata.version("tau2")
+except importlib.metadata.PackageNotFoundError:
+    raise SystemExit(
+        "installed tau2 distribution not found in this interpreter; "
+        "run 'uv sync' inside the tau2-bench checkout and set "
+        "AGENT_BENCH_PYTHON to <checkout>/.venv/bin/python3"
+    )
+`;
 const tauResultsOutput = { parse(events) {
 	let text = "";
 	for (const ev of events) {
@@ -32,14 +87,15 @@ function benchDir(config) {
 function benchDomain(config) {
 	return process.env[config.domainEnv] ?? config.defaultDomain;
 }
-function rowToTask(row, config, split) {
+function rowToTask(row, config, split, pin) {
 	const meta = {
 		taskId: row.id,
 		domain: row.domain,
 		split,
 		userScenario: row.user_scenario,
 		description: row.description,
-		evaluationCriteria: row.evaluation_criteria
+		evaluationCriteria: row.evaluation_criteria,
+		...pin ?? {}
 	};
 	return {
 		id: row.id,
@@ -61,8 +117,8 @@ function readMeta(task, benchName) {
 	if (!md || typeof md.taskId !== "string" || typeof md.domain !== "string") throw new Error(`${benchName} task ${task.id} missing metadata — loadTasks did not populate it`);
 	return md;
 }
-function selectRows(rows, opts, config, split) {
-	let tasks = rows.map((row) => rowToTask(row, config, split));
+function selectRows(rows, opts, config, split, pin) {
+	let tasks = rows.map((row) => rowToTask(row, config, split, pin));
 	if (opts.ids) {
 		const want = new Set(opts.ids);
 		tasks = tasks.filter((task) => want.has(task.id));
@@ -76,13 +132,15 @@ async function loadFixtures(config, opts) {
 	return selectRows(rows, opts, config, opts.split);
 }
 async function loadOfficialTasks(config, root, opts) {
-	const stdout = await runVenvPython(`
+	const domain = benchDomain(config);
+	const script = `
 import json, sys
 from pathlib import Path
 root = Path(sys.argv[1])
 domain = sys.argv[2]
 split = sys.argv[3] or None
 sys.path.insert(0, str(root / "src"))
+${upstreamVersionSnippet}
 from tau2.registry import registry
 loader = registry.get_tasks_loader(domain)
 tasks = loader(split)
@@ -91,13 +149,21 @@ for task in tasks:
     row = task.model_dump(mode="json")
     row["domain"] = domain
     rows.append(row)
-print(json.dumps(rows))
-`, [
+print(json.dumps({"upstreamVersion": upstream_version, "rows": rows}))
+`;
+	const upstreamCommit = await resolveUpstreamCommit(root, config.name);
+	const stdout = await runVenvPython(script, [
 		root,
-		benchDomain(config),
+		domain,
 		opts.split ?? ""
 	]);
-	return selectRows(JSON.parse(stdout), opts, config, opts.split);
+	const report = JSON.parse(stdout);
+	if (typeof report.upstreamVersion !== "string" || report.upstreamVersion.length === 0 || !Array.isArray(report.rows)) throw new Error(`${config.name}: upstream task loader returned no pinned version/rows`);
+	const pin = {
+		upstreamCommit,
+		upstreamVersion: report.upstreamVersion
+	};
+	return selectRows(report.rows, opts, config, opts.split, pin);
 }
 async function scoreOfficialTrajectory(root, meta, artifactPath) {
 	const stdout = await runVenvPython(`
@@ -107,6 +173,7 @@ root = Path(sys.argv[1])
 task_id = sys.argv[2]
 artifact = Path(sys.argv[3])
 sys.path.insert(0, str(root / "src"))
+${upstreamVersionSnippet}
 from tau2.data_model.simulation import Results
 from tau2.scripts.evaluate_trajectories import compute_simulation_rewards
 results = Results.load(artifact)
@@ -117,7 +184,7 @@ for sim in updated.simulations:
         scores.append(float(sim.reward_info.reward))
 if not scores:
     raise SystemExit(f"no scored simulations for task_id={task_id} in {artifact}")
-print(json.dumps({"count": len(scores), "score": sum(scores) / len(scores), "scores": scores}))
+print(json.dumps({"count": len(scores), "score": sum(scores) / len(scores), "scores": scores, "upstreamVersion": upstream_version}))
 `, [
 		root,
 		meta.taskId,
@@ -150,7 +217,15 @@ function createTauBenchAdapter(config) {
 			const meta = readMeta(task, config.name);
 			const artifactPath = resolve(artifact.trim());
 			await assertPath(artifactPath, "tau results/trajectory artifact", config.name);
+			const upstreamCommit = await resolveUpstreamCommit(dir, config.name);
 			const report = await scoreOfficialTrajectory(dir, meta, artifactPath);
+			const upstreamVersion = report.upstreamVersion;
+			if (typeof upstreamVersion !== "string" || upstreamVersion.length === 0) throw new Error(`${config.name}: reward recomputation returned no pinned tau2 version`);
+			const live = {
+				upstreamCommit,
+				upstreamVersion
+			};
+			assertTauUpstreamPinUnchanged(config.name, meta.taskId, meta, live);
 			const score = typeof report.score === "number" ? report.score : 0;
 			return {
 				resolved: score === 1,
@@ -158,13 +233,15 @@ function createTauBenchAdapter(config) {
 				detail: JSON.stringify({
 					taskId: meta.taskId,
 					domain: meta.domain,
-					count: report.count
+					count: report.count,
+					upstreamCommit,
+					upstreamVersion
 				})
 			};
 		}
 	};
 }
 //#endregion
-export { createTauBenchAdapter, tauResultsOutput };
+export { assertTauUpstreamPinUnchanged, createTauBenchAdapter, tauResultsOutput };
 
 //# sourceMappingURL=tau-bench-shared.js.map