@tangle-network/agent-bench 0.3.6 → 0.3.8

package/CHANGELOG.md

@@ -1,5 +1,16 @@
 # Changelog
 
+## 0.3.8
+
+- Resolve package-owned fixtures and scripts from both source and compiled installs, and prove ToolLLM fixture loading from the packed package.
+
+## 0.3.7
+
+- Declare the public executable search path for Pier candidate entrypoints so the exact process contract can replay them against `@tangle-network/agent-runtime@0.102.0`.
+- Align the portable agent contract with `@tangle-network/agent-interface@0.32.0`.
+- Add the trace-driven SWE supervisor improvement loop with measured candidate worktrees and official Docker judging.
+- Settle cancelled worker trees, isolate each cell's state, serialize judges with a kernel lock, and reject unfinished processes before patch extraction or scoring.
+
 ## 0.3.6
 
 - Ship compiled ESM and declarations so Node can import the installed package without a TypeScript runtime.

package/HARNESS.md

@@ -71,6 +71,48 @@ this codebase. `agent-lab/runs/RUNS.md` is the index mapping artifacts → verdi
 findings gist. **Set `OUT=runs/<date>/<name>.json` (in agent-lab) on every run — never `/tmp`**
 (a reboot erases ramdisk; ~20 runs nearly died there once).
 
+## Run observability — never hand-grep a journal again
+
+Every supervisor cell writes `run-report.json` + `run-report.md` next to its artifacts, and
+appends a headline block to the run log, automatically: `outer-loop.mts` and both
+`run-experiment.mts` paths call `writeRunReportSafe` after each cell and `reportRound` at
+the gate. Source: `swe-arena/run-report.mts` (pure `computeRunReport` + an I/O collector).
+
+What it answers, all deterministic, all from artifacts already on disk (`journal.jsonl`,
+`workers/*.ndjson`, `workers/*.inbox.ndjson`, `state.json`, `result.json`, `judge.json`,
+the ledger, `driver.log`, the delivered patch, the opencode session store):
+
+- **orchestration** — workers spawned/settled/cancelled, **steer count with a per-worker
+  breakdown**, spawn waves + sizes, max concurrency, respawns, repeated labels, delegation
+  depth, time-to-first-spawn, supervisor wall, idle wall (zero live workers), worker
+  utilization (Σ worker wall ÷ supervisor wall).
+- **decision quality** — settled by status/verdict, accepted vs rejected vs empty-pass,
+  evidence→respawn sequences vs blind respawns, review actions, evidence bytes returned.
+- **economics** — tokens + USD by role (brain from journal `metered`; workers from journal
+  `settled` spend plus the opencode session join), per-worker rows, cost per accepted patch,
+  worker wall distribution.
+- **outcome** — supervisor status/verdict/delivered, judge verdict + score (from `judge.json`
+  or the matching ledger row, with provenance), verify gate, patch file/line/test-touch stats.
+
+Every metric is UNAVAILABLE-aware: a missing artifact yields `{"unavailable": "<reason>"}`
+and renders as `unavailable — <reason>`, never `0`. A supervisor that steered nobody and a
+supervisor whose worker logs were lost are visually different, because they have driven
+opposite conclusions about the same architecture.
+
+Standalone / backfill (use `--report-dir` when the run directory must stay READ-ONLY):
+```
+tsx src/swe-arena/run-report.mts <cellDir> [--patch p.diff] [--ledger ledger.jsonl] [--log run.log] [--report-dir out/]
+tsx src/swe-arena/run-report.mts --round <outDir> [--report-dir out/]
+```
+
+Relationship to the published `traces` CLI (`npx @tangle-network/traces analyze`): traces
+covers the HARNESS-SESSION layer — per-session model calls, token/cost tables, latency
+distributions, stuck-loop detection, tool-error rates, skill/subagent adoption — reading the
+opencode/claude-code/codex session stores. It has no supervision-tree reader: no journal
+parsing, no steer/wave/concurrency/utilization/delegation-depth metric, no accepted-vs-rejected
+worker accounting. The two are complementary; run-report prints the traces command for the
+same cell rather than re-deriving session-level facts.
+
 ## Data flow (the whole experiment in one line)
 `rollout (worker → answer) → adapter.judge (valid?) → CORPUS RunRecord (k attempts, output+valid each) → corpus-replay --selector (pick WITHOUT the judge) → corpus-report CI → gate verdict`
 The expensive part (rollouts) produces a **reusable corpus**; selection + stats are free
@@ -135,6 +177,7 @@ the gate + measurement tools:
   commit0-env-run.mts  the HARD domain through `runBenchmark` (the optimization suite)
   terminal-compare.ts  Terminal-Bench compare (own main)
   pnpm verify:pier  zero-model failure/pass Pier controls through a separate verifier
+  swe-arena/run-report.mts  deterministic per-cell run report (steers, waves, idle, cost by role) + round rollup
 unit tests (the only fully-green, cred-free runnable surface besides offline replay):
   node --test --import tsx src/{selector,refine-loop}.test.mts
   tsx src/gate.test.mts   # offline plumbing test (no creds)

package/dist/adapters.js

@@ -1,40 +1,40 @@
 import {
   ADAPTERS,
   resolveAdapter
-} from "./chunk-5SBJCB6W.js";
-import "./chunk-2PVVP7GN.js";
+} from "./chunk-V7AEBY6U.js";
+import "./chunk-XKEFIFIC.js";
 import "./chunk-VQRS7VUC.js";
-import "./chunk-Z7ML6L77.js";
-import "./chunk-R67DFVLO.js";
+import "./chunk-HWST3SED.js";
+import "./chunk-IFVINJ4B.js";
 import "./chunk-SEVJPLZC.js";
-import "./chunk-CKUVRZ2T.js";
-import "./chunk-Y6O2OCUO.js";
-import "./chunk-HBSWHQNJ.js";
-import "./chunk-UAIOHCUK.js";
-import "./chunk-R36V2VP7.js";
-import "./chunk-JRWWGMK7.js";
+import "./chunk-3U5TXJZS.js";
+import "./chunk-SHM6MRRF.js";
+import "./chunk-RCYQEFNX.js";
+import "./chunk-SFLA7OH3.js";
+import "./chunk-IZ5M6OAC.js";
+import "./chunk-XYA4XSNU.js";
 import "./chunk-INNOYXCP.js";
 import "./chunk-PA2ZKHJC.js";
-import "./chunk-PPYSEKFM.js";
+import "./chunk-5H5XV76F.js";
 import "./chunk-TBKU5XQI.js";
-import "./chunk-TSWPNOYM.js";
-import "./chunk-KDIKRJGB.js";
-import "./chunk-ZEWMTR5M.js";
+import "./chunk-NQG5XDSB.js";
+import "./chunk-SHYIRB7I.js";
+import "./chunk-MQMRLGOG.js";
 import "./chunk-PUIRNYI7.js";
 import "./chunk-C7T7WEK2.js";
 import "./chunk-JTHWEDEW.js";
-import "./chunk-2XU6OGEN.js";
-import "./chunk-HHXFIHXC.js";
+import "./chunk-Z4TZ76N7.js";
+import "./chunk-SVR2LKYI.js";
 import "./chunk-X3BTXCJ4.js";
-import "./chunk-J3KDJNX2.js";
-import "./chunk-X5YKXC6V.js";
-import "./chunk-YCGY7UIZ.js";
+import "./chunk-RH5F53JT.js";
+import "./chunk-YSMEKBTD.js";
+import "./chunk-7GRVHU22.js";
 import "./chunk-UPAMRDX4.js";
-import "./chunk-ODT47UAY.js";
-import "./chunk-7WSD27QQ.js";
-import "./chunk-SYDW647C.js";
-import "./chunk-IFAV6KEM.js";
-import "./chunk-LRRD7NAG.js";
+import "./chunk-K3BQGZCT.js";
+import "./chunk-PB64GYIG.js";
+import "./chunk-IA2FBTWC.js";
+import "./chunk-KP5KD6EN.js";
+import "./chunk-WSKWVEQB.js";
 import "./chunk-53UPUNBZ.js";
 export {
   ADAPTERS,

package/dist/benchmarks/_harness.d.ts

@@ -19,7 +19,7 @@ import '@tangle-network/agent-runtime/loops';
  * nonzero throws with the captured stderr — never a fabricated score.
  */
 
-/** Repo root for the bench package (…/bench), so `.venv` and `fixtures` resolve. */
+/** Package root for agent-bench, so `.venv`, scripts, and fixtures resolve. */
 declare const benchRoot: string;
 /** Resolve the shared interpreter without requiring an installed package to contain a venv. */
 declare function resolveBenchPython(env?: Readonly<{

package/dist/benchmarks/_harness.js

@@ -13,7 +13,7 @@ import {
   venvBin,
   venvPython,
   venvPythonAt
-} from "../chunk-LRRD7NAG.js";
+} from "../chunk-WSKWVEQB.js";
 export {
   StagedJudgeError,
   benchRoot,

package/dist/benchmarks/aec-bench.js

@@ -1,7 +1,7 @@
 import {
   createAecBenchAdapter
-} from "../chunk-ODT47UAY.js";
-import "../chunk-LRRD7NAG.js";
+} from "../chunk-K3BQGZCT.js";
+import "../chunk-WSKWVEQB.js";
 export {
   createAecBenchAdapter
 };

package/dist/benchmarks/agentbench.js

@@ -1,8 +1,8 @@
 import {
   agentbenchAnswerOutput,
   createAgentBenchAdapter
-} from "../chunk-7WSD27QQ.js";
-import "../chunk-LRRD7NAG.js";
+} from "../chunk-PB64GYIG.js";
+import "../chunk-WSKWVEQB.js";
 export {
   agentbenchAnswerOutput,
   createAgentBenchAdapter

package/dist/benchmarks/appworld.js

@@ -3,8 +3,8 @@ import {
   appworldToolLoopClient,
   createAppWorldAdapter,
   createAppWorldReactAdapter
-} from "../chunk-SYDW647C.js";
-import "../chunk-LRRD7NAG.js";
+} from "../chunk-IA2FBTWC.js";
+import "../chunk-WSKWVEQB.js";
 export {
   appworldSolutionOutput,
   appworldToolLoopClient,

package/dist/benchmarks/bfcl.js

@@ -1,8 +1,8 @@
 import {
   bfclOutput,
   createBfclAdapter
-} from "../chunk-IFAV6KEM.js";
-import "../chunk-LRRD7NAG.js";
+} from "../chunk-KP5KD6EN.js";
+import "../chunk-WSKWVEQB.js";
 export {
   bfclOutput,
   createBfclAdapter

package/dist/benchmarks/commit0.js

@@ -1,8 +1,8 @@
 import {
   commit0DiffOutput,
   createCommit0Adapter
-} from "../chunk-2XU6OGEN.js";
-import "../chunk-LRRD7NAG.js";
+} from "../chunk-Z4TZ76N7.js";
+import "../chunk-WSKWVEQB.js";
 export {
   commit0DiffOutput,
   createCommit0Adapter

package/dist/benchmarks/crag.js

@@ -1,8 +1,8 @@
 import {
   createCragAdapter
-} from "../chunk-HHXFIHXC.js";
+} from "../chunk-SVR2LKYI.js";
 import "../chunk-X3BTXCJ4.js";
-import "../chunk-LRRD7NAG.js";
+import "../chunk-WSKWVEQB.js";
 export {
   createCragAdapter
 };

package/dist/benchmarks/dabstep.js

@@ -1,8 +1,8 @@
 import {
   createDabstepAdapter,
   dabstepAnswerOutput
-} from "../chunk-J3KDJNX2.js";
-import "../chunk-LRRD7NAG.js";
+} from "../chunk-RH5F53JT.js";
+import "../chunk-WSKWVEQB.js";
 export {
   createDabstepAdapter,
   dabstepAnswerOutput

package/dist/benchmarks/enterpriseops-gym.js

@@ -1,8 +1,8 @@
 import {
   createEnterpriseOpsGymAdapter,
   enterpriseOpsTranscriptOutput
-} from "../chunk-X5YKXC6V.js";
-import "../chunk-LRRD7NAG.js";
+} from "../chunk-YSMEKBTD.js";
+import "../chunk-WSKWVEQB.js";
 export {
   createEnterpriseOpsGymAdapter,
   enterpriseOpsTranscriptOutput

package/dist/benchmarks/finresearchbench.js

@@ -1,7 +1,7 @@
 import {
   createFinResearchBenchAdapter
-} from "../chunk-YCGY7UIZ.js";
-import "../chunk-LRRD7NAG.js";
+} from "../chunk-7GRVHU22.js";
+import "../chunk-WSKWVEQB.js";
 export {
   createFinResearchBenchAdapter
 };

package/dist/benchmarks/humaneval.d.ts

@@ -45,9 +45,18 @@ interface CheckResult {
      *  execution-grounded feedback a self-repair loop steers on; ignored by selection. */
     detail?: string;
 }
+interface PythonProgramResult {
+    exitCode: number;
+    stdout: string;
+    stderr: string;
+}
+/** Run arbitrary Python in the same isolated container used by the HumanEval
+ * checker. This is also the execution-tool primitive for experiments that let
+ * a model test its own snippets; model-written code must never run on the host. */
+declare function runPythonProgram(program: string, timeoutMs?: number): Promise<PythonProgramResult>;
 declare function runChecker(task: HumanEvalTask, candidate: string): Promise<CheckResult>;
 /** The HumanEval `BenchmarkAdapter`. OFFSET (env) selects the correctable middle
  *  band; loadTasks honors `limit`/`ids`. The judge is the Docker deployable checker. */
 declare function createHumanEvalAdapter(): BenchmarkAdapter;
 
-export { type CheckResult, type HumanEvalTask, basePrompt, createHumanEvalAdapter, extractCode, loadHumanEval, runChecker };
+export { type CheckResult, type HumanEvalTask, type PythonProgramResult, basePrompt, createHumanEvalAdapter, extractCode, loadHumanEval, runChecker, runPythonProgram };

package/dist/benchmarks/humaneval.js

@@ -3,13 +3,15 @@ import {
   createHumanEvalAdapter,
   extractCode,
   loadHumanEval,
-  runChecker
-} from "../chunk-PPYSEKFM.js";
+  runChecker,
+  runPythonProgram
+} from "../chunk-5H5XV76F.js";
 export {
   basePrompt,
   createHumanEvalAdapter,
   extractCode,
   loadHumanEval,
-  runChecker
+  runChecker,
+  runPythonProgram
 };
 //# sourceMappingURL=humaneval.js.map

package/dist/benchmarks/nomiracl.js

@@ -1,8 +1,8 @@
 import {
   createNoMiraclAdapter
-} from "../chunk-TSWPNOYM.js";
+} from "../chunk-NQG5XDSB.js";
 import "../chunk-X3BTXCJ4.js";
-import "../chunk-LRRD7NAG.js";
+import "../chunk-WSKWVEQB.js";
 export {
   createNoMiraclAdapter
 };

package/dist/benchmarks/open-rag-bench.js

@@ -1,8 +1,8 @@
 import {
   createOpenRagBenchAdapter
-} from "../chunk-KDIKRJGB.js";
+} from "../chunk-SHYIRB7I.js";
 import "../chunk-X3BTXCJ4.js";
-import "../chunk-LRRD7NAG.js";
+import "../chunk-WSKWVEQB.js";
 export {
   createOpenRagBenchAdapter
 };

package/dist/benchmarks/programbench.js

@@ -1,8 +1,8 @@
 import {
   createProgrambenchAdapter,
   programbenchSubmissionOutput
-} from "../chunk-ZEWMTR5M.js";
-import "../chunk-LRRD7NAG.js";
+} from "../chunk-MQMRLGOG.js";
+import "../chunk-WSKWVEQB.js";
 export {
   createProgrambenchAdapter,
   programbenchSubmissionOutput

package/dist/benchmarks/ragbench.js

@@ -1,8 +1,8 @@
 import {
   createRagBenchAdapter
-} from "../chunk-R67DFVLO.js";
+} from "../chunk-IFVINJ4B.js";
 import "../chunk-X3BTXCJ4.js";
-import "../chunk-LRRD7NAG.js";
+import "../chunk-WSKWVEQB.js";
 export {
   createRagBenchAdapter
 };

package/dist/benchmarks/swe-bench.js

@@ -3,8 +3,8 @@ import {
   scoreSweReport,
   sweEvaluationArgv,
   swePatchOutput
-} from "../chunk-CKUVRZ2T.js";
-import "../chunk-LRRD7NAG.js";
+} from "../chunk-3U5TXJZS.js";
+import "../chunk-WSKWVEQB.js";
 export {
   createSweBenchAdapter,
   scoreSweReport,

package/dist/benchmarks/t2-ragbench.js

@@ -1,8 +1,8 @@
 import {
   createT2RagBenchAdapter
-} from "../chunk-Y6O2OCUO.js";
+} from "../chunk-SHM6MRRF.js";
 import "../chunk-X3BTXCJ4.js";
-import "../chunk-LRRD7NAG.js";
+import "../chunk-WSKWVEQB.js";
 export {
   createT2RagBenchAdapter
 };

package/dist/benchmarks/tau-bench-shared.js

@@ -1,8 +1,8 @@
 import {
   createTauBenchAdapter,
   tauResultsOutput
-} from "../chunk-R36V2VP7.js";
-import "../chunk-LRRD7NAG.js";
+} from "../chunk-IZ5M6OAC.js";
+import "../chunk-WSKWVEQB.js";
 export {
   createTauBenchAdapter,
   tauResultsOutput

package/dist/benchmarks/tau2-bench.js

@@ -1,9 +1,9 @@
 import {
   createTau2BenchAdapter,
   tau2ResultsOutput
-} from "../chunk-HBSWHQNJ.js";
-import "../chunk-R36V2VP7.js";
-import "../chunk-LRRD7NAG.js";
+} from "../chunk-RCYQEFNX.js";
+import "../chunk-IZ5M6OAC.js";
+import "../chunk-WSKWVEQB.js";
 export {
   createTau2BenchAdapter,
   tau2ResultsOutput

package/dist/benchmarks/tau3-banking.js

@@ -1,8 +1,8 @@
 import {
   createTau3BankingAdapter
-} from "../chunk-UAIOHCUK.js";
-import "../chunk-R36V2VP7.js";
-import "../chunk-LRRD7NAG.js";
+} from "../chunk-SFLA7OH3.js";
+import "../chunk-IZ5M6OAC.js";
+import "../chunk-WSKWVEQB.js";
 export {
   createTau3BankingAdapter
 };

package/dist/benchmarks/terminal-bench.js

@@ -1,7 +1,7 @@
 import {
   createTerminalBenchAdapter
-} from "../chunk-JRWWGMK7.js";
-import "../chunk-LRRD7NAG.js";
+} from "../chunk-XYA4XSNU.js";
+import "../chunk-WSKWVEQB.js";
 export {
   createTerminalBenchAdapter
 };

package/dist/benchmarks/toollm.js

@@ -1,8 +1,8 @@
 import {
   createToolLlmAdapter,
   toollmOutput
-} from "../chunk-2PVVP7GN.js";
-import "../chunk-LRRD7NAG.js";
+} from "../chunk-XKEFIFIC.js";
+import "../chunk-WSKWVEQB.js";
 export {
   createToolLlmAdapter,
   toollmOutput

package/dist/benchmarks/webarena-verified.js

@@ -1,8 +1,8 @@
 import {
   createWebArenaVerifiedAdapter,
   webarenaOutputDirOutput
-} from "../chunk-Z7ML6L77.js";
-import "../chunk-LRRD7NAG.js";
+} from "../chunk-HWST3SED.js";
+import "../chunk-WSKWVEQB.js";
 export {
   createWebArenaVerifiedAdapter,
   webarenaOutputDirOutput

package/dist/{chunk-CKUVRZ2T.js → chunk-3U5TXJZS.js}

@@ -5,7 +5,7 @@ import {
   runVenvPython,
   safeRunId,
   stageFile
-} from "./chunk-LRRD7NAG.js";
+} from "./chunk-WSKWVEQB.js";
 
 // src/benchmarks/swe-bench.ts
 import { join } from "path";
@@ -248,4 +248,4 @@ export {
   sweEvaluationArgv,
   createSweBenchAdapter
 };
-//# sourceMappingURL=chunk-CKUVRZ2T.js.map
+//# sourceMappingURL=chunk-3U5TXJZS.js.map

package/dist/{chunk-PPYSEKFM.js → chunk-5H5XV76F.js}

@@ -1,6 +1,6 @@
 // src/benchmarks/humaneval.ts
 import { execFile } from "child_process";
-import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "fs";
+import { chmodSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "fs";
 import { tmpdir } from "os";
 import { join } from "path";
 import { gunzipSync } from "zlib";
@@ -59,10 +59,12 @@ check(${task.entryPoint})
 `;
 }
 var dockerRunSeq = 0;
-function runChecker(task, candidate) {
+function runPythonProgram(program, timeoutMs = dockerTimeoutMs) {
   const dir = mkdtempSync(join(tmpdir(), "hev-"));
-  writeFileSync(join(dir, "p.py"), buildProgram(task, candidate));
+  chmodSync(dir, 493);
+  writeFileSync(join(dir, "p.py"), program, { mode: 420 });
   const name = `hev-${process.pid}-${dockerRunSeq++}`;
+  const startMarker = `__AGENT_RUNTIME_CANDIDATE_STARTED_${name}__`;
   return new Promise((resolvePromise, reject) => {
     let settled = false;
     const cleanup = () => {
@@ -84,49 +86,98 @@ function runChecker(task, candidate) {
       cleanup();
       reject(e);
     };
-    const backstop = setTimeout(() => finish({ pass: 0 }), dockerTimeoutMs + 3e3);
+    const outerTimeoutMs = timeoutMs + 3e3;
+    const backstop = setTimeout(
+      () => fail(new Error(`docker checker did not return within ${outerTimeoutMs + 3e3}ms`)),
+      outerTimeoutMs + 3e3
+    );
+    const inContainerSeconds = Math.max(1, Math.ceil(timeoutMs / 1e3));
     execFile(
       "docker",
       [
         "run",
         "--rm",
+        "--pull=never",
         "--name",
         name,
         "--network=none",
         "--cpus=1",
         "--memory=512m",
+        "--pids-limit=64",
+        "--cap-drop=ALL",
+        "--security-opt=no-new-privileges",
+        "--read-only",
+        "--tmpfs",
+        "/tmp:rw,nosuid,nodev,noexec,size=64m,mode=1777",
+        "--user",
+        "65534:65534",
         "-v",
         `${dir}:/w:ro`,
         "-w",
         "/w",
         dockerImage,
-        "python",
-        "/w/p.py"
+        "sh",
+        "-c",
+        'command -v timeout >/dev/null 2>&1 && command -v python >/dev/null 2>&1 && [ -r /w/p.py ] || exit 126; printf "%s\\n" "$1"; exec timeout -s KILL "$2" python /w/p.py',
+        "agent-runtime-checker",
+        startMarker,
+        `${inContainerSeconds}s`
       ],
-      { timeout: dockerTimeoutMs, killSignal: "SIGKILL", maxBuffer: 4 * 1024 * 1024 },
-      (err, _stdout, stderr) => {
+      { timeout: outerTimeoutMs, killSignal: "SIGKILL", maxBuffer: 4 * 1024 * 1024 },
+      (err, stdout, stderr) => {
+        const markerLine = `${startMarker}
+`;
+        const candidateStarted = stdout?.startsWith(markerLine) === true;
+        const candidateStdout = candidateStarted ? stdout.slice(markerLine.length) : stdout ?? "";
         if (err) {
           const e = err;
           if (e.code === "ENOENT") {
             fail(new Error("docker binary not found on PATH \u2014 cannot run the deployable checker"));
             return;
           }
-          if (/cannot connect to the docker daemon|is the docker daemon running|permission denied while trying to connect/i.test(stderr)) {
-            fail(new Error(`docker daemon unreachable: ${stderr.slice(0, 200)}`));
+          if (e.killed) {
+            fail(new Error(`docker checker exceeded its ${outerTimeoutMs}ms outer timeout`));
             return;
           }
-          if (/(unable to find image|pull access denied|manifest unknown|error response from daemon).*(pull|repository|registry)/i.test(stderr)) {
-            fail(new Error(`docker image ${dockerImage} unavailable: ${stderr.slice(0, 200)}`));
+          if (!candidateStarted) {
+            if (/cannot connect to the docker daemon|is the docker daemon running|permission denied while trying to connect/i.test(
+              stderr
+            )) {
+              fail(new Error(`docker daemon unreachable: ${stderr.slice(0, 200)}`));
+              return;
+            }
+            if (/unable to find image|no such image|pull access denied|manifest unknown/i.test(stderr)) {
+              fail(new Error(`docker image ${dockerImage} unavailable: ${stderr.slice(0, 200)}`));
+              return;
+            }
+            fail(
+              new Error(
+                `docker checker did not start the candidate: ${stderr.slice(0, 200) || e.message}`
+              )
+            );
             return;
           }
-          finish({ pass: 0, detail: (stderr || "").slice(-600) || "timed out (no output)" });
+          const exitCode = typeof e.code === "number" ? e.code : 1;
+          finish({
+            exitCode,
+            stdout: candidateStdout,
+            stderr: stderr ?? ""
+          });
           return;
         }
-        finish({ pass: 1 });
+        if (!candidateStarted) {
+          fail(new Error("docker checker exited without starting the candidate"));
+          return;
+        }
+        finish({ exitCode: 0, stdout: candidateStdout, stderr: stderr ?? "" });
       }
     );
   });
 }
+async function runChecker(task, candidate) {
+  const result = await runPythonProgram(buildProgram(task, candidate));
+  return result.exitCode === 0 ? { pass: 1 } : { pass: 0, detail: result.stderr.slice(-600) || "timed out (no output)" };
+}
 function toBenchTask(t) {
   const metadata = {
     promptHeader: t.prompt,
@@ -153,6 +204,12 @@ function createHumanEvalAdapter() {
           else resolve();
         });
       });
+      await new Promise((resolve, reject) => {
+        execFile("docker", ["image", "inspect", dockerImage], (err) => {
+          if (err) reject(new Error(`HumanEval judge needs the cached Docker image ${dockerImage}`));
+          else resolve();
+        });
+      });
     },
     async loadTasks(opts) {
       const offset = Number(process.env.OFFSET ?? 0);
@@ -176,7 +233,8 @@ export {
   loadHumanEval,
   basePrompt,
   extractCode,
+  runPythonProgram,
   runChecker,
   createHumanEvalAdapter
 };
-//# sourceMappingURL=chunk-PPYSEKFM.js.map
+//# sourceMappingURL=chunk-5H5XV76F.js.map