@hona/openeval 0.4.0 → 0.5.0

package/JUDGING.md

@@ -2,7 +2,7 @@
 
 Use the [canonical terminology](https://openev.al/docs/terminology/): a criterion
 is a named graded requirement, a score is awarded credit, and a metric is a
-measurement such as token count or cost. OpenEval 0.3.0 supports code judges,
+measurement such as token count or cost. OpenEval supports code judges,
 LLM judges, and additive use of both against one recorded EvalRun.
 
 ## File conventions
@@ -36,6 +36,9 @@ Only the optional scores object has grading semantics. Each key is a criterion
 ID, using lowercase letters, digits, and underscores, starting with a letter.
 Values are booleans, finite numbers from 0 to 1, or null. The host converts true
 to 1 and false to 0. It rejects invalid values rather than clamping them.
+Optional `{ value, reason, evidence, measurements }` objects make code verdicts
+readable without changing their scoring semantics. Declared code criterion IDs
+must match the returned scores. See [artifact verification](VERIFICATION.md).
 response.text is always a string; missing text becomes an empty string. The
 original execution outcome is available separately on context.run.
 
@@ -69,6 +72,8 @@ a JudgeRun error. A hybrid judgment is selected only after both sources succeed.
 | recording.export(sessionID?) | Native OpenCode session export |
 | workspace.files/read/text/diff | Verified initial and final file snapshots |
 | workspace.materialize(revision?) | A disposable workspace copy for author-owned verification |
+| verification.run(request) | Bounded commands over a restored artifact in an isolated OCI container |
+| verification.read/text(result, path) | Hash-checked retained output from that verification |
 | native.database() | Read-only SQLite access to a verified database copy |
 | native.sdk() | The pinned OpenCode SDK/API over a separate disposable archive copy |
 | native.schema() | The pinned OpenCode schema module |

package/VERIFICATION.md

@@ -0,0 +1,100 @@
+# Artifact verification
+
+Code judges are still plain functions. When the score depends on delivered code,
+run it in a disposable OCI container rather than on the runner host.
+
+```ts
+// benchmark.ts
+import { VERIFICATION_IMAGE, type Benchmark } from "@hona/openeval";
+export default {
+  models: ["example/model"],
+  judge: { verification: { image: VERIFICATION_IMAGE, cpus: 2, memoryMiB: 4096 } },
+} satisfies Benchmark;
+```
+
+Build the candidate and standard verification images with `openeval image`.
+`openeval image --verification` builds only the verification image. Custom images
+must be built separately. Images resolve to immutable IDs before planning or
+collection; a changed verification image invalidates judgment reuse, not the
+candidate's delivered artifacts.
+
+```ts
+// judge.ts — trusted script content is an ordinary frozen local import/string.
+import type { JudgeContext } from "@hona/openeval";
+export const criteria = { correct: { name: "Correct output", categories: ["coding"] } };
+export default async (ctx: JudgeContext) => {
+  const result = await ctx.verification.run({
+    revision: "final", cwd: ".", timeoutMs: 60_000,
+    commands: [["bun", "/verification/check.mjs"]],
+    files: { "check.mjs": "/* author's outcome checks */" },
+    artifacts: ["checks.json", "screenshot.png"],
+  });
+  return { scores: { correct: {
+    value: result.state === "completed" && result.exitCode === 0,
+    reason: "Explain what the checks established or why they failed.",
+    evidence: [{ kind: "verification", id: result.id }],
+    measurements: { elapsedMs: result.elapsedMs },
+  } } };
+};
+```
+
+The primitive restores an initial or final snapshot, uploads trusted inputs to
+`/verification`, runs argv arrays in order, and stops at a nonzero exit. It does
+not choose scoring thresholds or interpret success. Judge setup/transfer/image
+errors remain JudgeRun errors. A test exit or verification timeout is an
+observation the authored criterion must interpret. Interrupted candidate work
+still needs the rubric's evidence policy; a failed verification of an unfinished
+prefix does not necessarily establish final-task failure.
+
+## Isolation and bounds
+
+- No host bind mounts, credentials, published ports, or network. Browser/server
+  verification can use loopback **inside** the container.
+- Read-only image filesystem, non-root command user, no capabilities, no privilege
+  escalation, PID/CPU/memory limits, bounded workspace/temp storage, and a deadline.
+- Trusted check inputs are root-owned and are not writable by delivered code.
+- Commands, exit statuses, image/resource identity, logs, and requested regular
+  output files are retained with the JudgeRun. Viewer previews never execute HTML
+  or SVG from the delivered artifact.
+- Workspace transfer is limited to 128 MiB; each output archive is limited to
+  32 MiB; logs are limited to 1 MiB per stream and are marked when truncated.
+- Containers self-expire. The parent also removes containers bearing only its
+  unique execution label after worker interruption. No shared/broad cleanup.
+
+The standard image includes Bun 1.4.2, Python, Git, Node, and Playwright 1.63.0
+with Chromium. Import Playwright from
+`/opt/verify/node_modules/playwright/index.mjs`. Dependencies needed by a task
+must be available in the image or supplied as frozen inputs; network installs
+are intentionally unavailable. Materialization alone is not a sandbox.
+`verification.text(result, "stdout" | "stderr")` reads the retained bounded logs;
+those names are reserved and cannot also name output artifacts.
+
+The primitive verifies a reconstructed artifact. It does **not** prove what was
+alive in the original candidate container, whether the candidate ran a test,
+or whether original game actions were legal. Those claims need original recording
+evidence. Do not put evaluator scripts in candidate workspaces.
+
+## Readable judgments and controls
+
+`openeval prepare --output <new-directory>` assembles real candidate inputs and
+runs declared preparation in the candidate image, then archives the prepared
+workspace. `--only-eval` scopes it. This makes zero model calls, creates no
+EvalRun/JudgeRun, and does not change selections. Use it to prove fixture setup
+before collection; it does not prove agent success or human task duration.
+
+Existing boolean, numeric, and null scores remain sufficient. Optional structured
+scores add `reason`, `evidence`, and JSON `measurements`. These details appear in
+the normal viewer beside verification receipts; arbitrary returned JSON remains
+inspectable. Declared code criteria must be returned exactly. Missing evidence
+references are judging errors, not candidate zeros.
+
+`recordEvidence({ workspace: { initial, final }, ... })` can retain constructed
+artifact controls without executing them. `judgeEvidence` then uses the same
+public verification primitive. Constructed controls prove tested boundaries, not
+human-duration calibration or live candidate feasibility.
+
+Unused `criteria` labels/categories are removed from executable bundling. Debug
+source maps remain retained but do not affect code identity. If grading reads a
+metadata value, it is executable behavior and remains fingerprinted. Changes to
+criterion IDs, actual grading code, imported inputs, dependencies, or verification
+environment are not reporting-only edits.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hona/openeval",
-  "version": "0.4.0",
+  "version": "0.5.0",
   "description": "Code and LLM evaluations with isolated agents, recorded OpenCode data, and a results viewer",
   "license": "MIT",
   "repository": {
@@ -14,7 +14,7 @@
   "bin": {
     "openeval": "src/cli.ts"
   },
-  "files": ["src", "viewer", "LICENSE", "README.md", "JUDGING.md", "!src/**/*.test.ts"],
+  "files": ["src", "viewer", "LICENSE", "README.md", "JUDGING.md", "VERIFICATION.md", "!src/**/*.test.ts"],
   "publishConfig": {
     "access": "public"
   },

package/src/app/grade-recording.ts

@@ -11,6 +11,8 @@ import { executeCodeJudge } from "../infra/judging/code";
 import { codeJudgment, combineJudgments } from "../infra/judging/code-result";
 import { executeJudge } from "../infra/judging";
 import { errorMessage } from "../infra/files";
+import { CandidateEvidence } from "../infra/evidence";
+import { validateCitations } from "../infra/judging/contract";
 
 export type GradedRecording = {
   state: "completed" | "failed" | "timed_out";
@@ -37,10 +39,24 @@ export async function gradeRecording(
         recording,
         resolve(directory, "code"),
         input.timeoutMs,
+        input.verification,
       );
       if (code.state !== "completed")
         return { state: code.state, code, error: code.error };
       const judged = codeJudgment(code.output!);
+      if (input.codeCriteria?.length && (
+        Object.keys(judged.scores).length !== input.codeCriteria.length ||
+        input.codeCriteria.some(item => !Object.hasOwn(judged.scores, item.id))
+      )) throw new Error("judge.ts must return exactly its declared criterion IDs");
+      for (const score of Object.values(judged.scores)) for (const citation of score.evidence) {
+        if (citation.kind !== "verification") continue;
+        const result = code.verifications?.find(item => item.id === citation.id);
+        if (!result || (citation.path && !result.artifacts.some(item => item.path === citation.path)))
+          throw new Error("Code judgment cites missing verification evidence");
+      }
+      await validateCitations({ ...judged, scores: Object.fromEntries(Object.entries(judged.scores).map(([id, score]) =>
+        [id, { ...score, evidence: score.evidence.filter(citation => citation.kind !== "verification") }])) },
+        await CandidateEvidence.open(recording.evidence.directory, recording.evidence.hash));
       for (const id of Object.keys(judged.scores))
         if (input.criteria.some((criterion) => criterion.id === id))
           throw new Error(

package/src/app/input-fingerprints.ts

@@ -61,6 +61,7 @@ export function candidateFingerprint(
       timeoutMs: definition.candidate.timeoutMs,
       websearch: definition.candidate.websearch,
       provider: scope && { ...scope.settings, model: scope.override },
+      earlyStop: item.settings.earlyStop ?? false,
     },
     container: {
       engine: definition.container.engine,
@@ -76,12 +77,13 @@ export const judgeFingerprint = (
   fingerprint({
     rubric: definition.evals.find((item) => item.id === evalId)!.judge,
     code: definition.evals.find((item) => item.id === evalId)!.code?.hash,
+    codeIds: definition.evals.find((item) => item.id === evalId)!.codeCriteria?.map(item => item.id).sort(),
     agent: definition.evals.find((item) => item.id === evalId)!.judge
       ? JUDGE_AGENT
       : undefined,
     judge: definition.evals.find((item) => item.id === evalId)!.judge
       ? definition.judge
-      : { timeoutMs: definition.judge.timeoutMs },
+      : { timeoutMs: definition.judge.timeoutMs, verification: definition.judge.verification },
     protocol: JUDGE_PROTOCOL,
   });
 export const savedJudgeFingerprint = (
@@ -94,11 +96,14 @@ export const savedJudgeFingerprint = (
     | "timeoutMs"
     | "websearch"
     | "code"
+    | "codeCriteria"
+    | "verification"
   >,
 ) =>
   fingerprint({
     rubric: input.rubric,
     code: input.code?.hash,
+    codeIds: input.codeCriteria?.map(item => item.id).sort(),
     agent: input.agent,
     protocol: input.protocol,
     judge: input.rubric
@@ -106,6 +111,7 @@ export const savedJudgeFingerprint = (
           model: input.model,
           timeoutMs: input.timeoutMs,
           websearch: input.websearch,
+          verification: input.verification,
         }
-      : { timeoutMs: input.timeoutMs },
+      : { timeoutMs: input.timeoutMs, verification: input.verification },
   });

package/src/app/judge-evidence.ts

@@ -9,6 +9,8 @@ import { savedJudgeFingerprint } from "./input-fingerprints";
 import { mkdir, mkdtemp, rm } from "node:fs/promises";
 import { compileCodeJudge } from "../infra/judging/code-source";
 import { gradeRecording } from "./grade-recording";
+import { readCodeCriteria } from "../infra/judging/code-criteria";
+import { verificationRuntime } from "../infra/verification/image";
 
 /** Inspect retained evidence without changing benchmark selections.
  * Useful for calibration and second opinions: https://arxiv.org/abs/2410.12784
@@ -28,6 +30,7 @@ export async function judgeEvidence(options: {
   if (options.rubric && !options.judge?.model)
     throw new Error("A Markdown rubric requires judge.model");
   const code = options.code ? await compileCodeJudge(options.code) : undefined;
+  const declared = options.code ? await readCodeCriteria(options.code) : undefined;
   const request: Omit<JudgeRunInput, "judgeHash"> = {
     evalRunId: "retained-evidence",
     evidence: {
@@ -37,6 +40,8 @@ export async function judgeEvidence(options: {
     rubric: options.rubric ?? "",
     kind: code ? (options.rubric ? "hybrid" : "code") : "llm",
     code,
+    codeCriteria: declared ? Object.entries(declared as Record<string, { name?: string }>).map(([id, value]) => ({ id, name: value.name ?? id })) : undefined,
+    verification: await verificationRuntime(options.judge?.verification),
     agent: options.rubric ? JUDGE_AGENT : undefined,
     model: options.rubric ? options.judge?.model : undefined,
     timeoutMs: options.judge?.timeoutMs ?? 600_000,
@@ -67,17 +72,18 @@ export async function recordEvidence(options: {
   prompt: string;
   response: string;
   tools?: ToolCall[];
+  workspace?: { initial?: string; final?: string };
 }): Promise<EvidenceRef> {
   const directory = resolve(options.directory);
   await mkdir(dirname(directory), { recursive: true });
   const workspace = await mkdtemp(resolve(dirname(directory), ".recording-"));
   try {
-    const capture = await EvidenceCapture.create(directory);
+    const capture = await EvidenceCapture.create(directory, options.workspace?.initial);
     const result = await capture.finish({
       prompt: options.prompt,
       response: { text: options.response },
       tools: options.tools ?? [],
-      workspace,
+      workspace: options.workspace?.final ?? workspace,
     });
     return { directory, hash: result.sha256 };
   } finally {

package/src/app/judge-run.ts

@@ -27,6 +27,8 @@ export async function judgeEvalRun(
     model: definition.judge ? context.definition.judge.model : undefined,
     kind: definition.code ? (definition.judge ? "hybrid" : "code") : "llm",
     code: definition.code,
+    codeCriteria: definition.codeCriteria,
+    verification: context.runtime.verification,
     judgeHash: slot.judgeHash,
     timeoutMs: context.definition.judge.timeoutMs,
     websearch: context.definition.judge.websearch,
@@ -76,6 +78,14 @@ export async function judgeEvalRun(
             ...graded.code,
             stdout: relative(context.directory, graded.code.stdout),
             stderr: relative(context.directory, graded.code.stderr),
+            verifications: graded.code.verifications?.map(result => ({
+              ...result,
+              stdout: relative(context.directory, resolve(directory, "code", result.stdout)),
+              stderr: relative(context.directory, resolve(directory, "code", result.stderr)),
+              artifacts: result.artifacts.map(artifact => ({ ...artifact,
+                file: relative(context.directory, resolve(directory, "code", artifact.file)),
+              })),
+            })),
           }
         : undefined,
       session: graded.session

package/src/app/load-benchmark.ts

@@ -292,11 +292,11 @@ export async function loadBenchmark(
       typeof definition.judge !== "object" ||
       Array.isArray(definition.judge) ||
       Object.keys(definition.judge).some(
-        (key) => !["model", "timeoutMs", "websearch"].includes(key),
+        (key) => !["model", "timeoutMs", "websearch", "verification"].includes(key),
       ))
   )
     throw new Error(
-      "judge must be an object with model, timeoutMs, or websearch settings",
+      "judge must be an object with model, timeoutMs, websearch, or verification settings",
     );
   if (new Set(models).size !== models.length)
     throw new Error("Benchmark contains duplicate models");
@@ -337,6 +337,13 @@ export async function loadBenchmark(
   const engine = definition.container?.engine ?? "docker";
   if (engine !== "docker" && engine !== "podman")
     throw new Error("Container engine must be docker or podman");
+  const verification = definition.judge?.verification;
+  if (verification && (
+    typeof verification !== "object" || Array.isArray(verification) ||
+    Object.keys(verification).some(key => !["image", "engine", "cpus", "memoryMiB"].includes(key)) ||
+    typeof verification.image !== "string" || !verification.image.trim() || /\s/.test(verification.image) ||
+    !["docker", "podman"].includes(verification.engine ?? engine)
+  )) throw new Error("judge.verification requires an image and valid container limits");
   for (const search of [
     definition.candidate?.websearch,
     definition.judge?.websearch,
@@ -367,6 +374,12 @@ export async function loadBenchmark(
         "Judge timeout",
       ),
       websearch: definition.judge?.websearch ?? "exa",
+      ...(verification ? { verification: {
+        image: verification.image,
+        engine: verification.engine ?? engine,
+        cpus: positive(verification.cpus, 2, "Verification CPU count"),
+        memoryMiB: positive(verification.memoryMiB, 4096, "Verification memory"),
+      } } : {}),
     },
     container: {
       engine,

package/src/app/prepare-inputs.ts

@@ -0,0 +1,40 @@
+import { mkdir, mkdtemp, rm } from "node:fs/promises";
+import { resolve } from "node:path";
+import { tmpdir } from "node:os";
+import { loadBenchmark } from "./load-benchmark";
+import { prepareWorkspace } from "../infra/containers/workspace";
+import { CandidateContainer, inspectImage } from "../infra/containers/oci";
+import { createSessionDatabase } from "../infra/opencode/host";
+import { treeHash, writeJson } from "../infra/files";
+
+/** Prove preparation without prompting a model, creating EvalRuns, or changing selections. */
+export async function prepareInputs(path: string, options: { directory: string; onlyEvals?: readonly string[] }) {
+  const definition = await loadBenchmark(path);
+  if (options.onlyEvals?.some(id => !definition.evals.some(item => item.id === id)))
+    throw new Error("Select configured evals for preparation");
+  const directory = resolve(options.directory);
+  if (await Bun.file(resolve(directory, "prepared.json")).exists()) throw new Error("Choose a new prepared-input directory");
+  const imageId = await inspectImage(definition.container);
+  const scratch = await mkdtemp(resolve(process.platform === "win32" ? "C:/tmp/opencode" : tmpdir(), "openeval-prepare-"));
+  const inputs: Array<{ eval: string; directory: string; sourceHash: string; preparedHash: string }> = [];
+  try {
+    await mkdir(directory, { recursive: true });
+    for (const item of definition.evals.filter(item => !options.onlyEvals || options.onlyEvals.includes(item.id))) {
+      const staging = resolve(scratch, item.id);
+      await mkdir(staging, { recursive: true });
+      const workspace = resolve(staging, "workspace");
+      await prepareWorkspace(item, workspace);
+      const database = resolve(staging, "opencode.db");
+      // No credentials or model route are needed to prepare task inputs.
+      await createSessionDatabase(database, []);
+      await using container = await CandidateContainer.create(definition.container, imageId);
+      await container.prepare(workspace, database, definition.candidate.websearch, item.settings.prepare ?? [], staging, definition.candidate.providers);
+      const output = resolve(directory, item.id, "workspace");
+      await container.snapshot(output, staging);
+      inputs.push({ eval: item.id, directory: output, sourceHash: item.sourceHash, preparedHash: await treeHash(output) });
+    }
+    const result = { imageId, inputs, candidateExecutions: 0, judgeExecutions: 0 };
+    await writeJson(resolve(directory, "prepared.json"), result);
+    return result;
+  } finally { await rm(scratch, { recursive: true, force: true }); }
+}

package/src/app/read-results.ts

@@ -1,5 +1,5 @@
-import { readdir } from "node:fs/promises";
-import { resolve, dirname } from "node:path";
+import { readdir, realpath } from "node:fs/promises";
+import { resolve, dirname, sep } from "node:path";
 import { Results } from "../infra/sqlite";
 import type { BenchmarkRun, EvalRun, JudgeRun, Slot, Cost } from "../types";
 import type {
@@ -26,7 +26,7 @@ import {
 } from "../runtime";
 import { canJudgeEval } from "./eval-state";
 import { CandidateEvidence, type EvidenceQuery } from "../infra/evidence";
-import { contained } from "../infra/files";
+import { contained, hash } from "../infra/files";
 
 const scheduled = (slot: Slot, benchmark: BenchmarkRun, now: number) =>
   benchmark.state === "running" &&
@@ -167,7 +167,7 @@ export class ResultReader {
     if (!judge) throw new Error("Unknown judge run");
     const candidate = results.evalRun(judge.input.evalRunId);
     const criteria = new Map(
-      judge.input.criteria.map((criterion) => [criterion.id, criterion]),
+      [...judge.input.criteria, ...judge.input.codeCriteria ?? []].map((criterion) => [criterion.id, criterion]),
     );
     for (const id of Object.keys(judge.judgment?.scores ?? {}))
       if (!criteria.has(id))
@@ -225,6 +225,23 @@ export class ResultReader {
         })),
     };
   }
+  async verificationEvidence(benchmarkId: string, judgeRunId: string, id: string, path?: string) {
+    using results = await this.database(benchmarkId);
+    const judge = results.judgeRun(judgeRunId);
+    const verification = judge?.code?.verifications?.find(item => item.id === id);
+    if (!verification) throw new Error("Unknown verification");
+    if (!path) return { receipt: verification };
+    const artifact = verification.artifacts.find(item => item.path === path);
+    const file = path === "stdout" ? verification.stdout : path === "stderr" ? verification.stderr : artifact?.file;
+    if (!file) throw new Error("Unknown verification artifact");
+    const root = await realpath(dirname(results.path));
+    const target = await realpath(contained(root, file));
+    if (target !== root && !target.startsWith(root + sep)) throw new Error("Verification artifact leaves its run directory");
+    const bytes = await Bun.file(target).bytes();
+    if (bytes.length > 32 * 1024 * 1024 || (artifact && hash(bytes) !== artifact.sha256))
+      throw new Error("Verification artifact is too large or its hash differs");
+    return { bytes, path };
+  }
   async checkEvidence(
     benchmarkId: string,
     judgeRunId: string,

package/src/app/rejudge.ts

@@ -7,6 +7,7 @@ import { judgeEvalRun } from "./judge-run";
 import { executeQueue, finishBenchmark } from "./run-benchmark";
 import { judgingFingerprint } from "../infra/judging";
 import { canJudgeEval } from "./eval-state";
+import { verificationRuntime } from "../infra/verification/image";
 
 /** Rejudging changes only the selected judgment. The candidate execution is never repeated. */
 export async function judgeRun(
@@ -59,6 +60,8 @@ export async function judgeRuns(
             judgeHash: updated.judgeHash,
             criteria: updated.criteria,
             code: updated.code,
+            codeCriteria: updated.codeCriteria,
+            categories: updated.categories,
             name: updated.name,
           }
         : evalDefinition;
@@ -67,7 +70,9 @@ export async function judgeRuns(
   const runtime = {
     ...previous.runtime,
     judgeHash: await judgingFingerprint(),
+    verification: await verificationRuntime(definition.judge.verification),
   };
+  if (runtime.verification) definition.judge.verification = runtime.verification;
   const selected = new Set(candidates.map((candidate) => candidate.slotId));
   results.transaction(() => {
     if (results.benchmark?.state === "running")

package/src/app/run-benchmark.ts

@@ -176,7 +176,8 @@ export async function runBenchmark(
   const selected = options.onlyModels?.map(modelRef);
   if (selected && !selected.length)
     throw new Error("Select at least one model");
-  const runtime = await runtimeFingerprint(definition.container);
+  const runtime = await runtimeFingerprint(definition.container, definition.judge.verification);
+  if (runtime.verification) definition.judge.verification = runtime.verification;
   let directory =
     options.directory ??
     (!options.fresh

package/src/app/serve-results.ts

@@ -52,6 +52,20 @@ export async function serveResults(options: {
               url.searchParams.get("judge") ?? "",
             ),
           );
+        if (url.pathname === "/api/verification") {
+          const result = await reader.verificationEvidence(
+            url.searchParams.get("benchmark") ?? "", url.searchParams.get("judge") ?? "",
+            url.searchParams.get("id") ?? "", url.searchParams.get("path") ?? undefined,
+          );
+          if ("receipt" in result) return Response.json(result);
+          const mime = /\.png$/i.test(result.path) ? "image/png" : /\.jpe?g$/i.test(result.path) ? "image/jpeg" :
+            /\.webp$/i.test(result.path) ? "image/webp" : "text/plain; charset=utf-8";
+          return new Response(result.bytes, { headers: {
+            "Content-Type": mime, "X-Content-Type-Options": "nosniff",
+            "Content-Security-Policy": "default-src 'none'; sandbox",
+            "Cache-Control": "no-store",
+          } });
+        }
         if (url.pathname === "/api/check-evidence") {
           const query: EvidenceQuery = {
             action: (url.searchParams.get("action") ??

package/src/cli.ts

@@ -12,6 +12,9 @@ import {
   serveResults,
   mergeBenchmarkRuns,
   snapshotBenchmarkRun,
+  buildVerificationImage,
+  VERIFICATION_IMAGE,
+  prepareInputs,
 } from "./index";
 import type { ModelRef } from "./index";
 
@@ -35,6 +38,7 @@ try {
 Commands:
   image                 Build the candidate container image
   plan                  Show missing or changed work without executing it
+  prepare               Prepare and archive inputs without model calls
   run                   Execute missing or changed work in the current result
   view                  Serve the results viewer
   snapshot <run> <name>  Export a score snapshot
@@ -55,14 +59,29 @@ Options:
   --only-repetition <n>  Execute only this repetition (repeatable)
   --max-cost <usd>       Scheduling budget for this invocation
   --final-only          Judge only after candidates finish
+  --verification        Build only the standard verification image (image command)
+  --output <dir>         New prepared-input directory (prepare command)
   --port <port>         Viewer port (default: 4173)
 
 Requires Bun 1.4.2+, Docker, and an authenticated OpenCode installation.`);
   } else if (command === "--version") {
     console.log((await Bun.file(new URL("../package.json", import.meta.url)).json()).version);
   } else if (command === "image") {
-    await buildImage((await loadBenchmark(benchmark)).container);
-    console.log("Candidate image ready");
+    const definition = await loadBenchmark(benchmark);
+    if (!args.includes("--verification")) {
+      await buildImage(definition.container);
+      console.log("Candidate image ready");
+    }
+    const verification = definition.judge.verification ?? (args.includes("--verification")
+      ? { image: VERIFICATION_IMAGE, engine: definition.container.engine } : undefined);
+    if (verification) {
+      await buildVerificationImage(verification);
+      console.log("Verification image ready");
+    }
+  } else if (command === "prepare") {
+    if (!option("--output")) throw new Error("prepare requires --output with a new directory");
+    const onlyEvals = args.flatMap((arg, index) => arg === "--only-eval" ? [args[index + 1]] : []);
+    console.log(JSON.stringify(await prepareInputs(benchmark, { directory: option("--output")!, onlyEvals: onlyEvals.length ? onlyEvals : undefined }), null, 2));
   } else if (command === "view") {
     const viewer = await serveResults({
       resultsPath: resolve(benchmark, "results"),

package/src/index.ts

@@ -46,8 +46,16 @@ export type {
   RecordedEvent,
   RecordedMessage,
   RecordedFile,
+  CodeScore,
+  VerificationEnvironment,
+  VerificationRuntime,
+  VerificationRequest,
+  VerificationResult,
+  VerificationArtifact,
 } from "./judge-context";
+export { buildVerificationImage, VERIFICATION_IMAGE } from "./infra/verification/image";
 export { loadBenchmark } from "./app/load-benchmark";
+export { prepareInputs } from "./app/prepare-inputs";
 export {
   runBenchmark,
   currentBenchmarkRun,

package/src/infra/containers/oci.ts

@@ -12,6 +12,7 @@ import { extractWorkspaceArchive } from "./transfer";
 import { treeHash, fingerprint, writeJson } from "../files";
 import { OPENCODE_VERSION } from "../opencode/host";
 import { judgingFingerprint } from "../judging";
+import { verificationRuntime } from "../verification/image";
 
 const runtimeDirectory = fileURLToPath(new URL("./runtime/", import.meta.url));
 export async function buildImage(config: BenchmarkDefinition["container"]) {
@@ -46,8 +47,10 @@ export async function inspectImage(config: BenchmarkDefinition["container"]) {
 }
 export async function runtimeFingerprint(
   config: BenchmarkDefinition["container"],
+  verification?: BenchmarkDefinition["judge"]["verification"],
 ) {
   const imageId = await inspectImage(config);
+  const verified = await verificationRuntime(verification);
   return {
     imageId,
     candidateHash: fingerprint({
@@ -80,6 +83,7 @@ export async function runtimeFingerprint(
       ),
     }),
     judgeHash: await judgingFingerprint(),
+    ...(verified ? { verification: verified } : {}),
   };
 }
 

package/src/infra/judging/code-result.ts

@@ -1,6 +1,7 @@
 import type { JsonValue } from "../../judge-context";
 import type { CriterionScore, Judgment } from "../../types";
 import { criterionMean, isScored } from "../../judgment";
+import { parseCitation } from "./contract";
 
 /** Preserve author JSON exactly; reject values JSON would silently discard or coerce. */
 export function jsonOutput(
@@ -53,7 +54,12 @@ export function codeJudgment(output: JsonValue): Judgment {
     Object.entries(values).map(([id, supplied]): [string, CriterionScore] => {
       if (!/^[a-z][a-z0-9_]*$/.test(id))
         throw new Error(`Invalid criterion ID: ${id}`);
-      const value = typeof supplied === "boolean" ? Number(supplied) : supplied;
+      const detailed = supplied !== null && typeof supplied === "object" && !Array.isArray(supplied) ? supplied : undefined;
+      if (detailed && (Object.keys(detailed).some(key => !["value", "reason", "evidence", "measurements"].includes(key)) ||
+        typeof detailed.reason !== "string" || !detailed.reason.trim()))
+        throw new Error(`scores.${id} requires a non-empty reason and only value, reason, evidence, and measurements`);
+      const raw = detailed ? detailed.value : supplied;
+      const value = typeof raw === "boolean" ? Number(raw) : raw;
       if (value !== null && !isScored(value))
         throw new Error(
           `scores.${id} must be a boolean, a finite number from 0 to 1, or null`,
@@ -62,11 +68,23 @@ export function codeJudgment(output: JsonValue): Judgment {
         id,
         {
           value: value as number | null,
-          reason:
+          reason: detailed ? detailed.reason as string :
             value === null
               ? "judge.ts returned an unresolved score."
               : `judge.ts returned ${String(supplied)}.`,
-          evidence: value === null ? [] : [{ kind: "recording" }],
+          evidence: detailed?.evidence !== undefined
+            ? (() => {
+              if (!Array.isArray(detailed.evidence) || detailed.evidence.length > 32)
+                throw new Error(`scores.${id}.evidence must contain at most 32 citations`);
+              return detailed.evidence.map(parseCitation);
+            })()
+            : value === null ? [] : [{ kind: "recording" }],
+          ...(detailed?.measurements !== undefined ? { measurements: (() => {
+            const values = detailed.measurements;
+            if (!values || typeof values !== "object" || Array.isArray(values))
+              throw new Error(`scores.${id}.measurements must be an object`);
+            return values;
+          })() } : {}),
           source: "judge.ts",
         },
       ];