@hona/openeval 0.3.3 → 0.5.0

package/JUDGING.md

@@ -2,7 +2,7 @@
 
 Use the [canonical terminology](https://openev.al/docs/terminology/): a criterion
 is a named graded requirement, a score is awarded credit, and a metric is a
-measurement such as token count or cost. OpenEval 0.3.0 supports code judges,
+measurement such as token count or cost. OpenEval supports code judges,
 LLM judges, and additive use of both against one recorded EvalRun.
 
 ## File conventions
@@ -36,6 +36,9 @@ Only the optional scores object has grading semantics. Each key is a criterion
 ID, using lowercase letters, digits, and underscores, starting with a letter.
 Values are booleans, finite numbers from 0 to 1, or null. The host converts true
 to 1 and false to 0. It rejects invalid values rather than clamping them.
+Optional `{ value, reason, evidence, measurements }` objects make code verdicts
+readable without changing their scoring semantics. Declared code criterion IDs
+must match the returned scores. See [artifact verification](VERIFICATION.md).
 response.text is always a string; missing text becomes an empty string. The
 original execution outcome is available separately on context.run.
 
@@ -69,6 +72,8 @@ a JudgeRun error. A hybrid judgment is selected only after both sources succeed.
 | recording.export(sessionID?) | Native OpenCode session export |
 | workspace.files/read/text/diff | Verified initial and final file snapshots |
 | workspace.materialize(revision?) | A disposable workspace copy for author-owned verification |
+| verification.run(request) | Bounded commands over a restored artifact in an isolated OCI container |
+| verification.read/text(result, path) | Hash-checked retained output from that verification |
 | native.database() | Read-only SQLite access to a verified database copy |
 | native.sdk() | The pinned OpenCode SDK/API over a separate disposable archive copy |
 | native.schema() | The pinned OpenCode schema module |
@@ -152,6 +157,63 @@ must declare at least one `## Criterion: id — Label`. A normalized Judgment ha
 an aggregate value and a scores map of CriterionScore objects containing value,
 reason, evidence, and source. The original code output is retained separately.
 
+## Criterion categories
+
+A category is any non-empty string on a criterion. Categories group results in
+the viewer and can compose a benchmark. They never change judge input,
+fingerprints, or the headline weighting.
+
+In `judge.md`, put one optional line directly below a criterion heading:
+
+```md
+## Criterion: asked_dialect — Asks for the SQL dialect
+Categories: misalignment, general
+```
+
+OpenEval removes that line before hashing the rubric and before the judge reads
+it, so adding or changing categories does not rejudge recorded evidence. A
+`Categories:` line anywhere else is an error.
+
+In `judge.ts`, export labels and categories for the scores it returns:
+
+```ts
+import type { CodeCriteria, JudgeContext } from "@hona/openeval";
+
+export const criteria = {
+  correct_answer: { name: "Correct answer", categories: ["general"] },
+} satisfies CodeCriteria;
+
+export default ({ response }: JudgeContext) => ({
+  scores: { correct_answer: response.text.trim() === "42" },
+});
+```
+
+Declare each criterion ID in one judge file only. Names are trimmed and matched
+case-insensitively; the first spelling is displayed. A criterion may have
+several categories, but one is usually clearer. A category score uses the same
+rule as the overall score: criteria are averaged within each eval, then evals
+are weighted equally. Unscored checks keep that category's score a range.
+
+Compose a benchmark from categories in `benchmark.ts`. Evals without a matching
+criterion are not run, and scores use only matching criteria:
+
+```ts
+export default {
+  models: ["opencode/gpt-6-astra#high"],
+  categories: ["coding", "verification"],
+} satisfies Benchmark;
+```
+
+`openeval run --only-category <name>` limits one invocation to evals with a
+matching criterion, like `--only-eval`, without changing the composition.
+
+Prefer published category sets so results are comparable:
+
+| Kind | Source | Categories |
+| --- | --- | --- |
+| Capability | [Artificial Analysis Intelligence Index](https://artificialanalysis.ai/methodology/intelligence-benchmarking) | `agents`, `coding`, `general`, `scientific-reasoning`; also `multilingual`, `vision` |
+| Agent failure | [MAST](https://arxiv.org/abs/2503.13657) (Cemri et al., NeurIPS 2025) | `specification`, `misalignment`, `verification` |
+
 ## Structured tool submissions
 
 | Tool | Data |

package/README.md

@@ -86,6 +86,10 @@ or does not provide a parameterized query.
 | Only asks which database | **1** | **0** |
 | Required recording is unavailable | **null** | **null** |
 
+Add an optional `Categories: misalignment` line under a heading to compare models
+by category in the viewer's radar chart and heatmap. Categories never trigger
+rejudging.
+
 → [Write good rubrics](https://openev.al/docs/rubrics/) · [Download the SQL starter](https://openev.al/starter.zip)
 
 ## One vocabulary

package/VERIFICATION.md

@@ -0,0 +1,100 @@
+# Artifact verification
+
+Code judges are still plain functions. When the score depends on delivered code,
+run it in a disposable OCI container rather than on the runner host.
+
+```ts
+// benchmark.ts
+import { VERIFICATION_IMAGE, type Benchmark } from "@hona/openeval";
+export default {
+  models: ["example/model"],
+  judge: { verification: { image: VERIFICATION_IMAGE, cpus: 2, memoryMiB: 4096 } },
+} satisfies Benchmark;
+```
+
+Build the candidate and standard verification images with `openeval image`.
+`openeval image --verification` builds only the verification image. Custom images
+must be built separately. Images resolve to immutable IDs before planning or
+collection; a changed verification image invalidates judgment reuse, not the
+candidate's delivered artifacts.
+
+```ts
+// judge.ts — trusted script content is an ordinary frozen local import/string.
+import type { JudgeContext } from "@hona/openeval";
+export const criteria = { correct: { name: "Correct output", categories: ["coding"] } };
+export default async (ctx: JudgeContext) => {
+  const result = await ctx.verification.run({
+    revision: "final", cwd: ".", timeoutMs: 60_000,
+    commands: [["bun", "/verification/check.mjs"]],
+    files: { "check.mjs": "/* author's outcome checks */" },
+    artifacts: ["checks.json", "screenshot.png"],
+  });
+  return { scores: { correct: {
+    value: result.state === "completed" && result.exitCode === 0,
+    reason: "Explain what the checks established or why they failed.",
+    evidence: [{ kind: "verification", id: result.id }],
+    measurements: { elapsedMs: result.elapsedMs },
+  } } };
+};
+```
+
+The primitive restores an initial or final snapshot, uploads trusted inputs to
+`/verification`, runs argv arrays in order, and stops at a nonzero exit. It does
+not choose scoring thresholds or interpret success. Judge setup/transfer/image
+errors remain JudgeRun errors. A test exit or verification timeout is an
+observation the authored criterion must interpret. Interrupted candidate work
+still needs the rubric's evidence policy; a failed verification of an unfinished
+prefix does not necessarily establish final-task failure.
+
+## Isolation and bounds
+
+- No host bind mounts, credentials, published ports, or network. Browser/server
+  verification can use loopback **inside** the container.
+- Read-only image filesystem, non-root command user, no capabilities, no privilege
+  escalation, PID/CPU/memory limits, bounded workspace/temp storage, and a deadline.
+- Trusted check inputs are root-owned and are not writable by delivered code.
+- Commands, exit statuses, image/resource identity, logs, and requested regular
+  output files are retained with the JudgeRun. Viewer previews never execute HTML
+  or SVG from the delivered artifact.
+- Workspace transfer is limited to 128 MiB; each output archive is limited to
+  32 MiB; logs are limited to 1 MiB per stream and are marked when truncated.
+- Containers self-expire. The parent also removes containers bearing only its
+  unique execution label after worker interruption. No shared/broad cleanup.
+
+The standard image includes Bun 1.4.2, Python, Git, Node, and Playwright 1.63.0
+with Chromium. Import Playwright from
+`/opt/verify/node_modules/playwright/index.mjs`. Dependencies needed by a task
+must be available in the image or supplied as frozen inputs; network installs
+are intentionally unavailable. Materialization alone is not a sandbox.
+`verification.text(result, "stdout" | "stderr")` reads the retained bounded logs;
+those names are reserved and cannot also name output artifacts.
+
+The primitive verifies a reconstructed artifact. It does **not** prove what was
+alive in the original candidate container, whether the candidate ran a test,
+or whether original game actions were legal. Those claims need original recording
+evidence. Do not put evaluator scripts in candidate workspaces.
+
+## Readable judgments and controls
+
+`openeval prepare --output <new-directory>` assembles real candidate inputs and
+runs declared preparation in the candidate image, then archives the prepared
+workspace. `--only-eval` scopes it. This makes zero model calls, creates no
+EvalRun/JudgeRun, and does not change selections. Use it to prove fixture setup
+before collection; it does not prove agent success or human task duration.
+
+Existing boolean, numeric, and null scores remain sufficient. Optional structured
+scores add `reason`, `evidence`, and JSON `measurements`. These details appear in
+the normal viewer beside verification receipts; arbitrary returned JSON remains
+inspectable. Declared code criteria must be returned exactly. Missing evidence
+references are judging errors, not candidate zeros.
+
+`recordEvidence({ workspace: { initial, final }, ... })` can retain constructed
+artifact controls without executing them. `judgeEvidence` then uses the same
+public verification primitive. Constructed controls prove tested boundaries, not
+human-duration calibration or live candidate feasibility.
+
+Unused `criteria` labels/categories are removed from executable bundling. Debug
+source maps remain retained but do not affect code identity. If grading reads a
+metadata value, it is executable behavior and remains fingerprinted. Changes to
+criterion IDs, actual grading code, imported inputs, dependencies, or verification
+environment are not reporting-only edits.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@hona/openeval",
-  "version": "0.3.3",
+  "version": "0.5.0",
   "description": "Code and LLM evaluations with isolated agents, recorded OpenCode data, and a results viewer",
   "license": "MIT",
   "repository": {
@@ -14,7 +14,7 @@
   "bin": {
     "openeval": "src/cli.ts"
   },
-  "files": ["src", "viewer", "LICENSE", "README.md", "JUDGING.md", "!src/**/*.test.ts"],
+  "files": ["src", "viewer", "LICENSE", "README.md", "JUDGING.md", "VERIFICATION.md", "!src/**/*.test.ts"],
   "publishConfig": {
     "access": "public"
   },

package/src/app/grade-recording.ts

@@ -11,6 +11,8 @@ import { executeCodeJudge } from "../infra/judging/code";
 import { codeJudgment, combineJudgments } from "../infra/judging/code-result";
 import { executeJudge } from "../infra/judging";
 import { errorMessage } from "../infra/files";
+import { CandidateEvidence } from "../infra/evidence";
+import { validateCitations } from "../infra/judging/contract";
 
 export type GradedRecording = {
   state: "completed" | "failed" | "timed_out";
@@ -37,10 +39,24 @@ export async function gradeRecording(
         recording,
         resolve(directory, "code"),
         input.timeoutMs,
+        input.verification,
       );
       if (code.state !== "completed")
         return { state: code.state, code, error: code.error };
       const judged = codeJudgment(code.output!);
+      if (input.codeCriteria?.length && (
+        Object.keys(judged.scores).length !== input.codeCriteria.length ||
+        input.codeCriteria.some(item => !Object.hasOwn(judged.scores, item.id))
+      )) throw new Error("judge.ts must return exactly its declared criterion IDs");
+      for (const score of Object.values(judged.scores)) for (const citation of score.evidence) {
+        if (citation.kind !== "verification") continue;
+        const result = code.verifications?.find(item => item.id === citation.id);
+        if (!result || (citation.path && !result.artifacts.some(item => item.path === citation.path)))
+          throw new Error("Code judgment cites missing verification evidence");
+      }
+      await validateCitations({ ...judged, scores: Object.fromEntries(Object.entries(judged.scores).map(([id, score]) =>
+        [id, { ...score, evidence: score.evidence.filter(citation => citation.kind !== "verification") }])) },
+        await CandidateEvidence.open(recording.evidence.directory, recording.evidence.hash));
       for (const id of Object.keys(judged.scores))
         if (input.criteria.some((criterion) => criterion.id === id))
           throw new Error(

package/src/app/input-fingerprints.ts

@@ -61,6 +61,7 @@ export function candidateFingerprint(
       timeoutMs: definition.candidate.timeoutMs,
       websearch: definition.candidate.websearch,
       provider: scope && { ...scope.settings, model: scope.override },
+      earlyStop: item.settings.earlyStop ?? false,
     },
     container: {
       engine: definition.container.engine,
@@ -76,12 +77,13 @@ export const judgeFingerprint = (
   fingerprint({
     rubric: definition.evals.find((item) => item.id === evalId)!.judge,
     code: definition.evals.find((item) => item.id === evalId)!.code?.hash,
+    codeIds: definition.evals.find((item) => item.id === evalId)!.codeCriteria?.map(item => item.id).sort(),
     agent: definition.evals.find((item) => item.id === evalId)!.judge
       ? JUDGE_AGENT
       : undefined,
     judge: definition.evals.find((item) => item.id === evalId)!.judge
       ? definition.judge
-      : { timeoutMs: definition.judge.timeoutMs },
+      : { timeoutMs: definition.judge.timeoutMs, verification: definition.judge.verification },
     protocol: JUDGE_PROTOCOL,
   });
 export const savedJudgeFingerprint = (
@@ -94,11 +96,14 @@ export const savedJudgeFingerprint = (
     | "timeoutMs"
     | "websearch"
     | "code"
+    | "codeCriteria"
+    | "verification"
   >,
 ) =>
   fingerprint({
     rubric: input.rubric,
     code: input.code?.hash,
+    codeIds: input.codeCriteria?.map(item => item.id).sort(),
     agent: input.agent,
     protocol: input.protocol,
     judge: input.rubric
@@ -106,6 +111,7 @@ export const savedJudgeFingerprint = (
           model: input.model,
           timeoutMs: input.timeoutMs,
           websearch: input.websearch,
+          verification: input.verification,
         }
-      : { timeoutMs: input.timeoutMs },
+      : { timeoutMs: input.timeoutMs, verification: input.verification },
   });

package/src/app/judge-evidence.ts

@@ -9,6 +9,8 @@ import { savedJudgeFingerprint } from "./input-fingerprints";
 import { mkdir, mkdtemp, rm } from "node:fs/promises";
 import { compileCodeJudge } from "../infra/judging/code-source";
 import { gradeRecording } from "./grade-recording";
+import { readCodeCriteria } from "../infra/judging/code-criteria";
+import { verificationRuntime } from "../infra/verification/image";
 
 /** Inspect retained evidence without changing benchmark selections.
  * Useful for calibration and second opinions: https://arxiv.org/abs/2410.12784
@@ -28,6 +30,7 @@ export async function judgeEvidence(options: {
   if (options.rubric && !options.judge?.model)
     throw new Error("A Markdown rubric requires judge.model");
   const code = options.code ? await compileCodeJudge(options.code) : undefined;
+  const declared = options.code ? await readCodeCriteria(options.code) : undefined;
   const request: Omit<JudgeRunInput, "judgeHash"> = {
     evalRunId: "retained-evidence",
     evidence: {
@@ -37,6 +40,8 @@ export async function judgeEvidence(options: {
     rubric: options.rubric ?? "",
     kind: code ? (options.rubric ? "hybrid" : "code") : "llm",
     code,
+    codeCriteria: declared ? Object.entries(declared as Record<string, { name?: string }>).map(([id, value]) => ({ id, name: value.name ?? id })) : undefined,
+    verification: await verificationRuntime(options.judge?.verification),
     agent: options.rubric ? JUDGE_AGENT : undefined,
     model: options.rubric ? options.judge?.model : undefined,
     timeoutMs: options.judge?.timeoutMs ?? 600_000,
@@ -67,17 +72,18 @@ export async function recordEvidence(options: {
   prompt: string;
   response: string;
   tools?: ToolCall[];
+  workspace?: { initial?: string; final?: string };
 }): Promise<EvidenceRef> {
   const directory = resolve(options.directory);
   await mkdir(dirname(directory), { recursive: true });
   const workspace = await mkdtemp(resolve(dirname(directory), ".recording-"));
   try {
-    const capture = await EvidenceCapture.create(directory);
+    const capture = await EvidenceCapture.create(directory, options.workspace?.initial);
     const result = await capture.finish({
       prompt: options.prompt,
       response: { text: options.response },
       tools: options.tools ?? [],
-      workspace,
+      workspace: options.workspace?.final ?? workspace,
     });
     return { directory, hash: result.sha256 };
   } finally {

package/src/app/judge-run.ts

@@ -27,6 +27,8 @@ export async function judgeEvalRun(
     model: definition.judge ? context.definition.judge.model : undefined,
     kind: definition.code ? (definition.judge ? "hybrid" : "code") : "llm",
     code: definition.code,
+    codeCriteria: definition.codeCriteria,
+    verification: context.runtime.verification,
     judgeHash: slot.judgeHash,
     timeoutMs: context.definition.judge.timeoutMs,
     websearch: context.definition.judge.websearch,
@@ -76,6 +78,14 @@ export async function judgeEvalRun(
             ...graded.code,
             stdout: relative(context.directory, graded.code.stdout),
             stderr: relative(context.directory, graded.code.stderr),
+            verifications: graded.code.verifications?.map(result => ({
+              ...result,
+              stdout: relative(context.directory, resolve(directory, "code", result.stdout)),
+              stderr: relative(context.directory, resolve(directory, "code", result.stderr)),
+              artifacts: result.artifacts.map(artifact => ({ ...artifact,
+                file: relative(context.directory, resolve(directory, "code", artifact.file)),
+              })),
+            })),
           }
         : undefined,
       session: graded.session

package/src/app/load-benchmark.ts

@@ -10,12 +10,17 @@ import type {
 } from "../types";
 import { CANDIDATE_TIMEOUT_MS } from "../types";
 import { rubricCriteria } from "../judgment";
+import {
+  categoryKey,
+  categoryList,
+  rubricCategories,
+} from "../criterion-categories";
 import { compileCodeJudge } from "../infra/judging/code-source";
+import { readCodeCriteria } from "../infra/judging/code-criteria";
 import { RUNTIME_IMAGE } from "../infra/opencode/version";
 import { monitorPolicy } from "./monitor-policy";
 import {
   fingerprint,
-  hash,
   relativePath,
   treeHash,
   contained,
@@ -35,10 +40,13 @@ export const modelRef = (value: unknown): ModelRef => {
     throw new Error(`Invalid model reference: ${String(value)}`);
   return value as ModelRef;
 };
+/** Bun caches modules by path, ignoring URL queries; clear the entry so edits load. */
+async function freshModule(path: string): Promise<Record<string, unknown>> {
+  delete require.cache[path];
+  return import(pathToFileURL(path).href);
+}
 async function declaration<T>(path: string): Promise<T> {
-  const url = pathToFileURL(path);
-  url.searchParams.set("version", hash(await Bun.file(path).bytes()));
-  return (await import(url.href)).default;
+  return (await freshModule(path)).default as T;
 }
 async function loadEval(directory: string): Promise<EvalDefinition> {
   const id = basename(directory),
@@ -168,10 +176,23 @@ async function loadEval(directory: string): Promise<EvalDefinition> {
       throw new Error(`${id}: preparation requires argv`);
   }
   const promptText = await prompt.text(),
-    judgeText = hasMarkdown ? await judge.text() : "";
+    rubric = rubricCategories(hasMarkdown ? await judge.text() : ""),
+    judgeText = rubric.rubric;
   if (!promptText.trim() || (hasMarkdown && !judgeText.trim()))
     throw new Error(`${id}: prompt and judge must not be empty`);
   const code = hasCode ? await compileCodeJudge(codeFile) : undefined;
+  const criteria = hasMarkdown ? rubricCriteria(judgeText) : [];
+  const declared = hasCode ? await codeCriteria(id, codeFile) : [];
+  if (declared.some((item) => criteria.some(({ id }) => id === item.id)))
+    throw new Error(
+      `${id}: declare each criterion in either judge.md or judge.ts, not both`,
+    );
+  const categories = Object.fromEntries(
+    [
+      ...Object.entries(rubric.categories),
+      ...declared.map((item) => [item.id, item.categories] as const),
+    ].filter(([, names]) => names.length),
+  );
   return {
     id,
     directory,
@@ -183,12 +204,49 @@ async function loadEval(directory: string): Promise<EvalDefinition> {
       source,
     }),
     judgeHash: fingerprint({ rubric: judgeText, code: code?.hash }),
-    criteria: hasMarkdown ? rubricCriteria(judgeText) : [],
+    criteria,
+    ...(declared.length
+      ? { codeCriteria: declared.map(({ id, name }) => ({ id, name })) }
+      : {}),
+    ...(Object.keys(categories).length ? { categories } : {}),
     ...(code ? { code } : {}),
     name: /^# (.+)$/m.exec(judgeText)?.[1] ?? id,
   };
 }
 
+/** Reads judge.ts's optional `criteria` export as data; planning never executes judge code. */
+async function codeCriteria(id: string, path: string) {
+  const declared = await readCodeCriteria(path);
+  if (declared === undefined) return [];
+  if (!declared || typeof declared !== "object" || Array.isArray(declared))
+    throw new Error(`${id}/judge.ts criteria must be an object`);
+  return Object.entries(declared).map(([criterion, value]) => {
+    const where = `${id}/judge.ts criterion ${criterion}`;
+    if (!/^[a-z][a-z0-9_]*$/.test(criterion))
+      throw new Error(`${where}: use a lowercase snake_case ID`);
+    if (
+      !value ||
+      typeof value !== "object" ||
+      Array.isArray(value) ||
+      Object.keys(value).some((key) => !["name", "categories"].includes(key))
+    )
+      throw new Error(`${where}: declare only name and categories`);
+    const { name, categories = [] } = value as {
+      name?: unknown;
+      categories?: unknown;
+    };
+    if (name !== undefined && (typeof name !== "string" || !name.trim()))
+      throw new Error(`${where}: name must be a non-empty string`);
+    if (!Array.isArray(categories))
+      throw new Error(`${where}: categories must be an array`);
+    return {
+      id: criterion,
+      name: (name as string | undefined)?.trim() ?? criterion.replaceAll("_", " "),
+      categories: categoryList(categories, where),
+    };
+  });
+}
+
 export async function loadBenchmark(
   path: string,
 ): Promise<BenchmarkDefinition> {
@@ -214,10 +272,19 @@ export async function loadBenchmark(
           "concurrency",
           "candidate",
           "container",
+          "categories",
         ].includes(key),
     )
   )
     throw new Error("benchmark.ts contains unsupported settings");
+  if (
+    definition.categories !== undefined &&
+    (!Array.isArray(definition.categories) || !definition.categories.length)
+  )
+    throw new Error("benchmark.ts categories must be a non-empty array");
+  const categories = definition.categories
+    ? categoryList(definition.categories, "benchmark.ts")
+    : undefined;
   const models = definition.models.map(modelRef);
   if (
     definition.judge !== undefined &&
@@ -225,11 +292,11 @@ export async function loadBenchmark(
       typeof definition.judge !== "object" ||
       Array.isArray(definition.judge) ||
       Object.keys(definition.judge).some(
-        (key) => !["model", "timeoutMs", "websearch"].includes(key),
+        (key) => !["model", "timeoutMs", "websearch", "verification"].includes(key),
       ))
   )
     throw new Error(
-      "judge must be an object with model, timeoutMs, or websearch settings",
+      "judge must be an object with model, timeoutMs, websearch, or verification settings",
     );
   if (new Set(models).size !== models.length)
     throw new Error("Benchmark contains duplicate models");
@@ -245,9 +312,21 @@ export async function loadBenchmark(
     .filter((entry) => entry.isDirectory() && !entry.name.startsWith("."))
     .sort((a, b) => a.name.localeCompare(b.name));
   if (!directories.length) throw new Error("Benchmark has no eval folders");
-  const evals = await Promise.all(
+  const declared = await Promise.all(
     directories.map((entry) => loadEval(resolve(evalRoot, entry.name))),
   );
+  const keys = categories?.map(categoryKey);
+  const evals = keys
+    ? declared.filter((item) =>
+        Object.values(item.categories ?? {}).some((names) =>
+          names.some((name) => keys.includes(categoryKey(name))),
+        ),
+      )
+    : declared;
+  if (!evals.length)
+    throw new Error(
+      `No criteria match the benchmark categories: ${categories!.join(", ")}`,
+    );
   if (evals.some((item) => item.judge) && !definition.judge?.model)
     throw new Error("A benchmark containing judge.md requires judge.model");
   const concurrency = positive(definition.concurrency, 10, "Concurrency");
@@ -258,6 +337,13 @@ export async function loadBenchmark(
   const engine = definition.container?.engine ?? "docker";
   if (engine !== "docker" && engine !== "podman")
     throw new Error("Container engine must be docker or podman");
+  const verification = definition.judge?.verification;
+  if (verification && (
+    typeof verification !== "object" || Array.isArray(verification) ||
+    Object.keys(verification).some(key => !["image", "engine", "cpus", "memoryMiB"].includes(key)) ||
+    typeof verification.image !== "string" || !verification.image.trim() || /\s/.test(verification.image) ||
+    !["docker", "podman"].includes(verification.engine ?? engine)
+  )) throw new Error("judge.verification requires an image and valid container limits");
   for (const search of [
     definition.candidate?.websearch,
     definition.judge?.websearch,
@@ -288,6 +374,12 @@ export async function loadBenchmark(
         "Judge timeout",
       ),
       websearch: definition.judge?.websearch ?? "exa",
+      ...(verification ? { verification: {
+        image: verification.image,
+        engine: verification.engine ?? engine,
+        cpus: positive(verification.cpus, 2, "Verification CPU count"),
+        memoryMiB: positive(verification.memoryMiB, 4096, "Verification memory"),
+      } } : {}),
     },
     container: {
       engine,
@@ -295,5 +387,6 @@ export async function loadBenchmark(
       cpus: positive(definition.container?.cpus, 2, "CPU count"),
       memoryMiB: positive(definition.container?.memoryMiB, 4096, "Memory"),
     },
+    ...(categories ? { categories } : {}),
   };
 }

package/src/app/prepare-inputs.ts

@@ -0,0 +1,40 @@
+import { mkdir, mkdtemp, rm } from "node:fs/promises";
+import { resolve } from "node:path";
+import { tmpdir } from "node:os";
+import { loadBenchmark } from "./load-benchmark";
+import { prepareWorkspace } from "../infra/containers/workspace";
+import { CandidateContainer, inspectImage } from "../infra/containers/oci";
+import { createSessionDatabase } from "../infra/opencode/host";
+import { treeHash, writeJson } from "../infra/files";
+
+/** Prove preparation without prompting a model, creating EvalRuns, or changing selections. */
+export async function prepareInputs(path: string, options: { directory: string; onlyEvals?: readonly string[] }) {
+  const definition = await loadBenchmark(path);
+  if (options.onlyEvals?.some(id => !definition.evals.some(item => item.id === id)))
+    throw new Error("Select configured evals for preparation");
+  const directory = resolve(options.directory);
+  if (await Bun.file(resolve(directory, "prepared.json")).exists()) throw new Error("Choose a new prepared-input directory");
+  const imageId = await inspectImage(definition.container);
+  const scratch = await mkdtemp(resolve(process.platform === "win32" ? "C:/tmp/opencode" : tmpdir(), "openeval-prepare-"));
+  const inputs: Array<{ eval: string; directory: string; sourceHash: string; preparedHash: string }> = [];
+  try {
+    await mkdir(directory, { recursive: true });
+    for (const item of definition.evals.filter(item => !options.onlyEvals || options.onlyEvals.includes(item.id))) {
+      const staging = resolve(scratch, item.id);
+      await mkdir(staging, { recursive: true });
+      const workspace = resolve(staging, "workspace");
+      await prepareWorkspace(item, workspace);
+      const database = resolve(staging, "opencode.db");
+      // No credentials or model route are needed to prepare task inputs.
+      await createSessionDatabase(database, []);
+      await using container = await CandidateContainer.create(definition.container, imageId);
+      await container.prepare(workspace, database, definition.candidate.websearch, item.settings.prepare ?? [], staging, definition.candidate.providers);
+      const output = resolve(directory, item.id, "workspace");
+      await container.snapshot(output, staging);
+      inputs.push({ eval: item.id, directory: output, sourceHash: item.sourceHash, preparedHash: await treeHash(output) });
+    }
+    const result = { imageId, inputs, candidateExecutions: 0, judgeExecutions: 0 };
+    await writeJson(resolve(directory, "prepared.json"), result);
+    return result;
+  } finally { await rm(scratch, { recursive: true, force: true }); }
+}