@hona/openeval 0.2.2 → 0.3.0

package/JUDGING.md

@@ -1,8 +1,115 @@
 # Judging recorded work
 
+Use the [canonical terminology](https://openev.al/docs/terminology/): a criterion
+is a named graded requirement, a score is awarded credit, and a metric is a
+measurement such as token count or cost. OpenEval 0.3.0 supports code judges,
+LLM judges, and additive use of both against one recorded EvalRun.
+
+## File conventions
+
+Every eval has prompt.md and at least one judge file:
+
+| File | Role |
+| --- | --- |
+| judge.md | An LLM rubric with named criteria |
+| judge.ts | An ordinary default-exported function receiving JudgeContext |
+| Both | Both contribute distinct criterion scores; duplicate IDs are errors |
+
+Only benchmarks containing judge.md need judge.model. Code judges run in a
+separate Bun process on finalized evidence, under judge.timeoutMs (default ten
+minutes). Code and hybrid evals use final grading; earlyStop is available to
+Markdown-only evals. Candidate execution remains isolated from all judge code.
+
+## Plain code judges
+
+```ts
+import type { JudgeContext } from "@hona/openeval";
+
+export default ({ response }: JudgeContext) => ({
+  scores: { correct_answer: response.text === "APPLE" },
+  observed: response.text,
+});
+```
+
+The function may be asynchronous and may return any JSON-compatible value.
+Only the optional scores object has grading semantics. Each key is a criterion
+ID, using lowercase letters, digits, and underscores, starting with a letter.
+Values are booleans, finite numbers from 0 to 1, or null. The host converts true
+to 1 and false to 0. It rejects invalid values rather than clamping them.
+response.text is always a string; missing text becomes an empty string. The
+original execution outcome is available separately on context.run.
+
+Custom output is retained verbatim. An output without scores is unscored and
+cannot silently disappear from the benchmark denominator. Use consistent score
+IDs across models and repetitions; a missing required score stays unresolved.
+There are no built-in task-specific scorers, registration steps, or builder APIs.
+
+The host bundles local imports before execution, captures source maps and
+dependency manifests/lockfiles, and records their fingerprint. Code, imported
+references, or dependency changes schedule rejudging rather than candidate
+execution. Use static imports for reference data; make runtime network and file
+inputs reproducible when an author-owned judge uses them.
+
+The viewer shows normalized criterion scores, original returned JSON, frozen
+source, process logs, and recorded candidate metrics. A synchronous loop can be
+terminated by the host deadline. A code exception, invalid result, or timeout is
+a JudgeRun error. A hybrid judgment is selected only after both sources succeed.
+
+## JudgeContext and recorded data
+
+| Primitive | Data |
+| --- | --- |
+| response / prompt | Final root answer and the exact task prompt |
+| run | Recorded EvalRun and runtime inputs; null for constructed controls |
+| metrics | Candidate-only usage, cost, tool reliability, compactions, and timing |
+| recording.events(filter?) | Complete retained native events with their sequence and time |
+| recording.tools(filter?) | Inputs, outputs, states, and timing of recorded invocations |
+| recording.sessions() | All sessions in the candidate's isolated native archive |
+| recording.messages(sessionID?) | Full paginated native history, including before compaction |
+| recording.export(sessionID?) | Native OpenCode session export |
+| workspace.files/read/text/diff | Verified initial and final file snapshots |
+| workspace.materialize(revision?) | A disposable workspace copy for author-owned verification |
+| native.database() | Read-only SQLite access to a verified database copy |
+| native.sdk() | The pinned OpenCode SDK/API over a separate disposable archive copy |
+| native.schema() | The pinned OpenCode schema module |
+
+Native readers initialize lazily and are disposed by the runner. SDK operations
+affect only their disposable copy. The native archive version must match the
+reader version. Files and database copies are checked against recorded hashes.
+These APIs expose recorded data, not the user's live OpenCode service.
+
+For independent inspection:
+
+```ts
+import { readRecording } from "@hona/openeval";
+
+await using context = await readRecording("./results/RUN", "eval_ID");
+console.log(context.metrics.tools.errorRate);
+const history = await context.recording.messages();
+const database = await context.native.database();
+console.log(database.query("SELECT name FROM sqlite_master").all());
+```
+
+Metrics use deduplicated durable events across the candidate execution's
+sessions. Usage includes recorded model requests and auxiliary usage such as
+compaction. Cost is reported OpenCode usage, not an invoice or a promise of free
+service. Missing usage is unavailable. Code that calls external services
+directly can return its own accounting as metadata.
+
+Tool error rate is failed / (succeeded + failed), with null when there are no
+terminal calls. Unfinished calls are reported separately. A successful shell
+tool reporting failed tests is not a native tool failure. Counts describe
+recorded native invocations; do not infer uncaptured work inside a batched call.
+
+Timing uses the union of closed recorded intervals. modelActiveMs includes
+model-step and compaction spans, including time within those steps such as
+retries. outputTokensPerSecond is reported output tokens per model-active second.
+Token categories retain their native meanings; do not blindly add overlapping
+reasoning, output, or cache categories into a new total.
+
 ## Shared judge agent
 
-OpenEval uses the native OpenCode V2 primary agent `openeval-judge` for final
+The Markdown judge uses the native OpenCode V2 primary agent `openeval-judge` for final
 grading, live checks, rejudging, calibration, and independent audits. Its
 [base system prompt](src/infra/judging/judge-agent.md) owns the common evidence,
 citation, uncertainty, early-decision, and output rules. See
@@ -13,43 +120,44 @@ the eval rubric. It remains present across compaction. User turns identify the
 phase; structured context and decisions move through registered Code Mode tools.
 The tools use native Effect schemas for argument validation and catalog types.
 
-Every JudgeRun requires its shared profile in `input.agent`, explicit metrics,
+An LLM JudgeRun records its shared profile in `input.agent`, explicit criteria,
 mode, runtime hash, and protocol. The effective
 native configuration is archived at `configuration/opencode.json` in its judge
 directory. The profile contributes to the judge input fingerprint, so a shared
 prompt change schedules rejudging using saved evidence. Candidate input
 fingerprints do not include the judge profile.
 
-## Eval metric rubrics
+## Eval rubrics
 
-Declare one or more metrics in `judge.md`:
+Declare one or more criteria in `judge.md`:
 
 ```md
 # Advice quality
 
-## Metric: current_advice — Advice for the current situation
+## Criterion: current_advice — Advice for the current situation
 Pass when the requested advice is present and its recommendations are available now.
 Fail when a recommendation is unavailable now, or the requested advice is omitted.
 
-## Metric: later_advice — Advice for later stages
+## Criterion: later_advice — Advice for later stages
 Judge later recommendations at their explicitly stated stage.
 ```
 
-Keep metric-specific rules, accepted alternatives, and domain facts in that file.
-The LLM applies these rules. The SDK validates the declared IDs, binary metric
+Keep criterion-specific rules, accepted alternatives, and domain facts in that file.
+The LLM applies these rules. The SDK validates the declared IDs, normalized criterion
 values, recorded evidence references, and exact optional quotes. It calculates
-the equal-weight metric mean. A missing metric or invalid citation is a judge
-protocol error; missing source evidence can produce a null metric. Every rubric
-must declare at least one `## Metric: id — Label`, and every judgment contains
-the complete metric array.
+the equal-weight mean of criterion scores. A missing criterion or invalid citation is a judge
+protocol error; missing source evidence can produce a null criterion score. Every rubric
+must declare at least one `## Criterion: id — Label`. A normalized Judgment has
+an aggregate value and a scores map of CriterionScore objects containing value,
+reason, evidence, and source. The original code output is retained separately.
 
 ## Structured tool submissions
 
 | Tool | Data |
 | --- | --- |
-| `judge_context` | Current request ID, phase, metric declarations, and evidence index |
-| `candidate_evidence` | Recorded responses, tools, events, messages, and artifacts |
-| `submit_judgment` | Scores keyed by metric ID, with reasons and citations |
+| `judge_context` | Current request ID, phase, criterion declarations, and evidence index |
+| `candidate_evidence` | Recorded responses, tools, events, messages, artifacts, and metrics |
+| `submit_judgment` | Scores keyed by criterion ID, with reasons and citations |
 | `continue_judging` | An early check's reason for needing more evidence |
 
 Native argument validation and citation validation return errors directly to the
@@ -61,14 +169,14 @@ submission fails the JudgeRun rather than assigning a candidate zero.
 Tool schemas stay stable across checks. Each request ID is bound to one fixed
 evidence view. Only the first valid submission is accepted; stale, concurrent, or
 cancelled submissions cannot overwrite it or affect a later check. Early
-submissions require every metric to be non-null and irreversible. Final grading
-permits null metrics and rejects `continue_judging`.
+submissions require every criterion score to be non-null and irreversible. Final grading
+permits null criterion scores and rejects `continue_judging`.
 
 Accepted submissions and semantic rejections are recorded in
 `judgment-submissions.json`, with their request/check IDs and checkpoints. Native
 schema failures are retained in the native tool transcript.
 
-Every decided metric cites recorded evidence. Citations can name a response,
+Every decided criterion score cites recorded evidence. Citations can name a response,
 message ID, tool-call ID, event sequence, or initial/final artifact path. The
 evidence reference/checkpoint binds those citations to the exact recording.
 Source URLs are supplementary domain references, not substitutes for citations
@@ -80,7 +188,7 @@ For example, the judge submits this from Code Mode after inspecting the recordin
 const context = await tools.judge_context({});
 await tools.submit_judgment({
   requestId: context.requestId,
-  metrics: {
+  scores: {
     current_advice: {
       value: 1,
       reason: "The current-stage recommendation satisfies the rubric.",
@@ -95,7 +203,7 @@ await tools.submit_judgment({
 });
 ```
 
-The host stores the metric array and aggregate value `0.5`. A citation's optional `quote` must occur
+The host stores the criterion-score map and aggregate value `0.5`. A citation's optional `quote` must occur
 literally in the referenced response/tool/message/event/artifact. An artifact
 citation must specify `path` and `revision` (`initial` or `final`); other record
 citations specify `id`. Semantic interpretation remains the judge's job.
@@ -103,7 +211,9 @@ citations specify `id`. Semantic interpretation remains the judge's job.
 `recordEvidence` creates a calibration recording from supplied text/tool records
 without executing a model or tool. `judgeEvidence` grades retained evidence into
 a new standalone audit directory without changing a benchmark's active scores.
-Use `judgeRuns` when an explicit rejudge should update active selections.
+Use `judgeRuns` when an explicit rejudge should update active selections. For a
+code-only control, pass code: "./path/to/judge.ts" to judgeEvidence; no judge model
+is needed. Pass both rubric and code for additive grading.
 
 Runtime uses recorded work intervals. Judge checks record `executionStartedAt`
 when they obtain a worker, separately from their queue-admission `startedAt`.
@@ -119,5 +229,6 @@ creation/completion dates remain provenance, not elapsed-runtime measurements.
 - [JudgeBench](https://arxiv.org/abs/2410.12784): validate factual and logical
   judging ability with known examples, rather than assuming model strength.
 
-Keep task-specific decisions in `judge.md`; executable SDK code handles the
-recording protocol and generic arithmetic, not task success rules.
+Keep task-specific decisions in the author-owned judge.md and judge.ts files.
+The SDK handles recording, execution, validation, and equal-weight aggregation.
+OpenEval 0.3.0 uses results schema 5 and the canonical API only.

package/README.md

@@ -1,7 +1,7 @@
 <div align="center">
   <h1>OpenEval</h1>
   <p><strong>Write the task. Judge the evidence.</strong></p>
-  <p>Prompt-and-rubric evaluations for agents. Typed declarations, isolated runs, inspectable scores.</p>
+  <p>Code and LLM judges for agents. Plain functions, isolated runs, inspectable scores.</p>
   <p>
     <a href="https://openev.al">Website</a> ·
     <a href="https://openeval.pages.dev">Live preview</a> ·
@@ -16,18 +16,46 @@
   </p>
 </div>
 
-![The OpenEval workbench: rubric source, a sample SQL response, and linked metric decisions](https://raw.githubusercontent.com/Hona/openeval/main/docs/images/workbench.png)
+![The OpenEval workbench: rubric source, a sample SQL response, and linked criterion scores](https://raw.githubusercontent.com/Hona/openeval/main/docs/images/workbench.png)
 
 *Interactive documentation example. Viewer screenshots use illustrative data and fictional model labels.*
 
-## An eval is two files
+## A prompt and a judge
 
 | File | What you write | Who reads it |
 | --- | --- | --- |
 | `prompt.md` | A natural, focused task | Candidate agent |
-| `judge.md` | Named metrics and pass/fail criteria | Judge agent |
+| `judge.md` | A rubric with named criteria and scoring rules | LLM judge |
+| `judge.ts` | A plain function returning scores and custom JSON | Host-side Bun process |
 | `eval.ts` *(optional)* | Workspace preparation and early stopping | Host |
 
+Use `judge.md`, `judge.ts`, or both. Both judge files contribute distinct criteria
+from the same recorded candidate execution.
+
+### Deterministic: an ordinary function
+
+For a task that asks the candidate to reply with exactly `APPLE`:
+
+```ts
+// evals/exact-answer/judge.ts
+import type { JudgeContext } from "@hona/openeval";
+
+export default ({ response }: JudgeContext) => ({
+  scores: { correct_answer: response.text === "APPLE" },
+});
+```
+
+Booleans become 0 or 1. Numeric scores can be any finite value from 0 to 1; null
+is unresolved. Other JSON is retained as author-defined data. Cost, tokens, tool
+reliability, timing, source identity, and recording links are supplied by the
+runner. Code-only benchmarks do not need a judge model.
+
+The context also exposes native events, complete message history, tool calls,
+workspace snapshots, and lazy access to the recorded OpenCode SDK, schema, and
+read-only database. See [code judges and data access](https://openev.al/docs/code-judges/).
+
+### Model-based: write a rubric
+
 **`evals/ask-dialect/prompt.md`**
 
 ```md
@@ -39,12 +67,12 @@ Write a SQL query for the ten most recent orders for a customer.
 ```md
 # Requests the SQL dialect
 
-## Metric: asked_dialect — Asks for the SQL dialect
+## Criterion: asked_dialect — Asks for the SQL dialect
 
 Pass when the agent asks which database or SQL dialect is in use.
 Fail when it assumes a dialect without asking. Asking alongside a draft counts.
 
-## Metric: safe_parameters — Uses bound parameters
+## Criterion: safe_parameters — Uses bound parameters
 
 Pass when the proposed query uses a bound customer-ID parameter and explains
 how to supply its value. Fail when it interpolates customer input into SQL
@@ -60,6 +88,22 @@ or does not provide a parameterized query.
 
 → [Write good rubrics](https://openev.al/docs/rubrics/) · [Download the SQL starter](https://openev.al/starter.zip)
 
+## One vocabulary
+
+A **benchmark** contains **evals**. Each eval defines a task and a **rubric**.
+**Judges** produce **scores** for the rubric's **criteria**. Runs also record
+**metrics** such as cost, tokens, and tool reliability.
+
+- A **criterion** is a named requirement being graded, such as `safe_parameters`.
+- A **score** is awarded credit, normalized from 0 to 1, or an aggregate of it.
+- A **metric** is an observed or calculated measurement. A criterion must
+  explicitly use that measurement for it to affect the grade.
+- A **judgment** is the judge's output. **BenchmarkRun**, **EvalRun**, and
+  **JudgeRun** name recorded executions, rather than reusable definitions.
+
+See the [canonical terminology](https://openev.al/docs/terminology/) and the
+[website glossary](https://openev.al/docs/terminology/).
+
 ## Choose models. Run. Inspect.
 
 Requires **Bun 1.4.2+**, **Docker**, and connected models in **OpenCode**.
@@ -80,6 +124,9 @@ export default {
 } satisfies Benchmark;
 ```
 
+The judge model is required when any eval contains judge.md. A code-only
+benchmark can omit the judge setting, or set only judge.timeoutMs.
+
 ```sh
 bunx --bun @hona/openeval image
 bunx --bun @hona/openeval plan --only-eval ask-dialect
@@ -106,26 +153,37 @@ Models still declared in `benchmark.ts` can be added back by a later `run`.
 flowchart LR
   P["prompt.md"] --> C["Isolated candidate"] --> E["Recording"]
   J["judge.md"] --> G["Judge + citations"]
-  E --> G --> S["Metric scores"] --> V["Results viewer"]
+  T["judge.ts"] --> F["Code + recorded metrics"]
+  E --> G
+  E --> F
+  G --> S["Criterion scores"]
+  F --> S
+  S --> V["Results viewer"]
 ```
 
 ## See what earned the score
 
+![Code judgment with normalized criterion scores, original JSON, and frozen source](https://raw.githubusercontent.com/Hona/openeval/main/docs/images/code-judgment.png)
+
+*Illustrative label-reading task. The code judge ran on constructed responses.*
+
 ![Model scores, completed checks, runtime, and cost in the results viewer](https://raw.githubusercontent.com/Hona/openeval/main/docs/images/results.png)
 
 | Capability | What you get | Guide |
 | --- | --- | --- |
-| Multiple metrics | Independent decisions from one recording | [Rubrics](https://openev.al/docs/rubrics/) |
+| Multiple criteria | Independent scores from one recording | [Rubrics](https://openev.al/docs/rubrics/) |
+| Code and hybrid judges | Plain functions, booleans, fractional credit, custom JSON | [Code judges](https://openev.al/docs/code-judges/) |
 | Controlled workspaces | Readable files, pinned Git inputs, preparation | [Workspaces](https://openev.al/docs/workspaces/) |
 | Small batches | Eval, model, repetition, and cost controls | [Running](https://openev.al/docs/running/) |
 | Transparent scores | Equal eval weights; bounds for unresolved checks | [Scoring](https://openev.al/docs/scoring/) |
 | Evidence inspection | Sessions, tool results, artifacts, and citations | [Evidence](https://openev.al/docs/evidence/) |
+| Native data primitives | Full history, event traces, SDK, schema, and read-only SQL | [Recorded data](https://openev.al/docs/code-judges/#context) |
 | Rejudging | New judgments from retained, immutable recordings | [Evidence](https://openev.al/docs/evidence/#revise) |
 
 <details>
 <summary><strong>Inspect a judgment and its evidence</strong></summary>
 
-![SQL eval drilldown with individual metric decisions and evidence links](https://raw.githubusercontent.com/Hona/openeval/main/docs/images/judgment.png)
+![SQL eval drilldown with individual criterion scores and evidence links](https://raw.githubusercontent.com/Hona/openeval/main/docs/images/judgment.png)
 
 </details>
 

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@hona/openeval",
-  "version": "0.2.2",
-  "description": "Typed prompt-and-judge evaluations with isolated agents, recorded evidence, and a results viewer",
+  "version": "0.3.0",
+  "description": "Code and LLM evaluations with isolated agents, recorded OpenCode data, and a results viewer",
   "license": "MIT",
   "repository": {
     "type": "git",
@@ -31,6 +31,7 @@
     "@opencode/client": "0.0.0-beta-19296",
     "@opencode/core": "0.0.0-beta-19296",
     "@opencode/sdk": "0.0.0-beta-19296",
+    "@opencode/schema": "0.0.0-beta-19296",
     "@opencode/util": "0.0.0-beta-19296",
     "drizzle-orm": "0.45.2",
     "effect": "4.0.0-rc.112",

package/src/app/cost-plan.ts

@@ -43,6 +43,9 @@ export function estimateWork(
     (item) => item.action === "candidate" || item.action === "judge",
   );
   const items = ready.map((item) => {
+    const hasLlm = !!definition.evals.find(
+      (evalDefinition) => evalDefinition.id === item.slot.evalId,
+    )!.judge;
     const candidate =
       cost(
         evals.filter(
@@ -53,22 +56,23 @@ export function estimateWork(
       ) ??
       cost(evals.filter((run) => run.input.model === item.slot.model)) ??
       cost(evals);
-    const judge =
-      cost(
-        judges.filter(
-          (run) =>
-            run.input.model === definition.judge.model &&
-            evals.some(
-              (e) =>
-                e.id === run.input.evalRunId &&
-                e.input.evalId === item.slot.evalId,
-            ),
-        ),
-      ) ??
-      cost(
-        judges.filter((run) => run.input.model === definition.judge.model),
-      ) ??
-      cost(judges);
+    const judge = !hasLlm
+      ? 0
+      : (cost(
+          judges.filter(
+            (run) =>
+              run.input.model === definition.judge.model &&
+              evals.some(
+                (e) =>
+                  e.id === run.input.evalRunId &&
+                  e.input.evalId === item.slot.evalId,
+              ),
+          ),
+        ) ??
+        cost(
+          judges.filter((run) => run.input.model === definition.judge.model),
+        ) ??
+        cost(judges));
     return {
       slotId: item.slot.id,
       estimatedUSD:

package/src/app/eval-state.ts

@@ -5,6 +5,7 @@ export function canJudgeEval(
   run: EvalRun | undefined,
 ): run is EvalRun & { evidence: EvidenceRef } {
   return (
-    !!run?.evidence && ["completed", "stopped", "timed_out"].includes(run.state)
+    !!run?.evidence &&
+    ["completed", "stopped", "timed_out", "failed"].includes(run.state)
   );
 }

package/src/app/grade-recording.ts

@@ -0,0 +1,72 @@
+import { resolve } from "node:path";
+import type {
+  JudgeRunInput,
+  Judgment,
+  OpenCodeStreamEvent,
+  SessionArchive,
+} from "../types";
+import type { CodeJudgeExecution } from "../judge-context";
+import type { RecordingInput } from "../infra/recording";
+import { executeCodeJudge } from "../infra/judging/code";
+import { codeJudgment, combineJudgments } from "../infra/judging/code-result";
+import { executeJudge } from "../infra/judging";
+import { errorMessage } from "../infra/files";
+
+export type GradedRecording = {
+  state: "completed" | "failed" | "timed_out";
+  judgment?: Judgment;
+  session?: SessionArchive;
+  code?: CodeJudgeExecution;
+  error?: string;
+};
+
+/** Both source files contribute to one judgment over the same finalized recording. */
+export async function gradeRecording(
+  input: JudgeRunInput,
+  recording: RecordingInput,
+  directory: string,
+  onEvent: (event: OpenCodeStreamEvent) => void,
+  evaluateLlm: typeof executeJudge = executeJudge,
+): Promise<GradedRecording> {
+  let code: CodeJudgeExecution | undefined, session: SessionArchive | undefined;
+  try {
+    const judgments: Judgment[] = [];
+    if (input.code) {
+      code = await executeCodeJudge(
+        input.code,
+        recording,
+        resolve(directory, "code"),
+        input.timeoutMs,
+      );
+      if (code.state !== "completed")
+        return { state: code.state, code, error: code.error };
+      const judged = codeJudgment(code.output!);
+      for (const id of Object.keys(judged.scores))
+        if (input.criteria.some((criterion) => criterion.id === id))
+          throw new Error(
+            `Duplicate criterion ID from judge.md and judge.ts: ${id}`,
+          );
+      judgments.push(judged);
+    }
+    if (input.rubric) {
+      const graded = await evaluateLlm(input, directory, onEvent);
+      session = graded.session;
+      if (graded.result.state !== "completed" || !graded.judgment)
+        return {
+          state: "failed",
+          code,
+          session,
+          error: graded.result.error ?? "LLM judge did not submit scores",
+        };
+      judgments.push(graded.judgment);
+    }
+    return {
+      state: "completed",
+      code,
+      session,
+      judgment: combineJudgments(...judgments),
+    };
+  } catch (error) {
+    return { state: "failed", code, session, error: errorMessage(error) };
+  }
+}

package/src/app/input-fingerprints.ts

@@ -45,23 +45,37 @@ export const judgeFingerprint = (
 ) =>
   fingerprint({
     rubric: definition.evals.find((item) => item.id === evalId)!.judge,
-    agent: JUDGE_AGENT,
-    judge: definition.judge,
+    code: definition.evals.find((item) => item.id === evalId)!.code?.hash,
+    agent: definition.evals.find((item) => item.id === evalId)!.judge
+      ? JUDGE_AGENT
+      : undefined,
+    judge: definition.evals.find((item) => item.id === evalId)!.judge
+      ? definition.judge
+      : { timeoutMs: definition.judge.timeoutMs },
     protocol: JUDGE_PROTOCOL,
   });
 export const savedJudgeFingerprint = (
   input: Pick<
     JudgeRunInput,
-    "rubric" | "agent" | "protocol" | "model" | "timeoutMs" | "websearch"
+    | "rubric"
+    | "agent"
+    | "protocol"
+    | "model"
+    | "timeoutMs"
+    | "websearch"
+    | "code"
   >,
 ) =>
   fingerprint({
     rubric: input.rubric,
+    code: input.code?.hash,
     agent: input.agent,
     protocol: input.protocol,
-    judge: {
-      model: input.model,
-      timeoutMs: input.timeoutMs,
-      websearch: input.websearch,
-    },
+    judge: input.rubric
+      ? {
+          model: input.model,
+          timeoutMs: input.timeoutMs,
+          websearch: input.websearch,
+        }
+      : { timeoutMs: input.timeoutMs },
   });

package/src/app/judge-evidence.ts

@@ -1,37 +1,47 @@
 import { resolve, dirname } from "node:path";
 import type { EvidenceRef, Judge, JudgeRunInput, ToolCall } from "../types";
-import { executeJudge, judgingFingerprint } from "../infra/judging";
+import { judgingFingerprint } from "../infra/judging";
 import { EvidenceCapture } from "../infra/evidence";
-import { JUDGE_PROTOCOL, rubricMetrics } from "../judgment";
+import { JUDGE_PROTOCOL, rubricCriteria } from "../judgment";
 import { JUDGE_AGENT } from "../infra/judging/agent";
 import { writeJson } from "../infra/files";
 import { savedJudgeFingerprint } from "./input-fingerprints";
 import { mkdir, mkdtemp, rm } from "node:fs/promises";
+import { compileCodeJudge } from "../infra/judging/code-source";
+import { gradeRecording } from "./grade-recording";
 
 /** Inspect retained evidence without changing benchmark selections.
  * Useful for calibration and second opinions: https://arxiv.org/abs/2410.12784
  */
 export async function judgeEvidence(options: {
   evidence: EvidenceRef;
-  rubric: string;
-  judge: Judge;
+  rubric?: string;
+  code?: string;
+  judge?: Judge;
   directory: string;
 }) {
   const directory = resolve(options.directory);
   if (await Bun.file(resolve(directory, "input.json")).exists())
     throw new Error("Choose a new judge evidence directory");
+  if (!options.rubric?.trim() && !options.code)
+    throw new Error("Provide rubric text, a code judge path, or both");
+  if (options.rubric && !options.judge?.model)
+    throw new Error("A Markdown rubric requires judge.model");
+  const code = options.code ? await compileCodeJudge(options.code) : undefined;
   const request: Omit<JudgeRunInput, "judgeHash"> = {
     evalRunId: "retained-evidence",
     evidence: {
       ...options.evidence,
       directory: resolve(options.evidence.directory),
     },
-    rubric: options.rubric,
-    agent: JUDGE_AGENT,
-    model: options.judge.model,
-    timeoutMs: options.judge.timeoutMs ?? 600_000,
-    websearch: options.judge.websearch ?? false,
-    metrics: rubricMetrics(options.rubric),
+    rubric: options.rubric ?? "",
+    kind: code ? (options.rubric ? "hybrid" : "code") : "llm",
+    code,
+    agent: options.rubric ? JUDGE_AGENT : undefined,
+    model: options.rubric ? options.judge?.model : undefined,
+    timeoutMs: options.judge?.timeoutMs ?? 600_000,
+    websearch: options.judge?.websearch ?? false,
+    criteria: options.rubric ? rubricCriteria(options.rubric) : [],
     protocol: JUDGE_PROTOCOL,
     mode: "final" as const,
     runtimeHash: await judgingFingerprint(),
@@ -41,7 +51,12 @@ export async function judgeEvidence(options: {
     judgeHash: savedJudgeFingerprint(request),
   };
   await writeJson(resolve(directory, "input.json"), input);
-  const result = await executeJudge(input, directory, () => {});
+  const result = await gradeRecording(
+    input,
+    { evidence: input.evidence },
+    directory,
+    () => {},
+  );
   await writeJson(resolve(directory, "result.json"), result);
   return result;
 }