@leostera/evalkit 0.9.0 → 0.10.0

package/CHANGELOG.md

@@ -2,6 +2,52 @@
 
 All notable changes to `@leostera/evalkit`. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses [Semantic Versioning](https://semver.org/). Before 1.0, a minor version may include breaking changes; they are marked **Breaking**.
 
+## [0.10.0] - 2026-11-04
+
+You can see what a judge checks before a run and why it passed or failed after. A judge is now a rubric plus what it sees and how it answers: EvalKit builds the prompt, enforces the answer, and records the prompt, reply, model and cost, which a matrix axis can vary. Run pages put why a run failed beside what it did, a scorer can report several checks from one exercise, and run workspaces move out of the project.
+
+### Breaking: judges name their agent and EvalKit builds the prompt
+
+- **`judge()` takes `agent`, `inputs`, `answer` and `passAt`.** A scenario's `judge:` field is gone. Each judge uses its own `agent`, or `judge` in `evalkit.config.ts` for every judge that names none. `inputs` (`{ name: (ctx) => value }`, default `conversation` and `reply`) says what the judge sees, and `answer` how it answers: `choices({ A: [label, value], ... })`, `criteria({ name: weight })`, or by default a value from 0 to 1. EvalKit writes the prompt from these and the rubric, reads the reply, and asks once more if it cannot. The judge agent receives the prompt as its message and replies in plain text; it no longer has to return a JSON verdict, except for the default answer.
+- **Judges pass at `passAt`** (default only 1), as predicates do. A judge that returned `passed` itself is no longer trusted with the pass mark.
+- **The judge agent gets the run's `parameters`**, so a matrix axis such as `judgeModel` chooses its model. It should return `{ reply, usage, model }`; the model is recorded with each verdict.
+- `EvalCatalogEntry.judgeAgent` is gone; each catalog scorer has its own `agent`, and judges add `inputs`, `answer` and `prompt` (the template).
+
+### Breaking: run workspaces live outside the project
+
+Run workspaces used to live in `_evalkit-sandbox/` inside the project, so an agent that looked around found the project's source, its `AGENTS.md` and skills, its results, and other runs' workspaces, and some read them unprompted. They now default to the system temp directory, `$TMPDIR/evalkit/<project>-<hash>/<run>/{candidate,evaluator}`, one directory per checkout so `tidy`, `dev` and the Workspaces page still find the ones a run kept. Setting `sandboxDir` inside the project still works, and `run` warns. This removes what an agent finds by accident; it is not confinement. Runs of a project still share a parent directory, and the evaluator workspace sits beside the candidate one, so an agent that goes looking can reach them. Confine the agent in its adapter, with a container or an OS sandbox.
+
+### Migrating from 0.9
+
+1. Move the scenario's `judge: myJudge` to `judge: myJudge` in `evalkit.config.ts`, or to `agent: myJudge` on each `judge()` rule. A scenario that still sets it fails with a message saying where it went.
+2. If your judge agent parsed EvalKit's old JSON prompt, make it send the message it receives to its model as it is, and return the model's reply. Return `model` and `usage` too.
+3. If a judge returned partial values and relied on its own `passed`, set `passAt` to the lowest value that should pass.
+4. If a script reads `_evalkit-sandbox/` by path, read the dashboard's Workspaces page or the run's saved artifacts instead, or set `sandboxDir: '_evalkit-sandbox'` to keep the old place (and its warning). Delete the old `_evalkit-sandbox/` and its Git ignore line.
+5. Several predicates sharing one cached exercise can become one predicate that returns `criteria`, declared with `predicate(name, fn, { criteria })`; see below.
+
+### Added
+
+- **Descriptions.** `predicate(name, fn, { description })` and `judge(name, { description })` say what a scorer checks, in the dashboard, `--dry-run` (which now lists each scenario's scorers, pass marks and judge agents) and the Judges page.
+- **Criteria: one exercise, several checks.** A predicate can return `criteria` instead of `value`, one per check: `{ name, value, weight?, explanation?, evidence? }`. The score is their weighted mean (weight 1 by default), and the run page shows the formula. `criteria()` judges answer the same way.
+  - `predicate(name, fn, { criteria: { 'cannot push': 'the share token gets 403' } })` declares the checks, so they show before a run.
+  - A declared check that the result leaves out, or one returned as `{ name, reached: false }`, is **not reached**: it counts as 0 and shows as "not reached", with the score's explanation as the one root cause. A check that wasn't declared is an error.
+  - Each criterion's evidence opens under it on the run page.
+- **A run page that answers why, in one window.** A rail lists what the run ran (status, scenario, agent, each parameter, commit, time, agent and judge tokens) above a menu of its sections: **Trajectory & scoring**, **Judging**, **Metrics**, **Workspace files**, **Logs**, **Datasets** and **Raw report**, each with its own URL (`/runs/:id/metrics`); `[` and `]` step through them. The first section puts what the agent did beside every verdict, failures first and open, under "Why it failed", "How it scored" or "Why it errored" with "2 of 3 checks did not pass"; on a narrower window they become tabs, opening on the verdicts when the run did not pass. The page is never taller than the window: each column scrolls on its own. Clicking a checkpoint in the trajectory points at its verdict, and `?verdict=` links to one.
+- **Workspace files and logs, named for what they hold.** The run page calls the snapshot of what the agent left in its workspace **Workspace files** (one browser; the separate artifacts table is gone), and the files saved on purpose with `attach()` **Logs**, at `/runs/:id/logs`; `/runs/:id/attachments` redirects there. The API keeps `attach()` and `attachments`.
+- **No title bar.** The bar at the top of every page, which repeated the section name, is gone; the breadcrumb says where you are. A shared page's version picker moved beside the "Shared results for…" line.
+- **Verdicts.** Each scorer and checkpoint opens (failed ones open by default) to the verdict in words ("answered C (0), passes at A or B → failed"), the explanation, criteria, and for a judge the agent, model, tokens, time and retry, then tabs with the prompt it saw, its reply, its conversation and the evidence as a tree or JSON.
+- **Judging cost apart from the agent's.** Results save the judge's prompt, reply, conversation, usage, model and the `passAt` applied. A **Judging** section on the run page totals judge tokens and time by judge, and the built-in `judge tokens` metric puts it on the scoreboard and in Compare.
+- **Judges page.** The list shows each judge's description, pass mark, latest model and last 10 runs. A judge's page shows its agent, what it sees, how it answers, the choice or criteria table, the rubric, the prompt EvalKit sends (as a template or as a recent run sent it), and recent verdicts with their reasons.
+- **"How it is scored"** on a scenario's page: every scorer, what it checks, its pass mark and how it did in the last 10 runs. Checkpoints in the transcript show their description and pass mark.
+- **Failure lines say why, in one line.** The CLI's failure line and JUnit messages give each failing scorer's value, pass mark and choice, which criteria failed or weren't reached, and the first line of its explanation, without terminal colours. A 300-line stderr in an explanation or error no longer floods the terminal.
+- **Attachments next to the verdict.** A score or checkpoint records the files its scorer saved with `attach()` (`attachments`), even when the scorer then failed. Its verdict links them, and a failed one also links the run's other attachments, such as the agent's transcript.
+- **Graders that read external state**, documented: reset before you exercise, or compare before and after, so the agent's own test leftovers don't decide a check.
+- [`examples/judges`](https://github.com/leostera/evalkit/tree/main/examples/judges): choices and criteria judges on a chat assistant, one predicate with declared checks (failed, and not reached) on a built service, and a judge model chosen by the matrix, all provider-free.
+
+### Fixed
+
+- A run's score on its page is rounded to two places.
+
 ## [0.9.0] - 2026-11-02
 
 A run is one execution now: `repeat` starts several as a batch, and `evalkit upgrade` converts older results. Also attachments, typed checkpoint lookups, Markdown replies, one definition of `tokens`, and a share server that can live below a path on your own site and show a single repository.
@@ -235,7 +281,8 @@ Commit provenance and team sharing ([RFD0006](docs/rfds/RFD0006-commit-provenanc
 - **Breaking:** the npm package moved to `@leostera/evalkit`. Replace the `@leostera-js/evalkit` dependency and update imports, including `/runner`. Eval definitions and local reports are unchanged.
 - Dashboard workflow refinements.
 
-[0.9.0]: https://github.com/leostera/evalkit/compare/2c62b85...main
+[0.10.0]: https://github.com/leostera/evalkit/compare/f06b26a...main
+[0.9.0]: https://github.com/leostera/evalkit/compare/2c62b85...f06b26a
 [0.8.0]: https://github.com/leostera/evalkit/compare/7e8c3e5...2c62b85
 [0.7.0]: https://github.com/leostera/evalkit/compare/1ca784e...7e8c3e5
 [0.6.1]: https://github.com/leostera/evalkit/compare/91c2c86...1ca784e

package/README.md

@@ -5,7 +5,7 @@
 EvalKit lets you write agent evaluations as TypeScript: give an Agent Under Test (AUT) a scenario, inspect what it actually said and did, and score the result. Write each scenario as a small file, share agents, judges and datasets between them, and keep them in Git alongside your code; run them locally and inspect the evidence in a dashboard or ordinary report files.
 
 - **Test the journey, not just the final answer.** Put `predicate(...)`, `judge(...)`, or `expectToolCall(...)` between user turns to check an intermediate reply, observed tool call, or live file state before continuing. `policy.failfast` can skip later turns after a failed checkpoint.
-- **Choose how to score.** Run deterministic predicates without model credentials, or provide an independent judge agent for rubric-based verdicts. A judge can have its own tools; its events stay separate from the AUT trajectory. The same rule can run inline and in final scoring.
+- **Choose how to score.** Run deterministic predicates without model credentials, or write a rubric and let a judge agent grade it: EvalKit builds the prompt, enforces the answer format (a value, `choices` or weighted `criteria`) and records the prompt, reply, model and judging cost apart from the agent's. A matrix axis can pick the judge's model. The same rule can run inline and in final scoring.
 - **Make failures inspectable.** Reports record turns, checkpoints, skipped steps, final scores, and errors. Runs have isolated candidate and evaluator workspaces, so private expected answers need not be shown to the AUT.
 - **Repeat and compare.** Run multiple runs or define a matrix of parameters. An adapter must explicitly apply those parameters to its AUT; EvalKit does not silently switch models.
 - **Gate merge requests.** `--junit` gives CI a test report, and `evalkit compare` shows what regressed or improved against the base branch and can fail the pipeline.
@@ -42,13 +42,14 @@ The generator creates `scenarios/`, `agents/`, `datasets/`, and `judges/`, plus
 
 In-place setup preserves existing dependencies, scripts, README, and scenario files; it refuses to overwrite any of the generated paths, including an existing config. The project installs `@leostera/evalkit` from npm. For more runnable examples, see [`examples/starter`](examples/starter/) and [`examples/interleaved-scenario`](examples/interleaved-scenario/).
 
-Runs create `_evalkit-results/` and `_evalkit-sandbox/` in the generated project directory. Scenario definitions can be versioned in Git; reports and workspaces remain local and are **not automatically shared**. Review them for sensitive data before sharing.
+Runs write reports to `_evalkit-results/` in the project, and run their agents in workspaces outside it (`$TMPDIR/evalkit/<project>-<hash>/`), so an agent cannot read the project's source, instructions or other runs by looking around. Scenario definitions can be versioned in Git; reports and workspaces remain local and are **not automatically shared**. Review them for sensitive data before sharing.
 
 ## Explore more
 
 - [Get started](sites/evalkit.leostera.dev/src/content/docs/index.md) and the [manual](sites/evalkit.leostera.dev/src/content/docs/manual/) explain adapters, datasets, scoring, reports, and the CLI. Upgrading from 0.5? See the [migration guide](CHANGELOG.md#migrating-from-05).
 - [Interleaved scenario](examples/interleaved-scenario/) has a passing write/revise flow, an intentional failfast failure, and a provider-free fake judge agent.
 - [Configured matrix](examples/configured-matrix/) shows a runnable parameter sweep without model credentials.
+- [Judges](examples/judges/) grades a chat assistant with A/B/C and weighted-criteria judges and checks a built service with evidence-returning predicates, a stronger judge model chosen by the matrix, all provider-free.
 - [Support suite](examples/support-suite/) is a few dozen short scenarios in three suites, sharing one agent, its judges, and static, computed and evaluator-only datasets.
 - [HACKING.md](HACKING.md) covers the repository layout, builds, and tests for contributors.
 

package/dist/cli.mjs

@@ -1,12 +1,14 @@
 #!/bin/sh
 ':' //; exec "${EVALKIT_RUNTIME:-$(command -v bun || command -v node)}" "$0" "$@"
-import { H as canonicalParameters, V as defineEvalMatrix, a as authoringId, c as defineSuite, i as agentRuntimes, m as registerEvals, p as recordError } from "./src-BrfIXfO2.mjs";
-import { a as readRunEvents, c as readRunSummary, d as staticDatasetFile, f as staticDatasetFiles, i as ownerIsGone, l as localReportStore, n as upgradeReports, o as readRunManifest, p as globMatcher, r as runMatrix, t as runEval, u as runWorkspacePath } from "./src-CcdLZqFk.mjs";
-import { f as encodeBundle, i as normalizeBasePath, l as dashboardIndex, s as repoKey } from "./index-db-BbVblaHz.mjs";
-import { t as createDirectoryShareApp } from "./src-Bn6fnZnF.mjs";
+import { H as defineEvalMatrix, U as canonicalParameters, h as registerEvals, i as agentRuntimes, l as defineSuite, m as recordError, o as authoringId, tt as passMark } from "./src-B14fXAGp.mjs";
+import { a as readRunEvents, c as readRunSummary, d as staticDatasetFile, f as staticDatasetFiles, i as ownerIsGone, l as localReportStore, n as upgradeReports, o as readRunManifest, p as globMatcher, r as runMatrix, t as runEval, u as runWorkspacePath } from "./src-BfSQKOJP.mjs";
+import { f as encodeBundle, i as normalizeBasePath, l as dashboardIndex, s as repoKey } from "./index-db-B1FeXzmS.mjs";
+import { t as createDirectoryShareApp } from "./src-Bccv9igQ.mjs";
 import { Effect, LogLevel, Logger } from "effect";
+import { tmpdir } from "node:os";
 import { cp, lstat, mkdir, readFile, readdir, realpath, rm, stat, writeFile } from "node:fs/promises";
-import path, { basename, dirname, extname, join, relative, resolve, sep } from "node:path";
+import path, { basename, dirname, extname, isAbsolute, join, relative, resolve, sep } from "node:path";
+import { createHash } from "node:crypto";
 import fs, { existsSync } from "node:fs";
 import { fileURLToPath, pathToFileURL } from "node:url";
 import { Hono } from "hono";
@@ -166,6 +168,19 @@ function shortSource(source) {
 //#endregion
 //#region ../cli/src/dashboard-reports.ts
 const UUID_PATTERN = /^[0-9a-f]{8}-[0-9a-f]{4}-[1-8][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i;
+const ANSI = /\u001b\[[0-9;?]*[ -/]*[@-~]|\u001b\][^\u0007]*(?:\u0007|\u001b\\)|\u001b[@-Z\\-_]/g;
+const CONTROL = /[\u0000-\u0008\u000b-\u001f\u007f]/g;
+/**
+* The first non-empty line of a message, without terminal colours, cut to
+* `max` characters: a scorer may explain itself with 300 lines of stderr,
+* and a summary line must stay one line. The run page has the rest.
+*/
+function firstLine(text, max = 100) {
+	const lines = (text ?? "").replace(ANSI, "").split(/\r?\n/).map((line) => line.replace(CONTROL, "").trim()).filter(Boolean);
+	const line = lines[0] ?? "";
+	const more = lines.length > 1 ? " …" : "";
+	return line.length > max ? `${line.slice(0, max - 1)}…` : `${line}${more}`;
+}
 /** Written into a run directory by `evalkit tidy`; deleting it restores the run. */
 const ARCHIVE_MARKER = "archived.json";
 function isMissing(error) {
@@ -266,7 +281,12 @@ function createDashboardReportReader(root) {
 				name: result.name,
 				kind: result.kind,
 				...result.skipped ? { skipped: true } : {},
-				...result.value === void 0 || result.skipped ? {} : { value: result.value }
+				...result.value === void 0 || result.skipped ? {} : { value: result.value },
+				...result.passed === void 0 || result.skipped ? {} : { passed: result.passed },
+				...result.error ? { errored: true } : {},
+				...result.choice ? { choice: result.choice } : {},
+				...result.explanation ? { explanation: firstLine(result.explanation, 200) } : {},
+				...result.judge?.model ? { model: result.judge.model } : {}
 			})) } : {},
 			...scoring?.metrics?.length ? { metrics: scoring.metrics.map((metric) => ({
 				name: metric.name,
@@ -456,13 +476,22 @@ function staticRoutes(ids) {
 		...under("library/scenarios", ids.scenarios),
 		...under("library/agents", ids.agents),
 		...under("library/datasets", ids.datasets),
-		...ids.runs.filter(safe).flatMap((id) => [
-			`runs/${id}`,
-			`runs/${id}/workspace`,
-			`runs/${id}/attachments`
-		])
+		...ids.runs.filter(safe).flatMap((id) => [`runs/${id}`, ...RUN_SECTIONS.map((section) => `runs/${id}/${section}`)])
 	];
 }
+/**
+* A run's sections, each its own page; `attachments` is the logs' URL from
+* before 0.10, kept so old links redirect.
+*/
+const RUN_SECTIONS = [
+	"judging",
+	"metrics",
+	"workspace",
+	"logs",
+	"datasets",
+	"raw",
+	"attachments"
+];
 /** `runs/abc.html` is served at `<mount>/runs/abc`; its base is one level up. */
 function baseFor(route) {
 	const depth = route.split("/").length - 1;
@@ -843,6 +872,7 @@ async function loadConfig(cwd, configPath) {
 	const config = file ? (await importProjectModule(file)).default : {};
 	if (!config || typeof config !== "object" || Array.isArray(config)) throw new Error("Config must default-export defineConfig({...})");
 	if (config.execution && "trials" in config.execution) throw new Error("execution.trials was renamed to execution.repeat in EvalKit 0.9");
+	if (config.judge !== void 0 && typeof config.judge?.start !== "function") throw new Error("judge in the config must be an agent made with defineAgent()");
 	for (const [key, value] of Object.entries(config.execution ?? {})) if (value !== void 0 && (!Number.isSafeInteger(value) || value < 1)) throw new Error(`execution.${key} must be a positive integer`);
 	return {
 		root,
@@ -880,6 +910,17 @@ async function loadProject(cwd, configPath) {
 			matrices: [...registry.matrices, matrix]
 		};
 	}
+	const judge = config.judge;
+	if (judge) {
+		const catalog = registry.catalog;
+		registry = {
+			...registry,
+			catalog: (options) => catalog({
+				judge,
+				...options
+			})
+		};
+	}
 	return {
 		root,
 		config,
@@ -887,6 +928,34 @@ async function loadProject(cwd, configPath) {
 		loadErrors
 	};
 }
+/**
+* Where a project's run workspaces live. By default outside the project, in
+* the system temp directory (one directory per project, so `tidy` and `dev`
+* find the ones it kept): an agent that looks around its workspace finds no
+* project source, no `AGENTS.md` or skills above it, and no results.
+* `sandboxDir` overrides it; `inside` says when that puts them back in the
+* project, which `run` warns about.
+*/
+function sandboxLocation(project) {
+	if (project.config.sandboxDir === void 0) return {
+		path: defaultSandboxDir(project.root),
+		inside: false
+	};
+	const path = resolve(project.root, project.config.sandboxDir);
+	const fromRoot = relative(project.root, path);
+	return {
+		path,
+		inside: !fromRoot.startsWith("..") && !isAbsolute(fromRoot)
+	};
+}
+/** `<tmp>/evalkit/<project name>-<hash of its path>`: stable per checkout. */
+function defaultSandboxDir(root) {
+	const hash = createHash("sha256").update(root).digest("hex").slice(0, 10);
+	const name = basename(root).replace(/[^A-Za-z0-9._-]/g, "-") || "project";
+	return join(tmpdir(), "evalkit", `${name}-${hash}`);
+}
+/** The warning `run` prints when workspaces sit inside the project. */
+const SANDBOX_INSIDE_WARNING = "sandboxDir is inside the project: agents can read its source, its AGENTS.md and skills, its results and other runs’ workspaces from there. Leave sandboxDir unset to keep workspaces in the system temp directory.";
 //#endregion
 //#region ../cli/src/terminal.ts
 /** Human CLI output. Colors and live progress only appear on an interactive terminal. */
@@ -1011,28 +1080,47 @@ function runOutcome(result) {
 	if (result.status === "cancelled") return "cancelled";
 	return result.status === "passed" ? "passed" : "failed";
 }
-function passAtFor(definition, name) {
-	const rule = [...definition.transcript, ...definition.scoring].find((step) => step.kind === "predicate" && step.name === name);
-	return rule?.kind === "predicate" ? rule.passAt : void 0;
-}
 function value(score) {
 	return score.value === void 0 ? "" : score.value.toFixed(1);
 }
+/**
+* One failed verdict in words: `history 0.0 (passes at 1): got 2 entries`,
+* or for a choices judge `Correctness C (passes at A or B): wrong product`.
+*/
+function verdictReason(definition, verdict) {
+	const rule = [...definition.transcript, ...definition.scoring].find((step) => step.kind === "judge" && step.name === verdict.name);
+	const answer = rule?.kind === "judge" ? rule.answer : void 0;
+	const mark = verdict.passAt === void 0 ? "" : answer?.kind === "choices" ? ` (passes at ${passMark(verdict.passAt, answer)})` : ` (passes at ${verdict.passAt})`;
+	const why = [criteriaReason(verdict.criteria), firstLine(verdict.explanation)].filter(Boolean).join(" · ");
+	return `${verdict.name} ${verdict.choice ?? value(verdict)}${mark}${why ? `: ${why}` : ""}`;
+}
+/**
+* Which criteria let the score down, so seven checks the app never reached
+* read as one cause (`not reached: saves, history`), not seven failures.
+*/
+function criteriaReason(criteria) {
+	if (!criteria?.length) return "";
+	const failed = criteria.filter((criterion) => "value" in criterion && criterion.value < 1);
+	const unreached = criteria.filter((criterion) => !("value" in criterion));
+	return [failed.length ? `failed ${failed.map((criterion) => {
+		const why = firstLine(criterion.explanation, 60);
+		return why ? `${criterion.name} (${why})` : criterion.name;
+	}).join(", ")}` : "", unreached.length ? `not reached ${unreached.map((criterion) => criterion.name).join(", ")}` : ""].filter(Boolean).join(" · ");
+}
 function scoreReason(definition, score) {
-	if (score.error) return `${score.name} error: ${score.error.message}`;
-	const passAt = passAtFor(definition, score.name);
-	return `${score.name} ${value(score)}${passAt === void 0 ? "" : ` (passes at ${passAt})`}`;
+	if (score.error) return `${score.name} error: ${firstLine(score.error.message)}`;
+	return verdictReason(definition, score);
 }
 function checkpointReason(definition, checkpoint) {
-	if (checkpoint.status === "error") return `step ${checkpoint.step} ${checkpoint.name} error: ${checkpoint.error?.message ?? ""}`;
-	const passAt = passAtFor(definition, checkpoint.name);
-	return `step ${checkpoint.step} ${checkpoint.name} ${checkpoint.value === void 0 ? checkpoint.status : value(checkpoint)}${passAt === void 0 ? "" : ` (passes at ${passAt})`}`;
+	if (checkpoint.status === "error") return `step ${checkpoint.step} ${checkpoint.name} error: ${firstLine(checkpoint.error?.message)}`;
+	if (checkpoint.value === void 0 && !checkpoint.choice) return `step ${checkpoint.step} ${checkpoint.name} ${checkpoint.status}`;
+	return `step ${checkpoint.step} ${verdictReason(definition, checkpoint)}`;
 }
 /** What went wrong in a run. */
 function reasonsFor(definition, run) {
 	const reasons = /* @__PURE__ */ new Set();
 	const skipped = /* @__PURE__ */ new Set();
-	if (run.status === "errored" && run.error) reasons.add(`error: ${run.error.message}`);
+	if (run.status === "errored" && run.error) reasons.add(`error: ${firstLine(run.error.message, 160)}`);
 	for (const checkpoint of run.scoring?.checkpoints ?? []) if (checkpoint.status === "failed" || checkpoint.status === "error") reasons.add(checkpointReason(definition, checkpoint));
 	for (const score of run.scoring?.results ?? []) if (score.skipped) skipped.add(score.name);
 	else if (!score.passed) reasons.add(scoreReason(definition, score));
@@ -1205,7 +1293,7 @@ function allSkipped(scenario) {
 //#endregion
 //#region ../cli/src/version.ts
 /** The CLI version; kept in sync with package.json when releasing. */
-const VERSION = "0.9.0";
+const VERSION = "0.10.0";
 //#endregion
 //#region ../cli/src/run-command.ts
 /** Validates `run` flags into a selection, parameter overrides and limits. */
@@ -1332,7 +1420,8 @@ async function runProjectCommand(cwd, parsed) {
 		runs,
 		parameters: selection.parameters,
 		overrides: selection.overrides,
-		repeat: repeat ?? "scenario"
+		repeat: repeat ?? "scenario",
+		scoring: Object.fromEntries(registry.catalog({ root: project.root }).filter((entry) => matrix.evals.some((definition) => definition.id === entry.id)).map((entry) => [entry.id, plannedChecks(entry)]))
 	};
 	if (values["dry-run"]) {
 		console.log(JSON.stringify(plan, null, 2));
@@ -1341,7 +1430,9 @@ async function runProjectCommand(cwd, parsed) {
 	const sweeps = Object.keys(matrix.parameters).length > 0 || Boolean(matrix.cases?.length);
 	if (sweeps && cells > (project.config.execution?.maxCells ?? 100) && !values.all) throw new Error(`Selected ${cells} cells. Narrow the selection, inspect --dry-run, or pass --all.`);
 	const reportDir = resolve(project.root, project.config.reportDir ?? "_evalkit-results");
-	const sandboxDir = resolve(project.root, project.config.sandboxDir ?? "_evalkit-sandbox");
+	const sandbox = sandboxLocation(project);
+	const sandboxDir = sandbox.path;
+	if (sandbox.inside) console.error(`  ${yellow("!")} ${SANDBOX_INSIDE_WARNING}`);
 	const source = await detectSource(project.root, { outputDirs: [
 		reportDir,
 		sandboxDir,
@@ -1385,6 +1476,7 @@ async function runProjectCommand(cwd, parsed) {
 			keepWorkspaces: project.config.keepSandboxes,
 			artifactExclude: project.config.artifacts?.exclude,
 			attachmentLimits: project.config.artifacts?.attachments,
+			judge: project.config.judge,
 			signal: controller.signal,
 			onEvent: (run, event) => reporter.onEvent(run, event),
 			onResult: (cell, results) => {
@@ -1423,6 +1515,23 @@ async function runProjectCommand(cwd, parsed) {
 	if (controller.signal.aborted) process.exitCode = 130;
 	else if (summary.failed || broken.length) process.exitCode = 1;
 }
+/** A scenario's checkpoints and final scorers, as `--dry-run` lists them. */
+function plannedChecks(entry) {
+	return [...entry.transcript.flatMap((step, index) => step.kind === "checkpoint" ? [{
+		scorer: step.scorer,
+		at: `step ${index}`
+	}] : []), ...entry.scorers.map((scorer) => ({
+		scorer,
+		at: "final"
+	}))].map(({ scorer, at }) => ({
+		name: scorer.name,
+		kind: scorer.kind,
+		at,
+		...scorer.description ? { description: scorer.description } : {},
+		passes: passMark(scorer.passAt, scorer.answer),
+		...scorer.agent ? { agent: scorer.agent.name ?? scorer.agent.id ?? scorer.agent.kind } : {}
+	}));
+}
 //#endregion
 //#region ../cli/src/tidy.ts
 /**
@@ -1831,11 +1940,10 @@ scores, evidence, and candidate files (never evaluator workspaces): review it be
 }
 //#endregion
 //#region ../cli/src/new-project.ts
-const npmDependency = "^0.9.0";
+const npmDependency = "^0.10.0";
 const ignored = [
 	"node_modules/",
 	"_evalkit-results/",
-	"_evalkit-sandbox/",
 	"_evalkit-dashboard/",
 	"_evalkit-share/"
 ];
@@ -5318,7 +5426,7 @@ function runInput(evalIds, flags) {
 //#region ../cli/src/index.ts
 let projectRoot = process.cwd();
 let reportRoot = resolve(projectRoot, "_evalkit-results");
-let sandboxRoot = resolve(projectRoot, "_evalkit-sandbox");
+let sandboxRoot = defaultSandboxDir(projectRoot);
 let project;
 async function loadRegistry() {
 	project ??= await loadProject(projectRoot);
@@ -5345,6 +5453,7 @@ async function runEvals(options) {
 			keepWorkspaces: project?.config?.keepSandboxes,
 			artifactExclude: project?.config?.artifacts?.exclude,
 			attachmentLimits: project?.config?.artifacts?.attachments,
+			judge: project?.config?.judge,
 			suiteId: options.suiteId,
 			runtime: runtimeFor(evaluation),
 			concurrency: options.concurrency,
@@ -5418,6 +5527,7 @@ function registerMatrixRoutes(app, registry, dashboardMatrix, repeat) {
 				keepWorkspaces: project?.config?.keepSandboxes,
 				artifactExclude: project?.config?.artifacts?.exclude,
 				attachmentLimits: project?.config?.artifacts?.attachments,
+				judge: project?.config?.judge,
 				...provenance,
 				onResult: (cell, result) => runLog.cell(cell, result, performance.now() - started)
 			})))).catch((error) => runLog.failed(name, error)).finally(() => {
@@ -5550,7 +5660,7 @@ async function useProject(configPath) {
 	project = await loadProject(projectRoot, configPath);
 	projectRoot = project.root;
 	reportRoot = resolve(projectRoot, project.config.reportDir ?? "_evalkit-results");
-	sandboxRoot = resolve(projectRoot, project.config.sandboxDir ?? "_evalkit-sandbox");
+	sandboxRoot = sandboxLocation(project).path;
 }
 /**
 * `--port` is honoured exactly; without it the server takes the first free