@leostera/evalkit 0.8.0 → 0.10.0

package/CHANGELOG.md

@@ -2,6 +2,96 @@
 
 All notable changes to `@leostera/evalkit`. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses [Semantic Versioning](https://semver.org/). Before 1.0, a minor version may include breaking changes; they are marked **Breaking**.
 
+## [0.10.0] - 2026-11-04
+
+You can see what a judge checks before a run and why it passed or failed after. A judge is now a rubric plus what it sees and how it answers: EvalKit builds the prompt, enforces the answer, and records the prompt, reply, model and cost, which a matrix axis can vary. Run pages put why a run failed beside what it did, a scorer can report several checks from one exercise, and run workspaces move out of the project.
+
+### Breaking: judges name their agent and EvalKit builds the prompt
+
+- **`judge()` takes `agent`, `inputs`, `answer` and `passAt`.** A scenario's `judge:` field is gone. Each judge uses its own `agent`, or `judge` in `evalkit.config.ts` for every judge that names none. `inputs` (`{ name: (ctx) => value }`, default `conversation` and `reply`) says what the judge sees, and `answer` how it answers: `choices({ A: [label, value], ... })`, `criteria({ name: weight })`, or by default a value from 0 to 1. EvalKit writes the prompt from these and the rubric, reads the reply, and asks once more if it cannot. The judge agent receives the prompt as its message and replies in plain text; it no longer has to return a JSON verdict, except for the default answer.
+- **Judges pass at `passAt`** (default only 1), as predicates do. A judge that returned `passed` itself is no longer trusted with the pass mark.
+- **The judge agent gets the run's `parameters`**, so a matrix axis such as `judgeModel` chooses its model. It should return `{ reply, usage, model }`; the model is recorded with each verdict.
+- `EvalCatalogEntry.judgeAgent` is gone; each catalog scorer has its own `agent`, and judges add `inputs`, `answer` and `prompt` (the template).
+
+### Breaking: run workspaces live outside the project
+
+Run workspaces used to live in `_evalkit-sandbox/` inside the project, so an agent that looked around found the project's source, its `AGENTS.md` and skills, its results, and other runs' workspaces, and some read them unprompted. They now default to the system temp directory, `$TMPDIR/evalkit/<project>-<hash>/<run>/{candidate,evaluator}`, one directory per checkout so `tidy`, `dev` and the Workspaces page still find the ones a run kept. Setting `sandboxDir` inside the project still works, and `run` warns. This removes what an agent finds by accident; it is not confinement. Runs of a project still share a parent directory, and the evaluator workspace sits beside the candidate one, so an agent that goes looking can reach them. Confine the agent in its adapter, with a container or an OS sandbox.
+
+### Migrating from 0.9
+
+1. Move the scenario's `judge: myJudge` to `judge: myJudge` in `evalkit.config.ts`, or to `agent: myJudge` on each `judge()` rule. A scenario that still sets it fails with a message saying where it went.
+2. If your judge agent parsed EvalKit's old JSON prompt, make it send the message it receives to its model as it is, and return the model's reply. Return `model` and `usage` too.
+3. If a judge returned partial values and relied on its own `passed`, set `passAt` to the lowest value that should pass.
+4. If a script reads `_evalkit-sandbox/` by path, read the dashboard's Workspaces page or the run's saved artifacts instead, or set `sandboxDir: '_evalkit-sandbox'` to keep the old place (and its warning). Delete the old `_evalkit-sandbox/` and its Git ignore line.
+5. Several predicates sharing one cached exercise can become one predicate that returns `criteria`, declared with `predicate(name, fn, { criteria })`; see below.
+
+### Added
+
+- **Descriptions.** `predicate(name, fn, { description })` and `judge(name, { description })` say what a scorer checks, in the dashboard, `--dry-run` (which now lists each scenario's scorers, pass marks and judge agents) and the Judges page.
+- **Criteria: one exercise, several checks.** A predicate can return `criteria` instead of `value`, one per check: `{ name, value, weight?, explanation?, evidence? }`. The score is their weighted mean (weight 1 by default), and the run page shows the formula. `criteria()` judges answer the same way.
+  - `predicate(name, fn, { criteria: { 'cannot push': 'the share token gets 403' } })` declares the checks, so they show before a run.
+  - A declared check that the result leaves out, or one returned as `{ name, reached: false }`, is **not reached**: it counts as 0 and shows as "not reached", with the score's explanation as the one root cause. A check that wasn't declared is an error.
+  - Each criterion's evidence opens under it on the run page.
+- **A run page that answers why, in one window.** A rail lists what the run ran (status, scenario, agent, each parameter, commit, time, agent and judge tokens) above a menu of its sections: **Trajectory & scoring**, **Judging**, **Metrics**, **Workspace files**, **Logs**, **Datasets** and **Raw report**, each with its own URL (`/runs/:id/metrics`); `[` and `]` step through them. The first section puts what the agent did beside every verdict, failures first and open, under "Why it failed", "How it scored" or "Why it errored" with "2 of 3 checks did not pass"; on a narrower window they become tabs, opening on the verdicts when the run did not pass. The page is never taller than the window: each column scrolls on its own. Clicking a checkpoint in the trajectory points at its verdict, and `?verdict=` links to one.
+- **Workspace files and logs, named for what they hold.** The run page calls the snapshot of what the agent left in its workspace **Workspace files** (one browser; the separate artifacts table is gone), and the files saved on purpose with `attach()` **Logs**, at `/runs/:id/logs`; `/runs/:id/attachments` redirects there. The API keeps `attach()` and `attachments`.
+- **No title bar.** The bar at the top of every page, which repeated the section name, is gone; the breadcrumb says where you are. A shared page's version picker moved beside the "Shared results for…" line.
+- **Verdicts.** Each scorer and checkpoint opens (failed ones open by default) to the verdict in words ("answered C (0), passes at A or B → failed"), the explanation, criteria, and for a judge the agent, model, tokens, time and retry, then tabs with the prompt it saw, its reply, its conversation and the evidence as a tree or JSON.
+- **Judging cost apart from the agent's.** Results save the judge's prompt, reply, conversation, usage, model and the `passAt` applied. A **Judging** section on the run page totals judge tokens and time by judge, and the built-in `judge tokens` metric puts it on the scoreboard and in Compare.
+- **Judges page.** The list shows each judge's description, pass mark, latest model and last 10 runs. A judge's page shows its agent, what it sees, how it answers, the choice or criteria table, the rubric, the prompt EvalKit sends (as a template or as a recent run sent it), and recent verdicts with their reasons.
+- **"How it is scored"** on a scenario's page: every scorer, what it checks, its pass mark and how it did in the last 10 runs. Checkpoints in the transcript show their description and pass mark.
+- **Failure lines say why, in one line.** The CLI's failure line and JUnit messages give each failing scorer's value, pass mark and choice, which criteria failed or weren't reached, and the first line of its explanation, without terminal colours. A 300-line stderr in an explanation or error no longer floods the terminal.
+- **Attachments next to the verdict.** A score or checkpoint records the files its scorer saved with `attach()` (`attachments`), even when the scorer then failed. Its verdict links them, and a failed one also links the run's other attachments, such as the agent's transcript.
+- **Graders that read external state**, documented: reset before you exercise, or compare before and after, so the agent's own test leftovers don't decide a check.
+- [`examples/judges`](https://github.com/leostera/evalkit/tree/main/examples/judges): choices and criteria judges on a chat assistant, one predicate with declared checks (failed, and not reached) on a built service, and a judge model chosen by the matrix, all provider-free.
+
+### Fixed
+
+- A run's score on its page is rounded to two places.
+
+## [0.9.0] - 2026-11-02
+
+A run is one execution now: `repeat` starts several as a batch, and `evalkit upgrade` converts older results. Also attachments, typed checkpoint lookups, Markdown replies, one definition of `tokens`, and a share server that can live below a path on your own site and show a single repository.
+
+### Breaking: runs without trials
+
+A **run** is now one execution of a scenario: one agent session in one sandbox, scored. The level above it, a run of trials, is gone ([RFD0008](https://github.com/leostera/evalkit/blob/main/docs/rfds/RFD0008-runs-without-trials.md)). Most runs had a single trial, so their pages had one row to click through, and Runs and Trials listed the same thing twice.
+
+- **`repeat` replaces `policy.trials`.** `repeat: 3` on a scenario, `-n, --repeat 3` (was `--trials`) and `execution.repeat` (was `execution.trials`) start three ordinary runs that share a **batch**: `batch: { id, index, size }` in their manifests. A config or scenario that still sets the old option fails with a message naming the new one.
+- **Statuses.** A finished run is `passed`, `failed` (it finished, but a score or checkpoint did not pass), `errored` (it could not finish) or `cancelled`. Old `completed` became `passed` or `failed`, and old `failed` became `errored`.
+- **Report schema version 6.** A run's directory holds `manifest.json`, `catalog.json`, `trajectory.jsonl`, `scoring.json`, `summary.json` and `artifacts/` directly, with no `trials/`. The summary is the old trial summary. Readers accept version 6 only; `evalkit upgrade` converts older results.
+- **API.** `runEval` resolves to `RunResult[]`, one per run. `readTrialManifest`, `readTrialSummary`, `readTrialScoring` and `readTrialEvents` are `readRunManifest`, `readRunSummary`, `readRunScoring` and `readRunEvents(root, runId)`. `trialWorkspacePath` is `runWorkspacePath`, `maxTrialBytes` is `maxRunBytes`, and the schemas are `RunManifestSchema`, `RunSummarySchema`, `RunScoringSchema`, `BatchSchema` and `RunDatasetSchema`. The adapter, scoring and dataset contexts have an optional `batch` instead of `trialId` and `trialIndex`, as does a computed `user()` message. `runMatrix` resolves to `{ cells, runs, passed, failed, cancelled }` and `onResult(cell, runs)`. The `evalkit:trial:` URI kind and the `trial-started` and `trial-completed` events (now `run-started` and `run-completed`) are gone; `AggregateScoring` and `TrialResult` are removed.
+- **Counting.** The CLI summary, JUnit (one testcase per run, `#2` for a batch's second), `evalkit compare` (a cell is compared by its latest batch's pass rate), the scoreboard and Compare count runs.
+- **Dashboard.** A run's page is the old trial page: its trajectory beside what it ran (scenario, agent, parameters, commit, batch) and how it scored. The Trials page is gone; a batch shows as `2/3` beside a run's ID in the Runs list, which filters the list to it (`/runs?batch=<id>`). Run and trial links from before 0.9 still lead to the same run, or to the batch for a run of several trials.
+- **Snapshots and the share server.** `evalkit build` and `evalkit share` write snapshot format version 2 (`runs/<id>/detail.json`, `events.json`, `workspace/` and so on). The share server converts uploads from older CLIs as they arrive, and `POST /api/v1/upgrade` converts the runs it already stores.
+
+### Migrating from 0.8
+
+1. Rename `policy: { trials: n }` to `repeat: n`, `execution.trials` to `execution.repeat`, and `--trials` to `--repeat` in scripts and CI.
+2. Run `bun run evalkit upgrade` in each eval project (`--dry-run` first to see what it would change). An old run of one trial keeps its ID; an old run of several becomes runs with the trials' IDs, in a batch with the old run's ID. Either way, old run and trial links keep working. It only reads the config, is safe to interrupt and run again, and leaves runs that are still running for later.
+3. In code, handle `runEval`'s array of runs, the new statuses and the renamed readers above; `status === 'passed'` replaces `status === 'completed' && scoring.passed`.
+4. If you host a share server, redeploy it, then call `POST /api/v1/upgrade` until `remaining` is 0. Until a stored run is converted it is not shown.
+
+### Added
+
+- **Selecting a slice in one line.** `-m` and `-s` combine: `evalkit run -m models -s testing -a model=luna -a toolchain=riot` runs the testing suite's scenarios on Luna with Riot. `-s` alone narrows the config's matrix the same way, eval IDs (as arguments or `-e`) narrow further, and selections that cannot be met are errors. `--select` has a short form, `-a` (for axis), and takes comma-separated values: `-a model=luna,kimi`.
+- **Built-in metric names are typed.** `BuiltInMetricName` is `'agent time' | 'tokens' | 'tool calls'`, `MetricResult.name` suggests them, and `metric('tokens', ...)` is a type error. The docs define each one precisely.
+- `Usage.cachedInputTokens`: how many of a turn's input tokens came from a prompt cache, shown on the run page. `turnTokens(usage)` computes what a turn counts toward `tokens`.
+- **Typed checkpoint lookups.** `checkpoint(rule)` on the scoring context returns a checkpoint's latest result by the rule itself, not its name, with the predicate's evidence type: `checkpoint(builds)?.evidence?.stdout`. Renaming the check cannot break it, and a rule outside the transcript throws instead of reading as "not run".
+- **Attachments.** Adapters get `attach(name, content)` in `start()`, and scorers and checkpoints on their context, to save files such as the agent's raw log with the run's results. They survive when a passing run's sandbox is removed, show on the run page and its new **Attachments** page, and are included by `build` and `share`. Names are checked relative paths; a file over 5 MB keeps its last bytes, and a run's attachments are capped at 20 MB (`artifacts.attachments` sets both).
+- **Agent replies render as Markdown** on the run page: headings, lists, code blocks, tables, task lists and links, with single newlines kept. "Source" shows the exact text. Raw HTML is shown as text, links open in a new tab without a referrer, and images are never loaded, only linked.
+- `evalkit upgrade [--dry-run]` and `POST /api/v1/upgrade` on the share server; see above.
+- **A share server below a path, for one repository.** `basePath` (`BASE_PATH` in the Worker's `vars`, `--base-path` for `share serve`) serves everything below a path such as `/evals`, so the server can be routed into a site's own domain. `repo` (`REPO`, `--repo`) shows only that repository: pages are `/evals/` and `/evals/-/<sha>/runs/…`, there is no list of repositories, and uploads for any other repository are rejected. Set `share.url` to the full address, e.g. `https://example.com/evals`.
+
+### Changed
+
+- **`tokens` has one definition:** each turn's `inputTokens + outputTokens`, with cached input counted as input, and `totalTokens` only when a turn reports neither. Before, a reported `totalTokens` took precedence.
+- `-s` with a config matrix records its runs under that matrix's ID, as `-m` does, instead of `default`.
+- **Missing metrics are "not measured", never zero.** Runs from before a metric existed, or that recorded nothing for it, stay out of its means, and the dashboard shows how many contributed: "1 of 2 runs" on the scoreboard and in Compare cells, and "not measured" on a run page or when none did.
+
+### Fixed
+
+- **Agent-written files are never rendered.** The local dashboard served candidate workspace and dataset files with a type from their extension, so an `.html` file an agent wrote would open as a page on the dashboard's origin. They are now served as bytes with `nosniff` and a sandboxing CSP, as the share server already did.
+
 ## [0.8.0] - 2026-10-31
 
 Shell commands in trajectories read like a terminal, and each scenario can choose what its snapshots leave out.
@@ -9,7 +99,6 @@ Shell commands in trajectories read like a terminal, and each scenario can choos
 ### Added
 
 - **Per-scenario snapshot ignores.** `defineEval({ workspace: { ignore: [...] } })` adds gitignore-style patterns after the project's `artifacts.exclude`, so each scenario decides what in its workspace is noise. Later patterns win, and `!pattern` keeps a path even inside an ignored directory: `['!_build/doc/**']` saves generated docs and drops the rest of `_build`. Ignored directories are walked only when a `!` pattern could keep something in them. Each trial manifest records the effective list as `workspaceIgnore`.
-
 - **Shell commands as terminals.** On the trial page, a tool call that ran a shell command shows as a small terminal: the command, its output (the last lines first, where errors are, with earlier ones a click away) and stderr in red, with the exit code beside the tool name. Any tool whose arguments have a `command`, `cmd` or `script` qualifies.
 
 ## [0.7.0] - 2026-10-31
@@ -192,7 +281,9 @@ Commit provenance and team sharing ([RFD0006](docs/rfds/RFD0006-commit-provenanc
 - **Breaking:** the npm package moved to `@leostera/evalkit`. Replace the `@leostera-js/evalkit` dependency and update imports, including `/runner`. Eval definitions and local reports are unchanged.
 - Dashboard workflow refinements.
 
-[0.8.0]: https://github.com/leostera/evalkit/compare/7e8c3e5...main
+[0.10.0]: https://github.com/leostera/evalkit/compare/f06b26a...main
+[0.9.0]: https://github.com/leostera/evalkit/compare/2c62b85...f06b26a
+[0.8.0]: https://github.com/leostera/evalkit/compare/7e8c3e5...2c62b85
 [0.7.0]: https://github.com/leostera/evalkit/compare/1ca784e...7e8c3e5
 [0.6.1]: https://github.com/leostera/evalkit/compare/91c2c86...1ca784e
 [0.6.0]: https://github.com/leostera/evalkit/compare/99008d5...91c2c86

package/README.md

@@ -5,9 +5,9 @@
 EvalKit lets you write agent evaluations as TypeScript: give an Agent Under Test (AUT) a scenario, inspect what it actually said and did, and score the result. Write each scenario as a small file, share agents, judges and datasets between them, and keep them in Git alongside your code; run them locally and inspect the evidence in a dashboard or ordinary report files.
 
 - **Test the journey, not just the final answer.** Put `predicate(...)`, `judge(...)`, or `expectToolCall(...)` between user turns to check an intermediate reply, observed tool call, or live file state before continuing. `policy.failfast` can skip later turns after a failed checkpoint.
-- **Choose how to score.** Run deterministic predicates without model credentials, or provide an independent judge agent for rubric-based verdicts. A judge can have its own tools; its events stay separate from the AUT trajectory. The same rule can run inline and in final scoring.
-- **Make failures inspectable.** Reports record turns, checkpoints, skipped steps, final scores, and errors. Trials have isolated candidate and evaluator workspaces, so private expected answers need not be shown to the AUT.
-- **Repeat and compare.** Run multiple trials or define a matrix of parameters. An adapter must explicitly apply those parameters to its AUT; EvalKit does not silently switch models.
+- **Choose how to score.** Run deterministic predicates without model credentials, or write a rubric and let a judge agent grade it: EvalKit builds the prompt, enforces the answer format (a value, `choices` or weighted `criteria`) and records the prompt, reply, model and judging cost apart from the agent's. A matrix axis can pick the judge's model. The same rule can run inline and in final scoring.
+- **Make failures inspectable.** Reports record turns, checkpoints, skipped steps, final scores, and errors. Runs have isolated candidate and evaluator workspaces, so private expected answers need not be shown to the AUT.
+- **Repeat and compare.** Run multiple runs or define a matrix of parameters. An adapter must explicitly apply those parameters to its AUT; EvalKit does not silently switch models.
 - **Gate merge requests.** `--junit` gives CI a test report, and `evalkit compare` shows what regressed or improved against the base branch and can fail the pipeline.
 
 ## Try it locally
@@ -42,13 +42,14 @@ The generator creates `scenarios/`, `agents/`, `datasets/`, and `judges/`, plus
 
 In-place setup preserves existing dependencies, scripts, README, and scenario files; it refuses to overwrite any of the generated paths, including an existing config. The project installs `@leostera/evalkit` from npm. For more runnable examples, see [`examples/starter`](examples/starter/) and [`examples/interleaved-scenario`](examples/interleaved-scenario/).
 
-Runs create `_evalkit-results/` and `_evalkit-sandbox/` in the generated project directory. Scenario definitions can be versioned in Git; reports and workspaces remain local and are **not automatically shared**. Review them for sensitive data before sharing.
+Runs write reports to `_evalkit-results/` in the project, and run their agents in workspaces outside it (`$TMPDIR/evalkit/<project>-<hash>/`), so an agent cannot read the project's source, instructions or other runs by looking around. Scenario definitions can be versioned in Git; reports and workspaces remain local and are **not automatically shared**. Review them for sensitive data before sharing.
 
 ## Explore more
 
 - [Get started](sites/evalkit.leostera.dev/src/content/docs/index.md) and the [manual](sites/evalkit.leostera.dev/src/content/docs/manual/) explain adapters, datasets, scoring, reports, and the CLI. Upgrading from 0.5? See the [migration guide](CHANGELOG.md#migrating-from-05).
 - [Interleaved scenario](examples/interleaved-scenario/) has a passing write/revise flow, an intentional failfast failure, and a provider-free fake judge agent.
 - [Configured matrix](examples/configured-matrix/) shows a runnable parameter sweep without model credentials.
+- [Judges](examples/judges/) grades a chat assistant with A/B/C and weighted-criteria judges and checks a built service with evidence-returning predicates, a stronger judge model chosen by the matrix, all provider-free.
 - [Support suite](examples/support-suite/) is a few dozen short scenarios in three suites, sharing one agent, its judges, and static, computed and evaluator-only datasets.
 - [HACKING.md](HACKING.md) covers the repository layout, builds, and tests for contributors.