@leostera/evalkit 0.9.0 → 0.11.0

package/CHANGELOG.md

@@ -2,6 +2,117 @@
 
 All notable changes to `@leostera/evalkit`. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses [Semantic Versioning](https://semver.org/). Before 1.0, a minor version may include breaking changes; they are marked **Breaking**.
 
+## [Unreleased]
+
+## [0.11.0] - 2026-11-04
+
+Runs record what the agent did as an [ATXF](https://atxf.leostera.dev) transcript, an open format other tools read, stop at the limits you declare, and can run in a container of their own. Pi, Codex, opencode and Claude Code can be the agent under test as they ship.
+
+### Breaking: runs record an ATXF transcript
+
+- **Adapters write to `transcript` instead of calling `onEvent`.** `start()` receives a `TranscriptWriter`: `assistant({ content, model?, usage? })` for each model response (text, reasoning and `tool_call` blocks), `toolResult(callId, result, { name?, isError? })`, `turn('start' | 'end')`, `append(record)` for any other ATXF record, `import(transcript)` for agents whose sessions ATXF converts (Pi, Claude Code, Codex, Gemini CLI, OpenCode), and `subagent(transcript)` to keep a sub-agent's transcript with the run. Returning `{ reply, toolCalls, usage, model }` from `send()` still works.
+- **Usage is in ATXF's buckets**: `{ input, output, cacheRead?, cacheWrite?, reasoning?, total? }`. `input` no longer includes cached tokens: a provider reporting `input_tokens: 200, cache_read_input_tokens: 9000` reports `input: 200, cacheRead: 9000`. `inputTokens`, `outputTokens`, `cachedInputTokens` and `totalTokens` are gone, and so is `turnTokens()`.
+- **Scorers read `transcript` instead of `trajectory`.** `ScoringContext` has `transcript` (ATXF), `subagents` and `runEvents` (EvalKit's step, checkpoint, scorer and limit events). New helpers read it: `messages(ctx, role?)`, `toolCalls(ctx, name?)`, `lastReply(ctx)`, `usage(ctx)` and `modelContext(ctx)`. `turn.events` is `turn.records`, and a checkpoint's `matchedToolCall` is `{ messageId, id }`.
+- **Each run saves `transcript.atxf.jsonl` and `run-events.jsonl`** instead of `trajectory.jsonl`, plus `transcripts/` for sub-agents. The transcript is sealed with a hash of every record when the run ends, and a run whose transcript is not valid ATXF errors. Report schema version 7; `readRunTranscript` and `readRunSubagents` read the new files, and `readRunEvents` returns run events.
+- **The `tokens` metric leaves out cache reads**, which used to dominate it. New built-in metrics: `cached tokens`, `output tokens`, `cache hit rate` and `steps` (one per model call). Sub-agents count toward them all.
+- **Judge results keep the judge's own transcript** (`judge.transcript`) instead of `judge.events`.
+- The dashboard reads `GET /v1/runs/:id/transcript` (and `transcript.json` in a static build) instead of `/events`. Share snapshots are version 3; the share server converts version 2 runs on upload, on read and with `POST /api/v1/upgrade`.
+
+### Breaking: limits are declared and enforced
+
+- **`limits: { steps?, tokens?, time? }`** in `evalkit.config.ts` and on a scenario caps every run. A matrix sweeps one with `limits: { steps: [40, 80] }` (an axis named `limits.steps`), and `evalkit run --limit steps=40,time=10m` overrides them. Each kind resolves on its own: project, scenario, matrix cell, then `--limit`.
+- **A run past a limit ends `limited`**, a new status: EvalKit records a `limit-reached` event, aborts `context.signal`, gives the turn in progress 10 seconds, and grades what the agent left with the scorers that support partial runs. It is a failure for the exit code and in JUnit. The run page shows steps and tokens against their limits.
+- **`context.signal` and `context.limits`** are new on the run context: pass the signal to model calls and tool processes so they stop with the run.
+- **`policy.timeoutMs` is gone**; it was never enforced. Use `limits: { time }`. A scenario that still sets it fails with a message saying so.
+- **`--max-tokens`, `--turn-budget` and `--chat-timeout-ms` are gone.** They only set unenforced parameters; use `--limit`, or `--param` for a value your adapter reads.
+
+### Breaking: coding agents run as they ship
+
+- **`piAgent()`, `codexAgent()`, `opencodeAgent()` and `claudeCodeAgent()`** run those CLIs with their own tools, system prompts and context handling, one session per run. Each user message is one invocation through `context.shell`, with the message on stdin, and the CLI's own session record is imported through ATXF after each turn, so tool calls, tokens and steps are in the transcript and count toward limits, even when a run fails or is stopped.
+- **They start from a private, empty config**, in the run's private home: your extensions, skills, instructions and logins never reach the agent. Give it a provider with `configFiles` and `env`, or use your own with `config: 'user'`.
+- `model` takes a string or a function of the run's parameters, for a matrix model axis.
+- `harnessAgent(spec, options)` adds another CLI.
+- **`piAgent()` changed:** it had no tools, no session and your own config. Pass `tools: false` for a tool-less Pi, such as a judge, and `config: 'user'` for your own setup. `args` is still added to every turn.
+
+### Added: isolation and fixtures
+
+- **`context.shell`** runs a run's commands: `exec`, `spawn` for long-lived processes, and `readFile`, `writeFile` and `list` on the workspace. Adapters and scorers share it, commands stop at a limit, and everything it started is stopped when the run ends, background children included. Without isolation it runs on the host, as adapters did before.
+- **`isolation: container({ image, network })`** gives each run its own container through the `docker` CLI (Docker Desktop, OrbStack, Colima, Rancher Desktop or podman): the candidate workspace at `/workspace`, the evaluator workspace never mounted, its own process table so a `pkill` reaches only that run, and `network: 'none'` by default. A Dockerfile is built once per invocation and reused while unchanged. Containers of runs that did not pass are kept, stopped, with their workspaces, and `evalkit tidy` removes them.
+- **`isolation: sandboxExec({ allow, read, deny, env })`** confines runs on macOS without containers: no reading or writing the project, the home directory, other runs or `deny`, except read-write `allow` and read-only `read` holes; signals only within the run; and with `env`, a scrubbed environment.
+- **`fixtures`** in the config start services on the host before the first run and stop them after the last. Runs find them in `context.fixtures`; a container on `network: 'fixtures'` reaches them as `fixtures.<name>` and nothing else. Their output is attached to each run.
+- **`context.shell.home`** is the run's private directory outside the workspace (`/evalkit-home` in a container), for an agent CLI's config and sessions; it is never snapshotted.
+- The manifest records each run's isolation, and the run page shows it.
+- Dashboard runs now apply the config's `limits`, as `evalkit run` does.
+
+### Migrating from 0.10
+
+1. Run `evalkit upgrade` after updating. It converts each run's `trajectory.jsonl` into a sealed `transcript.atxf.jsonl` and `run-events.jsonl`, and lists any run it could not convert, which it leaves as it was. Until then the CLI and dashboard skip those runs.
+2. In adapters, replace `onEvent({ kind: 'message', ... })` with `transcript.assistant({ content })`, `onEvent({ kind: 'tool-call', ... })` with a `tool_call` block in that message, and `onEvent({ kind: 'tool-result', ... })` with `transcript.toolResult(id, result)`. Drop lifecycle events. Move usage onto the assistant message.
+3. Rename usage fields: `inputTokens` becomes `input` **minus cached tokens**, `cachedInputTokens` becomes `cacheRead`, `outputTokens` becomes `output`.
+4. In scorers, replace `trajectory.events` filters with `toolCalls(ctx, name)`, `messages(ctx, role)` or `lastReply(ctx)`.
+5. Replace `policy: { timeoutMs }` with `limits: { time }`, and the removed CLI flags with `--limit`.
+6. The `tokens` metric drops for runs that read from a prompt cache; compare old and new runs on `tokens` plus `cached tokens`.
+7. Redeploy a share server, then call `POST /api/v1/upgrade` to convert stored runs.
+8. `piAgent()` now runs Pi with tools and a private config: pass `config: 'user'` to keep using your Pi setup, and `tools: false` where Pi judges.
+
+### Fixed
+
+- **Commands run in their own directory.** `context.shell` sets `$PWD` to each command's working directory; programs that trust `$PWD` (opencode does) worked in the directory EvalKit was started from instead of the run's workspace.
+
+- **The hint after a run uses your package manager.** It says `pnpm run evalkit dev` in a pnpm project (or yarn, or `npm run evalkit -- dev`), from the manager that started the run or the nearest lockfile, instead of always `bun run`.
+- **Overriding a matrix axis says how to pick it.** `--param model=…` on a matrix axis now answers "model is a matrix axis: pick its value with -a model=<value> (--select), not --param". `run --help` shows running one cell.
+
+### Documentation
+
+- Confining concurrent runs: processes as well as files, since one agent's `pkill` can stop every run's shared server.
+- Graders that read live state cannot re-score saved runs; after fixing one, run the affected scenarios again.
+
+## [0.10.0] - 2026-11-04
+
+You can see what a judge checks before a run and why it passed or failed after. A judge is now a rubric plus what it sees and how it answers: EvalKit builds the prompt, enforces the answer, and records the prompt, reply, model and cost, which a matrix axis can vary. Run pages put why a run failed beside what it did, a scorer can report several checks from one exercise, and run workspaces move out of the project.
+
+### Breaking: judges name their agent and EvalKit builds the prompt
+
+- **`judge()` takes `agent`, `inputs`, `answer` and `passAt`.** A scenario's `judge:` field is gone. Each judge uses its own `agent`, or `judge` in `evalkit.config.ts` for every judge that names none. `inputs` (`{ name: (ctx) => value }`, default `conversation` and `reply`) says what the judge sees, and `answer` how it answers: `choices({ A: [label, value], ... })`, `criteria({ name: weight })`, or by default a value from 0 to 1. EvalKit writes the prompt from these and the rubric, reads the reply, and asks once more if it cannot. The judge agent receives the prompt as its message and replies in plain text; it no longer has to return a JSON verdict, except for the default answer.
+- **Judges pass at `passAt`** (default only 1), as predicates do. A judge that returned `passed` itself is no longer trusted with the pass mark.
+- **The judge agent gets the run's `parameters`**, so a matrix axis such as `judgeModel` chooses its model. It should return `{ reply, usage, model }`; the model is recorded with each verdict.
+- `EvalCatalogEntry.judgeAgent` is gone; each catalog scorer has its own `agent`, and judges add `inputs`, `answer` and `prompt` (the template).
+
+### Breaking: run workspaces live outside the project
+
+Run workspaces used to live in `_evalkit-sandbox/` inside the project, so an agent that looked around found the project's source, its `AGENTS.md` and skills, its results, and other runs' workspaces, and some read them unprompted. They now default to the system temp directory, `$TMPDIR/evalkit/<project>-<hash>/<run>/{candidate,evaluator}`, one directory per checkout so `tidy`, `dev` and the Workspaces page still find the ones a run kept. Setting `sandboxDir` inside the project still works, and `run` warns. This removes what an agent finds by accident; it is not confinement. Runs of a project still share a parent directory, and the evaluator workspace sits beside the candidate one, so an agent that goes looking can reach them. Confine the agent in its adapter, with a container or an OS sandbox.
+
+### Migrating from 0.9
+
+1. Move the scenario's `judge: myJudge` to `judge: myJudge` in `evalkit.config.ts`, or to `agent: myJudge` on each `judge()` rule. A scenario that still sets it fails with a message saying where it went.
+2. If your judge agent parsed EvalKit's old JSON prompt, make it send the message it receives to its model as it is, and return the model's reply. Return `model` and `usage` too.
+3. If a judge returned partial values and relied on its own `passed`, set `passAt` to the lowest value that should pass.
+4. If a script reads `_evalkit-sandbox/` by path, read the dashboard's Workspaces page or the run's saved artifacts instead, or set `sandboxDir: '_evalkit-sandbox'` to keep the old place (and its warning). Delete the old `_evalkit-sandbox/` and its Git ignore line.
+5. Several predicates sharing one cached exercise can become one predicate that returns `criteria`, declared with `predicate(name, fn, { criteria })`; see below.
+
+### Added
+
+- **Descriptions.** `predicate(name, fn, { description })` and `judge(name, { description })` say what a scorer checks, in the dashboard, `--dry-run` (which now lists each scenario's scorers, pass marks and judge agents) and the Judges page.
+- **Criteria: one exercise, several checks.** A predicate can return `criteria` instead of `value`, one per check: `{ name, value, weight?, explanation?, evidence? }`. The score is their weighted mean (weight 1 by default), and the run page shows the formula. `criteria()` judges answer the same way.
+  - `predicate(name, fn, { criteria: { 'cannot push': 'the share token gets 403' } })` declares the checks, so they show before a run.
+  - A declared check that the result leaves out, or one returned as `{ name, reached: false }`, is **not reached**: it counts as 0 and shows as "not reached", with the score's explanation as the one root cause. A check that wasn't declared is an error.
+  - Each criterion's evidence opens under it on the run page.
+- **A run page that answers why, in one window.** A rail lists what the run ran (status, scenario, agent, each parameter, commit, time, agent and judge tokens) above a menu of its sections: **Trajectory & scoring**, **Judging**, **Metrics**, **Workspace files**, **Logs**, **Datasets** and **Raw report**, each with its own URL (`/runs/:id/metrics`); `[` and `]` step through them. The first section puts what the agent did beside every verdict, failures first and open, under "Why it failed", "How it scored" or "Why it errored" with "2 of 3 checks did not pass"; on a narrower window they become tabs, opening on the verdicts when the run did not pass. The page is never taller than the window: each column scrolls on its own. Clicking a checkpoint in the trajectory points at its verdict, and `?verdict=` links to one.
+- **Workspace files and logs, named for what they hold.** The run page calls the snapshot of what the agent left in its workspace **Workspace files** (one browser; the separate artifacts table is gone), and the files saved on purpose with `attach()` **Logs**, at `/runs/:id/logs`; `/runs/:id/attachments` redirects there. The API keeps `attach()` and `attachments`.
+- **No title bar.** The bar at the top of every page, which repeated the section name, is gone; the breadcrumb says where you are. A shared page's version picker moved beside the "Shared results for…" line.
+- **Verdicts.** Each scorer and checkpoint opens (failed ones open by default) to the verdict in words ("answered C (0), passes at A or B → failed"), the explanation, criteria, and for a judge the agent, model, tokens, time and retry, then tabs with the prompt it saw, its reply, its conversation and the evidence as a tree or JSON.
+- **Judging cost apart from the agent's.** Results save the judge's prompt, reply, conversation, usage, model and the `passAt` applied. A **Judging** section on the run page totals judge tokens and time by judge, and the built-in `judge tokens` metric puts it on the scoreboard and in Compare.
+- **Judges page.** The list shows each judge's description, pass mark, latest model and last 10 runs. A judge's page shows its agent, what it sees, how it answers, the choice or criteria table, the rubric, the prompt EvalKit sends (as a template or as a recent run sent it), and recent verdicts with their reasons.
+- **"How it is scored"** on a scenario's page: every scorer, what it checks, its pass mark and how it did in the last 10 runs. Checkpoints in the transcript show their description and pass mark.
+- **Failure lines say why, in one line.** The CLI's failure line and JUnit messages give each failing scorer's value, pass mark and choice, which criteria failed or weren't reached, and the first line of its explanation, without terminal colours. A 300-line stderr in an explanation or error no longer floods the terminal.
+- **Attachments next to the verdict.** A score or checkpoint records the files its scorer saved with `attach()` (`attachments`), even when the scorer then failed. Its verdict links them, and a failed one also links the run's other attachments, such as the agent's transcript.
+- **Graders that read external state**, documented: reset before you exercise, or compare before and after, so the agent's own test leftovers don't decide a check.
+- [`examples/judges`](https://github.com/leostera/evalkit/tree/main/examples/judges): choices and criteria judges on a chat assistant, one predicate with declared checks (failed, and not reached) on a built service, and a judge model chosen by the matrix, all provider-free.
+
+### Fixed
+
+- A run's score on its page is rounded to two places.
+
 ## [0.9.0] - 2026-11-02
 
 A run is one execution now: `repeat` starts several as a batch, and `evalkit upgrade` converts older results. Also attachments, typed checkpoint lookups, Markdown replies, one definition of `tokens`, and a share server that can live below a path on your own site and show a single repository.
@@ -235,7 +346,9 @@ Commit provenance and team sharing ([RFD0006](docs/rfds/RFD0006-commit-provenanc
 - **Breaking:** the npm package moved to `@leostera/evalkit`. Replace the `@leostera-js/evalkit` dependency and update imports, including `/runner`. Eval definitions and local reports are unchanged.
 - Dashboard workflow refinements.
 
-[0.9.0]: https://github.com/leostera/evalkit/compare/2c62b85...main
+[Unreleased]: https://github.com/leostera/evalkit/compare/498c5a5...main
+[0.10.0]: https://github.com/leostera/evalkit/compare/f06b26a...498c5a5
+[0.9.0]: https://github.com/leostera/evalkit/compare/2c62b85...f06b26a
 [0.8.0]: https://github.com/leostera/evalkit/compare/7e8c3e5...2c62b85
 [0.7.0]: https://github.com/leostera/evalkit/compare/1ca784e...7e8c3e5
 [0.6.1]: https://github.com/leostera/evalkit/compare/91c2c86...1ca784e

package/README.md

@@ -5,7 +5,7 @@
 EvalKit lets you write agent evaluations as TypeScript: give an Agent Under Test (AUT) a scenario, inspect what it actually said and did, and score the result. Write each scenario as a small file, share agents, judges and datasets between them, and keep them in Git alongside your code; run them locally and inspect the evidence in a dashboard or ordinary report files.
 
 - **Test the journey, not just the final answer.** Put `predicate(...)`, `judge(...)`, or `expectToolCall(...)` between user turns to check an intermediate reply, observed tool call, or live file state before continuing. `policy.failfast` can skip later turns after a failed checkpoint.
-- **Choose how to score.** Run deterministic predicates without model credentials, or provide an independent judge agent for rubric-based verdicts. A judge can have its own tools; its events stay separate from the AUT trajectory. The same rule can run inline and in final scoring.
+- **Choose how to score.** Run deterministic predicates without model credentials, or write a rubric and let a judge agent grade it: EvalKit builds the prompt, enforces the answer format (a value, `choices` or weighted `criteria`) and records the prompt, reply, model and judging cost apart from the agent's. A matrix axis can pick the judge's model. The same rule can run inline and in final scoring.
 - **Make failures inspectable.** Reports record turns, checkpoints, skipped steps, final scores, and errors. Runs have isolated candidate and evaluator workspaces, so private expected answers need not be shown to the AUT.
 - **Repeat and compare.** Run multiple runs or define a matrix of parameters. An adapter must explicitly apply those parameters to its AUT; EvalKit does not silently switch models.
 - **Gate merge requests.** `--junit` gives CI a test report, and `evalkit compare` shows what regressed or improved against the base branch and can fail the pipeline.
@@ -42,13 +42,14 @@ The generator creates `scenarios/`, `agents/`, `datasets/`, and `judges/`, plus
 
 In-place setup preserves existing dependencies, scripts, README, and scenario files; it refuses to overwrite any of the generated paths, including an existing config. The project installs `@leostera/evalkit` from npm. For more runnable examples, see [`examples/starter`](examples/starter/) and [`examples/interleaved-scenario`](examples/interleaved-scenario/).
 
-Runs create `_evalkit-results/` and `_evalkit-sandbox/` in the generated project directory. Scenario definitions can be versioned in Git; reports and workspaces remain local and are **not automatically shared**. Review them for sensitive data before sharing.
+Runs write reports to `_evalkit-results/` in the project, and run their agents in workspaces outside it (`$TMPDIR/evalkit/<project>-<hash>/`), so an agent cannot read the project's source, instructions or other runs by looking around. Scenario definitions can be versioned in Git; reports and workspaces remain local and are **not automatically shared**. Review them for sensitive data before sharing.
 
 ## Explore more
 
 - [Get started](sites/evalkit.leostera.dev/src/content/docs/index.md) and the [manual](sites/evalkit.leostera.dev/src/content/docs/manual/) explain adapters, datasets, scoring, reports, and the CLI. Upgrading from 0.5? See the [migration guide](CHANGELOG.md#migrating-from-05).
 - [Interleaved scenario](examples/interleaved-scenario/) has a passing write/revise flow, an intentional failfast failure, and a provider-free fake judge agent.
 - [Configured matrix](examples/configured-matrix/) shows a runnable parameter sweep without model credentials.
+- [Judges](examples/judges/) grades a chat assistant with A/B/C and weighted-criteria judges and checks a built service with evidence-returning predicates, a stronger judge model chosen by the matrix, all provider-free.
 - [Support suite](examples/support-suite/) is a few dozen short scenarios in three suites, sharing one agent, its judges, and static, computed and evaluator-only datasets.
 - [HACKING.md](HACKING.md) covers the repository layout, builds, and tests for contributors.