@leostera/evalkit 0.10.0 → 0.11.0

package/CHANGELOG.md

@@ -2,6 +2,71 @@
 
 All notable changes to `@leostera/evalkit`. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses [Semantic Versioning](https://semver.org/). Before 1.0, a minor version may include breaking changes; they are marked **Breaking**.
 
+## [Unreleased]
+
+## [0.11.0] - 2026-11-04
+
+Runs record what the agent did as an [ATXF](https://atxf.leostera.dev) transcript, an open format other tools read, stop at the limits you declare, and can run in a container of their own. Pi, Codex, opencode and Claude Code can be the agent under test as they ship.
+
+### Breaking: runs record an ATXF transcript
+
+- **Adapters write to `transcript` instead of calling `onEvent`.** `start()` receives a `TranscriptWriter`: `assistant({ content, model?, usage? })` for each model response (text, reasoning and `tool_call` blocks), `toolResult(callId, result, { name?, isError? })`, `turn('start' | 'end')`, `append(record)` for any other ATXF record, `import(transcript)` for agents whose sessions ATXF converts (Pi, Claude Code, Codex, Gemini CLI, OpenCode), and `subagent(transcript)` to keep a sub-agent's transcript with the run. Returning `{ reply, toolCalls, usage, model }` from `send()` still works.
+- **Usage is in ATXF's buckets**: `{ input, output, cacheRead?, cacheWrite?, reasoning?, total? }`. `input` no longer includes cached tokens: a provider reporting `input_tokens: 200, cache_read_input_tokens: 9000` reports `input: 200, cacheRead: 9000`. `inputTokens`, `outputTokens`, `cachedInputTokens` and `totalTokens` are gone, and so is `turnTokens()`.
+- **Scorers read `transcript` instead of `trajectory`.** `ScoringContext` has `transcript` (ATXF), `subagents` and `runEvents` (EvalKit's step, checkpoint, scorer and limit events). New helpers read it: `messages(ctx, role?)`, `toolCalls(ctx, name?)`, `lastReply(ctx)`, `usage(ctx)` and `modelContext(ctx)`. `turn.events` is `turn.records`, and a checkpoint's `matchedToolCall` is `{ messageId, id }`.
+- **Each run saves `transcript.atxf.jsonl` and `run-events.jsonl`** instead of `trajectory.jsonl`, plus `transcripts/` for sub-agents. The transcript is sealed with a hash of every record when the run ends, and a run whose transcript is not valid ATXF errors. Report schema version 7; `readRunTranscript` and `readRunSubagents` read the new files, and `readRunEvents` returns run events.
+- **The `tokens` metric leaves out cache reads**, which used to dominate it. New built-in metrics: `cached tokens`, `output tokens`, `cache hit rate` and `steps` (one per model call). Sub-agents count toward them all.
+- **Judge results keep the judge's own transcript** (`judge.transcript`) instead of `judge.events`.
+- The dashboard reads `GET /v1/runs/:id/transcript` (and `transcript.json` in a static build) instead of `/events`. Share snapshots are version 3; the share server converts version 2 runs on upload, on read and with `POST /api/v1/upgrade`.
+
+### Breaking: limits are declared and enforced
+
+- **`limits: { steps?, tokens?, time? }`** in `evalkit.config.ts` and on a scenario caps every run. A matrix sweeps one with `limits: { steps: [40, 80] }` (an axis named `limits.steps`), and `evalkit run --limit steps=40,time=10m` overrides them. Each kind resolves on its own: project, scenario, matrix cell, then `--limit`.
+- **A run past a limit ends `limited`**, a new status: EvalKit records a `limit-reached` event, aborts `context.signal`, gives the turn in progress 10 seconds, and grades what the agent left with the scorers that support partial runs. It is a failure for the exit code and in JUnit. The run page shows steps and tokens against their limits.
+- **`context.signal` and `context.limits`** are new on the run context: pass the signal to model calls and tool processes so they stop with the run.
+- **`policy.timeoutMs` is gone**; it was never enforced. Use `limits: { time }`. A scenario that still sets it fails with a message saying so.
+- **`--max-tokens`, `--turn-budget` and `--chat-timeout-ms` are gone.** They only set unenforced parameters; use `--limit`, or `--param` for a value your adapter reads.
+
+### Breaking: coding agents run as they ship
+
+- **`piAgent()`, `codexAgent()`, `opencodeAgent()` and `claudeCodeAgent()`** run those CLIs with their own tools, system prompts and context handling, one session per run. Each user message is one invocation through `context.shell`, with the message on stdin, and the CLI's own session record is imported through ATXF after each turn, so tool calls, tokens and steps are in the transcript and count toward limits, even when a run fails or is stopped.
+- **They start from a private, empty config**, in the run's private home: your extensions, skills, instructions and logins never reach the agent. Give it a provider with `configFiles` and `env`, or use your own with `config: 'user'`.
+- `model` takes a string or a function of the run's parameters, for a matrix model axis.
+- `harnessAgent(spec, options)` adds another CLI.
+- **`piAgent()` changed:** it had no tools, no session and your own config. Pass `tools: false` for a tool-less Pi, such as a judge, and `config: 'user'` for your own setup. `args` is still added to every turn.
+
+### Added: isolation and fixtures
+
+- **`context.shell`** runs a run's commands: `exec`, `spawn` for long-lived processes, and `readFile`, `writeFile` and `list` on the workspace. Adapters and scorers share it, commands stop at a limit, and everything it started is stopped when the run ends, background children included. Without isolation it runs on the host, as adapters did before.
+- **`isolation: container({ image, network })`** gives each run its own container through the `docker` CLI (Docker Desktop, OrbStack, Colima, Rancher Desktop or podman): the candidate workspace at `/workspace`, the evaluator workspace never mounted, its own process table so a `pkill` reaches only that run, and `network: 'none'` by default. A Dockerfile is built once per invocation and reused while unchanged. Containers of runs that did not pass are kept, stopped, with their workspaces, and `evalkit tidy` removes them.
+- **`isolation: sandboxExec({ allow, read, deny, env })`** confines runs on macOS without containers: no reading or writing the project, the home directory, other runs or `deny`, except read-write `allow` and read-only `read` holes; signals only within the run; and with `env`, a scrubbed environment.
+- **`fixtures`** in the config start services on the host before the first run and stop them after the last. Runs find them in `context.fixtures`; a container on `network: 'fixtures'` reaches them as `fixtures.<name>` and nothing else. Their output is attached to each run.
+- **`context.shell.home`** is the run's private directory outside the workspace (`/evalkit-home` in a container), for an agent CLI's config and sessions; it is never snapshotted.
+- The manifest records each run's isolation, and the run page shows it.
+- Dashboard runs now apply the config's `limits`, as `evalkit run` does.
+
+### Migrating from 0.10
+
+1. Run `evalkit upgrade` after updating. It converts each run's `trajectory.jsonl` into a sealed `transcript.atxf.jsonl` and `run-events.jsonl`, and lists any run it could not convert, which it leaves as it was. Until then the CLI and dashboard skip those runs.
+2. In adapters, replace `onEvent({ kind: 'message', ... })` with `transcript.assistant({ content })`, `onEvent({ kind: 'tool-call', ... })` with a `tool_call` block in that message, and `onEvent({ kind: 'tool-result', ... })` with `transcript.toolResult(id, result)`. Drop lifecycle events. Move usage onto the assistant message.
+3. Rename usage fields: `inputTokens` becomes `input` **minus cached tokens**, `cachedInputTokens` becomes `cacheRead`, `outputTokens` becomes `output`.
+4. In scorers, replace `trajectory.events` filters with `toolCalls(ctx, name)`, `messages(ctx, role)` or `lastReply(ctx)`.
+5. Replace `policy: { timeoutMs }` with `limits: { time }`, and the removed CLI flags with `--limit`.
+6. The `tokens` metric drops for runs that read from a prompt cache; compare old and new runs on `tokens` plus `cached tokens`.
+7. Redeploy a share server, then call `POST /api/v1/upgrade` to convert stored runs.
+8. `piAgent()` now runs Pi with tools and a private config: pass `config: 'user'` to keep using your Pi setup, and `tools: false` where Pi judges.
+
+### Fixed
+
+- **Commands run in their own directory.** `context.shell` sets `$PWD` to each command's working directory; programs that trust `$PWD` (opencode does) worked in the directory EvalKit was started from instead of the run's workspace.
+
+- **The hint after a run uses your package manager.** It says `pnpm run evalkit dev` in a pnpm project (or yarn, or `npm run evalkit -- dev`), from the manager that started the run or the nearest lockfile, instead of always `bun run`.
+- **Overriding a matrix axis says how to pick it.** `--param model=…` on a matrix axis now answers "model is a matrix axis: pick its value with -a model=<value> (--select), not --param". `run --help` shows running one cell.
+
+### Documentation
+
+- Confining concurrent runs: processes as well as files, since one agent's `pkill` can stop every run's shared server.
+- Graders that read live state cannot re-score saved runs; after fixing one, run the affected scenarios again.
+
 ## [0.10.0] - 2026-11-04
 
 You can see what a judge checks before a run and why it passed or failed after. A judge is now a rubric plus what it sees and how it answers: EvalKit builds the prompt, enforces the answer, and records the prompt, reply, model and cost, which a matrix axis can vary. Run pages put why a run failed beside what it did, a scorer can report several checks from one exercise, and run workspaces move out of the project.
@@ -281,7 +346,8 @@ Commit provenance and team sharing ([RFD0006](docs/rfds/RFD0006-commit-provenanc
 - **Breaking:** the npm package moved to `@leostera/evalkit`. Replace the `@leostera-js/evalkit` dependency and update imports, including `/runner`. Eval definitions and local reports are unchanged.
 - Dashboard workflow refinements.
 
-[0.10.0]: https://github.com/leostera/evalkit/compare/f06b26a...main
+[Unreleased]: https://github.com/leostera/evalkit/compare/498c5a5...main
+[0.10.0]: https://github.com/leostera/evalkit/compare/f06b26a...498c5a5
 [0.9.0]: https://github.com/leostera/evalkit/compare/2c62b85...f06b26a
 [0.8.0]: https://github.com/leostera/evalkit/compare/7e8c3e5...2c62b85
 [0.7.0]: https://github.com/leostera/evalkit/compare/1ca784e...7e8c3e5