@leostera/evalkit 0.6.1 → 0.8.0

package/CHANGELOG.md

@@ -2,6 +2,51 @@
 
 All notable changes to `@leostera/evalkit`. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses [Semantic Versioning](https://semver.org/). Before 1.0, a minor version may include breaking changes; they are marked **Breaking**.
 
+## [0.8.0] - 2026-10-31
+
+Shell commands in trajectories read like a terminal, and each scenario can choose what its snapshots leave out.
+
+### Added
+
+- **Per-scenario snapshot ignores.** `defineEval({ workspace: { ignore: [...] } })` adds gitignore-style patterns after the project's `artifacts.exclude`, so each scenario decides what in its workspace is noise. Later patterns win, and `!pattern` keeps a path even inside an ignored directory: `['!_build/doc/**']` saves generated docs and drops the rest of `_build`. Ignored directories are walked only when a `!` pattern could keep something in them. Each trial manifest records the effective list as `workspaceIgnore`.
+
+- **Shell commands as terminals.** On the trial page, a tool call that ran a shell command shows as a small terminal: the command, its output (the last lines first, where errors are, with earlier ones a click away) and stderr in red, with the exit code beside the tool name. Any tool whose arguments have a `command`, `cmd` or `script` qualifies.
+
+## [0.7.0] - 2026-10-31
+
+Metrics, computed prompts, comparing runs by parameter, smaller results, and a clearer command line. This release has breaking changes; see [Migrating from 0.6](#migrating-from-06).
+
+### Migrating from 0.6
+
+1. **Environment variables are flags.** Replace `PORT=n evalkit dev` with `evalkit dev -p n`, `EVALKIT_NO_OPEN=1` with `--no-open`, `EVALKIT_LOG_LEVEL=info` with `--log-level info`, and `EVALKIT_SHARE_URL=…` with `share --server …` / `compare --server …` (or `share.url` in the config).
+2. **`{{randomSeed}}`** in a user message becomes a function: `user(({ seed }) => `Pick number ${seed}.`)`.
+3. **Snapshots and sandboxes.** Trial snapshots no longer include `node_modules`, `.git`, build output and the like, and `_evalkit-sandbox/` keeps only trials that did not pass. If a scorer or your tooling read those, set `artifacts.exclude` or `keepSandboxes: 'all'`.
+
+### Added
+
+- **API reference** on the website: every export with its signature, kept complete by a test. The results guide now says plainly where scores live (`scoring.json`, not the manifests).
+- **Compare by parameter.** A scenario page pivots its runs by matrix parameters: one down, optionally one across, showing the pass rate, mean score or any metric. Works in shared snapshots too.
+- **Prompts computed per trial.** `user()` takes a function of `{ parameters, data, seed, trialIndex }`, so one scenario can vary its prompt by matrix axis or candidate dataset: `user(({ parameters }) => `Set up OCaml with ${parameters.toolchain}.`)`. `data` holds only candidate datasets. The trajectory records the message each trial was sent.
+- **Metrics.** `metric(name, run, { unit, better })` in a scenario's `metrics` records a number for every trial (tokens, retries, mistakes, web searches) without affecting pass or fail. EvalKit also records **agent time**, **tokens** (when the agent reports usage) and **tool calls** for every trial. Metrics are saved in `scoring.json`, shown on trial pages, averaged on run pages, and get a column each on the scoreboard.
+- **Scoreboard means.** The scoreboard shows the mean score and each metric's mean over the last 10 runs, coloured green to red when a metric says which direction is better, and its numeric columns sort.
+- **Scorers see checkpoint results.** The scoring context has `checkpoints`: the results so far for inline checkpoints, all of them for final scorers. A checkpoint can inspect live state before the session closes and pass what it saw to final scorers in `evidence`.
+
+### Changed
+
+- **Smaller results.** Candidate snapshots leave out dependencies, build output and VCS metadata (`.git`, `node_modules`, `.venv`, `__pycache__`, `_build`, `_opam`, `target`); set `artifacts.exclude` to change the list. Sandboxes in `_evalkit-sandbox/` are kept only for trials that did not pass; set `keepSandboxes: 'all' | 'none'` to change that. `evalkit tidy` removes the sandboxes this setting would not keep, freeing space used by earlier versions.
+- `evalkit dev` and `evalkit share serve` use the next free port when their default (4317, 4318) is taken, and print the one they chose, so several projects can run at once.
+- **A clearer command line.** Every command and flag has a description in `--help`, with examples; common flags have short forms (`-c`, `-e`, `-s`, `-m`, `-n`, `-j`, `-v`, `-p`, `-o`, `-b`, `-d`); `--opt=value` works; a mistyped command suggests the right one (`Did you mean run?`); and `evalkit help <command>` works.
+- **Breaking:** settings that were environment variables are flags: `dev --port <n>` (`-p`, used exactly) replaces `PORT`, `dev --no-open` replaces `EVALKIT_NO_OPEN=1`, `--log-level <level>` on `run`, `dev` and `build` replaces `EVALKIT_LOG_LEVEL`, and `share --server <url>` / `compare --server <url>` replace `EVALKIT_SHARE_URL`. `EVALKIT_SOURCE_*` (commit provenance outside GitHub and GitLab CI) and `EVALKIT_RUNTIME` stay environment variables.
+- **Breaking:** the `{{randomSeed}}` placeholder in user messages is removed; use `user(({ seed }) => `... ${seed} ...`)`.
+
+### Fixed
+
+- A symbolic link or special file in the candidate workspace no longer fails the trial's snapshot; it is skipped, never followed.
+- `evalkit dev` honours `sandboxDir` for runs started from the dashboard.
+- `evalkit new` accepts any directory, such as `./ocaml.ai` or a nested path; the package name is made npm-safe (`ocaml-ai`).
+- `run --dry-run` lists the scenarios a matrix will run instead of `"evals": []`.
+- An empty scenario file reports `scenarios/x.eval.ts is empty` instead of a missing default export.
+
 ## [0.6.1] - 2026-10-30
 
 Readable trajectories on the trial page.
@@ -147,7 +192,9 @@ Commit provenance and team sharing ([RFD0006](docs/rfds/RFD0006-commit-provenanc
 - **Breaking:** the npm package moved to `@leostera/evalkit`. Replace the `@leostera-js/evalkit` dependency and update imports, including `/runner`. Eval definitions and local reports are unchanged.
 - Dashboard workflow refinements.
 
-[0.6.1]: https://github.com/leostera/evalkit/compare/91c2c86...main
+[0.8.0]: https://github.com/leostera/evalkit/compare/7e8c3e5...main
+[0.7.0]: https://github.com/leostera/evalkit/compare/1ca784e...7e8c3e5
+[0.6.1]: https://github.com/leostera/evalkit/compare/91c2c86...1ca784e
 [0.6.0]: https://github.com/leostera/evalkit/compare/99008d5...91c2c86
 [0.5.0]: https://github.com/leostera/evalkit/compare/49ad689...99008d5
 [0.4.0]: https://github.com/leostera/evalkit/compare/175bfcd...49ad689