@leostera/evalkit 0.6.0 → 0.7.0

package/CHANGELOG.md

@@ -2,6 +2,50 @@
 
 All notable changes to `@leostera/evalkit`. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses [Semantic Versioning](https://semver.org/). Before 1.0, a minor version may include breaking changes; they are marked **Breaking**.
 
+## [0.7.0] - 2026-10-31
+
+Metrics, computed prompts, comparing runs by parameter, smaller results, and a clearer command line. This release has breaking changes; see [Migrating from 0.6](#migrating-from-06).
+
+### Migrating from 0.6
+
+1. **Environment variables are flags.** Replace `PORT=n evalkit dev` with `evalkit dev -p n`, `EVALKIT_NO_OPEN=1` with `--no-open`, `EVALKIT_LOG_LEVEL=info` with `--log-level info`, and `EVALKIT_SHARE_URL=…` with `share --server …` / `compare --server …` (or `share.url` in the config).
+2. **`{{randomSeed}}`** in a user message becomes a function: `user(({ seed }) => `Pick number ${seed}.`)`.
+3. **Snapshots and sandboxes.** Trial snapshots no longer include `node_modules`, `.git`, build output and the like, and `_evalkit-sandbox/` keeps only trials that did not pass. If a scorer or your tooling read those, set `artifacts.exclude` or `keepSandboxes: 'all'`.
+
+### Added
+
+- **API reference** on the website: every export with its signature, kept complete by a test. The results guide now says plainly where scores live (`scoring.json`, not the manifests).
+- **Compare by parameter.** A scenario page pivots its runs by matrix parameters: one down, optionally one across, showing the pass rate, mean score or any metric. Works in shared snapshots too.
+- **Prompts computed per trial.** `user()` takes a function of `{ parameters, data, seed, trialIndex }`, so one scenario can vary its prompt by matrix axis or candidate dataset: `user(({ parameters }) => `Set up OCaml with ${parameters.toolchain}.`)`. `data` holds only candidate datasets. The trajectory records the message each trial was sent.
+- **Metrics.** `metric(name, run, { unit, better })` in a scenario's `metrics` records a number for every trial (tokens, retries, mistakes, web searches) without affecting pass or fail. EvalKit also records **agent time**, **tokens** (when the agent reports usage) and **tool calls** for every trial. Metrics are saved in `scoring.json`, shown on trial pages, averaged on run pages, and get a column each on the scoreboard.
+- **Scoreboard means.** The scoreboard shows the mean score and each metric's mean over the last 10 runs, coloured green to red when a metric says which direction is better, and its numeric columns sort.
+- **Scorers see checkpoint results.** The scoring context has `checkpoints`: the results so far for inline checkpoints, all of them for final scorers. A checkpoint can inspect live state before the session closes and pass what it saw to final scorers in `evidence`.
+
+### Changed
+
+- **Smaller results.** Candidate snapshots leave out dependencies, build output and VCS metadata (`.git`, `node_modules`, `.venv`, `__pycache__`, `_build`, `_opam`, `target`); set `artifacts.exclude` to change the list. Sandboxes in `_evalkit-sandbox/` are kept only for trials that did not pass; set `keepSandboxes: 'all' | 'none'` to change that. `evalkit tidy` removes the sandboxes this setting would not keep, freeing space used by earlier versions.
+- `evalkit dev` and `evalkit share serve` use the next free port when their default (4317, 4318) is taken, and print the one they chose, so several projects can run at once.
+- **A clearer command line.** Every command and flag has a description in `--help`, with examples; common flags have short forms (`-c`, `-e`, `-s`, `-m`, `-n`, `-j`, `-v`, `-p`, `-o`, `-b`, `-d`); `--opt=value` works; a mistyped command suggests the right one (`Did you mean run?`); and `evalkit help <command>` works.
+- **Breaking:** settings that were environment variables are flags: `dev --port <n>` (`-p`, used exactly) replaces `PORT`, `dev --no-open` replaces `EVALKIT_NO_OPEN=1`, `--log-level <level>` on `run`, `dev` and `build` replaces `EVALKIT_LOG_LEVEL`, and `share --server <url>` / `compare --server <url>` replace `EVALKIT_SHARE_URL`. `EVALKIT_SOURCE_*` (commit provenance outside GitHub and GitLab CI) and `EVALKIT_RUNTIME` stay environment variables.
+- **Breaking:** the `{{randomSeed}}` placeholder in user messages is removed; use `user(({ seed }) => `... ${seed} ...`)`.
+
+### Fixed
+
+- A symbolic link or special file in the candidate workspace no longer fails the trial's snapshot; it is skipped, never followed.
+- `evalkit dev` honours `sandboxDir` for runs started from the dashboard.
+- `evalkit new` accepts any directory, such as `./ocaml.ai` or a nested path; the package name is made npm-safe (`ocaml-ai`).
+- `run --dry-run` lists the scenarios a matrix will run instead of `"evals": []`.
+- An empty scenario file reports `scenarios/x.eval.ts is empty` instead of a missing default export.
+
+## [0.6.1] - 2026-10-30
+
+Readable trajectories on the trial page.
+
+### Added
+
+- **Readable trajectories.** The trial page shows the trajectory as a conversation grouped by turn: the user's message, tool calls paired with their results (arguments and results expand), replies as text, and each checkpoint as one verdict with its duration. Lifecycle events are folded away; **All events** lists every raw event, each expandable to its JSON. Turns show latency and token usage when the agent reports them.
+- The runner records each user step's message on its `transcript-step-started` event (`message`), so trajectories include what the agent was asked.
+
 ## [0.6.0] - 2026-10-30
 
 Scenarios, datasets and CI reporting ([RFD0007](docs/rfds/RFD0007-scenarios-datasets-and-ci-reporting.md)). This release has breaking changes; see [Migrating from 0.5](#migrating-from-05) below.
@@ -138,10 +182,12 @@ Commit provenance and team sharing ([RFD0006](docs/rfds/RFD0006-commit-provenanc
 - **Breaking:** the npm package moved to `@leostera/evalkit`. Replace the `@leostera-js/evalkit` dependency and update imports, including `/runner`. Eval definitions and local reports are unchanged.
 - Dashboard workflow refinements.
 
-[0.6.0]: https://github.com/leostera/evalkit/compare/72148d4...main
-[0.5.0]: https://github.com/leostera/evalkit/compare/0dd8717...72148d4
-[0.4.0]: https://github.com/leostera/evalkit/compare/5e67e72...0dd8717
-[0.3.0]: https://github.com/leostera/evalkit/compare/38c34be...5e67e72
-[0.2.0]: https://github.com/leostera/evalkit/compare/3f8855a...38c34be
-[0.1.0]: https://github.com/leostera/evalkit/compare/12d5a1e...3f8855a
-[0.0.2]: https://github.com/leostera/evalkit/compare/0016e72...12d5a1e
+[0.7.0]: https://github.com/leostera/evalkit/compare/1ca784e...main
+[0.6.1]: https://github.com/leostera/evalkit/compare/91c2c86...1ca784e
+[0.6.0]: https://github.com/leostera/evalkit/compare/99008d5...91c2c86
+[0.5.0]: https://github.com/leostera/evalkit/compare/49ad689...99008d5
+[0.4.0]: https://github.com/leostera/evalkit/compare/175bfcd...49ad689
+[0.3.0]: https://github.com/leostera/evalkit/compare/d45b221...175bfcd
+[0.2.0]: https://github.com/leostera/evalkit/compare/49471c3...d45b221
+[0.1.0]: https://github.com/leostera/evalkit/compare/b03233b...49471c3
+[0.0.2]: https://github.com/leostera/evalkit/compare/f13f20b...b03233b