evals-lab 0.4.1 → 0.6.0

package/CHANGELOG.md

@@ -5,6 +5,87 @@ Newest first. Read a release's **Upgrade notes** before installing it: an
 upgrade can rewrite what the lab keeps in your data directory, and an older
 version cannot always read it back.
 
+## 0.6.0
+
+### Added
+
+- Library › Evals (was Library › Datasets) lists every eval group: its
+  Source, cases, scoring, and the pipelines that link it. A group's page
+  holds its Grades, Every item, Whole run and cases.
+- Runs › Evals links eval groups to a pipeline. Link eval group picks from
+  the Library and marks the groups made for this content. A link follows a
+  group's latest version, or is pinned to one. A pipeline's own group is
+  edited in place, and Move to Library shares it.
+- A pipeline can link several eval groups. Its pass rule says whether every
+  one must pass, or at least a number. Results, History and `evals-lab run`
+  all grade by it.
+- An Undo button in the header takes back the last 20 actions, newest
+  first, until the page reloads.
+- Choose pipeline lists the 5 used most recently, then the rest, searchable
+  and sortable.
+- A Profile, Source or Grader field opens a picker, most recent first.
+
+### Changed
+
+- Results shows a row per eval group, with each Target's share and verdict,
+  then the overall verdict. History's Outcome is the overall verdict.
+- A dialog is a draft: nothing reaches the lab until Save & Close. Cancel,
+  × and Esc discard it.
+- An unnamed target is Target A, B, C.
+- The run bar sits under the pipeline's header. It says "12 of 42" while a
+  run goes and "42 in 2m" when it is done.
+- Verdicts show as a check or a cross, in Results and History.
+- A metric's "How it counts" is Options, and Grades is Reads: Parsed reply
+  or Raw reply.
+
+### Fixed
+
+- Add profile, then ×, added a profile.
+
+## 0.5.0
+
+### Upgrade notes
+
+- **Copy your data directory before upgrading** (see "User data" in the
+  README). 0.5.0 keeps versions of each eval group and each run's verdict
+  beside what 0.4.x kept, and saves pipelines as pipeline version 13, which
+  0.4.x cannot read. To go back, reinstall 0.4.1 and restore the copy.
+- Pipelines are read as version 13 and saved as it at their next edit. An
+  eval that graded by a dataset becomes a link to that eval group, or a
+  group of the pipeline's own when it had metrics of its own.
+- Export JSON writes an eval group file, which 0.4.x refuses to import.
+  Import still reads dataset files of versions 1 to 7.
+
+### Added
+
+- `evals-lab run <bundle>` runs a pipeline's evals with no lab, for CI: the
+  pipeline's menu › Export for CI… downloads the bundle. It exits 0 when
+  every eval passes, 1 when one fails, 2 when the run is incomplete and 3
+  when it cannot run, and writes a JSON report (`--json`), JUnit XML
+  (`--junit`) and a Markdown summary (`--summary`). See "In CI" in the
+  README.
+- `evals-lab run --lab <url> --pipeline <name> --wait` runs a lab's own
+  pipeline there, so the run is in its History, and exits with its verdict.
+- Each Target profile has a slug in Setup. CI reads the profile's key from
+  `EVALSLAB_API_KEY_<SLUG>`.
+- An eval group keeps its versions, and a run keeps the version it graded
+  with: Re-run grades with the same one.
+- A run's row in History keeps its verdict.
+
+### Changed
+
+- A Target profile's key is never sent back to the page: Setup shows it as
+  Set, and typing replaces it.
+- Tables look the same on every tab, every named column sorts, and lists
+  can select many rows at once.
+- A dialog's buttons sit at its bottom right, and one that edits in place
+  closes with Close, or Save & Close once something changed.
+
+### Fixed
+
+- History skipped a run queued at the same moment as another when it loaded
+  older runs.
+
 ## 0.4.1
 
 ### Changed

package/README.md

@@ -1,15 +1,19 @@
 # evals-lab
 
-A lab in the browser for evaluating prompts and models. A pipeline takes items
-from a Source — uploaded files, a Power Automate flow's run history, or a
-prompt alone — and sends each to one or more targets, every target a model
-with its own prompt. Its evals score the replies with metrics — text and JSON
-checks, item counts, agreement with production's reply, or a grading model —
-and a dataset holds each item's expected answers. Every run is kept in
-History, and every prompt version in the Library.
+A lab for evaluating prompts and models in a friendly UI. A pipeline takes items from a Source (files, Power Automate runs, or a prompt) and sends each to one or more Targets, then runs evaluates Responses. Allows for quick iterations to compare changes in prompts and settings.
 
-An early test release. It runs on your machine and talks only to the models
-and services you connect it to.
+Model connections can be local (Ollama, llama.cpp) or online (Anthropic, OpenAI, any compatible HTTP endpoint).
+
+The pipeline can be scored against metrics. The metrics include parsing checks, item counts, prod reply verification, grading models, eval groups (datasets), and more.
+
+Runs are kept in history which can be exported.
+
+Currently in early release. Runs locally and doesn't send your data anywhere else.
+
+Example use-cases:
+- Providing images to an LLM and comparing responses for accuracy
+- Checking which model can meet evals in a Power Automate workflow
+- Determining which prompt gets the highest score in metrics
 
 ## Install and run
 
@@ -19,7 +23,7 @@ evals-lab
 ```
 
 Open <http://localhost:8080>. A new lab starts with two demo datasets and
-their pipelines; the first start takes longer while it installs them. Without
+their pipelines. Note the first start takes longer while it installs them. Without
 installing: `npx evals-lab`.
 
 | Needs | Windows | Linux |
@@ -31,10 +35,9 @@ installing: `npx evals-lab`.
 `evals-lab` names anything it cannot find. Open a new terminal after
 installing one. macOS should work and is untried.
 
-## Where your data is kept
+## User data
 
-Datasets, pipelines, Setup profiles (with their API keys), prompts, uploaded
-files and runs are kept in one directory, which upgrades leave alone:
+User data is kept in a single folder. Upgrades intend to keep user data. The folders are:
 
 | | |
 |---|---|
@@ -42,7 +45,7 @@ files and runs are kept in one directory, which upgrades leave alone:
 | Linux | `~/.local/share/evals-lab` (or `$XDG_DATA_HOME/evals-lab`) |
 | macOS | `~/Library/Application Support/evals-lab` |
 
-Delete it to start again from a new lab.
+Delete this folder to start again fresh.
 
 ## Options
 
@@ -59,24 +62,118 @@ evals-lab [--port <n>] [--host <address>] [--data-dir <dir>]
 | | `OLLAMA_URL` | `http://127.0.0.1:11434` |
 | | `PYTHON` | `py -3`, `python` or `python3`, whichever is found |
 
-`--host 0.0.0.0` makes the lab reachable from your network and is refused
-without `LAB_PASSWORD`. Never put it on the open internet: the lab holds your
-API keys and forwards requests to any address a profile names.
+`--host 0.0.0.0` makes the lab reachable from your network and needs `LAB_PASSWORD` to be set (otherwise it refuses connections). 
+
+## In CI
+
+A pipeline can run with no lab, from a repository. In the lab, the pipeline's
+menu › **Export for CI…** downloads it as a bundle: unzip it into the
+repository, then:
+
+```bash
+EVALSLAB_API_KEY_ANTHROPIC=... evals-lab run evals/
+```
+
+```text
+evals-lab run <bundle-dir | pipeline.yaml> [--items <dir>] [--json <file|->]
+              [--junit <file>] [--summary <file>] [--min-pass <0..1>] [--progress]
+```
+
+| Option | |
+|---|---|
+| `--items` | the items the pipeline's Source reads; default the bundle's `items/` |
+| `--json` | the JSON report; `-` for stdout |
+| `--junit` | JUnit XML, for a CI test panel |
+| `--summary` | the results as Markdown, added to the file: for GitHub, `"$GITHUB_STEP_SUMMARY"` |
+| `--min-pass` | an eval passes when this share of its items pass |
+| `--progress` | a line per item as it finishes |
+
+The bundle holds no key. Each Target profile's key comes from
+`EVALSLAB_API_KEY_` and its slug in Setup, in capitals with hyphens as
+underscores: a profile slugged `anthropic` reads `EVALSLAB_API_KEY_ANTHROPIC`,
+and `gpt-4o-mini` reads `EVALSLAB_API_KEY_GPT_4O_MINI`. A table of results
+goes to stderr.
+
+| Exit code | |
+|---|---|
+| `0` | every eval passed |
+| `1` | an eval failed |
+| `2` | incomplete: an item is missing, or did not run |
+| `3` | broken or refused |
+
+An item a case names that is not in the items is reported as Missing.
+Images need ImageMagick, as in the lab.
+
+### GitHub Actions
+
+```yaml
+name: Evals
+on: pull_request
+
+jobs:
+  evals:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 24
+      # Only when the items are images:
+      # - run: sudo apt-get update && sudo apt-get install -y imagemagick
+      - run: >
+          npx --yes evals-lab@0.5 run evals/
+          --junit evals.xml --json evals.json --summary "$GITHUB_STEP_SUMMARY"
+        env:
+          EVALSLAB_API_KEY_ANTHROPIC: ${{ secrets.ANTHROPIC_API_KEY }}
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: evals
+          path: evals.*
+```
+
+A failing eval fails the job, and the results show on the run's page. Other
+CI systems run the same command and read `--junit` in their test panel.
+
+### Against a running lab
+
+CI can run a lab's own pipeline there instead, so the run shows in that lab's
+History and its keys never leave it:
 
-## Models
+```bash
+LAB_PASSWORD=... evals-lab run --lab https://lab.example --pipeline "Demo 1" --wait
+```
 
-A model is a Setup profile: **Setup › Add profile**, then pick its type —
+```text
+evals-lab run --lab <url> --pipeline <name> [--wait] [--json <file|->]
+              [--junit <file>] [--summary <file>] [--min-pass <0..1>] [--progress]
+```
+
+| Option | |
+|---|---|
+| `--lab` | the lab's address; `LAB_PASSWORD` is sent when it is set |
+| `--pipeline` | the pipeline, by its name or its id |
+| `--wait` | wait for the run, and exit with the lab's verdict on it |
+
+Without `--wait` the run's id is printed once it is queued, and the exit code
+is `0`; `--json`, `--junit`, `--summary`, `--min-pass` and `--progress` need
+`--wait`. The exit codes are the ones above, and a cancelled run is `2`.
+
+## Getting started
+
+First set up a Setup profile: **Setup › Add profile**, then pick its type —
 Ollama (local or cloud), OpenAI-compatible, Anthropic, llama.cpp or an HTTP
-endpoint — and give its address, model and key. **Test connection** checks
-it. An Ollama profile left without an address uses `OLLAMA_URL`.
+endpoint — and give its address, model and key. NB: An Ollama profile without an address uses `OLLAMA_URL`.
+
+Set up your data source(s) in Library.
+
+Then create a Pipeline to Run. Changes are auto-saved. Use the "Restore" menu to go back and undo any changes you don't want to keep.
 
 ## Power Automate
 
 The lab can read a Power Automate cloud flow and its run history from your
-organisation's Microsoft 365 tenant, then grade the flow's HTTP step. It needs
-a work or school account; personal accounts (outlook.com, hotmail.com) have no
-cloud flows. A small business, a trial or a Microsoft 365 Developer Program
-tenant follows the same steps. The sign-in stays in your browser tab.
+org's Microsoft 365 tenant then grade the flow's HTTP step. It needs
+a work or school account. Personal accounts (outlook.com, hotmail.com) have no cloud flows. The sign-in data is kept in your browser to avoid it being stored elsewhere, the downside is you need to re-sign in if you close/reopen your web browser. The evals can keep running however, the sign-in is just to get the workflow data, which does persist in the evals-lab user data locally.
 
 ### Register an app
 
@@ -105,12 +202,12 @@ A tenant administrator does this once, at <https://entra.microsoft.com>:
 Or set `M365_CLIENT_ID` and `M365_TENANT` when starting the lab, for the
 whole machine; Setup then shows them read-only.
 
-## Upgrading and removing
+## Upgrading
 
 Read `CHANGELOG.md` first, in the package or on
 <https://www.npmjs.com/package/evals-lab?activeTab=code>. When its
-**Upgrade notes** say a release rewrites your data, copy the data directory
-before upgrading if you may want to go back.
+**Upgrade notes** say a release rewrites your data, backup the data directory
+before upgrading.
 
 ```bash
 npm install -g evals-lab@latest   # upgrade

package/bin/evals-lab.js

@@ -1,6 +1,8 @@
 #!/usr/bin/env node
 // The npm package's launcher: `evals-lab` starts the lab's server and says
-// where it is.
+// where it is, and `evals-lab run <bundle>` runs an Export for CI bundle with
+// no server at all, or `evals-lab run --lab <url>` a running lab's pipeline
+// (run.js).
 //
 // The lab itself is server.py, a stdlib Python server, staged beside this file
 // in ../lab by build.js. What this adds is what a lab run by someone who only
@@ -26,6 +28,8 @@ const PYTHON_FLOOR = [3, 10];
 const LOCAL_OLLAMA = "http://127.0.0.1:11434";
 
 const USAGE = `Usage: evals-lab [options]
+       evals-lab run <bundle> [options]   (evals-lab run --help)
+       evals-lab run --lab <url> --pipeline <name> [--wait] [options]
 
   --port <n>        the port to listen on. Default: $PORT, else 8080.
   --host <address>  the address to listen on. Default: $LAB_HOST, else
@@ -152,6 +156,10 @@ function fail(message) {
 }
 
 async function main() {
+  if (process.argv[2] === "run") {
+    process.exitCode = await require("./run.js").run(process.argv.slice(3), { lab: LAB });
+    return;
+  }
   let o;
   try { o = options(process.argv.slice(2)); } catch (e) { fail(`${e.message}\n\n${USAGE}`); }
   if (o.help) return console.log(USAGE);