evals-lab 0.4.0 → 0.5.0

package/CHANGELOG.md

@@ -5,6 +5,57 @@ Newest first. Read a release's **Upgrade notes** before installing it: an
 upgrade can rewrite what the lab keeps in your data directory, and an older
 version cannot always read it back.
 
+## 0.5.0
+
+### Upgrade notes
+
+- **Copy your data directory before upgrading** (see "User data" in the
+  README). 0.5.0 keeps versions of each eval group and each run's verdict
+  beside what 0.4.x kept, and saves pipelines as pipeline version 13, which
+  0.4.x cannot read. To go back, reinstall 0.4.1 and restore the copy.
+- Pipelines are read as version 13 and saved as it at their next edit. An
+  eval that graded by a dataset becomes a link to that eval group, or a
+  group of the pipeline's own when it had metrics of its own.
+- Export JSON writes an eval group file, which 0.4.x refuses to import.
+  Import still reads dataset files of versions 1 to 7.
+
+### Added
+
+- `evals-lab run <bundle>` runs a pipeline's evals with no lab, for CI: the
+  pipeline's menu › Export for CI… downloads the bundle. It exits 0 when
+  every eval passes, 1 when one fails, 2 when the run is incomplete and 3
+  when it cannot run, and writes a JSON report (`--json`), JUnit XML
+  (`--junit`) and a Markdown summary (`--summary`). See "In CI" in the
+  README.
+- `evals-lab run --lab <url> --pipeline <name> --wait` runs a lab's own
+  pipeline there, so the run is in its History, and exits with its verdict.
+- Each Target profile has a slug in Setup. CI reads the profile's key from
+  `EVALSLAB_API_KEY_<SLUG>`.
+- An eval group keeps its versions, and a run keeps the version it graded
+  with: Re-run grades with the same one.
+- A run's row in History keeps its verdict.
+
+### Changed
+
+- A Target profile's key is never sent back to the page: Setup shows it as
+  Set, and typing replaces it.
+- Tables look the same on every tab, every named column sorts, and lists
+  can select many rows at once.
+- A dialog's buttons sit at its bottom right, and one that edits in place
+  closes with Close, or Save & Close once something changed.
+
+### Fixed
+
+- History skipped a run queued at the same moment as another when it loaded
+  older runs.
+
+## 0.4.1
+
+### Changed
+
+- The README is shorter, and describes the whole lab rather than one use
+  of it.
+
 ## 0.4.0
 
 ### Upgrade notes

package/README.md

@@ -1,10 +1,19 @@
 # evals-lab
 
-A browser bench for grading what a model answers: two wordings side by side
-over a set of files, scored against answers fixed in advance.
+A lab for evaluating prompts and models in a friendly UI. A pipeline takes items from a Source (files, Power Automate runs, or a prompt) and sends each to one or more Targets, then runs evaluates Responses. Allows for quick iterations to compare changes in prompts and settings.
 
-This is an early test release. It runs on your own machine and talks to the
-models you point it at.
+Model connections can be local (Ollama, llama.cpp) or online (Anthropic, OpenAI, any compatible HTTP endpoint).
+
+The pipeline can be scored against metrics. The metrics include parsing checks, item counts, prod reply verification, grading models, eval groups (datasets), and more.
+
+Runs are kept in history which can be exported.
+
+Currently in early release. Runs locally and doesn't send your data anywhere else.
+
+Example use-cases:
+- Providing images to an LLM and comparing responses for accuracy
+- Checking which model can meet evals in a Power Automate workflow
+- Determining which prompt gets the highest score in metrics
 
 ## Install and run
 
@@ -13,31 +22,22 @@ npm install -g evals-lab
 evals-lab
 ```
 
-Then open <http://localhost:8080>. A new lab opens on two demo datasets, each
-with its images and a starter pipeline. The first start takes a little longer
-than later ones while it installs them.
-
-Without installing: `npx evals-lab`.
+Open <http://localhost:8080>. A new lab starts with two demo datasets and
+their pipelines. Note the first start takes longer while it installs them. Without
+installing: `npx evals-lab`.
 
-## What it needs
-
-| | | Windows | Linux |
-|---|---|---|---|
-| **Node.js** 22.18 or newer | runs the lab's worker | you have it if you have `npm` | same |
-| **Python** 3.10 or newer | runs the lab's server | `winget install Python.Python.3.13` | your distribution's `python3` |
-| **ImageMagick** | sizes images before a model is sent them, and makes thumbnails | `winget install ImageMagick.ImageMagick` | your distribution's `imagemagick` |
-
-`evals-lab` says which of these it could not find. It starts without
-ImageMagick, but a run that sends images needs it. Open a new terminal after
-installing one, so it is on your `PATH`.
+| Needs | Windows | Linux |
+|---|---|---|
+| **Node.js** 22.18+ | comes with `npm` | same |
+| **Python** 3.10+ | `winget install Python.Python.3.13` | your distribution's `python3` |
+| **ImageMagick**, for runs that send images | `winget install ImageMagick.ImageMagick` | your distribution's `imagemagick` |
 
-macOS should work the same way and has not been tried.
+`evals-lab` names anything it cannot find. Open a new terminal after
+installing one. macOS should work and is untried.
 
-## Where your data is kept
+## User data
 
-Datasets, pipelines, Setup profiles (API keys among them), prompts, uploaded
-files and runs are kept on your machine, in one directory, so the lab opens
-as you left it:
+User data is kept in a single folder. Upgrades intend to keep user data. The folders are:
 
 | | |
 |---|---|
@@ -45,8 +45,7 @@ as you left it:
 | Linux | `~/.local/share/evals-lab` (or `$XDG_DATA_HOME/evals-lab`) |
 | macOS | `~/Library/Application Support/evals-lab` |
 
-Upgrading or reinstalling the package leaves it alone. Delete the directory
-to start again from a new lab.
+Delete this folder to start again fresh.
 
 ## Options
 
@@ -61,125 +60,154 @@ evals-lab [--port <n>] [--host <address>] [--data-dir <dir>]
 | `--data-dir` | `DATA_DIR` | the directory above |
 | | `LAB_PASSWORD` | none |
 | | `OLLAMA_URL` | `http://127.0.0.1:11434` |
-| | `M365_CLIENT_ID`, `M365_TENANT` | none; see Power Automate below |
 | | `PYTHON` | `py -3`, `python` or `python3`, whichever is found |
 
-`OLLAMA_URL` is only the connection a new lab starts with. Every other
-connection — an OpenAI-compatible endpoint, Anthropic, llama.cpp — is a Setup
-profile you add in the page.
+`--host 0.0.0.0` makes the lab reachable from your network and needs `LAB_PASSWORD` to be set (otherwise it refuses connections). 
+
+## In CI
+
+A pipeline can run with no lab, from a repository. In the lab, the pipeline's
+menu › **Export for CI…** downloads it as a bundle: unzip it into the
+repository, then:
 
-## Power Automate (Microsoft 365)
+```bash
+EVALSLAB_API_KEY_ANTHROPIC=... evals-lab run evals/
+```
 
-The lab can read a Power Automate cloud flow, and its run history, from your
-organisation's Microsoft 365 tenant, then grade the flow's HTTP step against
-the same model and prompt. The sign-in stays in your browser tab: the lab's
-server never receives a Microsoft token.
+```text
+evals-lab run <bundle-dir | pipeline.yaml> [--items <dir>] [--json <file|->]
+              [--junit <file>] [--summary <file>] [--min-pass <0..1>] [--progress]
+```
 
-It needs a work or school account. Microsoft retired cloud flows for personal
-accounts (outlook.com, hotmail.com) in July 2025, so a personal account cannot
-be used. For a small business, a trial or a Microsoft 365 Developer Program
-tenant, follow the same steps as a company.
+| Option | |
+|---|---|
+| `--items` | the items the pipeline's Source reads; default the bundle's `items/` |
+| `--json` | the JSON report; `-` for stdout |
+| `--junit` | JUnit XML, for a CI test panel |
+| `--summary` | the results as Markdown, added to the file: for GitHub, `"$GITHUB_STEP_SUMMARY"` |
+| `--min-pass` | an eval passes when this share of its items pass |
+| `--progress` | a line per item as it finishes |
+
+The bundle holds no key. Each Target profile's key comes from
+`EVALSLAB_API_KEY_` and its slug in Setup, in capitals with hyphens as
+underscores: a profile slugged `anthropic` reads `EVALSLAB_API_KEY_ANTHROPIC`,
+and `gpt-4o-mini` reads `EVALSLAB_API_KEY_GPT_4O_MINI`. A table of results
+goes to stderr.
+
+| Exit code | |
+|---|---|
+| `0` | every eval passed |
+| `1` | an eval failed |
+| `2` | incomplete: an item is missing, or did not run |
+| `3` | broken or refused |
+
+An item a case names that is not in the items is reported as Missing.
+Images need ImageMagick, as in the lab.
+
+### GitHub Actions
+
+```yaml
+name: Evals
+on: pull_request
+
+jobs:
+  evals:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 24
+      # Only when the items are images:
+      # - run: sudo apt-get update && sudo apt-get install -y imagemagick
+      - run: >
+          npx --yes evals-lab@0.5 run evals/
+          --junit evals.xml --json evals.json --summary "$GITHUB_STEP_SUMMARY"
+        env:
+          EVALSLAB_API_KEY_ANTHROPIC: ${{ secrets.ANTHROPIC_API_KEY }}
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: evals
+          path: evals.*
+```
 
-### Register an app in your tenant
+A failing eval fails the job, and the results show on the run's page. Other
+CI systems run the same command and read `--junit` in their test panel.
 
-An administrator of the tenant registers it once, at
-<https://entra.microsoft.com>:
+### Against a running lab
 
-1. **App registrations › New registration.**
-   - Supported account types: *Accounts in this organizational directory only*.
-   - Redirect URI: platform **Single-page application**, value `http://localhost/`.
-     Microsoft ignores the port on a `localhost` redirect, so this covers
-     `--port` too. A lab served on another address needs that address here
-     as well, over `https`.
-2. **API permissions › Add a permission › APIs my organization uses.** Search
-   for *Microsoft Flow Service* and add the delegated permission
-   **Flows.Read.All**. If the service isn't listed, open
-   <https://make.powerautomate.com> once, then search again.
-3. **Grant admin consent** for the tenant.
-4. No client secret or certificate is needed: the lab signs in as a public
-   client.
-
-### Tell the lab about it
-
-In the lab, open **Setup › Connections**, then use the Microsoft 365 row's
-menu › **Configure…**. Enter:
+CI can run a lab's own pipeline there instead, so the run shows in that lab's
+History and its keys never leave it:
 
-| | |
+```bash
+LAB_PASSWORD=... evals-lab run --lab https://lab.example --pipeline "Demo 1" --wait
+```
+
+```text
+evals-lab run --lab <url> --pipeline <name> [--wait] [--json <file|->]
+              [--junit <file>] [--summary <file>] [--min-pass <0..1>] [--progress]
+```
+
+| Option | |
 |---|---|
-| Client ID | the app's *Application (client) ID* |
-| Tenant | the *Directory (tenant) ID*, or one of the tenant's domains |
-
-Fill in Tenant for an app registered as above. Left empty, it means any
-organisation's tenant, which only a multitenant app accepts: Microsoft
-refuses the sign-in of a single-tenant app with `AADSTS50194`.
-
-Both are saved as you type, and neither is secret. **Sign in** from the same
-menu, then add a Source in the Sources tab with **Add source › Power Automate
-workflow**.
-
-To set the app for the whole machine instead, start the lab with
-`M365_CLIENT_ID` and `M365_TENANT` set. Setup then shows those values and
-can't change them.
-
-Open the lab at `localhost`, the address `evals-lab` prints. A sign-in begun
-at `127.0.0.1` reloads the page at `localhost`, the address registered above,
-and carries on from there. Anything open in the tab, such as an Add source
-dialog, does not carry over.
-
-## Google Drive
-
-**Not usable yet.** The lab can sign in to Google Drive, but nothing reads
-from Drive yet: a Source made from a Drive folder is still to come. Until it
-arrives, there is no reason to set up a Google app, and the steps below are
-provisional. They haven't been tried against a real Google project.
-
-When it arrives, the sign-in will be held by the lab's server, on your
-machine. It will reach only the files you pick: the lab asks for
-`drive.file` access, never your whole Drive.
-
-To sign in, open **Setup › Connections**, then use the Google Drive row's
-menu › **Sign in**. If the row says *Not Configured*, give the lab a Google
-app first, once:
-
-1. At <https://console.cloud.google.com>, make a project. Enable the
-   **Google Drive API** and the **Google Picker API**.
-2. **OAuth consent screen:** choose *External*. Give the app a name and your
-   email, add the scope `.../auth/drive.file`, then **Publish app**. It needs
-   no review from Google, because that scope is non-sensitive. Left in
-   *Testing*, a sign-in lapses after 7 days.
-3. **Credentials › Create credentials › OAuth client ID**, of type
-   **Desktop app**. Download its JSON file.
-4. **Credentials › Create credentials › API key.** Restrict it to the Google
-   Picker API.
-5. In the lab, use the Google Drive row's menu › **Configure…**, then
-   **Load client file…**, and choose the file from step 3. Paste the key
-   from step 4 into **API key**.
-
-A Desktop app needs no redirect registered: Google sends the browser back to
-any port on this machine. A lab you reach at its own address, from another
-machine, needs a *Web application* client instead. Choose that type in
-Configure…, then register the **Redirect URI** it shows with the client.
-
-Everything saves as you type. The client's secret is kept by the lab and
-never shown again. The window says only that one is saved.
-
-## Reaching it from another machine
-
-By default only your own machine can reach the lab. That is deliberate: the
-lab's server forwards requests to whatever address a Setup profile names, so
-anyone who can reach the lab can reach those addresses through it, and the
-lab holds your API keys.
-
-`--host 0.0.0.0` makes it reachable from your network, and is refused unless
-`LAB_PASSWORD` is set. Do not put it on the open internet.
-
-## Upgrading and removing
-
-Read `CHANGELOG.md` in the package, or on
-<https://www.npmjs.com/package/evals-lab?activeTab=code>, before upgrading.
-Its **Upgrade notes** say when a release rewrites what is in your data
-directory in a form an older version cannot read; copy the directory first
-if you may want to go back.
+| `--lab` | the lab's address; `LAB_PASSWORD` is sent when it is set |
+| `--pipeline` | the pipeline, by its name or its id |
+| `--wait` | wait for the run, and exit with the lab's verdict on it |
+
+Without `--wait` the run's id is printed once it is queued, and the exit code
+is `0`; `--json`, `--junit`, `--summary`, `--min-pass` and `--progress` need
+`--wait`. The exit codes are the ones above, and a cancelled run is `2`.
+
+## Getting started
+
+First set up a Setup profile: **Setup › Add profile**, then pick its type —
+Ollama (local or cloud), OpenAI-compatible, Anthropic, llama.cpp or an HTTP
+endpoint — and give its address, model and key. NB: An Ollama profile without an address uses `OLLAMA_URL`.
+
+Set up your data source(s) in Library.
+
+Then create a Pipeline to Run. Changes are auto-saved. Use the "Restore" menu to go back and undo any changes you don't want to keep.
+
+## Power Automate
+
+The lab can read a Power Automate cloud flow and its run history from your
+org's Microsoft 365 tenant then grade the flow's HTTP step. It needs
+a work or school account. Personal accounts (outlook.com, hotmail.com) have no cloud flows. The sign-in data is kept in your browser to avoid it being stored elsewhere, the downside is you need to re-sign in if you close/reopen your web browser. The evals can keep running however, the sign-in is just to get the workflow data, which does persist in the evals-lab user data locally.
+
+### Register an app
+
+A tenant administrator does this once, at <https://entra.microsoft.com>:
+
+1. **App registrations › New registration.**
+   - Supported account types: *Accounts in this organizational directory only*.
+   - Redirect URI: **Single-page application**, `http://localhost/`. This
+     covers any `--port`. A lab served elsewhere needs its own address here
+     too, over `https`.
+2. **API permissions › Add a permission › APIs my organization uses ›
+   Microsoft Flow Service**, delegated **Flows.Read.All**. If it isn't listed,
+   open <https://make.powerautomate.com> once and search again.
+3. **Grant admin consent.** No client secret is needed.
+
+### Connect the lab
+
+1. **Setup › Connections**, the Microsoft 365 row's menu › **Configure…**:
+   - Client ID: the app's *Application (client) ID*.
+   - Tenant: the *Directory (tenant) ID* or one of its domains. Left empty, a
+     single-tenant app's sign-in fails with `AADSTS50194`.
+2. The same menu › **Sign in**, with the lab open at `localhost`. A sign-in
+   from `127.0.0.1` reloads the page there and loses any open dialog.
+3. **Library › Sources › Add source › Power Automate workflow.**
+
+Or set `M365_CLIENT_ID` and `M365_TENANT` when starting the lab, for the
+whole machine; Setup then shows them read-only.
+
+## Upgrading
+
+Read `CHANGELOG.md` first, in the package or on
+<https://www.npmjs.com/package/evals-lab?activeTab=code>. When its
+**Upgrade notes** say a release rewrites your data, backup the data directory
+before upgrading.
 
 ```bash
 npm install -g evals-lab@latest   # upgrade
@@ -188,13 +216,9 @@ npm uninstall -g evals-lab        # remove; your data directory stays
 
 ## Licence
 
-Copyright 2026 Grainbox Limited. Licensed under the
+Copyright 2026 Grainbox Limited, under the
 [PolyForm Internal Use License 1.0.0](https://polyformproject.org/licenses/internal-use/1.0.0)
-(`LICENSE.md` in the package): you and your company may use it, and change it,
-for your own internal business. You may not give it, or a changed copy, to
-anyone else, sell it, or offer it as a service.
-
-## The demo images
-
-Every image in the demo datasets is in the public domain. Each has its line in
-`lab/demo/CREDITS.md` inside the package.
+(`LICENSE.md`): you and your company may use and change it for your own
+internal business, but not give it, or a changed copy, to anyone else, sell
+it, or offer it as a service. The demo images are public domain; each is
+credited in `lab/demo/CREDITS.md`.

package/bin/evals-lab.js

@@ -1,6 +1,8 @@
 #!/usr/bin/env node
 // The npm package's launcher: `evals-lab` starts the lab's server and says
-// where it is.
+// where it is, and `evals-lab run <bundle>` runs an Export for CI bundle with
+// no server at all, or `evals-lab run --lab <url>` a running lab's pipeline
+// (run.js).
 //
 // The lab itself is server.py, a stdlib Python server, staged beside this file
 // in ../lab by build.js. What this adds is what a lab run by someone who only
@@ -26,6 +28,8 @@ const PYTHON_FLOOR = [3, 10];
 const LOCAL_OLLAMA = "http://127.0.0.1:11434";
 
 const USAGE = `Usage: evals-lab [options]
+       evals-lab run <bundle> [options]   (evals-lab run --help)
+       evals-lab run --lab <url> --pipeline <name> [--wait] [options]
 
   --port <n>        the port to listen on. Default: $PORT, else 8080.
   --host <address>  the address to listen on. Default: $LAB_HOST, else
@@ -152,6 +156,10 @@ function fail(message) {
 }
 
 async function main() {
+  if (process.argv[2] === "run") {
+    process.exitCode = await require("./run.js").run(process.argv.slice(3), { lab: LAB });
+    return;
+  }
   let o;
   try { o = options(process.argv.slice(2)); } catch (e) { fail(`${e.message}\n\n${USAGE}`); }
   if (o.help) return console.log(USAGE);