evals-lab 0.3.0 → 0.4.1

package/CHANGELOG.md

@@ -5,6 +5,42 @@ Newest first. Read a release's **Upgrade notes** before installing it: an
 upgrade can rewrite what the lab keeps in your data directory, and an older
 version cannot always read it back.
 
+## 0.4.1
+
+### Changed
+
+- The README is shorter, and describes the whole lab rather than one use
+  of it.
+
+## 0.4.0
+
+### Upgrade notes
+
+- **Copy your data directory before upgrading** (see "Where your data is
+  kept" in the README). A dataset 0.4.0 saves is dataset version 7, which
+  0.3.x cannot read. To go back, reinstall 0.3.0 and restore the copy.
+- Stored datasets are not rewritten on upgrade: 0.4.0 reads a version-6
+  dataset as version 7, and writes version 7 the next time it is saved.
+- Export JSON writes dataset version 7, which 0.3.x refuses to import.
+  Import reads versions 1 to 7.
+
+### Added
+
+- Re-run, in a History row's menu and in Results' run menu: a new run of the
+  same pipeline over the same file revisions and the same dataset version.
+  It is refused while the run is still going, or when one of the files it
+  read is gone. The new run says which run it re-ran, and links back to it
+  in Results. Undo cancels it while it is still queued.
+- A dataset carries four new sections, `scoring`, `grader`, `every` and
+  `run`: how it scores, which model grades it, and metrics for every item
+  and for the whole run. Edit raw… shows them. Runs do not read them yet.
+  A dataset from 0.3.x has them as scored All, by the lab's grader, with no
+  metrics of its own.
+
+### Changed
+
+- Resume is in a History row's menu, beside Re-run.
+
 ## 0.3.0
 
 ### Upgrade notes

package/README.md

@@ -1,10 +1,15 @@
 # evals-lab
 
-A browser bench for grading what a model answers: two wordings side by side
-over a set of files, scored against answers fixed in advance.
+A lab in the browser for evaluating prompts and models. A pipeline takes items
+from a Source — uploaded files, a Power Automate flow's run history, or a
+prompt alone — and sends each to one or more targets, every target a model
+with its own prompt. Its evals score the replies with metrics — text and JSON
+checks, item counts, agreement with production's reply, or a grading model —
+and a dataset holds each item's expected answers. Every run is kept in
+History, and every prompt version in the Library.
 
-This is an early test release. It runs on your own machine and talks to the
-models you point it at.
+An early test release. It runs on your machine and talks only to the models
+and services you connect it to.
 
 ## Install and run
 
@@ -13,31 +18,23 @@ npm install -g evals-lab
 evals-lab
 ```
 
-Then open <http://localhost:8080>. A new lab opens on two demo datasets, each
-with its images and a starter pipeline. The first start takes a little longer
-than later ones while it installs them.
+Open <http://localhost:8080>. A new lab starts with two demo datasets and
+their pipelines; the first start takes longer while it installs them. Without
+installing: `npx evals-lab`.
 
-Without installing: `npx evals-lab`.
-
-## What it needs
-
-| | | Windows | Linux |
-|---|---|---|---|
-| **Node.js** 22.18 or newer | runs the lab's worker | you have it if you have `npm` | same |
-| **Python** 3.10 or newer | runs the lab's server | `winget install Python.Python.3.13` | your distribution's `python3` |
-| **ImageMagick** | sizes images before a model is sent them, and makes thumbnails | `winget install ImageMagick.ImageMagick` | your distribution's `imagemagick` |
-
-`evals-lab` says which of these it could not find. It starts without
-ImageMagick, but a run that sends images needs it. Open a new terminal after
-installing one, so it is on your `PATH`.
+| Needs | Windows | Linux |
+|---|---|---|
+| **Node.js** 22.18+ | comes with `npm` | same |
+| **Python** 3.10+ | `winget install Python.Python.3.13` | your distribution's `python3` |
+| **ImageMagick**, for runs that send images | `winget install ImageMagick.ImageMagick` | your distribution's `imagemagick` |
 
-macOS should work the same way and has not been tried.
+`evals-lab` names anything it cannot find. Open a new terminal after
+installing one. macOS should work and is untried.
 
 ## Where your data is kept
 
-Datasets, pipelines, Setup profiles (API keys among them), prompts, uploaded
-files and runs are kept on your machine, in one directory, so the lab opens
-as you left it:
+Datasets, pipelines, Setup profiles (with their API keys), prompts, uploaded
+files and runs are kept in one directory, which upgrades leave alone:
 
 | | |
 |---|---|
@@ -45,8 +42,7 @@ as you left it:
 | Linux | `~/.local/share/evals-lab` (or `$XDG_DATA_HOME/evals-lab`) |
 | macOS | `~/Library/Application Support/evals-lab` |
 
-Upgrading or reinstalling the package leaves it alone. Delete the directory
-to start again from a new lab.
+Delete it to start again from a new lab.
 
 ## Options
 
@@ -61,125 +57,60 @@ evals-lab [--port <n>] [--host <address>] [--data-dir <dir>]
 | `--data-dir` | `DATA_DIR` | the directory above |
 | | `LAB_PASSWORD` | none |
 | | `OLLAMA_URL` | `http://127.0.0.1:11434` |
-| | `M365_CLIENT_ID`, `M365_TENANT` | none; see Power Automate below |
 | | `PYTHON` | `py -3`, `python` or `python3`, whichever is found |
 
-`OLLAMA_URL` is only the connection a new lab starts with. Every other
-connection — an OpenAI-compatible endpoint, Anthropic, llama.cpp — is a Setup
-profile you add in the page.
+`--host 0.0.0.0` makes the lab reachable from your network and is refused
+without `LAB_PASSWORD`. Never put it on the open internet: the lab holds your
+API keys and forwards requests to any address a profile names.
+
+## Models
 
-## Power Automate (Microsoft 365)
+A model is a Setup profile: **Setup › Add profile**, then pick its type —
+Ollama (local or cloud), OpenAI-compatible, Anthropic, llama.cpp or an HTTP
+endpoint — and give its address, model and key. **Test connection** checks
+it. An Ollama profile left without an address uses `OLLAMA_URL`.
 
-The lab can read a Power Automate cloud flow, and its run history, from your
-organisation's Microsoft 365 tenant, then grade the flow's HTTP step against
-the same model and prompt. The sign-in stays in your browser tab: the lab's
-server never receives a Microsoft token.
+## Power Automate
 
-It needs a work or school account. Microsoft retired cloud flows for personal
-accounts (outlook.com, hotmail.com) in July 2025, so a personal account cannot
-be used. For a small business, a trial or a Microsoft 365 Developer Program
-tenant, follow the same steps as a company.
+The lab can read a Power Automate cloud flow and its run history from your
+organisation's Microsoft 365 tenant, then grade the flow's HTTP step. It needs
+a work or school account; personal accounts (outlook.com, hotmail.com) have no
+cloud flows. A small business, a trial or a Microsoft 365 Developer Program
+tenant follows the same steps. The sign-in stays in your browser tab.
 
-### Register an app in your tenant
+### Register an app
 
-An administrator of the tenant registers it once, at
-<https://entra.microsoft.com>:
+A tenant administrator does this once, at <https://entra.microsoft.com>:
 
 1. **App registrations › New registration.**
    - Supported account types: *Accounts in this organizational directory only*.
-   - Redirect URI: platform **Single-page application**, value `http://localhost/`.
-     Microsoft ignores the port on a `localhost` redirect, so this covers
-     `--port` too. A lab served on another address needs that address here
-     as well, over `https`.
-2. **API permissions › Add a permission › APIs my organization uses.** Search
-   for *Microsoft Flow Service* and add the delegated permission
-   **Flows.Read.All**. If the service isn't listed, open
-   <https://make.powerautomate.com> once, then search again.
-3. **Grant admin consent** for the tenant.
-4. No client secret or certificate is needed: the lab signs in as a public
-   client.
-
-### Tell the lab about it
-
-In the lab, open **Setup › Connections**, then use the Microsoft 365 row's
-menu › **Configure…**. Enter:
-
-| | |
-|---|---|
-| Client ID | the app's *Application (client) ID* |
-| Tenant | the *Directory (tenant) ID*, or one of the tenant's domains |
-
-Fill in Tenant for an app registered as above. Left empty, it means any
-organisation's tenant, which only a multitenant app accepts: Microsoft
-refuses the sign-in of a single-tenant app with `AADSTS50194`.
-
-Both are saved as you type, and neither is secret. **Sign in** from the same
-menu, then add a Source in the Sources tab with **Add source › Power Automate
-workflow**.
-
-To set the app for the whole machine instead, start the lab with
-`M365_CLIENT_ID` and `M365_TENANT` set. Setup then shows those values and
-can't change them.
-
-Open the lab at `localhost`, the address `evals-lab` prints. A sign-in begun
-at `127.0.0.1` reloads the page at `localhost`, the address registered above,
-and carries on from there. Anything open in the tab, such as an Add source
-dialog, does not carry over.
-
-## Google Drive
-
-**Not usable yet.** The lab can sign in to Google Drive, but nothing reads
-from Drive yet: a Source made from a Drive folder is still to come. Until it
-arrives, there is no reason to set up a Google app, and the steps below are
-provisional. They haven't been tried against a real Google project.
-
-When it arrives, the sign-in will be held by the lab's server, on your
-machine. It will reach only the files you pick: the lab asks for
-`drive.file` access, never your whole Drive.
-
-To sign in, open **Setup › Connections**, then use the Google Drive row's
-menu › **Sign in**. If the row says *Not Configured*, give the lab a Google
-app first, once:
-
-1. At <https://console.cloud.google.com>, make a project. Enable the
-   **Google Drive API** and the **Google Picker API**.
-2. **OAuth consent screen:** choose *External*. Give the app a name and your
-   email, add the scope `.../auth/drive.file`, then **Publish app**. It needs
-   no review from Google, because that scope is non-sensitive. Left in
-   *Testing*, a sign-in lapses after 7 days.
-3. **Credentials › Create credentials › OAuth client ID**, of type
-   **Desktop app**. Download its JSON file.
-4. **Credentials › Create credentials › API key.** Restrict it to the Google
-   Picker API.
-5. In the lab, use the Google Drive row's menu › **Configure…**, then
-   **Load client file…**, and choose the file from step 3. Paste the key
-   from step 4 into **API key**.
-
-A Desktop app needs no redirect registered: Google sends the browser back to
-any port on this machine. A lab you reach at its own address, from another
-machine, needs a *Web application* client instead. Choose that type in
-Configure…, then register the **Redirect URI** it shows with the client.
-
-Everything saves as you type. The client's secret is kept by the lab and
-never shown again. The window says only that one is saved.
-
-## Reaching it from another machine
-
-By default only your own machine can reach the lab. That is deliberate: the
-lab's server forwards requests to whatever address a Setup profile names, so
-anyone who can reach the lab can reach those addresses through it, and the
-lab holds your API keys.
-
-`--host 0.0.0.0` makes it reachable from your network, and is refused unless
-`LAB_PASSWORD` is set. Do not put it on the open internet.
+   - Redirect URI: **Single-page application**, `http://localhost/`. This
+     covers any `--port`. A lab served elsewhere needs its own address here
+     too, over `https`.
+2. **API permissions › Add a permission › APIs my organization uses ›
+   Microsoft Flow Service**, delegated **Flows.Read.All**. If it isn't listed,
+   open <https://make.powerautomate.com> once and search again.
+3. **Grant admin consent.** No client secret is needed.
+
+### Connect the lab
+
+1. **Setup › Connections**, the Microsoft 365 row's menu › **Configure…**:
+   - Client ID: the app's *Application (client) ID*.
+   - Tenant: the *Directory (tenant) ID* or one of its domains. Left empty, a
+     single-tenant app's sign-in fails with `AADSTS50194`.
+2. The same menu › **Sign in**, with the lab open at `localhost`. A sign-in
+   from `127.0.0.1` reloads the page there and loses any open dialog.
+3. **Library › Sources › Add source › Power Automate workflow.**
+
+Or set `M365_CLIENT_ID` and `M365_TENANT` when starting the lab, for the
+whole machine; Setup then shows them read-only.
 
 ## Upgrading and removing
 
-Read `CHANGELOG.md` in the package, or on
-<https://www.npmjs.com/package/evals-lab?activeTab=code>, before upgrading.
-Its **Upgrade notes** say when a release rewrites what is in your data
-directory in a form an older version cannot read; copy the directory first
-if you may want to go back.
+Read `CHANGELOG.md` first, in the package or on
+<https://www.npmjs.com/package/evals-lab?activeTab=code>. When its
+**Upgrade notes** say a release rewrites your data, copy the data directory
+before upgrading if you may want to go back.
 
 ```bash
 npm install -g evals-lab@latest   # upgrade
@@ -188,13 +119,9 @@ npm uninstall -g evals-lab        # remove; your data directory stays
 
 ## Licence
 
-Copyright 2026 Grainbox Limited. Licensed under the
+Copyright 2026 Grainbox Limited, under the
 [PolyForm Internal Use License 1.0.0](https://polyformproject.org/licenses/internal-use/1.0.0)
-(`LICENSE.md` in the package): you and your company may use it, and change it,
-for your own internal business. You may not give it, or a changed copy, to
-anyone else, sell it, or offer it as a service.
-
-## The demo images
-
-Every image in the demo datasets is in the public domain. Each has its line in
-`lab/demo/CREDITS.md` inside the package.
+(`LICENSE.md`): you and your company may use and change it for your own
+internal business, but not give it, or a changed copy, to anyone else, sell
+it, or offer it as a service. The demo images are public domain; each is
+credited in `lab/demo/CREDITS.md`.

package/lab/VERSION

@@ -1 +1 @@
-0.3.0 (2026.10.02-355)
+0.4.1 (2026.10.03-363)

package/lab/evals-core.mjs

@@ -983,22 +983,37 @@ const SLOTS                      = ["content", "target", "responses"];
                             
                                        
                                                                           
-                      
+                                                                              
                      
-                          
+                                                                     
+                                                                   
+                        
+                                                                              
+                     
+                                                       
+                  
+                                                                
+                
+                                                                           
                 
  
 
-/** The dataset body's version: 6 marks itself; 5 named its Source; 4 and
-    earlier, neither. */
-const DATASET_BODY_VERSION = 6         ;
+/** An eval group's scoring (docs/pipeline-model.md §17). */
+                               
+                           
+                           
+ 
+
+/** The dataset body's version: 7 is an eval group (§17); 6 marks itself; 5
+    named its Source; 4 and earlier, neither. */
+const DATASET_BODY_VERSION = 7         ;
 
 /** The metrics whose Ignore case version 6 made mean what it says for a
     reply read as a list, as well as one read as text. */
 const CASE_FOLDING = ["contains", "contains-all", "contains-any"];
 
 /**
- * [body] as this version of a dataset (6), from any earlier one. Version 1
+ * [body] as this version of a dataset (7), from any earlier one. Version 1
  * held `imageCases`, each with `minTags`/`maxTags`, and the parser's
  * `replays` and `conformance` (now fixtures/replays.json beside the checks).
  * Version 2 held `rules`, which clean a job's answer and so belong to the
@@ -1010,7 +1025,9 @@ const CASE_FOLDING = ["contains", "contains-all", "contains-any"];
  * `note`, `traits` go, and the body names no Source yet. Version 5 matched
  * a list's items ignoring case whatever a Contains metric's Ignore case
  * said, so each of its Contains metrics says Ignore case (`caseOfV5`), and
- * the body says its version. Every reader of a
+ * the body says its version. Version 6 is a group's cases alone: version 7
+ * adds its scoring, grader, Every item and Whole run (`groupOfV6`), and a
+ * stored row is read that way rather than rewritten. Every reader of a
  * body calls this: the runner, the page, a run's kept copy. A reader that
  * needs a version-2 body's rules -- to upgrade a pipeline graded against
  * it -- takes them first (`datasetRules`). Pure: the same body gives the
@@ -1021,14 +1038,26 @@ function upgradeDatasetBody   (body   )                  {
   if (!isObj(body)) return body;
   if (body.version === DATASET_BODY_VERSION) return body;
   const b = body             ;
+  // A body saying any other version is one this lab does not read.
+  if ("version" in b) return b.version === 6 ? groupOfV6(b) : body;
   // A body that names its Source, even as null, is version 5.
   if ("source" in b) {
-    return { version: DATASET_BODY_VERSION, ...b, ...(Array.isArray(b.cases) ? { cases: b.cases.map(caseOfV5) } : {}) };
+    return groupOfV6({ ...b, ...(Array.isArray(b.cases) ? { cases: b.cases.map(caseOfV5) } : {}) });
   }
   const raw = Array.isArray(b.cases) ? b.cases : Array.isArray(b.imageCases) ? b.imageCases : null;
   // Not a body of any version -- a copy kept while a dataset was an overlay.
   if (!raw) return body;
-  return { version: DATASET_BODY_VERSION, source: null, cases: raw.map(caseOfV4).map(caseOfV5) };
+  return groupOfV6({ source: null, cases: raw.map(caseOfV4).map(caseOfV5) });
+}
+
+/** A version-6 body as a version-7 eval group: scored All, the lab's
+    grader, and no metrics of its own for every item or the whole run --
+    what a Metrics eval naming the dataset with none of its own graded, which
+    is what every Graded set converted to. */
+function groupOfV6(b           )              {
+  const { version: _v, ...rest } = b;
+  return { version: DATASET_BODY_VERSION, source: null, scoring: { mode: "all", threshold: null }, grader: null, every: [], run: [],
+           ...rest }               ;
 }
 
 /** A version-5 case as a version-6 one: each Contains metric says Ignore
@@ -3187,15 +3216,7 @@ EVAL_TYPES.metrics = {
     ? { ...t, grader: { id: ctx.grader.id, name: ctx.grader.name } } : t),
   wholeRun: t => t.over === "run",
   // Over the whole run: every metric over the replies together, or each alone.
-  verdict(t, ress, kind){
-    const run = runInput(ress, !(kind != null && OUTPUT_KINDS[kind]?.terms));
-    const metrics = readRun(t.metrics || [], run);
-    const s = scoreOf(metrics, t.mode, t.threshold);
-    const off = metrics.filter(r => !r.pass);
-    return { pass: s ? s.pass : true, detail: off.length ? off.map(r => `${r.label}: ${r.reason}`).join(" · ") : "all matched",
-             ran: run.replies.length,
-             checks: metrics.map((r, x) => ({ key: `m${x}`, label: r.label, want: "", pass: r.pass, ...(r.pass ? {} : { note: r.reason }) })) };
-  },
+  verdict: (t, ress, kind) => wholeRunVerdict(t.metrics || [], ress, kind, t.mode, t.threshold),
   // Rule m{x} is the eval's own metric x, read in that place on every item.
   // A score stored before Metrics -- a Graded set's, read as its conversion --
   // has no readings of its own: its one metric's reading is the score's.
@@ -3296,6 +3317,38 @@ LEGACY_TESTS.single .toMetrics = t => {
   return { ...common(t), type: "metrics", mode: "all", threshold: null, grader: null, dataset: null, over: "run", metrics };
 };
 
+/** [list] read over a whole run's replies so far, scored the way [mode]
+    says: a Metrics eval's verdict over the run, and an eval group's Whole run. */
+function wholeRunVerdict(list                   , ress                                       , kind               ,
+                         mode = "all", threshold                = null)                  {
+  const run = runInput(ress, !(kind != null && OUTPUT_KINDS[kind]?.terms));
+  const metrics = readRun(list, run);
+  const s = scoreOf(metrics, mode, threshold);
+  const off = metrics.filter(r => !r.pass);
+  return { pass: s ? s.pass : true, detail: off.length ? off.map(r => `${r.label}: ${r.reason}`).join(" · ") : "all matched",
+           ran: run.replies.length,
+           checks: metrics.map((r, x) => ({ key: `m${x}`, label: r.label, want: "", pass: r.pass, ...(r.pass ? {} : { note: r.reason }) })) };
+}
+
+/**
+ * An eval group's verdict on one item (docs/pipeline-model.md §17): its
+ * Every item metrics and [kase]'s own -- the item's case in the group, or
+ * null where it has none -- read of the reply under the group's scoring.
+ * The reading a Metrics eval naming the group as its dataset gives, which
+ * groups-check.js holds it to. Null where no metric had anything to read.
+ */
+function readGroup(group             , kase             , res                    , more           = {})                              {
+  return readMetrics([...group.every, ...(kase?.metrics ?? [])], metricInput(res, kase, more), graderCtx(group, more),
+                     group.scoring.mode, group.scoring.threshold);
+}
+
+/** An eval group's Whole run verdict, from the replies so far: its `run`
+    metrics under its scoring. Null for a group with no Whole run metrics,
+    which has nothing to say of a run. */
+function readGroupRun(group             , ress                                       , kind                = null)                         {
+  return group.run.length ? wholeRunVerdict(group.run, ress, kind, group.scoring.mode, group.scoring.threshold) : null;
+}
+
 /** The grader a Metrics eval names, reached through what the runner hands it. */
 function graderCtx(t           , more          )            {
   const ask = isRef(t.grader) ? more.grader?.(t.grader) : undefined;
@@ -4768,6 +4821,24 @@ function validateEvals(ev           , files
   }
   if (ev.cases != null && !Array.isArray(ev.cases)) bad.push("cases has to be a list");
   if (ev.source != null && !isRef(ev.source)) bad.push("a dataset names its Source as { id, name }, or null");
+  // An eval group's own sections (version 7): what a Metrics eval held.
+  if (ev.scoring != null) {
+    const sc = ev.scoring;
+    if (!isObj(sc) || !SCORING_MODES.includes(sc.mode)) bad.push(`a group is scored ${SCORING_MODES.join(" or ")}`);
+    else if (sc.mode === "weighted" && typeof sc.threshold !== "number") bad.push("a group scored in points needs Pass at: the points an item has to reach");
+    else if (sc.threshold != null && typeof sc.threshold !== "number") bad.push("a group's Pass at has to be a number");
+  }
+  if (ev.grader != null && !isRef(ev.grader)) bad.push("a group names its grader as { id, name }, or null");
+  if (ev.every != null) metricsProblems(ev.every, "Every item", bad);
+  if (ev.run != null) {
+    metricsProblems(ev.run, "Whole run", bad);
+    // Read once over every reply: nothing that reads one item, or asks a grader.
+    for (const m of Array.isArray(ev.run) ? ev.run : []) {
+      const entry = isObj(m) ? METRICS[m.type] : undefined;
+      if (entry?.graded) bad.push(`Whole run: ${entry.label} is model-graded, so it cannot read a whole run`);
+      else if (entry?.perItem) bad.push(`Whole run: ${entry.label} reads one item at a time, so it cannot read a whole run`);
+    }
+  }
   if (bad.length) return bad;
 
   const graded              = ev.cases || [];
@@ -4915,7 +4986,7 @@ registerMetrics({ registerKinds });
 export {
     TOKEN_DEFAULTS, TOKEN_TYPES, tokenMapping, tokenNames, resolvePrompt, tokenSet, textPrompt, words,
     loopReplyError, preparedSize,
-    termIn, forbiddenIn, readCase, gradedSetFrom, caseItem, caseMetrics, metricLines, upgradeDatasetBody, datasetRules, upgradeResults, emptyTally, addToTally, isHostedUrl,
+    termIn, forbiddenIn, readCase, gradedSetFrom, caseItem, caseMetrics, metricLines, upgradeDatasetBody, readGroup, readGroupRun, datasetRules, upgradeResults, emptyTally, addToTally, isHostedUrl,
     SEED, REPLY_TOKENS_BEFORE, ANTHROPIC_MAX_TOKENS, asNumber, pinReplyTokens, mappingsFor,
     CONNECTION_TYPES, HTTP_APIS, httpApiFor, httpRequestOf, httpReplyOf, readFlowReply, WdlError, localAnswer, typeOf, profileType, convertProfile, splitOllama, DECODING_KEYS, apiBase,
     EDGE_448, budgetLabel, IMAGE_FORMATS, encoderQuality,

package/lab/run-evals.js

@@ -296,8 +296,8 @@ function readDataset(file) {
     broken(`${file}: ${e.message}`);
   }
   if (isObject(doc) && doc.format === "evals-lab/dataset") {
-    if (![1, 2, 3, 4, 5].includes(doc.version)) {
-      broken(`${file} is version ${JSON.stringify(doc.version)}, and this reads versions 1 to 5`);
+    if (![1, 2, 3, 4, 5, 6, 7].includes(doc.version)) {
+      broken(`${file} is version ${JSON.stringify(doc.version)}, and this reads versions 1 to 7`);
     }
     doc = isObject(doc.dataset) ? doc.dataset.body : null;
   }