evals-lab 0.11.0 → 0.13.0

package/CHANGELOG.md

@@ -5,6 +5,102 @@ Newest first. Read a release's **Upgrade notes** before installing it: an
 upgrade can rewrite what the lab keeps in your data directory, and an older
 version cannot always read it back.
 
+## 0.13.0
+
+### Upgrade notes
+
+- Pipelines and datasets keep their versions (17 and 9), so nothing in your
+  data directory is rewritten. A pipeline that uses the new Refused metric
+  needs 0.13.0 to grade it.
+
+### Added
+
+- A Refused metric, for a case that expects the target's call to be refused.
+- A PDF item goes to the model whole, as a document; a connection that can't
+  read one is refused before the run.
+- AI Connections and Connection Profiles select several rows to remove at
+  once, with one Undo; Connection Profiles show their connection.
+- Copy or move… on an eval group's, a filter set's and a Source's own page.
+
+### Changed
+
+- One date format everywhere, and Home lists recent runs.
+- Dropdowns that name things read A–Z; Model, Environment and Flow open the
+  Switcher.
+- Empty values read as a short parenthetical, such as (No grader), and
+  explanations are shorter.
+- A run's jobs sit in one Jobs group, with Add job inside it.
+- A Supplied Source takes several files in one Upload.
+- Enter in a dialog's field presses its main button; in Run settings it
+  leaves the field.
+- Stop reads Stopping… until the run ends, and History can stop a run.
+
+### Fixed
+
+- An errored result fails on its error and is not graded.
+- Metrics and the grader read the reply after its Pre-format filters, and
+  Results shows that reply.
+- A linked group whose grader is gone can't run, and Results says why a
+  graded metric never ran; an off-scale grader score is the grader's error.
+- An Edited target whose expression doesn't parse is refused before the run.
+- Pin names the newest version the server holds.
+- Demo targets answer on a connection the workspace can use.
+- Loop repeats a live capture, and the Comment stays with its pipeline.
+- A Re-sync that needed a Microsoft sign-in runs once the page is back.
+- A dynamic Source's fetch names the lab as its User-Agent.
+- Emptying the trash leaves a connection shared only with the purged
+  workspace shared with none, not All; a lab import's rollback leaves
+  another import's connection intact.
+- A stored Echo connection is no longer listed beside the built-in Echo
+  target.
+
+## 0.12.0
+
+### Upgrade notes
+
+- **Copy your data directory before upgrading** (see "User data" in the
+  README). A pipeline 0.12.0 saves is pipeline version 17, which 0.11.x
+  cannot read. To go back, reinstall 0.11.0 and restore the copy.
+- An HTTP Request step no longer picks an API of its own: it uses its job's
+  flow step and connection. A step that named a different API keeps the
+  request it would have sent, and a request in a job with no flow step
+  becomes a Prompt keeping its words. Pipelines read as version 17, and are
+  saved as it at their next edit.
+
+### Added
+
+- `evals-lab export --lab <url>` writes a lab's settings to a file:
+  workspaces, connections with who each is shared with, profiles and the
+  grader, never a key. `--workspace <slug>` exports one workspace's.
+  `evals-lab import <file> --lab <url>` adds what the lab lacks, giving each
+  connection the key in `$EVALSLAB_API_KEY_<SLUG>`, and sends keys only to
+  an https or loopback lab.
+- `LAB_IMPORT=<file>` imports a lab file each time the lab starts.
+- A new lab opens on five demos: describe images (twice), classify
+  sentiment, compare wordings, and sort weather alerts from a live HTTP GET
+  Source. A pack can now carry a dynamic Source.
+- Create test… on a dynamic Source's Calls makes a dataset that is graded
+  and ready to run.
+
+### Changed
+
+- A new lab starts in about half the time.
+- Names, slugs and model ids get the full width of their field, the tables
+  in one dialog share one label column, and Undo is amber and labelled.
+- A dynamic Source names its provider once, and the grader picker says
+  Connection.
+
+### Fixed
+
+- An API call Source sends an HTTP endpoint connection's key in the header
+  the connection names, not always `Authorization: Bearer`.
+- An API call Source can use only a connection shared with its workspace.
+- A run is refused when two connections take their key from the same
+  variable, on a re-run too.
+- Seeding from production reads a gateway flow's replies in its
+  connection's request format.
+- The Runs stop keeps its icon on Home.
+
 ## 0.11.0
 
 ### Upgrade notes

package/README.md

@@ -45,8 +45,8 @@ npm install -g evals-lab
 evals-lab
 ```
 
-Open <http://localhost:8080>. A new lab starts with two demo datasets and
-their pipelines. Note the first start takes longer while it installs them. Without
+Open <http://localhost:8080>. A new lab starts with five demos, each a dataset
+and its pipeline, which run once you add an AI Connection in Setup. Note the first start takes longer while it installs them. Without
 installing: `npx evals-lab`.
 
 | Needs | Windows | Linux |
@@ -164,7 +164,7 @@ CI can run a lab's own pipeline there instead, so the run shows in that lab's
 History and its keys never leave it:
 
 ```bash
-LAB_PASSWORD=... evals-lab run --lab https://lab.example --pipeline "Demo 1" --wait
+LAB_PASSWORD=... evals-lab run --lab https://lab.example --pipeline "Demo: describe images" --wait
 ```
 
 ```text
@@ -249,5 +249,5 @@ Copyright 2026 Grainbox Limited, under the
 [PolyForm Internal Use License 1.0.0](https://polyformproject.org/licenses/internal-use/1.0.0)
 (`LICENSE.md`): you and your company may use and change it for your own
 internal business, but not give it, or a changed copy, to anyone else, sell
-it, or offer it as a service. The demo images are public domain; each is
+it, or offer it as a service. The demo files are public domain; each is
 credited in `lab/demo/CREDITS.md`.

package/bin/evals-lab.js

@@ -30,6 +30,8 @@ const LOCAL_OLLAMA = "http://127.0.0.1:11434";
 const USAGE = `Usage: evals-lab [options]
        evals-lab run <bundle> [options]   (evals-lab run --help)
        evals-lab run --lab <url> --pipeline <name> [--wait] [options]
+       evals-lab export --lab <url> [--workspace <slug>] [--out <file>]
+       evals-lab import <file> --lab <url>   (evals-lab import --help)
 
   --port <n>        the port to listen on. Default: $PORT, else 8080.
   --host <address>  the address to listen on. Default: $LAB_HOST, else
@@ -43,7 +45,9 @@ Environment: LAB_PASSWORD puts a password in front of the lab, OLLAMA_URL is
 the connection a new lab starts with, and PYTHON names the Python to run.
 SOURCE_ALLOW_LOCAL_FILES=1 and SOURCE_ALLOW_LOCAL_COMMANDS=1 let a Source read a
 file or run a command on this machine; both are off unless set.
-SOURCE_LOCAL_ROOT=<dir> lets a file Source read only under that directory.`;
+SOURCE_LOCAL_ROOT=<dir> lets a file Source read only under that directory.
+LAB_IMPORT=<file> imports a lab file each time the lab starts, each
+connection's key from $EVALSLAB_API_KEY_<SLUG>.`;
 
 // Where a lab keeps its store when nobody said: the platform's per-user data
 // directory, so it survives the package being upgraded or reinstalled.
@@ -159,6 +163,10 @@ function fail(message) {
 }
 
 async function main() {
+  if (process.argv[2] === "export" || process.argv[2] === "import") {
+    process.exitCode = await require("./lab-file.js").labFile(process.argv[2], process.argv.slice(3));
+    return;
+  }
   if (process.argv[2] === "run") {
     process.exitCode = await require("./run.js").run(process.argv.slice(3), { lab: LAB });
     return;

package/bin/lab-file.js

@@ -0,0 +1,97 @@
+// `evals-lab export` and `evals-lab import` (docs/lab-export.md): a running
+// lab's settings -- its AI Connections, Connection Profiles, grader and
+// workspaces -- as a lab file, and a lab file into a lab, adding what it
+// lacks. Setup › Workspaces › Data does the same through the same two routes.
+//
+// THE KEYS. A lab file holds none, and the lab serves none. `import` sends
+// each connection's from $EVALSLAB_API_KEY_<SLUG>, as a CI run reads it, and
+// only to an https or loopback lab: anything else would put the key on the
+// wire in the clear.
+const fs = require("fs");
+const path = require("path");
+const { labApi, Refused } = require("./run.js");
+
+const EXIT = { broken: 3 };
+
+const USAGE = `Usage: evals-lab export --lab <url> [--workspace <slug>] [--out <file>]
+       evals-lab import <file> --lab <url>
+
+  --lab <url>        the running lab. LAB_PASSWORD is sent when it is set.
+  --workspace <slug> export only the workspace at /w/<slug>. Default: every one.
+  --out <file>       where the export goes. Default: stdout.
+
+A lab file holds no key. import gives each connection the key in
+$EVALSLAB_API_KEY_<SLUG> when it is set, and sends keys only to an https or
+loopback lab. An import only adds: what the lab has already is left as it is.`;
+
+function options(argv, verb) {
+  const o = {};
+  const value = { "--lab": "lab", "--workspace": "workspace", "--out": "out" };
+  for (let i = 0; i < argv.length; i++) {
+    const a = argv[i];
+    if (a === "--help" || a === "-h") o.help = true;
+    else if (value[a]) {
+      if (argv[i + 1] === undefined) throw new Refused(`${a} needs a value`, true);
+      o[value[a]] = argv[++i];
+    } else if (a.startsWith("-")) throw new Refused(`no option ${a}`, true);
+    else if (verb === "import" && o.file === undefined) o.file = a;
+    else throw new Refused(`one lab file, not ${a} as well`, true);
+  }
+  if (o.help) return o;
+  if (o.lab == null) throw new Refused("name the lab with --lab", true);
+  if (!/^https?:\/\/[^/]/i.test(o.lab)) throw new Refused(`--lab ${o.lab} is not an http or https address`, true);
+  if (verb === "import") {
+    if (o.file == null) throw new Refused("name the lab file to import", true);
+    if (o.workspace != null || o.out != null) throw new Refused("--workspace and --out are export's", true);
+  }
+  return o;
+}
+
+const keyVar = (id, slug) => "EVALSLAB_API_KEY_"
+  + String(slug || id).toUpperCase().replace(/[^A-Z0-9]+/g, "_").replace(/^_+|_+$/g, "");
+
+// Loopback or https: where a key may go.
+function keySafe(base) {
+  const u = new URL(base);
+  return u.protocol === "https:" || ["localhost", "127.0.0.1", "[::1]"].includes(u.hostname);
+}
+
+async function labFile(verb, argv, { env = process.env, stdout = process.stdout, stderr = process.stderr } = {}) {
+  const say = text => stderr.write(text + "\n");
+  try {
+    const o = options(argv, verb);
+    if (o.help) { stdout.write(USAGE + "\n"); return 0; }
+    const base = o.lab.replace(/\/+$/, "");
+    const api = labApi(base, env);
+    if (verb === "export") {
+      const out = await api("GET", "/api/lab-export" + (o.workspace ? `?workspace=${encodeURIComponent(o.workspace)}` : ""));
+      const text = JSON.stringify(out, null, 2) + "\n";
+      if (o.out) fs.writeFileSync(path.resolve(o.out), text);
+      else stdout.write(text);
+      if (o.out) say(`${out.connections.length} connections, ${out.workspaces.length} workspaces to ${o.out}`);
+      return 0;
+    }
+    let lab;
+    try { lab = JSON.parse(fs.readFileSync(path.resolve(o.file), "utf8")); }
+    catch (e) { throw new Refused(`${o.file} is not a lab file: ${e.message}`); }
+    const keys = {};
+    for (const c of Array.isArray(lab?.connections) ? lab.connections : []) {
+      const name = keyVar(c?.id, c?.slug);
+      if (env[name]) keys[name] = env[name];
+    }
+    if (Object.keys(keys).length && !keySafe(base)) {
+      throw new Refused(`${base} is neither https nor loopback, so no key goes to it: `
+        + `unset ${Object.keys(keys).join(", ")} to import without them`);
+    }
+    const { said } = await api("POST", "/api/lab-import", { lab, keys });
+    for (const line of said) say(line);
+    return 0;
+  } catch (e) {
+    if (!(e instanceof Refused)) throw e;
+    say(`evals-lab ${verb}: ${e.message}`);
+    if (e.usage) say(`\n${USAGE}`);
+    return EXIT.broken;
+  }
+}
+
+module.exports = { labFile, options, keyVar, keySafe, USAGE };

package/bin/run.js

@@ -573,4 +573,4 @@ async function run(argv, { lab, env = process.env, stdout = process.stdout, stde
   }
 }
 
-module.exports = { run, runOptions, USAGE };
+module.exports = { run, runOptions, USAGE, labApi, Refused };

package/lab/VERSION

@@ -1 +1 @@
-0.11.0 (2026.10.11-671)
+0.13.0 (2026.10.13-794)

package/lab/demo/CREDITS.md

@@ -1,9 +1,13 @@
 # Credits
 
-Every file in `images/`, where it came from and under what terms. All
+Every file in `sources/`, where it came from and under what terms. All
 need no attribution; the sources are listed so each one can be traced.
-Files were resized to about 3 megapixels and their metadata stripped;
-the panorama is a crop, and the PNG and TIFF are conversions.
+
+## Images
+
+The images of `Demo 1/` and `Demo 2/`. Files were resized to about 3
+megapixels and their metadata stripped; the panorama is a crop, and the PNG
+and TIFF are conversions.
 
 | File | Source | Author | Licence |
 |---|---|---|---|
@@ -90,3 +94,33 @@ the panorama is a crop, and the PNG and TIFF are conversions.
 | `whiteboard-meeting-notes.jpg` | https://www.flickr.com/photo.gne?id=186873704 |  | Public domain |
 | `whiteboard-messages.jpg` | https://www.flickr.com/photo.gne?id=23162308061 |  | Public domain |
 | `wooden-house-fence.jpg` | https://www.flickr.com/photo.gne?id=26595748497 |  | Public domain |
+
+## Texts
+
+The short texts of `sentiment/`, written for this pack, and the alert
+types of `weather-alerts/`, fetched from the National Weather Service's API
+on 2026-10-11. A work of the US federal government is in the public domain.
+
+| File | Source | Author | Licence |
+|---|---|---|---|
+| `bakery-croissant.txt` | Written for this pack | Evals Lab | Public domain |
+| `bus-timetable.txt` | Written for this pack | Evals Lab | Public domain |
+| `cafe-coffee.txt` | Written for this pack | Evals Lab | Public domain |
+| `concert-night.txt` | Written for this pack | Evals Lab | Public domain |
+| `film-sequel.txt` | Written for this pack | Evals Lab | Public domain |
+| `garden-tomatoes.txt` | Written for this pack | Evals Lab | Public domain |
+| `gym-contract.txt` | Written for this pack | Evals Lab | Public domain |
+| `headphones-review.txt` | Written for this pack | Evals Lab | Public domain |
+| `hotel-room.txt` | Written for this pack | Evals Lab | Public domain |
+| `library-hours.txt` | Written for this pack | Evals Lab | Public domain |
+| `meeting-minutes.txt` | Written for this pack | Evals Lab | Public domain |
+| `parcel-lost.txt` | Written for this pack | Evals Lab | Public domain |
+| `phone-update.txt` | Written for this pack | Evals Lab | Public domain |
+| `puppy-home.txt` | Written for this pack | Evals Lab | Public domain |
+| `recipe-note.txt` | Written for this pack | Evals Lab | Public domain |
+| `road-closure.txt` | Written for this pack | Evals Lab | Public domain |
+| `teacher-thanks.txt` | Written for this pack | Evals Lab | Public domain |
+| `train-delay.txt` | Written for this pack | Evals Lab | Public domain |
+| `volunteer-day.txt` | Written for this pack | Evals Lab | Public domain |
+| `weather-report.txt` | Written for this pack | Evals Lab | Public domain |
+| `https-api.weather.gov-alerts-types.txt` | https://api.weather.gov/alerts/types | National Weather Service | Public domain |

package/lab/demo/datasets/demo-1.json

@@ -2,12 +2,12 @@
   "format": "evals-lab/dataset",
   "version": 6,
   "dataset": {
-    "name": "Demo 1",
+    "name": "Demo: describe images",
     "body": {
       "version": 6,
       "source": {
         "id": "Demo 1",
-        "name": "Demo 1"
+        "name": "Demo: describe images"
       },
       "cases": [
         {

package/lab/demo/datasets/demo-2.json

@@ -2,12 +2,12 @@
   "format": "evals-lab/dataset",
   "version": 6,
   "dataset": {
-    "name": "Demo 2",
+    "name": "Demo: describe images 2",
     "body": {
       "version": 6,
       "source": {
         "id": "Demo 2",
-        "name": "Demo 2"
+        "name": "Demo: describe images 2"
       },
       "cases": [
         {