evals-lab 0.10.0 → 0.12.0

package/CHANGELOG.md

@@ -5,6 +5,91 @@ Newest first. Read a release's **Upgrade notes** before installing it: an
 upgrade can rewrite what the lab keeps in your data directory, and an older
 version cannot always read it back.
 
+## 0.12.0
+
+### Upgrade notes
+
+- **Copy your data directory before upgrading** (see "User data" in the
+  README). A pipeline 0.12.0 saves is pipeline version 17, which 0.11.x
+  cannot read. To go back, reinstall 0.11.0 and restore the copy.
+- An HTTP Request step no longer picks an API of its own: it uses its job's
+  flow step and connection. A step that named a different API keeps the
+  request it would have sent, and a request in a job with no flow step
+  becomes a Prompt keeping its words. Pipelines read as version 17, and are
+  saved as it at their next edit.
+
+### Added
+
+- `evals-lab export --lab <url>` writes a lab's settings to a file:
+  workspaces, connections with who each is shared with, profiles and the
+  grader, never a key. `--workspace <slug>` exports one workspace's.
+  `evals-lab import <file> --lab <url>` adds what the lab lacks, giving each
+  connection the key in `$EVALSLAB_API_KEY_<SLUG>`, and sends keys only to
+  an https or loopback lab.
+- `LAB_IMPORT=<file>` imports a lab file each time the lab starts.
+- A new lab opens on five demos: describe images (twice), classify
+  sentiment, compare wordings, and sort weather alerts from a live HTTP GET
+  Source. A pack can now carry a dynamic Source.
+- Create test… on a dynamic Source's Calls makes a dataset that is graded
+  and ready to run.
+
+### Changed
+
+- A new lab starts in about half the time.
+- Names, slugs and model ids get the full width of their field, the tables
+  in one dialog share one label column, and Undo is amber and labelled.
+- A dynamic Source names its provider once, and the grader picker says
+  Connection.
+
+### Fixed
+
+- An API call Source sends an HTTP endpoint connection's key in the header
+  the connection names, not always `Authorization: Bearer`.
+- An API call Source can use only a connection shared with its workspace.
+- A run is refused when two connections take their key from the same
+  variable, on a re-run too.
+- Seeding from production reads a gateway flow's replies in its
+  connection's request format.
+- The Runs stop keeps its icon on Home.
+
+## 0.11.0
+
+### Upgrade notes
+
+- **Copy your data directory before upgrading** (see "User data" in the
+  README). A pipeline 0.11.0 saves is pipeline version 16, which 0.10.x
+  cannot read. To go back, reinstall 0.10.0 and restore the copy.
+- On first start, each Target profile becomes an AI Connection (its type,
+  address, key and model), keeping its key and who it is shared with, plus a
+  Connection Profile where it changed any model or image setting. The old
+  profiles stay stored, unused.
+- A pipeline's target names a Connection and a Profile. Pipelines read as
+  version 16, and are saved as it at their next edit; a run from before
+  shows its Target profile as its Profile.
+
+### Added
+
+- Setup › Connections holds AI Connections and Connection Profiles: a
+  connection is where to send and with which key, and a profile is how to
+  ask. A target picks one of each, so one connection serves several
+  profiles.
+- Every pipeline, prompt, run, profile and wizard has its own address, and
+  Copy link copies it.
+- A run's Responses read as a flow, one ordered table a stage, and name the
+  stage and filter that discarded a reply. Move to filter set… and Move all
+  turn a filter into part of a filter set.
+- A Call is viewed, edited, renamed and cloned in one dialog, and a
+  workflow or dynamic Source's settings are edited where they were made.
+- The package page gives a contact address and links to its home page.
+
+### Fixed
+
+- File on disk never reads `/proc`, and a Call holding the lab's own secret
+  is refused.
+- Importing a version-15 YAML pipeline keeps each target's profile.
+- A Target profile from before reply limits keeps its 160-token cap when it
+  becomes a connection.
+
 ## 0.10.0
 
 ### Upgrade notes
@@ -42,7 +127,9 @@ version cannot always read it back.
     lab warns at start when one is on while it listens beyond this machine
     without `LAB_PASSWORD`. `SOURCE_LOCAL_ROOT=<dir>` lets File on disk read only
     under that directory. A command runs without a shell, in an empty
-    directory, without the lab's own secrets.
+    directory, with the lab's own secrets out of its environment -- but it
+    can still read them from the lab's, so turn Command on only where
+    everyone who can reach the lab may see them.
 - Copy or move… copies or moves a pipeline, dataset, filter set, prompt or
   Source to another workspace, and says first what a move would leave broken.
 - A run's view opens on an Evals summary, and Show details shows what came

package/README.md

@@ -45,8 +45,8 @@ npm install -g evals-lab
 evals-lab
 ```
 
-Open <http://localhost:8080>. A new lab starts with two demo datasets and
-their pipelines. Note the first start takes longer while it installs them. Without
+Open <http://localhost:8080>. A new lab starts with five demos, each a dataset
+and its pipeline, which run once you add an AI Connection in Setup. Note the first start takes longer while it installs them. Without
 installing: `npx evals-lab`.
 
 | Needs | Windows | Linux |
@@ -111,9 +111,9 @@ evals-lab run <bundle-dir | pipeline.yaml> [--items <dir>] [--json <file|->]
 | `--min-pass` | an eval passes when this share of its items pass |
 | `--progress` | a line per item as it finishes |
 
-The bundle holds no key. Each Target profile's key comes from
+The bundle holds no key. Each AI Connection's key comes from
 `EVALSLAB_API_KEY_` and its slug in Setup, in capitals with hyphens as
-underscores: a profile slugged `anthropic` reads `EVALSLAB_API_KEY_ANTHROPIC`,
+underscores: a connection slugged `anthropic` reads `EVALSLAB_API_KEY_ANTHROPIC`,
 and `gpt-4o-mini` reads `EVALSLAB_API_KEY_GPT_4O_MINI`. A table of results
 goes to stderr.
 
@@ -164,7 +164,7 @@ CI can run a lab's own pipeline there instead, so the run shows in that lab's
 History and its keys never leave it:
 
 ```bash
-LAB_PASSWORD=... evals-lab run --lab https://lab.example --pipeline "Demo 1" --wait
+LAB_PASSWORD=... evals-lab run --lab https://lab.example --pipeline "Demo: describe images" --wait
 ```
 
 ```text
@@ -237,11 +237,17 @@ npm install -g evals-lab@latest   # upgrade
 npm uninstall -g evals-lab        # remove; your data directory stays
 ```
 
+## Contact
+
+[evals-lab@grainbox.com](mailto:evals-lab@grainbox.com) for questions,
+problems and feedback. More at
+<https://www.grainbox.com/products/evals-lab/>.
+
 ## Licence
 
 Copyright 2026 Grainbox Limited, under the
 [PolyForm Internal Use License 1.0.0](https://polyformproject.org/licenses/internal-use/1.0.0)
 (`LICENSE.md`): you and your company may use and change it for your own
 internal business, but not give it, or a changed copy, to anyone else, sell
-it, or offer it as a service. The demo images are public domain; each is
+it, or offer it as a service. The demo files are public domain; each is
 credited in `lab/demo/CREDITS.md`.

package/bin/evals-lab.js

@@ -30,6 +30,8 @@ const LOCAL_OLLAMA = "http://127.0.0.1:11434";
 const USAGE = `Usage: evals-lab [options]
        evals-lab run <bundle> [options]   (evals-lab run --help)
        evals-lab run --lab <url> --pipeline <name> [--wait] [options]
+       evals-lab export --lab <url> [--workspace <slug>] [--out <file>]
+       evals-lab import <file> --lab <url>   (evals-lab import --help)
 
   --port <n>        the port to listen on. Default: $PORT, else 8080.
   --host <address>  the address to listen on. Default: $LAB_HOST, else
@@ -43,7 +45,9 @@ Environment: LAB_PASSWORD puts a password in front of the lab, OLLAMA_URL is
 the connection a new lab starts with, and PYTHON names the Python to run.
 SOURCE_ALLOW_LOCAL_FILES=1 and SOURCE_ALLOW_LOCAL_COMMANDS=1 let a Source read a
 file or run a command on this machine; both are off unless set.
-SOURCE_LOCAL_ROOT=<dir> lets a file Source read only under that directory.`;
+SOURCE_LOCAL_ROOT=<dir> lets a file Source read only under that directory.
+LAB_IMPORT=<file> imports a lab file each time the lab starts, each
+connection's key from $EVALSLAB_API_KEY_<SLUG>.`;
 
 // Where a lab keeps its store when nobody said: the platform's per-user data
 // directory, so it survives the package being upgraded or reinstalled.
@@ -159,6 +163,10 @@ function fail(message) {
 }
 
 async function main() {
+  if (process.argv[2] === "export" || process.argv[2] === "import") {
+    process.exitCode = await require("./lab-file.js").labFile(process.argv[2], process.argv.slice(3));
+    return;
+  }
   if (process.argv[2] === "run") {
     process.exitCode = await require("./run.js").run(process.argv.slice(3), { lab: LAB });
     return;

package/bin/lab-file.js

@@ -0,0 +1,97 @@
+// `evals-lab export` and `evals-lab import` (docs/lab-export.md): a running
+// lab's settings -- its AI Connections, Connection Profiles, grader and
+// workspaces -- as a lab file, and a lab file into a lab, adding what it
+// lacks. Setup › Workspaces › Data does the same through the same two routes.
+//
+// THE KEYS. A lab file holds none, and the lab serves none. `import` sends
+// each connection's from $EVALSLAB_API_KEY_<SLUG>, as a CI run reads it, and
+// only to an https or loopback lab: anything else would put the key on the
+// wire in the clear.
+const fs = require("fs");
+const path = require("path");
+const { labApi, Refused } = require("./run.js");
+
+const EXIT = { broken: 3 };
+
+const USAGE = `Usage: evals-lab export --lab <url> [--workspace <slug>] [--out <file>]
+       evals-lab import <file> --lab <url>
+
+  --lab <url>        the running lab. LAB_PASSWORD is sent when it is set.
+  --workspace <slug> export only the workspace at /w/<slug>. Default: every one.
+  --out <file>       where the export goes. Default: stdout.
+
+A lab file holds no key. import gives each connection the key in
+$EVALSLAB_API_KEY_<SLUG> when it is set, and sends keys only to an https or
+loopback lab. An import only adds: what the lab has already is left as it is.`;
+
+function options(argv, verb) {
+  const o = {};
+  const value = { "--lab": "lab", "--workspace": "workspace", "--out": "out" };
+  for (let i = 0; i < argv.length; i++) {
+    const a = argv[i];
+    if (a === "--help" || a === "-h") o.help = true;
+    else if (value[a]) {
+      if (argv[i + 1] === undefined) throw new Refused(`${a} needs a value`, true);
+      o[value[a]] = argv[++i];
+    } else if (a.startsWith("-")) throw new Refused(`no option ${a}`, true);
+    else if (verb === "import" && o.file === undefined) o.file = a;
+    else throw new Refused(`one lab file, not ${a} as well`, true);
+  }
+  if (o.help) return o;
+  if (o.lab == null) throw new Refused("name the lab with --lab", true);
+  if (!/^https?:\/\/[^/]/i.test(o.lab)) throw new Refused(`--lab ${o.lab} is not an http or https address`, true);
+  if (verb === "import") {
+    if (o.file == null) throw new Refused("name the lab file to import", true);
+    if (o.workspace != null || o.out != null) throw new Refused("--workspace and --out are export's", true);
+  }
+  return o;
+}
+
+const keyVar = (id, slug) => "EVALSLAB_API_KEY_"
+  + String(slug || id).toUpperCase().replace(/[^A-Z0-9]+/g, "_").replace(/^_+|_+$/g, "");
+
+// Loopback or https: where a key may go.
+function keySafe(base) {
+  const u = new URL(base);
+  return u.protocol === "https:" || ["localhost", "127.0.0.1", "[::1]"].includes(u.hostname);
+}
+
+async function labFile(verb, argv, { env = process.env, stdout = process.stdout, stderr = process.stderr } = {}) {
+  const say = text => stderr.write(text + "\n");
+  try {
+    const o = options(argv, verb);
+    if (o.help) { stdout.write(USAGE + "\n"); return 0; }
+    const base = o.lab.replace(/\/+$/, "");
+    const api = labApi(base, env);
+    if (verb === "export") {
+      const out = await api("GET", "/api/lab-export" + (o.workspace ? `?workspace=${encodeURIComponent(o.workspace)}` : ""));
+      const text = JSON.stringify(out, null, 2) + "\n";
+      if (o.out) fs.writeFileSync(path.resolve(o.out), text);
+      else stdout.write(text);
+      if (o.out) say(`${out.connections.length} connections, ${out.workspaces.length} workspaces to ${o.out}`);
+      return 0;
+    }
+    let lab;
+    try { lab = JSON.parse(fs.readFileSync(path.resolve(o.file), "utf8")); }
+    catch (e) { throw new Refused(`${o.file} is not a lab file: ${e.message}`); }
+    const keys = {};
+    for (const c of Array.isArray(lab?.connections) ? lab.connections : []) {
+      const name = keyVar(c?.id, c?.slug);
+      if (env[name]) keys[name] = env[name];
+    }
+    if (Object.keys(keys).length && !keySafe(base)) {
+      throw new Refused(`${base} is neither https nor loopback, so no key goes to it: `
+        + `unset ${Object.keys(keys).join(", ")} to import without them`);
+    }
+    const { said } = await api("POST", "/api/lab-import", { lab, keys });
+    for (const line of said) say(line);
+    return 0;
+  } catch (e) {
+    if (!(e instanceof Refused)) throw e;
+    say(`evals-lab ${verb}: ${e.message}`);
+    if (e.usage) say(`\n${USAGE}`);
+    return EXIT.broken;
+  }
+}
+
+module.exports = { labFile, options, keyVar, keySafe, USAGE };

package/bin/run.js

@@ -62,7 +62,7 @@ const USAGE = `Usage: evals-lab run <bundle-dir | pipeline.yaml> [options]
                      --progress need it.
   --help             this.
 
-A bundle's Target profile keys come from $EVALSLAB_API_KEY_<SLUG>. Exit
+A bundle's connection keys come from $EVALSLAB_API_KEY_<SLUG>. Exit
 codes: 0 pass (or queued, without --wait), 1 an eval failed, 2 incomplete (an
 item missing, one that did not run, or the run cancelled), 3 broken or
 refused.`;
@@ -392,16 +392,20 @@ async function runOnLab(core, o, { env, stdout, say }) {
   }
   const call = labApi(base, env, workspace);
 
-  // What the page reads before it submits: the lab's pipelines and Target
-  // profiles, the Source the pipeline reads and the eval groups it may link.
+  // What the page reads before it submits: the lab's pipelines, AI
+  // Connections and Connection Profiles, the Source the pipeline reads and
+  // the eval groups it may link. Slugged, as RunsTab's submit has them, so
+  // the run carries the slug its keys are spelt from (#253).
   const docs = (await call("GET", "/api/state")).docs ?? {};
-  const setup = docs["promptlab.profiles"]?.body ?? {};
-  const profiles = Array.isArray(setup.list) ? setup.list : [];
+  const setup = docs["promptlab.connections"]?.body ?? {};
+  const connections = core.withSlugs(Array.isArray(setup.list) ? setup.list : []);
+  const held = docs["promptlab.connectionProfiles"]?.body?.list;
+  const answers = core.answersOf(connections, Array.isArray(held) ? held : []);
   const saved = docs["promptlab.workflows"]?.body?.list;
   const found = pipelineNamed(Array.isArray(saved) ? saved : [], o.pipeline, base);
-  const types = new Map(profiles.map(p => [p.id, p.type]));
-  const doc = core.upgradePipeline(found.work, { profileType: id => types.get(id) });
-  const g = profiles.find(p => p.id === setup.grader);
+  const doc = core.upgradePipeline(found.work, { profileType: id => answers.connection(id)?.type,
+                                                 connectionSplit: answers.split });
+  const g = answers.connection(setup.grader);
   const grader = g ? { id: g.id, name: g.name } : null;
   const content = core.contentOf(doc);
   let source = null;
@@ -411,18 +415,14 @@ async function runOnLab(core, o, { env, stdout, say }) {
   const files = source ? source.files.map(f => f.name) : undefined;
   const { datasets } = await call("GET", "/api/datasets");
   const problems = core.validatePipeline(doc, {
-    profiles: id => profiles.find(p => p.id === id) ?? null,
+    profiles: answers.resolve, connections: answers.connection,
     sources: id => (source && id === content.ref.id ? { name: source.name, type: source.type, files } : null),
     grader, groups: datasets ?? [],
   });
   if (problems.length) throw new Refused(`${found.name} cannot run: ${problems.join("; ")}`);
-  // As RunsTab's submit resolves it: slugged, so the run carries the slug
-  // its keys are spelt from (#253), and keyless -- the lab fills each key in
+  // As RunsTab's submit resolves it: keyless -- the lab fills each key in
   // from its own store.
-  const slugged = core.withSlugs(profiles);
-  const run = core.resolvePipeline(doc, {
-    profiles: id => slugged.find(p => p.id === id) ?? null, grader, ...(files ? { files } : {}),
-  });
+  const run = core.resolvePipeline(doc, { profiles: answers.resolve, grader, ...(files ? { files } : {}) });
 
   const queued = (await call("POST", "/api/queue", run)).run;
   say(`evals-lab run: ${found.name} is queued on ${base} as run ${queued.id}`);
@@ -573,4 +573,4 @@ async function run(argv, { lab, env = process.env, stdout = process.stdout, stde
   }
 }
 
-module.exports = { run, runOptions, USAGE };
+module.exports = { run, runOptions, USAGE, labApi, Refused };

package/lab/VERSION

@@ -1 +1 @@
-0.10.0 (2026.10.09-614)
+0.12.0 (2026.10.12-718)

package/lab/demo/CREDITS.md

@@ -1,9 +1,13 @@
 # Credits
 
-Every file in `images/`, where it came from and under what terms. All
+Every file in `sources/`, where it came from and under what terms. All
 need no attribution; the sources are listed so each one can be traced.
-Files were resized to about 3 megapixels and their metadata stripped;
-the panorama is a crop, and the PNG and TIFF are conversions.
+
+## Images
+
+The images of `Demo 1/` and `Demo 2/`. Files were resized to about 3
+megapixels and their metadata stripped; the panorama is a crop, and the PNG
+and TIFF are conversions.
 
 | File | Source | Author | Licence |
 |---|---|---|---|
@@ -90,3 +94,33 @@ the panorama is a crop, and the PNG and TIFF are conversions.
 | `whiteboard-meeting-notes.jpg` | https://www.flickr.com/photo.gne?id=186873704 |  | Public domain |
 | `whiteboard-messages.jpg` | https://www.flickr.com/photo.gne?id=23162308061 |  | Public domain |
 | `wooden-house-fence.jpg` | https://www.flickr.com/photo.gne?id=26595748497 |  | Public domain |
+
+## Texts
+
+The short texts of `sentiment/`, written for this pack, and the alert
+types of `weather-alerts/`, fetched from the National Weather Service's API
+on 2026-10-11. A work of the US federal government is in the public domain.
+
+| File | Source | Author | Licence |
+|---|---|---|---|
+| `bakery-croissant.txt` | Written for this pack | Evals Lab | Public domain |
+| `bus-timetable.txt` | Written for this pack | Evals Lab | Public domain |
+| `cafe-coffee.txt` | Written for this pack | Evals Lab | Public domain |
+| `concert-night.txt` | Written for this pack | Evals Lab | Public domain |
+| `film-sequel.txt` | Written for this pack | Evals Lab | Public domain |
+| `garden-tomatoes.txt` | Written for this pack | Evals Lab | Public domain |
+| `gym-contract.txt` | Written for this pack | Evals Lab | Public domain |
+| `headphones-review.txt` | Written for this pack | Evals Lab | Public domain |
+| `hotel-room.txt` | Written for this pack | Evals Lab | Public domain |
+| `library-hours.txt` | Written for this pack | Evals Lab | Public domain |
+| `meeting-minutes.txt` | Written for this pack | Evals Lab | Public domain |
+| `parcel-lost.txt` | Written for this pack | Evals Lab | Public domain |
+| `phone-update.txt` | Written for this pack | Evals Lab | Public domain |
+| `puppy-home.txt` | Written for this pack | Evals Lab | Public domain |
+| `recipe-note.txt` | Written for this pack | Evals Lab | Public domain |
+| `road-closure.txt` | Written for this pack | Evals Lab | Public domain |
+| `teacher-thanks.txt` | Written for this pack | Evals Lab | Public domain |
+| `train-delay.txt` | Written for this pack | Evals Lab | Public domain |
+| `volunteer-day.txt` | Written for this pack | Evals Lab | Public domain |
+| `weather-report.txt` | Written for this pack | Evals Lab | Public domain |
+| `https-api.weather.gov-alerts-types.txt` | https://api.weather.gov/alerts/types | National Weather Service | Public domain |

package/lab/demo/datasets/demo-1.json

@@ -2,12 +2,12 @@
   "format": "evals-lab/dataset",
   "version": 6,
   "dataset": {
-    "name": "Demo 1",
+    "name": "Demo: describe images",
     "body": {
       "version": 6,
       "source": {
         "id": "Demo 1",
-        "name": "Demo 1"
+        "name": "Demo: describe images"
       },
       "cases": [
         {

package/lab/demo/datasets/demo-2.json

@@ -2,12 +2,12 @@
   "format": "evals-lab/dataset",
   "version": 6,
   "dataset": {
-    "name": "Demo 2",
+    "name": "Demo: describe images 2",
     "body": {
       "version": 6,
       "source": {
         "id": "Demo 2",
-        "name": "Demo 2"
+        "name": "Demo: describe images 2"
       },
       "cases": [
         {