evals-lab 0.14.0 → 0.16.0

package/CHANGELOG.md

@@ -5,6 +5,103 @@ Newest first. Read a release's **Upgrade notes** before installing it: an
 upgrade can rewrite what the lab keeps in your data directory, and an older
 version cannot always read it back.
 
+## 0.16.0
+
+### Upgrade notes
+
+- **Copy your data directory before upgrading** (see "User data" in the
+  README). An experiment 0.16.0 saves is pipeline version 20, which 0.15.x
+  cannot read. To go back, reinstall 0.15.0 and restore the copy.
+- An experiment's jobs and targets become one Steps list in run order (Get
+  items, Request, Read response, Filter), with Variants beside it. Every
+  experiment runs and grades as it did.
+- The lab no longer ships presets of its own: presets come from a pack.
+
+### Added
+
+- The Experiment page is a Steps list: Add step, Move up and down, Clone and
+  Remove, and a step out of order says where it goes.
+- A Variants table under the steps: each Variant reads (Same) or holds an
+  override, its prompt included. Add variants from presets fills it from a
+  pack's presets.
+- An experiment can have several Get items steps; Results groups rows by
+  Source.
+- A flow's Request chooses who answers it: the Request, its Recorded
+  answer, or Echo.
+- Score with latest Eval: Results shows how kept responses grade against the
+  Eval as it reads now, without asking a model.
+- Add eval case on a Results row that has none.
+- History opens on Overview: an experiment's variants over its recent runs,
+  Previous performance with the best marked, New failures and New passing,
+  and an Items matrix.
+- Guides on Home, replacing Setup › Wizards. Old links still land.
+- `evals-lab presets compare` sets an app's decoding settings beside a lab's
+  or a pack's presets.
+
+### Changed
+
+- On screen: Experiments, Eval, Eval Cases, Variant, Item and Response.
+  Nothing stored is renamed.
+- Setup calls an AI connection only when a button is pressed. Model is
+  listed as its picker opens.
+- Errors and statuses read the same everywhere, with field errors on the
+  field.
+- Empty values read as a short parenthetical, such as (Not set).
+- Runs are named by when they ran.
+- History, Library and Setup reopen the section last used.
+- The Evals Lab title links to Home.
+
+### Fixed
+
+- An open Request fits a phone's width.
+- History's experiment filter lists each experiment once.
+- Items strips keep their width in Firefox.
+- An experiment opened unchanged is no longer saved again.
+
+## 0.15.0
+
+### Upgrade notes
+
+- **Copy your data directory before upgrading** (see "User data" in the
+  README). A pipeline 0.15.0 saves is pipeline version 19 and a filter set is
+  version 4, neither of which 0.14.x can read. To go back, reinstall 0.14.0
+  and restore the copy.
+- A job's Pre-format and Post-format filters become one Filters list, in run
+  order around its Response Format, and each filter becomes one of seven
+  verbs. Every reply reads as it did, except a filter that was linked where
+  it never ran: it now runs where it sits.
+
+### Added
+
+- Filters are seven verbs, in one Filters table with the Response Format as
+  a row.
+- A typed prompt edited between runs becomes its next version, not a new
+  prompt.
+- Results show a spinner on the row a run is processing, and (Not run) on
+  rows a stopped run never reached.
+- Enter in a picker's search picks the top match.
+
+### Changed
+
+- A picker ends in Cancel and a viewer in Close.
+- Escape closes the innermost dialog or menu, and focus returns to what
+  opened it.
+- Enter presses the main button in more dialogs.
+- Links are amber.
+- Keyboard and screen readers: focus rings are no longer clipped, menus
+  move with the arrow keys, and Results' run tabs are a proper tab list.
+- When another tab saved first, the conflict bar keeps that copy by default,
+  with one button to override it.
+- A dynamic Source's Status is its newest fetch's, and its button says Get
+  response.
+- The Sources toolbar stays on one row on a phone, and a tile's quota says
+  what it measures.
+
+### Fixed
+
+- Add source adds one Source, not two.
+- A focus trap skips disabled buttons.
+
 ## 0.14.0
 
 ### Upgrade notes

package/README.md

@@ -4,10 +4,10 @@ Test what a model, an HTTP endpoint or an API answers, side by side, and score
 it with evals you define in advance. Runs locally in a friendly UI, and
 doesn't send your data anywhere you haven't pointed it.
 
-A pipeline takes items from a **Source** and sends each one to one or more
-**Targets**, then scores the **Responses**.
+An experiment takes items from a **Source** and sends each one to one or more
+**Variants**, then scores the **Responses**.
 
-- **Targets** can be models, local (Ollama, llama.cpp) or hosted (Anthropic,
+- **Variants** can be models, local (Ollama, llama.cpp) or hosted (Anthropic,
   OpenAI, any compatible API), or **any HTTP endpoint**: your own service,
   an internal API, a gateway in front of a model. The lab sends each request
   as you write it, with the key held in a connection rather than in the
@@ -18,8 +18,8 @@ A pipeline takes items from a **Source** and sends each one to one or more
   a fresh one at run time.
 
 Evals include parsing checks, item counts, comparison with production
-replies, grading by a model, and eval groups (datasets) shared between
-pipelines. Run history is kept and can be exported.
+responses, grading by a model, and evals (evals) shared between
+experiments. Run history is kept and can be exported.
 
 The same evals run in CI: `evals-lab run` fails the build when an eval
 fails, like a failing unit test.
@@ -33,7 +33,7 @@ Example uses:
   keywords it returns
 - Comparing two models, or two prompts, on the same items and seeing which
   scores higher
-- Grading what an API returns against a dataset of expected answers
+- Grading what an API returns against an eval of expected answers
 - Checking which model can meet the evals in a Power Automate workflow
 - Sending images to a model with vision and comparing its answers for
   accuracy
@@ -45,8 +45,8 @@ npm install -g evals-lab
 evals-lab
 ```
 
-Open <http://localhost:8080>. A new lab starts with six demos, each a dataset
-and its pipeline, which run once you add an AI Connection in Setup. Note the first start takes longer while it installs them. Without
+Open <http://localhost:8080>. A new lab starts with six demos, each an eval
+and its experiment, which run once you add an AI Connection in Setup. Note the first start takes longer while it installs them. Without
 installing: `npx evals-lab`.
 
 | Needs | Windows | Linux |
@@ -89,7 +89,7 @@ evals-lab [--port <n>] [--host <address>] [--data-dir <dir>]
 
 ## In CI
 
-A pipeline can run with no lab, from a repository. In the lab, the pipeline's
+An experiment can run with no lab, from a repository. In the lab, the experiment's
 menu › **Export for CI…** downloads it as a bundle: unzip it into the
 repository, then:
 
@@ -104,7 +104,7 @@ evals-lab run <bundle-dir | pipeline.yaml> [--items <dir>] [--json <file|->]
 
 | Option | |
 |---|---|
-| `--items` | the items the pipeline's Source reads; default the bundle's `items/` |
+| `--items` | the items the experiment's Source reads; default the bundle's `items/` |
 | `--json` | the JSON report; `-` for stdout |
 | `--junit` | JUnit XML, for a CI test panel |
 | `--summary` | the results as Markdown, added to the file: for GitHub, `"$GITHUB_STEP_SUMMARY"` |
@@ -124,7 +124,7 @@ goes to stderr.
 | `2` | incomplete: an item is missing, or did not run |
 | `3` | broken or refused |
 
-An item a case names that is not in the items is reported as Missing.
+An item an eval case names that is not in the items is reported as Missing.
 Images need ImageMagick, as in the lab.
 
 ### GitHub Actions
@@ -160,7 +160,7 @@ CI systems run the same command and read `--junit` in their test panel.
 
 ### Against a running lab
 
-CI can run a lab's own pipeline there instead, so the run shows in that lab's
+CI can run a lab's own experiment there instead, so the run shows in that lab's
 History and its keys never leave it:
 
 ```bash
@@ -175,13 +175,34 @@ evals-lab run --lab <url> --pipeline <name> [--wait] [--json <file|->]
 | Option | |
 |---|---|
 | `--lab` | the lab's address; `LAB_PASSWORD` is sent when it is set |
-| `--pipeline` | the pipeline, by its name or its id |
+| `--pipeline` | the experiment, by its name or its id |
 | `--wait` | wait for the run, and exit with the lab's verdict on it |
 
 Without `--wait` the run's id is printed once it is queued, and the exit code
 is `0`; `--json`, `--junit`, `--summary`, `--min-pass` and `--progress` need
 `--wait`. The exit codes are the ones above, and a cancelled run is `2`.
 
+### Presets
+
+An app that sends decoding settings of its own can compare them with the
+presets its pack gives the lab, as the request fields each sends:
+
+```bash
+evals-lab presets compare app-presets.json --pack my-app.labpack [--fail-on-difference]
+evals-lab presets compare app-presets.json --lab https://lab.example
+```
+
+It reports each preset as the same, different or on one side only, and exits
+`0`; `--fail-on-difference` exits `1` on a difference. The app's file names
+each preset's request fields:
+
+```json
+{ "format": "evals-lab/presets", "version": 1,
+  "presets": [{ "id": "repeat-1.2", "request": { "repeat_penalty": 1.2 } }] }
+```
+
+`evals-lab presets --help` has the rest.
+
 ## Getting started
 
 First set up a Setup profile: **Setup › Add profile**, then pick its type —
@@ -190,7 +211,7 @@ endpoint — and give its address, model and key. NB: An Ollama profile without
 
 Set up your data source(s) in Library.
 
-Then create a Pipeline to Run. Changes are auto-saved. Use the "Restore" menu to go back and undo any changes you don't want to keep.
+Then create an Experiment to Run. Changes are auto-saved. Use the "Restore" menu to go back and undo any changes you don't want to keep.
 
 ## Power Automate
 

package/bin/evals-lab.js

@@ -2,7 +2,8 @@
 // The npm package's launcher: `evals-lab` starts the lab's server and says
 // where it is, and `evals-lab run <bundle>` runs an Export for CI bundle with
 // no server at all, or `evals-lab run --lab <url>` a running lab's pipeline
-// (run.js).
+// (run.js). `evals-lab presets compare` sets an app's decoding settings
+// beside a lab's or a pack's presets (presets.js).
 //
 // The lab itself is server.py, a stdlib Python server, staged beside this file
 // in ../lab by build.js. What this adds is what a lab run by someone who only
@@ -32,6 +33,8 @@ const USAGE = `Usage: evals-lab [options]
        evals-lab run --lab <url> --pipeline <name> [--wait] [options]
        evals-lab export --lab <url> [--workspace <slug>] [--out <file>]
        evals-lab import <file> --lab <url>   (evals-lab import --help)
+       evals-lab presets compare <file> (--lab <url> | --pack <file>)
+                                         (evals-lab presets --help)
 
   --port <n>        the port to listen on. Default: $PORT, else 8080.
   --host <address>  the address to listen on. Default: $LAB_HOST, else
@@ -167,6 +170,10 @@ async function main() {
     process.exitCode = await require("./lab-file.js").labFile(process.argv[2], process.argv.slice(3));
     return;
   }
+  if (process.argv[2] === "presets") {
+    process.exitCode = await require("./presets.js").presets(process.argv.slice(3), { lab: LAB });
+    return;
+  }
   if (process.argv[2] === "run") {
     process.exitCode = await require("./run.js").run(process.argv.slice(3), { lab: LAB });
     return;

package/bin/presets.js

@@ -0,0 +1,234 @@
+// `evals-lab presets compare <file>` (#652, docs/presets.md): an app's
+// decoding settings beside the presets a lab or a pack offers, compared as
+// what each one sends.
+//
+// WHAT A PRESET SENDS. A preset is written in the lab's spelling
+// (repeatPenalty: "1.2") and an app's in whatever its own code says, so the
+// two are compared where they meet: the request fields. A lab preset's are
+// the fields its type's request carries with the preset applied and not
+// without it -- the core builds both, so this is what a run would send. The
+// app's file names its fields directly.
+//
+// A COMPARISON, NOT AN ASSERTION. It says what is the same, what differs and
+// what is only on one side, and exits 0; --fail-on-difference exits 1 when
+// anything is not the same, for a CI that wants that. 3 is a comparison
+// refused: a file that is not one, a lab that did not answer.
+const fs = require("fs");
+const path = require("path");
+const zlib = require("zlib");
+const { labApi, Refused } = require("./run.js");
+
+const EXIT = { same: 0, different: 1, broken: 3 };
+const FORMAT = "evals-lab/presets";
+
+const USAGE = `Usage: evals-lab presets compare <file> (--lab <url> | --pack <file>) [options]
+
+  <file>              the app's presets: { "format": "${FORMAT}", "version": 1,
+                      "presets": [{ "id", "type"?, "request": { field: value } }] }
+  --lab <url>         compare with a running lab's presets. LAB_PASSWORD is sent
+                      when it is set.
+  --workspace <slug>  the lab's workspace, by its /w/<slug> address. Default:
+                      the lab's default workspace.
+  --pack <file>       compare with a pack's presets: a .labpack, or its
+                      presets.json.
+  --json <file|->     write the comparison as JSON. "-" means stdout.
+  --fail-on-difference
+                      exit 1 when anything differs or is on one side only.
+  --help              this.
+
+Exit codes: 0 compared, 1 a difference with --fail-on-difference, 3 refused.`;
+
+function options(argv) {
+  const o = { file: null, lab: null, workspace: null, pack: null, json: null, failOnDifference: false };
+  const value = { "--lab": "lab", "--workspace": "workspace", "--pack": "pack", "--json": "json" };
+  const [verb, ...rest] = argv;
+  if (verb === "--help" || verb === "-h" || verb === undefined) return { help: true };
+  if (verb !== "compare") throw new Refused(`no presets command ${verb}`, true);
+  for (let i = 0; i < rest.length; i++) {
+    const a = rest[i];
+    if (a === "--help" || a === "-h") return { help: true };
+    if (a === "--fail-on-difference") o.failOnDifference = true;
+    else if (value[a]) {
+      if (rest[i + 1] === undefined) throw new Refused(`${a} needs a value`, true);
+      o[value[a]] = rest[++i];
+    } else if (a.startsWith("-") && a !== "-") throw new Refused(`no option ${a}`, true);
+    else if (o.file === null) o.file = a;
+    else throw new Refused(`one file to compare, not ${a} as well`, true);
+  }
+  if (o.file === null) throw new Refused("name the app's presets file", true);
+  if ((o.lab === null) === (o.pack === null)) throw new Refused("compare with one of --lab or --pack", true);
+  if (o.lab !== null && !/^https?:\/\/[^/]/i.test(o.lab)) throw new Refused(`--lab ${o.lab} is not an http or https address`, true);
+  if (o.workspace !== null && o.lab === null) throw new Refused("--workspace is --lab's", true);
+  return o;
+}
+
+// The entries of a zip, by name, as Buffers. A pack is a plain zip, stored
+// or deflated, so this reads the central directory and nothing fancier.
+function unzip(buf) {
+  let end = buf.length - 22;
+  while (end >= 0 && buf.readUInt32LE(end) !== 0x06054b50) end--;
+  if (end < 0) throw new Error("not a zip");
+  const count = buf.readUInt16LE(end + 10);
+  let at = buf.readUInt32LE(end + 16);
+  const files = {};
+  for (let n = 0; n < count; n++) {
+    if (buf.readUInt32LE(at) !== 0x02014b50) throw new Error("a broken zip directory");
+    const method = buf.readUInt16LE(at + 10), size = buf.readUInt32LE(at + 20);
+    const nameLen = buf.readUInt16LE(at + 28), extraLen = buf.readUInt16LE(at + 30), noteLen = buf.readUInt16LE(at + 32);
+    const local = buf.readUInt32LE(at + 42);
+    const name = buf.toString("utf8", at + 46, at + 46 + nameLen);
+    const start = local + 30 + buf.readUInt16LE(local + 26) + buf.readUInt16LE(local + 28);
+    const raw = buf.subarray(start, start + size);
+    if (method === 0) files[name] = raw;
+    else if (method === 8) files[name] = zlib.inflateRawSync(raw);
+    at += 46 + nameLen + extraLen + noteLen;
+  }
+  return files;
+}
+
+// A pack's presets: from its .labpack, or from its presets.json as it is.
+function packPresets(file) {
+  let buf;
+  try { buf = fs.readFileSync(path.resolve(file)); }
+  catch (e) { throw new Refused(`${file}: ${e.message}`); }
+  let text = buf.toString("utf8");
+  if (buf.readUInt32LE(0) === 0x04034b50) {
+    let entries;
+    try { entries = unzip(buf); } catch (e) { throw new Refused(`${file}: ${e.message}`); }
+    if (!entries["presets.json"]) return [];
+    text = entries["presets.json"].toString("utf8");
+  }
+  let list;
+  try { list = JSON.parse(text); } catch (e) { throw new Refused(`${file}: ${e.message}`); }
+  if (!Array.isArray(list) || !list.every(p => p && typeof p.id === "string" && p.opts && typeof p.opts === "object")) {
+    throw new Refused(`${file}: a pack's presets are a list of { id, label, type?, opts }`);
+  }
+  return list;
+}
+
+// The app's file, or a Refused saying what is wrong with it.
+function appPresets(file, text) {
+  let doc;
+  try { doc = JSON.parse(text); } catch (e) { throw new Refused(`${file}: ${e.message}`); }
+  if (doc?.format !== FORMAT) throw new Refused(`${file} is not an ${FORMAT} file`);
+  if (doc.version !== 1) throw new Refused(`${file} is ${FORMAT} version ${doc.version}, and this reads 1`);
+  const list = doc.presets;
+  if (!Array.isArray(list) || !list.every(p => p && typeof p.id === "string"
+      && p.request && typeof p.request === "object" && !Array.isArray(p.request))) {
+    throw new Refused(`${file}: presets are a list of { id, type?, request }`);
+  }
+  return list;
+}
+
+const typeOf = p => p.type ?? "llama.cpp";
+
+// The request fields a lab [preset] sends: what its type's request carries
+// with it applied and not without.
+function presetRequest(core, preset) {
+  const conn = { type: typeOf(preset), model: "m", url: "" };
+  const body = c => {
+    const b = { ...core.connectionRequest(c, "p", null, "").body };
+    delete b.messages;
+    return b;
+  };
+  const base = body(conn), sent = body({ ...conn, ...preset.opts });
+  return Object.fromEntries(Object.entries(sent).filter(([k, v]) => JSON.stringify(base[k]) !== JSON.stringify(v)));
+}
+
+const same = (a, b) => JSON.stringify(a) === JSON.stringify(b);
+
+function unique(list, where) {
+  const ids = list.map(p => p.id);
+  const twice = ids.find((id, n) => ids.indexOf(id) !== n);
+  if (twice !== undefined) throw new Refused(`${where} has two presets named ${JSON.stringify(twice)}`);
+}
+
+/**
+ * The comparison of the lab's [lab] presets (its spelling) with the app's
+ * [app] ones (request fields): a row per preset id, the lab's in their
+ * order and then the app's only, each with its fields.
+ */
+function compare(core, lab, app) {
+  unique(lab, "the lab"); unique(app, "the file");
+  const theirs = new Map(app.map(p => [p.id, p]));
+  const rows = [];
+  for (const p of lab) {
+    let sent;
+    try { sent = presetRequest(core, p); }
+    catch (e) { throw new Refused(`the lab's preset ${p.id}: ${e.message}`); }
+    const other = theirs.get(p.id);
+    if (!other) { rows.push({ id: p.id, status: "lab only", lab: sent }); continue; }
+    const fields = [];
+    if (typeOf(p) !== typeOf(other)) fields.push({ field: "(type)", status: "different", lab: typeOf(p), file: typeOf(other) });
+    for (const k of [...new Set([...Object.keys(sent), ...Object.keys(other.request)])].sort()) {
+      if (!(k in other.request)) fields.push({ field: k, status: "lab only", lab: sent[k] });
+      else if (!(k in sent)) fields.push({ field: k, status: "file only", file: other.request[k] });
+      else fields.push({ field: k, status: same(sent[k], other.request[k]) ? "same" : "different", lab: sent[k], file: other.request[k] });
+    }
+    rows.push({ id: p.id, status: fields.every(f => f.status === "same") ? "same" : "different", fields });
+  }
+  const ours = new Set(lab.map(p => p.id));
+  for (const p of app) if (!ours.has(p.id)) rows.push({ id: p.id, status: "file only", file: p.request });
+  return rows;
+}
+
+const shown = v => {
+  const s = JSON.stringify(v);
+  return s.length > 60 ? `${s.slice(0, 59)}…` : s;
+};
+
+// The comparison as text: a line per preset, then one per field not the same.
+function table(rows) {
+  if (!rows.length) return "No presets on either side.";
+  const width = Math.max(...rows.map(r => r.id.length));
+  const lines = [];
+  for (const r of rows) {
+    lines.push(`${r.id.padEnd(width)}  ${r.status}`);
+    const fields = r.fields ? r.fields.filter(f => f.status !== "same")
+      : Object.entries(r.lab ?? r.file).map(([field, v]) => ({ field, status: r.status, [r.lab ? "lab" : "file"]: v }));
+    for (const f of fields) {
+      const said = f.status === "different" ? `lab ${shown(f.lab)}, file ${shown(f.file)}`
+        : f.status === "lab only" ? `lab ${shown(f.lab)}` : `file ${shown(f.file)}`;
+      lines.push(`${"".padEnd(width)}    ${f.field}: ${said}`);
+    }
+  }
+  return lines.join("\n");
+}
+
+/**
+ * Runs [argv] -- what follows `evals-lab presets` -- against the staged core
+ * in [lab], and resolves to the exit code.
+ */
+async function presets(argv, { lab, env = process.env, stdout = process.stdout, stderr = process.stderr } = {}) {
+  const say = text => stderr.write(text + "\n");
+  try {
+    const o = options(argv);
+    if (o.help) { stdout.write(USAGE + "\n"); return 0; }
+    const corePath = path.join(lab, "evals-core.mjs");
+    if (!fs.existsSync(corePath)) {
+      throw new Refused(`no ${corePath} -- in a checkout the package is staged by \`node npm/build.js\``);
+    }
+    const core = require(corePath);
+    let text;
+    try { text = fs.readFileSync(path.resolve(o.file), "utf8"); }
+    catch (e) { throw new Refused(`${o.file}: ${e.message}`); }
+    const app = appPresets(o.file, text);
+    const ours = o.pack !== null ? packPresets(o.pack)
+      : await labApi(o.lab.replace(/\/+$/, ""), env, o.workspace)("GET", "/api/presets");
+    const rows = compare(core, ours, app);
+    const out = table(rows);
+    if (o.json === "-") { stdout.write(JSON.stringify({ presets: rows }, null, 2) + "\n"); say(out); }
+    else {
+      if (o.json) fs.writeFileSync(path.resolve(o.json), JSON.stringify({ presets: rows }, null, 2) + "\n");
+      stdout.write(out + "\n");
+    }
+    return o.failOnDifference && rows.some(r => r.status !== "same") ? EXIT.different : EXIT.same;
+  } catch (e) {
+    if (!(e instanceof Refused)) throw e;
+    say(`evals-lab presets: ${e.message}`);
+    if (e.usage) say(`\n${USAGE}`);
+    return EXIT.broken;
+  }
+}
+
+module.exports = { presets, options, compare, presetRequest, packPresets, appPresets, unzip, table, USAGE, FORMAT };

package/lab/VERSION

@@ -1 +1 @@
-0.14.0 (2026.10.14-830)
+0.16.0 (2026.10.16-944)

package/lab/demo/manifest.json

@@ -3,9 +3,9 @@
   "version": 1,
   "id": "demo",
   "name": "Demo",
-  "packVersion": "2026.10.14.2",
+  "packVersion": "2026.10.14.3",
   "requires": {
-    "lab": ">=18",
+    "lab": ">=19",
     "plugins": []
   },
   "description": "Six demos, each a dataset with a starter pipeline: two sets of public-domain images, short texts to classify, one question in four wordings, a dynamic Source's snapshot, and PDFs to check and sort."