evals-lab 0.7.0 → 0.9.0

package/CHANGELOG.md

@@ -5,6 +5,116 @@ Newest first. Read a release's **Upgrade notes** before installing it: an
 upgrade can rewrite what the lab keeps in your data directory, and an older
 version cannot always read it back.
 
+## 0.9.0
+
+### Upgrade notes
+
+- **Copy your data directory before upgrading** (see "User data" in the
+  README). A pipeline 0.9.0 saves is pipeline version 14, which 0.8.x
+  cannot read. To go back, reinstall 0.8.0 and restore the copy.
+- A job's Drop items and Reject the answer rules become one private filter
+  set in that job, which keeps and rejects exactly what they did. Pipelines
+  read as version 14, and are saved as it at their next edit.
+- On first start, 0.9.0 adds the tables filter sets are kept in, and each
+  run keeps the filter sets it ran with.
+
+### Added
+
+- Library › Filter sets: named, versioned lists of filters that clean a list
+  reply. A Drop item filter drops the items its conditions match; a Reject
+  reply filter rejects the whole reply. Filters run in their list's order.
+- A job's Responses stage links filter sets, following each one's latest
+  version or pinned to one, or holds a private set of its own.
+- Each Target profile shows its connection status: Not tested, Connected,
+  Key needed or Unreachable, checked quietly when Setup opens. The dot opens
+  the Test connection result.
+- Target profiles list each profile's Model and what uses it, with quick
+  filters: All, Connected, Needs attention and Grader. A profile's edit
+  sheet links the pipelines and eval groups that use it.
+- Selected profiles can be tested, or have their workspaces changed, all
+  at once.
+- An HTTP Request target's prompt can be edited. Insert field… adds one of
+  the item's fields, and the preview highlights the words changed from the
+  recorded prompt.
+- The Recorded target shows the flow's original prompt.
+
+### Changed
+
+- Target profiles no longer show Type, Address and Key columns, or the
+  Type, Key and Sort selects.
+- The Edited target's request template is in Flow's request, folded until
+  opened.
+- Drop items and Reject the answer are no longer offered under a job's
+  Responses: link a filter set instead.
+
+## 0.8.0
+
+### Upgrade notes
+
+- No stored document changes format: pipelines stay version 13, and eval
+  groups and exports stay version 8, so 0.7.0 still reads what 0.8.0 saves.
+- On first start, 0.8.0 adds a table recording which workspaces may use each
+  Target profile and account. Nothing is narrowed: every connection stays
+  shared with every workspace until you change it.
+
+### Added
+
+- Several workspaces. Make, rename and archive them in Setup › Workspaces,
+  and switch between them from the header's workspace menu. Each one has its
+  own address (`/w/<slug>`) and its own datasets, prompts, Sources, pipelines
+  and runs. The theme choice (Dark, Light, Match system) has moved into that
+  menu.
+- Target profiles and accounts can be shared with every workspace, with some,
+  or with each new one. A run in a workspace a profile isn't shared with says
+  so and offers to share it.
+- `evals-lab run --lab … --workspace <slug>` runs a lab's pipeline in a named
+  workspace. Without it, the default workspace is used, as before.
+- Setup › Wizards, and a header pill showing a wizard's progress. The first
+  wizard, Test a Power Automate workflow, ticks off each step as you do it.
+  Plugins can add workflow platforms and wizards.
+- Judged against recorded has an **Expected** setting: tick which of Better,
+  Same and Worse pass (Better and Same by default, as before).
+- A Recorded target's menu has **Preview replies…**, showing what each call
+  recorded.
+- Test… asks for a grader when a check needs one. You can create the test
+  without one and add it later.
+
+### Changed
+
+- Model-graded checks no longer need a prompt written. The grader is told
+  what the target was asked; Judged against recorded's Task is now optional
+  Additional guidance.
+- A grader's answer is held to a fixed JSON shape where its API supports
+  that (Anthropic, OpenAI-compatible, llama.cpp, Ollama). An answer that
+  still can't be read is asked for once more, then reported as a grader
+  error rather than a verdict.
+- Test… on a step whose reply is plain words judges the reply against the
+  recorded one, instead of requiring the exact same words.
+- Rubric's and Factual's threshold reads **Expected: score at least**.
+- New profile: Anthropic and Ollama (Cloud) no longer ask for an address. A
+  proxy or gateway address goes under Advanced. Temperature is hidden, and
+  never sent, for models that reject it (Claude Opus 4.7 and later, the
+  5-series Claude models, OpenAI reasoning models). Image settings are now
+  the folded Vision Capabilities group.
+- The Test… dialog is "Create new pipeline test", with a Create button.
+- Library › Sources › a flow: Actions come before Calls, and each call's
+  request reads In sync or Out of sync.
+- Run results: Take B as expected is in each call's row menu, and the
+  Results table no longer has checkboxes.
+- The wizard pill's button is Open wizard, which shows the wizard's steps.
+
+### Fixed
+
+- An HTTP endpoint profile whose address included a path (such as
+  `https://api.anthropic.com/v1/messages`) sent the path twice, so every
+  call came back Not found.
+- An Anthropic profile with a blank address was sent to the lab's own Ollama.
+- A run said "this run has none" for a grader that its Library eval group
+  named.
+- The wizard pill didn't tick a step until the page was reloaded.
+- A Microsoft 365 sign-in now shows which tenants it reaches, so a lab that
+  works across tenants is not tied to one.
+
 ## 0.7.0
 
 ### Upgrade notes

package/README.md

@@ -11,6 +11,7 @@ Runs are kept in history which can be exported.
 Currently in early release. Runs locally and doesn't send your data anywhere else.
 
 Example use-cases:
+- Keywording images using a model's vision capability, and running evals on the tags it returns
 - Providing images to an LLM and comparing responses for accuracy
 - Checking which model can meet evals in a Power Automate workflow
 - Determining which prompt gets the highest score in metrics

package/bin/run.js

@@ -51,6 +51,8 @@ const USAGE = `Usage: evals-lab run <bundle-dir | pipeline.yaml> [options]
                      and their keys, and the run in its History. LAB_PASSWORD
                      is sent when it is set.
   --pipeline <name>  the lab's pipeline to run, by its name or its id.
+  --workspace <slug> the lab's workspace to run in, by its /w/<slug> address.
+                     Default: the lab's default workspace.
   --wait             wait for the lab's run to finish, and exit with its
                      verdict. Without it the run's id is printed once it is
                      queued. --json, --junit, --summary, --min-pass and
@@ -70,9 +72,9 @@ class Refused extends Error {
 // What the command line asks for, or a Refused saying what is wrong with it.
 function runOptions(argv) {
   const o = { bundle: null, items: null, json: null, junit: null, summary: null, minPass: null, progress: false,
-              lab: null, pipeline: null, wait: false };
+              lab: null, pipeline: null, workspace: null, wait: false };
   const value = { "--items": "items", "--json": "json", "--junit": "junit", "--summary": "summary", "--min-pass": "minPass",
-                  "--lab": "lab", "--pipeline": "pipeline" };
+                  "--lab": "lab", "--pipeline": "pipeline", "--workspace": "workspace" };
   for (let i = 0; i < argv.length; i++) {
     const a = argv[i];
     if (a === "--help" || a === "-h") o.help = true;
@@ -96,6 +98,8 @@ function runOptions(argv) {
     if (waits.length && !o.wait) {
       throw new Refused(`--${waits[0].replace("minPass", "min-pass")} reads the finished run: add --wait`, true);
     }
+  } else if (o.workspace != null) {
+    throw new Refused("--workspace is for a lab's run: name the lab with --lab", true);
   } else if (o.wait) {
     throw new Refused("--wait is for a lab's run: a bundle's always waits", true);
   } else if (o.bundle == null) {
@@ -276,15 +280,18 @@ const POLL_MISSES = 20;
 
 // The lab's API at [base], as [env]'s LAB_PASSWORD opens it: HTTP Basic, the
 // password alone, as server.py takes it. A refusal says what the lab said.
-function labApi(base, env) {
+// [workspace], when given, is the slug every call carries as X-Workspace, so
+// the run reads and queues in that workspace rather than the lab's default.
+function labApi(base, env, workspace) {
   const auth = env.LAB_PASSWORD
     ? { Authorization: `Basic ${Buffer.from(`:${env.LAB_PASSWORD}`).toString("base64")}` } : {};
+  const scope = workspace ? { "X-Workspace": workspace } : {};
   return async (method, route, body, { missing = false } = {}) => {
     let res;
     try {
       res = await fetch(base + route, {
         method, redirect: "manual",
-        headers: { ...auth, ...(body !== undefined ? { "Content-Type": "application/json" } : {}) },
+        headers: { ...auth, ...scope, ...(body !== undefined ? { "Content-Type": "application/json" } : {}) },
         ...(body !== undefined ? { body: JSON.stringify(body) } : {}),
       });
     } catch (e) {
@@ -319,6 +326,21 @@ function pipelineNamed(saved, name, base) {
   return found[0];
 }
 
+// The lab's workspace --workspace [slug] names, by its /w/<slug> address -- or
+// a Refused naming the slugs it has, as pipelineNamed does for pipelines. The
+// server falls back to its default for an unknown slug, so the CLI checks the
+// slug itself; an archived workspace cannot be run against.
+function workspaceNamed(workspaces, slug, base) {
+  const active = workspaces.filter(w => w && !w.archived);
+  const found = active.find(w => w.slug === slug);
+  if (found) return found;
+  if (workspaces.some(w => w && w.slug === slug)) {
+    throw new Refused(`${base}'s workspace at /w/${slug} is archived`);
+  }
+  throw new Refused(`${base} has no workspace at /w/${slug}`
+    + (active.length ? `: it has ${active.map(w => JSON.stringify(w.slug)).join(", ")}` : ""));
+}
+
 // The row's verdicts with --min-pass relaxing each eval read item by item,
 // as the worker's --min-pass does: it only ever turns a fail into a pass.
 const relaxed = (verdicts, minPass) => minPass == null ? verdicts : verdicts.map(evals =>
@@ -356,7 +378,16 @@ function rowSuites(core, run, verdicts, items) {
 /** [o]'s pipeline, queued on its lab; with --wait, followed to its verdict. */
 async function runOnLab(core, o, { env, stdout, say }) {
   const base = o.lab.replace(/\/+$/, "");
-  const call = labApi(base, env);
+  // Resolve --workspace to the slug every call carries; the lab's pipelines,
+  // Sources and eval groups are per-workspace, so the whole run reads and
+  // queues inside it. Without it, the lab's default workspace is used and the
+  // X-Workspace header is omitted, so existing CI is unchanged.
+  let workspace = null;
+  if (o.workspace != null) {
+    const { workspaces } = await labApi(base, env)("GET", "/api/workspaces");
+    workspace = workspaceNamed(Array.isArray(workspaces) ? workspaces : [], o.workspace, base).slug;
+  }
+  const call = labApi(base, env, workspace);
 
   // What the page reads before it submits: the lab's pipelines and Target
   // profiles, the Source the pipeline reads and the eval groups it may link.

package/lab/VERSION

@@ -1 +1 @@
-0.7.0 (2026.10.06-456)
+0.9.0 (2026.10.07-516)

package/lab/demo/pipelines/demo-1.json

@@ -1,5 +1,5 @@
 {
-  "version": 13,
+  "version": 14,
   "id": "demo-1",
   "name": "Demo 1",
   "jobs": [

package/lab/demo/pipelines/demo-2.json

@@ -1,5 +1,5 @@
 {
-  "version": 13,
+  "version": 14,
   "id": "demo-2",
   "name": "Demo 2",
   "jobs": [