evals-lab 0.6.0 → 0.8.0

package/CHANGELOG.md

@@ -5,6 +5,114 @@ Newest first. Read a release's **Upgrade notes** before installing it: an
 upgrade can rewrite what the lab keeps in your data directory, and an older
 version cannot always read it back.
 
+## 0.8.0
+
+### Upgrade notes
+
+- No stored document changes format: pipelines stay version 13, and eval
+  groups and exports stay version 8, so 0.7.0 still reads what 0.8.0 saves.
+- On first start, 0.8.0 adds a table recording which workspaces may use each
+  Target profile and account. Nothing is narrowed: every connection stays
+  shared with every workspace until you change it.
+
+### Added
+
+- Several workspaces. Make, rename and archive them in Setup › Workspaces,
+  and switch between them from the header's workspace menu. Each one has its
+  own address (`/w/<slug>`) and its own datasets, prompts, Sources, pipelines
+  and runs. The theme choice (Dark, Light, Match system) has moved into that
+  menu.
+- Target profiles and accounts can be shared with every workspace, with some,
+  or with each new one. A run in a workspace a profile isn't shared with says
+  so and offers to share it.
+- `evals-lab run --lab … --workspace <slug>` runs a lab's pipeline in a named
+  workspace. Without it, the default workspace is used, as before.
+- Setup › Wizards, and a header pill showing a wizard's progress. The first
+  wizard, Test a Power Automate workflow, ticks off each step as you do it.
+  Plugins can add workflow platforms and wizards.
+- Judged against recorded has an **Expected** setting: tick which of Better,
+  Same and Worse pass (Better and Same by default, as before).
+- A Recorded target's menu has **Preview replies…**, showing what each call
+  recorded.
+- Test… asks for a grader when a check needs one. You can create the test
+  without one and add it later.
+
+### Changed
+
+- Model-graded checks no longer need a prompt written. The grader is told
+  what the target was asked; Judged against recorded's Task is now optional
+  Additional guidance.
+- A grader's answer is held to a fixed JSON shape where its API supports
+  that (Anthropic, OpenAI-compatible, llama.cpp, Ollama). An answer that
+  still can't be read is asked for once more, then reported as a grader
+  error rather than a verdict.
+- Test… on a step whose reply is plain words judges the reply against the
+  recorded one, instead of requiring the exact same words.
+- Rubric's and Factual's threshold reads **Expected: score at least**.
+- New profile: Anthropic and Ollama (Cloud) no longer ask for an address. A
+  proxy or gateway address goes under Advanced. Temperature is hidden, and
+  never sent, for models that reject it (Claude Opus 4.7 and later, the
+  5-series Claude models, OpenAI reasoning models). Image settings are now
+  the folded Vision Capabilities group.
+- The Test… dialog is "Create new pipeline test", with a Create button.
+- Library › Sources › a flow: Actions come before Calls, and each call's
+  request reads In sync or Out of sync.
+- Run results: Take B as expected is in each call's row menu, and the
+  Results table no longer has checkboxes.
+- The wizard pill's button is Open wizard, which shows the wizard's steps.
+
+### Fixed
+
+- An HTTP endpoint profile whose address included a path (such as
+  `https://api.anthropic.com/v1/messages`) sent the path twice, so every
+  call came back Not found.
+- An Anthropic profile with a blank address was sent to the lab's own Ollama.
+- A run said "this run has none" for a grader that its Library eval group
+  named.
+- The wizard pill didn't tick a step until the page was reloaded.
+- A Microsoft 365 sign-in now shows which tenants it reaches, so a lab that
+  works across tenants is not tied to one.
+
+## 0.7.0
+
+### Upgrade notes
+
+- **Copy your data directory before upgrading** (see "User data" in the
+  README). An eval group 0.7.0 saves is dataset version 8, which 0.6.x
+  cannot read. To go back, reinstall 0.6.0 and restore the copy.
+- On its first start, 0.7.0 moves everything the lab keeps into one
+  workspace, Default. Nothing on the page changes.
+- Stored eval groups are not rewritten on upgrade: a version-7 group reads
+  as version 8, and is saved as version 8 at its next edit.
+- Export JSON writes version 8, which 0.6.x refuses to import. Import reads
+  versions 1 to 8.
+- The metrics that compare a reply with production's are now the Recorded
+  reply metrics. Groups that use them grade as before.
+
+### Added
+
+- Test… on a Power Automate Source's action makes an eval group from the
+  Source's calls, graded against the replies production recorded, and a
+  pipeline to run it, then opens it in Runs.
+- The Recorded target replays each call's recorded reply and sends nothing,
+  so a run compares an edited request with what production did.
+- Adding a Power Automate Source imports the calls of its last five runs in
+  the background, while the tab stays open.
+- Results of a run with a Recorded target: a table of each check with a
+  column per Target (n/a where a check cannot apply), a call that opens both
+  replies side by side, and Take B as expected, which makes the selected
+  calls expect Target B's reply.
+- An HTTP Request target shows its words with the item's fields as chips.
+  Select prompt… and Save to library… use the Prompt library, and Copy
+  request… writes the request back for Power Automate, or as JSON.
+
+### Changed
+
+- A Power Automate Source's records are its Calls, with Add call.
+- Targets sit side by side on a wide screen. A job's Content and Responses
+  fold to one line until opened.
+- A row with one action shows it as a button; two or more are in its ⋯ menu.
+
 ## 0.6.0
 
 ### Added

package/bin/run.js

@@ -51,6 +51,8 @@ const USAGE = `Usage: evals-lab run <bundle-dir | pipeline.yaml> [options]
                      and their keys, and the run in its History. LAB_PASSWORD
                      is sent when it is set.
   --pipeline <name>  the lab's pipeline to run, by its name or its id.
+  --workspace <slug> the lab's workspace to run in, by its /w/<slug> address.
+                     Default: the lab's default workspace.
   --wait             wait for the lab's run to finish, and exit with its
                      verdict. Without it the run's id is printed once it is
                      queued. --json, --junit, --summary, --min-pass and
@@ -70,9 +72,9 @@ class Refused extends Error {
 // What the command line asks for, or a Refused saying what is wrong with it.
 function runOptions(argv) {
   const o = { bundle: null, items: null, json: null, junit: null, summary: null, minPass: null, progress: false,
-              lab: null, pipeline: null, wait: false };
+              lab: null, pipeline: null, workspace: null, wait: false };
   const value = { "--items": "items", "--json": "json", "--junit": "junit", "--summary": "summary", "--min-pass": "minPass",
-                  "--lab": "lab", "--pipeline": "pipeline" };
+                  "--lab": "lab", "--pipeline": "pipeline", "--workspace": "workspace" };
   for (let i = 0; i < argv.length; i++) {
     const a = argv[i];
     if (a === "--help" || a === "-h") o.help = true;
@@ -96,6 +98,8 @@ function runOptions(argv) {
     if (waits.length && !o.wait) {
       throw new Refused(`--${waits[0].replace("minPass", "min-pass")} reads the finished run: add --wait`, true);
     }
+  } else if (o.workspace != null) {
+    throw new Refused("--workspace is for a lab's run: name the lab with --lab", true);
   } else if (o.wait) {
     throw new Refused("--wait is for a lab's run: a bundle's always waits", true);
   } else if (o.bundle == null) {
@@ -276,15 +280,18 @@ const POLL_MISSES = 20;
 
 // The lab's API at [base], as [env]'s LAB_PASSWORD opens it: HTTP Basic, the
 // password alone, as server.py takes it. A refusal says what the lab said.
-function labApi(base, env) {
+// [workspace], when given, is the slug every call carries as X-Workspace, so
+// the run reads and queues in that workspace rather than the lab's default.
+function labApi(base, env, workspace) {
   const auth = env.LAB_PASSWORD
     ? { Authorization: `Basic ${Buffer.from(`:${env.LAB_PASSWORD}`).toString("base64")}` } : {};
+  const scope = workspace ? { "X-Workspace": workspace } : {};
   return async (method, route, body, { missing = false } = {}) => {
     let res;
     try {
       res = await fetch(base + route, {
         method, redirect: "manual",
-        headers: { ...auth, ...(body !== undefined ? { "Content-Type": "application/json" } : {}) },
+        headers: { ...auth, ...scope, ...(body !== undefined ? { "Content-Type": "application/json" } : {}) },
         ...(body !== undefined ? { body: JSON.stringify(body) } : {}),
       });
     } catch (e) {
@@ -319,6 +326,21 @@ function pipelineNamed(saved, name, base) {
   return found[0];
 }
 
+// The lab's workspace --workspace [slug] names, by its /w/<slug> address -- or
+// a Refused naming the slugs it has, as pipelineNamed does for pipelines. The
+// server falls back to its default for an unknown slug, so the CLI checks the
+// slug itself; an archived workspace cannot be run against.
+function workspaceNamed(workspaces, slug, base) {
+  const active = workspaces.filter(w => w && !w.archived);
+  const found = active.find(w => w.slug === slug);
+  if (found) return found;
+  if (workspaces.some(w => w && w.slug === slug)) {
+    throw new Refused(`${base}'s workspace at /w/${slug} is archived`);
+  }
+  throw new Refused(`${base} has no workspace at /w/${slug}`
+    + (active.length ? `: it has ${active.map(w => JSON.stringify(w.slug)).join(", ")}` : ""));
+}
+
 // The row's verdicts with --min-pass relaxing each eval read item by item,
 // as the worker's --min-pass does: it only ever turns a fail into a pass.
 const relaxed = (verdicts, minPass) => minPass == null ? verdicts : verdicts.map(evals =>
@@ -356,7 +378,16 @@ function rowSuites(core, run, verdicts, items) {
 /** [o]'s pipeline, queued on its lab; with --wait, followed to its verdict. */
 async function runOnLab(core, o, { env, stdout, say }) {
   const base = o.lab.replace(/\/+$/, "");
-  const call = labApi(base, env);
+  // Resolve --workspace to the slug every call carries; the lab's pipelines,
+  // Sources and eval groups are per-workspace, so the whole run reads and
+  // queues inside it. Without it, the lab's default workspace is used and the
+  // X-Workspace header is omitted, so existing CI is unchanged.
+  let workspace = null;
+  if (o.workspace != null) {
+    const { workspaces } = await labApi(base, env)("GET", "/api/workspaces");
+    workspace = workspaceNamed(Array.isArray(workspaces) ? workspaces : [], o.workspace, base).slug;
+  }
+  const call = labApi(base, env, workspace);
 
   // What the page reads before it submits: the lab's pipelines and Target
   // profiles, the Source the pipeline reads and the eval groups it may link.

package/lab/VERSION

@@ -1 +1 @@
-0.6.0 (2026.10.05-430)
+0.8.0 (2026.10.07-484)