evals-lab 0.5.0 → 0.7.0

package/CHANGELOG.md

@@ -5,6 +5,83 @@ Newest first. Read a release's **Upgrade notes** before installing it: an
 upgrade can rewrite what the lab keeps in your data directory, and an older
 version cannot always read it back.
 
+## 0.7.0
+
+### Upgrade notes
+
+- **Copy your data directory before upgrading** (see "User data" in the
+  README). An eval group 0.7.0 saves is dataset version 8, which 0.6.x
+  cannot read. To go back, reinstall 0.6.0 and restore the copy.
+- On its first start, 0.7.0 moves everything the lab keeps into one
+  workspace, Default. Nothing on the page changes.
+- Stored eval groups are not rewritten on upgrade: a version-7 group reads
+  as version 8, and is saved as version 8 at its next edit.
+- Export JSON writes version 8, which 0.6.x refuses to import. Import reads
+  versions 1 to 8.
+- The metrics that compare a reply with production's are now the Recorded
+  reply metrics. Groups that use them grade as before.
+
+### Added
+
+- Test… on a Power Automate Source's action makes an eval group from the
+  Source's calls, graded against the replies production recorded, and a
+  pipeline to run it, then opens it in Runs.
+- The Recorded target replays each call's recorded reply and sends nothing,
+  so a run compares an edited request with what production did.
+- Adding a Power Automate Source imports the calls of its last five runs in
+  the background, while the tab stays open.
+- Results of a run with a Recorded target: a table of each check with a
+  column per Target (n/a where a check cannot apply), a call that opens both
+  replies side by side, and Take B as expected, which makes the selected
+  calls expect Target B's reply.
+- An HTTP Request target shows its words with the item's fields as chips.
+  Select prompt… and Save to library… use the Prompt library, and Copy
+  request… writes the request back for Power Automate, or as JSON.
+
+### Changed
+
+- A Power Automate Source's records are its Calls, with Add call.
+- Targets sit side by side on a wide screen. A job's Content and Responses
+  fold to one line until opened.
+- A row with one action shows it as a button; two or more are in its ⋯ menu.
+
+## 0.6.0
+
+### Added
+
+- Library › Evals (was Library › Datasets) lists every eval group: its
+  Source, cases, scoring, and the pipelines that link it. A group's page
+  holds its Grades, Every item, Whole run and cases.
+- Runs › Evals links eval groups to a pipeline. Link eval group picks from
+  the Library and marks the groups made for this content. A link follows a
+  group's latest version, or is pinned to one. A pipeline's own group is
+  edited in place, and Move to Library shares it.
+- A pipeline can link several eval groups. Its pass rule says whether every
+  one must pass, or at least a number. Results, History and `evals-lab run`
+  all grade by it.
+- An Undo button in the header takes back the last 20 actions, newest
+  first, until the page reloads.
+- Choose pipeline lists the 5 used most recently, then the rest, searchable
+  and sortable.
+- A Profile, Source or Grader field opens a picker, most recent first.
+
+### Changed
+
+- Results shows a row per eval group, with each Target's share and verdict,
+  then the overall verdict. History's Outcome is the overall verdict.
+- A dialog is a draft: nothing reaches the lab until Save & Close. Cancel,
+  × and Esc discard it.
+- An unnamed target is Target A, B, C.
+- The run bar sits under the pipeline's header. It says "12 of 42" while a
+  run goes and "42 in 2m" when it is done.
+- Verdicts show as a check or a cross, in Results and History.
+- A metric's "How it counts" is Options, and Grades is Reads: Parsed reply
+  or Raw reply.
+
+### Fixed
+
+- Add profile, then ×, added a profile.
+
 ## 0.5.0
 
 ### Upgrade notes

package/bin/run.js

@@ -487,10 +487,21 @@ async function run(argv, { lab, env = process.env, stdout = process.stdout, stde
     const args = [path.join(lab, "run-evals.js"), "--run", path.join(tmp, "run.json"),
                   "--json", path.join(tmp, "report.json")];
     if (items) args.push("--source", items);
-    const ref = core.evalsDataset(read.run);
-    if (ref && read.datasets[ref.id]) {
-      fs.writeFileSync(path.join(tmp, "dataset.json"), JSON.stringify(read.datasets[ref.id]));
+    // The eval group bodies the run grades against: its one, as --dataset, or
+    // a body per group it links, as --groups (#233), keyed by the id its
+    // reference carries (a bundle's refs record no version).
+    const ids = [...new Set(core.evalsOf(read.run).map(core.casesRef).filter(Boolean).map(r => r.id))];
+    if (ids.length === 1 && read.datasets[ids[0]]) {
+      fs.writeFileSync(path.join(tmp, "dataset.json"), JSON.stringify(read.datasets[ids[0]]));
       args.push("--dataset", path.join(tmp, "dataset.json"));
+    } else if (ids.length > 1) {
+      const groups = {};
+      for (const id of ids) {
+        if (!read.datasets[id]) throw new Refused(`${o.bundle}: the bundle holds no eval group ${id}`);
+        groups[id] = read.datasets[id];
+      }
+      fs.writeFileSync(path.join(tmp, "groups.json"), JSON.stringify(groups));
+      args.push("--groups", path.join(tmp, "groups.json"));
     }
     if (plugins.length) args.push("--plugins", path.join(dir, "plugins"));
     if (o.minPass != null) args.push("--min-pass", o.minPass);

package/lab/VERSION

@@ -1 +1 @@
-0.5.0 (2026.10.04-397)
+0.7.0 (2026.10.06-456)