evals-lab 0.13.0 → 0.15.0

package/CHANGELOG.md

@@ -5,6 +5,89 @@ Newest first. Read a release's **Upgrade notes** before installing it: an
 upgrade can rewrite what the lab keeps in your data directory, and an older
 version cannot always read it back.
 
+## 0.15.0
+
+### Upgrade notes
+
+- **Copy your data directory before upgrading** (see "User data" in the
+  README). A pipeline 0.15.0 saves is pipeline version 19 and a filter set is
+  version 4, neither of which 0.14.x can read. To go back, reinstall 0.14.0
+  and restore the copy.
+- A job's Pre-format and Post-format filters become one Filters list, in run
+  order around its Response Format, and each filter becomes one of seven
+  verbs. Every reply reads as it did, except a filter that was linked where
+  it never ran: it now runs where it sits.
+
+### Added
+
+- Filters are seven verbs, in one Filters table with the Response Format as
+  a row.
+- A typed prompt edited between runs becomes its next version, not a new
+  prompt.
+- Results show a spinner on the row a run is processing, and (Not run) on
+  rows a stopped run never reached.
+- Enter in a picker's search picks the top match.
+
+### Changed
+
+- A picker ends in Cancel and a viewer in Close.
+- Escape closes the innermost dialog or menu, and focus returns to what
+  opened it.
+- Enter presses the main button in more dialogs.
+- Links are amber.
+- Keyboard and screen readers: focus rings are no longer clipped, menus
+  move with the arrow keys, and Results' run tabs are a proper tab list.
+- When another tab saved first, the conflict bar keeps that copy by default,
+  with one button to override it.
+- A dynamic Source's Status is its newest fetch's, and its button says Get
+  response.
+- The Sources toolbar stays on one row on a phone, and a tile's quota says
+  what it measures.
+
+### Fixed
+
+- Add source adds one Source, not two.
+- A focus trap skips disabled buttons.
+
+## 0.14.0
+
+### Upgrade notes
+
+- **Copy your data directory before upgrading** (see "User data" in the
+  README). A pipeline 0.14.0 saves is pipeline version 18, which 0.13.x
+  cannot read. To go back, reinstall 0.13.0 and restore the copy.
+- A job that graded a dynamic Source's saved content now grades one saved
+  response: on upgrade it is pinned to the Source's newest successful one.
+  Live data is unchanged. To grade several responses, clone the job.
+
+### Added
+
+- An endpoint or call keeps its responses: Fetch now, an import or an edit
+  adds one, and Restore brings an earlier one back.
+- A Source page lists Responses and History, and Saved response pins the
+  one response a job grades.
+- Results show one block per eval group, export as JSON or PDF, say when a
+  reply was cut off, and show where content goes in the prompt.
+- Post-format filters for a text reply, and more Rewrite text operations.
+- A sixth demo: sort documents, with valid and invalid PDFs.
+- The (Default) Connection Profile can be edited and applies to (Any)
+  connection.
+- History filters by pipeline and content.
+
+### Changed
+
+- A dialog says Close until something in it changes.
+- Demo datasets are named for their kind, such as "Demo: sentiment evals".
+- Response Format offers one text option, Plain text.
+- A link to something that is gone says so.
+- History fits a tablet, and Home and Results show long names on two lines.
+- More colours reach WCAG AA contrast in both themes.
+
+### Fixed
+
+- A job hands on its reply as its filters left it.
+- A Read Reply step's `preamble: false` is dropped on upgrade and on read.
+
 ## 0.13.0
 
 ### Upgrade notes

package/README.md

@@ -45,7 +45,7 @@ npm install -g evals-lab
 evals-lab
 ```
 
-Open <http://localhost:8080>. A new lab starts with five demos, each a dataset
+Open <http://localhost:8080>. A new lab starts with six demos, each a dataset
 and its pipeline, which run once you add an AI Connection in Setup. Note the first start takes longer while it installs them. Without
 installing: `npx evals-lab`.
 
@@ -164,7 +164,7 @@ CI can run a lab's own pipeline there instead, so the run shows in that lab's
 History and its keys never leave it:
 
 ```bash
-LAB_PASSWORD=... evals-lab run --lab https://lab.example --pipeline "Demo: describe images" --wait
+LAB_PASSWORD=... evals-lab run --lab https://lab.example --pipeline "Demo: images pipeline" --wait
 ```
 
 ```text

package/bin/run.js

@@ -399,8 +399,10 @@ async function runOnLab(core, o, { env, stdout, say }) {
   const docs = (await call("GET", "/api/state")).docs ?? {};
   const setup = docs["promptlab.connections"]?.body ?? {};
   const connections = core.withSlugs(Array.isArray(setup.list) ? setup.list : []);
-  const held = docs["promptlab.connectionProfiles"]?.body?.list;
-  const answers = core.answersOf(connections, Array.isArray(held) ? held : []);
+  const held = docs["promptlab.connectionProfiles"]?.body;
+  // (Default)'s settings, under each target that names no profile (#603).
+  const answers = core.answersOf(connections, Array.isArray(held?.list) ? held.list : [],
+                                 held?.default && typeof held.default === "object" ? held.default : null);
   const saved = docs["promptlab.workflows"]?.body?.list;
   const found = pipelineNamed(Array.isArray(saved) ? saved : [], o.pipeline, base);
   const doc = core.upgradePipeline(found.work, { profileType: id => answers.connection(id)?.type,

package/lab/VERSION

@@ -1 +1 @@
-0.13.0 (2026.10.13-794)
+0.15.0 (2026.10.14-867)

package/lab/demo/CREDITS.md

@@ -124,3 +124,17 @@ on 2026-10-11. A work of the US federal government is in the public domain.
 | `volunteer-day.txt` | Written for this pack | Evals Lab | Public domain |
 | `weather-report.txt` | Written for this pack | Evals Lab | Public domain |
 | `https-api.weather.gov-alerts-types.txt` | https://api.weather.gov/alerts/types | National Weather Service | Public domain |
+
+## Documents
+
+The PDFs of `documents/`, made for this pack: four short documents, a PDF cut
+short, and a text file with a `.pdf` name.
+
+| File | Source | Author | Licence |
+|---|---|---|---|
+| `invoice.pdf` | Written for this pack | Evals Lab | Public domain |
+| `letter.pdf` | Written for this pack | Evals Lab | Public domain |
+| `not-a-pdf.pdf` | Written for this pack | Evals Lab | Public domain |
+| `receipt.pdf` | Written for this pack | Evals Lab | Public domain |
+| `report.pdf` | Written for this pack | Evals Lab | Public domain |
+| `truncated.pdf` | Written for this pack | Evals Lab | Public domain |

package/lab/demo/datasets/demo-1.json

@@ -2,12 +2,12 @@
   "format": "evals-lab/dataset",
   "version": 6,
   "dataset": {
-    "name": "Demo: describe images",
+    "name": "Demo: images evals",
     "body": {
       "version": 6,
       "source": {
         "id": "Demo 1",
-        "name": "Demo: describe images"
+        "name": "Demo: images source"
       },
       "cases": [
         {

package/lab/demo/datasets/demo-2.json

@@ -2,12 +2,12 @@
   "format": "evals-lab/dataset",
   "version": 6,
   "dataset": {
-    "name": "Demo: describe images 2",
+    "name": "Demo: images 2 evals",
     "body": {
       "version": 6,
       "source": {
         "id": "Demo 2",
-        "name": "Demo: describe images 2"
+        "name": "Demo: images 2 source"
       },
       "cases": [
         {

package/lab/demo/datasets/documents-type.json

@@ -0,0 +1,79 @@
+{
+  "format": "evals-lab/eval-group",
+  "version": 9,
+  "group": {
+    "name": "Document type",
+    "preview": {
+      "id": "documents",
+      "name": "Demo: documents source"
+    },
+    "body": {
+      "version": 9,
+      "scoring": {
+        "mode": "all",
+        "threshold": null
+      },
+      "grader": null,
+      "every": [],
+      "run": [],
+      "cases": [
+        {
+          "id": "invoice",
+          "item": "invoice.pdf",
+          "todo": false,
+          "note": "",
+          "metrics": [
+            {
+              "type": "json-field",
+              "path": "type",
+              "op": "equals",
+              "value": "invoice"
+            }
+          ]
+        },
+        {
+          "id": "letter",
+          "item": "letter.pdf",
+          "todo": false,
+          "note": "",
+          "metrics": [
+            {
+              "type": "json-field",
+              "path": "type",
+              "op": "equals",
+              "value": "letter"
+            }
+          ]
+        },
+        {
+          "id": "receipt",
+          "item": "receipt.pdf",
+          "todo": false,
+          "note": "",
+          "metrics": [
+            {
+              "type": "json-field",
+              "path": "type",
+              "op": "equals",
+              "value": "receipt"
+            }
+          ]
+        },
+        {
+          "id": "report",
+          "item": "report.pdf",
+          "todo": false,
+          "note": "",
+          "metrics": [
+            {
+              "type": "json-field",
+              "path": "type",
+              "op": "equals",
+              "value": "report"
+            }
+          ]
+        }
+      ]
+    }
+  }
+}

package/lab/demo/datasets/documents-valid.json

@@ -0,0 +1,117 @@
+{
+  "format": "evals-lab/eval-group",
+  "version": 9,
+  "group": {
+    "name": "PDF is valid",
+    "preview": {
+      "id": "documents",
+      "name": "Demo: documents source"
+    },
+    "body": {
+      "version": 9,
+      "scoring": {
+        "mode": "weighted",
+        "threshold": 1
+      },
+      "grader": null,
+      "every": [],
+      "run": [],
+      "cases": [
+        {
+          "id": "invoice",
+          "item": "invoice.pdf",
+          "todo": false,
+          "note": "",
+          "metrics": [
+            {
+              "type": "json-field",
+              "path": "valid",
+              "op": "equals",
+              "value": "true"
+            }
+          ]
+        },
+        {
+          "id": "letter",
+          "item": "letter.pdf",
+          "todo": false,
+          "note": "",
+          "metrics": [
+            {
+              "type": "json-field",
+              "path": "valid",
+              "op": "equals",
+              "value": "true"
+            }
+          ]
+        },
+        {
+          "id": "receipt",
+          "item": "receipt.pdf",
+          "todo": false,
+          "note": "",
+          "metrics": [
+            {
+              "type": "json-field",
+              "path": "valid",
+              "op": "equals",
+              "value": "true"
+            }
+          ]
+        },
+        {
+          "id": "report",
+          "item": "report.pdf",
+          "todo": false,
+          "note": "",
+          "metrics": [
+            {
+              "type": "json-field",
+              "path": "valid",
+              "op": "equals",
+              "value": "true"
+            }
+          ]
+        },
+        {
+          "id": "truncated",
+          "item": "truncated.pdf",
+          "todo": false,
+          "note": "A real PDF cut short. Either the API refuses it or the model says it can't read it.",
+          "metrics": [
+            {
+              "type": "refused",
+              "weight": 1
+            },
+            {
+              "type": "json-field",
+              "path": "valid",
+              "op": "equals",
+              "value": "false",
+              "weight": 1
+            }
+          ]
+        },
+        {
+          "id": "not-a-pdf",
+          "item": "not-a-pdf.pdf",
+          "todo": false,
+          "note": "Plain text with a .pdf name. Either the API refuses it or the model says it can't read it.",
+          "metrics": [
+            {
+              "type": "refused",
+              "weight": 1
+            },
+            {
+              "type": "json-field",
+              "path": "valid",
+              "op": "equals",
+              "value": "false",
+              "weight": 1
+            }
+          ]
+        }
+      ]
+    }
+  }
+}

package/lab/demo/datasets/sentiment.json

@@ -2,7 +2,7 @@
   "format": "evals-lab/eval-group",
   "version": 9,
   "group": {
-    "name": "Demo: classify sentiment",
+    "name": "Demo: sentiment evals",
     "body": {
       "version": 9,
       "scoring": {
@@ -377,7 +377,7 @@
     },
     "preview": {
       "id": "sentiment",
-      "name": "Demo: classify sentiment"
+      "name": "Demo: sentiment source"
     }
   }
 }

package/lab/demo/datasets/weather-alerts.json

@@ -2,7 +2,7 @@
   "format": "evals-lab/eval-group",
   "version": 9,
   "group": {
-    "name": "Demo: sort weather alerts",
+    "name": "Demo: weather evals",
     "body": {
       "version": 9,
       "scoring": {
@@ -55,7 +55,7 @@
     },
     "preview": {
       "id": "weather-alerts",
-      "name": "Demo: sort weather alerts"
+      "name": "Demo: weather source"
     }
   }
 }

package/lab/demo/datasets/wordings.json

@@ -2,7 +2,7 @@
   "format": "evals-lab/eval-group",
   "version": 9,
   "group": {
-    "name": "Demo: compare wordings",
+    "name": "Demo: wordings evals",
     "body": {
       "version": 9,
       "scoring": {

package/lab/demo/manifest.json

@@ -3,10 +3,10 @@
   "version": 1,
   "id": "demo",
   "name": "Demo",
-  "packVersion": "2026.10.11",
+  "packVersion": "2026.10.14.3",
   "requires": {
-    "lab": ">=17",
+    "lab": ">=19",
     "plugins": []
   },
-  "description": "Five demos, each a dataset with a starter pipeline: two sets of public-domain images, short texts to classify, one question in four wordings, and a dynamic Source's snapshot."
+  "description": "Six demos, each a dataset with a starter pipeline: two sets of public-domain images, short texts to classify, one question in four wordings, a dynamic Source's snapshot, and PDFs to check and sort."
 }

package/lab/demo/pipelines/demo-1.json

@@ -1,7 +1,7 @@
 {
-  "version": 17,
+  "version": 19,
   "id": "demo-1",
-  "name": "Demo: describe images",
+  "name": "Demo: images pipeline",
   "jobs": [
     {
       "type": "job",
@@ -14,7 +14,7 @@
             "type": "source",
             "ref": {
               "id": "Demo 1",
-              "name": "Demo: describe images"
+              "name": "Demo: images source"
             },
             "first": null,
             "loops": 1
@@ -32,18 +32,17 @@
         },
         {
           "type": "filters",
-          "stage": "items",
           "filterSets": [
             {
               "id": "filters-j1",
               "set": null,
               "pin": null,
               "own": {
-                "version": 3,
+                "version": 4,
                 "filters": [
                   {
                     "id": "tidy",
-                    "type": "rewrite-item",
+                    "type": "trim",
                     "match": {},
                     "options": {
                       "ignoreFailures": false,
@@ -51,12 +50,12 @@
                     },
                     "tests": [],
                     "op": {
-                      "kind": "tidy"
+                      "bullets": true
                     }
                   },
                   {
                     "id": "already there",
-                    "type": "drop-item",
+                    "type": "remove",
                     "match": {
                       "duplicate": true
                     },
@@ -64,7 +63,10 @@
                       "ignoreFailures": false,
                       "repeat": 1
                     },
-                    "tests": []
+                    "tests": [],
+                    "op": {
+                      "kind": "items"
+                    }
                   },
                   {
                     "id": "keep first",
@@ -76,12 +78,13 @@
                     },
                     "tests": [],
                     "op": {
-                      "n": 12
+                      "n": 12,
+                      "unit": "items"
                     }
                   },
                   {
                     "id": "case",
-                    "type": "rewrite-item",
+                    "type": "change-case",
                     "match": {},
                     "options": {
                       "ignoreFailures": false,
@@ -89,14 +92,14 @@
                     },
                     "tests": [],
                     "op": {
-                      "kind": "case",
                       "mode": "LOWER"
                     }
                   }
                 ]
               }
             }
-          ]
+          ],
+          "format": 0
         }
       ]
     }
@@ -141,7 +144,7 @@
       "continueOnFailure": true,
       "group": {
         "id": "demo-1",
-        "name": "Demo: describe images"
+        "name": "Demo: images evals"
       },
       "pin": null
     }

package/lab/demo/pipelines/demo-2.json

@@ -1,7 +1,7 @@
 {
-  "version": 17,
+  "version": 19,
   "id": "demo-2",
-  "name": "Demo: describe images 2",
+  "name": "Demo: images 2 pipeline",
   "jobs": [
     {
       "type": "job",
@@ -14,7 +14,7 @@
             "type": "source",
             "ref": {
               "id": "Demo 2",
-              "name": "Demo: describe images 2"
+              "name": "Demo: images 2 source"
             },
             "first": null,
             "loops": 1
@@ -32,18 +32,17 @@
         },
         {
           "type": "filters",
-          "stage": "items",
           "filterSets": [
             {
               "id": "filters-j1",
               "set": null,
               "pin": null,
               "own": {
-                "version": 3,
+                "version": 4,
                 "filters": [
                   {
                     "id": "tidy",
-                    "type": "rewrite-item",
+                    "type": "trim",
                     "match": {},
                     "options": {
                       "ignoreFailures": false,
@@ -51,12 +50,12 @@
                     },
                     "tests": [],
                     "op": {
-                      "kind": "tidy"
+                      "bullets": true
                     }
                   },
                   {
                     "id": "already there",
-                    "type": "drop-item",
+                    "type": "remove",
                     "match": {
                       "duplicate": true
                     },
@@ -64,7 +63,10 @@
                       "ignoreFailures": false,
                       "repeat": 1
                     },
-                    "tests": []
+                    "tests": [],
+                    "op": {
+                      "kind": "items"
+                    }
                   },
                   {
                     "id": "keep first",
@@ -76,12 +78,13 @@
                     },
                     "tests": [],
                     "op": {
-                      "n": 12
+                      "n": 12,
+                      "unit": "items"
                     }
                   },
                   {
                     "id": "case",
-                    "type": "rewrite-item",
+                    "type": "change-case",
                     "match": {},
                     "options": {
                       "ignoreFailures": false,
@@ -89,14 +92,14 @@
                     },
                     "tests": [],
                     "op": {
-                      "kind": "case",
                       "mode": "LOWER"
                     }
                   }
                 ]
               }
             }
-          ]
+          ],
+          "format": 0
         }
       ]
     }
@@ -141,7 +144,7 @@
       "continueOnFailure": true,
       "group": {
         "id": "demo-2",
-        "name": "Demo: describe images 2"
+        "name": "Demo: images 2 evals"
       },
       "pin": null
     }