evals-lab 0.12.0 → 0.14.0

package/CHANGELOG.md

@@ -5,6 +5,94 @@ Newest first. Read a release's **Upgrade notes** before installing it: an
 upgrade can rewrite what the lab keeps in your data directory, and an older
 version cannot always read it back.
 
+## 0.14.0
+
+### Upgrade notes
+
+- **Copy your data directory before upgrading** (see "User data" in the
+  README). A pipeline 0.14.0 saves is pipeline version 18, which 0.13.x
+  cannot read. To go back, reinstall 0.13.0 and restore the copy.
+- A job that graded a dynamic Source's saved content now grades one saved
+  response: on upgrade it is pinned to the Source's newest successful one.
+  Live data is unchanged. To grade several responses, clone the job.
+
+### Added
+
+- An endpoint or call keeps its responses: Fetch now, an import or an edit
+  adds one, and Restore brings an earlier one back.
+- A Source page lists Responses and History, and Saved response pins the
+  one response a job grades.
+- Results show one block per eval group, export as JSON or PDF, say when a
+  reply was cut off, and show where content goes in the prompt.
+- Post-format filters for a text reply, and more Rewrite text operations.
+- A sixth demo: sort documents, with valid and invalid PDFs.
+- The (Default) Connection Profile can be edited and applies to (Any)
+  connection.
+- History filters by pipeline and content.
+
+### Changed
+
+- A dialog says Close until something in it changes.
+- Demo datasets are named for their kind, such as "Demo: sentiment evals".
+- Response Format offers one text option, Plain text.
+- A link to something that is gone says so.
+- History fits a tablet, and Home and Results show long names on two lines.
+- More colours reach WCAG AA contrast in both themes.
+
+### Fixed
+
+- A job hands on its reply as its filters left it.
+- A Read Reply step's `preamble: false` is dropped on upgrade and on read.
+
+## 0.13.0
+
+### Upgrade notes
+
+- Pipelines and datasets keep their versions (17 and 9), so nothing in your
+  data directory is rewritten. A pipeline that uses the new Refused metric
+  needs 0.13.0 to grade it.
+
+### Added
+
+- A Refused metric, for a case that expects the target's call to be refused.
+- A PDF item goes to the model whole, as a document; a connection that can't
+  read one is refused before the run.
+- AI Connections and Connection Profiles select several rows to remove at
+  once, with one Undo; Connection Profiles show their connection.
+- Copy or move… on an eval group's, a filter set's and a Source's own page.
+
+### Changed
+
+- One date format everywhere, and Home lists recent runs.
+- Dropdowns that name things read A–Z; Model, Environment and Flow open the
+  Switcher.
+- Empty values read as a short parenthetical, such as (No grader), and
+  explanations are shorter.
+- A run's jobs sit in one Jobs group, with Add job inside it.
+- A Supplied Source takes several files in one Upload.
+- Enter in a dialog's field presses its main button; in Run settings it
+  leaves the field.
+- Stop reads Stopping… until the run ends, and History can stop a run.
+
+### Fixed
+
+- An errored result fails on its error and is not graded.
+- Metrics and the grader read the reply after its Pre-format filters, and
+  Results shows that reply.
+- A linked group whose grader is gone can't run, and Results says why a
+  graded metric never ran; an off-scale grader score is the grader's error.
+- An Edited target whose expression doesn't parse is refused before the run.
+- Pin names the newest version the server holds.
+- Demo targets answer on a connection the workspace can use.
+- Loop repeats a live capture, and the Comment stays with its pipeline.
+- A Re-sync that needed a Microsoft sign-in runs once the page is back.
+- A dynamic Source's fetch names the lab as its User-Agent.
+- Emptying the trash leaves a connection shared only with the purged
+  workspace shared with none, not All; a lab import's rollback leaves
+  another import's connection intact.
+- A stored Echo connection is no longer listed beside the built-in Echo
+  target.
+
 ## 0.12.0
 
 ### Upgrade notes

package/README.md

@@ -45,7 +45,7 @@ npm install -g evals-lab
 evals-lab
 ```
 
-Open <http://localhost:8080>. A new lab starts with five demos, each a dataset
+Open <http://localhost:8080>. A new lab starts with six demos, each a dataset
 and its pipeline, which run once you add an AI Connection in Setup. Note the first start takes longer while it installs them. Without
 installing: `npx evals-lab`.
 
@@ -164,7 +164,7 @@ CI can run a lab's own pipeline there instead, so the run shows in that lab's
 History and its keys never leave it:
 
 ```bash
-LAB_PASSWORD=... evals-lab run --lab https://lab.example --pipeline "Demo: describe images" --wait
+LAB_PASSWORD=... evals-lab run --lab https://lab.example --pipeline "Demo: images pipeline" --wait
 ```
 
 ```text

package/bin/run.js

@@ -399,8 +399,10 @@ async function runOnLab(core, o, { env, stdout, say }) {
   const docs = (await call("GET", "/api/state")).docs ?? {};
   const setup = docs["promptlab.connections"]?.body ?? {};
   const connections = core.withSlugs(Array.isArray(setup.list) ? setup.list : []);
-  const held = docs["promptlab.connectionProfiles"]?.body?.list;
-  const answers = core.answersOf(connections, Array.isArray(held) ? held : []);
+  const held = docs["promptlab.connectionProfiles"]?.body;
+  // (Default)'s settings, under each target that names no profile (#603).
+  const answers = core.answersOf(connections, Array.isArray(held?.list) ? held.list : [],
+                                 held?.default && typeof held.default === "object" ? held.default : null);
   const saved = docs["promptlab.workflows"]?.body?.list;
   const found = pipelineNamed(Array.isArray(saved) ? saved : [], o.pipeline, base);
   const doc = core.upgradePipeline(found.work, { profileType: id => answers.connection(id)?.type,

package/lab/VERSION

@@ -1 +1 @@
-0.12.0 (2026.10.12-718)
+0.14.0 (2026.10.14-830)

package/lab/demo/CREDITS.md

@@ -124,3 +124,17 @@ on 2026-10-11. A work of the US federal government is in the public domain.
 | `volunteer-day.txt` | Written for this pack | Evals Lab | Public domain |
 | `weather-report.txt` | Written for this pack | Evals Lab | Public domain |
 | `https-api.weather.gov-alerts-types.txt` | https://api.weather.gov/alerts/types | National Weather Service | Public domain |
+
+## Documents
+
+The PDFs of `documents/`, made for this pack: four short documents, a PDF cut
+short, and a text file with a `.pdf` name.
+
+| File | Source | Author | Licence |
+|---|---|---|---|
+| `invoice.pdf` | Written for this pack | Evals Lab | Public domain |
+| `letter.pdf` | Written for this pack | Evals Lab | Public domain |
+| `not-a-pdf.pdf` | Written for this pack | Evals Lab | Public domain |
+| `receipt.pdf` | Written for this pack | Evals Lab | Public domain |
+| `report.pdf` | Written for this pack | Evals Lab | Public domain |
+| `truncated.pdf` | Written for this pack | Evals Lab | Public domain |

package/lab/demo/datasets/demo-1.json

@@ -2,12 +2,12 @@
   "format": "evals-lab/dataset",
   "version": 6,
   "dataset": {
-    "name": "Demo: describe images",
+    "name": "Demo: images evals",
     "body": {
       "version": 6,
       "source": {
         "id": "Demo 1",
-        "name": "Demo: describe images"
+        "name": "Demo: images source"
       },
       "cases": [
         {

package/lab/demo/datasets/demo-2.json

@@ -2,12 +2,12 @@
   "format": "evals-lab/dataset",
   "version": 6,
   "dataset": {
-    "name": "Demo: describe images 2",
+    "name": "Demo: images 2 evals",
     "body": {
       "version": 6,
       "source": {
         "id": "Demo 2",
-        "name": "Demo: describe images 2"
+        "name": "Demo: images 2 source"
       },
       "cases": [
         {

package/lab/demo/datasets/documents-type.json

@@ -0,0 +1,79 @@
+{
+  "format": "evals-lab/eval-group",
+  "version": 9,
+  "group": {
+    "name": "Document type",
+    "preview": {
+      "id": "documents",
+      "name": "Demo: documents source"
+    },
+    "body": {
+      "version": 9,
+      "scoring": {
+        "mode": "all",
+        "threshold": null
+      },
+      "grader": null,
+      "every": [],
+      "run": [],
+      "cases": [
+        {
+          "id": "invoice",
+          "item": "invoice.pdf",
+          "todo": false,
+          "note": "",
+          "metrics": [
+            {
+              "type": "json-field",
+              "path": "type",
+              "op": "equals",
+              "value": "invoice"
+            }
+          ]
+        },
+        {
+          "id": "letter",
+          "item": "letter.pdf",
+          "todo": false,
+          "note": "",
+          "metrics": [
+            {
+              "type": "json-field",
+              "path": "type",
+              "op": "equals",
+              "value": "letter"
+            }
+          ]
+        },
+        {
+          "id": "receipt",
+          "item": "receipt.pdf",
+          "todo": false,
+          "note": "",
+          "metrics": [
+            {
+              "type": "json-field",
+              "path": "type",
+              "op": "equals",
+              "value": "receipt"
+            }
+          ]
+        },
+        {
+          "id": "report",
+          "item": "report.pdf",
+          "todo": false,
+          "note": "",
+          "metrics": [
+            {
+              "type": "json-field",
+              "path": "type",
+              "op": "equals",
+              "value": "report"
+            }
+          ]
+        }
+      ]
+    }
+  }
+}

package/lab/demo/datasets/documents-valid.json

@@ -0,0 +1,117 @@
+{
+  "format": "evals-lab/eval-group",
+  "version": 9,
+  "group": {
+    "name": "PDF is valid",
+    "preview": {
+      "id": "documents",
+      "name": "Demo: documents source"
+    },
+    "body": {
+      "version": 9,
+      "scoring": {
+        "mode": "weighted",
+        "threshold": 1
+      },
+      "grader": null,
+      "every": [],
+      "run": [],
+      "cases": [
+        {
+          "id": "invoice",
+          "item": "invoice.pdf",
+          "todo": false,
+          "note": "",
+          "metrics": [
+            {
+              "type": "json-field",
+              "path": "valid",
+              "op": "equals",
+              "value": "true"
+            }
+          ]
+        },
+        {
+          "id": "letter",
+          "item": "letter.pdf",
+          "todo": false,
+          "note": "",
+          "metrics": [
+            {
+              "type": "json-field",
+              "path": "valid",
+              "op": "equals",
+              "value": "true"
+            }
+          ]
+        },
+        {
+          "id": "receipt",
+          "item": "receipt.pdf",
+          "todo": false,
+          "note": "",
+          "metrics": [
+            {
+              "type": "json-field",
+              "path": "valid",
+              "op": "equals",
+              "value": "true"
+            }
+          ]
+        },
+        {
+          "id": "report",
+          "item": "report.pdf",
+          "todo": false,
+          "note": "",
+          "metrics": [
+            {
+              "type": "json-field",
+              "path": "valid",
+              "op": "equals",
+              "value": "true"
+            }
+          ]
+        },
+        {
+          "id": "truncated",
+          "item": "truncated.pdf",
+          "todo": false,
+          "note": "A real PDF cut short. Either the API refuses it or the model says it can't read it.",
+          "metrics": [
+            {
+              "type": "refused",
+              "weight": 1
+            },
+            {
+              "type": "json-field",
+              "path": "valid",
+              "op": "equals",
+              "value": "false",
+              "weight": 1
+            }
+          ]
+        },
+        {
+          "id": "not-a-pdf",
+          "item": "not-a-pdf.pdf",
+          "todo": false,
+          "note": "Plain text with a .pdf name. Either the API refuses it or the model says it can't read it.",
+          "metrics": [
+            {
+              "type": "refused",
+              "weight": 1
+            },
+            {
+              "type": "json-field",
+              "path": "valid",
+              "op": "equals",
+              "value": "false",
+              "weight": 1
+            }
+          ]
+        }
+      ]
+    }
+  }
+}

package/lab/demo/datasets/sentiment.json

@@ -2,7 +2,7 @@
   "format": "evals-lab/eval-group",
   "version": 9,
   "group": {
-    "name": "Demo: classify sentiment",
+    "name": "Demo: sentiment evals",
     "body": {
       "version": 9,
       "scoring": {
@@ -377,7 +377,7 @@
     },
     "preview": {
       "id": "sentiment",
-      "name": "Demo: classify sentiment"
+      "name": "Demo: sentiment source"
     }
   }
 }

package/lab/demo/datasets/weather-alerts.json

@@ -2,7 +2,7 @@
   "format": "evals-lab/eval-group",
   "version": 9,
   "group": {
-    "name": "Demo: sort weather alerts",
+    "name": "Demo: weather evals",
     "body": {
       "version": 9,
       "scoring": {
@@ -55,7 +55,7 @@
     },
     "preview": {
       "id": "weather-alerts",
-      "name": "Demo: sort weather alerts"
+      "name": "Demo: weather source"
     }
   }
 }

package/lab/demo/datasets/wordings.json

@@ -2,7 +2,7 @@
   "format": "evals-lab/eval-group",
   "version": 9,
   "group": {
-    "name": "Demo: compare wordings",
+    "name": "Demo: wordings evals",
     "body": {
       "version": 9,
       "scoring": {

package/lab/demo/manifest.json

@@ -3,10 +3,10 @@
   "version": 1,
   "id": "demo",
   "name": "Demo",
-  "packVersion": "2026.10.11",
+  "packVersion": "2026.10.14.2",
   "requires": {
-    "lab": ">=17",
+    "lab": ">=18",
     "plugins": []
   },
-  "description": "Five demos, each a dataset with a starter pipeline: two sets of public-domain images, short texts to classify, one question in four wordings, and a dynamic Source's snapshot."
+  "description": "Six demos, each a dataset with a starter pipeline: two sets of public-domain images, short texts to classify, one question in four wordings, a dynamic Source's snapshot, and PDFs to check and sort."
 }

package/lab/demo/pipelines/demo-1.json

@@ -1,7 +1,7 @@
 {
-  "version": 17,
+  "version": 18,
   "id": "demo-1",
-  "name": "Demo: describe images",
+  "name": "Demo: images pipeline",
   "jobs": [
     {
       "type": "job",
@@ -14,7 +14,7 @@
             "type": "source",
             "ref": {
               "id": "Demo 1",
-              "name": "Demo: describe images"
+              "name": "Demo: images source"
             },
             "first": null,
             "loops": 1
@@ -141,7 +141,7 @@
       "continueOnFailure": true,
       "group": {
         "id": "demo-1",
-        "name": "Demo: describe images"
+        "name": "Demo: images evals"
       },
       "pin": null
     }

package/lab/demo/pipelines/demo-2.json

@@ -1,7 +1,7 @@
 {
-  "version": 17,
+  "version": 18,
   "id": "demo-2",
-  "name": "Demo: describe images 2",
+  "name": "Demo: images 2 pipeline",
   "jobs": [
     {
       "type": "job",
@@ -14,7 +14,7 @@
             "type": "source",
             "ref": {
               "id": "Demo 2",
-              "name": "Demo: describe images 2"
+              "name": "Demo: images 2 source"
             },
             "first": null,
             "loops": 1
@@ -141,7 +141,7 @@
       "continueOnFailure": true,
       "group": {
         "id": "demo-2",
-        "name": "Demo: describe images 2"
+        "name": "Demo: images 2 evals"
       },
       "pin": null
     }

package/lab/demo/pipelines/documents.json

@@ -0,0 +1,102 @@
+{
+  "version": 18,
+  "id": "documents",
+  "name": "Demo: documents pipeline",
+  "jobs": [
+    {
+      "type": "job",
+      "id": "j1",
+      "name": "",
+      "steps": [
+        {
+          "type": "attachContent",
+          "content": {
+            "type": "source",
+            "ref": {
+              "id": "documents",
+              "name": "Demo: documents source"
+            },
+            "first": null,
+            "loops": 1
+          }
+        },
+        {
+          "type": "attachImage"
+        },
+        {
+          "type": "filters",
+          "stage": "items",
+          "filterSets": [
+            {
+              "id": "filters-j1",
+              "set": null,
+              "pin": null,
+              "own": {
+                "version": 3,
+                "filters": [
+                  {
+                    "id": "unfence",
+                    "type": "rewrite-text",
+                    "match": {},
+                    "options": {
+                      "ignoreFailures": false,
+                      "repeat": 1
+                    },
+                    "tests": [],
+                    "op": {
+                      "kind": "fences"
+                    }
+                  }
+                ]
+              }
+            }
+          ]
+        }
+      ]
+    }
+  ],
+  "targets": [
+    {
+      "id": "s1",
+      "name": "",
+      "connection": {
+        "id": "model",
+        "name": "Model"
+      },
+      "profile": null,
+      "steps": [
+        {
+          "type": "prompt",
+          "prompt": "Read the attached document. Reply with JSON only: {\"valid\": true or false (false if you can't read it as a document), \"type\": \"invoice\" | \"letter\" | \"receipt\" | \"report\" | \"other\"}"
+        }
+      ]
+    }
+  ],
+  "evals": [
+    {
+      "id": "t1",
+      "name": "",
+      "type": "group",
+      "continueOnFailure": true,
+      "group": {
+        "id": "documents-valid",
+        "name": "PDF is valid"
+      },
+      "pin": null
+    },
+    {
+      "id": "t2",
+      "name": "",
+      "type": "group",
+      "continueOnFailure": true,
+      "group": {
+        "id": "documents-type",
+        "name": "Document type"
+      },
+      "pin": null
+    }
+  ],
+  "pass": {
+    "mode": "all"
+  }
+}

package/lab/demo/pipelines/sentiment.json

@@ -1,7 +1,7 @@
 {
-  "version": 17,
+  "version": 18,
   "id": "sentiment",
-  "name": "Demo: classify sentiment",
+  "name": "Demo: sentiment pipeline",
   "jobs": [
     {
       "type": "job",
@@ -14,7 +14,7 @@
             "type": "source",
             "ref": {
               "id": "sentiment",
-              "name": "Demo: classify sentiment"
+              "name": "Demo: sentiment source"
             },
             "first": null,
             "loops": 1
@@ -54,7 +54,7 @@
       "continueOnFailure": true,
       "group": {
         "id": "sentiment",
-        "name": "Demo: classify sentiment"
+        "name": "Demo: sentiment evals"
       },
       "pin": null
     }

package/lab/demo/pipelines/weather-alerts.json

@@ -1,7 +1,7 @@
 {
-  "version": 17,
+  "version": 18,
   "id": "weather-alerts",
-  "name": "Demo: sort weather alerts",
+  "name": "Demo: weather pipeline",
   "jobs": [
     {
       "type": "job",
@@ -14,11 +14,15 @@
             "type": "source",
             "ref": {
               "id": "weather-alerts",
-              "name": "Demo: sort weather alerts"
+              "name": "Demo: weather source"
             },
             "first": null,
             "loops": 1,
-            "live": false
+            "live": false,
+            "pin": {
+              "name": "https-api.weather.gov-alerts-types.txt",
+              "n": 0
+            }
           }
         },
         {
@@ -55,7 +59,7 @@
       "continueOnFailure": true,
       "group": {
         "id": "weather-alerts",
-        "name": "Demo: sort weather alerts"
+        "name": "Demo: weather evals"
       },
       "pin": null
     }