evals-lab 0.1.3 → 0.2.0

package/README.md

@@ -13,7 +13,7 @@ npm install -g evals-lab
 evals-lab
 ```
 
-Then open <http://127.0.0.1:8080>. A new lab opens on two demo datasets, each
+Then open <http://localhost:8080>. A new lab opens on two demo datasets, each
 with its images and a starter pipeline. The first start takes a little longer
 than later ones while it installs them.
 
@@ -61,12 +61,106 @@ evals-lab [--port <n>] [--host <address>] [--data-dir <dir>]
 | `--data-dir` | `DATA_DIR` | the directory above |
 | | `LAB_PASSWORD` | none |
 | | `OLLAMA_URL` | `http://127.0.0.1:11434` |
+| | `M365_CLIENT_ID`, `M365_TENANT` | none; see Power Automate below |
 | | `PYTHON` | `py -3`, `python` or `python3`, whichever is found |
 
 `OLLAMA_URL` is only the connection a new lab starts with. Every other
 connection — an OpenAI-compatible endpoint, Anthropic, llama.cpp — is a Setup
 profile you add in the page.
 
+## Power Automate (Microsoft 365)
+
+The lab can read a Power Automate cloud flow, and its run history, from your
+organisation's Microsoft 365 tenant, then grade the flow's HTTP step against
+the same model and prompt. The sign-in stays in your browser tab: the lab's
+server never receives a Microsoft token.
+
+It needs a work or school account. Microsoft retired cloud flows for personal
+accounts (outlook.com, hotmail.com) in July 2025, so a personal account cannot
+be used. For a small business, a trial or a Microsoft 365 Developer Program
+tenant, follow the same steps as a company.
+
+### Register an app in your tenant
+
+An administrator of the tenant registers it once, at
+<https://entra.microsoft.com>:
+
+1. **App registrations › New registration.**
+   - Supported account types: *Accounts in this organizational directory only*.
+   - Redirect URI: platform **Single-page application**, value `http://localhost/`.
+     Microsoft ignores the port on a `localhost` redirect, so this covers
+     `--port` too. A lab served on another address needs that address here
+     as well, over `https`.
+2. **API permissions › Add a permission › APIs my organization uses.** Search
+   for *Microsoft Flow Service* and add the delegated permission
+   **Flows.Read.All**. If the service isn't listed, open
+   <https://make.powerautomate.com> once, then search again.
+3. **Grant admin consent** for the tenant.
+4. No client secret or certificate is needed: the lab signs in as a public
+   client.
+
+### Tell the lab about it
+
+In the lab, open **Setup › Connections**, then use the Microsoft 365 row's
+menu › **Configure…**. Enter:
+
+| | |
+|---|---|
+| Client ID | the app's *Application (client) ID* |
+| Tenant | the *Directory (tenant) ID*, or one of the tenant's domains |
+
+Fill in Tenant for an app registered as above. Left empty, it means any
+organisation's tenant, which only a multitenant app accepts: Microsoft
+refuses the sign-in of a single-tenant app with `AADSTS50194`.
+
+Both are saved as you type, and neither is secret. **Sign in** from the same
+menu, then add a Source in the Sources tab with **Add source › Power Automate
+workflow**.
+
+To set the app for the whole machine instead, start the lab with
+`M365_CLIENT_ID` and `M365_TENANT` set. Setup then shows those values and
+can't change them.
+
+Open the lab at `localhost`, the address `evals-lab` prints. A sign-in begun
+at `127.0.0.1` reloads the page at `localhost`, the address registered above,
+and carries on from there. Anything open in the tab, such as an Add source
+dialog, does not carry over.
+
+## Google Drive
+
+The lab can sign in to Google Drive, so a Source can read files from a Drive
+folder. The sign-in is held by the lab's server, on your machine, and it can
+reach only the files you pick: the lab asks for `drive.file` access, never
+your whole Drive.
+
+Open **Setup › Connections**, then use the Google Drive row's menu ›
+**Sign in**. If the row says *Not Configured*, give the lab a Google app
+first, once. These steps are provisional: they haven't yet been tried
+against a real Google project, and the Drive Source that uses the sign-in
+is still to come.
+
+1. At <https://console.cloud.google.com>, make a project. Enable the
+   **Google Drive API** and the **Google Picker API**.
+2. **OAuth consent screen:** choose *External*. Give the app a name and your
+   email, add the scope `.../auth/drive.file`, then **Publish app**. It needs
+   no review from Google, because that scope is non-sensitive. Left in
+   *Testing*, a sign-in lapses after 7 days.
+3. **Credentials › Create credentials › OAuth client ID**, of type
+   **Desktop app**. Download its JSON file.
+4. **Credentials › Create credentials › API key.** Restrict it to the Google
+   Picker API.
+5. In the lab, use the Google Drive row's menu › **Configure…**, then
+   **Load client file…**, and choose the file from step 3. Paste the key
+   from step 4 into **API key**.
+
+A Desktop app needs no redirect registered: Google sends the browser back to
+any port on this machine. A lab you reach at its own address, from another
+machine, needs a *Web application* client instead. Choose that type in
+Configure…, then register the **Redirect URI** it shows with the client.
+
+Everything saves as you type. The client's secret is kept by the lab and
+never shown again. The window says only that one is saved.
+
 ## Reaching it from another machine
 
 By default only your own machine can reach the lab. That is deliberate: the

package/bin/evals-lab.js

@@ -53,6 +53,13 @@ function isLoopback(host) {
   return host === "localhost" || /^127(?:\.\d{1,3}){3}$/.test(host);
 }
 
+// The address to open the lab at. Loopback reads as localhost: it is the one
+// http address Entra takes as a sign-in redirect from its portal, on any port,
+// so a Microsoft 365 sign-in works from the address printed (npm/README.md).
+function advertised(host, port) {
+  return `http://${host === "127.0.0.1" ? "localhost" : host}:${port}`;
+}
+
 // What the command line and the environment ask for, or an Error saying what
 // is wrong with it.
 function options(argv, env = process.env, platform = process.platform, home = os.homedir()) {
@@ -191,11 +198,12 @@ async function main() {
   process.on("SIGTERM", () => { stopping = true; child.kill("SIGTERM"); });
 
   const address = `http://${o.host}:${o.port}`;
+  const open = advertised(o.host, o.port);
   // A first start installs the demo pack before it listens, which takes longer
   // than any later one.
   for (let i = 0; i < 1200; i++) {
     if (await answers(`${address}/healthz`)) {
-      console.log(`\nevals-lab ${version()} is running at ${address}\n  data:  ${o.dataDir}\n  stop:  Ctrl+C`);
+      console.log(`\nevals-lab ${version()} is running at ${open}\n  data:  ${o.dataDir}\n  stop:  Ctrl+C`);
       return;
     }
     await new Promise(r => setTimeout(r, 250));
@@ -204,4 +212,4 @@ async function main() {
 }
 
 if (require.main === module) main();
-module.exports = { dataDir, isLoopback, options, pythonCandidates, findPython, atLeast, PYTHON_FLOOR };
+module.exports = { dataDir, isLoopback, advertised, options, pythonCandidates, findPython, atLeast, PYTHON_FLOOR };

package/lab/VERSION

@@ -1 +1 @@
-0.1.3 (2026.10.01-289)
+0.2.0 (2026.10.02-323)

package/lab/demo/pipelines/demo-1.json

@@ -1,5 +1,5 @@
 {
-  "version": 9,
+  "version": 10,
   "id": "demo-1",
   "name": "Demo 1",
   "jobs": [
@@ -21,38 +21,46 @@
           }
         },
         {
-          "type": "prompt",
-          "withImage": true,
-          "tokenMappings": []
+          "type": "attachImage"
         },
         {
           "type": "readReply",
           "out": {
             "kind": "list",
             "parse": "csv",
-            "preamble": false,
-            "modifiers": [
-              {
-                "type": "tidy"
-              },
-              {
-                "type": "dedupe"
-              },
-              {
-                "type": "keep-first",
-                "n": 12
-              },
-              {
-                "type": "case",
-                "mode": "LOWER"
-              }
-            ]
+            "preamble": false
+          }
+        },
+        {
+          "type": "modifier",
+          "modifier": {
+            "type": "tidy"
+          }
+        },
+        {
+          "type": "modifier",
+          "modifier": {
+            "type": "dedupe"
+          }
+        },
+        {
+          "type": "modifier",
+          "modifier": {
+            "type": "keep-first",
+            "n": 12
+          }
+        },
+        {
+          "type": "modifier",
+          "modifier": {
+            "type": "case",
+            "mode": "LOWER"
           }
         }
       ]
     }
   ],
-  "scenarios": [
+  "targets": [
     {
       "id": "s1",
       "name": "",
@@ -60,8 +68,9 @@
         "id": "vision",
         "name": "Vision"
       },
-      "stages": [
+      "steps": [
         {
+          "type": "prompt",
           "prompt": "List the things you can see in this image, as a comma-separated list of short items (one to three words each), most important first. Up to 12 items."
         }
       ]
@@ -73,8 +82,9 @@
         "id": "vision",
         "name": "Vision"
       },
-      "stages": [
+      "steps": [
         {
+          "type": "prompt",
           "prompt": "Name what is in this image. Reply with a comma-separated list only: up to 12 short items, most important first."
         }
       ]

package/lab/demo/pipelines/demo-2.json

@@ -1,5 +1,5 @@
 {
-  "version": 9,
+  "version": 10,
   "id": "demo-2",
   "name": "Demo 2",
   "jobs": [
@@ -21,38 +21,46 @@
           }
         },
         {
-          "type": "prompt",
-          "withImage": true,
-          "tokenMappings": []
+          "type": "attachImage"
         },
         {
           "type": "readReply",
           "out": {
             "kind": "list",
             "parse": "csv",
-            "preamble": false,
-            "modifiers": [
-              {
-                "type": "tidy"
-              },
-              {
-                "type": "dedupe"
-              },
-              {
-                "type": "keep-first",
-                "n": 12
-              },
-              {
-                "type": "case",
-                "mode": "LOWER"
-              }
-            ]
+            "preamble": false
+          }
+        },
+        {
+          "type": "modifier",
+          "modifier": {
+            "type": "tidy"
+          }
+        },
+        {
+          "type": "modifier",
+          "modifier": {
+            "type": "dedupe"
+          }
+        },
+        {
+          "type": "modifier",
+          "modifier": {
+            "type": "keep-first",
+            "n": 12
+          }
+        },
+        {
+          "type": "modifier",
+          "modifier": {
+            "type": "case",
+            "mode": "LOWER"
           }
         }
       ]
     }
   ],
-  "scenarios": [
+  "targets": [
     {
       "id": "s1",
       "name": "",
@@ -60,8 +68,9 @@
         "id": "vision",
         "name": "Vision"
       },
-      "stages": [
+      "steps": [
         {
+          "type": "prompt",
           "prompt": "List the things you can see in this image, as a comma-separated list of short items (one to three words each), most important first. Up to 12 items."
         }
       ]
@@ -73,8 +82,9 @@
         "id": "vision",
         "name": "Vision"
       },
-      "stages": [
+      "steps": [
         {
+          "type": "prompt",
           "prompt": "Name what is in this image. Reply with a comma-separated list only: up to 12 short items, most important first."
         }
       ]