evals-lab 0.1.3 → 0.1.4

package/README.md

@@ -13,7 +13,7 @@ npm install -g evals-lab
 evals-lab
 ```
 
-Then open <http://127.0.0.1:8080>. A new lab opens on two demo datasets, each
+Then open <http://localhost:8080>. A new lab opens on two demo datasets, each
 with its images and a starter pipeline. The first start takes a little longer
 than later ones while it installs them.
 
@@ -61,12 +61,71 @@ evals-lab [--port <n>] [--host <address>] [--data-dir <dir>]
 | `--data-dir` | `DATA_DIR` | the directory above |
 | | `LAB_PASSWORD` | none |
 | | `OLLAMA_URL` | `http://127.0.0.1:11434` |
+| | `M365_CLIENT_ID`, `M365_TENANT` | none; see Power Automate below |
 | | `PYTHON` | `py -3`, `python` or `python3`, whichever is found |
 
 `OLLAMA_URL` is only the connection a new lab starts with. Every other
 connection — an OpenAI-compatible endpoint, Anthropic, llama.cpp — is a Setup
 profile you add in the page.
 
+## Power Automate (Microsoft 365)
+
+The lab can read a Power Automate cloud flow, and its run history, from your
+organisation's Microsoft 365 tenant, then grade the flow's HTTP step against
+the same model and prompt. The sign-in stays in your browser tab: the lab's
+server never receives a Microsoft token.
+
+It needs a work or school account. Microsoft retired cloud flows for personal
+accounts (outlook.com, hotmail.com) in July 2025, so a personal account cannot
+be used. For a small business, a trial or a Microsoft 365 Developer Program
+tenant, follow the same steps as a company.
+
+### Register an app in your tenant
+
+An administrator of the tenant registers it once, at
+<https://entra.microsoft.com>:
+
+1. **App registrations › New registration.**
+   - Supported account types: *Accounts in this organizational directory only*.
+   - Redirect URI: platform **Single-page application**, value `http://localhost/`.
+     Microsoft ignores the port on a `localhost` redirect, so this covers
+     `--port` too. A lab served on another address needs that address here
+     as well, over `https`.
+2. **API permissions › Add a permission › APIs my organization uses.** Search
+   for *Microsoft Flow Service* and add the delegated permission
+   **Flows.Read.All**. If the service isn't listed, open
+   <https://make.powerautomate.com> once, then search again.
+3. **Grant admin consent** for the tenant.
+4. No client secret or certificate is needed: the lab signs in as a public
+   client.
+
+### Tell the lab about it
+
+In the lab, open **Setup › Connections**, then use the Microsoft 365 row's
+menu › **Configure…**. Enter:
+
+| | |
+|---|---|
+| Client ID | the app's *Application (client) ID* |
+| Tenant | the *Directory (tenant) ID*, or one of the tenant's domains |
+
+Fill in Tenant for an app registered as above. Left empty, it means any
+organisation's tenant, which only a multitenant app accepts: Microsoft
+refuses the sign-in of a single-tenant app with `AADSTS50194`.
+
+Both are saved as you type, and neither is secret. **Sign in** from the same
+menu, then add a Source in the Sources tab with **Add source › Power Automate
+workflow**.
+
+To set the app for the whole machine instead, start the lab with
+`M365_CLIENT_ID` and `M365_TENANT` set. Setup then shows those values and
+can't change them.
+
+Open the lab at `localhost`, the address `evals-lab` prints. A sign-in begun
+at `127.0.0.1` reloads the page at `localhost`, the address registered above,
+and carries on from there. Anything open in the tab, such as an Add source
+dialog, does not carry over.
+
 ## Reaching it from another machine
 
 By default only your own machine can reach the lab. That is deliberate: the

package/bin/evals-lab.js

@@ -53,6 +53,13 @@ function isLoopback(host) {
   return host === "localhost" || /^127(?:\.\d{1,3}){3}$/.test(host);
 }
 
+// The address to open the lab at. Loopback reads as localhost: it is the one
+// http address Entra takes as a sign-in redirect from its portal, on any port,
+// so a Microsoft 365 sign-in works from the address printed (npm/README.md).
+function advertised(host, port) {
+  return `http://${host === "127.0.0.1" ? "localhost" : host}:${port}`;
+}
+
 // What the command line and the environment ask for, or an Error saying what
 // is wrong with it.
 function options(argv, env = process.env, platform = process.platform, home = os.homedir()) {
@@ -191,11 +198,12 @@ async function main() {
   process.on("SIGTERM", () => { stopping = true; child.kill("SIGTERM"); });
 
   const address = `http://${o.host}:${o.port}`;
+  const open = advertised(o.host, o.port);
   // A first start installs the demo pack before it listens, which takes longer
   // than any later one.
   for (let i = 0; i < 1200; i++) {
     if (await answers(`${address}/healthz`)) {
-      console.log(`\nevals-lab ${version()} is running at ${address}\n  data:  ${o.dataDir}\n  stop:  Ctrl+C`);
+      console.log(`\nevals-lab ${version()} is running at ${open}\n  data:  ${o.dataDir}\n  stop:  Ctrl+C`);
       return;
     }
     await new Promise(r => setTimeout(r, 250));
@@ -204,4 +212,4 @@ async function main() {
 }
 
 if (require.main === module) main();
-module.exports = { dataDir, isLoopback, options, pythonCandidates, findPython, atLeast, PYTHON_FLOOR };
+module.exports = { dataDir, isLoopback, advertised, options, pythonCandidates, findPython, atLeast, PYTHON_FLOOR };

package/lab/VERSION

@@ -1 +1 @@
-0.1.3 (2026.10.01-289)
+0.1.4 (2026.10.01-291)

package/lab/server.py

@@ -288,9 +288,44 @@ DEFAULT_SOURCE_TYPE = "files"
 # The Entra app the page signs in to Microsoft 365 with (docs/power-automate.md
 # § Registering the app). A single-page app has no secret, so both are public
 # and served to the page; the token it gets stays in the browser, and the
-# server never sees one. Unset, a Power Automate workflow cannot be made.
+# server never sees one. The deployment's own, when it names one, then the one
+# entered in Setup (MicrosoftApp), then the published app below; with none, a
+# Power Automate workflow cannot be made.
 M365_CLIENT_ID = os.environ.get("M365_CLIENT_ID") or None
 M365_TENANT = os.environ.get("M365_TENANT") or "organizations"
+# The multitenant app published for every lab, a tenant's admin consenting to
+# it once. None until it is registered and its publisher verified: an
+# unverified app from another organisation is one most tenants' users may not
+# consent to, so shipping it earlier would only move the failure to sign-in.
+PUBLIC_M365_CLIENT_ID = None
+
+GUID = re.compile(r"[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}")
+TENANT_DOMAIN = re.compile(r"(?=.{4,253}$)(?:[A-Za-z0-9](?:[A-Za-z0-9-]{0,61}[A-Za-z0-9])?\.)+[A-Za-z]{2,63}")
+
+
+def microsoft_config():
+    """The Microsoft app this lab signs in with, and where it came from."""
+    if M365_CLIENT_ID:
+        return {"clientId": M365_CLIENT_ID, "tenant": M365_TENANT, "from": "env"}
+    kept = MICROSOFT.get() if MICROSOFT is not None else None
+    if kept:
+        return {**kept, "from": "lab"}
+    if PUBLIC_M365_CLIENT_ID:
+        return {"clientId": PUBLIC_M365_CLIENT_ID, "tenant": "organizations", "from": "default"}
+    return None
+
+
+def microsoft_problem(client_id, tenant):
+    """Why a client and tenant cannot be this lab's Microsoft app, or None."""
+    if not GUID.fullmatch(client_id):
+        return "a client ID is a GUID, the app registration's Application (client) ID"
+    if tenant.lower() in ("common", "consumers"):
+        # Microsoft retired cloud flows for personal accounts on 2025-07-26,
+        # so an authority that admits them only admits a sign-in with no flows.
+        return "a personal Microsoft account has no Power Automate flows: name a tenant, or organizations"
+    if not (tenant == "organizations" or GUID.fullmatch(tenant) or TENANT_DOMAIN.fullmatch(tenant)):
+        return "a tenant is its Directory (tenant) ID, one of its domains, or organizations"
+    return None
 
 # How much of a flow a Source keeps: a definition's size, and how many of its
 # versions -- the documents rule's ten.
@@ -420,7 +455,7 @@ SOURCE_TYPES["power-automate"]["take"] = take_record
 
 # The sign-ins a Source type may be made with, and whether this lab has each:
 # a type naming one this lab lacks cannot be made.
-SIGN_INS = {"microsoft": lambda: M365_CLIENT_ID is not None}
+SIGN_INS = {"microsoft": lambda: microsoft_config() is not None}
 
 
 # The browser stores that follow their user between devices, by their
@@ -2668,6 +2703,43 @@ class Connections:
         self.problem = None
 
 
+class MicrosoftApp:
+    """The Microsoft app entered in Setup: a client ID and a tenant, neither
+    secret, kept so a lab installed rather than deployed needs no environment
+    variable to read a flow."""
+
+    KEYS = ("m365.clientId", "m365.tenant")
+
+    def __init__(self, store: Store):
+        self.store = store
+        with store.lock, closing(sqlite3.connect(store.path)) as db, db:
+            db.execute("CREATE TABLE IF NOT EXISTS settings (key TEXT PRIMARY KEY, value TEXT NOT NULL)")
+
+    def get(self):
+        with self.store.lock, closing(sqlite3.connect(self.store.path)) as db:
+            rows = dict(db.execute("SELECT key, value FROM settings WHERE key IN (?, ?)", self.KEYS))
+        client = rows.get(self.KEYS[0])
+        return {"clientId": client, "tenant": rows.get(self.KEYS[1]) or "organizations"} if client else None
+
+    def set(self, client_id, tenant):
+        """Keeps them, or with no client ID forgets them: (kept, error)."""
+        if not isinstance(client_id, str) or not isinstance(tenant, str):
+            return None, (400, "a Microsoft app is { clientId, tenant }")
+        client_id, tenant = client_id.strip(), tenant.strip() or "organizations"
+        if tenant.lower() == "organizations":
+            tenant = "organizations"
+        with self.store.lock, closing(sqlite3.connect(self.store.path)) as db, db:
+            if not client_id:
+                db.execute("DELETE FROM settings WHERE key IN (?, ?)", self.KEYS)
+                return None, None
+            bad = microsoft_problem(client_id, tenant)
+            if bad:
+                return None, (400, bad)
+            db.executemany("INSERT INTO settings VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value = excluded.value",
+                           zip(self.KEYS, (client_id, tenant)))
+        return self.get(), None
+
+
 class Plugins:
     """The installed plugins: their files on disk, their manifests in the store."""
 
@@ -3873,10 +3945,11 @@ if DATA_DIR:
     PACKS = Packs(STORE)
     PLUGINS = Plugins(STORE)
     CONNECTIONS = Connections(STORE)
+    MICROSOFT = MicrosoftApp(STORE)
     QUEUE = Queue(STORE)
     QUEUE.prompts = PROMPTS
 else:
-    STORE = SOURCES = PROMPTS = DATASETS = PACKS = PLUGINS = CONNECTIONS = QUEUE = None
+    STORE = SOURCES = PROMPTS = DATASETS = PACKS = PLUGINS = CONNECTIONS = MICROSOFT = QUEUE = None
 
 
 class NoRedirects(urllib.request.HTTPRedirectHandler):
@@ -4030,9 +4103,10 @@ class Handler(BaseHTTPRequestHandler):
             # The lab's own limits, for the page to disable rather than
             # hard-code: how many scenarios a run may have.
             # And the Microsoft app a flow is read with, when there is one.
-            return self._json(200, {"scenarioCap": SCENARIO_CAP,
-                                    "microsoft": {"clientId": M365_CLIENT_ID, "tenant": M365_TENANT}
-                                    if M365_CLIENT_ID else None})
+            # The app may be entered in Setup unless the deployment names
+            # its own, and only in a lab with a store to keep it in.
+            return self._json(200, {"scenarioCap": SCENARIO_CAP, "microsoft": microsoft_config(),
+                                    "microsoftEditable": MICROSOFT is not None and not M365_CLIENT_ID})
         if path == "/api/state":
             if STORE is None:
                 return self._send(404, b"not found", "text/plain")
@@ -4307,6 +4381,15 @@ class Handler(BaseHTTPRequestHandler):
         parts = self.path.split("?", 1)[0].split("/")
         if len(parts) == 4 and parts[:3] == ["", "api", "prompts"] and PROMPTS is not None:
             return self._prompts_put(parts[3])
+        if parts == ["", "api", "microsoft"] and MICROSOFT is not None:
+            if M365_CLIENT_ID:
+                return self._json(409, {"error": "this lab's Microsoft app is its deployment's (M365_CLIENT_ID)"})
+            payload = self._payload() or {}
+            _, err = MICROSOFT.set(payload.get("clientId"), payload.get("tenant", ""))
+            if err:
+                code, message = err
+                return self._json(code, {"error": message})
+            return self._json(200, {"microsoft": microsoft_config()})
         if len(parts) == 5 and parts[:3] == ["", "api", "sources"] and parts[4] == "definition" \
                 and SOURCES is not None:
             payload, err = self._dataset_payload(DEFINITION_CAP * 2, "a definition")
@@ -4857,7 +4940,7 @@ class Handler(BaseHTTPRequestHandler):
             # made in a lab without that sign-in.
             sign_in = (SOURCE_TYPES.get(kind) or {}).get("signIn")
             if sign_in and not SIGN_INS.get(sign_in, lambda: False)():
-                return self._json(400, {"error": "this lab has no Microsoft app to sign in with (M365_CLIENT_ID)"
+                return self._json(400, {"error": "this lab has no Microsoft app to sign in with (Setup › Connections)"
                                          if sign_in == "microsoft" else f"this lab has no {sign_in} sign-in"})
             source, err = SOURCES.create(payload.get("name"), kind, payload.get("config"))
             if err: