scenescout 3.10.0 → 3.11.0

package/CHANGELOG.md

@@ -1,5 +1,15 @@
 # scenescout
 
+## 3.11.0
+
+### Minor Changes
+
+- f61141e: `scenescout check` replays saved flows and re-tests open findings. Flows saved as `.scenescout/flows/*.json` use `scout_run_plan`'s steps (navigate, click, type, select, press) plus `expect-text`, `expect-url` and `expect-request`; each is replayed after the crawl with no model, and a flow whose step breaks fails the gate with a high `flow-step-failed` issue naming the flow and the step. A flow file that is not valid stops the check with exit 2, naming the file and the field. Open findings in the project's memory that a page load can reproduce (a failed GET, nothing done on the page but looking) are re-tested and reported as still reproducing or possibly fixed; the memory is never written. New options, also inputs of the GitHub Action: `--flows <dir|off>`, `--retest on|off`, and three settings for what a check may do, whose effective values are printed under the verdict and in `check.json`: `--flow-writes never|allow` (default `never`: flows replay under observe's rule whatever `--mode` says; `allow` replays them under `--mode`, so their form submissions are sent), `--on-refused-step report|stop` (default `report`: a flow whose step was refused is marked "could not run" in the report, JSON and SARIF, every other verdict is kept, and the check exits 2; `stop` exits 2 at once with no results) and `--gate-retests never|high|all` (default `high`: a re-tested finding filed high that still reproduces fails the gate). The defaults are what an unconfigured check does, for a first try or an agent running it unattended: its flows send no HTTP write and it never silently hides a result. A refused beacon or ping (`navigator.sendBeacon`, `<a ping>`) is listed in the flow's result as a background request and charged to no step, whatever its origin, while every other refused write (fetch, XHR, form post) is charged to its step wherever it goes; a write the last step sets off within 750 ms is still charged to it; the WebSocket connections a flow's page opened are listed; a `role=` target must name an ARIA role; anything else in the flows directory is listed as skipped with its reason. Pages loaded only to re-test a finding are measured and nothing else: they are not checked routes, do not count towards `--max-routes`, and add nothing to the route list. The GitHub Action also outputs `could-not-run` and `retests-failing`, and its annotation says when re-tested findings are among what failed the gate. `scout_run_plan` targets accept `role=<role>[name="…"]`, and the `.gitignore` SceneScout writes in `.scenescout/` no longer ignores `flows/*.json`, so flows can be committed.
+
+### Patch Changes
+
+- a1f8011: The benchmark has a second, held-out app (`npm run holdout:serve`), a library loans desk with its own planted defects and answer key, which nothing in the engine or the skill is tuned against. `npm run bench` takes `--app demo|holdout` to choose the key a run is scored with (the demo stays the default), records the app in every new archive, refuses to score an archive against another app's key, and `--all` scores each archive against its own app's key, one table per app.
+
 ## 3.10.0
 
 ### Minor Changes

package/README.md

@@ -363,7 +363,7 @@ npx scenescout check http://127.0.0.1:3000 --fail-on high
 |---|---|
 | 0 | Passed the gate |
 | 1 | Failed it: something at the `--fail-on` severity or worse |
-| 2 | Could not run: a bad argument, an app that never answered, or only the sign-in page reached |
+| 2 | Could not run, or not all of it: a bad argument, an app that never answered, only the sign-in page reached, a saved flow that is not valid, or a flow step the write policy refused |
 
 It writes `report.md`, `check.sarif` (for code-scanning dashboards) and `check.json` to `.scenescout/check/`, and on GitHub Actions it also puts the report on the job's summary page.
 
@@ -377,16 +377,24 @@ On GitHub Actions, this repository is also an action that installs everything an
 
 [docs/ci.md](docs/ci.md) has a complete workflow (start the app, wait for it, check it), the action's inputs and outputs, code-scanning upload, and the same check on GitLab CI, CircleCI or any shell.
 
-It never writes to the app (`--mode observe` or `read-only`), and by default it fails only on facts that mean a page is broken: a page that did not load, an uncaught exception, a 5xx, a failure shown as success. Other options:
+With the default settings its saved flows send no HTTP write (they replay under observe's rule), and its crawl runs under `--mode observe` or `read-only`; `--flow-writes allow` lets flows write as `--mode` allows. By default it fails only on facts that mean a page is broken: a page that did not load, an uncaught exception, a 5xx, a failure shown as success. Other options:
 
 - `--fail-on medium` or `low` makes the gate stricter.
 - `--ignore <rule>` drops a rule.
 - `--paths /a,/b` checks only those pages.
 - `--storage-state <file>` checks while signed in.
+- `--flows <dir>` or `off` chooses which saved flows to replay; `--retest off` skips re-testing open findings.
+- `--flow-writes never|allow` (default `never`): `never` replays flows under observe's rule whatever `--mode` says; `allow` replays them under `--mode`, so in `read-only` a flow's form submissions are sent to the target on every run.
+- `--on-refused-step report|stop` (default `report`): `report` marks a flow whose step was refused "could not run", keeps every other verdict and exits 2; `stop` exits 2 at that step with no results.
+- `--gate-retests never|high|all` (default `high`): which still-reproducing re-tested findings fail the gate.
+
+The defaults are what an unconfigured check does, for a first try or an AI agent running it unattended: its flows send no HTTP write and it never silently hides a result. Each setting is a choice for the project; the report and `check.json` print the values a check ran with.
 
 `scenescout check --help` lists every option. Why the defaults are what they are: [ADR 11](docs/adr/0011-a-gate-is-deterministic-and-fails-only-on-what-it-can-prove.md).
 
-The check fills no forms and follows no flows. That is the exploratory run's job, and its findings belong in a report, not a gate.
+It also replays the flows saved in `.scenescout/flows/*.json`, with no model: the steps `scout_run_plan` takes (navigate, click, type, select, press) plus `expect-text`, `expect-url` and `expect-request`. A flow whose step breaks fails the gate, naming the flow and the step. And it re-tests the open findings earlier runs left in the project's memory that a page load can reproduce, reporting each as still reproducing or possibly fixed; by default a finding filed high that still reproduces fails the gate. [docs/ci.md](docs/ci.md#saved-flows) has the flow format; [ADR 12](docs/adr/0012-a-check-replays-saved-flows-and-reports-re-tests.md) says why it works this way.
+
+Beyond those flows it explores nothing and fills no forms. That is the exploratory run's job, and its findings belong in a report, not a gate.
 
 ---
 
@@ -652,6 +660,7 @@ The load-bearing choices are recorded as ADRs — read the relevant one before c
 - [9 · A refused write is answered, not dropped](docs/adr/0009-a-refused-write-is-answered-not-dropped.md)
 - [10 · A lane's confidence is checked, not trusted](docs/adr/0010-a-confidence-is-checked-not-trusted.md)
 - [11 · A gate is deterministic, and fails only on what it can prove](docs/adr/0011-a-gate-is-deterministic-and-fails-only-on-what-it-can-prove.md)
+- [12 · A check replays saved flows and re-tests open findings, within settings whose defaults do the least harm](docs/adr/0012-a-check-replays-saved-flows-and-reports-re-tests.md)
 
 ---
 

package/dist/browsers.js

@@ -172,6 +172,26 @@ export function focusAdvanceKey(engine, platform) {
 export function echoesFailedLoads(engine) {
     return engine !== "firefox";
 }
+/**
+ * The resource type a navigator.sendBeacon request carries when it is
+ * intercepted: Chromium calls it "ping", Firefox and WebKit "beacon". A saved
+ * flow excuses a refused beacon from its step (flow.ts `splitRefusals`), so
+ * every engine's name for it must be known.
+ */
+export function beaconResourceType(engine) {
+    return engine === "chromium" ? "ping" : "beacon";
+}
+/**
+ * Whether a beacon a page sends as it is being left (sendBeacon, or a
+ * keepalive fetch, on `pagehide`) escapes interception. In Chromium it is
+ * never routed, so the write policy does not see it and it reaches the server
+ * in every mode; Firefox and WebKit route it, and the policy refuses it like
+ * any other. A limit of the network-layer policy (ADR 2), reported in ADR 12;
+ * the check smoke suite asserts both directions per engine.
+ */
+export function unloadBeaconsEscapePolicy(engine) {
+    return engine === "chromium";
+}
 /**
  * Run in every page before its own scripts: takes shared workers away.
  *

package/dist/check-run.js

@@ -1,17 +1,48 @@
 /**
  * Runs `scenescout check`: attach, crawl every route the engine can find,
- * measure each one, and write the verdict. The rules live in engine/check.ts;
- * this file only drives the browser and the files.
+ * measure each one, replay the saved flows, re-test the open findings a page
+ * load can reproduce, and write the verdict. The rules live in
+ * engine/check.ts, engine/flow.ts and engine/verify.ts; this file only drives
+ * the browser and the files.
  */
 import fs from "node:fs";
 import os from "node:os";
 import path from "node:path";
 import { BrowserEngine } from "./engine/browser.js";
-import { issuesFromRoutes, redactRoute, redactRoutes, withoutOwnResponse } from "./engine/check.js";
+import { issuesFromRoutes, redactFlowRuns, redactRoute, redactRoutes, settingsOf, withoutOwnResponse, } from "./engine/check.js";
+import { loadFlows, resolveFlowsDir } from "./engine/flow.js";
 import { MemoryStore, MEMORY_DIRNAME } from "./engine/memory.js";
+import { checkRetestPlan, retestResults, wellFormedFindings } from "./engine/verify.js";
 /** Link discovery rounds: each crawl reveals the routes its pages link to. Past a few, a site is paginating rather than revealing. */
 const MAX_ROUNDS = 6;
-export async function runCheck(options, log = () => { }) {
+/**
+ * Read the flows and the findings, before any browser starts. Throws with a
+ * sentence naming the file and the field on anything it cannot read: a flow
+ * that is silently skipped is a flow that silently passes.
+ */
+export function readCheckInputs(options) {
+    const where = resolveFlowsDir(options.flows, options.projectDir, (p) => fs.existsSync(p) && fs.statSync(p).isDirectory());
+    if ("error" in where)
+        throw new Error(where.error);
+    const { flows, skipped: skippedFlows } = where.dir ? loadFlows(where.dir) : { flows: [], skipped: [] };
+    if (!options.retest)
+        return { flows, skippedFlows, findings: null };
+    // Read, never written: a check leaves the project's memory as it found it.
+    const memoryPath = path.join(options.projectDir, MEMORY_DIRNAME, "memory.json");
+    if (!fs.existsSync(memoryPath))
+        return { flows, skippedFlows, findings: null };
+    let findings;
+    try {
+        findings = JSON.parse(fs.readFileSync(memoryPath, "utf8")).findings;
+    }
+    catch (err) {
+        throw new Error(`could not read ${memoryPath} to re-test its findings (${err instanceof Error ? err.message : String(err)}). Pass --retest off to check without it`);
+    }
+    if (findings !== undefined && !Array.isArray(findings))
+        throw new Error(`${memoryPath}: "findings" is not a list. Pass --retest off to check without it`);
+    return { flows, skippedFlows, findings: wellFormedFindings(findings ?? []) };
+}
+export async function runCheck(options, log = () => { }, inputs = { flows: [], findings: null }) {
     // A throwaway memory: a check is one run, and a store shared with earlier
     // exploratory runs would count their visits as this check's and skip those routes.
     const scratch = fs.mkdtempSync(path.join(os.tmpdir(), "scenescout-check-"));
@@ -52,17 +83,60 @@ export async function runCheck(options, log = () => { }) {
                 log(`  ${routes.length} route(s) checked`);
             }
         }
+        // Pages of open findings the crawl did not load exactly: loaded now, so each re-test has its own measurement.
+        // Kept apart from `routes`: they are measured only to re-test, never checked against the page rules, and do not count
+        // towards --max-routes. With --paths the check stays on the paths it was given.
+        const plan = inputs.findings ? checkRetestPlan(inputs.findings) : null;
+        const retestPages = [];
+        if (plan && !options.paths) {
+            const missing = [...new Set(plan.candidates.map((c) => c.path))].filter((p) => !routes.some((r) => r.path === p));
+            if (missing.length > 0) {
+                await engine.crawl(missing, { limit: missing.length, measureOnly: true });
+                retestPages.push(...engine.lastCrawlHealth);
+                log(`  ${missing.length} page(s) of open findings loaded only to re-test them`);
+            }
+        }
+        const flowRuns = [];
+        for (const flow of inputs.flows) {
+            // --flow-writes never: observe's rule, whatever --mode lets the crawl do.
+            const replay = await engine.replayFlow(flow.steps, options.flowWrites === "never" ? "observe" : options.mode);
+            flowRuns.push({ name: flow.name, file: flow.file, steps: flow.steps.length, ...replay });
+            log(`  flow ${flow.name}: ${replay.outcome.status}${replay.outcome.status === "passed" ? "" : ` at step ${replay.outcome.step}`}`);
+            // --on-refused-step stop: nothing after a refused step runs, and the check ends without a verdict.
+            if (replay.outcome.status === "refused" && options.onRefusedStep === "stop")
+                break;
+        }
+        const retest = plan
+            ? {
+                open: plan.open,
+                extraPages: retestPages.length,
+                results: retestResults(plan.candidates, [...routes, ...retestPages].map((r) => ({
+                    path: r.path,
+                    status: r.status,
+                    ...(r.loadError !== undefined ? { loadError: r.loadError } : {}),
+                    loginRedirect: r.loginRedirect,
+                    httpErrors: withoutOwnResponse(r)
+                        .violations.filter((v) => v.kind === "http_error")
+                        .map((v) => v.detail),
+                }))),
+            }
+            : null;
         const measured = redactRoutes(routes.map(withoutOwnResponse));
+        const flows = redactFlowRuns(flowRuns);
         return {
             url: redactRoute(options.url),
             generatedAt: new Date().toISOString(),
             mode: options.mode,
             failOn: options.failOn,
             routes: measured,
-            issues: issuesFromRoutes(measured, start.origin, options.ignore),
+            issues: issuesFromRoutes(measured, start.origin, options.ignore, flows),
             // Routes that failed to load are issues already; "not visited" is only what --max-routes left out.
             unvisited: options.paths ? [] : engine.crawlableRoutes().map(redactRoute),
             ignored: options.ignore,
+            flows,
+            skippedFlows: inputs.skippedFlows ?? [],
+            retest,
+            settings: settingsOf(options),
         };
     }
     finally {

package/dist/cli.js

@@ -17,8 +17,8 @@ import { fileURLToPath } from "node:url";
 import { APPROX_DISK_MB, BROWSER_ENGINES, browserPresence, defaultAttachNote, defaultEngine, launchTarget, parseBrowserSelection, playwrightInstallArgs, } from "./browsers.js";
 import { CLIENT_LABELS, firstMessageHint, manualFor, parseClients, registerWithClient, vscodeBinary } from "./clients.js";
 import { CLI_NAME, diagnose, ensureCommand, findOnUserPath, installSkill, isEphemeralRoot, launchCommand, manualRegisterCommand, planCommand, registerMcp, resolveClaudeDir, spawnRunner, } from "./installer.js";
-import { defaultCheckDir, runCheck } from "./check-run.js";
-import { EXIT, formatCheck, parseCheckArgs, summarise, toSarif, toSummaryJson, unmeasuredReason } from "./engine/check.js";
+import { defaultCheckDir, readCheckInputs, runCheck } from "./check-run.js";
+import { EXIT, exitCodeOf, formatCheck, parseCheckArgs, refusedFlowReason, toSarif, toSummaryJson, unmeasuredReason } from "./engine/check.js";
 import { LEGACY_MEMORY_DIRNAME, MEMORY_DIRNAME, writeSelfIgnore } from "./engine/memory.js";
 import { formatStatus, liveEngines, liveTokenFileName, localClock, LIVE_TOKEN_FILE, pidAlive, watchTarget, wholeSessions, } from "./engine/live.js";
 import { formatScan, scanProject } from "./scan.js";
@@ -53,7 +53,16 @@ Usage:
                                      --max-routes N (default 50); --paths /a,/b to check only those;
                                      --ignore rule,rule; --storage-state file to check signed in;
                                      --project dir (default: here); --out dir (default: .scenescout/check);
-                                     --browser chromium|firefox|webkit)
+                                     --browser chromium|firefox|webkit;
+                                     --flows dir|off: replay the flows saved there (default: .scenescout/flows);
+                                     --retest on|off: re-test open findings a page load reproduces (default on);
+                                     --flow-writes never|allow: never (default) replays flows under observe's
+                                      rule whatever --mode says; allow replays them under --mode, so their
+                                      form submissions are sent;
+                                     --on-refused-step report|stop: report (default) marks a flow whose step was
+                                      refused "could not run", keeps every other verdict and exits 2; stop exits 2 there;
+                                     --gate-retests never|high|all: which still-reproducing findings fail the gate
+                                      (default high: those filed high))
                                     Exit code: 0 passed, 1 failed the gate, 2 could not run.
   scenescout status [projectPath]   What is the engine doing right now? (every session + recent actions)
   scenescout watch [projectPath]    Open the live view in a browser: what each session is doing, a thumbnail
@@ -471,10 +480,18 @@ async function check(args) {
     }
     const options = parsed.options;
     const outDir = options.outDir ?? defaultCheckDir(options.projectDir);
+    let inputs;
+    try {
+        inputs = readCheckInputs(options);
+    }
+    catch (err) {
+        console.error(`scenescout check: ${err instanceof Error ? err.message : String(err)}`);
+        process.exit(EXIT.error);
+    }
     let result;
     try {
-        console.log(`Checking ${options.url} …`);
-        result = await runCheck(options, (line) => console.log(line));
+        console.log(`Checking ${options.url} …${inputs.flows.length > 0 ? ` (and ${inputs.flows.length} saved flow(s))` : ""}`);
+        result = await runCheck(options, (line) => console.log(line), inputs);
     }
     catch (err) {
         console.error(`scenescout check: could not run: ${err instanceof Error ? err.message : String(err)}`);
@@ -485,6 +502,12 @@ async function check(args) {
         console.error(`scenescout check: could not measure ${options.url}: ${unmeasured}`);
         process.exit(EXIT.error);
     }
+    const refused = refusedFlowReason(result);
+    // --on-refused-step stop: no verdict is written at all.
+    if (refused && options.onRefusedStep === "stop") {
+        console.error(`scenescout check: could not run a saved flow: ${refused}`);
+        process.exit(EXIT.error);
+    }
     const markdown = formatCheck(result);
     try {
         const version = packageVersion();
@@ -505,7 +528,10 @@ async function check(args) {
     }
     console.log("\n" + markdown);
     console.log(`Wrote report.md, check.sarif and check.json to ${outDir}`);
-    process.exit(summarise(result).passed ? EXIT.pass : EXIT.gateFailed);
+    // --on-refused-step report: everything else has its verdict in the files, and the exit code still says the run was incomplete.
+    if (refused)
+        console.error(`scenescout check: could not run a saved flow: ${refused}`);
+    process.exit(exitCodeOf(result));
 }
 const [, , command, ...args] = process.argv;
 // A CLI's failure mode should be a sentence, not a stack trace. `scan` on a

package/dist/engine/bench.js

@@ -79,6 +79,19 @@ const NonDefect = z
     counterExamples: z.array(z.string()).default([]),
 })
     .strict();
+const Contextual = z
+    .object({
+    id: z.string().min(1),
+    route: z.string().startsWith("/"),
+    title: z.string().min(1),
+    category: z.string().min(1),
+    /** The convention that decides it: where it would be a defect, and why the run cannot tell whether it holds here. */
+    reason: z.string().min(1),
+    match: z.array(Pattern).min(1),
+    examples: z.array(z.string()).default([]),
+    counterExamples: z.array(z.string()).default([]),
+})
+    .strict();
 const Key = z
     .object({
     app: z.string().min(1),
@@ -92,6 +105,14 @@ const Key = z
      */
     alsoReal: z.array(Entry).default([]),
     nonDefects: z.array(NonDefect).default([]),
+    /**
+     * Observations that are defects only under a convention the run cannot
+     * see — a spacing scale the project may or may not declare, say. Neither
+     * lane is wrong about them, so they are set aside: a finding matching one
+     * is neither correct nor a false positive, and a verdict on one is not
+     * scored for calibration.
+     */
+    contextual: z.array(Contextual).default([]),
 })
     .strict();
 /**
@@ -105,7 +126,7 @@ export function parseKey(raw) {
         const issue = parsed.error.issues[0];
         throw new Error(`The answer key is not valid at ${issue.path.join(".") || "(root)"}: ${issue.message}`);
     }
-    const ids = [...parsed.data.defects, ...parsed.data.alsoReal, ...parsed.data.nonDefects].map((e) => e.id);
+    const ids = [...parsed.data.defects, ...parsed.data.alsoReal, ...parsed.data.nonDefects, ...parsed.data.contextual].map((e) => e.id);
     const dup = ids.find((id, i) => ids.indexOf(id) !== i);
     if (dup)
         throw new Error(`The answer key uses the id ${JSON.stringify(dup)} twice.`);
@@ -119,7 +140,7 @@ export function parseKey(raw) {
 }
 /** A short, stable fingerprint of a key, so a scorecard says which key produced it. */
 export function keyHash(key) {
-    const canonical = JSON.stringify({ defects: key.defects, alsoReal: key.alsoReal, nonDefects: key.nonDefects });
+    const canonical = JSON.stringify({ defects: key.defects, alsoReal: key.alsoReal, nonDefects: key.nonDefects, contextual: key.contextual });
     return createHash("sha256").update(canonical).digest("hex").slice(0, 10);
 }
 const RANK = { minimal: 0, medium: 1, extensive: 2 };
@@ -138,12 +159,19 @@ function matches(patterns, text) {
  * match would quietly count a finding about two defects as one, and letting a
  * non-defect win everywhere turned realistic rewordings of real defects into
  * false positives with a confident explanation beside them.
+ *
+ * A contextual entry is one more claimant on the same terms as a real one: it
+ * wins over nothing, no non-defect can override it, and text it shares with
+ * any other entry is ambiguous. Setting a finding aside removes it from
+ * precision, so a pattern that let it win would hide real defects from the
+ * score as quietly as a greedy non-defect once did.
  */
 export function classify(text, key) {
     const nd = key.nonDefects.filter((e) => matches(e.match, text));
     const real = [
         ...key.defects.filter((e) => matches(e.match, text)).map((entry) => ({ kind: "defect", entry })),
         ...key.alsoReal.filter((e) => matches(e.match, text)).map((entry) => ({ kind: "alsoReal", entry })),
+        ...key.contextual.filter((e) => matches(e.match, text)).map((entry) => ({ kind: "contextual", entry })),
     ];
     if (nd.length > 1)
         return { kind: "ambiguous", ids: [...nd, ...real.map((m) => m.entry)].map((e) => e.id) };
@@ -171,7 +199,7 @@ export function decisionText(d) {
  * A key that disagrees with its own examples is wrong before any run is
  * scored. Returns every title or example — non-defects' included — that
  * classifies anywhere but its own entry, and every counter-example that
- * classifies TO its entry.
+ * classifies TO its entry. Contextual entries are held to the same rule.
  */
 export function lintKey(key) {
     const problems = [];
@@ -195,11 +223,16 @@ export function lintKey(key) {
                 check(e.id, kind, t, false);
         }
     }
-    for (const e of key.nonDefects) {
-        for (const t of [e.title, ...e.examples])
-            check(e.id, "nonDefect", t, true);
-        for (const t of e.counterExamples)
-            check(e.id, "nonDefect", t, false);
+    for (const [kind, list] of [
+        ["nonDefect", key.nonDefects],
+        ["contextual", key.contextual],
+    ]) {
+        for (const e of list) {
+            for (const t of [e.title, ...e.examples])
+                check(e.id, kind, t, true);
+            for (const t of e.counterExamples)
+                check(e.id, kind, t, false);
+        }
     }
     return problems;
 }
@@ -208,8 +241,8 @@ export function lintKey(key) {
  *
  * "defect" on a planted or also-real defect is right, and on a known
  * non-defect is wrong. "not_a_defect" is the reverse. An "unsure" verdict, a
- * dismissal as another lane's, a decision the key does not name, and one it
- * names ambiguously are not scored — the same rule the in-product calibration
+ * dismissal as another lane's, a decision the key does not name, one it
+ * names ambiguously, and one about a contextual entry are not scored — the same rule the in-product calibration
  * follows, for the same reason.
  */
 export function judgeDecision(d, key) {
@@ -218,7 +251,7 @@ export function judgeDecision(d, key) {
     if (isScopeDismissal(d))
         return null;
     const m = classify(decisionText(d), key);
-    if (!m || m.kind === "ambiguous")
+    if (!m || m.kind === "ambiguous" || m.kind === "contextual")
         return null;
     const isDefect = m.kind !== "nonDefect";
     return d.verdict === "defect" ? isDefect : !isDefect;
@@ -233,7 +266,19 @@ export function calibrateAgainstKey(decisions, key) {
     if (decisions.length === 0)
         return null;
     const buckets = BUCKET_EDGES.map(() => ({ n: 0, conf: 0, right: 0 }));
-    const out = { judged: 0, correct: 0, notInKey: 0, ambiguous: 0, badConfidence: 0, unsure: 0, outOfScope: 0, buckets: [], ece: 0, brier: 0 };
+    const out = {
+        judged: 0,
+        correct: 0,
+        notInKey: 0,
+        ambiguous: 0,
+        badConfidence: 0,
+        unsure: 0,
+        outOfScope: 0,
+        contextual: 0,
+        buckets: [],
+        ece: 0,
+        brier: 0,
+    };
     for (const d of decisions) {
         if (d.verdict === "unsure") {
             out.unsure += 1;
@@ -257,6 +302,10 @@ export function calibrateAgainstKey(decisions, key) {
             out.ambiguous += 1;
             continue;
         }
+        if (m.kind === "contextual") {
+            out.contextual += 1;
+            continue;
+        }
         const right = d.verdict === "defect" ? m.kind !== "nonDefect" : m.kind === "nonDefect";
         out.judged += 1;
         if (right)
@@ -290,6 +339,7 @@ export function score(key, findings, decisions, level = "medium") {
     const falsePositives = [];
     const unknown = [];
     const ambiguous = [];
+    const contextual = [];
     let correct = 0;
     for (const f of findings) {
         const m = classify(findingText(f), key);
@@ -305,6 +355,10 @@ export function score(key, findings, decisions, level = "medium") {
             falsePositives.push({ id: m.entry.id, title: f.title, why: m.entry.why, severity: f.severity });
             continue;
         }
+        if (m.kind === "contextual") {
+            contextual.push({ id: m.entry.id, title: f.title });
+            continue;
+        }
         correct += 1;
         if (m.kind === "defect") {
             const list = hitsByDefect.get(m.entry.id) ?? [];
@@ -355,6 +409,7 @@ export function score(key, findings, decisions, level = "medium") {
         findings: findings.length,
         correct,
         falsePositives,
+        contextual,
         unknown,
         ambiguous,
         duplicates,
@@ -367,12 +422,14 @@ const pct = (n, d) => (d === 0 ? "—" : `${Math.round((n / d) * 100)}%`);
  * Precision, with its bounds. The labelled ratio leaves unlabelled findings out
  * of the denominator, so on its own it cannot move when a change adds five new
  * false claims nobody has judged yet. The bounds can: the lower one counts
- * every open finding as wrong, the upper one as right.
+ * every open finding as wrong, the upper one as right. Findings set aside as
+ * contextual are in neither: they are not claims the run could get right.
  */
 export function precisionBounds(c) {
     const labelled = c.correct + c.falsePositives.length;
     const open = c.unknown.length + c.ambiguous.length;
-    return { labelled: `${c.correct}/${labelled} (${pct(c.correct, labelled)})`, low: pct(c.correct, c.findings), high: pct(c.correct + open, c.findings) };
+    const scored = c.findings - c.contextual.length;
+    return { labelled: `${c.correct}/${labelled} (${pct(c.correct, labelled)})`, low: pct(c.correct, scored), high: pct(c.correct + open, scored) };
 }
 /** The scorecard as a person reads it. Leads with the two numbers, then says what each is made of. */
 export function formatScorecard(c) {
@@ -383,9 +440,13 @@ export function formatScorecard(c) {
         ``,
         `Recall     ${c.found.length}/${c.expected} (${pct(c.found.length, c.expected)}) of the planted defects expected at this level`,
         `Precision  ${p.labelled} of the findings the key can label` +
-            (open ? ` — ${open} of ${c.findings} unlabelled, so between ${p.low} and ${p.high} of all findings` : ""),
-        ``,
+            (open
+                ? ` — ${open} of ${c.findings - c.contextual.length} unlabelled, so between ${p.low} and ${p.high} of ${c.contextual.length ? "the findings not set aside" : "all findings"}`
+                : ""),
     ];
+    if (c.contextual.length)
+        lines.push(`Set aside  ${c.contextual.length} of ${c.findings} finding(s): defects only under a convention the run cannot see, so in neither count`);
+    lines.push(``);
     if (c.missed.length)
         lines.push(`Missed: ${c.missed.join(", ")}`);
     if (c.judgedNotFiled.length)
@@ -397,6 +458,11 @@ export function formatScorecard(c) {
         for (const fp of c.falsePositives)
             lines.push(`  [${fp.severity}] ${fp.title} — ${fp.why}`);
     }
+    if (c.contextual.length) {
+        lines.push(``, `Set aside as contextual (${c.contextual.length}):`);
+        for (const x of c.contextual)
+            lines.push(`  ${x.title} (${x.id})`);
+    }
     if (c.ambiguous.length) {
         lines.push(``, `Ambiguous (${c.ambiguous.length}) — the key claims each twice; sharpen it:`);
         for (const a of c.ambiguous)
@@ -425,6 +491,7 @@ export function formatScorecard(c) {
             k.badConfidence && `${k.badConfidence} with an unusable confidence`,
             k.unsure && `${k.unsure} unsure`,
             k.outOfScope && `${k.outOfScope} dismissed as another lane's`,
+            k.contextual && `${k.contextual} about things that are defects only under a convention the run cannot see`,
         ].filter(Boolean);
         lines.push(``, k.judged === 0
             ? `Lane calibration against the key: nothing the key could judge` + (skipped.length ? ` (${skipped.join(", ")})` : "")
@@ -460,6 +527,13 @@ export function runDate(decisions, given) {
         throw new Error(`The run's last decision was made on ${last}; it cannot be dated ${given}.`);
     return given ?? last;
 }
+/**
+ * The order `--all` lists runs in: by date, then by name with its numbers
+ * compared as numbers, so run-10 follows run-9 rather than run-1.
+ */
+export function byRunOrder(a, b) {
+    return a.date.localeCompare(b.date) || a.run.localeCompare(b.run, undefined, { numeric: true });
+}
 /**
  * Paths into a machine's own directories, which a finding's evidence can pick
  * up from a stack trace or an upload. An archive is committed, and a home
@@ -469,8 +543,9 @@ const LOCAL_PATH_RE = /(?:\/Users\/[^/\s"']+|\/home\/[^/\s"']+|\/private\/tmp|\/
 export function sanitize(text) {
     return text.replace(LOCAL_PATH_RE, "<path>");
 }
-export function toArchive(run, date, note, findings, decisions) {
+export function toArchive(run, date, note, findings, decisions, app) {
     return {
+        app,
         run,
         date,
         note,
@@ -483,3 +558,65 @@ export function toArchive(run, date, note, findings, decisions) {
         decisions: decisions.map((d) => ({ ...d, observation: sanitize(d.observation), evidence: d.evidence === null ? null : sanitize(d.evidence) })),
     };
 }
+// ── which app, and so which key ─────────────────────────────────────────────
+/**
+ * The app a run is for when nothing says otherwise. Every archive made before
+ * there was a second benchmark app is this one's, and carries no `app`.
+ */
+export const DEFAULT_APP = "demo";
+/** The app an archive was made against. */
+export function archiveApp(a) {
+    return a.app ?? DEFAULT_APP;
+}
+/**
+ * The app whose key scores a run. Scoring a run against another app's key
+ * produces a plausible wrong number — every planted defect "missed", every
+ * finding "unlabelled" — so a request that disagrees with what the archive
+ * records is refused rather than obeyed.
+ *
+ * `requested` is what the caller asked for (`--app`), `archived` what the
+ * run's archive records, and `known` the apps that have a key.
+ */
+export function chooseApp(opts) {
+    const { requested, archived, known } = opts;
+    for (const [what, app] of [
+        ["--app", requested],
+        ["The archive's app", archived],
+    ]) {
+        if (app !== undefined && !known.includes(app))
+            throw new Error(`${what} ${JSON.stringify(app)} is not a benchmark app; the apps are ${known.join(", ")}.`);
+    }
+    if (requested !== undefined && archived !== undefined && requested !== archived)
+        throw new Error(`This run was archived for the ${archived} app; scoring it against the ${requested} app's key would score the wrong answers.`);
+    return requested ?? archived ?? DEFAULT_APP;
+}
+/**
+ * Which benchmark app a key file belongs to, by the name the key gives its
+ * app, or undefined for a key no benchmark app ships (a draft, a test key).
+ * A copy of one app's key, edited or not, still names that app.
+ */
+export function appOfKey(key, keys) {
+    return Object.keys(keys).find((app) => keys[app].app === key.app);
+}
+/**
+ * Refuse a key file that belongs to another benchmark app than the archive
+ * being scored: `--key` naming the held-out key scored a demo archive against
+ * the wrong answers without a word.
+ */
+export function checkKeyForArchive(key, archive, keys) {
+    const keyApp = appOfKey(key, keys);
+    const runApp = archiveApp(archive);
+    if (keyApp !== undefined && keyApp !== runApp)
+        throw new Error(`This run was archived for the ${runApp} app, and that key is the ${keyApp} app's; scoring one against the other would score the wrong answers.`);
+}
+/** Archives grouped by app, in the order each app is first met, each group in `byRunOrder`. */
+export function groupByApp(archives) {
+    const groups = new Map();
+    for (const a of archives) {
+        const app = archiveApp(a);
+        groups.set(app, [...(groups.get(app) ?? []), a]);
+    }
+    for (const list of groups.values())
+        list.sort(byRunOrder);
+    return groups;
+}