scenescout 3.10.0 → 3.11.1

package/CHANGELOG.md

@@ -1,5 +1,23 @@
 # scenescout
 
+## 3.11.1
+
+### Patch Changes
+
+- 847c149: Two different defects on one element now stay two findings. Finding dedup compares what is wrong as well as where: identical evidence, a shared quoted label or a reworded title merges two findings only when they are one kind of defect (the same category, or neighbouring categories of one family such as `page-error` and `console-error`). `visual`, `ux-polish`, `a11y` and `missing-testid` are each their own kind, so a link clipped out of view and the same link styled like body text are no longer merged. When `scout_finding` merges a filing it now names the category of the finding it joined and which categories the filing could have merged with.
+- aca7a28: When a lane report is folded, a judged defect on a failing request now counts as filed when a finding names that request with its path written as a template: `GET /api/things/{id} 500` (or `:id`, or `*`) covers a lane's `GET /api/things/7 500`. A template stands only for one id segment (a number, a UUID or a long hex id), never for a word such as `me` or `export`. A defect naming several failing requests counts as filed only when every one of them is. On a request that did not fail, the lane's evidence, once the paths are aligned, must be the finding's, a restatement of part of it, or the finding's evidence followed only by the status the call should have returned, as in `… 200 as role=viewer (expected 403)`.
+- 2b8ac9a: The write policy now judges writes a page sends as it is being left. In Chromium, a `navigator.sendBeacon` or `fetch(..., { keepalive: true })` sent on `pagehide`, `visibilitychange` or `unload` was never intercepted and reached the server in every mode, observe included; the engine now catches it at the browser level and applies the same rules, so under observe (and under read-only, for a destructive write) it is refused and reported with the other refused writes, and a mode that allows it still sends it. Such a write is recognised by its headers, since the page that sent it is gone: one out of the app whose Origin is another site's, or `null`, is treated as an embed's and refused outside destructive, and the exception for a captcha on the app's sign-in page is decided on the Referer, the page it was sent from. Browsers send only the origin as the Referer of a request to another site by default, so that exception seldom applies and such a captcha write is refused; a sign-in the tester completes while the page is open is not affected. In Chromium, a write a 307 or 308 redirect carries on to a new address is now judged there as well. In every browser, the pages the engine closes (at the end of a session, on a re-attach, and a popup of another site) are left for `about:blank` first, so what they send on the way out meets the policy too: in Firefox and WebKit such a write had also gone out unseen.
+
+## 3.11.0
+
+### Minor Changes
+
+- f61141e: `scenescout check` replays saved flows and re-tests open findings. Flows saved as `.scenescout/flows/*.json` use `scout_run_plan`'s steps (navigate, click, type, select, press) plus `expect-text`, `expect-url` and `expect-request`; each is replayed after the crawl with no model, and a flow whose step breaks fails the gate with a high `flow-step-failed` issue naming the flow and the step. A flow file that is not valid stops the check with exit 2, naming the file and the field. Open findings in the project's memory that a page load can reproduce (a failed GET, nothing done on the page but looking) are re-tested and reported as still reproducing or possibly fixed; the memory is never written. New options, also inputs of the GitHub Action: `--flows <dir|off>`, `--retest on|off`, and three settings for what a check may do, whose effective values are printed under the verdict and in `check.json`: `--flow-writes never|allow` (default `never`: flows replay under observe's rule whatever `--mode` says; `allow` replays them under `--mode`, so their form submissions are sent), `--on-refused-step report|stop` (default `report`: a flow whose step was refused is marked "could not run" in the report, JSON and SARIF, every other verdict is kept, and the check exits 2; `stop` exits 2 at once with no results) and `--gate-retests never|high|all` (default `high`: a re-tested finding filed high that still reproduces fails the gate). The defaults are what an unconfigured check does, for a first try or an agent running it unattended: its flows send no HTTP write and it never silently hides a result. A refused beacon or ping (`navigator.sendBeacon`, `<a ping>`) is listed in the flow's result as a background request and charged to no step, whatever its origin, while every other refused write (fetch, XHR, form post) is charged to its step wherever it goes; a write the last step sets off within 750 ms is still charged to it; the WebSocket connections a flow's page opened are listed; a `role=` target must name an ARIA role; anything else in the flows directory is listed as skipped with its reason. Pages loaded only to re-test a finding are measured and nothing else: they are not checked routes, do not count towards `--max-routes`, and add nothing to the route list. The GitHub Action also outputs `could-not-run` and `retests-failing`, and its annotation says when re-tested findings are among what failed the gate. `scout_run_plan` targets accept `role=<role>[name="…"]`, and the `.gitignore` SceneScout writes in `.scenescout/` no longer ignores `flows/*.json`, so flows can be committed.
+
+### Patch Changes
+
+- a1f8011: The benchmark has a second, held-out app (`npm run holdout:serve`), a library loans desk with its own planted defects and answer key, which nothing in the engine or the skill is tuned against. `npm run bench` takes `--app demo|holdout` to choose the key a run is scored with (the demo stays the default), records the app in every new archive, refuses to score an archive against another app's key, and `--all` scores each archive against its own app's key, one table per app.
+
 ## 3.10.0
 
 ### Minor Changes

package/README.md

@@ -363,7 +363,7 @@ npx scenescout check http://127.0.0.1:3000 --fail-on high
 |---|---|
 | 0 | Passed the gate |
 | 1 | Failed it: something at the `--fail-on` severity or worse |
-| 2 | Could not run: a bad argument, an app that never answered, or only the sign-in page reached |
+| 2 | Could not run, or not all of it: a bad argument, an app that never answered, only the sign-in page reached, a saved flow that is not valid, or a flow step the write policy refused |
 
 It writes `report.md`, `check.sarif` (for code-scanning dashboards) and `check.json` to `.scenescout/check/`, and on GitHub Actions it also puts the report on the job's summary page.
 
@@ -377,16 +377,24 @@ On GitHub Actions, this repository is also an action that installs everything an
 
 [docs/ci.md](docs/ci.md) has a complete workflow (start the app, wait for it, check it), the action's inputs and outputs, code-scanning upload, and the same check on GitLab CI, CircleCI or any shell.
 
-It never writes to the app (`--mode observe` or `read-only`), and by default it fails only on facts that mean a page is broken: a page that did not load, an uncaught exception, a 5xx, a failure shown as success. Other options:
+With the default settings its saved flows send no HTTP write (they replay under observe's rule), and its crawl runs under `--mode observe` or `read-only`; `--flow-writes allow` lets flows write as `--mode` allows. By default it fails only on facts that mean a page is broken: a page that did not load, an uncaught exception, a 5xx, a failure shown as success. Other options:
 
 - `--fail-on medium` or `low` makes the gate stricter.
 - `--ignore <rule>` drops a rule.
 - `--paths /a,/b` checks only those pages.
 - `--storage-state <file>` checks while signed in.
+- `--flows <dir>` or `off` chooses which saved flows to replay; `--retest off` skips re-testing open findings.
+- `--flow-writes never|allow` (default `never`): `never` replays flows under observe's rule whatever `--mode` says; `allow` replays them under `--mode`, so in `read-only` a flow's form submissions are sent to the target on every run.
+- `--on-refused-step report|stop` (default `report`): `report` marks a flow whose step was refused "could not run", keeps every other verdict and exits 2; `stop` exits 2 at that step with no results.
+- `--gate-retests never|high|all` (default `high`): which still-reproducing re-tested findings fail the gate.
+
+The defaults are what an unconfigured check does, for a first try or an AI agent running it unattended: its flows send no HTTP write and it never silently hides a result. Each setting is a choice for the project; the report and `check.json` print the values a check ran with.
 
 `scenescout check --help` lists every option. Why the defaults are what they are: [ADR 11](docs/adr/0011-a-gate-is-deterministic-and-fails-only-on-what-it-can-prove.md).
 
-The check fills no forms and follows no flows. That is the exploratory run's job, and its findings belong in a report, not a gate.
+It also replays the flows saved in `.scenescout/flows/*.json`, with no model: the steps `scout_run_plan` takes (navigate, click, type, select, press) plus `expect-text`, `expect-url` and `expect-request`. A flow whose step breaks fails the gate, naming the flow and the step. And it re-tests the open findings earlier runs left in the project's memory that a page load can reproduce, reporting each as still reproducing or possibly fixed; by default a finding filed high that still reproduces fails the gate. [docs/ci.md](docs/ci.md#saved-flows) has the flow format; [ADR 12](docs/adr/0012-a-check-replays-saved-flows-and-reports-re-tests.md) says why it works this way.
+
+Beyond those flows it explores nothing and fills no forms. That is the exploratory run's job, and its findings belong in a report, not a gate.
 
 ---
 
@@ -652,6 +660,7 @@ The load-bearing choices are recorded as ADRs — read the relevant one before c
 - [9 · A refused write is answered, not dropped](docs/adr/0009-a-refused-write-is-answered-not-dropped.md)
 - [10 · A lane's confidence is checked, not trusted](docs/adr/0010-a-confidence-is-checked-not-trusted.md)
 - [11 · A gate is deterministic, and fails only on what it can prove](docs/adr/0011-a-gate-is-deterministic-and-fails-only-on-what-it-can-prove.md)
+- [12 · A check replays saved flows and re-tests open findings, within settings whose defaults do the least harm](docs/adr/0012-a-check-replays-saved-flows-and-reports-re-tests.md)
 
 ---
 

package/dist/browsers.js

@@ -172,6 +172,70 @@ export function focusAdvanceKey(engine, platform) {
 export function echoesFailedLoads(engine) {
     return engine !== "firefox";
 }
+/**
+ * The resource type a navigator.sendBeacon request carries when it is
+ * intercepted: Chromium calls it "ping", Firefox and WebKit "beacon". A saved
+ * flow excuses a refused beacon from its step (flow.ts `splitRefusals`), so
+ * every engine's name for it must be known.
+ */
+export function beaconResourceType(engine) {
+    return engine === "chromium" ? "ping" : "beacon";
+}
+/**
+ * Where the write policy meets a write a page sends as it is being left (a
+ * sendBeacon or keepalive fetch on `pagehide`, `visibilitychange` or
+ * `unload`, or one still leaving as a page is closed).
+ *
+ * - `route`: Firefox and WebKit hand it to the context's route handler like
+ *   any other request.
+ * - `browser-fetch`: Chromium never routes it (it arrives with no network id,
+ *   or after the page's session has gone, and the driver sends it on), so on
+ *   its own it reached the server in every mode. The engine also pauses
+ *   requests at the browser level with the DevTools Fetch domain and judges
+ *   there, by the same rules, every write the route handler did not
+ *   (unload.ts). Only in modes where the policy can refuse: destructive
+ *   installs neither.
+ *
+ * Either way the write is judged and a refusal is reported like any other
+ * (ADR 2); the unload smoke suite asserts it per engine.
+ */
+export function unloadWriteInterception(engine) {
+    return engine === "chromium" ? "browser-fetch" : "route";
+}
+/**
+ * Whether a write a page sends as it is being left may be lost after the
+ * write policy lets it through. WebKit cancels such a request when the route
+ * handler answers after the page has gone, so a mode that allows the write
+ * does not promise it arrives; nothing is sent that the policy refused. Firefox
+ * sends it on, and so does Chromium's browser-level interception. In
+ * destructive mode nothing is intercepted and every engine sends it.
+ */
+export function allowedUnloadWritesMayBeLost(engine) {
+    return engine === "webkit";
+}
+/**
+ * Whether the writes another site's frame sends on `pagehide` may never be
+ * issued when the top page is left. Firefox can tear such a frame down before
+ * its requests reach the route handler, so there is nothing to refuse or to
+ * report, and nothing is sent. It depends on timing: seen in CI and under load
+ * locally, not on an idle machine. Chromium's browser-level interception and
+ * WebKit's route handler met them every time in the same runs. The writes the
+ * top page itself sends are not affected.
+ */
+export function frameUnloadWritesMayGoUnissued(engine) {
+    return engine === "firefox";
+}
+/**
+ * Whether a write carried on by a redirect (a 307 or 308 keeps the method and
+ * the body) is judged at its new address. The route handler sees only the
+ * first request of a redirect in every engine. In Chromium the browser-level
+ * interception (`unloadWriteInterception`) sees each hop and judges it like any
+ * write; Firefox and WebKit send the later hops on unjudged, a known limit of
+ * the policy there (ADR 2).
+ */
+export function writeRedirectHopsJudged(engine) {
+    return engine === "chromium";
+}
 /**
  * Run in every page before its own scripts: takes shared workers away.
  *

package/dist/check-run.js

@@ -1,17 +1,48 @@
 /**
  * Runs `scenescout check`: attach, crawl every route the engine can find,
- * measure each one, and write the verdict. The rules live in engine/check.ts;
- * this file only drives the browser and the files.
+ * measure each one, replay the saved flows, re-test the open findings a page
+ * load can reproduce, and write the verdict. The rules live in
+ * engine/check.ts, engine/flow.ts and engine/verify.ts; this file only drives
+ * the browser and the files.
  */
 import fs from "node:fs";
 import os from "node:os";
 import path from "node:path";
 import { BrowserEngine } from "./engine/browser.js";
-import { issuesFromRoutes, redactRoute, redactRoutes, withoutOwnResponse } from "./engine/check.js";
+import { issuesFromRoutes, redactFlowRuns, redactRoute, redactRoutes, settingsOf, withoutOwnResponse, } from "./engine/check.js";
+import { loadFlows, resolveFlowsDir } from "./engine/flow.js";
 import { MemoryStore, MEMORY_DIRNAME } from "./engine/memory.js";
+import { checkRetestPlan, retestResults, wellFormedFindings } from "./engine/verify.js";
 /** Link discovery rounds: each crawl reveals the routes its pages link to. Past a few, a site is paginating rather than revealing. */
 const MAX_ROUNDS = 6;
-export async function runCheck(options, log = () => { }) {
+/**
+ * Read the flows and the findings, before any browser starts. Throws with a
+ * sentence naming the file and the field on anything it cannot read: a flow
+ * that is silently skipped is a flow that silently passes.
+ */
+export function readCheckInputs(options) {
+    const where = resolveFlowsDir(options.flows, options.projectDir, (p) => fs.existsSync(p) && fs.statSync(p).isDirectory());
+    if ("error" in where)
+        throw new Error(where.error);
+    const { flows, skipped: skippedFlows } = where.dir ? loadFlows(where.dir) : { flows: [], skipped: [] };
+    if (!options.retest)
+        return { flows, skippedFlows, findings: null };
+    // Read, never written: a check leaves the project's memory as it found it.
+    const memoryPath = path.join(options.projectDir, MEMORY_DIRNAME, "memory.json");
+    if (!fs.existsSync(memoryPath))
+        return { flows, skippedFlows, findings: null };
+    let findings;
+    try {
+        findings = JSON.parse(fs.readFileSync(memoryPath, "utf8")).findings;
+    }
+    catch (err) {
+        throw new Error(`could not read ${memoryPath} to re-test its findings (${err instanceof Error ? err.message : String(err)}). Pass --retest off to check without it`);
+    }
+    if (findings !== undefined && !Array.isArray(findings))
+        throw new Error(`${memoryPath}: "findings" is not a list. Pass --retest off to check without it`);
+    return { flows, skippedFlows, findings: wellFormedFindings(findings ?? []) };
+}
+export async function runCheck(options, log = () => { }, inputs = { flows: [], findings: null }) {
     // A throwaway memory: a check is one run, and a store shared with earlier
     // exploratory runs would count their visits as this check's and skip those routes.
     const scratch = fs.mkdtempSync(path.join(os.tmpdir(), "scenescout-check-"));
@@ -52,17 +83,60 @@ export async function runCheck(options, log = () => { }) {
                 log(`  ${routes.length} route(s) checked`);
             }
         }
+        // Pages of open findings the crawl did not load exactly: loaded now, so each re-test has its own measurement.
+        // Kept apart from `routes`: they are measured only to re-test, never checked against the page rules, and do not count
+        // towards --max-routes. With --paths the check stays on the paths it was given.
+        const plan = inputs.findings ? checkRetestPlan(inputs.findings) : null;
+        const retestPages = [];
+        if (plan && !options.paths) {
+            const missing = [...new Set(plan.candidates.map((c) => c.path))].filter((p) => !routes.some((r) => r.path === p));
+            if (missing.length > 0) {
+                await engine.crawl(missing, { limit: missing.length, measureOnly: true });
+                retestPages.push(...engine.lastCrawlHealth);
+                log(`  ${missing.length} page(s) of open findings loaded only to re-test them`);
+            }
+        }
+        const flowRuns = [];
+        for (const flow of inputs.flows) {
+            // --flow-writes never: observe's rule, whatever --mode lets the crawl do.
+            const replay = await engine.replayFlow(flow.steps, options.flowWrites === "never" ? "observe" : options.mode);
+            flowRuns.push({ name: flow.name, file: flow.file, steps: flow.steps.length, ...replay });
+            log(`  flow ${flow.name}: ${replay.outcome.status}${replay.outcome.status === "passed" ? "" : ` at step ${replay.outcome.step}`}`);
+            // --on-refused-step stop: nothing after a refused step runs, and the check ends without a verdict.
+            if (replay.outcome.status === "refused" && options.onRefusedStep === "stop")
+                break;
+        }
+        const retest = plan
+            ? {
+                open: plan.open,
+                extraPages: retestPages.length,
+                results: retestResults(plan.candidates, [...routes, ...retestPages].map((r) => ({
+                    path: r.path,
+                    status: r.status,
+                    ...(r.loadError !== undefined ? { loadError: r.loadError } : {}),
+                    loginRedirect: r.loginRedirect,
+                    httpErrors: withoutOwnResponse(r)
+                        .violations.filter((v) => v.kind === "http_error")
+                        .map((v) => v.detail),
+                }))),
+            }
+            : null;
         const measured = redactRoutes(routes.map(withoutOwnResponse));
+        const flows = redactFlowRuns(flowRuns);
         return {
             url: redactRoute(options.url),
             generatedAt: new Date().toISOString(),
             mode: options.mode,
             failOn: options.failOn,
             routes: measured,
-            issues: issuesFromRoutes(measured, start.origin, options.ignore),
+            issues: issuesFromRoutes(measured, start.origin, options.ignore, flows),
             // Routes that failed to load are issues already; "not visited" is only what --max-routes left out.
             unvisited: options.paths ? [] : engine.crawlableRoutes().map(redactRoute),
             ignored: options.ignore,
+            flows,
+            skippedFlows: inputs.skippedFlows ?? [],
+            retest,
+            settings: settingsOf(options),
         };
     }
     finally {

package/dist/cli.js

@@ -17,8 +17,8 @@ import { fileURLToPath } from "node:url";
 import { APPROX_DISK_MB, BROWSER_ENGINES, browserPresence, defaultAttachNote, defaultEngine, launchTarget, parseBrowserSelection, playwrightInstallArgs, } from "./browsers.js";
 import { CLIENT_LABELS, firstMessageHint, manualFor, parseClients, registerWithClient, vscodeBinary } from "./clients.js";
 import { CLI_NAME, diagnose, ensureCommand, findOnUserPath, installSkill, isEphemeralRoot, launchCommand, manualRegisterCommand, planCommand, registerMcp, resolveClaudeDir, spawnRunner, } from "./installer.js";
-import { defaultCheckDir, runCheck } from "./check-run.js";
-import { EXIT, formatCheck, parseCheckArgs, summarise, toSarif, toSummaryJson, unmeasuredReason } from "./engine/check.js";
+import { defaultCheckDir, readCheckInputs, runCheck } from "./check-run.js";
+import { EXIT, exitCodeOf, formatCheck, parseCheckArgs, refusedFlowReason, toSarif, toSummaryJson, unmeasuredReason } from "./engine/check.js";
 import { LEGACY_MEMORY_DIRNAME, MEMORY_DIRNAME, writeSelfIgnore } from "./engine/memory.js";
 import { formatStatus, liveEngines, liveTokenFileName, localClock, LIVE_TOKEN_FILE, pidAlive, watchTarget, wholeSessions, } from "./engine/live.js";
 import { formatScan, scanProject } from "./scan.js";
@@ -53,7 +53,16 @@ Usage:
                                      --max-routes N (default 50); --paths /a,/b to check only those;
                                      --ignore rule,rule; --storage-state file to check signed in;
                                      --project dir (default: here); --out dir (default: .scenescout/check);
-                                     --browser chromium|firefox|webkit)
+                                     --browser chromium|firefox|webkit;
+                                     --flows dir|off: replay the flows saved there (default: .scenescout/flows);
+                                     --retest on|off: re-test open findings a page load reproduces (default on);
+                                     --flow-writes never|allow: never (default) replays flows under observe's
+                                      rule whatever --mode says; allow replays them under --mode, so their
+                                      form submissions are sent;
+                                     --on-refused-step report|stop: report (default) marks a flow whose step was
+                                      refused "could not run", keeps every other verdict and exits 2; stop exits 2 there;
+                                     --gate-retests never|high|all: which still-reproducing findings fail the gate
+                                      (default high: those filed high))
                                     Exit code: 0 passed, 1 failed the gate, 2 could not run.
   scenescout status [projectPath]   What is the engine doing right now? (every session + recent actions)
   scenescout watch [projectPath]    Open the live view in a browser: what each session is doing, a thumbnail
@@ -471,10 +480,18 @@ async function check(args) {
     }
     const options = parsed.options;
     const outDir = options.outDir ?? defaultCheckDir(options.projectDir);
+    let inputs;
+    try {
+        inputs = readCheckInputs(options);
+    }
+    catch (err) {
+        console.error(`scenescout check: ${err instanceof Error ? err.message : String(err)}`);
+        process.exit(EXIT.error);
+    }
     let result;
     try {
-        console.log(`Checking ${options.url} …`);
-        result = await runCheck(options, (line) => console.log(line));
+        console.log(`Checking ${options.url} …${inputs.flows.length > 0 ? ` (and ${inputs.flows.length} saved flow(s))` : ""}`);
+        result = await runCheck(options, (line) => console.log(line), inputs);
     }
     catch (err) {
         console.error(`scenescout check: could not run: ${err instanceof Error ? err.message : String(err)}`);
@@ -485,6 +502,12 @@ async function check(args) {
         console.error(`scenescout check: could not measure ${options.url}: ${unmeasured}`);
         process.exit(EXIT.error);
     }
+    const refused = refusedFlowReason(result);
+    // --on-refused-step stop: no verdict is written at all.
+    if (refused && options.onRefusedStep === "stop") {
+        console.error(`scenescout check: could not run a saved flow: ${refused}`);
+        process.exit(EXIT.error);
+    }
     const markdown = formatCheck(result);
     try {
         const version = packageVersion();
@@ -505,7 +528,10 @@ async function check(args) {
     }
     console.log("\n" + markdown);
     console.log(`Wrote report.md, check.sarif and check.json to ${outDir}`);
-    process.exit(summarise(result).passed ? EXIT.pass : EXIT.gateFailed);
+    // --on-refused-step report: everything else has its verdict in the files, and the exit code still says the run was incomplete.
+    if (refused)
+        console.error(`scenescout check: could not run a saved flow: ${refused}`);
+    process.exit(exitCodeOf(result));
 }
 const [, , command, ...args] = process.argv;
 // A CLI's failure mode should be a sentence, not a stack trace. `scan` on a