scenescout 3.18.0 → 3.19.1

package/CHANGELOG.md

@@ -1,5 +1,31 @@
 # scenescout
 
+## 3.19.1
+
+### Patch Changes
+
+- 3a302b4: A saved flow that fails after typing into a form with an unsaved-changes guard no longer stops the flows after it. During a flow, a page asking to confirm leaving is left, as the flow's `navigate` and its hand-back mean, so the next flow's first `navigate` is not cancelled (`net::ERR_ABORTED`). The session's own answer is restored once the flow hands back.
+
+## 3.19.0
+
+### Minor Changes
+
+- fb7a3e5: An optional Claude Code mod, `scenescout-mod`, is listed in the plugin marketplace beside `scenescout`. In the Claude Code CLI and the desktop Code tab, `/scenescout-pane` opens a pane that refreshes every 2.5 seconds from the server's `scout_status_poll` tool: each session with its task and objective, open findings by severity, coverage, and a link to the live view. Its `lane_model` setting, empty by default, sets the model SceneScout lane agents start on. The `scenescout` plugin, skill and server are unchanged and work the same without it.
+- 1a2db2f: `scenescout export --from <file>` exports a `check.json` from `scenescout check` or a `ci.json` from `scenescout ci`, so the unattended runs can file GitHub or Jira issues without an interactive session's memory. Their issues get the same body, label and marker as memory findings: a check issue is keyed on its fingerprint and a ci finding keeps its memory id, so a re-export files nothing twice. The project's memory stays the default source.
+- 498dbdf: A saved flow can take a `type` or `select` value from the environment with `${env:NAME}`, so a one-time code or a password stays in the CI's secret store rather than in the flow file. A variable that is not set stops the check before it starts, naming the flow and the variable, and every substituted value of four characters or more is masked as `[$NAME]` in everything the check writes and prints.
+- ea8fbfa: Saved flows can assert an element's state with a new step, `expect-element`: a target and a state of `visible`, `hidden`, `enabled`, `disabled`, `checked` or `unchecked`. `hidden` also holds when nothing matches, so a flow can say a dialog closed or a receipt was never shown, and a failure says what is true instead (`testid=receipt is visible, expected hidden`).
+- c07941d: Saved flows have a `repeat` step: run a few click, type, select or press steps until an `expect-text`, `expect-element` or `expect-url` step holds, at most `max` times. It is for gates a fixed script cannot walk, such as paging through a document until its Continue button is enabled. The condition is checked first, so a page already in that state runs nothing, and a failure names what never held and how many rounds were tried.
+- 54257fc: A saved flow can name the `role` it runs as. `scenescout check` replays it in a browser of its own, signed in with the profile `scenescout login --role <name>` saved in the project, while flows naming no role keep the check's own session. Flows run in file-name order, so a journey that passes between people (one submits, another approves) is a sequence of flows. A role with no saved profile stops the check before it starts, with exit 2 and the command that saves one.
+- d439d28: Saved flows have an `upload` step: attach a small valid file generated on the spot (`pdf`, `png`, `txt`, `csv` or `json`, or the kind the input's `accept` asks for) to a file input, the control that opens its chooser, or the page's only file input. A journey that starts from an attached file can now be saved and replayed by `scenescout check`. Submitting the upload is a write, so it reaches the server only under `--flow-writes allow`.
+
+### Patch Changes
+
+- 36c6e11: `scenescout check` no longer passes when the only page it measured was a start page showing at most one control and linking nowhere, which is what an app measured before it finished drawing looks like. Without `--paths`, that check now exits 2 saying only the start page was measured, rather than going green having checked nothing.
+- 87fa5bf: The contrast rule no longer reports text whose own colour is fully transparent, such as the selectable text layer a PDF viewer lays over the rendered page. That text is not painted, so its 1.00:1 ratio is not what anyone sees, and how many of those spans were measured depended on whether the viewer had finished loading. Translucent text that is painted, a faint watermark say, is still measured.
+- 8857bab: `scenescout export` no longer files a finding twice when an export runs straight after another, or after a create whose result was uncertain, while the tracker's issue listing has not caught up. Each filed issue is recorded in `.scenescout/exported.json` and read back by number on the next export; GitHub's newest issues are also read without the label filter; and in Jira a finding whose create may have been made is held back for 15 minutes unless its issue is found.
+- 82f775c: A saved flow step on a control that is shown but stays disabled (or, for `type`, read-only) until the action limit runs out now fails saying so first, `testid=save is visible but disabled after 5s`, followed by the limit hint. It used to report only an action timeout, which never said the control was there but disabled.
+- 830b663: `scenescout check --baseline` now takes each picture again, a frame later, until two in a row are the same, so a baseline is no longer of a moment the page was still drawing: the last frame of an animation it had just stopped, or a script still filling something in after the load. A page that never holds still within the action limit keeps its last picture, and the report says so on that target's line.
+
 ## 3.18.0
 
 ### Minor Changes

package/README.md

@@ -154,6 +154,8 @@ Either way it downloads the browser and registers the server with the client you
 
 Then start a new chat to use SceneScout; the test browser downloads on first use (to have it ready beforehand: `npx -y scenescout install --browser-only`). The command becomes `/scenescout:scenescout`. A plugin's skill comes from this repository and its server from the latest npm release, so right after a release lands here the two can differ for a short while; `/plugin marketplace update scenescout-marketplace` brings the skill up to date.
 
+An optional second plugin, `scenescout-mod@scenescout-marketplace`, adds a run pane (`/scenescout-pane`) and a setting for the model lane agents run on, in the Claude Code CLI and the desktop Code tab. It is a mod: unsandboxed JavaScript that runs inside Claude Code, so it is opt-in. [The Claude Code mod](docs/guide/Ways-to-use-it.md#the-claude-code-mod).
+
 **Using Claude Desktop?** Install the extension: download `scenescout-X.Y.Z.mcpb` from the [latest release](https://github.com/brunoboto96/SceneScout/releases/latest) and open it (or Settings > Extensions > Advanced settings > Install Extension). It works as soon as it is installed, with no terminal step: the test browser downloads on first use. Start a new chat and ask *"Use SceneScout to test http://localhost:3000"*. [More in the guide](docs/guide/Start-here.md#as-a-claude-desktop-extension).
 
 **A client that is not in that list?** [Add the server to its config by hand](#-other-mcp-clients); the test browser downloads on first use.
@@ -457,7 +459,7 @@ The defaults are what an unconfigured check does, for a first try or an AI agent
 
 `scenescout check --help` lists every option. Why the defaults are what they are: [ADR 11](docs/adr/0011-a-gate-is-deterministic-and-fails-only-on-what-it-can-prove.md).
 
-It also replays the flows saved in `.scenescout/flows/*.json`, with no model: the steps `scout_run_plan` takes (navigate, click, type, select, press) plus `expect-text`, `expect-url` and `expect-request`. A flow whose step breaks fails the gate, naming the flow and the step. And it re-tests the open findings earlier runs left in the project's memory that a page load can reproduce, reporting each as still reproducing or possibly fixed; by default a finding filed high that still reproduces fails the gate. [docs/ci.md](docs/ci.md#saved-flows) has the flow format; [ADR 12](docs/adr/0012-a-check-replays-saved-flows-and-reports-re-tests.md) says why it works this way.
+It also replays the flows saved in `.scenescout/flows/*.json`, with no model: the steps `scout_run_plan` takes (navigate, click, type, select, press, and upload, which attaches a generated file) plus `expect-text`, `expect-element` (a control is visible, hidden, enabled, disabled, checked or unchecked), `expect-url` and `expect-request`, and `repeat` to run actions until a condition holds (page through a document until Continue is enabled). A flow can name the `role` it runs as, signed in with a profile `scenescout login --role` saved, so a journey that passes between people (one submits, another approves) is a sequence of flows. Values can come from the environment, `${env:NAME}`, so a code or a password stays in the CI's secret store and is masked in everything the check writes. A flow whose step breaks fails the gate, naming the flow and the step. And it re-tests the open findings earlier runs left in the project's memory that a page load can reproduce, reporting each as still reproducing or possibly fixed; by default a finding filed high that still reproduces fails the gate. [docs/ci.md](docs/ci.md#saved-flows) has the flow format; [ADR 12](docs/adr/0012-a-check-replays-saved-flows-and-reports-re-tests.md) says why it works this way.
 
 With `--baseline compare` it also holds pages and elements to approved pictures: list them in a `targets.json`, take the baselines once with `--baseline update`, and a later check that finds one changed past `--baseline-threshold` fails the gate with the share of pixels changed and a diff picture beside the report. Baselines are kept per browser, in a git-ignored folder unless `--baselines` names one the project commits, and only `--baseline update` ever writes one. [The guide](docs/guide/Ways-to-use-it.md#visual-baselines) has the details; [ADR 19](docs/adr/0019-a-visual-baseline-changes-only-when-asked.md) says why.
 

package/dist/check-run.js

@@ -12,7 +12,8 @@ import path from "node:path";
 import { BASELINE_CAPTURE, baselineFiles, baselineMeta, defaultBaselinesDir, elementTarget, isVisualPicture, judgeBaseline, missingTargetsMessage, parseBaselineTargets, readStoredBaseline, TARGETS_FILE, VISUAL_DIRNAME, visualFiles, } from "./engine/baseline.js";
 import { BrowserEngine } from "./engine/browser.js";
 import { checkFindings, redactBaselineRun, MAX_DISCOVERY_ROUNDS, redactFlowRuns, redactRoute, redactRoutes, settingsOf, splitResources, unmeasuredReason, withoutOwnResponse, } from "./engine/check.js";
-import { loadFlows, resolveFlowsDir } from "./engine/flow.js";
+import { flowRoles, loadFlows, maskEnvValues, resolveFlowEnv, resolveFlowsDir } from "./engine/flow.js";
+import { resolveAttachAuth } from "./engine/profiles.js";
 import { MemoryStore, MEMORY_DIRNAME, writeSelfIgnore } from "./engine/memory.js";
 import { decodePng, encodePng } from "./engine/png.js";
 import { firstLineOf } from "./engine/limits.js";
@@ -81,7 +82,23 @@ export async function runCheck(options, log = () => { }, inputs = { flows: [], f
     // Also on an exit the finally below never reaches, such as Ctrl+C, which the browser's driver answers with process.exit.
     const removeScratch = () => fs.rmSync(scratch, { recursive: true, force: true });
     process.once("exit", removeScratch);
+    // A flow that names a role whose sign-in is not saved would only fail after the crawl: refuse before anything starts,
+    // naming the command that saves it.
+    for (const role of flowRoles(inputs.flows))
+        resolveAttachAuth({ projectDir: options.projectDir, url: options.url, role });
+    // ${env:NAME} in a flow's values, from the environment; an unset name stops the check before anything starts.
+    const envValues = new Map();
+    const runFlows = inputs.flows.map((flow) => {
+        const resolved = resolveFlowEnv(flow, process.env);
+        if (!resolved.ok)
+            throw new Error(`flow "${flow.name}" (${flow.file}) takes ${resolved.missing.map((n) => `\${env:${n}}`).join(", ")} from the environment, which is not set`);
+        for (const [name, value] of resolved.values)
+            envValues.set(name, value);
+        return resolved.flow;
+    });
     const engine = new BrowserEngine();
+    /** One browser per role a flow runs as, each attached once, on first use, and closed with the check's own. */
+    const roleEngines = new Map();
     const start = new URL(options.url);
     // From the start of the run, browser launch included: the budget is wall-clock time a person waits.
     const deadline = options.timeBudgetMs !== undefined ? Date.now() + options.timeBudgetMs : undefined;
@@ -147,15 +164,45 @@ export async function runCheck(options, log = () => { }, inputs = { flows: [], f
         // reached only sign-in pages: the check has no verdict then, and an update would write sign-in pages as baselines.
         if (options.baseline !== "off" && !inputs.baselineTargets)
             throw new Error(`--baseline ${options.baseline} was given no targets: read them with readCheckInputs`);
-        const baselines = options.baseline !== "off" && inputs.baselineTargets && unmeasuredReason(routes, !options.paths) === null
+        const baselines = options.baseline !== "off" && inputs.baselineTargets && unmeasuredReason(routes, !options.paths, options.paths ? [] : engine.crawlableRoutes()) === null
             ? await takeBaselines(engine, options, options.baseline, inputs.baselineTargets, log)
             : null;
         const flowRuns = [];
-        for (const flow of inputs.flows) {
+        // A flow that names a role runs in that role's own browser, so the check's session (and what its crawl found) is
+        // left as it was.
+        const engineFor = async (role) => {
+            if (role === undefined)
+                return engine;
+            const known = roleEngines.get(role);
+            if (known)
+                return known;
+            const roleEngine = new BrowserEngine();
+            roleEngines.set(role, roleEngine);
+            const roleDir = path.join(scratch, `role-${role}`);
+            fs.mkdirSync(roleDir, { recursive: true });
+            const attachedAs = await roleEngine.attach({
+                url: start.origin,
+                projectDir: options.projectDir,
+                memoryStore: new MemoryStore(roleDir),
+                mode: options.mode,
+                role,
+                ...(options.browser ? { browser: options.browser } : {}),
+                actionTimeoutMs: options.actionTimeoutMs,
+                navTimeoutMs: options.navTimeoutMs,
+                objective: `Deterministic check: replay the saved flows that run as ${role}`,
+                task: `Replaying flows as ${role}`,
+            });
+            const lost = attachedAs.split("\n").find((line) => line.startsWith("⚠ AUTH FAILED"));
+            if (lost)
+                throw new Error(`the saved sign-in for role "${role}" no longer signs in: ${lost.replace(/ Continuing now tests a logged-out app\.$/, "")}`);
+            return roleEngine;
+        };
+        for (const flow of runFlows) {
+            const runner = await engineFor(flow.role);
             // --flow-writes never: observe's rule, whatever --mode lets the crawl do.
-            const replay = await engine.replayFlow(flow.steps, options.flowWrites === "never" ? "observe" : options.mode);
-            flowRuns.push({ name: flow.name, file: flow.file, steps: flow.steps.length, ...replay });
-            log(`  flow ${flow.name}: ${replay.outcome.status}${replay.outcome.status === "passed" ? "" : ` at step ${replay.outcome.step}`}`);
+            const replay = await runner.replayFlow(flow.steps, options.flowWrites === "never" ? "observe" : options.mode);
+            flowRuns.push({ name: flow.name, file: flow.file, steps: flow.steps.length, ...(flow.role === undefined ? {} : { role: flow.role }), ...replay });
+            log(`  flow ${flow.name}${flow.role === undefined ? "" : ` (as ${flow.role})`}: ${replay.outcome.status}${replay.outcome.status === "passed" ? "" : ` at step ${replay.outcome.step}`}`);
             // --on-refused-step stop: nothing after a refused step runs, and the check ends without a verdict.
             if (replay.outcome.status === "refused" && options.onRefusedStep === "stop")
                 break;
@@ -181,7 +228,7 @@ export async function runCheck(options, log = () => { }, inputs = { flows: [], f
         // Signed in only when given a session: with none, a route that sends the browser to sign-in needs one, and is not a lost session.
         const signedIn = options.storageStatePath !== undefined;
         const { issues, worthALook, needsSignIn } = checkFindings(measured, start.origin, options.ignore, flows, pictured, options.ignorePaths, signedIn);
-        return {
+        const result = {
             url: redactRoute(options.url),
             generatedAt: new Date().toISOString(),
             mode: options.mode,
@@ -202,8 +249,12 @@ export async function runCheck(options, log = () => { }, inputs = { flows: [], f
             settings: settingsOf(options),
             baselines: pictured,
         };
+        // A value taken from the environment (a code, a password) is never written, wherever the page echoed it.
+        return envValues.size > 0 ? JSON.parse(maskEnvValues(JSON.stringify(result), envValues)) : result;
     }
     finally {
+        for (const [role, roleEngine] of roleEngines)
+            await roleEngine.close().catch((err) => log(`closing the browser for role ${role} failed: ${err instanceof Error ? err.message : String(err)}`));
         await engine.close().catch((err) => log(`closing the browser failed: ${err instanceof Error ? err.message : String(err)}`));
         process.off("exit", removeScratch);
         removeScratch();
@@ -315,7 +366,12 @@ async function takeBaselines(engine, options, mode, targets, log) {
                 stored,
                 now: { capture, platform: process.platform, image },
             });
-            const result = { ...own, ...verdict, ...(shot.cut ? { partial: shot.cut } : {}) };
+            const result = {
+                ...own,
+                ...verdict,
+                ...(shot.cut ? { partial: shot.cut } : {}),
+                ...(shot.unsteady ? { unsteady: shot.unsteady } : {}),
+            };
             if (verdict.status === "updated") {
                 fs.mkdirSync(path.dirname(at(dir, files.png)), { recursive: true });
                 fs.writeFileSync(at(dir, files.png), shot.png);

package/dist/cli.js

@@ -162,8 +162,9 @@ Usage:
                                     from SCENESCOUT_LOGIN_<FLAG>, e.g. SCENESCOUT_LOGIN_SUCCESS_URL;
                                     --timeout seconds (default 60))
   scenescout export --to github|jira
-                                    File the project's open findings (from .scenescout/memory.json) as issues,
-                                    each once: a finding whose marker is already on an issue is skipped. A dry run
+                                    File the project's open findings (from .scenescout/memory.json, or from a
+                                    check.json or ci.json given with --from file) as issues, each once: a finding
+                                    whose marker is already on an issue is skipped. A dry run
                                     that lists what it would file unless --yes is given. Credentials come from the
                                     environment only: GH_TOKEN or GITHUB_TOKEN; JIRA_EMAIL and JIRA_API_TOKEN.
                                     (--repo owner/name for GitHub (GITHUB_API_URL for GitHub Enterprise Server);
@@ -181,7 +182,8 @@ Usage:
                                       in Jira (default severity: high… / High, Medium, Low); --labels a,b;
                                      --screenshots on|off (default on: the finding's picture and the run's frames,
                                       attached in Jira, named on GitHub);
-                                     --project dir (default: here); --dry-run; --yes)
+                                     --project dir (default: here; its .scenescout folder keeps the record of
+                                      filed issues); --from check.json|ci.json; --dry-run; --yes)
                                     Exit code: 0 done (findings over the cap wait for the next export), 2 could not
                                     export (it lists what it filed before it stopped), or a screenshot was not
                                     attached or a ticket not linked.
@@ -587,7 +589,7 @@ async function check(args) {
         console.error(`scenescout check: could not run: ${err instanceof Error ? err.message : String(err)}`);
         process.exit(EXIT.error);
     }
-    const unmeasured = unmeasuredReason(result.routes, !options.paths);
+    const unmeasured = unmeasuredReason(result.routes, !options.paths, result.unvisited);
     if (unmeasured) {
         console.error(`scenescout check: could not measure ${options.url}: ${unmeasured}`);
         process.exit(EXIT.error);

package/dist/engine/baseline.js

@@ -74,6 +74,49 @@ export const STOP_ANIMATIONS_SCRIPT = `(() => {
   }
   return true;
 })()`;
+/**
+ * Page-side source that resolves once the page has drawn a frame after this
+ * call (two animation frames: the first starts one, the second runs once it
+ * is done). A page whose frames are not running at all resolves after
+ * NEXT_FRAME_WAIT_MS instead; the browser side bounds the wait by the same
+ * time, for a page too busy to run either.
+ */
+export const NEXT_FRAME_WAIT_MS = 1000;
+export const NEXT_FRAME_SCRIPT = `new Promise((resolve) => {
+  requestAnimationFrame(() => requestAnimationFrame(() => resolve(true)));
+  setTimeout(() => resolve(false), ${NEXT_FRAME_WAIT_MS});
+})`;
+/**
+ * A picture of a page that has stopped changing: pictures are taken, a frame
+ * apart, until two in a row are the same, and that one is kept. Stopping CSS
+ * animations does not make a page still. A frame drawn before the stop took
+ * effect, or a script that draws for a moment after the load (a count-up, an
+ * entrance), each make the first picture depend on how fast the machine is,
+ * and a baseline taken on a fast run then fails on a slow one. No new
+ * picture is started once `budgetMs` has passed (the one in progress still
+ * finishes): a page that never holds still keeps its last picture, and
+ * `steady` is false so the result can say so. `take` and `nextFrame` are the
+ * browser's (BrowserEngine.captureForBaseline); the loop is table-tested.
+ */
+export async function steadyPicture(take, nextFrame, budgetMs, now = Date.now) {
+    const until = now() + budgetMs;
+    let last = await take();
+    let takes = 1;
+    for (;;) {
+        await nextFrame();
+        const png = await take();
+        takes += 1;
+        if (png.equals(last))
+            return { png, steady: true, takes };
+        last = png;
+        if (now() >= until)
+            return { png, steady: false, takes };
+    }
+}
+/** What a result says when its picture never held still (steadyPicture): a comparison of it may not repeat. */
+export function unsteadyNote(budgetMs) {
+    return `the picture was still changing after about ${budgetMs / 1000}s, so the last one taken is used: something on the page keeps moving, and a comparison of it may not repeat`;
+}
 const ELEMENT_HELP = `${PAGE_ELEMENT} or ${TARGET_HELP}`;
 /** What an element names: null for the page itself, else the target a saved flow would use. Undefined when it is neither. */
 export function elementTarget(element) {