scenescout 3.20.1 → 3.21.0

package/CHANGELOG.md

@@ -1,5 +1,18 @@
 # scenescout
 
+## 3.21.0
+
+### Minor Changes
+
+- 64a719b: `scenescout ci` and its GitHub Action now default to two lanes sharing 80 model turns and 3,000,000 tokens (from one loop, 40 turns and 1,500,000 tokens). On the benchmark's demo app the new defaults found 5 to 7 of 13 planted defects in three runs, against 2 to 4 for the old ones, and 4 to 5 of 10 on the held-out app, at about $0.03 a run on `gpt-6-luna` instead of about $0.011. Without `--lanes`, a run with `--show` or with `--max-turns 1` runs as one loop. `--lanes 1 --max-turns 40 --max-tokens 1500000` restores the old behaviour.
+- a4531a8: `/scenescout qa check [focus]` runs a project's own recorded check from a pull-request comment, for projects with no preview deployments. Set the repository variable `SCENESCOUT_QA_CHECK_WORKFLOW` to a workflow that runs `scenescout check --record --video` (`examples/workflows/scenescout-qa-check.yml` is one): the comment workflow dispatches it on the pull request's branch, waits for it, and replies with the verdict, each journey's result, the findings by severity, the first failing step and links to the run and its artifact. No model key is involved, and pull requests from forks are refused. `/scenescout qa`, `show` and `compare` are unchanged.
+
+## 3.20.2
+
+### Patch Changes
+
+- f8f7ad2: A recorded `scenescout check` (`--record` or `--video`) no longer overwrites a `replay.html` in the output folder that it did not write: it stops before it starts, with exit code 2 and a message naming the file. With `--video`, a video that fails to start no longer stops every later flow from being filmed.
+
 ## 3.20.1
 
 ### Patch Changes

package/README.md

@@ -479,14 +479,14 @@ npx scenescout ci http://127.0.0.1:3000
 
 - **It reports and never gates.** Exit 0 when the run ran, whatever it found; exit 2 when it could not run (no key, a key the API refused, an app that never answered). Two runs of the same app find different things, so a finding is something to read, never a reason to fail a build. `scenescout check` is the gate.
 - **Providers:** the Anthropic Messages API (default model `claude-sonnet-5`) or the OpenAI Responses API (default `gpt-6-luna`), chosen by which key is set; with both set, `--provider` decides. `--model` and `--effort` (default `low`) override; `--base-url` points at another endpoint that implements the same API.
-- **Caps:** at most 40 model turns, 1,500,000 tokens and 20 minutes (`--max-turns`, `--max-tokens`, `--max-minutes`). The first cap reached ends the exploration; the report is still written, and says which cap ended it.
+- **Caps:** at most 80 model turns, 3,000,000 tokens and 20 minutes (`--max-turns`, `--max-tokens`, `--max-minutes`), shared by two model loops that each explore their own part of the app (`--lanes`, default 2). The first cap reached ends the exploration; the report is still written, and says which cap ended it. On the benchmark's demo app a run at these defaults cost about $0.03 on `gpt-6-luna` ([docs/benchmark.md](docs/benchmark.md#choosing-the-defaults-issue-419)).
 - **Mode:** `read-only` by default; `--mode observe` sends no form at all, `--mode safe-write` lets the run create records and change only the ones it created. `--mode destructive` runs only with `--allow-destructive` as well.
 - **Duplicates:** when the dedup rule keeps a filed finding apart, the run's model is asked at its lowest effort whether it is one already open on the same page, and merges it on a "same", keeping the filing's title, category, severity and evidence under that finding. The two findings' titles, categories and evidence, and the page's path, are sent; `--dedup rule` turns it off ([ADR 17](docs/adr/0017-a-model-judges-only-the-merges-the-rule-misses.md)).
 - **Output**, in `.scenescout/ci/` (or `--out`): `report.md` and `report.html` (the report an agent's run writes), `summary.md` (also appended to the GitHub job summary), `ci.json` and `ci.sarif`, with a usage line: turns, tokens, time and an estimated cost where the model's price is known (`--price-in`, `--price-out` give one for any model).
 
 There is a GitHub Action for it (`uses: brunoboto96/SceneScout/ci@…`). [docs/ci.md](docs/ci.md#an-unattended-exploratory-run) has the workflow and every option; [ADR 14](docs/adr/0014-an-unattended-run-reports-and-never-gates.md) says why it works this way.
 
-On a pull request, an allowed account can comment `/scenescout qa` to run it against that pull request's deployed preview and get the results as a reply. The job that holds the key checks out nothing and runs SceneScout from an exact release tag, so the pull request's code never runs beside the key. [docs/ci.md](docs/ci.md#a-qa-review-from-a-pull-request-comment) has the workflow to copy and what a project configures; [ADR 15](docs/adr/0015-a-qa-comment-tests-a-preview-and-never-runs-the-pull-requests-code.md) says why.
+On a pull request, an allowed account can comment `/scenescout qa` to run it against that pull request's deployed preview and get the results as a reply. The job that holds the key checks out nothing and runs SceneScout from an exact release tag, so the pull request's code never runs beside the key. A project without previews can comment `/scenescout qa check` instead: it dispatches the project's own recorded `scenescout check` workflow on the pull request's branch and replies with the verdict, per journey, with no model key involved. [docs/ci.md](docs/ci.md#a-qa-review-from-a-pull-request-comment) has the workflows to copy and what a project configures; [ADR 15](docs/adr/0015-a-qa-comment-tests-a-preview-and-never-runs-the-pull-requests-code.md) says why.
 
 ## 📮 Filing findings as issues
 

package/dist/check-run.js

@@ -512,6 +512,27 @@ export function clearReplayOutput(outDir) {
     if (fs.readdirSync(root).length === 0)
         fs.rmdirSync(root);
 }
+/**
+ * Why a recorded check (--record or --video) must not start, or null: a
+ * replay.html in the output folder that a check did not write is the
+ * project's own, and is neither removed nor overwritten. Asked before the
+ * browser starts, so the run is not spent first.
+ */
+export function replayPageConflict(outDir) {
+    const page = path.join(outDir, REPLAY_FILE);
+    let text;
+    try {
+        text = fs.readFileSync(page, "utf8");
+    }
+    catch (err) {
+        if (err.code === "ENOENT")
+            return null;
+        throw err;
+    }
+    if (isGeneratedReplay(text))
+        return null;
+    return `${page} is not a page SceneScout wrote, so a recorded check will not overwrite it. Move or rename it, or pass --out to write the check somewhere else`;
+}
 /**
  * The replay page's model for a recorded check, with its frames copied from
  * the browsers' scratch folders to replay-frames/ beside the report: routes

package/dist/cli.js

@@ -21,7 +21,7 @@ import { APPROX_DISK_MB, defaultAttachNote, defaultEngine, launchTarget, parseBr
 import { CLIENT_LABELS, firstMessageHint, manualFor, parseClients, registerWithClient, vscodeBinary } from "./clients.js";
 import { CLAUDE_CODE_NOT_NEEDED, CLI_NAME, desktopExtensionRoots, diagnose, doctorAllGood, findDesktopExtension, installClosing, ensureCommand, findOnUserPath, installSkill, isEphemeralRoot, launchCommand, manualRegisterCommand, planCommand, registerMcp, resolveClaudeDir, spawnRunner, } from "./installer.js";
 import { downloadBrowsers, presentBrowsers } from "./installer.js";
-import { baselinesDirOf, clearReplayOutput, defaultCheckDir, readCheckInputs, runCheck } from "./check-run.js";
+import { baselinesDirOf, clearReplayOutput, replayPageConflict, defaultCheckDir, readCheckInputs, runCheck } from "./check-run.js";
 import { buildCheckReplayHtml, commitOf, REPLAY_FILE, replayFrames, replayVideos } from "./engine/check-replay.js";
 import { recordChoice } from "./engine/capture.js";
 import { httpClient, httpJudgeAsk, runCi } from "./ci-run.js";
@@ -122,10 +122,10 @@ Usage:
                                      --model id (default claude-sonnet-5 / gpt-6-luna); --effort none|low|medium|
                                       high|xhigh|max (default low; none is OpenAI only); --base-url https://…/v1 for
                                       another endpoint that implements the same API;
-                                     --max-turns N (default 40); --max-tokens N (default 1500000);
+                                     --max-turns N (default 80); --max-tokens N (default 3000000);
                                      --max-minutes N (default 20): the run stops at the first cap reached and still
                                       writes the report;
-                                     --lanes N (default 1, at most 8): split the app between N model loops that
+                                     --lanes N (default 2, 1 with --show; at most 8): split the app between N model loops that
                                       explore at once, each in its own browser, sharing those caps;
                                      --price-in, --price-cached-in, --price-out: US dollars per million tokens,
                                       over the built-in prices, for the cost estimate of any model;
@@ -583,6 +583,10 @@ async function check(args) {
     try {
         // --record, else SCENESCOUT_RECORD, else off.
         options = { ...options, record: recordChoice(options.record, process.env) };
+        // A replay.html the project keeps there is its own: a recorded check refuses to start rather than overwrite it.
+        const conflict = options.record || options.video ? replayPageConflict(outDir) : null;
+        if (conflict)
+            throw new Error(conflict);
         // A replay page an earlier run left must never be read, or uploaded, as this run's.
         clearReplayOutput(outDir);
         inputs = readCheckInputs(options);

package/dist/engine/browser.js

@@ -16,6 +16,7 @@ import { extractCreatedIds, isOwnedResource, normalizeId } from "./ownership.js"
 import { formatJourney, journeyTime, measureJourney } from "./journey.js";
 import { describeStep, elementStateMatches, FLOW_AFTER_LAST_STEP_MS, isAction, matchRequest, notActionable, parseTarget, repeatFailure, splitRefusals, TARGET_HELP, urlMatches, } from "./flow.js";
 import { framePath, RECORD_MAX_FRAMES } from "./replay.js";
+import { film } from "./film.js";
 import { InFlightRequests, keepWatchingUrl, normalizePace, SETTLE_TICK_MS, shouldKeepWaiting } from "./settle.js";
 import { crawledRoute, crawlLine, isFileMediaType, isNonPageResource, mainStateFlag, mediaTypeOf } from "./crawl.js";
 import { authToRemember, BODY_FETCH_MAX, buildRequestScript, formatPageRequests, formatReplay, PageRequests, replaySignature, requestHeaders, resolveMethod, resolveRequestUrl, resolveTarget, staleCredentialNote, toReplayResult, wantsView, } from "./request.js";
@@ -1689,30 +1690,11 @@ export class BrowserEngine {
      */
     async filming(videoTo, work) {
         const page = this.requirePage();
-        const why = (err) => (err instanceof Error ? err.message.split("\n")[0] : String(err));
-        let videoError;
-        let filming = false;
-        try {
-            await fs.promises.mkdir(path.dirname(videoTo), { recursive: true });
-            await page.screencast.start({ path: videoTo, ...(page.viewportSize() ? { size: page.viewportSize() } : {}) });
-            filming = true;
-        }
-        catch (err) {
-            videoError = `the video could not be started: ${why(err)}`;
-        }
-        let value;
-        try {
-            value = await work();
-        }
-        finally {
-            // The page the filming began on, even if the work moved the session to a popup.
-            if (filming)
-                await page.screencast.stop().catch((err) => (videoError = `the video could not be saved: ${why(err)}`));
-        }
-        if (videoError)
-            return { value, video: null, videoError };
-        const saved = await fs.promises.stat(videoTo).then((st) => st.size > 0, () => false);
-        return saved ? { value, video: videoTo } : { value, video: null, videoError: "no video was written" };
+        // The page the filming began on is the one stopped, even if the work moved the session to a popup.
+        return film(page.screencast, videoTo, page.viewportSize(), work, {
+            prepare: (file) => fs.promises.mkdir(path.dirname(file), { recursive: true }).then(() => undefined),
+            written: (file) => fs.promises.stat(file).then((st) => st.size > 0, () => false),
+        });
     }
     requirePage() {
         if (!this.page || !this.memory) {

package/dist/engine/ci.js

@@ -53,15 +53,19 @@ export const DEFAULT_CI_DEDUP = "judge";
 export const DEDUP_ENV = "SCENESCOUT_DEDUP";
 /** Which provider the MCP server's dedup judge uses when both keys are in its environment. */
 export const DEDUP_PROVIDER_ENV = "SCENESCOUT_DEDUP_PROVIDER";
-export const DEFAULT_CAPS = { turns: 40, tokens: 1_500_000, wallMs: 20 * 60_000 };
+/** Why these are the defaults: docs/benchmark.md, "Choosing the defaults (issue 419)". */
+export const DEFAULT_CAPS = { turns: 80, tokens: 3_000_000, wallMs: 20 * 60_000 };
 const CAP_BOUNDS = { turns: [1, 500], tokens: [1_000, 20_000_000], minutes: [1, 360] };
 /**
  * How many model loops explore at once, each in its own browser session and
  * its own part of the app (engine/ci-lanes.ts). 1 is the single loop. The most
  * is scout_lane_brief's, since the split is the same one. Why the default is
- * what it is: docs/benchmark.md, "Unattended runs".
+ * what it is: docs/benchmark.md, "Choosing the defaults (issue 419)". Without
+ * --lanes, a run that cannot take this many (--show, or fewer turns than
+ * lanes) runs as one loop rather than failing; --lanes given explicitly is
+ * held to those rules.
  */
-export const DEFAULT_LANES = 1;
+export const DEFAULT_LANES = 2;
 export const MAX_CI_LANES = MAX_LANES;
 /** Every option `scenescout ci` accepts; the ci action's inputs are these names (ci-test holds them equal). */
 export const CI_OPTION_NAMES = [
@@ -183,7 +187,9 @@ export function parseCiArgs(args, cwd) {
     const turns = whole("max-turns", CAP_BOUNDS.turns, DEFAULT_CAPS.turns);
     const tokens = whole("max-tokens", CAP_BOUNDS.tokens, DEFAULT_CAPS.tokens);
     const minutes = whole("max-minutes", CAP_BOUNDS.minutes, DEFAULT_CAPS.wallMs / 60_000);
-    const lanes = whole("lanes", [1, MAX_CI_LANES], DEFAULT_LANES);
+    // The default split yields to --show and to a turn cap below it; an explicit --lanes is checked against them below.
+    const defaultLanes = flags.has("show") || typeof turns !== "number" ? 1 : Math.min(DEFAULT_LANES, turns);
+    const lanes = whole("lanes", [1, MAX_CI_LANES], defaultLanes);
     for (const v of [turns, tokens, minutes, lanes])
         if (typeof v === "string")
             return { ok: false, error: v };

package/dist/engine/film.js

@@ -0,0 +1,41 @@
+/**
+ * Filming one piece of work into a video file (`scenescout check --video`):
+ * start, run the work, stop, and say whether a video was saved. The browser
+ * hands in its page's screencast; nothing here needs Playwright, so the
+ * failure paths are table-tested.
+ */
+const why = (err) => (err instanceof Error ? err.message.split("\n")[0] : String(err));
+/**
+ * Run `work` while `screencast` films it into `videoTo`. The work's own result
+ * and errors are never touched by the filming: a video that cannot be started
+ * or saved is reported in `videoError`, and the work runs regardless.
+ *
+ * A start that fails is followed by a stop, its error ignored: a screencast
+ * left half-started would refuse every later start ("already started"), so
+ * one failure would cost the video of every flow after it.
+ */
+export async function film(screencast, videoTo, size, work, io) {
+    let videoError;
+    let filming = false;
+    try {
+        await io.prepare(videoTo);
+        await screencast.start({ path: videoTo, ...(size ? { size } : {}) });
+        filming = true;
+    }
+    catch (err) {
+        videoError = `the video could not be started: ${why(err)}`;
+        // Whatever the failed start left running is stopped, so the next flow can start afresh; there is nothing to save.
+        await screencast.stop().catch(() => undefined);
+    }
+    let value;
+    try {
+        value = await work();
+    }
+    finally {
+        if (filming)
+            await screencast.stop().catch((err) => (videoError = `the video could not be saved: ${why(err)}`));
+    }
+    if (videoError)
+        return { value, video: null, videoError };
+    return (await io.written(videoTo)) ? { value, video: videoTo } : { value, video: null, videoError: "no video was written" };
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "scenescout",
-  "version": "3.20.1",
+  "version": "3.21.0",
   "description": "SceneScout — exploratory UI testing for AI coding agents. An MCP server that gives any agent (Claude Code, Cursor, VS Code Copilot, Codex, Gemini CLI and others) a structured view of a running web app, always-on oracles, a network-level write policy, memory across runs and a gap-checked report.",
   "license": "MIT",
   "author": "brunoboto96",