scenescout 3.3.0 → 3.5.0

package/CHANGELOG.md

@@ -1,5 +1,32 @@
 # scenescout
 
+## 3.5.0
+
+### Minor Changes
+
+- 4fe8889: Parallel lanes now share what a run knows, and the planner is told what a lane left undone.
+  
+  - A markup value typed in one session is watched for in every session on the same project, so a stored injection is caught when another lane opens the list that renders it.
+  - The report gate counts design audits from every session in the run, so a planner whose lanes audited the pages is no longer refused.
+  - Folding a lane report lists each judged defect that no finding matches yet, so it can be filed before the lane's session closes.
+  - A lane report wrapped in prose around one fenced JSON block is accepted, with the prose discarded unread.
+  - `scout_lane_brief` gives each lane a landing route of its own and the rules a measured run found worth stating.
+  - The pace section shows how long each session held a browser idle before its first action and after its last, including sessions that attached and never acted.
+  - An empty live region (`role="status"` and similar) is no longer shown or counted as an unnamed control.
+- 6ec8fee: The write policy now answers a page's blocked `fetch` or XHR write with a `403` in the server's place instead of dropping it. The server is still never contacted, but the page's handling of a refusal really runs, so a page that reports a refused save or delete as a success is caught as a `false_success` (and says the refusal was the policy's stand-in). Blocked navigations are still dropped. The stand-in 403 is not reported as an HTTP error of the app.
+
+## 3.4.0
+
+### Minor Changes
+
+- 04f0f22: Measure whether a lane's confidence means anything, and make one lane rubric serve a whole wave.
+  
+  Every lane has been told its confidence must be calibrated — "0.5 means a coin flip, 0.95 means you would bet on it" — and the number was then averaged into one line and discarded. Nothing was stored, so nothing could ever be checked, and a confidence nobody checks is decoration.
+  
+  Lane decisions are now kept, and the report carries a calibration section: how often a decision at a stated confidence matched a finding the project holds, bucketed, with an expected calibration error. It says plainly what the number is not — agreement between the lanes and the bar the project applies, over its whole history, not evidence that the app is broken — and names the two ways a lane is counted wrong through no fault of its own. A decision naming no failing endpoint cannot be looked up at all, so it is excluded and disclosed rather than scored as a miss. Below eight checkable decisions the figure is withheld, but the section says so rather than vanishing. Where findings have since been re-tested through `scout_verify`, those verdicts are reported beside it, because they *are* evidence about the app.
+  
+  Separately, the lane name used to sit in the second sentence of the instruction every lane receives, so two lanes' prompts diverged almost immediately and shared no prefix. The rubric is now identical for every lane in a wave and the name is the last thing said, which makes it one cacheable prefix instead of one per lane.
+
 ## 3.3.0
 
 ### Minor Changes

package/README.md

@@ -205,7 +205,8 @@ The view is served on `127.0.0.1` only, behind a token that changes every time t
 
 **Try it with parallel agents.** The demo app has three roles and several separate areas, so a run can be split between agents. Start it with `npm run demo:serve`, then ask your agent to explore it with several agents in parallel, one role and one area each. The pictures above come from a run of three. Two things keep a parallel run efficient:
 
-- **Each agent opens its own session when it starts and closes it when it is done.** An agent waiting for its turn then holds no browser. Opening every session up front leaves browsers idling while the machine runs out of memory for the agents that are working.
+- **Each agent opens its own session when it starts, and the planner closes it once it has folded that agent's report.** An agent waiting for its turn then holds no browser. Opening every session up front leaves browsers idling while the machine runs out of memory for the agents that are working. Closing before the fold loses the lane's decisions, which have nowhere to be kept.
+- **Slow it down to follow along.** `scout_attach {paceMs}` (or `scout_session {paceMs}` mid-run) sets a floor between actions, for when you want to watch a flow rather than let it run as fast as the page allows.
 - **Run about as many agents at once as your machine has cores, less two.** Each one drives a real browser.
 
 ---
@@ -272,7 +273,7 @@ Snapshots are cheap: re-snapshotting a route returns only *what changed*, with s
 
 ## 🧰 The toolbox
 
-26 deterministic tools. The agent picks; you rarely call these by hand.
+29 deterministic tools. The agent picks; you rarely call these by hand.
 
 | Phase | Tools | What they do |
 |---|---|---|
@@ -283,6 +284,8 @@ Snapshots are cheap: re-snapshotting a route returns only *what changed*, with s
 | **Act** | `scout_click` `scout_type` `scout_select` `scout_upload` `scout_press` `scout_scroll` `scout_navigate` `scout_back` `scout_run_plan` | Drive the UI like a user; `scout_run_plan` batches a whole mechanical sequence into one call |
 | **Assess** | `scout_design_audit` `scout_journey` | Score a page's craft/a11y/consistency; measure how hard a task is to complete |
 | **Record** | `scout_note` `scout_finding` `scout_resolve` `scout_report` | Curate durable notes; file deduped findings; mark fixes; write the report, and on a recorded run the whole run as one page |
+| **Re-test** | `scout_verify` | List the findings earlier runs left open, worst route first, and record whether each is gone, still present, or changed |
+| **Split the work** | `scout_lane_brief` `scout_lane_report` | Divide the app between parallel agents by whole module, each with its own landing route and rules; fold what each hands back as one typed JSON object, and name any defect it judged but never filed |
 | **Close** | `scout_close` | Tear down one session or all |
 
 A few that punch above their weight:
@@ -292,20 +295,23 @@ A few that punch above their weight:
 - **`scout_journey`** — wraps one goal and reports interaction count, screens seen, and **backtracks**; an abandoned journey is a finding no passing E2E suite can produce.
 - **`scout_upload`** — generates a *valid* in-memory fixture (real PDF/PNG, kind inferred from `accept`) so file-upload flows stop being a blind spot.
 - **`scout_click {clicks: 2}`** — the impatient-user probe: states whether a double-click fired the same state-changing request twice (the classic double-submit bug).
+- **`scout_request`** — calls the app's own API as the session, so "the button is hidden" becomes "the server refuses it" (or doesn't).
+
+Beyond crashes and HTTP errors, two oracles catch a page **contradicting the server**: `refused_empty` (a list request was refused and the page shows its empty state with no error) and `false_success` (a save was refused and the page says it worked). A third, `dom_injection`, reports a typed markup value coming back as an element on any page any session opens.
 
 ---
 
 ## 📊 Test levels
 
-Each level is an **enforced contract** — `scout_report` checks it before finalizing.
+Each level is a contract. `scout_report` enforces what the engine can see for itself — routes visited, pages audited, and at `extensive` an empty gap ledger (every route exercised and audited, every filled form submitted, a completed journey, two roles) — and the report's gap ledger discloses the rest.
 
 | Level | What it guarantees | Rough size |
 |---|---|---|
 | `minimal` | Every route visited, ≥1 design audit, key journeys as plans, crawl problems triaged. Remaining gaps **disclosed**. | ~40 actions |
-| `medium` *(default)* | minimal + design audits across several routes + every element class exercised + every form submitted valid **and** invalid | ~150 actions |
-| `extensive` | medium + fuzzing, back/refresh/deep-link resilience, keyboard-only pass, a journey per module, ≥2 roles compared, anonymous auth-surface walk. **Refuses to finalize while any gap remains.** | budget-capped |
+| `medium` *(default)* | minimal + design audits across several routes (enforced) + every element class exercised + every form submitted valid **and** invalid (asked of the agent) | ~150 actions |
+| `extensive` | medium + fuzzing, back/refresh/deep-link resilience, keyboard-only pass, a journey per module, ≥2 roles compared, anonymous auth-surface walk. **Refuses to finalize while any gap in the ledger remains.** | budget-capped |
 
-That refusal *is* the guarantee: an extensive report can only exist when nothing known was left untested.
+That refusal *is* the guarantee: an extensive report can only exist when nothing the engine can measure was left untested. Fuzzing, the keyboard pass and the auth-surface walk are the agent's to do; the engine cannot see whether they were done well.
 
 ---
 
@@ -317,7 +323,7 @@ That refusal *is* the guarantee: an extensive report can only exist when nothing
 - 🔴 **`destructive`** (`--allow-destructive`) allows everything, and only ever when *you* confirm the environment is disposable. The skill will never choose this itself.
 - 📂 Findings, memory, and reports live in a `.scenescout/` folder where you ran it. It ignores itself in git, so a stray `git add -A` never commits test data.
 
-A `🛡 WRITE-POLICY blocked` notice is the safety net doing its job, not an app bug.
+A `🛡 WRITE-POLICY blocked` notice is the safety net doing its job, not an app bug. The server never sees a blocked request, but a page's own `fetch` or XHR is answered with a `403` in its place rather than dropped, so the page's handling of a refusal really runs: a page that then claims success is reported as a `false_success` ([ADR 9](docs/adr/0009-a-refused-write-is-answered-not-dropped.md)).
 
 ---
 
@@ -329,6 +335,8 @@ A `🛡 WRITE-POLICY blocked` notice is the safety net doing its job, not an app
 - 💯 **Page scores** (0–100: a11y · craft · consistency · task-clarity), ranked worst-first, with stale scores from old runs marked as such.
 - 👥 **A role capability matrix** — what each role could and couldn't reach.
 - 🧾 **A gap ledger** — everything *not* done, so the report is honest about its own coverage.
+- ⏱️ **How the run was paced** — actions, median gap, idle share and held-idle time per session, so a browser held open for nothing is visible.
+- 🎯 **How well the lanes judged** — on a parallel run, whether the confidence each lane stated matched what the project went on to file, beside what later re-tests found ([ADR 10](docs/adr/0010-a-confidence-is-checked-not-trusted.md)).
 
 `.scenescout/report.html` — the same report as one self-contained page, with every session's trail beside it, and on a [recorded run](#-recording-a-run-and-reading-it-back) the screenshots under each finding.
 
@@ -539,7 +547,7 @@ npx -y scenescout watch <path>      # the same, live in your browser, with each
 
 ```
 src/
-  mcp-server.ts     the 25 tools + per-session dispatch
+  mcp-server.ts     the 29 tools + per-session dispatch
   scan.ts           project discovery (framework, routes, auth)
   cli.ts            scan · serve · install · doctor · status
   installer.ts      setup logic (skill link, MCP registration, diagnostics)
@@ -549,8 +557,15 @@ src/
     fingerprint.ts  route + element-set identity (state hashing)
     oracles.ts      console/page/network/HTTP error detection
     injection.ts    the DOM-injection oracle's rules (what to watch for, how to find it)
+    claims.ts       when the page contradicts the server (refused_empty, false_success)
+    request.ts      what a replayed API call may be and where it may go
+    brief.ts        splitting the app between parallel lanes
+    lane.ts         the typed report a lane hands back
+    calibration.ts  whether a lane's confidence held up; what it judged and never filed
+    pace.ts         how a run spent its time
+    bench.ts        scoring a run against the demo app's answer key
     policy.ts       the write-policy safety net
-    ownership.ts    safe-write: which records did this run create?
+    ownership.ts    safe-write: which records were created in this process?
     uploads.ts      disk uploads, fenced to the project by real path
     journey.ts      task-ease measurement from the action log
     design.ts       the design audit + page scoring
@@ -558,9 +573,11 @@ src/
     report.ts       the gap ledger + report generation
     replay.ts       the run as one page: steps, tasks, frames under each finding
     …               collector · dispatch · fixtures · authloss · reaper
-scripts/            the 17 test suites (smoke/ holds the real-browser ones)
+scripts/            the 23 test suites (smoke/ holds the real-browser ones)
 test-app/           fixtures for the real-browser smoke tests
 skills/scenescout/   the testing method (SKILL.md): a skill in Claude Code, served by the server everywhere else
+docs/how-it-works.md  what happens at each stage, in diagrams
+docs/benchmark.md   measuring whether a change made runs better
 docs/adr/           why it's built this way
 ```
 
@@ -570,6 +587,10 @@ docs/adr/           why it's built this way
 
 ## 🧠 Design decisions
 
+**[How it works, stage by stage](docs/how-it-works.md)** — diagrams of the run lifecycle, what happens inside one action, the write policy on the wire, how a violation becomes a finding, how a parallel run is split and folded, how roles hand work to each other, where a run's time goes, and how a lane's confidence is checked afterwards.
+
+**[Measuring whether a change helped](docs/benchmark.md)** — the demo app's answer key, the scorecard (recall, precision, judged-not-filed, severity, calibration), and the results log of every run, including what did not help.
+
 The load-bearing choices are recorded as ADRs — read the relevant one before changing a rule it covers:
 
 - [1 · Completion is an enforced contract, not a claim](docs/adr/0001-completion-is-a-contract-not-a-vibe.md)
@@ -578,6 +599,10 @@ The load-bearing choices are recorded as ADRs — read the relevant one before c
 - [4 · Findings dedup on machine signals, and a merge must never lose a finding](docs/adr/0004-dedup-on-machine-signals-not-prose.md)
 - [5 · Testable logic lives outside `browser.ts`](docs/adr/0005-keep-testable-logic-out-of-the-browser-module.md)
 - [6 · Nothing in this repo names or is tuned for a tested app](docs/adr/0006-stay-project-agnostic.md)
+- [7 · The live view is local, read-only, and leaves nothing behind](docs/adr/0007-the-live-view-is-local-read-only-and-leaves-nothing-behind.md)
+- [8 · Recording is opt-in, and a recorded run is one self-contained page](docs/adr/0008-a-recorded-run-is-evidence-and-must-be-asked-for.md)
+- [9 · A refused write is answered, not dropped](docs/adr/0009-a-refused-write-is-answered-not-dropped.md)
+- [10 · A lane's confidence is checked, not trusted](docs/adr/0010-a-confidence-is-checked-not-trusted.md)
 
 ---
 
@@ -589,8 +614,9 @@ Working on SceneScout itself is the only reason to clone it:
 git clone https://github.com/brunoboto96/SceneScout.git scenescout && cd scenescout
 npm install        # installs dependencies and builds
 npm run setup      # same as `scenescout install`, but registers THIS checkout (the skill is linked, so edits are live)
-npm test           # build + 14 suites: scan, oracle, policy, fixture, dispatch, design,
-                   #                     contract, memory, install, hygiene, smoke, mcp-check
+npm test           # build + 23 suites: 21 pure-logic suites (scan, oracle, policy, … bench, hygiene),
+                   #                     then smoke and mcp-check (the server over stdio), both with real browsers
+npm run bench -- --all   # re-score every archived benchmark run against the current answer key
 npm run demo       # regenerate examples/ from the demo app
 ```
 

package/dist/engine/bench.js

@@ -0,0 +1,485 @@
+/**
+ * Scoring a run against an answer key.
+ *
+ * Every change to this engine and to its skill has been made on judgement:
+ * something looked wrong in a run, it was fixed, and the next run looked
+ * better — or looked different, which is not the same thing. There was no
+ * number that could go up or down, so there was no way to tell a fix from a
+ * coincidence, or to notice a change that made things worse.
+ *
+ * An app with KNOWN defects makes a number possible. Given what the app
+ * actually contains and what a run reported, this computes recall (what it
+ * found of what was there), precision (how much of what it reported was
+ * real), how often it judged a defect and then never filed it, whether the
+ * severities it chose agree with the key, and whether the confidence its lanes
+ * stated was worth what they said — measured against the key, which is the
+ * one ground truth a run cannot talk itself into.
+ *
+ * Every future change is accepted or rejected by the number this produces, so
+ * the failure that matters most here is a PLAUSIBLE WRONG number. Three rules
+ * follow from that. A piece of text two key entries both claim is reported as
+ * ambiguous and scored as neither. The key is validated when it is read, so a
+ * mistyped level cannot silently drop a defect from recall. And the key is
+ * hashed into every scorecard, so two runs scored against different keys are
+ * never compared by accident.
+ *
+ * The key is data, not code: everything specific to one app lives in its key
+ * file, so this module stays as generic as the rest of the engine.
+ *
+ * Pure, so every rule is table-tested.
+ */
+import { createHash } from "node:crypto";
+import { z } from "zod";
+import { BUCKET_EDGES, bucketLabel, bucketOf } from "./calibration.js";
+export const LEVELS = ["minimal", "medium", "extensive"];
+const SEVERITIES = ["high", "medium", "low"];
+const Pattern = z.string().refine((p) => {
+    try {
+        new RegExp(p, "i");
+        return true;
+    }
+    catch {
+        return false;
+    }
+}, { message: "is not a valid regular expression" });
+const Entry = z
+    .object({
+    id: z.string().min(1),
+    route: z.string().startsWith("/"),
+    title: z.string().min(1),
+    /** The category a finding for it should carry. Reported, not enforced: two categories can both be defensible. */
+    category: z.string().min(1),
+    /** The severity the key's author would give it. A judgement, and labelled as one wherever it is reported. */
+    severity: z.enum(SEVERITIES),
+    /** The lowest level whose contract is expected to find it. A defect only a fuzzing pass can reach is not a miss at `medium`. */
+    level: z.enum(LEVELS),
+    /** Case-insensitive regular expressions tried against a finding's evidence and title. */
+    match: z.array(Pattern).min(1),
+    /** Phrasings that MUST classify to this entry. The title is always one of them. */
+    examples: z.array(z.string()).default([]),
+    /** Near misses that must NOT classify to this entry — the other half of each contrastive pair. */
+    counterExamples: z.array(z.string()).default([]),
+})
+    .strict();
+const NonDefect = z
+    .object({
+    id: z.string().min(1),
+    title: z.string().min(1),
+    /** Why it is not a defect, for the reader of a scorecard. */
+    why: z.string().min(1),
+    match: z.array(Pattern).min(1),
+    /**
+     * The defects this non-defect is a narrowing of: text that matches both is
+     * the false claim, not the defect. Only these. A non-defect that happens to
+     * overlap any other entry makes the text ambiguous, because a phrasing
+     * nobody anticipated is likelier to be the real defect than the false claim.
+     */
+    overrides: z.array(z.string()).default([]),
+    examples: z.array(z.string()).default([]),
+    counterExamples: z.array(z.string()).default([]),
+})
+    .strict();
+const Key = z
+    .object({
+    app: z.string().min(1),
+    _comment: z.string().optional(),
+    /** The defects the app was built to contain. Recall is measured against these. */
+    defects: z.array(Entry).min(1),
+    /**
+     * Real problems a run found that nobody planted. They count as correct for
+     * precision but not toward recall, so the number the key was designed around
+     * stays comparable from one run to the next.
+     */
+    alsoReal: z.array(Entry).default([]),
+    nonDefects: z.array(NonDefect).default([]),
+})
+    .strict();
+/**
+ * Read and validate a key. A key that fails here fails loudly: a mistyped
+ * `"level": "Minimal"` used to drop the defect out of recall with no error,
+ * and the run scored as if the app had one defect fewer.
+ */
+export function parseKey(raw) {
+    const parsed = Key.safeParse(raw);
+    if (!parsed.success) {
+        const issue = parsed.error.issues[0];
+        throw new Error(`The answer key is not valid at ${issue.path.join(".") || "(root)"}: ${issue.message}`);
+    }
+    const ids = [...parsed.data.defects, ...parsed.data.alsoReal, ...parsed.data.nonDefects].map((e) => e.id);
+    const dup = ids.find((id, i) => ids.indexOf(id) !== i);
+    if (dup)
+        throw new Error(`The answer key uses the id ${JSON.stringify(dup)} twice.`);
+    const real = new Set([...parsed.data.defects, ...parsed.data.alsoReal].map((e) => e.id));
+    for (const nd of parsed.data.nonDefects) {
+        const unknown = nd.overrides.find((id) => !real.has(id));
+        if (unknown)
+            throw new Error(`The non-defect ${JSON.stringify(nd.id)} overrides ${JSON.stringify(unknown)}, which is not a defect in the key.`);
+    }
+    return parsed.data;
+}
+/** A short, stable fingerprint of a key, so a scorecard says which key produced it. */
+export function keyHash(key) {
+    const canonical = JSON.stringify({ defects: key.defects, alsoReal: key.alsoReal, nonDefects: key.nonDefects });
+    return createHash("sha256").update(canonical).digest("hex").slice(0, 10);
+}
+const RANK = { minimal: 0, medium: 1, extensive: 2 };
+const SEVERITY_RANK = { low: 0, medium: 1, high: 2 };
+function matches(patterns, text) {
+    return patterns.some((p) => new RegExp(p, "i").test(text));
+}
+/**
+ * What a piece of reported text is, according to the key.
+ *
+ * A known non-defect can be a deliberate narrowing of a real defect — "the
+ * filter is stuck after the Archived 500" names the Archived failure but claims
+ * something false about it — so a non-defect wins over the defects it lists in
+ * `overrides`, and over nothing else. Anything else that more than one entry
+ * claims is AMBIGUOUS, reported, and scored as neither: crediting the first
+ * match would quietly count a finding about two defects as one, and letting a
+ * non-defect win everywhere turned realistic rewordings of real defects into
+ * false positives with a confident explanation beside them.
+ */
+export function classify(text, key) {
+    const nd = key.nonDefects.filter((e) => matches(e.match, text));
+    const real = [
+        ...key.defects.filter((e) => matches(e.match, text)).map((entry) => ({ kind: "defect", entry })),
+        ...key.alsoReal.filter((e) => matches(e.match, text)).map((entry) => ({ kind: "alsoReal", entry })),
+    ];
+    if (nd.length > 1)
+        return { kind: "ambiguous", ids: [...nd, ...real.map((m) => m.entry)].map((e) => e.id) };
+    if (nd.length === 1) {
+        const rest = real.filter((m) => !nd[0].overrides.includes(m.entry.id));
+        if (rest.length === 0)
+            return { kind: "nonDefect", entry: nd[0] };
+        return { kind: "ambiguous", ids: [nd[0].id, ...rest.map((m) => m.entry.id)] };
+    }
+    if (real.length === 1)
+        return real[0];
+    if (real.length > 1)
+        return { kind: "ambiguous", ids: real.map((m) => m.entry.id) };
+    return null;
+}
+/** The text a finding is classified on. Evidence leads, because it is the machine signature; the title backs it up. */
+export function findingText(f) {
+    return `${f.evidence ?? ""}\n${f.title}`;
+}
+/** The text a lane decision is classified on. */
+export function decisionText(d) {
+    return `${d.evidence ?? ""}\n${d.observation}`;
+}
+/**
+ * A key that disagrees with its own examples is wrong before any run is
+ * scored. Returns every title or example — non-defects' included — that
+ * classifies anywhere but its own entry, and every counter-example that
+ * classifies TO its entry.
+ */
+export function lintKey(key) {
+    const problems = [];
+    const describe = (got) => got === null ? "nothing" : got.kind === "ambiguous" ? `ambiguous (${got.ids.join(", ")})` : got.entry.id;
+    const check = (id, kind, text, want) => {
+        const got = classify(text, key);
+        const hit = got !== null && got.kind !== "ambiguous" && got.kind === kind && got.entry.id === id;
+        if (want && !hit)
+            problems.push(`${id}: ${JSON.stringify(text)} classified as ${describe(got)}`);
+        if (!want && hit)
+            problems.push(`${id}: counter-example ${JSON.stringify(text)} classified TO it`);
+    };
+    for (const [kind, list] of [
+        ["defect", key.defects],
+        ["alsoReal", key.alsoReal],
+    ]) {
+        for (const e of list) {
+            for (const t of [e.title, ...e.examples])
+                check(e.id, kind, t, true);
+            for (const t of e.counterExamples)
+                check(e.id, kind, t, false);
+        }
+    }
+    for (const e of key.nonDefects) {
+        for (const t of [e.title, ...e.examples])
+            check(e.id, "nonDefect", t, true);
+        for (const t of e.counterExamples)
+            check(e.id, "nonDefect", t, false);
+    }
+    return problems;
+}
+/**
+ * Whether a lane's verdict was right, according to the key.
+ *
+ * "defect" on a planted or also-real defect is right, and on a known
+ * non-defect is wrong. "not_a_defect" is the reverse. An "unsure" verdict, a
+ * dismissal as another lane's, a decision the key does not name, and one it
+ * names ambiguously are not scored — the same rule the in-product calibration
+ * follows, for the same reason.
+ */
+export function judgeDecision(d, key) {
+    if (d.verdict === "unsure")
+        return null;
+    if (isScopeDismissal(d))
+        return null;
+    const m = classify(decisionText(d), key);
+    if (!m || m.kind === "ambiguous")
+        return null;
+    const isDefect = m.kind !== "nonDefect";
+    return d.verdict === "defect" ? isDefect : !isDefect;
+}
+/** How a lane says "not mine": the wording lanes actually used, none of it about a particular app. */
+const SCOPE_DISMISSAL_RE = /out.of.lane.scope|out-of-scope|out of (my|this) (lane|scope)|belongs?.to.{0,20}\b(lane|route|page)\b|belongs-to-|(handled|owned) by (the |another )?[\w-]* ?lane|lane owns it|not (in |on |from |part of )?(my|this) (lane|assigned routes|routes?|pages?)\b|outside (my|this) (lane|routes?|pages?)|not part of my (assigned )?routes/i;
+/** A not-a-defect verdict whose stated reason is that the thing is another lane's. */
+export function isScopeDismissal(d) {
+    return d.verdict === "not_a_defect" && SCOPE_DISMISSAL_RE.test(decisionText(d));
+}
+export function calibrateAgainstKey(decisions, key) {
+    if (decisions.length === 0)
+        return null;
+    const buckets = BUCKET_EDGES.map(() => ({ n: 0, conf: 0, right: 0 }));
+    const out = { judged: 0, correct: 0, notInKey: 0, ambiguous: 0, badConfidence: 0, unsure: 0, outOfScope: 0, buckets: [], ece: 0, brier: 0 };
+    for (const d of decisions) {
+        if (d.verdict === "unsure") {
+            out.unsure += 1;
+            continue;
+        }
+        if (isScopeDismissal(d)) {
+            out.outOfScope += 1;
+            continue;
+        }
+        const c = d.confidence;
+        if (typeof c !== "number" || !Number.isFinite(c) || c < 0 || c > 1) {
+            out.badConfidence += 1;
+            continue;
+        }
+        const m = classify(decisionText(d), key);
+        if (!m) {
+            out.notInKey += 1;
+            continue;
+        }
+        if (m.kind === "ambiguous") {
+            out.ambiguous += 1;
+            continue;
+        }
+        const right = d.verdict === "defect" ? m.kind !== "nonDefect" : m.kind === "nonDefect";
+        out.judged += 1;
+        if (right)
+            out.correct += 1;
+        out.brier += (c - (right ? 1 : 0)) ** 2;
+        const b = buckets[bucketOf(c)];
+        b.n += 1;
+        b.conf += c;
+        if (right)
+            b.right += 1;
+    }
+    if (out.judged > 0)
+        out.brier /= out.judged;
+    buckets.forEach((b, i) => {
+        if (b.n === 0)
+            return;
+        out.ece += (b.n / out.judged) * Math.abs(b.conf / b.n - b.right / b.n);
+        out.buckets.push({ label: bucketLabel(i), decisions: b.n, stated: b.conf / b.n, correct: b.right / b.n });
+    });
+    return out;
+}
+/**
+ * Score one run.
+ *
+ * `findings` should be the run's own: a project memory accumulates findings
+ * across runs, and scoring an accumulated one credits a run with what an
+ * earlier run found. Point the scorer at a fresh project directory per run.
+ */
+export function score(key, findings, decisions, level = "medium") {
+    const hitsByDefect = new Map();
+    const falsePositives = [];
+    const unknown = [];
+    const ambiguous = [];
+    let correct = 0;
+    for (const f of findings) {
+        const m = classify(findingText(f), key);
+        if (!m) {
+            unknown.push({ title: f.title, severity: f.severity, evidence: f.evidence ?? "" });
+            continue;
+        }
+        if (m.kind === "ambiguous") {
+            ambiguous.push({ title: f.title, ids: m.ids });
+            continue;
+        }
+        if (m.kind === "nonDefect") {
+            falsePositives.push({ id: m.entry.id, title: f.title, why: m.entry.why, severity: f.severity });
+            continue;
+        }
+        correct += 1;
+        if (m.kind === "defect") {
+            const list = hitsByDefect.get(m.entry.id) ?? [];
+            list.push(f);
+            hitsByDefect.set(m.entry.id, list);
+        }
+    }
+    const expectedDefects = key.defects.filter((d) => RANK[d.level] <= RANK[level]);
+    const found = expectedDefects.filter((d) => hitsByDefect.has(d.id)).map((d) => d.id);
+    const missed = expectedDefects.filter((d) => !hitsByDefect.has(d.id)).map((d) => d.id);
+    const beyondLevel = key.defects.filter((d) => RANK[d.level] > RANK[level] && hitsByDefect.has(d.id)).map((d) => d.id);
+    // A defect a lane called out in its report, which no finding covers.
+    const judged = new Set();
+    for (const d of decisions) {
+        if (d.verdict !== "defect")
+            continue;
+        const m = classify(decisionText(d), key);
+        if (m?.kind === "defect" && !hitsByDefect.has(m.entry.id))
+            judged.add(m.entry.id);
+    }
+    const severity = { agree: 0, higher: 0, lower: 0, detail: [] };
+    let duplicates = 0;
+    for (const d of key.defects) {
+        const hits = hitsByDefect.get(d.id);
+        if (!hits)
+            continue;
+        duplicates += hits.length - 1;
+        // The most severe finding for it is the one a reader acts on.
+        const filed = hits.map((h) => h.severity).sort((a, b) => (SEVERITY_RANK[b] ?? 0) - (SEVERITY_RANK[a] ?? 0))[0];
+        const diff = (SEVERITY_RANK[filed] ?? 0) - SEVERITY_RANK[d.severity];
+        if (diff === 0)
+            severity.agree += 1;
+        else if (diff > 0)
+            severity.higher += 1;
+        else
+            severity.lower += 1;
+        severity.detail.push({ id: d.id, filed, key: d.severity });
+    }
+    return {
+        app: key.app,
+        key: keyHash(key),
+        level,
+        expected: expectedDefects.length,
+        found,
+        missed,
+        judgedNotFiled: [...judged],
+        beyondLevel,
+        findings: findings.length,
+        correct,
+        falsePositives,
+        unknown,
+        ambiguous,
+        duplicates,
+        severity,
+        calibration: calibrateAgainstKey(decisions, key),
+    };
+}
+const pct = (n, d) => (d === 0 ? "—" : `${Math.round((n / d) * 100)}%`);
+/**
+ * Precision, with its bounds. The labelled ratio leaves unlabelled findings out
+ * of the denominator, so on its own it cannot move when a change adds five new
+ * false claims nobody has judged yet. The bounds can: the lower one counts
+ * every open finding as wrong, the upper one as right.
+ */
+export function precisionBounds(c) {
+    const labelled = c.correct + c.falsePositives.length;
+    const open = c.unknown.length + c.ambiguous.length;
+    return { labelled: `${c.correct}/${labelled} (${pct(c.correct, labelled)})`, low: pct(c.correct, c.findings), high: pct(c.correct + open, c.findings) };
+}
+/** The scorecard as a person reads it. Leads with the two numbers, then says what each is made of. */
+export function formatScorecard(c) {
+    const p = precisionBounds(c);
+    const open = c.unknown.length + c.ambiguous.length;
+    const lines = [
+        `SCORECARD — ${c.app}, level ${c.level}, key ${c.key}`,
+        ``,
+        `Recall     ${c.found.length}/${c.expected} (${pct(c.found.length, c.expected)}) of the planted defects expected at this level`,
+        `Precision  ${p.labelled} of the findings the key can label` +
+            (open ? ` — ${open} of ${c.findings} unlabelled, so between ${p.low} and ${p.high} of all findings` : ""),
+        ``,
+    ];
+    if (c.missed.length)
+        lines.push(`Missed: ${c.missed.join(", ")}`);
+    if (c.judgedNotFiled.length)
+        lines.push(`Judged a defect in a lane report, never filed: ${c.judgedNotFiled.join(", ")}`);
+    if (c.beyondLevel.length)
+        lines.push(`Found above this level's contract: ${c.beyondLevel.join(", ")}`);
+    if (c.falsePositives.length) {
+        lines.push(``, `False positives (${c.falsePositives.length}):`);
+        for (const fp of c.falsePositives)
+            lines.push(`  [${fp.severity}] ${fp.title} — ${fp.why}`);
+    }
+    if (c.ambiguous.length) {
+        lines.push(``, `Ambiguous (${c.ambiguous.length}) — the key claims each twice; sharpen it:`);
+        for (const a of c.ambiguous)
+            lines.push(`  ${a.title} (${a.ids.join(" / ")})`);
+    }
+    if (c.unknown.length) {
+        lines.push(``, `Unlabelled (${c.unknown.length}) — add to the key once a person has judged them:`);
+        for (const u of c.unknown)
+            lines.push(`  [${u.severity}] ${u.title}`);
+    }
+    const s = c.severity;
+    lines.push(``, `Severity vs the key's judgement: ${s.agree} agree, ${s.higher} filed higher, ${s.lower} filed lower` +
+        (s.detail.some((d) => d.filed !== d.key)
+            ? ` (${s.detail
+                .filter((d) => d.filed !== d.key)
+                .map((d) => `${d.id} ${d.filed}→${d.key}`)
+                .join(", ")})`
+            : ""));
+    if (c.duplicates)
+        lines.push(`Duplicates: at least ${c.duplicates} extra finding(s) for a defect that already had one (after the store's own merge)`);
+    const k = c.calibration;
+    if (k) {
+        const skipped = [
+            k.notInKey && `${k.notInKey} about things the key does not name`,
+            k.ambiguous && `${k.ambiguous} the key names ambiguously`,
+            k.badConfidence && `${k.badConfidence} with an unusable confidence`,
+            k.unsure && `${k.unsure} unsure`,
+            k.outOfScope && `${k.outOfScope} dismissed as another lane's`,
+        ].filter(Boolean);
+        lines.push(``, k.judged === 0
+            ? `Lane calibration against the key: nothing the key could judge` + (skipped.length ? ` (${skipped.join(", ")})` : "")
+            : `Lane calibration against the key: ${k.correct}/${k.judged} verdicts right (${pct(k.correct, k.judged)}), expected calibration error ${k.ece.toFixed(2)}, Brier ${k.brier.toFixed(3)}` +
+                (skipped.length ? ` — not scored: ${skipped.join(", ")}` : ""));
+        for (const b of k.buckets)
+            lines.push(`  stated ${b.label}: ${b.decisions} decision(s), said ${b.stated.toFixed(2)}, right ${Math.round(b.correct * 100)}%`);
+    }
+    return lines.join("\n");
+}
+/**
+ * The date a run is archived under. A run's decisions carry the time each was
+ * made, so the date defaults to the last of them, and a date given by hand
+ * may not be later: `--all` sorts by it, and a run dated after it happened
+ * reads as the newer of two in the results log. A run with no decisions has
+ * nothing to check against and must be given one.
+ */
+export function runDate(decisions, given) {
+    if (given !== undefined && !/^\d{4}-\d{2}-\d{2}$/.test(given))
+        throw new Error(`The date must be YYYY-MM-DD, not ${JSON.stringify(given)}.`);
+    const last = decisions
+        .map((d) => d.at)
+        .filter((at) => typeof at === "string" && /^\d{4}-\d{2}-\d{2}/.test(at))
+        .sort()
+        .at(-1)
+        ?.slice(0, 10);
+    if (!last) {
+        if (given === undefined)
+            throw new Error("The run has no timestamped decisions to date it by; give it a date.");
+        return given;
+    }
+    if (given !== undefined && given > last)
+        throw new Error(`The run's last decision was made on ${last}; it cannot be dated ${given}.`);
+    return given ?? last;
+}
+/**
+ * Paths into a machine's own directories, which a finding's evidence can pick
+ * up from a stack trace or an upload. An archive is committed, and a home
+ * directory names a person.
+ */
+const LOCAL_PATH_RE = /(?:\/Users\/[^/\s"']+|\/home\/[^/\s"']+|\/private\/tmp|\/tmp|[A-Za-z]:\\Users\\[^\\\s"']+)[^\s"']*/g;
+export function sanitize(text) {
+    return text.replace(LOCAL_PATH_RE, "<path>");
+}
+export function toArchive(run, date, note, findings, decisions) {
+    return {
+        run,
+        date,
+        note,
+        findings: findings.map((f) => ({
+            title: sanitize(f.title),
+            severity: f.severity,
+            ...(f.category ? { category: f.category } : {}),
+            ...(f.evidence ? { evidence: sanitize(f.evidence) } : {}),
+        })),
+        decisions: decisions.map((d) => ({ ...d, observation: sanitize(d.observation), evidence: d.evidence === null ? null : sanitize(d.evidence) })),
+    };
+}