scenescout 3.11.1 → 3.12.0

package/CHANGELOG.md

@@ -1,5 +1,19 @@
 # scenescout
 
+## 3.12.0
+
+### Minor Changes
+
+- 967e98c: Add a "worth a look" tier for observations that are defects only under a convention of the project the run cannot see, such as a spacing scale, link styling in navigation, or test ids on every control. SceneScout reports each one with the convention that would decide it, and never counts it as a defect.
+  
+  - Lane reports accept the verdict `worth_a_look`, which must name its `convention`. A `convention` on any other verdict is ignored, and the fold says so. Lane calibration and the benchmark's key calibration leave it unscored and count it under its own reason.
+  - `scout_finding` takes an optional `convention`, which files the finding in this tier. The report lists these findings under "Worth a look", below the findings, as "a defect only if your project uses …", and leaves them out of every defect total. Filing the same thing again as a defect promotes it, at the defect's severity, and the reply says so. The benchmark sets such findings aside from recall and precision, and the fold lists a lane's judged defect as unfiled when it was filed only as worth a look.
+  - `scenescout check` reports two new rules, `off-grid-spacing` and `indistinct-link`, in this tier. They have no severity and never fail the gate at any `--fail-on`. SARIF reports them at level `note`. The report and the job summary list them in their own section, `check.json` puts them under `worthALook`, separate from `issues`, and the GitHub Action publishes their number as the `worth-a-look` output. Every existing rule is unchanged.
+
+### Patch Changes
+
+- c40892f: `scout_lane_report` keeps the routes each lane's accepted report lists in the project's memory (`laneRoutes` in `.scenescout/memory.json`), with query strings and fragments removed so no token in an address is stored, and `npm run bench -- --archive` carries them into the run archive. The benchmark then decides whether a lane's "not a defect" is a remark about another lane's page by where the matched defect is, not by how the verdict is worded: a defect none of whose pages the lane covered is set aside as another lane's and listed on the scorecard, and one on the lane's own page, or one every lane can reach, is scored. Answer-key entries can name further pages with `alsoOn` and mark a defect every lane can reach with `everyPage`. Archives made before routes were kept, and lanes with a route that names no page, are scored by wording, as before.
+
 ## 3.11.1
 
 ### Patch Changes

package/README.md

@@ -332,6 +332,7 @@ A `🛡 WRITE-POLICY blocked` notice is the safety net doing its job, not an app
 `.scenescout/report.md` — a deduplicated, worst-first report with:
 
 - 🐛 **Findings** with repro traces and generated Playwright regression-test skeletons.
+- 🔎 **Worth a look** — observations that are defects only under a convention of your project the run cannot see (a spacing scale, link styling in navigation, test ids on every control), each naming that convention. Listed below the findings and not counted as defects ([ADR 13](docs/adr/0013-a-convention-is-the-projects-to-decide.md)).
 - 💯 **Page scores** (0–100: a11y · craft · consistency · task-clarity), ranked worst-first, with stale scores from old runs marked as such.
 - 👥 **A role capability matrix** — what each role could and couldn't reach.
 - 🧾 **A gap ledger** — everything *not* done, so the report is honest about its own coverage.
@@ -355,6 +356,8 @@ An exploratory run is driven by a model, so two runs never find exactly the same
 - contrast and focus
 - pages with no way out
 
+Some of what it measures is a defect only under a convention the check cannot see: paddings off a 4px grid, and links styled like body text. Those are listed under **Worth a look**, each with the convention that would make it a defect. They are never counted and never fail the gate, at any `--fail-on`; SARIF reports them at level `note` ([ADR 13](docs/adr/0013-a-convention-is-the-projects-to-decide.md)).
+
 ```bash
 npx scenescout check http://127.0.0.1:3000 --fail-on high
 ```
@@ -380,7 +383,7 @@ On GitHub Actions, this repository is also an action that installs everything an
 With the default settings its saved flows send no HTTP write (they replay under observe's rule), and its crawl runs under `--mode observe` or `read-only`; `--flow-writes allow` lets flows write as `--mode` allows. By default it fails only on facts that mean a page is broken: a page that did not load, an uncaught exception, a 5xx, a failure shown as success. Other options:
 
 - `--fail-on medium` or `low` makes the gate stricter.
-- `--ignore <rule>` drops a rule.
+- `--ignore <rule>` drops a rule, worth-a-look rules included.
 - `--paths /a,/b` checks only those pages.
 - `--storage-state <file>` checks while signed in.
 - `--flows <dir>` or `off` chooses which saved flows to replay; `--retest off` skips re-testing open findings.
@@ -661,6 +664,7 @@ The load-bearing choices are recorded as ADRs — read the relevant one before c
 - [10 · A lane's confidence is checked, not trusted](docs/adr/0010-a-confidence-is-checked-not-trusted.md)
 - [11 · A gate is deterministic, and fails only on what it can prove](docs/adr/0011-a-gate-is-deterministic-and-fails-only-on-what-it-can-prove.md)
 - [12 · A check replays saved flows and re-tests open findings, within settings whose defaults do the least harm](docs/adr/0012-a-check-replays-saved-flows-and-reports-re-tests.md)
+- [13 · What depends on a project's convention is the project's to decide](docs/adr/0013-a-convention-is-the-projects-to-decide.md)
 
 ---
 

package/dist/check-run.js

@@ -9,7 +9,7 @@ import fs from "node:fs";
 import os from "node:os";
 import path from "node:path";
 import { BrowserEngine } from "./engine/browser.js";
-import { issuesFromRoutes, redactFlowRuns, redactRoute, redactRoutes, settingsOf, withoutOwnResponse, } from "./engine/check.js";
+import { checkFindings, redactFlowRuns, redactRoute, redactRoutes, settingsOf, withoutOwnResponse, } from "./engine/check.js";
 import { loadFlows, resolveFlowsDir } from "./engine/flow.js";
 import { MemoryStore, MEMORY_DIRNAME } from "./engine/memory.js";
 import { checkRetestPlan, retestResults, wellFormedFindings } from "./engine/verify.js";
@@ -123,13 +123,15 @@ export async function runCheck(options, log = () => { }, inputs = { flows: [], f
             : null;
         const measured = redactRoutes(routes.map(withoutOwnResponse));
         const flows = redactFlowRuns(flowRuns);
+        const { issues, worthALook } = checkFindings(measured, start.origin, options.ignore, flows);
         return {
             url: redactRoute(options.url),
             generatedAt: new Date().toISOString(),
             mode: options.mode,
             failOn: options.failOn,
             routes: measured,
-            issues: issuesFromRoutes(measured, start.origin, options.ignore, flows),
+            issues,
+            worthALook,
             // Routes that failed to load are issues already; "not visited" is only what --max-routes left out.
             unvisited: options.paths ? [] : engine.crawlableRoutes().map(redactRoute),
             ignored: options.ignore,

package/dist/engine/bench.js

@@ -31,6 +31,9 @@
 import { createHash } from "node:crypto";
 import { z } from "zod";
 import { BUCKET_EDGES, bucketLabel, bucketOf } from "./calibration.js";
+import { laneRoutePaths, stripRouteQuery } from "./fingerprint.js";
+import { isWorthALook } from "./memory.js";
+export { laneRoutePaths };
 export const LEVELS = ["minimal", "medium", "extensive"];
 const SEVERITIES = ["high", "medium", "low"];
 const Pattern = z.string().refine((p) => {
@@ -42,10 +45,24 @@ const Pattern = z.string().refine((p) => {
         return false;
     }
 }, { message: "is not a valid regular expression" });
+/**
+ * Other pages the same defect shows on, beyond `route`. A lane that owns any
+ * of them owns the defect, so its "not a defect" is a verdict, not a remark
+ * about another lane's page.
+ */
+const AlsoOn = z.array(z.string().startsWith("/")).min(1).optional();
+/**
+ * Every lane can reach the defect: it is in chrome every page carries (a
+ * shared nav, header or footer), or in an endpoint any lane can call. No lane
+ * can call it another lane's, so a verdict on it is always scored.
+ */
+const EveryPage = z.boolean().optional();
 const Entry = z
     .object({
     id: z.string().min(1),
     route: z.string().startsWith("/"),
+    alsoOn: AlsoOn,
+    everyPage: EveryPage,
     title: z.string().min(1),
     /** The category a finding for it should carry. Reported, not enforced: two categories can both be defensible. */
     category: z.string().min(1),
@@ -83,6 +100,8 @@ const Contextual = z
     .object({
     id: z.string().min(1),
     route: z.string().startsWith("/"),
+    alsoOn: AlsoOn,
+    everyPage: EveryPage,
     title: z.string().min(1),
     category: z.string().min(1),
     /** The convention that decides it: where it would be a defect, and why the run cannot tell whether it holds here. */
@@ -130,6 +149,14 @@ export function parseKey(raw) {
     const dup = ids.find((id, i) => ids.indexOf(id) !== i);
     if (dup)
         throw new Error(`The answer key uses the id ${JSON.stringify(dup)} twice.`);
+    for (const e of [...parsed.data.defects, ...parsed.data.alsoReal, ...parsed.data.contextual]) {
+        if (e.alsoOn && e.everyPage)
+            throw new Error(`The entry ${JSON.stringify(e.id)} has both alsoOn and everyPage; an entry on every page names no other pages.`);
+        const pages = [e.route, ...(e.alsoOn ?? [])];
+        const dup = pages.find((p, i) => pages.indexOf(p) !== i);
+        if (dup)
+            throw new Error(`The entry ${JSON.stringify(e.id)} names the page ${JSON.stringify(dup)} twice in route and alsoOn.`);
+    }
     const real = new Set([...parsed.data.defects, ...parsed.data.alsoReal].map((e) => e.id));
     for (const nd of parsed.data.nonDefects) {
         const unknown = nd.overrides.find((id) => !real.has(id));
@@ -241,14 +268,14 @@ export function lintKey(key) {
  *
  * "defect" on a planted or also-real defect is right, and on a known
  * non-defect is wrong. "not_a_defect" is the reverse. An "unsure" verdict, a
- * dismissal as another lane's, a decision the key does not name, one it
- * names ambiguously, and one about a contextual entry are not scored — the same rule the in-product calibration
+ * "worth_a_look", a dismissal as another lane's, a decision the key does not
+ * name, one it names ambiguously, and one about a contextual entry are not scored — the same rule the in-product calibration
  * follows, for the same reason.
  */
-export function judgeDecision(d, key) {
-    if (d.verdict === "unsure")
+export function judgeDecision(d, key, laneRoutes) {
+    if (d.verdict === "unsure" || d.verdict === "worth_a_look")
         return null;
-    if (isScopeDismissal(d))
+    if (isOwnershipRemark(d, key, laneRoutes))
         return null;
     const m = classify(decisionText(d), key);
     if (!m || m.kind === "ambiguous" || m.kind === "contextual")
@@ -258,13 +285,79 @@ export function judgeDecision(d, key) {
 }
 /** How a lane says "not mine": the wording lanes actually used, none of it about a particular app. */
 const SCOPE_DISMISSAL_RE = /out.of.lane.scope|out-of-scope|out of (my|this) (lane|scope)|belongs?.to.{0,20}\b(lane|route|page)\b|belongs-to-|(handled|owned) by (the |another )?[\w-]* ?lane|lane owns it|not (in |on |from |part of )?(my|this) (lane|assigned routes|routes?|pages?)\b|outside (my|this) (lane|routes?|pages?)|not part of my (assigned )?routes/i;
-/** A not-a-defect verdict whose stated reason is that the thing is another lane's. */
+/**
+ * A not-a-defect verdict whose stated reason is that the thing is another
+ * lane's. The WORDING rule: all a run archived before lane routes were kept
+ * can be judged by, and still what decides for a lane whose routes are not
+ * known. It cannot tell a lane's own page from another's, which is why it is
+ * not widened (see `isOwnershipRemark`).
+ */
 export function isScopeDismissal(d) {
     return d.verdict === "not_a_defect" && SCOPE_DISMISSAL_RE.test(decisionText(d));
 }
-export function calibrateAgainstKey(decisions, key) {
+/** The pages a key entry is on, in the same form as `laneRoutePaths`. */
+function entryPaths(e) {
+    return [e.route, ...(e.alsoOn ?? [])].flatMap(laneRoutePaths);
+}
+/**
+ * A lane's routes as a set of paths, or undefined when they are not known
+ * well enough to decide by: none were recorded, or ANY of them names no path
+ * ("the orders area"). A route that could not be read may be the very page a
+ * verdict is about, and deciding without it would set that verdict aside.
+ */
+function lanePaths(lane, laneRoutes) {
+    const routes = laneRoutes && Object.hasOwn(laneRoutes, lane) ? laneRoutes[lane] : undefined;
+    if (!routes || routes.length === 0)
+        return undefined;
+    const paths = new Set();
+    for (const r of routes) {
+        const named = laneRoutePaths(r);
+        if (named.length === 0)
+            return undefined;
+        for (const p of named)
+            paths.add(p);
+    }
+    return paths;
+}
+/**
+ * A not-a-defect that is a remark about who owns the thing, not a verdict on
+ * whether it is broken.
+ *
+ * Where the lane's routes are known, decided by WHERE the thing is: the
+ * verdict matches a planted or also-real defect whose pages are all outside
+ * the lane's routes. The wording no longer matters either way — "raised on /
+ * before navigating" from the orders lane is a remark about the dashboard's
+ * defect, and "handled by another lane" from the lane that owns the page is
+ * its own wrong verdict. A defect in chrome every page carries (`everyPage`)
+ * is every lane's, so it is never another lane's. A verdict that matches a
+ * known non-defect, a contextual entry, several entries or none is not about
+ * a defect with a page to own, and is judged as it would be otherwise.
+ *
+ * Where they are not known — every archive made before routes were kept, and
+ * any lane that reported no route — the wording rule decides, unchanged, so
+ * those runs score as they always have.
+ */
+export function ownershipRemark(d, key, laneRoutes) {
+    if (d.verdict !== "not_a_defect")
+        return null;
+    const own = lanePaths(d.lane, laneRoutes);
+    if (!own)
+        return isScopeDismissal(d) ? { by: "wording" } : null;
+    const m = classify(decisionText(d), key);
+    if (!m || (m.kind !== "defect" && m.kind !== "alsoReal"))
+        return null;
+    if (m.entry.everyPage)
+        return null;
+    return entryPaths(m.entry).some((p) => own.has(p)) ? null : { by: "routes", id: m.entry.id };
+}
+export function isOwnershipRemark(d, key, laneRoutes) {
+    return ownershipRemark(d, key, laneRoutes) !== null;
+}
+export function calibrateAgainstKey(decisions, key, laneRoutes) {
     if (decisions.length === 0)
         return null;
+    const lanes = [...new Set(decisions.map((d) => d.lane))];
+    const known = lanes.filter((l) => lanePaths(l, laneRoutes) !== undefined).length;
     const buckets = BUCKET_EDGES.map(() => ({ n: 0, conf: 0, right: 0 }));
     const out = {
         judged: 0,
@@ -274,7 +367,10 @@ export function calibrateAgainstKey(decisions, key) {
         badConfidence: 0,
         unsure: 0,
         outOfScope: 0,
+        ownership: known === 0 ? "wording" : known === lanes.length ? "routes" : "mixed",
+        setAsideByRoute: [],
         contextual: 0,
+        worthALook: 0,
         buckets: [],
         ece: 0,
         brier: 0,
@@ -284,8 +380,15 @@ export function calibrateAgainstKey(decisions, key) {
             out.unsure += 1;
             continue;
         }
-        if (isScopeDismissal(d)) {
+        if (d.verdict === "worth_a_look") {
+            out.worthALook += 1;
+            continue;
+        }
+        const remark = ownershipRemark(d, key, laneRoutes);
+        if (remark) {
             out.outOfScope += 1;
+            if (remark.by === "routes")
+                out.setAsideByRoute.push({ lane: d.lane, id: remark.id, confidence: d.confidence });
             continue;
         }
         const c = d.confidence;
@@ -334,14 +437,19 @@ export function calibrateAgainstKey(decisions, key) {
  * across runs, and scoring an accumulated one credits a run with what an
  * earlier run found. Point the scorer at a fresh project directory per run.
  */
-export function score(key, findings, decisions, level = "medium") {
+export function score(key, findings, decisions, level = "medium", laneRoutes) {
     const hitsByDefect = new Map();
     const falsePositives = [];
     const unknown = [];
     const ambiguous = [];
     const contextual = [];
+    const worthALook = [];
     let correct = 0;
     for (const f of findings) {
+        if (isWorthALook(f)) {
+            worthALook.push({ title: f.title, convention: f.convention ?? "" });
+            continue;
+        }
         const m = classify(findingText(f), key);
         if (!m) {
             unknown.push({ title: f.title, severity: f.severity, evidence: f.evidence ?? "" });
@@ -410,11 +518,12 @@ export function score(key, findings, decisions, level = "medium") {
         correct,
         falsePositives,
         contextual,
+        worthALook,
         unknown,
         ambiguous,
         duplicates,
         severity,
-        calibration: calibrateAgainstKey(decisions, key),
+        calibration: calibrateAgainstKey(decisions, key, laneRoutes),
     };
 }
 const pct = (n, d) => (d === 0 ? "—" : `${Math.round((n / d) * 100)}%`);
@@ -424,13 +533,18 @@ const pct = (n, d) => (d === 0 ? "—" : `${Math.round((n / d) * 100)}%`);
  * false claims nobody has judged yet. The bounds can: the lower one counts
  * every open finding as wrong, the upper one as right. Findings set aside as
  * contextual are in neither: they are not claims the run could get right.
+ * Nor are findings the run filed as worth a look: it did not claim them.
  */
 export function precisionBounds(c) {
     const labelled = c.correct + c.falsePositives.length;
     const open = c.unknown.length + c.ambiguous.length;
-    const scored = c.findings - c.contextual.length;
+    const scored = c.findings - setAside(c);
     return { labelled: `${c.correct}/${labelled} (${pct(c.correct, labelled)})`, low: pct(c.correct, scored), high: pct(c.correct + open, scored) };
 }
+/** Findings in neither precision count: contextual ones, and the run's own worth-a-looks. */
+function setAside(c) {
+    return c.contextual.length + (c.worthALook?.length ?? 0);
+}
 /** The scorecard as a person reads it. Leads with the two numbers, then says what each is made of. */
 export function formatScorecard(c) {
     const p = precisionBounds(c);
@@ -441,11 +555,13 @@ export function formatScorecard(c) {
         `Recall     ${c.found.length}/${c.expected} (${pct(c.found.length, c.expected)}) of the planted defects expected at this level`,
         `Precision  ${p.labelled} of the findings the key can label` +
             (open
-                ? ` — ${open} of ${c.findings - c.contextual.length} unlabelled, so between ${p.low} and ${p.high} of ${c.contextual.length ? "the findings not set aside" : "all findings"}`
+                ? ` — ${open} of ${c.findings - setAside(c)} unlabelled, so between ${p.low} and ${p.high} of ${setAside(c) ? "the findings not set aside" : "all findings"}`
                 : ""),
     ];
     if (c.contextual.length)
         lines.push(`Set aside  ${c.contextual.length} of ${c.findings} finding(s): defects only under a convention the run cannot see, so in neither count`);
+    if (c.worthALook.length)
+        lines.push(`Set aside  ${c.worthALook.length} of ${c.findings} finding(s) the run filed as worth a look: not claimed as defects, so in neither recall nor precision`);
     lines.push(``);
     if (c.missed.length)
         lines.push(`Missed: ${c.missed.join(", ")}`);
@@ -463,6 +579,11 @@ export function formatScorecard(c) {
         for (const x of c.contextual)
             lines.push(`  ${x.title} (${x.id})`);
     }
+    if (c.worthALook.length) {
+        lines.push(``, `Filed as worth a look (${c.worthALook.length}):`);
+        for (const x of c.worthALook)
+            lines.push(`  ${x.title}${x.convention ? ` — a defect only if the project uses ${x.convention}` : ""}`);
+    }
     if (c.ambiguous.length) {
         lines.push(``, `Ambiguous (${c.ambiguous.length}) — the key claims each twice; sharpen it:`);
         for (const a of c.ambiguous)
@@ -490,8 +611,10 @@ export function formatScorecard(c) {
             k.ambiguous && `${k.ambiguous} the key names ambiguously`,
             k.badConfidence && `${k.badConfidence} with an unusable confidence`,
             k.unsure && `${k.unsure} unsure`,
-            k.outOfScope && `${k.outOfScope} dismissed as another lane's`,
+            k.outOfScope &&
+                `${k.outOfScope} dismissed as another lane's (${k.ownership === "routes" ? "by the lanes' routes" : k.ownership === "wording" ? "by wording: no lane routes archived" : "by the lanes' routes where archived, else by wording"})`,
             k.contextual && `${k.contextual} about things that are defects only under a convention the run cannot see`,
+            k.worthALook && `${k.worthALook} the lane marked worth a look`,
         ].filter(Boolean);
         lines.push(``, k.judged === 0
             ? `Lane calibration against the key: nothing the key could judge` + (skipped.length ? ` (${skipped.join(", ")})` : "")
@@ -499,6 +622,11 @@ export function formatScorecard(c) {
                 (skipped.length ? ` — not scored: ${skipped.join(", ")}` : ""));
         for (const b of k.buckets)
             lines.push(`  stated ${b.label}: ${b.decisions} decision(s), said ${b.stated.toFixed(2)}, right ${Math.round(b.correct * 100)}%`);
+        if (k.setAsideByRoute.length) {
+            lines.push(`  Set aside as another lane's, by the lanes' routes (${k.setAsideByRoute.length}):`);
+            for (const r of k.setAsideByRoute)
+                lines.push(`    ${r.lane} on ${r.id}, stated ${Number.isFinite(r.confidence) ? r.confidence.toFixed(2) : "?"}`);
+        }
     }
     return lines.join("\n");
 }
@@ -543,17 +671,22 @@ const LOCAL_PATH_RE = /(?:\/Users\/[^/\s"']+|\/home\/[^/\s"']+|\/private\/tmp|\/
 export function sanitize(text) {
     return text.replace(LOCAL_PATH_RE, "<path>");
 }
-export function toArchive(run, date, note, findings, decisions, app) {
+export function toArchive(run, date, note, findings, decisions, app, laneRoutes = {}) {
+    // No query string or fragment reaches an archive: a lane copies routes from
+    // the address bar, and an address can carry a token.
+    const routes = Object.fromEntries(Object.entries(laneRoutes).map(([lane, list]) => [lane, list.map((r) => sanitize(stripRouteQuery(r)))]));
     return {
         app,
         run,
         date,
         note,
+        ...(Object.keys(routes).length ? { laneRoutes: routes } : {}),
         findings: findings.map((f) => ({
             title: sanitize(f.title),
             severity: f.severity,
             ...(f.category ? { category: f.category } : {}),
             ...(f.evidence ? { evidence: sanitize(f.evidence) } : {}),
+            ...(isWorthALook(f) ? { tier: "worth_a_look", ...(f.convention ? { convention: sanitize(f.convention) } : {}) } : {}),
         })),
         decisions: decisions.map((d) => ({ ...d, observation: sanitize(d.observation), evidence: d.evidence === null ? null : sanitize(d.evidence) })),
     };

package/dist/engine/calibration.js

@@ -21,7 +21,7 @@
  * Pure, so every rule here is table-tested.
  */
 import { requestsPair, statedRequests, templatedPathsMatch } from "./lane.js";
-import { failingSignatures } from "./memory.js";
+import { failingSignatures, isWorthALook } from "./memory.js";
 /**
  * Upper edge of each confidence bucket. Five is enough to see a shape and few
  * enough that each holds a usable count on a run of a few dozen decisions;
@@ -87,7 +87,8 @@ function usableConfidence(value) {
  */
 export function calibrate(decisions, findings) {
     const filedKeys = new Map();
-    for (const f of findings) {
+    // Filed as worth a look is not filed as a defect: a lane that called it a defect was not agreed with.
+    for (const f of findings.filter((x) => !isWorthALook(x))) {
         if (!f.evidence)
             continue;
         // A finding verified as still present is the most informative match, so it
@@ -102,8 +103,11 @@ export function calibrate(decisions, findings) {
     const claims = decisions.filter((d) => d.verdict === "defect" && d.evidence !== null);
     const checkable = claims.filter((d) => usableConfidence(d.confidence) !== null && joinKeys(d.evidence).size > 0);
     const unjoinable = claims.length - checkable.length;
+    const worthALook = decisions.filter((d) => d.verdict === "worth_a_look").length;
     if (checkable.length === 0)
-        return unjoinable > 0 ? { checkable: 0, filed: 0, stated: 0, buckets: [], ece: 0, verified: { present: 0, gone: 0, changed: 0 }, unjoinable } : null;
+        return unjoinable > 0 || worthALook > 0
+            ? { checkable: 0, filed: 0, stated: 0, buckets: [], ece: 0, verified: { present: 0, gone: 0, changed: 0 }, unjoinable, worthALook }
+            : null;
     const buckets = BUCKET_EDGES.map(() => ({ n: 0, conf: 0, hits: 0 }));
     const verified = { present: 0, gone: 0, changed: 0 };
     const matched = new Map();
@@ -147,7 +151,7 @@ export function calibrate(decisions, findings) {
         ece += (b.n / n) * Math.abs(meanConf - rate);
         out.push({ label: bucketLabel(i), decisions: b.n, stated: meanConf, filed: rate });
     });
-    return { checkable: n, filed, stated: stated / n, buckets: out, ece, verified, unjoinable };
+    return { checkable: n, filed, stated: stated / n, buckets: out, ece, verified, unjoinable, worthALook };
 }
 const pct = (x) => `${Math.round(x * 100)}%`;
 /**
@@ -163,7 +167,7 @@ export function formatCalibration(c) {
         // and a run that recorded none look identical when the section simply
         // vanishes, and the reader concludes the feature is broken — which is this
         // project's own definition of a silent path.
-        const had = c.checkable + c.unjoinable;
+        const had = c.checkable + c.unjoinable + c.worthALook;
         if (had === 0)
             return [];
         return [
@@ -171,6 +175,7 @@ export function formatCalibration(c) {
             ``,
             `Not enough to say yet: ${c.checkable} lane decision(s) could be checked${c.unjoinable > 0 ? ` (and ${c.unjoinable} could not be looked up at all)` : ""}, ` +
                 `and ${MIN_FOR_A_VERDICT} are needed before a calibration figure survives one of them changing.`,
+            ...worthALookNote(c),
             ``,
         ];
     }
@@ -194,6 +199,7 @@ export function formatCalibration(c) {
     if (c.unjoinable > 0) {
         lines.push(``, `${c.unjoinable} further decision(s) called a defect but named no failing endpoint, so nothing could be looked up for them. They are excluded above rather than counted as wrong.`);
     }
+    lines.push(...worthALookNote(c));
     const seen = c.verified.present + c.verified.gone + c.verified.changed;
     if (seen > 0) {
         lines.push(``, `${seen} of the findings those decisions matched ${seen === 1 ? "has" : "have"} since been re-tested with \`scout_verify\`: ${c.verified.present} still present, ${c.verified.gone} gone, ${c.verified.changed} changed. ` +
@@ -202,6 +208,16 @@ export function formatCalibration(c) {
     lines.push(``);
     return lines;
 }
+/** The line saying how many "worth a look" decisions were left out, and why; nothing when there were none. */
+function worthALookNote(c) {
+    if (c.worthALook === 0)
+        return [];
+    return [
+        ``,
+        `${c.worthALook} decision(s) were marked worth a look: real, and a defect only under a convention of the project the run cannot see. ` +
+            `Whether one was filed says nothing about whether the lane was right, so they are not scored.`,
+    ];
+}
 /** Below this, the number swings on a single decision and is worse than no number. */
 export const MIN_FOR_A_VERDICT = 8;
 /**
@@ -259,7 +275,8 @@ export const MAX_UNFILED_NAMED = 10;
 export function unfiledDefects(decisions, findings) {
     const keys = [];
     const filed = [];
-    for (const f of findings) {
+    // A defect filed only as worth a look is not in the report's findings, so it is still unfiled.
+    for (const f of findings.filter((x) => !isWorthALook(x))) {
         const text = `${f.evidence ?? ""} ${f.title}`;
         const requests = statedRequests(f.evidence ?? "");
         // The store's signatures, and the finding's own failing requests as