@qaguardian/playwright-score 2.1.1 → 2.3.0

package/CHANGELOG.md

@@ -10,6 +10,165 @@ parsing accuracy, not grading policy — can still move a suite's score
 without itself being a new model version; 2.1.0 below is exactly that
 case, and says so explicitly.
 
+## 2.3.0 — 2026-11-14
+
+Scoring weights and the scoring model (v4) are unchanged, but **scores can
+fall**: this release adds ten rules that catch tests which would miss a real
+bug or flake, and suites that use those patterns now lose points. Nothing
+gets more lenient. Two of the new rules are report-only (shown in the report,
+no effect on the score): `pwscore/no-self-fulfilling-fallback` (info, no real
+suite hit it yet, validated on fixtures only) and `pwscore/no-brittle-selector`
+(its cost is already counted by the raw-locator ratio), plus
+`pwscore/serial-mode-dependency` (info).
+
+New rules, flake risk (QAG-287 slice 2b):
+
+- `pwscore/no-unawaited-action` (error, hygiene): Playwright action or navigation on a Page/Locator as a bare statement without `await`.
+- `pwscore/no-brittle-selector` (warning, report-only): XPath, `:nth-child`, 4+ combinator CSS chains, generated class names, positional `.first()/.nth()/.last()` on a raw `.locator()`.
+- `pwscore/no-shared-mutable-state` (warning, structure) and report-only `pwscore/serial-mode-dependency` (info): state written in one test and read in another. Not flagged when the writing and the reading test both sit in a serial describe (`test.describe.serial` or `test.describe.configure({ mode: 'serial' })`, at file or describe level), wherever the variable is declared; flagged when either test is outside serial scope. Plus a report-only note for serial describe blocks.
+- `pwscore/no-excessive-timeout` (warning, hygiene): option timeouts over 60s (exactly 60s is not flagged), `test.setTimeout` >= 5 min, unconditional `test.slow()`.
+
+New rules, tests that would miss a real bug (QAG-287 slice 2a):
+
+- `pwscore/no-fixme-test` (structure, warning): unconditional `test.fixme('t', fn)`, `test.describe.fixme(...)`, in-body `test.fixme()` and `test.fixme(true, ...)`. `test.fixme(condition, reason)` stays clean.
+- `pwscore/no-swallowed-assertion` (assertions, warning): `expect(...)` inside a `try` whose `catch` neither rethrows, asserts, fails, returns a value, records state nor calls `.skip(...)`; and `await expect(...).catch(() => {})` with the result discarded. Retry loops are not flagged. Warning, not error, because some real suites use deliberate soft checks.
+- `pwscore/no-weak-assertion` (assertions, error): `expect(count).toBeGreaterThanOrEqual(0)`, `expect(<Locator>).toBeTruthy()/toBeDefined()/not.toBeNull()`, `expect(await x.isVisible()).toBeDefined()`.
+- `pwscore/no-unawaited-expect-value` (assertions, error): a Promise-returning Playwright call such as `page.title()` or `locator.innerText()` used without `await` inside `expect()` with a non-web-first matcher. Replaces the misleading `playwright/prefer-web-first-assertions` message on the same line.
+- `pwscore/no-self-fulfilling-fallback` (assertions, info, report-only): a negated state check (or a `catch` of a failed expect) whose fallback fills in or selects the value itself (click-only fallbacks are setup and stay clean), followed by an assertion on the same locator.
+
+Fix (test-factory recognition): tests declared through a factory function (`const t = testWithConfig({...}); t('x', ...)`, in the same or an imported file) are now recognised as tests, removing false `playwright/no-standalone-expect` errors.
+
+New report format (QAG-288), output only:
+
+- New `--format report`: a one-page, self-contained HTML suite report
+  (headline, dimension bars, top findings with code excerpts, flake risk,
+  items needing a human check, next steps). Light/dark, phone width, prints
+  to one or two pages. All source and paths are HTML-escaped. `--output` is
+  accepted as an alias of `--out`. New `--name <suite name>` sets the report
+  headline. Without it the label is the last two path segments (three when the
+  folder is generic, e.g. `e2e` or `tests`); it never includes a home directory
+  or anything above it, and a home directory itself is labelled "Playwright suite".
+- Findings are ranked with a display-only confidence tier (high / medium /
+  needs-check) and every rule has a plain-English title. The text and
+  markdown formats list findings by severity, then confidence, instead of
+  file order.
+- The report states how many tests need a human check (the report-only
+  `playwright-score/assertion-unverified` findings, kept out of the score),
+  groups them by file, and reads "why it matters" in business terms. A suite
+  hard-failed by an unparseable file says so in the headline.
+- Results gain an optional in-memory `baseDir`, used only by the report to
+  read code excerpts. It is never printed and is not part of the JSON, SARIF,
+  markdown or HTML output. A rule without plain-English copy renders with a
+  generic title instead of failing.
+
+Report and terminal polish (QAG-288 slice 3b), output only, scores unchanged:
+
+- The HTML report is now a Summary page plus a Details part. The Summary
+  (score, score by area, top 3 findings as rows without code, next steps) prints
+  on one A4 or Letter page; Details (code excerpts, findings 4-5, the
+  human-check list) starts on page 2. The duplicate "Flake risk" table is gone:
+  each top finding names the file with the most occurrences, and Details lists
+  the most flake-prone files.
+- A line under the score explains it (weighted average of four areas, the
+  grade bands, the pass bar; "your bar" when `--threshold` was passed). "clean"
+  became "have no issues".
+- Confidence reads Definite / Likely / Needs a human check, with one legend
+  line and neutral tags (fixes the 3.95:1 contrast failure of the amber tag).
+  The four areas are named Reliability, Checks, Resilience to UI changes and
+  Maintainability (the JSON keys are unchanged).
+- A suite with a syntax error or no tests shows "Not scored" instead of
+  "0/100 F", with no zero bars and consistent counts.
+- Rule copy says "can" instead of "will" for outcomes static analysis cannot
+  prove. The weakest area gets a slot in the top 3 when it scores under 75.
+- A rule that throws while scoring no longer vanishes silently: the run still
+  finishes, the CLI prints `warning: rule <id> crashed on <n> node(s); results
+  may be incomplete` to stderr, the text, markdown and report formats say "Some
+  checks could not run on this suite; results may be incomplete.", and the JSON
+  result gains `ruleCrashes: [{ rule, count }]`, only when a rule crashed (JSON
+  of a clean run is unchanged).
+- A suite with no tests no longer lists the working directory as an issue, and
+  it is "Not scored" rather than "0/100 F".
+- Without `--name`, the suite name falls back to the last two or three path
+  segments for a folder (never above the home directory), so
+  `e2e-tests/playwright` is no longer just `playwright`.
+- Terminal output (`--format text`) is grouped by rule, worst first, with a
+  count per rule, relative paths, at most three locations per rule (`+ N more`)
+  and at most ten rules; it opens with the verdict and repeats it on the last
+  line. `--format json`, `sarif` and `markdown` are byte-identical to before.
+
+## 2.2.0 — 2026-11-14
+
+False-positive fixes found by hand-checking error-level findings on public
+suites (7 of 18 were wrong), then tightened after review so the fixes cannot be used to hide a test with no assertion. No rule's weight or severity changed, so this
+is not a new scoring model (still v4), but **scores can rise**: findings
+that were wrong are no longer reported. Nothing gets stricter.
+
+- `playwright/expect-expect`: a test that asserts through a fixture
+  function injected via the test args is no longer reported as having no
+  assertion, but only when the fixture is a callable that really asserts:
+  a `test.extend({ check: async ({ page }, use) => { await use(async () =>
+  { await expect(...) }) } })` entry whose function handed to `use(...)`
+  runs `expect`, the test destructures it AND calls it, and the fixtures
+  module is reachable from that spec through relative imports. An `expect`
+  in a fixture's setup or teardown does not count, nor does any fixture
+  named like a built-in one (`page`, `context`, `request`, ...). A test
+  also counts as asserting through an object-literal method / class-property
+  arrow function in a file it imports (a fixture factory's returned object,
+  a page object), except that a helper whose name collides with a Playwright
+  method (`goto`, `click`, `fill`, `check`, `get`, `post`, ...) is never
+  treated as an assertion helper, so it cannot turn every `page.goto()` into
+  an assertion. When a test has no `expect` of its own but calls an
+  assertion-named method (`assert`, `verify`, `check`, `validate`, `expect`,
+  `ensure`, `should`, `confirm` followed by an upper-case letter, `_`, a digit
+  or nothing, such as `checkTotals` but not `checkout`; or `toBe*`, `toHave*`,
+  `toContain*`, `toMatch*`, `toEqual*`, `toThrow*`) on an object this scan
+  cannot read (an injected fixture from another package, an unresolvable
+  import, a third-party package), it is no longer an error: it is reported
+  as the report-only finding `playwright-score/assertion-unverified`
+  ("needs a human check"), which does not lower the score and is not
+  counted in `summary.unassertedTests`. Those tests are counted separately
+  in `summary.unverifiedTests`, a new field that is always present in the
+  JSON output (`0` when nothing is unverified; the text and Markdown reports
+  show those tests as unverified, never as asserting): they are not known to
+  assert. A fixture a test calls, whose `test` comes from a module the scan
+  does not resolve (a tsconfig `paths` alias, a workspace package), is
+  unverified the same way. Names bound by destructuring
+  (`const { channelsPage } = await pw.login(user)`) take the kind of the
+  expression they come from. Query-style names (`is...`, `has...`,
+  `wait...`), helper methods named like Playwright actions (`fill`, `check`,
+  `goto`, ...), built-in calls (`new Date().toISOString()`,
+  `user.email.toLowerCase()`, `x.toString()`), `shop.checkout()` and
+  `loginPage.rememberMe.check()` are not treated as assertions: a call whose
+  name is not assertion-like, a test that only calls Playwright API on
+  `page`/`request`/..., and helpers whose body we can read and which do not
+  assert are still `expect-expect` errors.
+- `pwscore/no-trivial-assertion`: `expect(pageErrors).toEqual([])` where
+  `pageErrors` is a `const` array that may be mutated (passed to a call as
+  an argument, the receiver of `push`/`pop`/`splice`/`sort`/..., an
+  assignment target, aliased, or captured by a closure that does one of
+  those, such as a listener, or an inner array reached through an index:
+  `rows[0].push(9)`) is no longer flagged. A `.length` or index
+  read, a non-mutating method such as `.map()`, and an unrelated property
+  with the same name (`obj.expected`) do not count as mutation; literals
+  and a `const` array that is only read are still flagged.
+- `pwscore/no-timer-sleep`: a retry/backoff pause inside a plain helper
+  function (declaration, `const f = () =>`, class method) is no longer
+  flagged, but only when the sleep sits in a `catch` clause or in a loop
+  whose body contains a non-empty `try` with a `catch`, and the helper is not declared inside
+  a test, hook, describe or step callback. Sleeps directly in a test body or
+  hook, any loop without `try`/`catch` (including a hand-rolled
+  `while (await x.isVisible()) sleep()` polling wait), helpers declared
+  inside a test callback, and bare `sleep()` helpers are still flagged.
+- `playwright/no-conditional-in-test`: a guard whose only body is `throw`
+  (`if (!id) throw new Error(...)`, no `else`) is no longer flagged.
+- `playwright/valid-expect`: a helper whose name merely ends in `Expect`
+  (`clickSortableHeaderAndExpect(...)`) is no longer reported as an
+  `expect()` without a matcher.
+- Known and not fixed: `playwright/no-standalone-expect` on tests declared
+  through a factory function that returns a `test.extend(...)` result
+  (`const t = testWithConfig({...}); t('name', ...)`). The alias detection
+  only follows direct `test.extend` chains.
+
 ## 2.1.1 — 2026-09-26
 
 Scoring model stays v4. Reviewer follow-ups from the 2.1.0 release review

package/METHODOLOGY.md

@@ -82,7 +82,7 @@ owns this dimension; counting them as demerits too would double-penalize.)
 | Dimension | Weight | Demerit sources |
 |---|---:|---|
 | playwrightHygiene | 40 | hard waits (`page.waitForTimeout`, `new Promise`+`setTimeout` and its respellings, `node:timers/promises` setTimeout), networkidle, force, coordinate-based mouse interaction (`*.mouse.click/dblclick/move/down/up(x, y)`, partial demerit — see below), missing await, element handles, eval, conditionals, page.pause, useless await |
-| assertions | 25 | a test with no recognized assertion is fully flawed (`expect-expect`, after all delegation resolution); an assertion whose subject/matcher combination can only ever pass (`expect(true).toBe(true)`, `expect([]).toEqual([])`, never a deliberate force-fail like `expect(true).toBe(false)`); `valid-expect`, `no-standalone-expect`, `prefer-web-first-assertions`, a soft-assertion-only test (`expect.poll` counts as hard, not soft) all add partial demerits |
+| assertions | 25 | a test with no recognized assertion is fully flawed (`expect-expect`, after all delegation resolution; a test with no `expect` of its own that calls an assertion-named method (`assert`, `verify`, `check`, `validate`, `expect`, `ensure`, `should`, `confirm` followed by an upper-case letter, `_`, a digit or nothing; or `toBe*`, `toHave*`, `toContain*`, `toMatch*`, `toEqual*`, `toThrow*`) on an object the scan cannot read, or calls a fixture that comes from a module the scan cannot resolve is a report-only `playwright-score/assertion-unverified` "needs a human check" finding, not a demerit, and is counted in `summary.unverifiedTests` rather than as asserting; any other call on such an object does not excuse the test, so query-style names (`is...`, `has...`, `wait...`), helper methods named like Playwright actions (`fill`, `check`, `goto`) and built-in calls are never treated as assertions; an assertion helper or fixture only counts when its body is readable, it is called by the test, and its name does not collide with a Playwright method); an assertion whose subject/matcher combination can only ever pass (`expect(true).toBe(true)`, `expect([]).toEqual([])`, never a deliberate force-fail like `expect(true).toBe(false)`, and never an array that may be mutated elsewhere: passed to a call, the receiver of a mutating method such as `push`, an assignment target, or captured by a closure such as an event listener doing so, including an inner array mutated through an index such as `rows[0].push(9)`); `valid-expect`, `no-standalone-expect`, `prefer-web-first-assertions`, a soft-assertion-only test (`expect.poll` counts as hard, not soft) all add partial demerits |
 | locators | 20 | usage ratio (above) |
 | structure | 15 | focused tests, always-skipped declarations, nested-describe depth, oversized files (module scope), an inline `eslint-disable` suppression comment (partial demerit — see "Known limitations" in VALIDATION.md) |
 
@@ -126,6 +126,20 @@ The exact ESLint rules that produce findings are enabled explicitly in
 eslint-plugin-playwright's `recommended` set at install time, so identical
 code scores identically regardless of which plugin version npm resolves.
 
+### Tests that would miss a real bug
+
+Five rules (QAG-287) target tests that pass whatever the app does. Each
+matches one exact syntactic shape and stays silent when unsure.
+
+| Rule | Dimension, severity | Exact shape |
+|---|---|---|
+| `pwscore/no-fixme-test` | structure, warning | `test.fixme('t', fn)` / `test.describe.fixme(...)` declarations; in-body `test.fixme()` or `test.fixme(true, ...)`. Conditional `test.fixme(cond, 'why')` is clean. |
+| `pwscore/no-swallowed-assertion` | assertions, warning | `expect(...)` in a `try` whose `catch` has no throw, expect, fail-ish call, returned value or assignment (not inside a loop); or `await expect(...).catch(fn)` as a statement where `fn` does not throw. |
+| `pwscore/no-weak-assertion` | assertions, error | `.count()`/`.length` `toBeGreaterThanOrEqual(0)`; a syntactic Locator (`page.locator`, `getBy*`, or a const from one) asserted `toBeTruthy`/`toBeDefined`/`not.toBeNull`; `isVisible()`-style and `count()` results asserted `toBeDefined`. `toBeTruthy()` on other values is not flagged. |
+| `pwscore/no-unawaited-expect-value` | assertions, error | `expect(<promise call>)` without `await`, receiver syntactically a Page/Locator, matcher not web-first (also an un-awaited promise call as the expected value of `toBe`/`toEqual`). `.resolves`/`.rejects` are exempt. |
+| `pwscore/no-self-fulfilling-fallback` | assertions, info, report-only (no score effect; no real-suite hits, validated on fixtures only) | Inside a test: `if (!check) { actions }`, the `else` of `if (check) {...} else { actions }`, or `try { expect/check } catch { actions }`, where the actions include `fill`/`type`/`pressSequentially`/`selectOption`/`setInputFiles` (click-only and key-press fallbacks are the usual "make sure the menu is open" setup and are not flagged), followed in the same block by an `expect` on a locator shared with the check or the actions. Positive-branch actions (dismiss a banner if visible) are never flagged. |
+
+
 ---
 
 ## Why v4
@@ -190,3 +204,30 @@ real inputs:
    diluted `findings/SLOC` and raised the score without fixing anything.
    A per-test ratio has no such lever — padding with fake tests costs
    assertions coverage instead.
+
+---
+
+## Flake-risk rules (unreleased)
+
+Five static checks for the patterns that make a test pass or fail by luck.
+Each is syntactic and conservative: when the receiver or value cannot be
+read from the source, nothing is reported.
+
+| Rule | Exact shape | Severity / dimension |
+|---|---|---|
+| `pwscore/no-unawaited-action` | An expression statement that is exactly `<receiver>.<action>(...)` inside an `async` function, where the receiver is `page`, a `.page` property, a `page.locator/getBy*/first/nth/filter` chain, or a `const` initialised from one, and the action is goto/reload/goBack/goForward/click/dblclick/fill/clear/press/type/check/uncheck/setChecked/selectOption/hover/focus/tap/dragTo/setInputFiles/setContent/waitFor*. Not matched: await, return, void, assignment, argument, `Promise.all`, `.catch()`/`.then()` chains, other receivers. Chosen over eslint-plugin-playwright's `missing-playwright-await` `includePageLocatorMethods` option because that one matches the method name on any receiver. | error / hygiene |
+| `pwscore/no-brittle-selector` | A fully static selector string passed to `.locator()`, `.frameLocator()` or a legacy selector method on a Page/Locator that is XPath (`xpath=`, `//`, `(//`), uses `:nth-child`/`:nth-of-type`, is a plain CSS chain with 4 or more combinators, or contains a generated class (`.css-1a2b3c`, `.sc-bdfBwQ`, `.jss12`, `.MuiBox-root-12`); plus `.nth()/.first()/.last()` directly on a raw `.locator('<string>')` (test-id, `role=` and `text=` strings exempt). | warning, **report-only** / locators |
+| `pwscore/no-shared-mutable-state` | A module- or describe-level `let`/`var` (or `const` object/array/Map/Set) that one `test()` body assigns or mutates and another `test()` body reads without writing it. Not matched: writes only in hooks, any `beforeEach` write, reader that also writes. | warning / structure |
+| `pwscore/serial-mode-dependency` | `test.describe.configure({ mode: 'serial' })` or `test.describe.serial(...)`. | info, **report-only** / structure |
+| `pwscore/no-excessive-timeout` | `{ timeout: N }` with N > 60,000 (strictly more than one minute) on an action, wait or matcher call; `test.setTimeout(N)`/`testInfo.setTimeout(N)` with N >= 300,000; `test.slow()` with no arguments directly at file or describe level. Literals and simple arithmetic only (`2 * 60 * 1000`, `60_000`). Config files are not read. | warning / hygiene |
+
+`no-brittle-selector` is report-only on purpose: the locators dimension is a
+native/raw ratio that already counts every raw `.locator()` and legacy
+selector call, so scoring the same call again would charge it twice. The
+finding is named evidence for why that raw call is fragile.
+
+**Test factories.** A function whose `return` is `<test>.extend(...)`, a known
+test alias, or a call of another such function (same file or imported through
+relative paths) is a test factory; `const t = factory(...)` makes `t` a test
+alias, so `t('name', ...)` is a test declaration and its `expect` calls are not
+"standalone".

package/README.md

@@ -99,7 +99,7 @@ npx playwright-score ./flow.spec.ts --format json --out report.json
 |---|---|
 | `--profile standard` | Scoring profile (default, and only, profile) |
 | `--threshold <n>` | Pass bar 0–100 |
-| `--format text\|json\|markdown\|sarif` | Output format |
+| `--format text\|json\|markdown\|sarif\|report` | Output format (`report` is a self-contained one-page HTML suite report) |
 | `--out <file>` | Write report to file |
 | `--version` | Print version |
 

package/dist/cli.js

@@ -9,6 +9,9 @@ import { formatJson } from './formatters/json.js';
 import { formatText } from './formatters/text.js';
 import { formatMarkdown } from './formatters/markdown.js';
 import { formatSarif } from './formatters/sarif.js';
+import { formatReport } from './formatters/report.js';
+import { makeSourceReader } from './report-source.js';
+import { suiteLabel } from './suite-name.js';
 function printHelp() {
     console.log(`@qaguardian/playwright-score — deterministic Playwright spec quality score
 
@@ -21,8 +24,12 @@ Options:
   --profile <standard>             Scoring profile (default, and only
                                    supported value, as of 2.0.0: standard).
   --threshold <n>                 Pass threshold 0-100 (default: 80)
-  --format <text|json|markdown|sarif>  Output format (default: text)
-  --out <file>                    Write report to file
+  --format <text|json|markdown|sarif|report>
+                                  Output format (default: text). report is a
+                                  self-contained one-page HTML suite report.
+  --out, --output <file>          Write report to file
+  --name <suite name>             Suite name for the report headline
+                                  (default: the directory basename)
   --version                       Print version
   --help                          Show help
 
@@ -43,6 +50,30 @@ Methodology: METHODOLOGY.md (scoring model v4)
 // size (64KB on macOS/Linux). Setting process.exitCode and returning lets
 // Node exit naturally once pending I/O actually completes, which is the
 // standard fix for this class of bug.
+async function packageVersion() {
+    try {
+        const { createRequire } = await import('node:module');
+        const require = createRequire(import.meta.url);
+        return require('../package.json').version ?? 'unknown';
+    }
+    catch {
+        return 'unknown';
+    }
+}
+// Never an absolute local path: see suite-name.ts.
+function suiteName(paths) {
+    const names = paths.slice(0, 2).map((p) => {
+        let isDir = false;
+        try {
+            isDir = fs.statSync(p).isDirectory();
+        }
+        catch {
+            /* missing path: scorePaths reports it */
+        }
+        return suiteLabel(p, isDir);
+    });
+    return names.join(', ') + (paths.length > 2 ? ` and ${paths.length - 2} more` : '');
+}
 async function main() {
     const args = process.argv.slice(2);
     if (args.includes('--version') || args.includes('-v')) {
@@ -67,6 +98,7 @@ async function main() {
     let threshold;
     let format = 'text';
     let out;
+    let name;
     const paths = [];
     for (let i = 0; i < args.length; i++) {
         const a = args[i];
@@ -99,16 +131,24 @@ async function main() {
         }
         else if (a === '--format') {
             const f = args[++i];
-            if (f !== 'text' && f !== 'json' && f !== 'markdown' && f !== 'sarif') {
+            if (f !== 'text' && f !== 'json' && f !== 'markdown' && f !== 'sarif' && f !== 'report') {
                 console.error(`Invalid format: ${f}`);
                 process.exitCode = 2;
                 return;
             }
             format = f;
         }
-        else if (a === '--out') {
+        else if (a === '--out' || a === '--output') {
             out = args[++i];
         }
+        else if (a === '--name') {
+            name = args[++i];
+            if (!name) {
+                console.error('--name requires a value');
+                process.exitCode = 2;
+                return;
+            }
+        }
         else if (a === '--version' || a === '-v') {
             continue;
         }
@@ -128,6 +168,9 @@ async function main() {
     }
     try {
         const result = await scorePaths({ paths, profile, threshold });
+        for (const c of result.ruleCrashes ?? []) {
+            console.error(`warning: rule ${c.rule} crashed on ${c.count} node(s); results may be incomplete`);
+        }
         let body;
         switch (format) {
             case 'json':
@@ -139,8 +182,21 @@ async function main() {
             case 'sarif':
                 body = formatSarif(result);
                 break;
+            case 'report':
+                body = formatReport(result, {
+                    date: new Date().toISOString().slice(0, 10),
+                    version: await packageVersion(),
+                    suiteName: name ?? suiteName(paths),
+                    customThreshold: threshold !== undefined,
+                    readSource: makeSourceReader(paths, process.cwd(), result.baseDir),
+                });
+                break;
             default:
-                body = formatText(result);
+                body = formatText(result, {
+                    suiteName: name ?? suiteName(paths),
+                    version: await packageVersion(),
+                    customThreshold: threshold !== undefined,
+                });
         }
         if (out) {
             fs.mkdirSync(path.dirname(path.resolve(out)), { recursive: true });

package/dist/confidence.d.ts

@@ -0,0 +1,30 @@
+import type { Finding, Severity } from './types.js';
+/**
+ * How sure we are that a finding is a real problem a reader would agree
+ * with. Derived from a hand check of 20 findings (QAG-288). Display and
+ * ranking only: it never changes the score.
+ *
+ * - high:        mechanical, almost never a false positive.
+ * - medium:      usually right, but there are legitimate uses.
+ * - needs-check: the scorer could not resolve something; a human should look.
+ */
+export type Confidence = 'high' | 'medium' | 'needs-check';
+export declare const CONFIDENCE_BY_RULE: Record<string, Confidence>;
+/** Display names (the Confidence values themselves never change). */
+export declare const CONFIDENCE_LABEL: Record<Confidence, string>;
+/** Rules not listed above (and not needs-check) are medium. */
+export declare const DEFAULT_CONFIDENCE: Confidence;
+/**
+ * A report-only finding normally means "ratio-scored" (the locator rules:
+ * they feed the locators dimension through native/raw counts, not penalty
+ * units) and is an ordinary finding. Any OTHER report-only finding is one
+ * the scorer deliberately left out of the score (e.g. an assertion helper it
+ * could not resolve), so it is a "needs a human check".
+ */
+export declare function isNeedsCheck(f: Pick<Finding, 'reportOnly' | 'dimension'>): boolean;
+export declare function confidenceOf(f: Pick<Finding, 'rule' | 'reportOnly' | 'dimension'>): Confidence;
+export declare function severityRank(s: Severity): number;
+export declare function confidenceRank(c: Confidence): number;
+/** Severity, then confidence, then file, line, rule: total and stable. */
+export declare function compareFindings(a: Finding, b: Finding): number;
+export declare function sortFindings(findings: readonly Finding[]): Finding[];

package/dist/confidence.js

@@ -0,0 +1,67 @@
+export const CONFIDENCE_BY_RULE = {
+    'playwright/missing-playwright-await': 'high',
+    'playwright/no-networkidle': 'high',
+    'playwright/no-wait-for-timeout': 'high',
+    'playwright/no-focused-test': 'high',
+    'playwright/no-page-pause': 'high',
+    'playwright-score/parse-error': 'high',
+    'pwscore/no-unawaited-action': 'high',
+    'pwscore/no-unawaited-expect-value': 'high',
+    'pwscore/no-weak-assertion': 'high',
+    'pwscore/no-fixme-test': 'high',
+    'pwscore/no-excessive-timeout': 'high',
+    'pwscore/no-swallowed-assertion': 'medium',
+    'pwscore/no-shared-mutable-state': 'medium',
+    'pwscore/no-self-fulfilling-fallback': 'needs-check',
+    'pwscore/no-brittle-selector': 'needs-check',
+    'pwscore/serial-mode-dependency': 'needs-check',
+    'playwright-score/assertion-unverified': 'needs-check',
+    'playwright/no-force-option': 'medium',
+    'playwright/no-wait-for-selector': 'medium',
+    'pwscore/no-timer-sleep': 'medium',
+    'pwscore/no-trivial-assertion': 'medium',
+    'playwright/no-conditional-in-test': 'medium',
+    'playwright/no-conditional-expect': 'medium',
+};
+/** Display names (the Confidence values themselves never change). */
+export const CONFIDENCE_LABEL = {
+    high: 'Definite',
+    medium: 'Likely',
+    'needs-check': 'Needs a human check',
+};
+/** Rules not listed above (and not needs-check) are medium. */
+export const DEFAULT_CONFIDENCE = 'medium';
+const CONFIDENCE_RANK = { high: 0, medium: 1, 'needs-check': 2 };
+const SEVERITY_RANK = { error: 0, warning: 1, info: 2 };
+/**
+ * A report-only finding normally means "ratio-scored" (the locator rules:
+ * they feed the locators dimension through native/raw counts, not penalty
+ * units) and is an ordinary finding. Any OTHER report-only finding is one
+ * the scorer deliberately left out of the score (e.g. an assertion helper it
+ * could not resolve), so it is a "needs a human check".
+ */
+export function isNeedsCheck(f) {
+    return f.reportOnly === true && f.dimension !== 'locators';
+}
+export function confidenceOf(f) {
+    if (isNeedsCheck(f))
+        return 'needs-check';
+    return CONFIDENCE_BY_RULE[f.rule] ?? DEFAULT_CONFIDENCE;
+}
+export function severityRank(s) {
+    return SEVERITY_RANK[s] ?? 3;
+}
+export function confidenceRank(c) {
+    return CONFIDENCE_RANK[c];
+}
+/** Severity, then confidence, then file, line, rule: total and stable. */
+export function compareFindings(a, b) {
+    return (severityRank(a.severity) - severityRank(b.severity) ||
+        confidenceRank(confidenceOf(a)) - confidenceRank(confidenceOf(b)) ||
+        a.file.localeCompare(b.file) ||
+        (a.line ?? 0) - (b.line ?? 0) ||
+        a.rule.localeCompare(b.rule));
+}
+export function sortFindings(findings) {
+    return [...findings].sort(compareFindings);
+}

package/dist/eslint-runner.js

@@ -154,6 +154,18 @@ function buildConfig(assertFunctionNames, testAliasNames) {
                 'pwscore/no-coordinate-click': 'warn',
                 'pwscore/no-trivial-assertion': 'error',
                 'pwscore/no-soft-assertion-only-test': ['warn', { testAliasNames }],
+                // QAG-287 slice 2b flake-risk rules, one file each under src/rules/.
+                'pwscore/no-unawaited-action': 'error',
+                'pwscore/no-brittle-selector': 'warn',
+                'pwscore/no-shared-mutable-state': ['warn', { testAliasNames }],
+                'pwscore/serial-mode-dependency': ['warn', { testAliasNames }],
+                'pwscore/no-excessive-timeout': ['warn', { testAliasNames }],
+                // QAG-287 slice 2a: tests that would miss a real bug.
+                'pwscore/no-fixme-test': ['warn', { testAliasNames }],
+                'pwscore/no-swallowed-assertion': 'warn',
+                'pwscore/no-weak-assertion': 'error',
+                'pwscore/no-unawaited-expect-value': 'error',
+                'pwscore/no-self-fulfilling-fallback': ['warn', { testAliasNames }],
             },
         },
     ];
@@ -215,6 +227,10 @@ export async function runEslint(options) {
     const results = await eslint.lintFiles(options.files);
     const findings = [];
     for (const result of results) {
+        // pwscore/no-unawaited-expect-value says "missing await" for the same
+        // expect(); upstream prefer-web-first-assertions says something
+        // misleading about it. Keep ours (same line).
+        const unawaitedLines = new Set(result.messages.filter((m) => m.ruleId === 'pwscore/no-unawaited-expect-value').map((m) => m.line));
         const absFile = path.resolve(result.filePath);
         const file = path.relative(cwd, absFile) || path.basename(absFile);
         for (const msg of result.messages) {
@@ -262,6 +278,8 @@ export async function runEslint(options) {
             }
             if (!msg.ruleId)
                 continue;
+            if (msg.ruleId === 'playwright/prefer-web-first-assertions' && unawaitedLines.has(msg.line))
+                continue;
             const mapping = mapRule(msg.ruleId);
             findings.push({
                 rule: msg.ruleId,

package/dist/finding-triage.d.ts

@@ -0,0 +1,13 @@
+import type { Finding } from './types.js';
+export declare const ASSERTION_UNVERIFIED_RULE = "playwright-score/assertion-unverified";
+export interface TriageContext {
+    /** Directory the findings' `file` paths are relative to. */
+    filesBase: string;
+    /** Scored spec files, absolute path -> source. */
+    sources: Map<string, string>;
+    /** Locally imported dependency files (page objects, fixtures), absolute path -> source. */
+    importedFiles: Map<string, string>;
+    /** Directory relative imports may not climb out of (defaults to filesBase). */
+    importBoundary?: string;
+}
+export declare function triageFindings(findings: Finding[], ctx: TriageContext): Finding[];