@qaguardian/playwright-score 2.2.0 → 2.3.0

package/CHANGELOG.md

@@ -10,6 +10,92 @@ parsing accuracy, not grading policy — can still move a suite's score
 without itself being a new model version; 2.1.0 below is exactly that
 case, and says so explicitly.
 
+## 2.3.0 — 2026-11-14
+
+Scoring weights and the scoring model (v4) are unchanged, but **scores can
+fall**: this release adds ten rules that catch tests which would miss a real
+bug or flake, and suites that use those patterns now lose points. Nothing
+gets more lenient. Two of the new rules are report-only (shown in the report,
+no effect on the score): `pwscore/no-self-fulfilling-fallback` (info, no real
+suite hit it yet, validated on fixtures only) and `pwscore/no-brittle-selector`
+(its cost is already counted by the raw-locator ratio), plus
+`pwscore/serial-mode-dependency` (info).
+
+New rules, flake risk (QAG-287 slice 2b):
+
+- `pwscore/no-unawaited-action` (error, hygiene): Playwright action or navigation on a Page/Locator as a bare statement without `await`.
+- `pwscore/no-brittle-selector` (warning, report-only): XPath, `:nth-child`, 4+ combinator CSS chains, generated class names, positional `.first()/.nth()/.last()` on a raw `.locator()`.
+- `pwscore/no-shared-mutable-state` (warning, structure) and report-only `pwscore/serial-mode-dependency` (info): state written in one test and read in another. Not flagged when the writing and the reading test both sit in a serial describe (`test.describe.serial` or `test.describe.configure({ mode: 'serial' })`, at file or describe level), wherever the variable is declared; flagged when either test is outside serial scope. Plus a report-only note for serial describe blocks.
+- `pwscore/no-excessive-timeout` (warning, hygiene): option timeouts over 60s (exactly 60s is not flagged), `test.setTimeout` >= 5 min, unconditional `test.slow()`.
+
+New rules, tests that would miss a real bug (QAG-287 slice 2a):
+
+- `pwscore/no-fixme-test` (structure, warning): unconditional `test.fixme('t', fn)`, `test.describe.fixme(...)`, in-body `test.fixme()` and `test.fixme(true, ...)`. `test.fixme(condition, reason)` stays clean.
+- `pwscore/no-swallowed-assertion` (assertions, warning): `expect(...)` inside a `try` whose `catch` neither rethrows, asserts, fails, returns a value, records state nor calls `.skip(...)`; and `await expect(...).catch(() => {})` with the result discarded. Retry loops are not flagged. Warning, not error, because some real suites use deliberate soft checks.
+- `pwscore/no-weak-assertion` (assertions, error): `expect(count).toBeGreaterThanOrEqual(0)`, `expect(<Locator>).toBeTruthy()/toBeDefined()/not.toBeNull()`, `expect(await x.isVisible()).toBeDefined()`.
+- `pwscore/no-unawaited-expect-value` (assertions, error): a Promise-returning Playwright call such as `page.title()` or `locator.innerText()` used without `await` inside `expect()` with a non-web-first matcher. Replaces the misleading `playwright/prefer-web-first-assertions` message on the same line.
+- `pwscore/no-self-fulfilling-fallback` (assertions, info, report-only): a negated state check (or a `catch` of a failed expect) whose fallback fills in or selects the value itself (click-only fallbacks are setup and stay clean), followed by an assertion on the same locator.
+
+Fix (test-factory recognition): tests declared through a factory function (`const t = testWithConfig({...}); t('x', ...)`, in the same or an imported file) are now recognised as tests, removing false `playwright/no-standalone-expect` errors.
+
+New report format (QAG-288), output only:
+
+- New `--format report`: a one-page, self-contained HTML suite report
+  (headline, dimension bars, top findings with code excerpts, flake risk,
+  items needing a human check, next steps). Light/dark, phone width, prints
+  to one or two pages. All source and paths are HTML-escaped. `--output` is
+  accepted as an alias of `--out`. New `--name <suite name>` sets the report
+  headline. Without it the label is the last two path segments (three when the
+  folder is generic, e.g. `e2e` or `tests`); it never includes a home directory
+  or anything above it, and a home directory itself is labelled "Playwright suite".
+- Findings are ranked with a display-only confidence tier (high / medium /
+  needs-check) and every rule has a plain-English title. The text and
+  markdown formats list findings by severity, then confidence, instead of
+  file order.
+- The report states how many tests need a human check (the report-only
+  `playwright-score/assertion-unverified` findings, kept out of the score),
+  groups them by file, and reads "why it matters" in business terms. A suite
+  hard-failed by an unparseable file says so in the headline.
+- Results gain an optional in-memory `baseDir`, used only by the report to
+  read code excerpts. It is never printed and is not part of the JSON, SARIF,
+  markdown or HTML output. A rule without plain-English copy renders with a
+  generic title instead of failing.
+
+Report and terminal polish (QAG-288 slice 3b), output only, scores unchanged:
+
+- The HTML report is now a Summary page plus a Details part. The Summary
+  (score, score by area, top 3 findings as rows without code, next steps) prints
+  on one A4 or Letter page; Details (code excerpts, findings 4-5, the
+  human-check list) starts on page 2. The duplicate "Flake risk" table is gone:
+  each top finding names the file with the most occurrences, and Details lists
+  the most flake-prone files.
+- A line under the score explains it (weighted average of four areas, the
+  grade bands, the pass bar; "your bar" when `--threshold` was passed). "clean"
+  became "have no issues".
+- Confidence reads Definite / Likely / Needs a human check, with one legend
+  line and neutral tags (fixes the 3.95:1 contrast failure of the amber tag).
+  The four areas are named Reliability, Checks, Resilience to UI changes and
+  Maintainability (the JSON keys are unchanged).
+- A suite with a syntax error or no tests shows "Not scored" instead of
+  "0/100 F", with no zero bars and consistent counts.
+- Rule copy says "can" instead of "will" for outcomes static analysis cannot
+  prove. The weakest area gets a slot in the top 3 when it scores under 75.
+- A rule that throws while scoring no longer vanishes silently: the run still
+  finishes, the CLI prints `warning: rule <id> crashed on <n> node(s); results
+  may be incomplete` to stderr, the text, markdown and report formats say "Some
+  checks could not run on this suite; results may be incomplete.", and the JSON
+  result gains `ruleCrashes: [{ rule, count }]`, only when a rule crashed (JSON
+  of a clean run is unchanged).
+- A suite with no tests no longer lists the working directory as an issue, and
+  it is "Not scored" rather than "0/100 F".
+- Without `--name`, the suite name falls back to the last two or three path
+  segments for a folder (never above the home directory), so
+  `e2e-tests/playwright` is no longer just `playwright`.
+- Terminal output (`--format text`) is grouped by rule, worst first, with a
+  count per rule, relative paths, at most three locations per rule (`+ N more`)
+  and at most ten rules; it opens with the verdict and repeats it on the last
+  line. `--format json`, `sarif` and `markdown` are byte-identical to before.
+
 ## 2.2.0 — 2026-11-14
 
 False-positive fixes found by hand-checking error-level findings on public

package/METHODOLOGY.md

@@ -126,6 +126,20 @@ The exact ESLint rules that produce findings are enabled explicitly in
 eslint-plugin-playwright's `recommended` set at install time, so identical
 code scores identically regardless of which plugin version npm resolves.
 
+### Tests that would miss a real bug
+
+Five rules (QAG-287) target tests that pass whatever the app does. Each
+matches one exact syntactic shape and stays silent when unsure.
+
+| Rule | Dimension, severity | Exact shape |
+|---|---|---|
+| `pwscore/no-fixme-test` | structure, warning | `test.fixme('t', fn)` / `test.describe.fixme(...)` declarations; in-body `test.fixme()` or `test.fixme(true, ...)`. Conditional `test.fixme(cond, 'why')` is clean. |
+| `pwscore/no-swallowed-assertion` | assertions, warning | `expect(...)` in a `try` whose `catch` has no throw, expect, fail-ish call, returned value or assignment (not inside a loop); or `await expect(...).catch(fn)` as a statement where `fn` does not throw. |
+| `pwscore/no-weak-assertion` | assertions, error | `.count()`/`.length` `toBeGreaterThanOrEqual(0)`; a syntactic Locator (`page.locator`, `getBy*`, or a const from one) asserted `toBeTruthy`/`toBeDefined`/`not.toBeNull`; `isVisible()`-style and `count()` results asserted `toBeDefined`. `toBeTruthy()` on other values is not flagged. |
+| `pwscore/no-unawaited-expect-value` | assertions, error | `expect(<promise call>)` without `await`, receiver syntactically a Page/Locator, matcher not web-first (also an un-awaited promise call as the expected value of `toBe`/`toEqual`). `.resolves`/`.rejects` are exempt. |
+| `pwscore/no-self-fulfilling-fallback` | assertions, info, report-only (no score effect; no real-suite hits, validated on fixtures only) | Inside a test: `if (!check) { actions }`, the `else` of `if (check) {...} else { actions }`, or `try { expect/check } catch { actions }`, where the actions include `fill`/`type`/`pressSequentially`/`selectOption`/`setInputFiles` (click-only and key-press fallbacks are the usual "make sure the menu is open" setup and are not flagged), followed in the same block by an `expect` on a locator shared with the check or the actions. Positive-branch actions (dismiss a banner if visible) are never flagged. |
+
+
 ---
 
 ## Why v4
@@ -190,3 +204,30 @@ real inputs:
    diluted `findings/SLOC` and raised the score without fixing anything.
    A per-test ratio has no such lever — padding with fake tests costs
    assertions coverage instead.
+
+---
+
+## Flake-risk rules (unreleased)
+
+Five static checks for the patterns that make a test pass or fail by luck.
+Each is syntactic and conservative: when the receiver or value cannot be
+read from the source, nothing is reported.
+
+| Rule | Exact shape | Severity / dimension |
+|---|---|---|
+| `pwscore/no-unawaited-action` | An expression statement that is exactly `<receiver>.<action>(...)` inside an `async` function, where the receiver is `page`, a `.page` property, a `page.locator/getBy*/first/nth/filter` chain, or a `const` initialised from one, and the action is goto/reload/goBack/goForward/click/dblclick/fill/clear/press/type/check/uncheck/setChecked/selectOption/hover/focus/tap/dragTo/setInputFiles/setContent/waitFor*. Not matched: await, return, void, assignment, argument, `Promise.all`, `.catch()`/`.then()` chains, other receivers. Chosen over eslint-plugin-playwright's `missing-playwright-await` `includePageLocatorMethods` option because that one matches the method name on any receiver. | error / hygiene |
+| `pwscore/no-brittle-selector` | A fully static selector string passed to `.locator()`, `.frameLocator()` or a legacy selector method on a Page/Locator that is XPath (`xpath=`, `//`, `(//`), uses `:nth-child`/`:nth-of-type`, is a plain CSS chain with 4 or more combinators, or contains a generated class (`.css-1a2b3c`, `.sc-bdfBwQ`, `.jss12`, `.MuiBox-root-12`); plus `.nth()/.first()/.last()` directly on a raw `.locator('<string>')` (test-id, `role=` and `text=` strings exempt). | warning, **report-only** / locators |
+| `pwscore/no-shared-mutable-state` | A module- or describe-level `let`/`var` (or `const` object/array/Map/Set) that one `test()` body assigns or mutates and another `test()` body reads without writing it. Not matched: writes only in hooks, any `beforeEach` write, reader that also writes. | warning / structure |
+| `pwscore/serial-mode-dependency` | `test.describe.configure({ mode: 'serial' })` or `test.describe.serial(...)`. | info, **report-only** / structure |
+| `pwscore/no-excessive-timeout` | `{ timeout: N }` with N > 60,000 (strictly more than one minute) on an action, wait or matcher call; `test.setTimeout(N)`/`testInfo.setTimeout(N)` with N >= 300,000; `test.slow()` with no arguments directly at file or describe level. Literals and simple arithmetic only (`2 * 60 * 1000`, `60_000`). Config files are not read. | warning / hygiene |
+
+`no-brittle-selector` is report-only on purpose: the locators dimension is a
+native/raw ratio that already counts every raw `.locator()` and legacy
+selector call, so scoring the same call again would charge it twice. The
+finding is named evidence for why that raw call is fragile.
+
+**Test factories.** A function whose `return` is `<test>.extend(...)`, a known
+test alias, or a call of another such function (same file or imported through
+relative paths) is a test factory; `const t = factory(...)` makes `t` a test
+alias, so `t('name', ...)` is a test declaration and its `expect` calls are not
+"standalone".

package/README.md

@@ -99,7 +99,7 @@ npx playwright-score ./flow.spec.ts --format json --out report.json
 |---|---|
 | `--profile standard` | Scoring profile (default, and only, profile) |
 | `--threshold <n>` | Pass bar 0–100 |
-| `--format text\|json\|markdown\|sarif` | Output format |
+| `--format text\|json\|markdown\|sarif\|report` | Output format (`report` is a self-contained one-page HTML suite report) |
 | `--out <file>` | Write report to file |
 | `--version` | Print version |
 

package/dist/cli.js

@@ -9,6 +9,9 @@ import { formatJson } from './formatters/json.js';
 import { formatText } from './formatters/text.js';
 import { formatMarkdown } from './formatters/markdown.js';
 import { formatSarif } from './formatters/sarif.js';
+import { formatReport } from './formatters/report.js';
+import { makeSourceReader } from './report-source.js';
+import { suiteLabel } from './suite-name.js';
 function printHelp() {
     console.log(`@qaguardian/playwright-score — deterministic Playwright spec quality score
 
@@ -21,8 +24,12 @@ Options:
   --profile <standard>             Scoring profile (default, and only
                                    supported value, as of 2.0.0: standard).
   --threshold <n>                 Pass threshold 0-100 (default: 80)
-  --format <text|json|markdown|sarif>  Output format (default: text)
-  --out <file>                    Write report to file
+  --format <text|json|markdown|sarif|report>
+                                  Output format (default: text). report is a
+                                  self-contained one-page HTML suite report.
+  --out, --output <file>          Write report to file
+  --name <suite name>             Suite name for the report headline
+                                  (default: the directory basename)
   --version                       Print version
   --help                          Show help
 
@@ -43,6 +50,30 @@ Methodology: METHODOLOGY.md (scoring model v4)
 // size (64KB on macOS/Linux). Setting process.exitCode and returning lets
 // Node exit naturally once pending I/O actually completes, which is the
 // standard fix for this class of bug.
+async function packageVersion() {
+    try {
+        const { createRequire } = await import('node:module');
+        const require = createRequire(import.meta.url);
+        return require('../package.json').version ?? 'unknown';
+    }
+    catch {
+        return 'unknown';
+    }
+}
+// Never an absolute local path: see suite-name.ts.
+function suiteName(paths) {
+    const names = paths.slice(0, 2).map((p) => {
+        let isDir = false;
+        try {
+            isDir = fs.statSync(p).isDirectory();
+        }
+        catch {
+            /* missing path: scorePaths reports it */
+        }
+        return suiteLabel(p, isDir);
+    });
+    return names.join(', ') + (paths.length > 2 ? ` and ${paths.length - 2} more` : '');
+}
 async function main() {
     const args = process.argv.slice(2);
     if (args.includes('--version') || args.includes('-v')) {
@@ -67,6 +98,7 @@ async function main() {
     let threshold;
     let format = 'text';
     let out;
+    let name;
     const paths = [];
     for (let i = 0; i < args.length; i++) {
         const a = args[i];
@@ -99,16 +131,24 @@ async function main() {
         }
         else if (a === '--format') {
             const f = args[++i];
-            if (f !== 'text' && f !== 'json' && f !== 'markdown' && f !== 'sarif') {
+            if (f !== 'text' && f !== 'json' && f !== 'markdown' && f !== 'sarif' && f !== 'report') {
                 console.error(`Invalid format: ${f}`);
                 process.exitCode = 2;
                 return;
             }
             format = f;
         }
-        else if (a === '--out') {
+        else if (a === '--out' || a === '--output') {
             out = args[++i];
         }
+        else if (a === '--name') {
+            name = args[++i];
+            if (!name) {
+                console.error('--name requires a value');
+                process.exitCode = 2;
+                return;
+            }
+        }
         else if (a === '--version' || a === '-v') {
             continue;
         }
@@ -128,6 +168,9 @@ async function main() {
     }
     try {
         const result = await scorePaths({ paths, profile, threshold });
+        for (const c of result.ruleCrashes ?? []) {
+            console.error(`warning: rule ${c.rule} crashed on ${c.count} node(s); results may be incomplete`);
+        }
         let body;
         switch (format) {
             case 'json':
@@ -139,8 +182,21 @@ async function main() {
             case 'sarif':
                 body = formatSarif(result);
                 break;
+            case 'report':
+                body = formatReport(result, {
+                    date: new Date().toISOString().slice(0, 10),
+                    version: await packageVersion(),
+                    suiteName: name ?? suiteName(paths),
+                    customThreshold: threshold !== undefined,
+                    readSource: makeSourceReader(paths, process.cwd(), result.baseDir),
+                });
+                break;
             default:
-                body = formatText(result);
+                body = formatText(result, {
+                    suiteName: name ?? suiteName(paths),
+                    version: await packageVersion(),
+                    customThreshold: threshold !== undefined,
+                });
         }
         if (out) {
             fs.mkdirSync(path.dirname(path.resolve(out)), { recursive: true });

package/dist/confidence.d.ts

@@ -0,0 +1,30 @@
+import type { Finding, Severity } from './types.js';
+/**
+ * How sure we are that a finding is a real problem a reader would agree
+ * with. Derived from a hand check of 20 findings (QAG-288). Display and
+ * ranking only: it never changes the score.
+ *
+ * - high:        mechanical, almost never a false positive.
+ * - medium:      usually right, but there are legitimate uses.
+ * - needs-check: the scorer could not resolve something; a human should look.
+ */
+export type Confidence = 'high' | 'medium' | 'needs-check';
+export declare const CONFIDENCE_BY_RULE: Record<string, Confidence>;
+/** Display names (the Confidence values themselves never change). */
+export declare const CONFIDENCE_LABEL: Record<Confidence, string>;
+/** Rules not listed above (and not needs-check) are medium. */
+export declare const DEFAULT_CONFIDENCE: Confidence;
+/**
+ * A report-only finding normally means "ratio-scored" (the locator rules:
+ * they feed the locators dimension through native/raw counts, not penalty
+ * units) and is an ordinary finding. Any OTHER report-only finding is one
+ * the scorer deliberately left out of the score (e.g. an assertion helper it
+ * could not resolve), so it is a "needs a human check".
+ */
+export declare function isNeedsCheck(f: Pick<Finding, 'reportOnly' | 'dimension'>): boolean;
+export declare function confidenceOf(f: Pick<Finding, 'rule' | 'reportOnly' | 'dimension'>): Confidence;
+export declare function severityRank(s: Severity): number;
+export declare function confidenceRank(c: Confidence): number;
+/** Severity, then confidence, then file, line, rule: total and stable. */
+export declare function compareFindings(a: Finding, b: Finding): number;
+export declare function sortFindings(findings: readonly Finding[]): Finding[];

package/dist/confidence.js

@@ -0,0 +1,67 @@
+export const CONFIDENCE_BY_RULE = {
+    'playwright/missing-playwright-await': 'high',
+    'playwright/no-networkidle': 'high',
+    'playwright/no-wait-for-timeout': 'high',
+    'playwright/no-focused-test': 'high',
+    'playwright/no-page-pause': 'high',
+    'playwright-score/parse-error': 'high',
+    'pwscore/no-unawaited-action': 'high',
+    'pwscore/no-unawaited-expect-value': 'high',
+    'pwscore/no-weak-assertion': 'high',
+    'pwscore/no-fixme-test': 'high',
+    'pwscore/no-excessive-timeout': 'high',
+    'pwscore/no-swallowed-assertion': 'medium',
+    'pwscore/no-shared-mutable-state': 'medium',
+    'pwscore/no-self-fulfilling-fallback': 'needs-check',
+    'pwscore/no-brittle-selector': 'needs-check',
+    'pwscore/serial-mode-dependency': 'needs-check',
+    'playwright-score/assertion-unverified': 'needs-check',
+    'playwright/no-force-option': 'medium',
+    'playwright/no-wait-for-selector': 'medium',
+    'pwscore/no-timer-sleep': 'medium',
+    'pwscore/no-trivial-assertion': 'medium',
+    'playwright/no-conditional-in-test': 'medium',
+    'playwright/no-conditional-expect': 'medium',
+};
+/** Display names (the Confidence values themselves never change). */
+export const CONFIDENCE_LABEL = {
+    high: 'Definite',
+    medium: 'Likely',
+    'needs-check': 'Needs a human check',
+};
+/** Rules not listed above (and not needs-check) are medium. */
+export const DEFAULT_CONFIDENCE = 'medium';
+const CONFIDENCE_RANK = { high: 0, medium: 1, 'needs-check': 2 };
+const SEVERITY_RANK = { error: 0, warning: 1, info: 2 };
+/**
+ * A report-only finding normally means "ratio-scored" (the locator rules:
+ * they feed the locators dimension through native/raw counts, not penalty
+ * units) and is an ordinary finding. Any OTHER report-only finding is one
+ * the scorer deliberately left out of the score (e.g. an assertion helper it
+ * could not resolve), so it is a "needs a human check".
+ */
+export function isNeedsCheck(f) {
+    return f.reportOnly === true && f.dimension !== 'locators';
+}
+export function confidenceOf(f) {
+    if (isNeedsCheck(f))
+        return 'needs-check';
+    return CONFIDENCE_BY_RULE[f.rule] ?? DEFAULT_CONFIDENCE;
+}
+export function severityRank(s) {
+    return SEVERITY_RANK[s] ?? 3;
+}
+export function confidenceRank(c) {
+    return CONFIDENCE_RANK[c];
+}
+/** Severity, then confidence, then file, line, rule: total and stable. */
+export function compareFindings(a, b) {
+    return (severityRank(a.severity) - severityRank(b.severity) ||
+        confidenceRank(confidenceOf(a)) - confidenceRank(confidenceOf(b)) ||
+        a.file.localeCompare(b.file) ||
+        (a.line ?? 0) - (b.line ?? 0) ||
+        a.rule.localeCompare(b.rule));
+}
+export function sortFindings(findings) {
+    return [...findings].sort(compareFindings);
+}

package/dist/eslint-runner.js

@@ -154,6 +154,18 @@ function buildConfig(assertFunctionNames, testAliasNames) {
                 'pwscore/no-coordinate-click': 'warn',
                 'pwscore/no-trivial-assertion': 'error',
                 'pwscore/no-soft-assertion-only-test': ['warn', { testAliasNames }],
+                // QAG-287 slice 2b flake-risk rules, one file each under src/rules/.
+                'pwscore/no-unawaited-action': 'error',
+                'pwscore/no-brittle-selector': 'warn',
+                'pwscore/no-shared-mutable-state': ['warn', { testAliasNames }],
+                'pwscore/serial-mode-dependency': ['warn', { testAliasNames }],
+                'pwscore/no-excessive-timeout': ['warn', { testAliasNames }],
+                // QAG-287 slice 2a: tests that would miss a real bug.
+                'pwscore/no-fixme-test': ['warn', { testAliasNames }],
+                'pwscore/no-swallowed-assertion': 'warn',
+                'pwscore/no-weak-assertion': 'error',
+                'pwscore/no-unawaited-expect-value': 'error',
+                'pwscore/no-self-fulfilling-fallback': ['warn', { testAliasNames }],
             },
         },
     ];
@@ -215,6 +227,10 @@ export async function runEslint(options) {
     const results = await eslint.lintFiles(options.files);
     const findings = [];
     for (const result of results) {
+        // pwscore/no-unawaited-expect-value says "missing await" for the same
+        // expect(); upstream prefer-web-first-assertions says something
+        // misleading about it. Keep ours (same line).
+        const unawaitedLines = new Set(result.messages.filter((m) => m.ruleId === 'pwscore/no-unawaited-expect-value').map((m) => m.line));
         const absFile = path.resolve(result.filePath);
         const file = path.relative(cwd, absFile) || path.basename(absFile);
         for (const msg of result.messages) {
@@ -262,6 +278,8 @@ export async function runEslint(options) {
             }
             if (!msg.ruleId)
                 continue;
+            if (msg.ruleId === 'playwright/prefer-web-first-assertions' && unawaitedLines.has(msg.line))
+                continue;
             const mapping = mapRule(msg.ruleId);
             findings.push({
                 rule: msg.ruleId,

package/dist/formatters/json.js

@@ -1,3 +1,6 @@
 export function formatJson(result) {
-    return JSON.stringify(result, null, 2);
+    // baseDir is an absolute local path used only by the report's excerpt
+    // reader; it must never reach a shareable artifact.
+    const { baseDir: _baseDir, ...shareable } = result;
+    return JSON.stringify(shareable, null, 2);
 }

package/dist/formatters/markdown.d.ts

@@ -1,2 +1,4 @@
 import type { ScoreResult } from '../types.js';
+/** Make a value safe inside a Markdown table cell: no pipe or newline can end the cell or row. */
+export declare function escapeCell(v: unknown): string;
 export declare function formatMarkdown(result: ScoreResult): string;

package/dist/formatters/markdown.js

@@ -1,3 +1,9 @@
+import { INCOMPLETE_NOTE, notScoredState } from '../report-model.js';
+import { sortFindings } from '../confidence.js';
+/** Make a value safe inside a Markdown table cell: no pipe or newline can end the cell or row. */
+export function escapeCell(v) {
+    return String(v).replace(/\|/g, '\\|').replace(/\r?\n|\r/g, ' ');
+}
 export function formatMarkdown(result) {
     const lines = [];
     lines.push(`# Playwright Spec Score (\`${result.scoreVersion}\`)`);
@@ -9,18 +15,23 @@ export function formatMarkdown(result) {
     for (const [k, v] of Object.entries(result.dimensions)) {
         if (v === undefined)
             continue;
-        lines.push(`| ${k} | ${v} |`);
+        lines.push(`| ${escapeCell(k)} | ${escapeCell(v)} |`);
     }
     lines.push('');
     lines.push(`Files: ${result.summary.files} · Tests: ${result.summary.tests} · SLOC: ${result.summary.sloc} · Findings: ${result.summary.findings}` +
         (result.summary.unverifiedTests ? ` · Unverified assertions: ${result.summary.unverifiedTests}` : ''));
-    if (result.findings.length) {
+    if (result.ruleCrashes?.length) {
+        lines.push('');
+        lines.push(INCOMPLETE_NOTE);
+    }
+    // 'no-tests' hard-fail findings carry the absolute working directory as their file.
+    if (result.findings.length && notScoredState(result) !== 'no-tests') {
         lines.push('');
         lines.push('## Findings');
         lines.push('');
-        for (const f of result.findings.slice(0, 40)) {
+        for (const f of sortFindings(result.findings).slice(0, 40)) {
             const loc = f.line ? `:${f.line}` : '';
-            lines.push(`- **${f.severity}** \`${f.rule}\` \`${f.file}${loc}\` — ${f.message}`);
+            lines.push(`- **${f.severity}** \`${f.rule}\` \`${escapeCell(f.file)}${loc}\` — ${escapeCell(f.message)}`);
         }
     }
     if (result.skippedFiles?.length) {

package/dist/formatters/report-css.d.ts

@@ -0,0 +1,2 @@
+/** Inline stylesheet for the suite report: light/dark, phone width, Letter/A4 print. */
+export declare const REPORT_CSS = "\n:root{--bg:#fff;--fg:#1a1d23;--muted:#5b6472;--line:#e3e6eb;--card:#f6f7f9;--accent:#2f5bea;--hit:#fff1c2;--hitfg:#1a1d23;--good:#1a7f4b;--warn:#b26a00;--bad:#c62828;--track:#e8ebf0}\n@media (prefers-color-scheme:dark){:root{--bg:#14161a;--fg:#e8eaee;--muted:#9aa3b2;--line:#2a2e36;--card:#1c1f25;--accent:#7c9bff;--hit:#4a3d10;--hitfg:#fff;--good:#4cc38a;--warn:#f0a63a;--bad:#ff6b6b;--track:#2a2e36}}\n*{box-sizing:border-box}\nbody{margin:0;background:var(--bg);color:var(--fg);font:15px/1.5 system-ui,-apple-system,\"Segoe UI\",Roboto,sans-serif}\nmain{max-width:880px;margin:0 auto;padding:20px 20px 40px}\nheader{border-bottom:1px solid var(--line);padding-bottom:14px}\n.hd{display:flex;gap:24px;align-items:center;flex-wrap:wrap}\n.eyebrow{margin:0;font-size:12px;letter-spacing:.06em;text-transform:uppercase;color:var(--muted)}\n.suite{margin:2px 0 10px;font-size:15px;font-weight:600;overflow-wrap:anywhere}\n.explain,.check{font-size:14px;margin:4px 0}\n.check a{color:var(--fg)}\n.sr-only{position:absolute;width:1px;height:1px;overflow:hidden;clip:rect(0 0 0 0);white-space:nowrap}\nh1{font-size:22px;margin:0 0 4px}\nh2{font-size:17px;margin:20px 0 8px;padding-bottom:6px;border-bottom:1px solid var(--line)}\nh3{font-size:15px;margin:0 0 4px}\np{margin:4px 0}\ncode,pre{font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;font-size:12.5px}\ncode{overflow-wrap:anywhere}\n.muted{color:var(--muted);font-size:13px}\n.meta{color:var(--muted);font-size:13px;margin-top:8px}\n.sub{font-weight:600;margin:12px 0 4px}\n.score{display:flex;align-items:baseline;gap:2px;padding:10px 18px;border-radius:12px;background:var(--card);border:1px solid var(--line);border-left:4px solid var(--muted)}\n.score .big{font-size:46px;font-weight:700;line-height:1}\n.score .of{color:var(--muted);font-size:16px}\n.score .grade{margin-left:12px;font-size:28px;font-weight:700}\n.g-A,.g-B{border-left-color:var(--good)}.g-A .grade,.g-B .grade{color:var(--good)}\n.g-C{border-left-color:var(--warn)}.g-C .grade{color:var(--warn)}\n.g-D,.g-F{border-left-color:var(--bad)}.g-D .grade,.g-F .grade{color:var(--bad)}\n.notscored{border-left-color:var(--bad)}.notscored .grade{margin-left:12px;font-size:16px;color:var(--fg)}\n@media (min-width:720px){.dims{display:grid;grid-template-columns:1fr 1fr;column-gap:28px}}\n.dim{margin:8px 0}.dimhead{display:flex;justify-content:space-between}\n.bar{height:10px;background:var(--track);border-radius:5px;overflow:hidden;margin:4px 0}\n.bar span{display:block;height:100%;background:var(--accent);-webkit-print-color-adjust:exact;print-color-adjust:exact}\n.finding{background:var(--card);border:1px solid var(--line);border-left:4px solid var(--warn);border-radius:8px;padding:12px 14px;margin:12px 0;break-inside:avoid}\n.finding.sev-error{border-left-color:var(--bad)}\n.rank{display:inline-block;width:22px;height:22px;line-height:22px;text-align:center;border-radius:50%;background:var(--fg);color:var(--bg);font-size:12px;margin-right:8px}\n.tag{font-size:11px;font-weight:600;border:1px solid var(--line);border-radius:10px;padding:1px 8px;margin-left:6px;color:var(--fg);border-color:var(--muted);white-space:nowrap}\n.where{margin:2px 0 6px}\nol.top3{list-style:none;padding:0;margin:8px 0}\nli.top{display:flex;gap:10px;align-items:flex-start;margin:8px 0;break-inside:avoid}\nli.top .rank{flex:none;margin-top:1px}\nli.top h3{margin:0 0 2px}li.top p{margin:2px 0}li.top .why{font-size:14px}\n.legend{margin:4px 0 8px}\npre.code{margin:6px 0;padding:8px 0;background:var(--bg);border:1px solid var(--line);border-radius:6px;overflow-x:auto}\n.ln{display:block;white-space:pre;padding:0 10px}\n.ln i{display:inline-block;width:3.2em;color:var(--muted);font-style:normal;user-select:none}\n.ln.hit{box-shadow:inset 3px 0 0 var(--bad);background:var(--hit);color:var(--hitfg);-webkit-print-color-adjust:exact;print-color-adjust:exact}\n.why{margin:6px 0}\nul.plain{list-style:none;padding:0;margin:6px 0}ul.plain li{padding:5px 0;border-bottom:1px solid var(--line);overflow-wrap:anywhere}\n.note{background:var(--card);border-radius:6px;padding:8px 12px}\nol{padding-left:22px}ol li{margin:6px 0}\nfooter{margin-top:32px;padding-top:12px;border-top:1px solid var(--line);color:var(--muted);font-size:12px}\n@media (max-width:520px){main{padding:16px 12px}.score .big{font-size:44px}.tag{display:inline-block;margin:4px 0 0}pre.code{font-size:11.5px}}\n@media print{\n@page{margin:14mm}\nbody{font-size:11.5px}main{max-width:none;padding:0}\nh2{margin:12px 0 6px;break-after:avoid}\n.summary .dims{display:grid;grid-template-columns:1fr 1fr;column-gap:16px}\n.summary pre.code{display:none}\n.finding{padding:6px 10px;margin:6px 0;break-inside:avoid}\nli.top{margin:6px 0}li.top .why{font-size:inherit}.explain,.check{font-size:inherit}.suite{margin-bottom:8px}p{margin:2px 0}ol li{margin:3px 0}.legend{font-size:11px;margin:2px 0 4px}\n.score .big{font-size:40px}header{padding-bottom:10px}\n.details{break-before:page}\npre.code{overflow:visible;white-space:pre-wrap}\n}\n";

package/dist/formatters/report-css.js

@@ -0,0 +1,68 @@
+/** Inline stylesheet for the suite report: light/dark, phone width, Letter/A4 print. */
+export const REPORT_CSS = `
+:root{--bg:#fff;--fg:#1a1d23;--muted:#5b6472;--line:#e3e6eb;--card:#f6f7f9;--accent:#2f5bea;--hit:#fff1c2;--hitfg:#1a1d23;--good:#1a7f4b;--warn:#b26a00;--bad:#c62828;--track:#e8ebf0}
+@media (prefers-color-scheme:dark){:root{--bg:#14161a;--fg:#e8eaee;--muted:#9aa3b2;--line:#2a2e36;--card:#1c1f25;--accent:#7c9bff;--hit:#4a3d10;--hitfg:#fff;--good:#4cc38a;--warn:#f0a63a;--bad:#ff6b6b;--track:#2a2e36}}
+*{box-sizing:border-box}
+body{margin:0;background:var(--bg);color:var(--fg);font:15px/1.5 system-ui,-apple-system,"Segoe UI",Roboto,sans-serif}
+main{max-width:880px;margin:0 auto;padding:20px 20px 40px}
+header{border-bottom:1px solid var(--line);padding-bottom:14px}
+.hd{display:flex;gap:24px;align-items:center;flex-wrap:wrap}
+.eyebrow{margin:0;font-size:12px;letter-spacing:.06em;text-transform:uppercase;color:var(--muted)}
+.suite{margin:2px 0 10px;font-size:15px;font-weight:600;overflow-wrap:anywhere}
+.explain,.check{font-size:14px;margin:4px 0}
+.check a{color:var(--fg)}
+.sr-only{position:absolute;width:1px;height:1px;overflow:hidden;clip:rect(0 0 0 0);white-space:nowrap}
+h1{font-size:22px;margin:0 0 4px}
+h2{font-size:17px;margin:20px 0 8px;padding-bottom:6px;border-bottom:1px solid var(--line)}
+h3{font-size:15px;margin:0 0 4px}
+p{margin:4px 0}
+code,pre{font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;font-size:12.5px}
+code{overflow-wrap:anywhere}
+.muted{color:var(--muted);font-size:13px}
+.meta{color:var(--muted);font-size:13px;margin-top:8px}
+.sub{font-weight:600;margin:12px 0 4px}
+.score{display:flex;align-items:baseline;gap:2px;padding:10px 18px;border-radius:12px;background:var(--card);border:1px solid var(--line);border-left:4px solid var(--muted)}
+.score .big{font-size:46px;font-weight:700;line-height:1}
+.score .of{color:var(--muted);font-size:16px}
+.score .grade{margin-left:12px;font-size:28px;font-weight:700}
+.g-A,.g-B{border-left-color:var(--good)}.g-A .grade,.g-B .grade{color:var(--good)}
+.g-C{border-left-color:var(--warn)}.g-C .grade{color:var(--warn)}
+.g-D,.g-F{border-left-color:var(--bad)}.g-D .grade,.g-F .grade{color:var(--bad)}
+.notscored{border-left-color:var(--bad)}.notscored .grade{margin-left:12px;font-size:16px;color:var(--fg)}
+@media (min-width:720px){.dims{display:grid;grid-template-columns:1fr 1fr;column-gap:28px}}
+.dim{margin:8px 0}.dimhead{display:flex;justify-content:space-between}
+.bar{height:10px;background:var(--track);border-radius:5px;overflow:hidden;margin:4px 0}
+.bar span{display:block;height:100%;background:var(--accent);-webkit-print-color-adjust:exact;print-color-adjust:exact}
+.finding{background:var(--card);border:1px solid var(--line);border-left:4px solid var(--warn);border-radius:8px;padding:12px 14px;margin:12px 0;break-inside:avoid}
+.finding.sev-error{border-left-color:var(--bad)}
+.rank{display:inline-block;width:22px;height:22px;line-height:22px;text-align:center;border-radius:50%;background:var(--fg);color:var(--bg);font-size:12px;margin-right:8px}
+.tag{font-size:11px;font-weight:600;border:1px solid var(--line);border-radius:10px;padding:1px 8px;margin-left:6px;color:var(--fg);border-color:var(--muted);white-space:nowrap}
+.where{margin:2px 0 6px}
+ol.top3{list-style:none;padding:0;margin:8px 0}
+li.top{display:flex;gap:10px;align-items:flex-start;margin:8px 0;break-inside:avoid}
+li.top .rank{flex:none;margin-top:1px}
+li.top h3{margin:0 0 2px}li.top p{margin:2px 0}li.top .why{font-size:14px}
+.legend{margin:4px 0 8px}
+pre.code{margin:6px 0;padding:8px 0;background:var(--bg);border:1px solid var(--line);border-radius:6px;overflow-x:auto}
+.ln{display:block;white-space:pre;padding:0 10px}
+.ln i{display:inline-block;width:3.2em;color:var(--muted);font-style:normal;user-select:none}
+.ln.hit{box-shadow:inset 3px 0 0 var(--bad);background:var(--hit);color:var(--hitfg);-webkit-print-color-adjust:exact;print-color-adjust:exact}
+.why{margin:6px 0}
+ul.plain{list-style:none;padding:0;margin:6px 0}ul.plain li{padding:5px 0;border-bottom:1px solid var(--line);overflow-wrap:anywhere}
+.note{background:var(--card);border-radius:6px;padding:8px 12px}
+ol{padding-left:22px}ol li{margin:6px 0}
+footer{margin-top:32px;padding-top:12px;border-top:1px solid var(--line);color:var(--muted);font-size:12px}
+@media (max-width:520px){main{padding:16px 12px}.score .big{font-size:44px}.tag{display:inline-block;margin:4px 0 0}pre.code{font-size:11.5px}}
+@media print{
+@page{margin:14mm}
+body{font-size:11.5px}main{max-width:none;padding:0}
+h2{margin:12px 0 6px;break-after:avoid}
+.summary .dims{display:grid;grid-template-columns:1fr 1fr;column-gap:16px}
+.summary pre.code{display:none}
+.finding{padding:6px 10px;margin:6px 0;break-inside:avoid}
+li.top{margin:6px 0}li.top .why{font-size:inherit}.explain,.check{font-size:inherit}.suite{margin-bottom:8px}p{margin:2px 0}ol li{margin:3px 0}.legend{font-size:11px;margin:2px 0 4px}
+.score .big{font-size:40px}header{padding-bottom:10px}
+.details{break-before:page}
+pre.code{overflow:visible;white-space:pre-wrap}
+}
+`;

package/dist/formatters/report.d.ts

@@ -0,0 +1,20 @@
+import type { ScoreResult } from '../types.js';
+export { rankTopFindings, compareGroups, type RuleGroup } from '../report-model.js';
+/** Returns the source text of a finding's file (path as in Finding.file), or undefined. */
+export type SourceReader = (file: string) => string | undefined;
+export interface ReportOptions {
+    /** Shown in the header, e.g. "2026-11-14". Injected so output is reproducible. */
+    date: string;
+    /** Scorer package version. */
+    version: string;
+    /** Suite name or path shown under the headline. */
+    suiteName: string;
+    /** True when --threshold was passed, so the bar reads "your bar" instead of "the default bar". */
+    customThreshold?: boolean;
+    /** Source lookup for code excerpts; without it the excerpts are omitted. */
+    readSource?: SourceReader;
+}
+/** Rules that make a test flaky rather than merely untidy. */
+export declare const FLAKE_RULES: readonly string[];
+export declare function escapeHtml(s: string): string;
+export declare function formatReport(result: ScoreResult, opts: ReportOptions): string;