@qaguardian/playwright-score 1.0.0 → 2.1.0

package/METHODOLOGY.md

@@ -1,6 +1,6 @@
 # Playwright Spec Score Methodology
 
-**Scoring model:** v3 (2026-08-27)  
+**Scoring model:** v4 (2026-09-23)  
 **Package:** `playwright-score`  
 **Landing page:** https://qaguardian.com/open-source/playwright-score  
 **Maintainer:** [QA Guardian](https://qaguardian.com)  
@@ -8,9 +8,11 @@
 **AI-free:** no LLM calls in the scoring path
 
 Any change that alters what a given suite scores is a new model version
-(v4, …), documented in the CHANGELOG. Model history: v1/v2 (published as
+(v5, …), documented in the CHANGELOG. Model history: v1/v2 (published as
 "sqs-v1"/"sqs-v2" through package 0.4.0) scored finding *density per line
-of code*; v3 replaced that entirely — see "Why v3" at the bottom.
+of code*; v3 replaced that entirely — see "Why v3" below. v4 keeps v3's
+exact per-test ratio formula and changes only which findings feed it —
+see "Why v4" below.
 
 ---
 
@@ -79,10 +81,10 @@ owns this dimension; counting them as demerits too would double-penalize.)
 
 | Dimension | Weight | Demerit sources |
 |---|---:|---|
-| playwrightHygiene | 40 | hard waits, networkidle, force, missing await, element handles, eval, conditionals, page.pause, useless await |
-| assertions | 25 | a test with no recognized assertion is fully flawed (`expect-expect`, after all delegation resolution); `valid-expect`, `no-standalone-expect`, `prefer-web-first-assertions` add partial demerits |
+| playwrightHygiene | 40 | hard waits (`page.waitForTimeout`, `new Promise`+`setTimeout` and its respellings, `node:timers/promises` setTimeout), networkidle, force, coordinate-based mouse interaction (`*.mouse.click/dblclick/move/down/up(x, y)`, partial demerit — see below), missing await, element handles, eval, conditionals, page.pause, useless await |
+| assertions | 25 | a test with no recognized assertion is fully flawed (`expect-expect`, after all delegation resolution); an assertion whose subject/matcher combination can only ever pass (`expect(true).toBe(true)`, `expect([]).toEqual([])`, never a deliberate force-fail like `expect(true).toBe(false)`); `valid-expect`, `no-standalone-expect`, `prefer-web-first-assertions`, a soft-assertion-only test (`expect.poll` counts as hard, not soft) all add partial demerits |
 | locators | 20 | usage ratio (above) |
-| structure | 15 | focused tests, always-skipped declarations, nested-describe depth, oversized files (module scope) |
+| structure | 15 | focused tests, always-skipped declarations, nested-describe depth, oversized files (module scope), an inline `eslint-disable` suppression comment (partial demerit — see "Known limitations" in VALIDATION.md) |
 
 ```
 score = round(Σ (weight_d / 100) × dimensionScore_d)
@@ -126,6 +128,51 @@ code scores identically regardless of which plugin version npm resolves.
 
 ---
 
+## Why v4
+
+v4 keeps v3's per-test ratio formula unchanged. What changed is which
+findings feed it:
+
+1. **Four gameability rules, planned as report-only under `standard`,
+   are real demerits there instead.** 1.1.0 (never published) planned
+   `pwscore/no-timer-sleep`,
+   `pwscore/no-coordinate-click`, `pwscore/no-trivial-assertion` and
+   `pwscore/no-soft-assertion-only-test` behind a separate opt-in `strict`
+   profile, so the public default (`standard`) wouldn't move anyone's
+   score. Owner direction, 2026-09-23: public scores should reflect
+   quality, full stop — a spec that sleeps on a timer, clicks a fixed
+   pixel coordinate, or asserts on a literal constant should score worse
+   under the default profile, not only under an opt-in one nobody has to
+   run. `strict` is removed as a result: with nothing left for it to gate
+   (every rule now scores the same way in the only remaining profile), a
+   second profile identical to the first was dead weight, not a real
+   choice. See CHANGELOG.md's 2.0.0 entry for the exact corpus repos this
+   moved, including pass→fail flips.
+2. **`pwscore/no-coordinate-click` is a warning (0.4 demerit), not an
+   error (1.0).** The 50-repo corpus surfaced a real false-positive
+   class: two infinite-canvas editors (tldraw, AFFiNE) accounted for
+   428/459 of this rule's hits, and a fixed-pixel `page.mouse.click(x,
+   y)` is the *correct*, only way to interact with canvas content that
+   has no addressable DOM element — not a brittleness anti-pattern there.
+   A full error-level demerit would treat every legitimate canvas
+   interaction as a fully-flawed test; a partial one keeps it a real cost
+   (still visible, still moves the score) without a brittle "is this a
+   canvas app?" heuristic (filename/import sniffing) that would itself
+   become a new gameability surface — name your spec file to dodge it.
+3. **Inline `eslint-disable` suppression comments are now a demerit,
+   not invisible.** A pre-2.0.0 review found this project's own docs
+   recommending `// eslint-disable-next-line pwscore/no-coordinate-click`
+   as an escape hatch, while every rule this package ships can in fact be
+   silenced the same way — defeating the score for whatever it hides.
+   `pwscore/eslint-disable-comment` reports the presence of any
+   suppression directive (not, yet, which specific rule it targets — a
+   documented known limitation, see VALIDATION.md).
+
+Together these move `standard` scores — see CHANGELOG.md's 2.0.0 entry
+for the "scores changed" breakdown, and VALIDATION.md for the full
+before/after table across all 50 corpus repos, including which ones flip
+from pass to fail at the default threshold.
+
 ## Why v3
 
 v1/v2 scored findings per 25 lines of code through an exponential decay

package/README.md

@@ -2,7 +2,7 @@
 
 [![CI](https://github.com/qa-guardian/playwright-score/actions/workflows/ci.yml/badge.svg)](https://github.com/qa-guardian/playwright-score/actions/workflows/ci.yml)
 [![npm version](https://img.shields.io/npm/v/@qaguardian/playwright-score.svg)](https://www.npmjs.com/package/@qaguardian/playwright-score)
-[![license](https://img.shields.io/npm/l/@qaguardian/playwright-score.svg)](./LICENSE)
+[![license](https://img.shields.io/npm/l/@qaguardian/playwright-score.svg)](https://github.com/qa-guardian/playwright-score/blob/main/LICENSE)
 
 **Deterministic, AI-free quality score for Playwright specs** — the
 weighted share of your tests that are clean.
@@ -19,42 +19,63 @@ The score never calls an LLM. AI may *generate* or *repair* code using findings;
 
 ## Real-world results
 
-17 public Playwright suites, scored with the published package against each
+100 public Playwright suites, scored with the published package against each
 project's actual source — chosen to include both well-known,
-heavily-engineered platforms and smaller, less mature projects. Not curated
-to look good: 5 of 17 fail the default threshold, for real, verifiable
-reasons (see [VALIDATION.md](./VALIDATION.md)).
+heavily-engineered platforms and smaller, less mature projects, plus three
+named testing-infra competitors scored under the exact same rules as
+everyone else. Not curated to look good: 35 of 100 fail the default
+threshold, for real, verifiable reasons (see
+[VALIDATION.md](https://github.com/qa-guardian/playwright-score/blob/main/VALIDATION.md),
+which also has the full before/after against the last published version
+and every one of the 100 entries, not just the 15 shown below).
 
-**17 suites · 1,230 files · 5,654 tests · 12/17 passed (80% threshold)**
+**100 suites · 7,281 files · 39,059 tests · 65/100 pass (80% threshold) ·
+tested on 2026-09-24**
+
+Top scorers (everyone at 97+; the next tier down is a 6-way tie at 96 — see
+VALIDATION.md for the full 100):
+
+| Repo (source scanned) | Score | Grade | Result |
+|---|---:|:-:|:-:|
+| [pinterest/gestalt](https://github.com/pinterest/gestalt/tree/22874a7522d1/playwright/accessibility) | 100/100 | A | PASS |
+| [TryGhost/Ghost](https://github.com/TryGhost/Ghost/tree/9d7248b56d53/e2e/tests) | 99/100 | A | PASS |
+| [hashicorp/vault](https://github.com/hashicorp/vault/tree/f03da019aba5/ui/e2e) | 98/100 | A | PASS |
+| [microsoft/playwright (TodoMVC example)](https://github.com/microsoft/playwright/tree/7563c5a67eab/examples/todomvc) | 98/100 | A | PASS |
+| [plausible/analytics](https://github.com/plausible/analytics/tree/8f3970104aa7/e2e) | 98/100 | A | PASS |
+| [segmentio/analytics-next](https://github.com/segmentio/analytics-next/tree/2d6d55301804/packages/browser-integration-tests/src) | 98/100 | A | PASS |
+| [cloudflare/templates](https://github.com/cloudflare/templates/tree/a0bb6ef9a990/playwright-tests) | 97/100 | A | PASS |
+| [glpi-project/glpi](https://github.com/glpi-project/glpi/tree/9e8a9570a90b/tests/e2e/specs) | 97/100 | A | PASS |
+| [shopify/hydrogen](https://github.com/shopify/hydrogen/tree/3ef7ec23096a/e2e/specs) | 97/100 | A | PASS |
+
+Bottom 5 scorers:
 
 | Repo (source scanned) | Score | Grade | Result |
 |---|---:|:-:|:-:|
-| [Playwright (own TodoMVC example)](https://github.com/microsoft/playwright/tree/main/examples/todomvc) | 98/100 | A | PASS |
-| [Storybook](https://github.com/storybookjs/storybook/tree/next/code/e2e-internal) | 95/100 | A | PASS |
-| [n8n](https://github.com/n8n-io/n8n/tree/master/packages/testing/playwright) | 95/100 | A | PASS |
-| [freeCodeCamp](https://github.com/freeCodeCamp/freeCodeCamp/tree/main/e2e) | 94/100 | A | PASS |
-| [dub](https://github.com/dubinc/dub/tree/main/apps/web/playwright) | 94/100 | A | PASS |
-| [Documenso](https://github.com/documenso/documenso/tree/main/packages/app-tests) | 93/100 | A | PASS |
-| [Supabase](https://github.com/supabase/supabase/tree/master/e2e) | 91/100 | A | PASS |
-| [PostHog](https://github.com/PostHog/posthog/tree/master/playwright) | 88/100 | B | PASS |
-| [Grafana](https://github.com/grafana/grafana/tree/main/e2e-playwright) | 87/100 | B | PASS |
-| [sencho](https://github.com/Studio-Saelix/sencho/tree/main/e2e) | 85/100 | B | PASS |
-| [Mattermost](https://github.com/mattermost/mattermost/tree/master/e2e-tests/playwright) | 83/100 | B | PASS |
-| [novu](https://github.com/novuhq/novu/tree/next/apps/dashboard/tests) | 82/100 | B | PASS |
-| [Immich](https://github.com/immich-app/immich/tree/main/e2e/src/specs) | 78/100 | C | FAIL |
-| [cal.com](https://github.com/calcom/cal.com/tree/main/apps/web/playwright) | 73/100 | C | FAIL |
-| [openplayerjs](https://github.com/openplayerjs/openplayerjs/tree/master/e2e) | 73/100 | C | FAIL |
-| [livecodes](https://github.com/live-codes/livecodes/tree/develop/e2e) | 70/100 | C | FAIL |
-| [TheCyberHub](https://github.com/th3cyb3rhub/TheCyberHub/tree/dev/e2e) | 56/100 | F | FAIL |
-
-Repo names link to the exact source scanned. Full methodology, findings
-breakdown, and `scripts/validate-corpus.sh` to reproduce every number here
-yourself live in [VALIDATION.md](./VALIDATION.md). A self-contained visual
+| [Flagsmith/flagsmith](https://github.com/Flagsmith/flagsmith/tree/ea5ad826e85b/frontend/e2e/tests) | 62/100 | D | FAIL |
+| [activepieces/activepieces](https://github.com/activepieces/activepieces/tree/9e493a87f471/packages/tests-e2e) | 58/100 | F | FAIL |
+| [th3cyb3rhub/TheCyberHub](https://github.com/th3cyb3rhub/TheCyberHub/tree/1cf418654f16/e2e) | 56/100 | F | FAIL |
+| [DataDog/documentation](https://github.com/DataDog/documentation/tree/6cd10f40cf46/hugo/e2e) | 50/100 | F | FAIL |
+| [getsentry/spotlight](https://github.com/getsentry/spotlight/tree/3dd5eb5b315f/packages/spotlight/tests) | 33/100 | F | FAIL |
+
+Repo names link to the exact commit scanned. Selected by a mix of
+mechanical GitHub search (real `@playwright/test` usage, hand-verified)
+and, for the most recent 16, hand-picked brand recognition — three of the
+100 are named competitors' own example suites (Checkly, Currents,
+LambdaTest), scored with no separate bar. Full methodology, the complete
+100-repo table, findings breakdown, and
+`scripts/validate-corpus.sh` to reproduce every number here yourself live
+in
+[VALIDATION.md](https://github.com/qa-guardian/playwright-score/blob/main/VALIDATION.md).
+A machine-readable
+[`validation.json`](https://github.com/qa-guardian/playwright-score/blob/main/validation.json)
+carries the same per-repo data for tooling. See
+[CHANGELOG.md](https://github.com/qa-guardian/playwright-score/blob/main/CHANGELOG.md)
+for what changed release to release. A self-contained visual
 report of an earlier five-suite audit lives at
-[`docs/scorecard.html`](./docs/scorecard.html) — host it wherever
+[`docs/scorecard.html`](https://github.com/qa-guardian/playwright-score/blob/main/docs/scorecard.html) — host it wherever
 (GitHub Pages, qaguardian.com, ...).
 
-See [METHODOLOGY.md](./METHODOLOGY.md) for the full scoring model, or the product write-up on the [landing page](https://qaguardian.com/open-source/playwright-score).
+See [METHODOLOGY.md](https://github.com/qa-guardian/playwright-score/blob/main/METHODOLOGY.md) for the full scoring model, or the product write-up on the [landing page](https://qaguardian.com/open-source/playwright-score).
 
 ## Install
 
@@ -148,8 +169,8 @@ sets:
 
 - **`warn`**: log score; never fail the run; findings still inject into heal/generate repair prompts  
 - **`gate`**: fail the run when score < threshold  
-- After AI **generate**, specs below threshold get one automatic **heal** pass with score findings  
-- **Heal** prompts always include score findings when the scorer reports issues  
+- After AI **generate**, every spec is scored; a spec below threshold is repaired with its score findings and re-scored, up to 3 scored attempts (the initial generation plus up to two repairs). A spec still below threshold, or one that could not be scored, is flagged for Guardian review and never silently accepted.
+- **Heal** and repair prompts always include score findings when the scorer reports issues.
 
 ```bash
 # Local from monorepo

package/dist/cli.js

@@ -18,7 +18,8 @@ Usage:
   npx playwright-score <paths...> [options]
 
 Options:
-  --profile <standard>            Scoring profile (default, and only: standard)
+  --profile <standard>             Scoring profile (default, and only
+                                   supported value, as of 2.0.0: standard).
   --threshold <n>                 Pass threshold 0-100 (default: 80)
   --format <text|json|markdown|sarif>  Output format (default: text)
   --out <file>                    Write report to file
@@ -31,7 +32,7 @@ Exit codes:
   2  tool error
 
 Website: https://qaguardian.com/open-source/playwright-score
-Methodology: METHODOLOGY.md (scoring model v3)
+Methodology: METHODOLOGY.md (scoring model v4)
 `);
 }
 // Deliberately never process.exit() right after a console.log/console.error
@@ -76,6 +77,11 @@ async function main() {
                 process.exitCode = 2;
                 return;
             }
+            if (p === 'strict') {
+                console.error('Profile "strict" was removed in 2.0.0 — the four gameability checks it used to count on top of "standard" (timer-sleep sleeps, coordinate clicks, trivial assertions, soft-assertion-only tests) are now scored under "standard" directly, so a separate stricter profile has nothing left to gate. See CHANGELOG.md. Use --profile standard.');
+                process.exitCode = 2;
+                return;
+            }
             if (p !== 'standard') {
                 console.error(`Invalid profile: ${p}`);
                 process.exitCode = 2;

package/dist/eslint-runner.d.ts

@@ -9,4 +9,6 @@ export declare function runEslint(options: {
     cwd?: string;
     /** See findLocalAssertionHelperNames in metrics.ts. */
     assertFunctionNames?: string[];
+    /** See detectExtendedTestAliases in test-aliases.ts. */
+    testAliasNames?: string[];
 }): Promise<Finding[]>;

package/dist/eslint-runner.js

@@ -4,10 +4,40 @@ import { commonAncestorDir } from './fs-util.js';
 import playwright from 'eslint-plugin-playwright';
 import { basePlugin } from './rules/base-plugin.js';
 import { mapRule } from './profiles.js';
-function buildConfig(assertFunctionNames) {
+function buildConfig(assertFunctionNames, testAliasNames) {
     const base = [
         {
             files: ['**/*.{ts,tsx,js,jsx,mjs,cjs}'],
+            // Identifiers bound to a `test.extend(...)`-derived fixture under a
+            // custom name (test-aliases.ts) — upstream eslint-plugin-playwright
+            // already dereferences a *same-file* extend chain internally, but
+            // can't follow one across files (its own `dereference` is scoped to
+            // the current file); this setting is what makes a cross-file alias
+            // (`import { loggedTest } from './fixtures'`) resolve to `test` for
+            // every rule that keys off the literal identifier — no-standalone-
+            // expect, no-conditional-expect, no-conditional-in-test, valid-expect,
+            // no-focused-test, and others built on the same parseFnCall/
+            // resolveToPlaywrightFn helper. Verified against two real corpus
+            // repos (argos-ci/argos's `loggedTest`, apache/superset's
+            // `testWithAssets`) where this exact cross-file shape was the actual
+            // false-positive cause.
+            settings: {
+                playwright: {
+                    globalAliases: { test: testAliasNames },
+                },
+            },
+            // A pre-2.0.0 review found `/* eslint-disable */` on line 1 of a
+            // 10-test file (waitForTimeout + expect(true).toBe(true) in every
+            // test) took it from 35/F to 99/A — every rule this package enables
+            // is an ordinary ESLint rule, so an ordinary inline disable comment
+            // silences it exactly like it would in any host project, defeating
+            // every one of these rules at will. noInlineConfig makes every
+            // eslint-disable/-line/-next-line comment (and enable/env/global)
+            // inert for this run's rule set; pwscore/eslint-disable-comment
+            // (metrics.ts) still reports the comment itself as a demerit — see
+            // eslint-disable-comment's docs — but it can no longer also silence
+            // whatever it's aimed at.
+            linterOptions: { noInlineConfig: true },
             languageOptions: {
                 ecmaVersion: 2022,
                 sourceType: 'module',
@@ -71,7 +101,7 @@ function buildConfig(assertFunctionNames) {
                 // with the same severity for the actual anti-pattern (an
                 // always-skipped test declaration) only.
                 'playwright/no-skipped-test': 'off',
-                'pwscore/no-skipped-test-declaration': 'warn',
+                'pwscore/no-skipped-test-declaration': ['warn', { testAliasNames }],
                 'playwright/no-raw-locators': 'warn',
                 // Both were mapped in profiles.ts (report-only, so they don't
                 // double-penalize the ratio-based locators dimension) but never
@@ -112,6 +142,18 @@ function buildConfig(assertFunctionNames) {
                 'playwright/valid-expect': 'error',
                 'playwright/no-standalone-expect': 'error',
                 'playwright/max-nested-describe': 'warn',
+                // QAG-196 gameability fixes (2026-09-21), scored unconditionally
+                // as of 2.0.0/model v4 — see base-plugin.ts for what each one
+                // catches and profiles.ts for the severity → demerit mapping.
+                // no-coordinate-click is 'warn' here too (not 'error'): profiles.ts
+                // overrides its scoring severity to 'warning' regardless (the
+                // canvas-app false-positive mitigation), so setting 'error' here
+                // would only mislead the raw ESLint severity shown in some
+                // formatters without changing what it actually costs.
+                'pwscore/no-timer-sleep': 'error',
+                'pwscore/no-coordinate-click': 'warn',
+                'pwscore/no-trivial-assertion': 'error',
+                'pwscore/no-soft-assertion-only-test': ['warn', { testAliasNames }],
             },
         },
     ];
@@ -142,7 +184,7 @@ export async function runEslint(options) {
     const cwd = options.files.length > 0
         ? commonAncestorDir(options.files)
         : (options.cwd ?? process.cwd());
-    const overrideConfig = buildConfig(options.assertFunctionNames ?? []);
+    const overrideConfig = buildConfig(options.assertFunctionNames ?? [], options.testAliasNames ?? []);
     // Try to attach typescript-eslint parser for .ts files
     try {
         const tsParser = await import('@typescript-eslint/parser');
@@ -189,6 +231,18 @@ export async function runEslint(options) {
             if (/^Definition for rule '.*' was not found\.$/.test(msg.message)) {
                 continue;
             }
+            // noInlineConfig (above) reports every inline directive comment
+            // itself as a `ruleId: null` message ("'...' has no effect because
+            // you have 'noInlineConfig' setting in your config.") — already
+            // caught by the generic `!msg.ruleId` skip below, but matched
+            // explicitly here too (same treatment as "Definition for rule",
+            // above) so this stays true even if that generic skip is ever
+            // narrowed: it's a diagnostic about our own noInlineConfig setting,
+            // not the spec's quality, and pwscore/eslint-disable-comment already
+            // reports the comment itself as a real finding.
+            if (/has no effect because you have 'noInlineConfig' setting in your config\.$/.test(msg.message)) {
+                continue;
+            }
             if (msg.fatal) {
                 // A file that doesn't parse can't be meaningfully scored at all —
                 // surface it explicitly rather than silently dropping it (it has

package/dist/formatters/markdown.js

@@ -32,5 +32,13 @@ export function formatMarkdown(result) {
         lines.push('');
         lines.push('</details>');
     }
+    if (result.configWarnings?.length) {
+        lines.push('');
+        lines.push('## Config warnings');
+        lines.push('');
+        for (const w of result.configWarnings) {
+            lines.push(`- ${w}`);
+        }
+    }
     return lines.join('\n');
 }

package/dist/formatters/text.js

@@ -36,6 +36,13 @@ export function formatText(result) {
             lines.push(`  … and ${result.skippedFiles.length - 20} more`);
         }
     }
+    if (result.configWarnings?.length) {
+        lines.push('');
+        lines.push('Config warnings:');
+        for (const w of result.configWarnings) {
+            lines.push(`  ${w}`);
+        }
+    }
     return lines.join('\n');
 }
 function pathBasename(p) {

package/dist/import-graph.d.ts

@@ -1,3 +1,4 @@
+export declare function resolveRelativeImport(fromFile: string, specifier: string, boundaryDir: string): string | undefined;
 /**
  * A Page Object Model is common in real Playwright suites: spec files
  * interact with the page entirely through methods on an imported class

package/dist/import-graph.js

@@ -35,7 +35,7 @@ function extractRelativeImportSpecifiers(source, file) {
     }
     return specifiers;
 }
-function resolveRelativeImport(fromFile, specifier, boundaryDir) {
+export function resolveRelativeImport(fromFile, specifier, boundaryDir) {
     const base = path.resolve(path.dirname(fromFile), specifier);
     const candidates = [];
     for (const ext of RESOLVE_EXTENSIONS)

package/dist/index.js

@@ -5,6 +5,8 @@ import { runEslint } from './eslint-runner.js';
 import { commonAncestorDir } from './fs-util.js';
 import { collectLocallyImportedFiles } from './import-graph.js';
 import { analyzeSource, attributeFindingScopes, countLocators, findLocalAssertionHelperNames, getTestSpans, looksLikeNonPlaywrightTest, } from './metrics.js';
+import { detectExtendedTestAliases } from './test-aliases.js';
+import { findPlaywrightConfig, findRepoBoundary, importsPlaywrightTestTransitively, parsePlaywrightConfig, resolveConfigScopedRoot, testDirEscapesRepo, } from './playwright-config.js';
 import { DEFAULT_THRESHOLDS } from './profiles.js';
 import { computeScore } from './score-engine.js';
 export { computeScore, MODEL, locatorsScore, gradeFromScore, ratioDimensionScore, } from './score-engine.js';
@@ -23,8 +25,13 @@ export { formatSarif } from './formatters/sarif.js';
 // metrics.ts) is the safety net for a repo that uses `.e2e.` for a
 // *different* framework — Cypress specifically was common enough here to
 // get its own explicit check alongside the existing Jest/Vitest/RTL ones.
+// `.pw.` joined the list from the 50-repo corpus expansion (QAG-196,
+// 2026-09-21) — Flagsmith's entire 20-spec frontend/e2e suite
+// (billing-test.pw.ts, flag-tests.pw.ts, ...) hard-failed with "no files
+// matched" without it, the same failure mode cal.com/Immich already fixed
+// for their own suffixes.
 const SPEC_GLOBS = [
-    '**/*.{spec,test,e2e,e2e-spec,e2e-test}.{ts,tsx,js,jsx}',
+    '**/*.{spec,test,e2e,e2e-spec,e2e-test,pw}.{ts,tsx,js,jsx}',
     '**/*.spec.ts',
     '**/*.test.ts',
 ];
@@ -62,6 +69,7 @@ function expandPaths(inputs, cwd) {
     const explicit = new Set();
     const expanded = new Set();
     const scanRootDirs = new Set();
+    const configWarnings = [];
     for (const input of inputs) {
         const abs = path.isAbsolute(input) ? input : path.resolve(cwd, input);
         if (fs.existsSync(abs) && fs.statSync(abs).isFile()) {
@@ -76,14 +84,109 @@ function expandPaths(inputs, cwd) {
             // needs this wider boundary, not the narrower common ancestor of
             // only the matched spec files.
             scanRootDirs.add(abs);
-            for (const g of SPEC_GLOBS) {
+            // A playwright.config.* at or above this directory, when it can be
+            // read statically, is authoritative about what this target itself
+            // considers a spec — see playwright-config.ts. Falls back to the
+            // filename-suffix SPEC_GLOBS (today's behavior) whenever a config
+            // isn't found, doesn't overlap this scan, or can't be parsed
+            // statically (a dynamic/unsupported shape) — the last case surfaces
+            // a warning rather than silently guessing.
+            let globCwd = abs;
+            let specGlobs = SPEC_GLOBS;
+            let ignoreGlobs = DEFAULT_IGNORE_GLOBS;
+            let configApplied = false;
+            // Boundary for the transitive-import safety net below — widened to
+            // the config's own directory (not `abs`), for the same reason
+            // import-graph.ts already widens its own POM boundary past the
+            // matched spec files' common ancestor: a fixtures module a spec
+            // imports test/expect from commonly lives in a directory *sibling*
+            // to testDir (`playwright/fixtures` next to `playwright/tests`), not
+            // inside it. Verified against a real config (appsmithorg/appsmith)
+            // where every real spec imports `test`/`expect` from `../../fixtures`
+            // — bounding resolution to `abs` (testDir itself) made every file
+            // fail the check and silently zeroed the whole suite, the same
+            // false-negative failure mode this safety net exists to prevent.
+            let importSafetyNetBoundary = abs;
+            const configFile = findPlaywrightConfig(abs);
+            if (configFile) {
+                const parsedConfig = parsePlaywrightConfig(configFile);
+                if (!parsedConfig) {
+                    configWarnings.push(`Found ${path.relative(cwd, configFile) || configFile} but could not statically parse testDir/testMatch/testIgnore (a dynamic value or unsupported shape) — falling back to filename-based spec discovery for ${path.relative(cwd, abs) || abs}.`);
+                }
+                else {
+                    // Fallback boundary for findRepoBoundary when no .git turns up:
+                    // the config file's own directory when it sits above `abs`
+                    // (found by walking up from abs, so it's always an ancestor-or-
+                    // same of it), else `abs` itself. Never the filesystem root or
+                    // 20 levels up — see findRepoBoundary's doc comment.
+                    const repoRoot = findRepoBoundary(abs, path.dirname(configFile));
+                    if (testDirEscapesRepo(parsedConfig.testDirAbs, repoRoot)) {
+                        // The config's testDir resolves outside the repo boundary
+                        // (directly, or via a symlink — see testDirEscapesRepo). Its
+                        // testMatch/testIgnore were written for that escaped testDir,
+                        // so applying them to the clamped repoRoot instead would be a
+                        // silent mismatch; fall back to filename-based discovery.
+                        configWarnings.push(`Found ${path.relative(cwd, configFile) || configFile} but its testDir (${parsedConfig.testDirAbs}) resolves outside the repository (${repoRoot}) — falling back to filename-based spec discovery for ${path.relative(cwd, abs) || abs}.`);
+                    }
+                    else {
+                        const scopedRoot = resolveConfigScopedRoot(abs, parsedConfig.testDirAbs, repoRoot);
+                        if (scopedRoot) {
+                            globCwd = scopedRoot;
+                            if (parsedConfig.testMatch)
+                                specGlobs = parsedConfig.testMatch;
+                            if (parsedConfig.testIgnore) {
+                                ignoreGlobs = [...DEFAULT_IGNORE_GLOBS, ...parsedConfig.testIgnore];
+                            }
+                            configApplied = true;
+                            importSafetyNetBoundary = path.dirname(configFile);
+                        }
+                    }
+                }
+            }
+            for (const g of specGlobs) {
                 for (const f of globSync(g, {
-                    cwd: abs,
+                    cwd: globCwd,
                     absolute: true,
                     nodir: true,
-                    ignore: DEFAULT_IGNORE_GLOBS,
+                    ignore: ignoreGlobs,
+                    // glob's own default for `follow` is already false (don't
+                    // descend into a symlinked directory while expanding `**`), but
+                    // this is untrusted-repo territory (see testDirEscapesRepo
+                    // above), so pin it explicitly rather than relying on a default
+                    // that could change upstream.
+                    follow: false,
                 })) {
-                    expanded.add(path.resolve(f));
+                    const resolved = path.resolve(f);
+                    // resolveConfigScopedRoot always globs from testDir (Playwright's
+                    // own testMatch/testIgnore semantics — see its doc comment), which
+                    // can be wider than the directory the caller actually pointed
+                    // `scorePaths` at (a narrower `abs`, e.g. one subproject's specs
+                    // inside a monorepo testDir that also covers others). Filter back
+                    // down to what the caller asked for so scoping to testDir for
+                    // correct path matching never silently widens the scored set.
+                    if (configApplied && globCwd !== abs) {
+                        const rel = path.relative(abs, resolved);
+                        if (rel.startsWith('..') || path.isAbsolute(rel))
+                            continue;
+                    }
+                    // Extra safety net, only once config-based discovery is actually
+                    // driving this scan: testMatch can still be broader than intended
+                    // (e.g. a generic '**/*.ts'); a file that imports neither
+                    // '@playwright/test' directly nor (transitively, through a local
+                    // module) re-exports it isn't a Playwright spec regardless of
+                    // what the config's glob matched.
+                    if (configApplied) {
+                        let source;
+                        try {
+                            source = fs.readFileSync(resolved, 'utf8');
+                        }
+                        catch {
+                            continue;
+                        }
+                        if (!importsPlaywrightTestTransitively(resolved, source, importSafetyNetBoundary))
+                            continue;
+                    }
+                    expanded.add(resolved);
                 }
             }
             continue;
@@ -106,11 +209,12 @@ function expandPaths(inputs, cwd) {
         explicit: [...explicit].sort(),
         expanded: [...expanded].sort(),
         scanRootDirs: [...scanRootDirs].sort(),
+        configWarnings,
     };
 }
-function hardFail(profile, threshold, findings, extra = {}, skippedFiles) {
+function hardFail(profile, threshold, findings, extra = {}, skippedFiles, configWarnings) {
     return {
-        scoreVersion: 'v3',
+        scoreVersion: 'v4',
         profile,
         score: 0,
         grade: 'F',
@@ -137,6 +241,7 @@ function hardFail(profile, threshold, findings, extra = {}, skippedFiles) {
         },
         findings,
         ...(skippedFiles && skippedFiles.length > 0 ? { skippedFiles } : {}),
+        ...(configWarnings && configWarnings.length > 0 ? { configWarnings } : {}),
     };
 }
 /**
@@ -146,7 +251,7 @@ export async function scorePaths(options) {
     const cwd = options.cwd ?? process.cwd();
     const profile = options.profile ?? 'standard';
     const threshold = options.threshold ?? DEFAULT_THRESHOLDS[profile] ?? 80;
-    const { explicit, expanded, scanRootDirs } = expandPaths(options.paths, cwd);
+    const { explicit, expanded, scanRootDirs, configWarnings } = expandPaths(options.paths, cwd);
     if (explicit.length === 0 && expanded.length === 0) {
         // Hard-fail: empty match must never look like a healthy suite (was ~99 PASS).
         return hardFail(profile, threshold, [
@@ -157,7 +262,7 @@ export async function scorePaths(options) {
                 file: cwd,
                 dimension: 'structure',
             },
-        ]);
+        ], {}, undefined, configWarnings);
     }
     // Directory/glob matches are excluded only on positive evidence of a
     // non-Playwright test framework — see looksLikeNonPlaywrightTest. Paths
@@ -224,11 +329,20 @@ export async function scorePaths(options) {
             assertFunctionNames.add(name);
         }
     }
+    // Identifiers bound to a `test.extend(...)`-derived fixture under a
+    // custom name (same-file, or imported from a local fixtures module —
+    // see test-aliases.ts), so a suite that always declares its tests via
+    // `loggedTest(...)` instead of `test(...)` is recognized correctly by
+    // both eslint-plugin-playwright's rules (fed via runEslint's settings
+    // below) and this package's own AST checks (getTestSpans/countTests,
+    // base-plugin.ts).
+    const testAliasNames = detectExtendedTestAliases(sources, importedFiles, importBoundary);
     const eslintFindings = await runEslint({
         files,
         profile,
         cwd,
         assertFunctionNames: [...assertFunctionNames],
+        testAliasNames,
     });
     let totalSloc = 0;
     let totalTests = 0;
@@ -240,12 +354,12 @@ export async function scorePaths(options) {
     for (const file of files) {
         const source = sources.get(file) ?? '';
         const relFile = path.relative(filesBase, file) || path.basename(file);
-        const m = analyzeSource(source, relFile);
+        const m = analyzeSource(source, relFile, testAliasNames);
         totalSloc += m.sloc;
         native += m.locators.native;
         raw += m.locators.raw;
         metricFindings.push(...m.findings);
-        const spans = getTestSpans(source, relFile);
+        const spans = getTestSpans(source, relFile, testAliasNames);
         fileSpans.set(relFile, spans);
         // AST spans are the authoritative test count (they are also the
         // attribution targets); the regex count is the fallback for a file
@@ -288,5 +402,9 @@ export async function scorePaths(options) {
         nativeLocators: native,
         rawLocators: raw,
     });
-    return skippedFiles.length > 0 ? { ...result, skippedFiles } : result;
+    return {
+        ...result,
+        ...(skippedFiles.length > 0 ? { skippedFiles } : {}),
+        ...(configWarnings.length > 0 ? { configWarnings } : {}),
+    };
 }