@qaguardian/playwright-score 2.0.0 → 2.1.1

package/CHANGELOG.md

@@ -1,8 +1,244 @@
 # Changelog
 
 All notable changes to this project are documented here. Any change that
-alters what a given suite scores is a new scoring-model version (see
-[METHODOLOGY.md](./METHODOLOGY.md)), not a quiet patch.
+alters what a given suite scores *by changing a rule's weight or
+severity* is a new scoring-model version (see
+[METHODOLOGY.md](./METHODOLOGY.md)), not a quiet patch. A release that
+leaves every rule's weight and severity untouched but changes which files
+or which test names get recognized in the first place — discovery and
+parsing accuracy, not grading policy — can still move a suite's score
+without itself being a new model version; 2.1.0 below is exactly that
+case, and says so explicitly.
+
+## 2.1.1 — 2026-09-26
+
+Scoring model stays v4. Reviewer follow-ups from the 2.1.0 release review
+(QAG-230) — none of these blocked that release. They close symlink-escape
+gaps a crafted repo could otherwise use, quiet a false warning, and fix a
+false-positive drop the first round of these same fixes introduced. None
+of that changes what an ordinary suite with no symlinked spec files
+scores; the realpath dedupe fix below is the one exception — it can
+change the file/test count for a suite that has a symlinked spec file
+matching discovery alongside its own real target, and says so.
+
+### Fixed
+- **A match escaping the *repository* through a symlink is now dropped —
+  scoped to the repo root, not the narrower directory the caller pointed
+  at.** `testDirEscapesRepo` (added in 2.1.0) only clamps a config's
+  `testDir` itself before it becomes a glob root — it couldn't see a
+  symlinked directory the glob then walked *through* on the way to an
+  individual match. A `testMatch` entry that names a symlinked directory
+  by a literal path segment (not a `**` wildcard) is still followed even
+  with glob's `follow: false` (that option only stops `**` from
+  *expanding into* a symlinked directory, not a pattern from naming one
+  explicitly), so a crafted repo (`tests/evil -> /`, `testMatch:
+  ['evil/**/*.ts']`) could make discovery walk arbitrary parts of the
+  filesystem while every match still lexically looked like it lived
+  under the scanned directory. The same gap let an individually
+  symlinked spec file sitting directly in the repo (no `testMatch`
+  trickery needed) resolve to, and get scored from, a file outside it.
+  `src/index.ts`'s match loop realpath's every candidate match and drops
+  it unless it's still contained within the realpath of the **repository
+  root** (`safeRealpath(findRepoBoundary(abs))`) — not, as this package
+  first shipped it, the narrower directory the caller actually scanned.
+  That first version silently dropped a real, common monorepo shape too:
+  a spec reachable through an in-repo symlink that resolves to a sibling
+  directory *outside* the caller's scanned subpath (one package's specs
+  shared with another via a symlink) is legitimate and is kept; only a
+  match whose realpath lands outside the repository itself is dropped.
+  The same repo-root containment now also applies to a bare glob-pattern
+  input (e.g. `tests/*.spec.ts` passed directly to `scorePaths`, not a
+  directory it walks itself) — the first version of this fix only
+  covered the directory-scan branch, so switching an otherwise-identical
+  call from a directory argument to an equivalent glob argument silently
+  bypassed the guard entirely.
+- **The walk itself is now pruned at a symlink that escapes the repo
+  root, not just filtered after the fact.** The realpath containment
+  check above still has to pay to walk all the way through a symlinked
+  directory before dropping every match inside it one by one — for a
+  large or slow target (`tests/evil -> /`) that's a real cost, not just
+  a theoretical one (measured: ~6.4s to fully walk a large symlinked
+  directory before this fix). `RepoBoundaryIgnore` (`src/index.ts`) wraps
+  glob's own `ignore` matching with a `childrenIgnored` check that stops
+  the walk from ever descending into a symlinked directory whose
+  realpath lands outside the repo root, only paying the realpath cost at
+  an actual symlink (a plain nested directory can't walk outside a root
+  its own parent is already inside, so it's never checked).
+- **An in-repo symlinked spec and its own real target are no longer both
+  scored, when both independently match discovery under the same scan.**
+  Pre-existing, not new to the fixes above: `expandPaths` deduped a
+  literal path only against `explicit` inputs, never against another
+  `expanded` match that happens to resolve to the same realpath. A
+  symlink sitting alongside its target inside the same scanned directory
+  (or reachable via two different scan roots in one `scorePaths` call)
+  was counted, and scored, twice. Now deduped by realpath, keeping the
+  first lexical path in sorted order. **This can change file/test counts
+  for a repo that has such a link** — said plainly here rather than
+  folded into "doesn't change what a correctly-shaped suite scores",
+  since a suite with this exact shape is exactly what changes.
+- **False "testDir resolves outside the repository" warning on a
+  nonexistent `testDir` under a symlinked repo path.** `safeRealpath`
+  only resolves the portion of a path that exists on disk, so a
+  nonexistent `testDir` kept its literal, unresolved form while
+  `repoRoot` (which does exist) got fully resolved — on a host where the
+  repo itself sits under a symlinked prefix (macOS: `/tmp` ->
+  `/private/tmp`), that asymmetry alone made the two sides disagree and
+  fired a false escape warning for an ordinary missing `testDir`.
+  `testDirEscapesRepo` now checks existence first and returns "does not
+  escape" immediately for a `testDir` that isn't on disk — consistent
+  with `resolveConfigScopedRoot`'s own existing "nothing to score there"
+  handling for the same case.
+- **Escape warning now prints cwd-relative paths**, matching every other
+  `configWarning` in this package, instead of the runner's own absolute
+  filesystem paths.
+- **A raw glob-pattern input pointing at a different repo than `cwd` no
+  longer hard-fails with 0 matches.** The repo-root containment fix above
+  used `cwd`'s own repo boundary for a bare glob pattern regardless of
+  where the pattern actually pointed — an absolute pattern into a sibling
+  repo (`['/abs/repoB/e2e/*.spec.ts']`), or an equivalent `../`-relative
+  one, had every match rejected as an "escape" of `cwd`'s repo, where the
+  same target passed as a directory already scored fine. `src/index.ts`'s
+  glob-input branch now derives its containment boundary from *the
+  pattern's own* literal (non-magic) leading-directory prefix, resolved
+  against `cwd`, not from `cwd` itself — a literal prefix that lands
+  inside another repo entirely is scored against that repo's own
+  boundary, while a symlink an in-bounds prefix still reaches that escapes
+  *that* repo is still dropped.
+- **An explicit target and a directory scan that reaches it again through
+  a symlink are no longer scored twice.** The realpath dedupe added above
+  only ever populated its map from `expanded` matches; an `explicit` file
+  and a same-realpath `expanded` symlink are two different literal paths,
+  so the separate "explicit wins" step (an exact literal-path delete)
+  never caught that pairing. The dedupe map is now seeded with
+  `explicit`'s own realpaths first, so a same-realpath `expanded` match is
+  recognized as a duplicate instead of surviving as a second, independent
+  file.
+
+Corpus rerun (100 public repos, directory inputs, run on the round-2 build;
+the round-3 fixes only touch glob-pattern inputs): 96/100 repos score
+identically, 4 moved upstream, 0 changed.
+
+## 2.1.0 — 2026-09-24
+
+Scoring model stays v4 — nothing here changes a rule's weight or
+severity, only whether a suite is *recognized* correctly in the first
+place (see the note at the top of this file reconciling that with "any
+score-changing release is a new model version"). Every fix below made at
+least one real corpus repo's score more accurate — not uniformly higher,
+and not less strict or more lenient by policy; see "Behaviour change"
+below and VALIDATION.md's "Bugs found this way, and fixed" for the full
+story (corpus repo, exact before/after) behind each one.
+
+### Fixed
+- **Custom `test.extend()` fixture names.** A suite that always declares
+  its tests via a custom-named fixture (`const loggedTest =
+  test.extend(...)`, imported into every spec file — sometimes through a
+  re-export barrel) had every `expect()` inside a completely normal test
+  flagged by upstream `eslint-plugin-playwright`'s `no-standalone-expect`
+  and sibling rules (which key off the literal identifier `test`, and
+  can only dereference a same-file extend chain), and undercounted by
+  this package's own test-span/`countTests` logic too.
+  `src/test-aliases.ts` resolves these aliases — same-file and
+  cross-file, through imports and re-export barrels — and feeds both
+  `eslint-plugin-playwright`'s `globalAliases.test` setting and this
+  package's own AST rules. Scores rise for affected suites:
+  `argos-ci/argos` 57/F → 95/A, `apache/superset` 70/C → 77/C.
+- **Discovery now honours the target's own `playwright.config.*`
+  `testDir`/`testMatch`/`testIgnore`.** Spec discovery previously matched
+  by filename suffix only, so a directory holding a different test
+  runner's files under a similar naming convention got swept in
+  alongside the real suite. `src/playwright-config.ts` statically parses
+  (never executes — a scored repo's config is untrusted input) the
+  nearest `playwright.config.{ts,js,mjs,cjs}` at or above the scanned
+  directory and scopes discovery to what the target itself declares.
+  When no config is found at all, this falls back to the previous
+  filename-based behavior silently, exactly as before this release; when
+  a config *is* found but can't be read statically (a dynamic value or
+  an unsupported shape, e.g. a `RegExp` `testMatch`), it falls back the
+  same way but now also adds an entry to the new, additive, optional
+  `ScoreResult.configWarnings` field so the caller can see why. Includes
+  three fixes found making this work against real configs: matching
+  Playwright's own `createFileMatcher` auto-`**/`-prefix behavior for a
+  `testMatch`/`testIgnore` pattern with no leading `**/`; globbing from
+  the config's `testDir` rather than the caller's own (possibly narrower
+  or differently-rooted) scanned directory; and failing *open* (keeping a
+  file) rather than treating an unresolvable relative import as proof a
+  file isn't a real spec, since a sparse/partial checkout can legitimately
+  be missing a fixtures module the real project has. This moves a
+  suite's score in *either* direction depending on whether the old
+  filename-only discovery had been over- or under-matching relative to
+  what the config itself declares — see "Behaviour change" below for a
+  concrete case where it lowers a score.
+- **Discovery no longer misses a plain CommonJS suite.** A follow-on to
+  the config-scoped discovery fix above: the safety net that confirms a
+  config-matched file transitively imports `@playwright/test` only
+  walked ES `import`/`export ... from` syntax, so a suite written
+  entirely in CommonJS (`require()`, no `import` anywhere) read as
+  *definitive negative evidence* of not being a Playwright spec — the
+  opposite of the truth — and hard-failed to 0 files the moment
+  config-scoped discovery activated for it. Fixed by also recognizing a
+  top-level `require('x')` call the same way as an ES import.
+- **Config discovery and a config's `testDir` can no longer escape the
+  scanned repo.** Hardening fixes for a public tool that reads an
+  untrusted repo's own config. `findPlaywrightConfig` stops walking
+  upward at the first ancestor directory containing `.git` (a directory
+  for a normal clone, a file for a worktree/submodule), so a coincidental
+  `playwright.config.*` sitting above the actual repo root — a shared CI
+  runner, a monorepo checkout nested inside another one — is never picked
+  up (this guarantee needs a `.git` somewhere in the ancestor chain).
+  The bug this closes: when no `.git` existed anywhere in the ancestor
+  chain, the boundary used to clamp a config's `testDir` fell back to the
+  filesystem root (or 20 levels up), which is effectively no clamp at
+  all, so a `testDir: '/'` could make the glob walk the whole filesystem
+  (in the worst case, a hang). For the `testDir` clamp the fallback is
+  now the directory actually passed to the scorer, or the config file's
+  own directory when that's the wider of the two, never the filesystem
+  root. The config search itself (a bounded series of existence checks)
+  keeps its wider fallback so monorepo configs above the scanned
+  directory are still found.
+  Symlinks got the same treatment: a `testDir` is now checked *after
+  resolving symlinks* on both sides of the ancestor check, not just
+  lexically, so a `testDir` that's a symlink to somewhere outside the
+  repo can't launder an escape the plain path comparison would have
+  missed (the glob's own `follow` option, already `false` by default for
+  `**` patterns, is also now pinned explicitly). An escaping `testDir` —
+  absolute (`'/'`), climbing out (`'../../..'`), or via a symlink — gets a
+  `configWarnings` entry and falls back to filename-based discovery
+  rather than globbing the clamped repo root with testMatch/testIgnore
+  patterns written for a different directory. To be clear about the
+  actual exposure this closes: a scored *result* was never able to
+  include a file whose path lies outside the scan root either way (a
+  symlinked file inside the repo is a separate, tracked follow-up) — index.ts always
+  filters expanded matches back down to the caller's own requested
+  directory — the risk was the walk/glob itself reading (or hanging on)
+  parts of the filesystem well outside the repo, not a leaked finding in
+  the output. Neither case is exercised by any corpus repo (a legitimate
+  config never does this); found and fixed during this release's own
+  review, not from a corpus regression.
+
+### Behaviour change
+- **Config-scoped discovery can lower a suite's score, not just raise
+  it.** Scoping to what a target's own `playwright.config.*` actually
+  declares is strictly more accurate than the old filename-suffix match,
+  but "more accurate" isn't the same as "friendlier": `mattermost/
+  mattermost` moves **84/B → 79/C (a pass→fail flip)** because its
+  top-level config's `testDir` doesn't cover the real, correctly-written
+  `upgrade-specs/` suite it runs only through per-project `testDir`
+  overrides — `projects[].testDir` isn't read yet (see VALIDATION.md's
+  "Known limitations"). A future config parser that reads
+  `projects[].testDir` would recover this repo's full suite and its
+  higher score.
+
+### Corpus
+Grown from 50 (2.0.0) to 100 public suites: 34 added in an interim,
+unpublished pass (84 total — see VALIDATION.md for the full accounting of
+that pass), then 16 more here for brand
+recognition (Nextcloud, Ghost, Pinterest, Microsoft, The Guardian,
+Shopify ×2, Adobe, Automattic, BBC, Google, Datadog, Twilio) plus three
+named competitors (Checkly, Currents, LambdaTest), scored under the
+exact same rules as everyone else. Every one of the 100 entries in
+VALIDATION.md now carries the exact commit SHA scored and the date it
+was tested, not just an aggregate "last run" date.
 
 ## 2.0.0 — 2026-09-23
 
@@ -37,7 +273,7 @@ own official examples must still score >= 90/A (see
 `tests/official-examples.test.ts`, vendored from microsoft/playwright),
 and a hand-written "gold" fixture must still reach exactly 100.
 
-### Added — four rules narrowing the QAG-196 gameability gap (scored under `standard`)
+### Added — four rules narrowing the gameability gap (scored under `standard`)
 Narrowing, not closing: each rule matches specific known patterns (see the
 respellings each one closed in this same release), not the general class
 of "code that fakes stability or coverage" — a determined author can

package/README.md

@@ -2,7 +2,7 @@
 
 [![CI](https://github.com/qa-guardian/playwright-score/actions/workflows/ci.yml/badge.svg)](https://github.com/qa-guardian/playwright-score/actions/workflows/ci.yml)
 [![npm version](https://img.shields.io/npm/v/@qaguardian/playwright-score.svg)](https://www.npmjs.com/package/@qaguardian/playwright-score)
-[![license](https://img.shields.io/npm/l/@qaguardian/playwright-score.svg)](./LICENSE)
+[![license](https://img.shields.io/npm/l/@qaguardian/playwright-score.svg)](https://github.com/qa-guardian/playwright-score/blob/main/LICENSE)
 
 **Deterministic, AI-free quality score for Playwright specs** — the
 weighted share of your tests that are clean.
@@ -19,76 +19,63 @@ The score never calls an LLM. AI may *generate* or *repair* code using findings;
 
 ## Real-world results
 
-50 public Playwright suites, scored with the published package against each
+100 public Playwright suites, scored with the published package against each
 project's actual source — chosen to include both well-known,
-heavily-engineered platforms and smaller, less mature projects. Not curated
-to look good: 16 of 50 fail the default threshold, for real, verifiable
-reasons (see [VALIDATION.md](./VALIDATION.md), which also has the full
-before/after against the last published version).
+heavily-engineered platforms and smaller, less mature projects, plus three
+named testing-infra competitors scored under the exact same rules as
+everyone else. Not curated to look good: 35 of 100 fail the default
+threshold, for real, verifiable reasons (see
+[VALIDATION.md](https://github.com/qa-guardian/playwright-score/blob/main/VALIDATION.md),
+which also has the full before/after against the last published version
+and every one of the 100 entries, not just the 15 shown below).
 
-**50 suites · 4,159 files · 20,183 tests · 34/50 passed (80% threshold)**
+**100 suites · 7,281 files · 39,059 tests · 65/100 pass (80% threshold) ·
+tested on 2026-09-24**
+
+Top scorers (everyone at 97+; the next tier down is a 6-way tie at 96 — see
+VALIDATION.md for the full 100):
+
+| Repo (source scanned) | Score | Grade | Result |
+|---|---:|:-:|:-:|
+| [pinterest/gestalt](https://github.com/pinterest/gestalt/tree/22874a7522d1/playwright/accessibility) | 100/100 | A | PASS |
+| [TryGhost/Ghost](https://github.com/TryGhost/Ghost/tree/9d7248b56d53/e2e/tests) | 99/100 | A | PASS |
+| [hashicorp/vault](https://github.com/hashicorp/vault/tree/f03da019aba5/ui/e2e) | 98/100 | A | PASS |
+| [microsoft/playwright (TodoMVC example)](https://github.com/microsoft/playwright/tree/7563c5a67eab/examples/todomvc) | 98/100 | A | PASS |
+| [plausible/analytics](https://github.com/plausible/analytics/tree/8f3970104aa7/e2e) | 98/100 | A | PASS |
+| [segmentio/analytics-next](https://github.com/segmentio/analytics-next/tree/2d6d55301804/packages/browser-integration-tests/src) | 98/100 | A | PASS |
+| [cloudflare/templates](https://github.com/cloudflare/templates/tree/a0bb6ef9a990/playwright-tests) | 97/100 | A | PASS |
+| [glpi-project/glpi](https://github.com/glpi-project/glpi/tree/9e8a9570a90b/tests/e2e/specs) | 97/100 | A | PASS |
+| [shopify/hydrogen](https://github.com/shopify/hydrogen/tree/3ef7ec23096a/e2e/specs) | 97/100 | A | PASS |
+
+Bottom 5 scorers:
 
 | Repo (source scanned) | Score | Grade | Result |
 |---|---:|:-:|:-:|
-| [Plausible Analytics](https://github.com/plausible/analytics/tree/0f362bd84eaa/e2e) | 98/100 | A | PASS |
-| [Playwright (own TodoMVC example)](https://github.com/microsoft/playwright/tree/0facd9234452/examples/todomvc) | 98/100 | A | PASS |
-| [Penpot](https://github.com/penpot/penpot/tree/433f8774497a/frontend/playwright) | 96/100 | A | PASS |
-| [SigNoz](https://github.com/SigNoz/signoz/tree/64fff60d7e8c/tests/e2e) | 96/100 | A | PASS |
-| [Temporal UI](https://github.com/temporalio/ui/tree/ffee0375c167/tests) | 96/100 | A | PASS |
-| [dub](https://github.com/dubinc/dub/tree/bd69439da068/apps/web/playwright) | 96/100 | A | PASS |
-| [Keycloak (admin-ui)](https://github.com/keycloak/keycloak/tree/525b92e82211/js/apps/admin-ui/test) | 95/100 | A | PASS |
-| [Storybook](https://github.com/storybookjs/storybook/tree/0365105152a6/code/e2e-internal) | 95/100 | A | PASS |
-| [Strapi](https://github.com/strapi/strapi/tree/142d9c6b31f3/tests/e2e) | 95/100 | A | PASS |
-| [freeCodeCamp](https://github.com/freeCodeCamp/freeCodeCamp/tree/42e6beeed4e9/e2e) | 94/100 | A | PASS |
-| [n8n](https://github.com/n8n-io/n8n/tree/4e3a7720cf85/packages/testing/playwright) | 94/100 | A | PASS |
-| [Documenso](https://github.com/documenso/documenso/tree/c81bc72c4c27/packages/app-tests) | 93/100 | A | PASS |
-| [WordPress Gutenberg](https://github.com/WordPress/gutenberg/tree/736cfd3daf75/test/e2e) | 93/100 | A | PASS |
-| [Supabase](https://github.com/supabase/supabase/tree/e1bdcc99dbf8/e2e) | 92/100 | A | PASS |
-| [Umami](https://github.com/umami-software/umami/tree/ec0ff50388c2/tests/e2e) | 92/100 | A | PASS |
-| [Coder](https://github.com/coder/coder/tree/47d3e1c52b14/site/e2e) | 91/100 | A | PASS |
-| [Twenty CRM](https://github.com/twentyhq/twenty/tree/710f1632ae96/packages/twenty-e2e-testing/tests) | 91/100 | A | PASS |
-| [Vendure](https://github.com/vendure-ecommerce/vendure/tree/07d595e1b844/packages/dashboard/e2e) | 91/100 | A | PASS |
-| [Wire (wire-webapp)](https://github.com/wireapp/wire-webapp/tree/9b0f1ea71e1d/apps/webapp/test/e2e_tests) | 91/100 | A | PASS |
-| [WooCommerce](https://github.com/woocommerce/woocommerce/tree/aa5308378124/plugins/woocommerce/tests/e2e) | 90/100 | A | PASS |
-| [Element (Matrix client)](https://github.com/element-hq/element-web/tree/3153cc6ae123/apps/web/playwright) | 89/100 | B | PASS |
-| [Clerk](https://github.com/clerk/javascript/tree/81a641348101/integration/tests) | 87/100 | B | PASS |
-| [Grafana](https://github.com/grafana/grafana/tree/1fd1d6398053/e2e-playwright) | 87/100 | B | PASS |
-| [PostHog](https://github.com/PostHog/posthog/tree/854a4262ec8b/playwright) | 87/100 | B | PASS |
-| [Formbricks](https://github.com/formbricks/formbricks/tree/55ade3bc2a5a/apps/web/playwright) | 85/100 | B | PASS |
-| [Bruno](https://github.com/usebruno/bruno/tree/f1b433f7b06f/tests) | 84/100 | B | PASS |
-| [Mattermost](https://github.com/mattermost/mattermost/tree/f0b63f8ed11e/e2e-tests/playwright) | 84/100 | B | PASS |
-| [Saleor Dashboard](https://github.com/saleor/saleor-dashboard/tree/3508127f50dc/playwright) | 84/100 | B | PASS |
-| [WordPress.com Calypso](https://github.com/Automattic/wp-calypso/tree/af6245bcd266/test/e2e) | 84/100 | B | PASS |
-| [sencho](https://github.com/Studio-Saelix/sencho/tree/489e1a9a9a90/e2e) | 84/100 | B | PASS |
-| [Handsontable](https://github.com/handsontable/handsontable/tree/40c2738b2e87/tests/e2e) | 83/100 | B | PASS |
-| [Rocket.Chat](https://github.com/RocketChat/Rocket.Chat/tree/828fa06f72d5/apps/meteor/tests/e2e) | 83/100 | B | PASS |
-| [novu](https://github.com/novuhq/novu/tree/6fc9d02ea001/apps/dashboard/tests) | 82/100 | B | PASS |
-| [Appsmith](https://github.com/appsmithorg/appsmith/tree/a72a95b73a99/app/client/playwright/tests) | 80/100 | B | PASS |
-| [Payload CMS](https://github.com/payloadcms/payload/tree/5f9bcac2f07b/test/admin/e2e) | 79/100 | C | FAIL |
-| [Immich](https://github.com/immich-app/immich/tree/f48d4b332127/e2e/src/specs) | 78/100 | C | FAIL |
-| [VS Code (component fixtures)](https://github.com/microsoft/vscode/tree/7b24b6303075/test/componentFixtures/playwright) | 77/100 | C | FAIL |
-| [Astro](https://github.com/withastro/astro/tree/40896acb7449/packages/astro/e2e) | 75/100 | C | FAIL |
-| [AFFiNE](https://github.com/toeverything/AFFiNE/tree/d897bb3d8409/tests/blocksuite/e2e) | 72/100 | C | FAIL |
-| [Builder.io SDKs](https://github.com/BuilderIO/builder/tree/51cd0521bbf1/packages/sdks-tests/src/e2e-tests) | 72/100 | C | FAIL |
-| [openplayerjs](https://github.com/openplayerjs/openplayerjs/tree/ee1673106237/e2e) | 72/100 | C | FAIL |
-| [tldraw](https://github.com/tldraw/tldraw/tree/a6cfe996e1df/apps/examples/e2e) | 72/100 | C | FAIL |
-| [cal.com](https://github.com/calcom/cal.com/tree/54343aa685ae/apps/web/playwright) | 71/100 | C | FAIL |
-| [Qwik](https://github.com/QwikDev/qwik/tree/02b5c6b18fe7/e2e) | 70/100 | C | FAIL |
-| [Wekan](https://github.com/wekan/wekan/tree/84414711b7ae/tests/playwright/specs) | 70/100 | C | FAIL |
-| [livecodes](https://github.com/live-codes/livecodes/tree/747054828256/e2e) | 70/100 | C | FAIL |
-| [OpenReplay](https://github.com/openreplay/openreplay/tree/3fce37d89113/frontend/tests/playwright) | 66/100 | D | FAIL |
-| [Flagsmith](https://github.com/Flagsmith/flagsmith/tree/66dc2e9ed00b/frontend/e2e/tests) | 62/100 | D | FAIL |
-| [Activepieces](https://github.com/activepieces/activepieces/tree/ab8cd5c9069e/packages/tests-e2e) | 58/100 | F | FAIL |
-| [TheCyberHub](https://github.com/th3cyb3rhub/TheCyberHub/tree/1cf418654f16/e2e) | 56/100 | F | FAIL |
-
-Repo names link to the exact source scanned. Full methodology, findings
-breakdown, and `scripts/validate-corpus.sh` to reproduce every number here
-yourself live in [VALIDATION.md](./VALIDATION.md). A self-contained visual
+| [Flagsmith/flagsmith](https://github.com/Flagsmith/flagsmith/tree/ea5ad826e85b/frontend/e2e/tests) | 62/100 | D | FAIL |
+| [activepieces/activepieces](https://github.com/activepieces/activepieces/tree/9e493a87f471/packages/tests-e2e) | 58/100 | F | FAIL |
+| [th3cyb3rhub/TheCyberHub](https://github.com/th3cyb3rhub/TheCyberHub/tree/1cf418654f16/e2e) | 56/100 | F | FAIL |
+| [DataDog/documentation](https://github.com/DataDog/documentation/tree/6cd10f40cf46/hugo/e2e) | 50/100 | F | FAIL |
+| [getsentry/spotlight](https://github.com/getsentry/spotlight/tree/3dd5eb5b315f/packages/spotlight/tests) | 33/100 | F | FAIL |
+
+Repo names link to the exact commit scanned. Selected by a mix of
+mechanical GitHub search (real `@playwright/test` usage, hand-verified)
+and, for the most recent 16, hand-picked brand recognition — three of the
+100 are named competitors' own example suites (Checkly, Currents,
+LambdaTest), scored with no separate bar. Full methodology, the complete
+100-repo table, findings breakdown, and
+`scripts/validate-corpus.sh` to reproduce every number here yourself live
+in
+[VALIDATION.md](https://github.com/qa-guardian/playwright-score/blob/main/VALIDATION.md).
+A machine-readable
+[`validation.json`](https://github.com/qa-guardian/playwright-score/blob/main/validation.json)
+carries the same per-repo data for tooling. See
+[CHANGELOG.md](https://github.com/qa-guardian/playwright-score/blob/main/CHANGELOG.md)
+for what changed release to release. A self-contained visual
 report of an earlier five-suite audit lives at
-[`docs/scorecard.html`](./docs/scorecard.html) — host it wherever
+[`docs/scorecard.html`](https://github.com/qa-guardian/playwright-score/blob/main/docs/scorecard.html) — host it wherever
 (GitHub Pages, qaguardian.com, ...).
 
-See [METHODOLOGY.md](./METHODOLOGY.md) for the full scoring model, or the product write-up on the [landing page](https://qaguardian.com/open-source/playwright-score).
+See [METHODOLOGY.md](https://github.com/qa-guardian/playwright-score/blob/main/METHODOLOGY.md) for the full scoring model, or the product write-up on the [landing page](https://qaguardian.com/open-source/playwright-score).
 
 ## Install
 

package/dist/eslint-runner.d.ts

@@ -9,4 +9,6 @@ export declare function runEslint(options: {
     cwd?: string;
     /** See findLocalAssertionHelperNames in metrics.ts. */
     assertFunctionNames?: string[];
+    /** See detectExtendedTestAliases in test-aliases.ts. */
+    testAliasNames?: string[];
 }): Promise<Finding[]>;

package/dist/eslint-runner.js

@@ -4,10 +4,28 @@ import { commonAncestorDir } from './fs-util.js';
 import playwright from 'eslint-plugin-playwright';
 import { basePlugin } from './rules/base-plugin.js';
 import { mapRule } from './profiles.js';
-function buildConfig(assertFunctionNames) {
+function buildConfig(assertFunctionNames, testAliasNames) {
     const base = [
         {
             files: ['**/*.{ts,tsx,js,jsx,mjs,cjs}'],
+            // Identifiers bound to a `test.extend(...)`-derived fixture under a
+            // custom name (test-aliases.ts) — upstream eslint-plugin-playwright
+            // already dereferences a *same-file* extend chain internally, but
+            // can't follow one across files (its own `dereference` is scoped to
+            // the current file); this setting is what makes a cross-file alias
+            // (`import { loggedTest } from './fixtures'`) resolve to `test` for
+            // every rule that keys off the literal identifier — no-standalone-
+            // expect, no-conditional-expect, no-conditional-in-test, valid-expect,
+            // no-focused-test, and others built on the same parseFnCall/
+            // resolveToPlaywrightFn helper. Verified against two real corpus
+            // repos (argos-ci/argos's `loggedTest`, apache/superset's
+            // `testWithAssets`) where this exact cross-file shape was the actual
+            // false-positive cause.
+            settings: {
+                playwright: {
+                    globalAliases: { test: testAliasNames },
+                },
+            },
             // A pre-2.0.0 review found `/* eslint-disable */` on line 1 of a
             // 10-test file (waitForTimeout + expect(true).toBe(true) in every
             // test) took it from 35/F to 99/A — every rule this package enables
@@ -83,7 +101,7 @@ function buildConfig(assertFunctionNames) {
                 // with the same severity for the actual anti-pattern (an
                 // always-skipped test declaration) only.
                 'playwright/no-skipped-test': 'off',
-                'pwscore/no-skipped-test-declaration': 'warn',
+                'pwscore/no-skipped-test-declaration': ['warn', { testAliasNames }],
                 'playwright/no-raw-locators': 'warn',
                 // Both were mapped in profiles.ts (report-only, so they don't
                 // double-penalize the ratio-based locators dimension) but never
@@ -135,7 +153,7 @@ function buildConfig(assertFunctionNames) {
                 'pwscore/no-timer-sleep': 'error',
                 'pwscore/no-coordinate-click': 'warn',
                 'pwscore/no-trivial-assertion': 'error',
-                'pwscore/no-soft-assertion-only-test': 'warn',
+                'pwscore/no-soft-assertion-only-test': ['warn', { testAliasNames }],
             },
         },
     ];
@@ -166,7 +184,7 @@ export async function runEslint(options) {
     const cwd = options.files.length > 0
         ? commonAncestorDir(options.files)
         : (options.cwd ?? process.cwd());
-    const overrideConfig = buildConfig(options.assertFunctionNames ?? []);
+    const overrideConfig = buildConfig(options.assertFunctionNames ?? [], options.testAliasNames ?? []);
     // Try to attach typescript-eslint parser for .ts files
     try {
         const tsParser = await import('@typescript-eslint/parser');

package/dist/formatters/markdown.js

@@ -32,5 +32,13 @@ export function formatMarkdown(result) {
         lines.push('');
         lines.push('</details>');
     }
+    if (result.configWarnings?.length) {
+        lines.push('');
+        lines.push('## Config warnings');
+        lines.push('');
+        for (const w of result.configWarnings) {
+            lines.push(`- ${w}`);
+        }
+    }
     return lines.join('\n');
 }

package/dist/formatters/text.js

@@ -36,6 +36,13 @@ export function formatText(result) {
             lines.push(`  … and ${result.skippedFiles.length - 20} more`);
         }
     }
+    if (result.configWarnings?.length) {
+        lines.push('');
+        lines.push('Config warnings:');
+        for (const w of result.configWarnings) {
+            lines.push(`  ${w}`);
+        }
+    }
     return lines.join('\n');
 }
 function pathBasename(p) {

package/dist/import-graph.d.ts

@@ -1,3 +1,4 @@
+export declare function resolveRelativeImport(fromFile: string, specifier: string, boundaryDir: string): string | undefined;
 /**
  * A Page Object Model is common in real Playwright suites: spec files
  * interact with the page entirely through methods on an imported class

package/dist/import-graph.js

@@ -35,7 +35,7 @@ function extractRelativeImportSpecifiers(source, file) {
     }
     return specifiers;
 }
-function resolveRelativeImport(fromFile, specifier, boundaryDir) {
+export function resolveRelativeImport(fromFile, specifier, boundaryDir) {
     const base = path.resolve(path.dirname(fromFile), specifier);
     const candidates = [];
     for (const ext of RESOLVE_EXTENSIONS)