@qaguardian/playwright-score 2.1.0 → 2.2.0

package/CHANGELOG.md

@@ -10,6 +10,187 @@ parsing accuracy, not grading policy — can still move a suite's score
 without itself being a new model version; 2.1.0 below is exactly that
 case, and says so explicitly.
 
+## 2.2.0 — 2026-11-14
+
+False-positive fixes found by hand-checking error-level findings on public
+suites (7 of 18 were wrong), then tightened after review so the fixes cannot be used to hide a test with no assertion. No rule's weight or severity changed, so this
+is not a new scoring model (still v4), but **scores can rise**: findings
+that were wrong are no longer reported. Nothing gets stricter.
+
+- `playwright/expect-expect`: a test that asserts through a fixture
+  function injected via the test args is no longer reported as having no
+  assertion, but only when the fixture is a callable that really asserts:
+  a `test.extend({ check: async ({ page }, use) => { await use(async () =>
+  { await expect(...) }) } })` entry whose function handed to `use(...)`
+  runs `expect`, the test destructures it AND calls it, and the fixtures
+  module is reachable from that spec through relative imports. An `expect`
+  in a fixture's setup or teardown does not count, nor does any fixture
+  named like a built-in one (`page`, `context`, `request`, ...). A test
+  also counts as asserting through an object-literal method / class-property
+  arrow function in a file it imports (a fixture factory's returned object,
+  a page object), except that a helper whose name collides with a Playwright
+  method (`goto`, `click`, `fill`, `check`, `get`, `post`, ...) is never
+  treated as an assertion helper, so it cannot turn every `page.goto()` into
+  an assertion. When a test has no `expect` of its own but calls an
+  assertion-named method (`assert`, `verify`, `check`, `validate`, `expect`,
+  `ensure`, `should`, `confirm` followed by an upper-case letter, `_`, a digit
+  or nothing, such as `checkTotals` but not `checkout`; or `toBe*`, `toHave*`,
+  `toContain*`, `toMatch*`, `toEqual*`, `toThrow*`) on an object this scan
+  cannot read (an injected fixture from another package, an unresolvable
+  import, a third-party package), it is no longer an error: it is reported
+  as the report-only finding `playwright-score/assertion-unverified`
+  ("needs a human check"), which does not lower the score and is not
+  counted in `summary.unassertedTests`. Those tests are counted separately
+  in `summary.unverifiedTests`, a new field that is always present in the
+  JSON output (`0` when nothing is unverified; the text and Markdown reports
+  show those tests as unverified, never as asserting): they are not known to
+  assert. A fixture a test calls, whose `test` comes from a module the scan
+  does not resolve (a tsconfig `paths` alias, a workspace package), is
+  unverified the same way. Names bound by destructuring
+  (`const { channelsPage } = await pw.login(user)`) take the kind of the
+  expression they come from. Query-style names (`is...`, `has...`,
+  `wait...`), helper methods named like Playwright actions (`fill`, `check`,
+  `goto`, ...), built-in calls (`new Date().toISOString()`,
+  `user.email.toLowerCase()`, `x.toString()`), `shop.checkout()` and
+  `loginPage.rememberMe.check()` are not treated as assertions: a call whose
+  name is not assertion-like, a test that only calls Playwright API on
+  `page`/`request`/..., and helpers whose body we can read and which do not
+  assert are still `expect-expect` errors.
+- `pwscore/no-trivial-assertion`: `expect(pageErrors).toEqual([])` where
+  `pageErrors` is a `const` array that may be mutated (passed to a call as
+  an argument, the receiver of `push`/`pop`/`splice`/`sort`/..., an
+  assignment target, aliased, or captured by a closure that does one of
+  those, such as a listener, or an inner array reached through an index:
+  `rows[0].push(9)`) is no longer flagged. A `.length` or index
+  read, a non-mutating method such as `.map()`, and an unrelated property
+  with the same name (`obj.expected`) do not count as mutation; literals
+  and a `const` array that is only read are still flagged.
+- `pwscore/no-timer-sleep`: a retry/backoff pause inside a plain helper
+  function (declaration, `const f = () =>`, class method) is no longer
+  flagged, but only when the sleep sits in a `catch` clause or in a loop
+  whose body contains a non-empty `try` with a `catch`, and the helper is not declared inside
+  a test, hook, describe or step callback. Sleeps directly in a test body or
+  hook, any loop without `try`/`catch` (including a hand-rolled
+  `while (await x.isVisible()) sleep()` polling wait), helpers declared
+  inside a test callback, and bare `sleep()` helpers are still flagged.
+- `playwright/no-conditional-in-test`: a guard whose only body is `throw`
+  (`if (!id) throw new Error(...)`, no `else`) is no longer flagged.
+- `playwright/valid-expect`: a helper whose name merely ends in `Expect`
+  (`clickSortableHeaderAndExpect(...)`) is no longer reported as an
+  `expect()` without a matcher.
+- Known and not fixed: `playwright/no-standalone-expect` on tests declared
+  through a factory function that returns a `test.extend(...)` result
+  (`const t = testWithConfig({...}); t('name', ...)`). The alias detection
+  only follows direct `test.extend` chains.
+
+## 2.1.1 — 2026-09-26
+
+Scoring model stays v4. Reviewer follow-ups from the 2.1.0 release review
+(QAG-230) — none of these blocked that release. They close symlink-escape
+gaps a crafted repo could otherwise use, quiet a false warning, and fix a
+false-positive drop the first round of these same fixes introduced. None
+of that changes what an ordinary suite with no symlinked spec files
+scores; the realpath dedupe fix below is the one exception — it can
+change the file/test count for a suite that has a symlinked spec file
+matching discovery alongside its own real target, and says so.
+
+### Fixed
+- **A match escaping the *repository* through a symlink is now dropped —
+  scoped to the repo root, not the narrower directory the caller pointed
+  at.** `testDirEscapesRepo` (added in 2.1.0) only clamps a config's
+  `testDir` itself before it becomes a glob root — it couldn't see a
+  symlinked directory the glob then walked *through* on the way to an
+  individual match. A `testMatch` entry that names a symlinked directory
+  by a literal path segment (not a `**` wildcard) is still followed even
+  with glob's `follow: false` (that option only stops `**` from
+  *expanding into* a symlinked directory, not a pattern from naming one
+  explicitly), so a crafted repo (`tests/evil -> /`, `testMatch:
+  ['evil/**/*.ts']`) could make discovery walk arbitrary parts of the
+  filesystem while every match still lexically looked like it lived
+  under the scanned directory. The same gap let an individually
+  symlinked spec file sitting directly in the repo (no `testMatch`
+  trickery needed) resolve to, and get scored from, a file outside it.
+  `src/index.ts`'s match loop realpath's every candidate match and drops
+  it unless it's still contained within the realpath of the **repository
+  root** (`safeRealpath(findRepoBoundary(abs))`) — not, as this package
+  first shipped it, the narrower directory the caller actually scanned.
+  That first version silently dropped a real, common monorepo shape too:
+  a spec reachable through an in-repo symlink that resolves to a sibling
+  directory *outside* the caller's scanned subpath (one package's specs
+  shared with another via a symlink) is legitimate and is kept; only a
+  match whose realpath lands outside the repository itself is dropped.
+  The same repo-root containment now also applies to a bare glob-pattern
+  input (e.g. `tests/*.spec.ts` passed directly to `scorePaths`, not a
+  directory it walks itself) — the first version of this fix only
+  covered the directory-scan branch, so switching an otherwise-identical
+  call from a directory argument to an equivalent glob argument silently
+  bypassed the guard entirely.
+- **The walk itself is now pruned at a symlink that escapes the repo
+  root, not just filtered after the fact.** The realpath containment
+  check above still has to pay to walk all the way through a symlinked
+  directory before dropping every match inside it one by one — for a
+  large or slow target (`tests/evil -> /`) that's a real cost, not just
+  a theoretical one (measured: ~6.4s to fully walk a large symlinked
+  directory before this fix). `RepoBoundaryIgnore` (`src/index.ts`) wraps
+  glob's own `ignore` matching with a `childrenIgnored` check that stops
+  the walk from ever descending into a symlinked directory whose
+  realpath lands outside the repo root, only paying the realpath cost at
+  an actual symlink (a plain nested directory can't walk outside a root
+  its own parent is already inside, so it's never checked).
+- **An in-repo symlinked spec and its own real target are no longer both
+  scored, when both independently match discovery under the same scan.**
+  Pre-existing, not new to the fixes above: `expandPaths` deduped a
+  literal path only against `explicit` inputs, never against another
+  `expanded` match that happens to resolve to the same realpath. A
+  symlink sitting alongside its target inside the same scanned directory
+  (or reachable via two different scan roots in one `scorePaths` call)
+  was counted, and scored, twice. Now deduped by realpath, keeping the
+  first lexical path in sorted order. **This can change file/test counts
+  for a repo that has such a link** — said plainly here rather than
+  folded into "doesn't change what a correctly-shaped suite scores",
+  since a suite with this exact shape is exactly what changes.
+- **False "testDir resolves outside the repository" warning on a
+  nonexistent `testDir` under a symlinked repo path.** `safeRealpath`
+  only resolves the portion of a path that exists on disk, so a
+  nonexistent `testDir` kept its literal, unresolved form while
+  `repoRoot` (which does exist) got fully resolved — on a host where the
+  repo itself sits under a symlinked prefix (macOS: `/tmp` ->
+  `/private/tmp`), that asymmetry alone made the two sides disagree and
+  fired a false escape warning for an ordinary missing `testDir`.
+  `testDirEscapesRepo` now checks existence first and returns "does not
+  escape" immediately for a `testDir` that isn't on disk — consistent
+  with `resolveConfigScopedRoot`'s own existing "nothing to score there"
+  handling for the same case.
+- **Escape warning now prints cwd-relative paths**, matching every other
+  `configWarning` in this package, instead of the runner's own absolute
+  filesystem paths.
+- **A raw glob-pattern input pointing at a different repo than `cwd` no
+  longer hard-fails with 0 matches.** The repo-root containment fix above
+  used `cwd`'s own repo boundary for a bare glob pattern regardless of
+  where the pattern actually pointed — an absolute pattern into a sibling
+  repo (`['/abs/repoB/e2e/*.spec.ts']`), or an equivalent `../`-relative
+  one, had every match rejected as an "escape" of `cwd`'s repo, where the
+  same target passed as a directory already scored fine. `src/index.ts`'s
+  glob-input branch now derives its containment boundary from *the
+  pattern's own* literal (non-magic) leading-directory prefix, resolved
+  against `cwd`, not from `cwd` itself — a literal prefix that lands
+  inside another repo entirely is scored against that repo's own
+  boundary, while a symlink an in-bounds prefix still reaches that escapes
+  *that* repo is still dropped.
+- **An explicit target and a directory scan that reaches it again through
+  a symlink are no longer scored twice.** The realpath dedupe added above
+  only ever populated its map from `expanded` matches; an `explicit` file
+  and a same-realpath `expanded` symlink are two different literal paths,
+  so the separate "explicit wins" step (an exact literal-path delete)
+  never caught that pairing. The dedupe map is now seeded with
+  `explicit`'s own realpaths first, so a same-realpath `expanded` match is
+  recognized as a duplicate instead of surviving as a second, independent
+  file.
+
+Corpus rerun (100 public repos, directory inputs, run on the round-2 build;
+the round-3 fixes only touch glob-pattern inputs): 96/100 repos score
+identically, 4 moved upstream, 0 changed.
+
 ## 2.1.0 — 2026-09-24
 
 Scoring model stays v4 — nothing here changes a rule's weight or

package/METHODOLOGY.md

@@ -82,7 +82,7 @@ owns this dimension; counting them as demerits too would double-penalize.)
 | Dimension | Weight | Demerit sources |
 |---|---:|---|
 | playwrightHygiene | 40 | hard waits (`page.waitForTimeout`, `new Promise`+`setTimeout` and its respellings, `node:timers/promises` setTimeout), networkidle, force, coordinate-based mouse interaction (`*.mouse.click/dblclick/move/down/up(x, y)`, partial demerit — see below), missing await, element handles, eval, conditionals, page.pause, useless await |
-| assertions | 25 | a test with no recognized assertion is fully flawed (`expect-expect`, after all delegation resolution); an assertion whose subject/matcher combination can only ever pass (`expect(true).toBe(true)`, `expect([]).toEqual([])`, never a deliberate force-fail like `expect(true).toBe(false)`); `valid-expect`, `no-standalone-expect`, `prefer-web-first-assertions`, a soft-assertion-only test (`expect.poll` counts as hard, not soft) all add partial demerits |
+| assertions | 25 | a test with no recognized assertion is fully flawed (`expect-expect`, after all delegation resolution; a test with no `expect` of its own that calls an assertion-named method (`assert`, `verify`, `check`, `validate`, `expect`, `ensure`, `should`, `confirm` followed by an upper-case letter, `_`, a digit or nothing; or `toBe*`, `toHave*`, `toContain*`, `toMatch*`, `toEqual*`, `toThrow*`) on an object the scan cannot read, or calls a fixture that comes from a module the scan cannot resolve is a report-only `playwright-score/assertion-unverified` "needs a human check" finding, not a demerit, and is counted in `summary.unverifiedTests` rather than as asserting; any other call on such an object does not excuse the test, so query-style names (`is...`, `has...`, `wait...`), helper methods named like Playwright actions (`fill`, `check`, `goto`) and built-in calls are never treated as assertions; an assertion helper or fixture only counts when its body is readable, it is called by the test, and its name does not collide with a Playwright method); an assertion whose subject/matcher combination can only ever pass (`expect(true).toBe(true)`, `expect([]).toEqual([])`, never a deliberate force-fail like `expect(true).toBe(false)`, and never an array that may be mutated elsewhere: passed to a call, the receiver of a mutating method such as `push`, an assignment target, or captured by a closure such as an event listener doing so, including an inner array mutated through an index such as `rows[0].push(9)`); `valid-expect`, `no-standalone-expect`, `prefer-web-first-assertions`, a soft-assertion-only test (`expect.poll` counts as hard, not soft) all add partial demerits |
 | locators | 20 | usage ratio (above) |
 | structure | 15 | focused tests, always-skipped declarations, nested-describe depth, oversized files (module scope), an inline `eslint-disable` suppression comment (partial demerit — see "Known limitations" in VALIDATION.md) |
 

package/dist/finding-triage.d.ts

@@ -0,0 +1,13 @@
+import type { Finding } from './types.js';
+export declare const ASSERTION_UNVERIFIED_RULE = "playwright-score/assertion-unverified";
+export interface TriageContext {
+    /** Directory the findings' `file` paths are relative to. */
+    filesBase: string;
+    /** Scored spec files, absolute path -> source. */
+    sources: Map<string, string>;
+    /** Locally imported dependency files (page objects, fixtures), absolute path -> source. */
+    importedFiles: Map<string, string>;
+    /** Directory relative imports may not climb out of (defaults to filesBase). */
+    importBoundary?: string;
+}
+export declare function triageFindings(findings: Finding[], ctx: TriageContext): Finding[];