@qaguardian/playwright-score 0.5.0 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (17) hide show
  1. package/CHANGELOG.md +689 -0
  2. package/METHODOLOGY.md +54 -7
  3. package/README.md +57 -23
  4. package/dist/cli.js +8 -2
  5. package/dist/eslint-runner.js +36 -0
  6. package/dist/index.d.ts +1 -1
  7. package/dist/index.js +12 -27
  8. package/dist/metrics.d.ts +13 -0
  9. package/dist/metrics.js +71 -1
  10. package/dist/profiles.d.ts +15 -0
  11. package/dist/profiles.js +34 -0
  12. package/dist/rules/base-plugin.d.ts +4 -0
  13. package/dist/rules/base-plugin.js +657 -0
  14. package/dist/score-engine.d.ts +9 -1
  15. package/dist/score-engine.js +9 -1
  16. package/dist/types.d.ts +13 -1
  17. package/package.json +4 -3
package/METHODOLOGY.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Playwright Spec Score Methodology
2
2
 
3
- **Scoring model:** v3 (2026-08-27)
3
+ **Scoring model:** v4 (2026-09-23)
4
4
  **Package:** `playwright-score`
5
5
  **Landing page:** https://qaguardian.com/open-source/playwright-score
6
6
  **Maintainer:** [QA Guardian](https://qaguardian.com)
@@ -8,9 +8,11 @@
8
8
  **AI-free:** no LLM calls in the scoring path
9
9
 
10
10
  Any change that alters what a given suite scores is a new model version
11
- (v4, …), documented in the CHANGELOG. Model history: v1/v2 (published as
11
+ (v5, …), documented in the CHANGELOG. Model history: v1/v2 (published as
12
12
  "sqs-v1"/"sqs-v2" through package 0.4.0) scored finding *density per line
13
- of code*; v3 replaced that entirely — see "Why v3" at the bottom.
13
+ of code*; v3 replaced that entirely — see "Why v3" below. v4 keeps v3's
14
+ exact per-test ratio formula and changes only which findings feed it —
15
+ see "Why v4" below.
14
16
 
15
17
  ---
16
18
 
@@ -65,7 +67,7 @@ The **locators** dimension is independent of demerits — it is the usage
65
67
  ratio it always was:
66
68
 
67
69
  ```
68
- native = getByRole|getByLabel|getByTestId|getByText|getByPlaceholder calls
70
+ native = getByRole|getByLabel|getByTestId|getByText|getByPlaceholder|getByAltText|getByTitle calls
69
71
  raw = .locator(...) calls + legacy string-selector actions (page.click('#x'), …)
70
72
 
71
73
  locatorsScore = total === 0 ? 100 : round(100 × native / total)
@@ -79,10 +81,10 @@ owns this dimension; counting them as demerits too would double-penalize.)
79
81
 
80
82
  | Dimension | Weight | Demerit sources |
81
83
  |---|---:|---|
82
- | playwrightHygiene | 40 | hard waits, networkidle, force, missing await, element handles, eval, conditionals, page.pause, useless await |
83
- | assertions | 25 | a test with no recognized assertion is fully flawed (`expect-expect`, after all delegation resolution); `valid-expect`, `no-standalone-expect`, `prefer-web-first-assertions` add partial demerits |
84
+ | playwrightHygiene | 40 | hard waits (`page.waitForTimeout`, `new Promise`+`setTimeout` and its respellings, `node:timers/promises` setTimeout), networkidle, force, coordinate-based mouse interaction (`*.mouse.click/dblclick/move/down/up(x, y)`, partial demerit — see below), missing await, element handles, eval, conditionals, page.pause, useless await |
85
+ | assertions | 25 | a test with no recognized assertion is fully flawed (`expect-expect`, after all delegation resolution); an assertion whose subject/matcher combination can only ever pass (`expect(true).toBe(true)`, `expect([]).toEqual([])`, never a deliberate force-fail like `expect(true).toBe(false)`); `valid-expect`, `no-standalone-expect`, `prefer-web-first-assertions`, a soft-assertion-only test (`expect.poll` counts as hard, not soft) all add partial demerits |
84
86
  | locators | 20 | usage ratio (above) |
85
- | structure | 15 | focused tests, always-skipped declarations, nested-describe depth, oversized files (module scope) |
87
+ | structure | 15 | focused tests, always-skipped declarations, nested-describe depth, oversized files (module scope), an inline `eslint-disable` suppression comment (partial demerit — see "Known limitations" in VALIDATION.md) |
86
88
 
87
89
  ```
88
90
  score = round(Σ (weight_d / 100) × dimensionScore_d)
@@ -126,6 +128,51 @@ code scores identically regardless of which plugin version npm resolves.
126
128
 
127
129
  ---
128
130
 
131
+ ## Why v4
132
+
133
+ v4 keeps v3's per-test ratio formula unchanged. What changed is which
134
+ findings feed it:
135
+
136
+ 1. **Four gameability rules, planned as report-only under `standard`,
137
+ are real demerits there instead.** 1.1.0 (never published) planned
138
+ `pwscore/no-timer-sleep`,
139
+ `pwscore/no-coordinate-click`, `pwscore/no-trivial-assertion` and
140
+ `pwscore/no-soft-assertion-only-test` behind a separate opt-in `strict`
141
+ profile, so the public default (`standard`) wouldn't move anyone's
142
+ score. Owner direction, 2026-09-23: public scores should reflect
143
+ quality, full stop — a spec that sleeps on a timer, clicks a fixed
144
+ pixel coordinate, or asserts on a literal constant should score worse
145
+ under the default profile, not only under an opt-in one nobody has to
146
+ run. `strict` is removed as a result: with nothing left for it to gate
147
+ (every rule now scores the same way in the only remaining profile), a
148
+ second profile identical to the first was dead weight, not a real
149
+ choice. See CHANGELOG.md's 2.0.0 entry for the exact corpus repos this
150
+ moved, including pass→fail flips.
151
+ 2. **`pwscore/no-coordinate-click` is a warning (0.4 demerit), not an
152
+ error (1.0).** The 50-repo corpus surfaced a real false-positive
153
+ class: two infinite-canvas editors (tldraw, AFFiNE) accounted for
154
+ 428/459 of this rule's hits, and a fixed-pixel `page.mouse.click(x,
155
+ y)` is the *correct*, only way to interact with canvas content that
156
+ has no addressable DOM element — not a brittleness anti-pattern there.
157
+ A full error-level demerit would treat every legitimate canvas
158
+ interaction as a fully-flawed test; a partial one keeps it a real cost
159
+ (still visible, still moves the score) without a brittle "is this a
160
+ canvas app?" heuristic (filename/import sniffing) that would itself
161
+ become a new gameability surface — name your spec file to dodge it.
162
+ 3. **Inline `eslint-disable` suppression comments are now a demerit,
163
+ not invisible.** A pre-2.0.0 review found this project's own docs
164
+ recommending `// eslint-disable-next-line pwscore/no-coordinate-click`
165
+ as an escape hatch, while every rule this package ships can in fact be
166
+ silenced the same way — defeating the score for whatever it hides.
167
+ `pwscore/eslint-disable-comment` reports the presence of any
168
+ suppression directive (not, yet, which specific rule it targets — a
169
+ documented known limitation, see VALIDATION.md).
170
+
171
+ Together these move `standard` scores — see CHANGELOG.md's 2.0.0 entry
172
+ for the "scores changed" breakdown, and VALIDATION.md for the full
173
+ before/after table across all 50 corpus repos, including which ones flip
174
+ from pass to fail at the default threshold.
175
+
129
176
  ## Why v3
130
177
 
131
178
  v1/v2 scored findings per 25 lines of code through an exponential decay
package/README.md CHANGED
@@ -19,33 +19,67 @@ The score never calls an LLM. AI may *generate* or *repair* code using findings;
19
19
 
20
20
  ## Real-world results
21
21
 
22
- 17 public Playwright suites, scored with the published package against each
22
+ 50 public Playwright suites, scored with the published package against each
23
23
  project's actual source — chosen to include both well-known,
24
24
  heavily-engineered platforms and smaller, less mature projects. Not curated
25
- to look good: 5 of 17 fail the default threshold, for real, verifiable
26
- reasons (see [VALIDATION.md](./VALIDATION.md)).
25
+ to look good: 16 of 50 fail the default threshold, for real, verifiable
26
+ reasons (see [VALIDATION.md](./VALIDATION.md), which also has the full
27
+ before/after against the last published version).
27
28
 
28
- **17 suites · 1,228 files · 5,664 tests · 12/17 passed (80% threshold)**
29
+ **50 suites · 4,159 files · 20,183 tests · 34/50 passed (80% threshold)**
29
30
 
30
31
  | Repo (source scanned) | Score | Grade | Result |
31
32
  |---|---:|:-:|:-:|
32
- | [Playwright (own TodoMVC example)](https://github.com/microsoft/playwright/tree/main/examples/todomvc) | 98/100 | A | PASS |
33
- | [Storybook](https://github.com/storybookjs/storybook/tree/next/code/e2e-internal) | 95/100 | A | PASS |
34
- | [n8n](https://github.com/n8n-io/n8n/tree/master/packages/testing/playwright) | 95/100 | A | PASS |
35
- | [freeCodeCamp](https://github.com/freeCodeCamp/freeCodeCamp/tree/main/e2e) | 94/100 | A | PASS |
36
- | [dub](https://github.com/dubinc/dub/tree/main/apps/web/playwright) | 94/100 | A | PASS |
37
- | [Documenso](https://github.com/documenso/documenso/tree/main/packages/app-tests) | 93/100 | A | PASS |
38
- | [Supabase](https://github.com/supabase/supabase/tree/master/e2e) | 91/100 | A | PASS |
39
- | [PostHog](https://github.com/PostHog/posthog/tree/master/playwright) | 88/100 | B | PASS |
40
- | [Grafana](https://github.com/grafana/grafana/tree/main/e2e-playwright) | 87/100 | B | PASS |
41
- | [sencho](https://github.com/Studio-Saelix/sencho/tree/main/e2e) | 85/100 | B | PASS |
42
- | [Mattermost](https://github.com/mattermost/mattermost/tree/master/e2e-tests/playwright) | 83/100 | B | PASS |
43
- | [novu](https://github.com/novuhq/novu/tree/next/apps/dashboard/tests) | 82/100 | B | PASS |
44
- | [Immich](https://github.com/immich-app/immich/tree/main/e2e/src/specs) | 78/100 | C | FAIL |
45
- | [cal.com](https://github.com/calcom/cal.com/tree/main/apps/web/playwright) | 73/100 | C | FAIL |
46
- | [openplayerjs](https://github.com/openplayerjs/openplayerjs/tree/master/e2e) | 73/100 | C | FAIL |
47
- | [livecodes](https://github.com/live-codes/livecodes/tree/develop/e2e) | 70/100 | C | FAIL |
48
- | [TheCyberHub](https://github.com/th3cyb3rhub/TheCyberHub/tree/dev/e2e) | 56/100 | F | FAIL |
33
+ | [Plausible Analytics](https://github.com/plausible/analytics/tree/0f362bd84eaa/e2e) | 98/100 | A | PASS |
34
+ | [Playwright (own TodoMVC example)](https://github.com/microsoft/playwright/tree/0facd9234452/examples/todomvc) | 98/100 | A | PASS |
35
+ | [Penpot](https://github.com/penpot/penpot/tree/433f8774497a/frontend/playwright) | 96/100 | A | PASS |
36
+ | [SigNoz](https://github.com/SigNoz/signoz/tree/64fff60d7e8c/tests/e2e) | 96/100 | A | PASS |
37
+ | [Temporal UI](https://github.com/temporalio/ui/tree/ffee0375c167/tests) | 96/100 | A | PASS |
38
+ | [dub](https://github.com/dubinc/dub/tree/bd69439da068/apps/web/playwright) | 96/100 | A | PASS |
39
+ | [Keycloak (admin-ui)](https://github.com/keycloak/keycloak/tree/525b92e82211/js/apps/admin-ui/test) | 95/100 | A | PASS |
40
+ | [Storybook](https://github.com/storybookjs/storybook/tree/0365105152a6/code/e2e-internal) | 95/100 | A | PASS |
41
+ | [Strapi](https://github.com/strapi/strapi/tree/142d9c6b31f3/tests/e2e) | 95/100 | A | PASS |
42
+ | [freeCodeCamp](https://github.com/freeCodeCamp/freeCodeCamp/tree/42e6beeed4e9/e2e) | 94/100 | A | PASS |
43
+ | [n8n](https://github.com/n8n-io/n8n/tree/4e3a7720cf85/packages/testing/playwright) | 94/100 | A | PASS |
44
+ | [Documenso](https://github.com/documenso/documenso/tree/c81bc72c4c27/packages/app-tests) | 93/100 | A | PASS |
45
+ | [WordPress Gutenberg](https://github.com/WordPress/gutenberg/tree/736cfd3daf75/test/e2e) | 93/100 | A | PASS |
46
+ | [Supabase](https://github.com/supabase/supabase/tree/e1bdcc99dbf8/e2e) | 92/100 | A | PASS |
47
+ | [Umami](https://github.com/umami-software/umami/tree/ec0ff50388c2/tests/e2e) | 92/100 | A | PASS |
48
+ | [Coder](https://github.com/coder/coder/tree/47d3e1c52b14/site/e2e) | 91/100 | A | PASS |
49
+ | [Twenty CRM](https://github.com/twentyhq/twenty/tree/710f1632ae96/packages/twenty-e2e-testing/tests) | 91/100 | A | PASS |
50
+ | [Vendure](https://github.com/vendure-ecommerce/vendure/tree/07d595e1b844/packages/dashboard/e2e) | 91/100 | A | PASS |
51
+ | [Wire (wire-webapp)](https://github.com/wireapp/wire-webapp/tree/9b0f1ea71e1d/apps/webapp/test/e2e_tests) | 91/100 | A | PASS |
52
+ | [WooCommerce](https://github.com/woocommerce/woocommerce/tree/aa5308378124/plugins/woocommerce/tests/e2e) | 90/100 | A | PASS |
53
+ | [Element (Matrix client)](https://github.com/element-hq/element-web/tree/3153cc6ae123/apps/web/playwright) | 89/100 | B | PASS |
54
+ | [Clerk](https://github.com/clerk/javascript/tree/81a641348101/integration/tests) | 87/100 | B | PASS |
55
+ | [Grafana](https://github.com/grafana/grafana/tree/1fd1d6398053/e2e-playwright) | 87/100 | B | PASS |
56
+ | [PostHog](https://github.com/PostHog/posthog/tree/854a4262ec8b/playwright) | 87/100 | B | PASS |
57
+ | [Formbricks](https://github.com/formbricks/formbricks/tree/55ade3bc2a5a/apps/web/playwright) | 85/100 | B | PASS |
58
+ | [Bruno](https://github.com/usebruno/bruno/tree/f1b433f7b06f/tests) | 84/100 | B | PASS |
59
+ | [Mattermost](https://github.com/mattermost/mattermost/tree/f0b63f8ed11e/e2e-tests/playwright) | 84/100 | B | PASS |
60
+ | [Saleor Dashboard](https://github.com/saleor/saleor-dashboard/tree/3508127f50dc/playwright) | 84/100 | B | PASS |
61
+ | [WordPress.com Calypso](https://github.com/Automattic/wp-calypso/tree/af6245bcd266/test/e2e) | 84/100 | B | PASS |
62
+ | [sencho](https://github.com/Studio-Saelix/sencho/tree/489e1a9a9a90/e2e) | 84/100 | B | PASS |
63
+ | [Handsontable](https://github.com/handsontable/handsontable/tree/40c2738b2e87/tests/e2e) | 83/100 | B | PASS |
64
+ | [Rocket.Chat](https://github.com/RocketChat/Rocket.Chat/tree/828fa06f72d5/apps/meteor/tests/e2e) | 83/100 | B | PASS |
65
+ | [novu](https://github.com/novuhq/novu/tree/6fc9d02ea001/apps/dashboard/tests) | 82/100 | B | PASS |
66
+ | [Appsmith](https://github.com/appsmithorg/appsmith/tree/a72a95b73a99/app/client/playwright/tests) | 80/100 | B | PASS |
67
+ | [Payload CMS](https://github.com/payloadcms/payload/tree/5f9bcac2f07b/test/admin/e2e) | 79/100 | C | FAIL |
68
+ | [Immich](https://github.com/immich-app/immich/tree/f48d4b332127/e2e/src/specs) | 78/100 | C | FAIL |
69
+ | [VS Code (component fixtures)](https://github.com/microsoft/vscode/tree/7b24b6303075/test/componentFixtures/playwright) | 77/100 | C | FAIL |
70
+ | [Astro](https://github.com/withastro/astro/tree/40896acb7449/packages/astro/e2e) | 75/100 | C | FAIL |
71
+ | [AFFiNE](https://github.com/toeverything/AFFiNE/tree/d897bb3d8409/tests/blocksuite/e2e) | 72/100 | C | FAIL |
72
+ | [Builder.io SDKs](https://github.com/BuilderIO/builder/tree/51cd0521bbf1/packages/sdks-tests/src/e2e-tests) | 72/100 | C | FAIL |
73
+ | [openplayerjs](https://github.com/openplayerjs/openplayerjs/tree/ee1673106237/e2e) | 72/100 | C | FAIL |
74
+ | [tldraw](https://github.com/tldraw/tldraw/tree/a6cfe996e1df/apps/examples/e2e) | 72/100 | C | FAIL |
75
+ | [cal.com](https://github.com/calcom/cal.com/tree/54343aa685ae/apps/web/playwright) | 71/100 | C | FAIL |
76
+ | [Qwik](https://github.com/QwikDev/qwik/tree/02b5c6b18fe7/e2e) | 70/100 | C | FAIL |
77
+ | [Wekan](https://github.com/wekan/wekan/tree/84414711b7ae/tests/playwright/specs) | 70/100 | C | FAIL |
78
+ | [livecodes](https://github.com/live-codes/livecodes/tree/747054828256/e2e) | 70/100 | C | FAIL |
79
+ | [OpenReplay](https://github.com/openreplay/openreplay/tree/3fce37d89113/frontend/tests/playwright) | 66/100 | D | FAIL |
80
+ | [Flagsmith](https://github.com/Flagsmith/flagsmith/tree/66dc2e9ed00b/frontend/e2e/tests) | 62/100 | D | FAIL |
81
+ | [Activepieces](https://github.com/activepieces/activepieces/tree/ab8cd5c9069e/packages/tests-e2e) | 58/100 | F | FAIL |
82
+ | [TheCyberHub](https://github.com/th3cyb3rhub/TheCyberHub/tree/1cf418654f16/e2e) | 56/100 | F | FAIL |
49
83
 
50
84
  Repo names link to the exact source scanned. Full methodology, findings
51
85
  breakdown, and `scripts/validate-corpus.sh` to reproduce every number here
@@ -148,8 +182,8 @@ sets:
148
182
 
149
183
  - **`warn`**: log score; never fail the run; findings still inject into heal/generate repair prompts
150
184
  - **`gate`**: fail the run when score < threshold
151
- - After AI **generate**, specs below threshold get one automatic **heal** pass with score findings
152
- - **Heal** prompts always include score findings when the scorer reports issues
185
+ - After AI **generate**, every spec is scored; a spec below threshold is repaired with its score findings and re-scored, up to 3 scored attempts (the initial generation plus up to two repairs). A spec still below threshold, or one that could not be scored, is flagged for Guardian review and never silently accepted.
186
+ - **Heal** and repair prompts always include score findings when the scorer reports issues.
153
187
 
154
188
  ```bash
155
189
  # Local from monorepo
package/dist/cli.js CHANGED
@@ -18,7 +18,8 @@ Usage:
18
18
  npx playwright-score <paths...> [options]
19
19
 
20
20
  Options:
21
- --profile <standard> Scoring profile (default, and only: standard)
21
+ --profile <standard> Scoring profile (default, and only
22
+ supported value, as of 2.0.0: standard).
22
23
  --threshold <n> Pass threshold 0-100 (default: 80)
23
24
  --format <text|json|markdown|sarif> Output format (default: text)
24
25
  --out <file> Write report to file
@@ -31,7 +32,7 @@ Exit codes:
31
32
  2 tool error
32
33
 
33
34
  Website: https://qaguardian.com/open-source/playwright-score
34
- Methodology: METHODOLOGY.md (scoring model v3)
35
+ Methodology: METHODOLOGY.md (scoring model v4)
35
36
  `);
36
37
  }
37
38
  // Deliberately never process.exit() right after a console.log/console.error
@@ -76,6 +77,11 @@ async function main() {
76
77
  process.exitCode = 2;
77
78
  return;
78
79
  }
80
+ if (p === 'strict') {
81
+ console.error('Profile "strict" was removed in 2.0.0 — the four gameability checks it used to count on top of "standard" (timer-sleep sleeps, coordinate clicks, trivial assertions, soft-assertion-only tests) are now scored under "standard" directly, so a separate stricter profile has nothing left to gate. See CHANGELOG.md. Use --profile standard.');
82
+ process.exitCode = 2;
83
+ return;
84
+ }
79
85
  if (p !== 'standard') {
80
86
  console.error(`Invalid profile: ${p}`);
81
87
  process.exitCode = 2;
package/dist/eslint-runner.js CHANGED
@@ -8,6 +8,18 @@ function buildConfig(assertFunctionNames) {
8
8
  const base = [
9
9
  {
10
10
  files: ['**/*.{ts,tsx,js,jsx,mjs,cjs}'],
11
+ // A pre-2.0.0 review found `/* eslint-disable */` on line 1 of a
12
+ // 10-test file (waitForTimeout + expect(true).toBe(true) in every
13
+ // test) took it from 35/F to 99/A — every rule this package enables
14
+ // is an ordinary ESLint rule, so an ordinary inline disable comment
15
+ // silences it exactly like it would in any host project, defeating
16
+ // every one of these rules at will. noInlineConfig makes every
17
+ // eslint-disable/-line/-next-line comment (and enable/env/global)
18
+ // inert for this run's rule set; pwscore/eslint-disable-comment
19
+ // (metrics.ts) still reports the comment itself as a demerit — see
20
+ // eslint-disable-comment's docs — but it can no longer also silence
21
+ // whatever it's aimed at.
22
+ linterOptions: { noInlineConfig: true },
11
23
  languageOptions: {
12
24
  ecmaVersion: 2022,
13
25
  sourceType: 'module',
@@ -112,6 +124,18 @@ function buildConfig(assertFunctionNames) {
112
124
  'playwright/valid-expect': 'error',
113
125
  'playwright/no-standalone-expect': 'error',
114
126
  'playwright/max-nested-describe': 'warn',
127
+ // QAG-196 gameability fixes (2026-09-21), scored unconditionally
128
+ // as of 2.0.0/model v4 — see base-plugin.ts for what each one
129
+ // catches and profiles.ts for the severity → demerit mapping.
130
+ // no-coordinate-click is 'warn' here too (not 'error'): profiles.ts
131
+ // overrides its scoring severity to 'warning' regardless (the
132
+ // canvas-app false-positive mitigation), so setting 'error' here
133
+ // would only mislead the raw ESLint severity shown in some
134
+ // formatters without changing what it actually costs.
135
+ 'pwscore/no-timer-sleep': 'error',
136
+ 'pwscore/no-coordinate-click': 'warn',
137
+ 'pwscore/no-trivial-assertion': 'error',
138
+ 'pwscore/no-soft-assertion-only-test': 'warn',
115
139
  },
116
140
  },
117
141
  ];
@@ -189,6 +213,18 @@ export async function runEslint(options) {
189
213
  if (/^Definition for rule '.*' was not found\.$/.test(msg.message)) {
190
214
  continue;
191
215
  }
216
+ // noInlineConfig (above) reports every inline directive comment
217
+ // itself as a `ruleId: null` message ("'...' has no effect because
218
+ // you have 'noInlineConfig' setting in your config.") — already
219
+ // caught by the generic `!msg.ruleId` skip below, but matched
220
+ // explicitly here too (same treatment as "Definition for rule",
221
+ // above) so this stays true even if that generic skip is ever
222
+ // narrowed: it's a diagnostic about our own noInlineConfig setting,
223
+ // not the spec's quality, and pwscore/eslint-disable-comment already
224
+ // reports the comment itself as a real finding.
225
+ if (/has no effect because you have 'noInlineConfig' setting in your config\.$/.test(msg.message)) {
226
+ continue;
227
+ }
192
228
  if (msg.fatal) {
193
229
  // A file that doesn't parse can't be meaningfully scored at all —
194
230
  // surface it explicitly rather than silently dropping it (it has
package/dist/index.d.ts CHANGED
@@ -2,7 +2,7 @@ import type { ScoreOptions, ScoreResult } from './types.js';
2
2
  export type { ScoreResult, ScoreOptions, Finding, ProfileName } from './types.js';
3
3
  export { computeScore, MODEL, locatorsScore, gradeFromScore, ratioDimensionScore, } from './score-engine.js';
4
4
  export { countSloc } from './sloc.js';
5
- export { countLocators, analyzeSource } from './metrics.js';
5
+ export { countLocators, analyzeSource, attributeFindingScopes, getTestSpans, } from './metrics.js';
6
6
  export { formatJson } from './formatters/json.js';
7
7
  export { formatText } from './formatters/text.js';
8
8
  export { formatMarkdown } from './formatters/markdown.js';
package/dist/index.js CHANGED
@@ -4,12 +4,12 @@ import { globSync } from 'glob';
4
4
  import { runEslint } from './eslint-runner.js';
5
5
  import { commonAncestorDir } from './fs-util.js';
6
6
  import { collectLocallyImportedFiles } from './import-graph.js';
7
- import { analyzeSource, countLocators, findLocalAssertionHelperNames, getTestSpans, looksLikeNonPlaywrightTest, } from './metrics.js';
7
+ import { analyzeSource, attributeFindingScopes, countLocators, findLocalAssertionHelperNames, getTestSpans, looksLikeNonPlaywrightTest, } from './metrics.js';
8
8
  import { DEFAULT_THRESHOLDS } from './profiles.js';
9
9
  import { computeScore } from './score-engine.js';
10
10
  export { computeScore, MODEL, locatorsScore, gradeFromScore, ratioDimensionScore, } from './score-engine.js';
11
11
  export { countSloc } from './sloc.js';
12
- export { countLocators, analyzeSource } from './metrics.js';
12
+ export { countLocators, analyzeSource, attributeFindingScopes, getTestSpans, } from './metrics.js';
13
13
  export { formatJson } from './formatters/json.js';
14
14
  export { formatText } from './formatters/text.js';
15
15
  export { formatMarkdown } from './formatters/markdown.js';
@@ -23,8 +23,13 @@ export { formatSarif } from './formatters/sarif.js';
23
23
  // metrics.ts) is the safety net for a repo that uses `.e2e.` for a
24
24
  // *different* framework — Cypress specifically was common enough here to
25
25
  // get its own explicit check alongside the existing Jest/Vitest/RTL ones.
26
+ // `.pw.` joined the list from the 50-repo corpus expansion (QAG-196,
27
+ // 2026-09-21) — Flagsmith's entire 20-spec frontend/e2e suite
28
+ // (billing-test.pw.ts, flag-tests.pw.ts, ...) hard-failed with "no files
29
+ // matched" without it, the same failure mode cal.com/Immich already fixed
30
+ // for their own suffixes.
26
31
  const SPEC_GLOBS = [
27
- '**/*.{spec,test,e2e,e2e-spec,e2e-test}.{ts,tsx,js,jsx}',
32
+ '**/*.{spec,test,e2e,e2e-spec,e2e-test,pw}.{ts,tsx,js,jsx}',
28
33
  '**/*.spec.ts',
29
34
  '**/*.test.ts',
30
35
  ];
@@ -110,7 +115,7 @@ function expandPaths(inputs, cwd) {
110
115
  }
111
116
  function hardFail(profile, threshold, findings, extra = {}, skippedFiles) {
112
117
  return {
113
- scoreVersion: 'v3',
118
+ scoreVersion: 'v4',
114
119
  profile,
115
120
  score: 0,
116
121
  grade: 'F',
@@ -260,29 +265,9 @@ export async function scorePaths(options) {
260
265
  raw += counts.raw;
261
266
  }
262
267
  const findings = [...eslintFindings, ...metricFindings];
263
- // Attribute each finding to the test it sits in (model v3): a finding
264
- // inside a test span belongs to that test; inside a hook span it affects
265
- // every test in the file; anything else — imports, describe bodies,
266
- // file-level metrics like oversized-file — is module scope, counted once
267
- // per file.
268
- for (const f of findings) {
269
- const spans = fileSpans.get(f.file);
270
- if (!spans || f.line === undefined) {
271
- f.scope = 'module';
272
- continue;
273
- }
274
- const idx = spans.tests.findIndex((t) => f.line !== undefined && f.line >= t.startLine && f.line <= t.endLine);
275
- if (idx >= 0) {
276
- f.scope = 'test';
277
- f.testKey = `${f.file}#${idx}`;
278
- }
279
- else if (spans.hooks.some((h) => f.line !== undefined && f.line >= h.startLine && f.line <= h.endLine)) {
280
- f.scope = 'hook';
281
- }
282
- else {
283
- f.scope = 'module';
284
- }
285
- }
268
+ // Attribute each finding to the test it sits in (model v3) — see
269
+ // attributeFindingScopes in metrics.ts.
270
+ attributeFindingScopes(findings, fileSpans);
286
271
  const parseErrors = findings.filter((f) => f.rule === 'playwright-score/parse-error');
287
272
  if (parseErrors.length > 0) {
288
273
  // A file that isn't valid JS/TS can't be executed, let alone scored on
package/dist/metrics.d.ts CHANGED
@@ -63,9 +63,22 @@ export interface TestSpans {
63
63
  * malformed anyway) are dropped so each line maps to at most one test.
64
64
  */
65
65
  export declare function getTestSpans(source: string, file?: string): TestSpans;
66
+ /**
67
+ * Model-v3 finding attribution (see score-engine.ts): a finding inside a
68
+ * test span belongs to that test (`scope: 'test'` + a stable
69
+ * `file#index` testKey); inside a hook span it affects every test in the
70
+ * file (`scope: 'hook'`); anything else — imports, describe bodies,
71
+ * file-level metrics — is `scope: 'module'`, counted once per file.
72
+ * Exported so layers composing extra rules on top (e.g. a private
73
+ * house-rules dimension) attribute their own findings with the same
74
+ * logic the package itself uses, rather than reimplementing it.
75
+ * Mutates the findings in place and returns them.
76
+ */
77
+ export declare function attributeFindingScopes(findings: Finding[], spansByFile: Map<string, TestSpans>): Finding[];
66
78
  /** Count test( / test.only( / test.skip( / test.fixme( style declarations (approx). */
67
79
  export declare function countTests(source: string): number;
68
80
  export declare function findOversizedFile(source: string, file: string, maxSloc?: number): Finding[];
81
+ export declare function findEslintDisableComments(source: string, file: string): Finding[];
69
82
  export declare function analyzeSource(source: string, file: string): {
70
83
  sloc: number;
71
84
  tests: number;
package/dist/metrics.js CHANGED
@@ -35,12 +35,18 @@ const CYPRESS_API_RE = /\bcy\.(?:visit|get|contains|click|type|should|intercept|
35
35
  export function looksLikeNonPlaywrightTest(source) {
36
36
  return NON_PLAYWRIGHT_IMPORT_RE.test(source) || RTL_API_RE.test(source) || CYPRESS_API_RE.test(source);
37
37
  }
38
+ // The full getBy* family Playwright ships. getByAltText/getByTitle are
39
+ // just as native as the rest — leaving them out meant an image-heavy suite
40
+ // (alt/title locators plus one .locator() fallback) scored 0 on the
41
+ // locators dimension for perfectly idiomatic code.
38
42
  const NATIVE_METHODS = new Set([
39
43
  'getByRole',
40
44
  'getByLabel',
41
45
  'getByTestId',
42
46
  'getByText',
43
47
  'getByPlaceholder',
48
+ 'getByAltText',
49
+ 'getByTitle',
44
50
  ]);
45
51
  /**
46
52
  * Generic ESTree walk (no visitor-keys dependency). `parent` is excluded so
@@ -382,6 +388,39 @@ export function getTestSpans(source, file) {
382
388
  }
383
389
  return { tests: outer, hooks };
384
390
  }
391
+ /**
392
+ * Model-v3 finding attribution (see score-engine.ts): a finding inside a
393
+ * test span belongs to that test (`scope: 'test'` + a stable
394
+ * `file#index` testKey); inside a hook span it affects every test in the
395
+ * file (`scope: 'hook'`); anything else — imports, describe bodies,
396
+ * file-level metrics — is `scope: 'module'`, counted once per file.
397
+ * Exported so layers composing extra rules on top (e.g. a private
398
+ * house-rules dimension) attribute their own findings with the same
399
+ * logic the package itself uses, rather than reimplementing it.
400
+ * Mutates the findings in place and returns them.
401
+ */
402
+ export function attributeFindingScopes(findings, spansByFile) {
403
+ for (const f of findings) {
404
+ const spans = spansByFile.get(f.file);
405
+ if (!spans || f.line === undefined) {
406
+ f.scope = 'module';
407
+ continue;
408
+ }
409
+ const line = f.line;
410
+ const idx = spans.tests.findIndex((t) => line >= t.startLine && line <= t.endLine);
411
+ if (idx >= 0) {
412
+ f.scope = 'test';
413
+ f.testKey = `${f.file}#${idx}`;
414
+ }
415
+ else if (spans.hooks.some((h) => line >= h.startLine && line <= h.endLine)) {
416
+ f.scope = 'hook';
417
+ }
418
+ else {
419
+ f.scope = 'module';
420
+ }
421
+ }
422
+ return findings;
423
+ }
385
424
  const TEST_NON_DECL_RE = /\btest\.(describe|step|beforeEach|afterEach|beforeAll|afterAll|use|setTimeout|info|expect)\b/;
386
425
  /** Count test( / test.only( / test.skip( / test.fixme( style declarations (approx). */
387
426
  export function countTests(source) {
@@ -412,6 +451,37 @@ export function findOversizedFile(source, file, maxSloc = 400) {
412
451
  }
413
452
  return [];
414
453
  }
454
+ /**
455
+ * Matches every ESLint suppression-directive comment: `// eslint-disable`,
456
+ * `// eslint-disable-line`, `// eslint-disable-next-line`, and their
457
+ * `/* ... *\/` block-comment equivalents. Deliberately does NOT parse which
458
+ * rule(s) it names or narrow to only our own rule ids — a pre-2.0.0 review
459
+ * found this project's own docs (VALIDATION.md) recommending
460
+ * `eslint-disable-next-line pwscore/no-coordinate-click` as "the correct
461
+ * escape hatch", while every one of this project's rules can in fact be
462
+ * silenced this way, defeating the score entirely for whatever it hides.
463
+ * Reporting the comment itself, without attempting to resolve exactly what
464
+ * it suppresses, is a known-coarse but honest signal — see VALIDATION.md's
465
+ * "Known limitations" for the plan to make it precise. Scored (a real
466
+ * demerit, not report-only) as of 2.0.0/model v4 — see CHANGELOG.md.
467
+ */
468
+ const ESLINT_DISABLE_COMMENT_RE = /\/[/*]\s*eslint-disable(?:-next-line|-line)?\b/g;
469
+ export function findEslintDisableComments(source, file) {
470
+ const findings = [];
471
+ ESLINT_DISABLE_COMMENT_RE.lastIndex = 0;
472
+ let m;
473
+ while ((m = ESLINT_DISABLE_COMMENT_RE.exec(source)) !== null) {
474
+ findings.push({
475
+ rule: 'pwscore/eslint-disable-comment',
476
+ severity: 'warning',
477
+ message: 'Inline ESLint suppression comment. Disabling a rule hides whatever finding it silences from every tool that runs it, including this scorer — fix the underlying issue, or if the finding is a genuine false positive, report it as a bug in this project rather than relying on a permanent local workaround.',
478
+ file,
479
+ line: source.slice(0, m.index).split('\n').length,
480
+ dimension: 'structure',
481
+ });
482
+ }
483
+ return findings;
484
+ }
415
485
  export function analyzeSource(source, file) {
416
486
  return {
417
487
  sloc: countSloc(source),
@@ -426,6 +496,6 @@ export function analyzeSource(source, file) {
426
496
  // call and so false-positived at error severity on any file whose only
427
497
  // assertions were expect.poll()/expect.soft() — verified against real
428
498
  // production code.
429
- findings: [...findOversizedFile(source, file)],
499
+ findings: [...findOversizedFile(source, file), ...findEslintDisableComments(source, file)],
430
500
  };
431
501
  }
package/dist/profiles.d.ts CHANGED
@@ -1,4 +1,19 @@
1
1
  import type { DimensionName, ProfileName } from './types.js';
2
+ /**
3
+ * `strict` was planned for 1.1.0 (never published) to gate four
4
+ * contentious rules behind an opt-in profile, report-only under
5
+ * `standard`. 2.0.0 (model v4, see
6
+ * CHANGELOG.md) scores all of them under `standard` directly — owner
7
+ * direction: public scores should reflect quality, full stop, not be
8
+ * softened to avoid moving anyone's number. With nothing left for a
9
+ * second profile to gate (every rule now counts the same way regardless
10
+ * of which profile runs), `strict` would be byte-for-byte identical to
11
+ * `standard` — dead weight, not a real choice — so it's removed rather
12
+ * than kept as a confusing no-op alias. `--profile strict` errors with a
13
+ * migration message (see cli.ts); the library API falls back to
14
+ * `standard` weights for any unrecognized profile string, same as the
15
+ * `guardian` profile's removal in 1.0.0.
16
+ */
2
17
  export declare const PROFILE_WEIGHTS: Record<ProfileName, Partial<Record<DimensionName, number>>>;
3
18
  export declare const DEFAULT_THRESHOLDS: Record<ProfileName, number>;
4
19
  /** Map eslint ruleId → dimension + whether report-only for scoring */
package/dist/profiles.js CHANGED
@@ -1,3 +1,18 @@
1
+ /**
2
+ * `strict` was planned for 1.1.0 (never published) to gate four
3
+ * contentious rules behind an opt-in profile, report-only under
4
+ * `standard`. 2.0.0 (model v4, see
5
+ * CHANGELOG.md) scores all of them under `standard` directly — owner
6
+ * direction: public scores should reflect quality, full stop, not be
7
+ * softened to avoid moving anyone's number. With nothing left for a
8
+ * second profile to gate (every rule now counts the same way regardless
9
+ * of which profile runs), `strict` would be byte-for-byte identical to
10
+ * `standard` — dead weight, not a real choice — so it's removed rather
11
+ * than kept as a confusing no-op alias. `--profile strict` errors with a
12
+ * migration message (see cli.ts); the library API falls back to
13
+ * `standard` weights for any unrecognized profile string, same as the
14
+ * `guardian` profile's removal in 1.0.0.
15
+ */
1
16
  export const PROFILE_WEIGHTS = {
2
17
  standard: {
3
18
  playwrightHygiene: 40,
@@ -77,6 +92,25 @@ export const ESLINT_RULE_MAP = {
77
92
  dimension: 'structure',
78
93
  severityOverride: 'warning',
79
94
  },
95
+ // v4 (2.0.0): the four QAG-196 gameability rules and the
96
+ // eslint-disable-comment finding are scored the same as everything
97
+ // else above — no more report-only/profile split. See CHANGELOG.md's
98
+ // 2.0.0 entry and METHODOLOGY.md's "Why v4".
99
+ 'pwscore/no-timer-sleep': { dimension: 'playwrightHygiene' },
100
+ // Demoted to warning (0.4 demerit, not 1.0): page.mouse.click/move/down/
101
+ // up(x, y) is also the *correct* way to interact with a canvas/drawing
102
+ // surface (no addressable DOM element to target with a locator), not
103
+ // just a brittle-DOM-app anti-pattern. Verified against the 50-repo
104
+ // corpus: two infinite-canvas editors (tldraw, AFFiNE) accounted for
105
+ // 428/459 of this rule's hits pre-v4. A full per-test demerit would
106
+ // treat every legitimate canvas interaction as a full flaw; a partial
107
+ // one still costs real points (a canvas-heavy test loses up to 0.4 per
108
+ // dimension-capped test) without a brittle "is this a canvas app?"
109
+ // filename/import heuristic that would itself become a new gameability
110
+ // surface (name your test file "canvas-*" to suppress the rule).
111
+ 'pwscore/no-coordinate-click': { dimension: 'playwrightHygiene', severityOverride: 'warning' },
112
+ 'pwscore/no-trivial-assertion': { dimension: 'assertions' },
113
+ 'pwscore/no-soft-assertion-only-test': { dimension: 'assertions', severityOverride: 'warning' },
80
114
  };
81
115
  export function mapRule(ruleId) {
82
116
  if (ESLINT_RULE_MAP[ruleId])
package/dist/rules/base-plugin.d.ts CHANGED
@@ -10,5 +10,9 @@ export declare const basePlugin: {
10
10
  };
11
11
  rules: {
12
12
  'no-skipped-test-declaration': Rule.RuleModule;
13
+ 'no-timer-sleep': Rule.RuleModule;
14
+ 'no-coordinate-click': Rule.RuleModule;
15
+ 'no-trivial-assertion': Rule.RuleModule;
16
+ 'no-soft-assertion-only-test': Rule.RuleModule;
13
17
  };
14
18
  };