@qaguardian/playwright-score 0.2.0 → 0.4.0

package/METHODOLOGY.md

@@ -1,13 +1,18 @@
 # Playwright Spec Score (SQS) Methodology
 
-**Score version:** `sqs-v1`  
+**Score version:** `sqs-v2`  
 **Package:** `playwright-score`  
 **Landing page:** https://qaguardian.com/open-source/playwright-score  
 **Maintainer:** [QA Guardian](https://qaguardian.com)  
 **Deterministic:** same inputs → same score  
 **AI-free:** no LLM calls in the scoring path
 
-Any change to formulas, weights, SLOC definition, caps, or constants requires a new version (`sqs-v2`, …).
+Any change to formulas, weights, SLOC definition, caps, or constants requires a new version (`sqs-v3`, …).
+
+**sqs-v2 (2026-08-27)** changed exactly one thing versus sqs-v1: the
+assertions dimension is now a per-test coverage ratio instead of a
+finding-density penalty (see the Assertions dimension section). All other
+formulas, weights, and constants are unchanged from sqs-v1.
 
 ---
 
@@ -40,6 +45,43 @@ assertion-delegation tracing, Page Object Model import resolution).
 
 ---
 
+## Assertions dimension (coverage ratio × quality decay)
+
+sqs-v1 fed "test has no assertions" through the same density math as
+hygiene smells. That misread coverage badly on small suites: 2 of 3 tests
+asserting nothing scored 82/100 on this dimension, because two findings in
+a tiny file barely register per-SLOC. Whether a test asserts anything is a
+fraction of tests, not a findings-per-SLOC rate — so, like the locators
+dimension, the primary signal is a ratio:
+
+```
+unassertedTests = count(playwright/expect-expect findings)   // UNCAPPED census
+coverage        = tests === 0 ? 1 : clamp01((tests - unassertedTests) / tests)
+
+auxLoad         = penalty load (same units/caps/slots as below) of the
+                  OTHER assertion rules only: valid-expect,
+                  no-standalone-expect, prefer-web-first-assertions
+
+assertionsScore = clamp(0, 100, round(100 * coverage * e^(-K * auxLoad)))
+```
+
+Notes:
+
+- `expect-expect` findings are excluded from the density term — they ARE
+  the coverage term; counting them twice would double-penalize.
+- The census is deliberately uncapped: the per-(file, rule) cap exists to
+  stop one repeated smell from nuking a density score, but hiding 2 of 5
+  assertion-free tests would misstate coverage.
+- All of expect-expect's delegation machinery (same-file helper discovery,
+  imported-helper tracing, `assertFunctionPatterns`) applies before the
+  census is taken, so delegated assertions do not count as unasserted.
+- `tests === 0` (nothing matched but helpers) keeps coverage neutral at 1.
+
+Worked example (the motivating case): 3 tests, 2 without assertions, no
+aux findings → coverage = 1/3 → **assertions = 33** (sqs-v1 said 82).
+
+---
+
 ## Locators dimension (ratio)
 
 Independent of finding-penalty math:
@@ -54,7 +96,7 @@ locatorsScore = total === 0 ? 100 : round(100 * native / total)
 
 ESLint `no-raw-locators` / `prefer-native-locators` still appear in `findings[]` for repair guidance.  
 They are **not** double-counted into the locators dimension via exponential penalties.  
-For sqs-v1 they map to **report-only** for scoring (severity still shown in findings; units = 0 for penalty dimensions).
+For sqs-v2 (as in sqs-v1) they map to **report-only** for scoring (severity still shown in findings; units = 0 for penalty dimensions).
 
 ---
 
@@ -73,9 +115,10 @@ Used for penalty density. JSDoc / decorative comments do not inflate slots.
 
 ## Penalty dimensions
 
-Applies to: `playwrightHygiene`, `assertions`, `structure`.
+Applies to: `playwrightHygiene`, `structure`, and the *aux* term of
+`assertions` (see above; the assertions primary term is a coverage ratio).
 
-### Constants (frozen sqs-v1)
+### Constants (frozen sqs-v2; identical values to sqs-v1)
 
 | Constant | Value |
 |---|---|
@@ -159,6 +202,16 @@ private house-rules gate on top of `standard` — see the README's
 See package rule docs / source `profiles.ts`. High level:
 
 - **playwrightHygiene:** waits, force, networkidle, missing await, handles, conditionals  
-- **assertions:** expect-expect, empty tests, web-first preference  
+- **assertions:** per-test coverage census (expect-expect) + quality decay (valid-expect, standalone, web-first)  
 - **locators:** ratio metric only (for score); raw-locator ESLint in findings  
-- **structure:** focused/skipped, oversized file, describe shape  
+- **structure:** focused/skipped, oversized file, describe shape
+
+### Active rule set (frozen)
+
+The exact ESLint rules that produce findings are enabled explicitly
+in `eslint-runner.ts` with fixed severities — the score never inherits
+eslint-plugin-playwright's `recommended` set at install time, so identical
+code scores identically regardless of which plugin version npm resolves.
+(Versions ≤0.2.0 intended to inherit `recommended` but a config-shape bug
+dropped it entirely; 0.3.0 froze the intended set explicitly. See
+CHANGELOG.)  

package/README.md

@@ -4,7 +4,7 @@
 [![npm version](https://img.shields.io/npm/v/@qaguardian/playwright-score.svg)](https://www.npmjs.com/package/@qaguardian/playwright-score)
 [![license](https://img.shields.io/npm/l/@qaguardian/playwright-score.svg)](./LICENSE)
 
-**Deterministic, AI-free quality score for Playwright specs** (`sqs-v1`).
+**Deterministic, AI-free quality score for Playwright specs** (`sqs-v2`).
 
 **Website:** [qaguardian.com/open-source/playwright-score](https://qaguardian.com/open-source/playwright-score)  
 **Built by:** [QA Guardian](https://qaguardian.com) — managed Playwright E2E (AI drafts, engineers verify, you own the code)
@@ -16,7 +16,44 @@ Page Object Model import resolution, ...).
 
 The score never calls an LLM. AI may *generate* or *repair* code using findings; **rules grade code**.
 
-See [METHODOLOGY.md](./METHODOLOGY.md) for the frozen formula, [VALIDATION.md](./VALIDATION.md) for real-world results against public repos (cal.com, Grafana, Mattermost, ...) and a re-runnable script to reproduce them, or the full product write-up on the [landing page](https://qaguardian.com/open-source/playwright-score).
+## Real-world results
+
+17 public Playwright suites, scored with the published package against each
+project's actual source — chosen to include both well-known,
+heavily-engineered platforms and smaller, less mature projects. Not curated
+to look good: 3 of 17 fail the default threshold, for real, verifiable
+reasons (see [VALIDATION.md](./VALIDATION.md)).
+
+**17 suites · 1,228 files · 6,037 tests · 14/17 passed (80% threshold)**
+
+| Repo (source scanned) | Score | Grade | Result |
+|---|---:|:-:|:-:|
+| [Playwright (own TodoMVC example)](https://github.com/microsoft/playwright/tree/main/examples/todomvc) | 98/100 | A | PASS |
+| [Supabase](https://github.com/supabase/supabase/tree/master/e2e) | 97/100 | A | PASS |
+| [Documenso](https://github.com/documenso/documenso/tree/main/packages/app-tests) | 96/100 | A | PASS |
+| [Storybook](https://github.com/storybookjs/storybook/tree/next/code/e2e-internal) | 96/100 | A | PASS |
+| [freeCodeCamp](https://github.com/freeCodeCamp/freeCodeCamp/tree/main/e2e) | 96/100 | A | PASS |
+| [n8n](https://github.com/n8n-io/n8n/tree/master/packages/testing/playwright) | 95/100 | A | PASS |
+| [dub](https://github.com/dubinc/dub/tree/main/apps/web/playwright) | 95/100 | A | PASS |
+| [novu](https://github.com/novuhq/novu/tree/next/apps/dashboard/tests) | 93/100 | A | PASS |
+| [Grafana](https://github.com/grafana/grafana/tree/main/e2e-playwright) | 91/100 | A | PASS |
+| [PostHog](https://github.com/PostHog/posthog/tree/master/playwright) | 90/100 | A | PASS |
+| [sencho](https://github.com/Studio-Saelix/sencho/tree/main/e2e) | 89/100 | B | PASS |
+| [Mattermost](https://github.com/mattermost/mattermost/tree/master/e2e-tests/playwright) | 88/100 | B | PASS |
+| [Immich](https://github.com/immich-app/immich/tree/main/e2e/src/specs) | 85/100 | B | PASS |
+| [cal.com](https://github.com/calcom/cal.com/tree/main/apps/web/playwright) | 84/100 | B | PASS |
+| [livecodes](https://github.com/live-codes/livecodes/tree/develop/e2e) | 78/100 | C | FAIL |
+| [openplayerjs](https://github.com/openplayerjs/openplayerjs/tree/master/e2e) | 77/100 | C | FAIL |
+| [TheCyberHub](https://github.com/th3cyb3rhub/TheCyberHub/tree/dev/e2e) | 68/100 | D | FAIL |
+
+Repo names link to the exact source scanned. Full methodology, findings
+breakdown, and `scripts/validate-corpus.sh` to reproduce every number here
+yourself live in [VALIDATION.md](./VALIDATION.md). A self-contained visual
+report of an earlier five-suite audit lives at
+[`docs/scorecard.html`](./docs/scorecard.html) — host it wherever
+(GitHub Pages, qaguardian.com, ...).
+
+See [METHODOLOGY.md](./METHODOLOGY.md) for the frozen formula, or the full product write-up on the [landing page](https://qaguardian.com/open-source/playwright-score).
 
 ## Install
 

package/dist/eslint-runner.js

@@ -5,8 +5,6 @@ import playwright from 'eslint-plugin-playwright';
 import { basePlugin } from './rules/base-plugin.js';
 import { mapRule } from './profiles.js';
 function buildConfig(assertFunctionNames) {
-    const playwrightRecommended = playwright.configs?.['flat/recommended'] ??
-        playwright.configs?.recommended;
     const base = [
         {
             files: ['**/*.{ts,tsx,js,jsx,mjs,cjs}'],
@@ -89,14 +87,34 @@ function buildConfig(assertFunctionNames) {
                 'playwright/prefer-native-locators': 'warn',
                 // Matches eslint-plugin-playwright's own recommended severity.
                 'playwright/prefer-web-first-assertions': 'error',
+                // The rules below complete the sqs-v1 rule set from
+                // profiles.ts's ESLINT_RULE_MAP. Earlier versions (<=0.2.0)
+                // expected them to arrive via eslint-plugin-playwright's
+                // flat/recommended config, but the spread only handled an array
+                // shape while the plugin exports a single config object — so the
+                // entire recommended set was silently dropped and these rules
+                // never ran (METHODOLOGY.md's hygiene dimension lists
+                // "conditionals" and "handles", yet neither could ever fire).
+                // They are enabled explicitly and frozen here on purpose:
+                // eslint-plugin-playwright is a ^range dependency, and inheriting
+                // whatever its recommended set says at install time would make
+                // identical code score differently across installs. Severities
+                // mirror upstream flat/recommended as of eslint-plugin-playwright
+                // 2.x. require-top-level-describe stays disabled deliberately —
+                // it is stylistic, not in upstream recommended, and would
+                // penalize idiomatic describe-less suites.
+                'playwright/no-conditional-in-test': 'warn',
+                'playwright/no-conditional-expect': 'warn',
+                'playwright/no-element-handle': 'warn',
+                'playwright/no-eval': 'warn',
+                'playwright/no-page-pause': 'warn',
+                'playwright/no-useless-await': 'warn',
+                'playwright/valid-expect': 'error',
+                'playwright/no-standalone-expect': 'error',
+                'playwright/max-nested-describe': 'warn',
             },
         },
     ];
-    // Prefer spreading official recommended flat config when available
-    if (Array.isArray(playwrightRecommended)) {
-        const configs = playwrightRecommended;
-        return [...configs, base[0]];
-    }
     return base;
 }
 function toSeverity(eslintSeverity, override) {

package/dist/formatters/text.js

@@ -12,7 +12,8 @@ export function formatText(result) {
     lines.push('');
     lines.push(`Summary: files=${result.summary.files} tests=${result.summary.tests} sloc=${result.summary.sloc} ` +
         `findings=${result.summary.findings} (errors=${result.summary.errors} warnings=${result.summary.warnings}) ` +
-        `locators native/raw=${result.summary.nativeLocators}/${result.summary.rawLocators}`);
+        `locators native/raw=${result.summary.nativeLocators}/${result.summary.rawLocators} ` +
+        `tests asserting=${result.summary.tests - result.summary.unassertedTests}/${result.summary.tests}`);
     if (result.findings.length) {
         lines.push('');
         lines.push('Findings:');

package/dist/index.js

@@ -110,7 +110,7 @@ function expandPaths(inputs, cwd) {
 }
 function hardFail(profile, threshold, findings, extra = {}, skippedFiles) {
     return {
-        scoreVersion: 'sqs-v1',
+        scoreVersion: 'sqs-v2',
         profile,
         score: 0,
         grade: 'F',
@@ -125,6 +125,7 @@ function hardFail(profile, threshold, findings, extra = {}, skippedFiles) {
             warnings: findings.filter((f) => f.severity === 'warning').length,
             nativeLocators: 0,
             rawLocators: 0,
+            unassertedTests: 0,
             ...extra,
         },
         dimensions: {

package/dist/score-engine.d.ts

@@ -1,7 +1,19 @@
 import type { DimensionName, Finding, Grade, ProfileName, ScoreResult } from './types.js';
-/** sqs-v1 frozen constants — change requires sqs-v2 */
+/** sqs-v2 frozen constants — change requires sqs-v3 */
+export declare const SQS_V2: {
+    readonly scoreVersion: "sqs-v2";
+    readonly ERROR_UNIT: 1;
+    readonly WARNING_UNIT: 0.4;
+    readonly INFO_UNIT: 0;
+    readonly MAX_FINDINGS_PER_RULE_PER_FILE: 3;
+    readonly SLOT_DIVISOR: 25;
+    readonly MIN_SLOTS: 4;
+    readonly K: 0.4;
+};
+/** @deprecated sqs-v1 alias kept for one release so pinned callers get a
+ * type-level nudge instead of a hard break; constants are identical. */
 export declare const SQS_V1: {
-    readonly scoreVersion: "sqs-v1";
+    readonly scoreVersion: "sqs-v2";
     readonly ERROR_UNIT: 1;
     readonly WARNING_UNIT: 0.4;
     readonly INFO_UNIT: 0;
@@ -13,6 +25,28 @@ export declare const SQS_V1: {
 declare const PENALTY_DIMENSIONS: DimensionName[];
 export declare function gradeFromScore(score: number): Grade;
 export declare function locatorsScore(native: number, raw: number): number;
+/**
+ * sqs-v2 assertions dimension: a per-test coverage ratio, decayed by the
+ * density of the remaining assertion-quality findings.
+ *
+ * sqs-v1 fed "test has no assertions" through the same density math as
+ * hygiene smells, which misread coverage badly on small suites: 2 of 3
+ * tests asserting nothing scored 82/100 on this dimension. Whether a test
+ * asserts anything is a fraction of tests, not a findings-per-SLOC rate —
+ * so, like the locators dimension, the primary signal is now a ratio:
+ *
+ *   coverage = (tests - unassertedTests) / tests      // uncapped census
+ *   aux      = density load of the OTHER assertion rules
+ *              (valid-expect, no-standalone-expect,
+ *               prefer-web-first-assertions), capped as usual
+ *   score    = round(100 * coverage * e^(-K * aux))
+ *
+ * expect-expect findings are excluded from the density term — they ARE the
+ * coverage term, and counting them twice would double-penalize.
+ * tests === 0 (nothing but helpers matched) keeps coverage at 1 and lets
+ * the aux findings, if any, carry the dimension.
+ */
+export declare function assertionsScore(findings: Finding[], tests: number, sloc: number): number;
 /** Cap findings per (file, rule) for penalty math only */
 export declare function applyPerRuleCap(findings: Finding[]): Finding[];
 export declare function penaltyDimensionScore(findings: Finding[], dimension: DimensionName, sloc: number): number;

package/dist/score-engine.js

@@ -1,7 +1,7 @@
 import { PROFILE_WEIGHTS } from './profiles.js';
-/** sqs-v1 frozen constants — change requires sqs-v2 */
-export const SQS_V1 = {
-    scoreVersion: 'sqs-v1',
+/** sqs-v2 frozen constants — change requires sqs-v3 */
+export const SQS_V2 = {
+    scoreVersion: 'sqs-v2',
     ERROR_UNIT: 1.0,
     WARNING_UNIT: 0.4,
     INFO_UNIT: 0.0,
@@ -10,6 +10,18 @@ export const SQS_V1 = {
     MIN_SLOTS: 4,
     K: 0.4,
 };
+/** @deprecated sqs-v1 alias kept for one release so pinned callers get a
+ * type-level nudge instead of a hard break; constants are identical. */
+export const SQS_V1 = SQS_V2;
+/**
+ * The rule whose findings are, by construction, a census of "this test
+ * declaration contains no recognized assertion" — one finding per test,
+ * with delegation to same-file/imported helpers already resolved upstream
+ * (see eslint-runner.ts's assertFunctionNames plumbing). sqs-v2 counts
+ * these uncapped as the assertions-coverage numerator instead of feeding
+ * them through density math.
+ */
+const UNASSERTED_TEST_RULE = 'playwright/expect-expect';
 const PENALTY_DIMENSIONS = ['playwrightHygiene', 'assertions', 'structure'];
 export function gradeFromScore(score) {
     if (score >= 90)
@@ -28,12 +40,46 @@ export function locatorsScore(native, raw) {
         return 100;
     return Math.round((100 * native) / total);
 }
+/**
+ * sqs-v2 assertions dimension: a per-test coverage ratio, decayed by the
+ * density of the remaining assertion-quality findings.
+ *
+ * sqs-v1 fed "test has no assertions" through the same density math as
+ * hygiene smells, which misread coverage badly on small suites: 2 of 3
+ * tests asserting nothing scored 82/100 on this dimension. Whether a test
+ * asserts anything is a fraction of tests, not a findings-per-SLOC rate —
+ * so, like the locators dimension, the primary signal is now a ratio:
+ *
+ *   coverage = (tests - unassertedTests) / tests      // uncapped census
+ *   aux      = density load of the OTHER assertion rules
+ *              (valid-expect, no-standalone-expect,
+ *               prefer-web-first-assertions), capped as usual
+ *   score    = round(100 * coverage * e^(-K * aux))
+ *
+ * expect-expect findings are excluded from the density term — they ARE the
+ * coverage term, and counting them twice would double-penalize.
+ * tests === 0 (nothing but helpers matched) keeps coverage at 1 and lets
+ * the aux findings, if any, carry the dimension.
+ */
+export function assertionsScore(findings, tests, sloc) {
+    const unasserted = findings.filter((f) => f.rule === UNASSERTED_TEST_RULE && !f.reportOnly).length;
+    const coverage = tests <= 0 ? 1 : Math.min(1, Math.max(0, (tests - unasserted) / tests));
+    const aux = findings.filter((f) => f.dimension === 'assertions' && f.rule !== UNASSERTED_TEST_RULE);
+    const capped = applyPerRuleCap(aux);
+    let rawUnits = 0;
+    for (const f of capped)
+        rawUnits += severityUnit(f.severity);
+    const slots = Math.max(sloc / SQS_V2.SLOT_DIVISOR, SQS_V2.MIN_SLOTS);
+    const load = rawUnits / slots;
+    const score = Math.round(100 * coverage * Math.exp(-SQS_V2.K * load));
+    return Math.min(100, Math.max(0, score));
+}
 function severityUnit(severity) {
     if (severity === 'error')
-        return SQS_V1.ERROR_UNIT;
+        return SQS_V2.ERROR_UNIT;
     if (severity === 'warning')
-        return SQS_V1.WARNING_UNIT;
-    return SQS_V1.INFO_UNIT;
+        return SQS_V2.WARNING_UNIT;
+    return SQS_V2.INFO_UNIT;
 }
 /** Cap findings per (file, rule) for penalty math only */
 export function applyPerRuleCap(findings) {
@@ -44,7 +90,7 @@ export function applyPerRuleCap(findings) {
             continue;
         const key = `${f.file}::${f.rule}`;
         const n = counts.get(key) ?? 0;
-        if (n >= SQS_V1.MAX_FINDINGS_PER_RULE_PER_FILE)
+        if (n >= SQS_V2.MAX_FINDINGS_PER_RULE_PER_FILE)
             continue;
         counts.set(key, n + 1);
         out.push(f);
@@ -58,9 +104,9 @@ export function penaltyDimensionScore(findings, dimension, sloc) {
     for (const f of capped) {
         rawUnits += severityUnit(f.severity);
     }
-    const slots = Math.max(sloc / SQS_V1.SLOT_DIVISOR, SQS_V1.MIN_SLOTS);
+    const slots = Math.max(sloc / SQS_V2.SLOT_DIVISOR, SQS_V2.MIN_SLOTS);
     const load = rawUnits / slots;
-    const score = Math.round(100 * Math.exp(-SQS_V1.K * load));
+    const score = Math.round(100 * Math.exp(-SQS_V2.K * load));
     return Math.min(100, Math.max(0, score));
 }
 export function computeScore(input) {
@@ -72,7 +118,7 @@ export function computeScore(input) {
     const weights = PROFILE_WEIGHTS[input.profile] ?? PROFILE_WEIGHTS.standard;
     const dims = {
         playwrightHygiene: penaltyDimensionScore(input.findings, 'playwrightHygiene', input.sloc),
-        assertions: penaltyDimensionScore(input.findings, 'assertions', input.sloc),
+        assertions: assertionsScore(input.findings, input.tests, input.sloc),
         locators: locatorsScore(input.nativeLocators, input.rawLocators),
         structure: penaltyDimensionScore(input.findings, 'structure', input.sloc),
     };
@@ -88,7 +134,7 @@ export function computeScore(input) {
     const errors = input.findings.filter((f) => f.severity === 'error').length;
     const warnings = input.findings.filter((f) => f.severity === 'warning').length;
     return {
-        scoreVersion: SQS_V1.scoreVersion,
+        scoreVersion: SQS_V2.scoreVersion,
         profile: input.profile,
         score,
         grade: gradeFromScore(score),
@@ -103,6 +149,7 @@ export function computeScore(input) {
             warnings,
             nativeLocators: input.nativeLocators,
             rawLocators: input.rawLocators,
+            unassertedTests: input.findings.filter((f) => f.rule === UNASSERTED_TEST_RULE && !f.reportOnly).length,
         },
         dimensions: dims,
         findings: input.findings,

package/dist/types.d.ts

@@ -1,4 +1,4 @@
-export type ScoreVersion = 'sqs-v1';
+export type ScoreVersion = 'sqs-v2';
 export type ProfileName = 'standard';
 export type Grade = 'A' | 'B' | 'C' | 'D' | 'F';
 export type Severity = 'error' | 'warning' | 'info';
@@ -29,6 +29,10 @@ export interface ScoreSummary {
     warnings: number;
     nativeLocators: number;
     rawLocators: number;
+    /** Count of test declarations with no recognized assertion (sqs-v2
+     * assertions-coverage numerator input; equals the uncapped
+     * playwright/expect-expect finding count). */
+    unassertedTests: number;
 }
 export interface ScoreResult {
     scoreVersion: ScoreVersion;

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@qaguardian/playwright-score",
-  "version": "0.2.0",
-  "description": "Deterministic, AI-free Playwright spec quality score (sqs-v1).",
+  "version": "0.4.0",
+  "description": "Deterministic, AI-free Playwright spec quality score (sqs-v2).",
   "type": "module",
   "main": "./dist/index.js",
   "types": "./dist/index.d.ts",