@qaguardian/playwright-score 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 QA Guardian / playwright-score contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/METHODOLOGY.md

@@ -0,0 +1,178 @@
+# Playwright Spec Score (SQS) Methodology
+
+**Score version:** `sqs-v1`  
+**Package:** `playwright-score`  
+**Landing page:** https://qaguardian.com/open-source/playwright-score  
+**Maintainer:** [QA Guardian](https://qaguardian.com)  
+**Deterministic:** same inputs → same score  
+**AI-free:** no LLM calls in the scoring path
+
+Any change to formulas, weights, SLOC definition, caps, or constants requires a new version (`sqs-v2`, …).
+
+---
+
+## Profiles
+
+| Profile | Purpose |
+|---|---|
+| `standard` | Community Playwright best practices (`eslint-plugin-playwright`) + suite metrics |
+| `guardian` | `standard` + QA Guardian house conventions (optional; clearly labeled) |
+
+House rules are **not** claimed to be official Playwright doctrine.
+
+---
+
+## Output
+
+- **score:** integer 0–100  
+- **grade:** A (90–100), B (80–89), C (70–79), D (60–69), F (0–59)  
+- **pass:** `score >= threshold`  
+- **dimensions:** per-dimension 0–100  
+- **findings:** full list (uncapped) for humans/agents  
+
+---
+
+## Dimension weights
+
+### `standard`
+
+| Dimension | Weight |
+|---|---|
+| playwrightHygiene | 40 |
+| assertions | 25 |
+| locators | 20 |
+| structure | 15 |
+
+### `guardian`
+
+| Dimension | Weight |
+|---|---|
+| playwrightHygiene | 30 |
+| assertions | 15 |
+| locators | 15 |
+| structure | 15 |
+| guardianConventions | 25 |
+
+---
+
+## Locators dimension (ratio)
+
+Independent of finding-penalty math:
+
+```
+native = count(getByRole|getByLabel|getByTestId|getByText|getByPlaceholder)
+raw    = count(page.locator(|frame.locator()
+total  = native + raw
+
+locatorsScore = total === 0 ? 100 : round(100 * native / total)
+```
+
+ESLint `no-raw-locators` / `prefer-native-locators` still appear in `findings[]` for repair guidance.  
+They are **not** double-counted into the locators dimension via exponential penalties.  
+For sqs-v1 they map to **report-only** for scoring (severity still shown in findings; units = 0 for penalty dimensions).
+
+---
+
+## SLOC (source lines of code)
+
+```
+sloc = lines that are NOT:
+  - blank / whitespace-only
+  - pure comment lines (// or block-comment-only lines)
+sloc = max(sloc, 1)
+```
+
+Used for penalty density. JSDoc / decorative comments do not inflate slots.
+
+---
+
+## Penalty dimensions
+
+Applies to: `playwrightHygiene`, `assertions`, `structure`, `guardianConventions`.
+
+### Constants (frozen sqs-v1)
+
+| Constant | Value |
+|---|---|
+| `ERROR_UNIT` | 1.0 |
+| `WARNING_UNIT` | 0.4 |
+| `INFO_UNIT` | 0.0 |
+| `MAX_FINDINGS_PER_RULE_PER_FILE` | 3 (for penalty math only) |
+| `SLOT_DIVISOR` | 25 |
+| `MIN_SLOTS` | 4 |
+| `K` | 0.4 |
+
+### Steps
+
+1. **Cap** findings per `(file, ruleId)` to 3 for penalty math (report all in `findings[]`).
+2. **Units:** `rawUnits = E * 1.0 + W * 0.4`
+3. **Slots:** `slots = max(sloc / 25, 4)`
+4. **Load:** `load = rawUnits / slots`
+5. **Score:** `dimensionScore = clamp(0, 100, round(100 * exp(-0.4 * load)))`
+
+### Final score
+
+```
+score = round(Σ (weight_d / 100) * dimensionScore_d)
+pass  = score >= threshold
+```
+
+---
+
+## Worked examples (Phase 0 validation)
+
+Assumptions: single-file runs; weights as above.
+
+### 1. Tiny clean (~15 SLOC, 0 findings, all native locators)
+
+- All penalty dims = 100  
+- Locators = 100  
+- **standard score ≈ 100**
+
+### 2. Tiny file, 1 hygiene error (~20 SLOC)
+
+- slots = max(20/25, 4) = 4  
+- rawUnits = 1.0 → load = 0.25 → hygiene = round(100 * e^(-0.1)) ≈ **90**  
+- Other dims 100  
+- standard ≈ 0.4*90 + 0.25*100 + 0.2*100 + 0.15*100 = **96**
+
+### 3. Medium file, 5 distinct hygiene errors (~100 SLOC)
+
+- slots = max(100/25, 4) = 4  
+- rawUnits = 5 → load = 1.25 → hygiene = round(100 * e^(-0.5)) ≈ **61**  
+- standard ≈ 0.4*61 + 60 = **84** if other dims perfect  
+
+### 4. Large file, 20× same warning (capped to 3)
+
+- 20 warnings same rule → cap 3 → rawUnits = 3 * 0.4 = 1.2  
+- sloc = 500 → slots = 20 → load = 0.06 → score ≈ **98** on that dim  
+- Cap prevents auto-zero  
+
+### 5. Locator ratio 2 native / 10 total
+
+- locatorsScore = round(100 * 2/10) = **20**  
+- Independent of caps  
+
+These examples feel fair: tiny single error stays high; multi-error drops materialy; repeated same-rule does not nuke large files; locator mix is a pure ratio.
+
+---
+
+## Default thresholds
+
+| Context | Profile | Threshold |
+|---|---|---|
+| CLI default | standard | 80 |
+| QAG dogfood (warn) | guardian | 75 → 80 |
+| QAG gate (later) | guardian | 80 |
+
+---
+
+## Rule → dimension map (summary)
+
+See package rule docs / source `profiles.ts`. High level:
+
+- **playwrightHygiene:** waits, force, networkidle, missing await, handles, conditionals  
+- **assertions:** expect-expect, empty tests, web-first preference  
+- **locators:** ratio metric only (for score); raw-locator ESLint in findings  
+- **structure:** focused/skipped, oversized file, describe shape  
+- **guardianConventions:** load-state ban, Date.now ban, secrets, test.step, one-describe-one-test, etc.  

package/README.md

@@ -0,0 +1,109 @@
+# @qaguardian/playwright-score
+
+**Deterministic, AI-free quality score for Playwright specs** (`sqs-v1`).
+
+**Website:** [qaguardian.com/open-source/playwright-score](https://qaguardian.com/open-source/playwright-score)  
+**Built by:** [QA Guardian](https://qaguardian.com) — managed Playwright E2E (AI drafts, engineers verify, you own the code)
+
+Lint + score Playwright tests with:
+
+- **`standard`** — community best practices via [`eslint-plugin-playwright`](https://github.com/playwright-community/eslint-plugin-playwright) + suite metrics  
+- **`guardian`** — optional house conventions (QA Guardian codegen gate); not official Playwright doctrine  
+
+The score never calls an LLM. AI may *generate* or *repair* code using findings; **rules grade code**.
+
+See [METHODOLOGY.md](./METHODOLOGY.md) for the frozen formula, or the full product write-up on the [landing page](https://qaguardian.com/open-source/playwright-score).
+
+## Install
+
+```bash
+npm install -D @qaguardian/playwright-score
+# or run from this package
+npm install && npm run build
+```
+
+## CLI
+
+```bash
+npx @qaguardian/playwright-score ./tests --profile standard --threshold 80
+npx @qaguardian/playwright-score ./flow.spec.ts --profile guardian --format json --out report.json
+```
+
+| Flag | Description |
+|---|---|
+| `--profile standard\|guardian` | Scoring profile (default `standard`) |
+| `--threshold <n>` | Pass bar 0–100 |
+| `--format text\|json\|markdown\|sarif` | Output format |
+| `--out <file>` | Write report to file |
+
+**Exit codes:** `0` pass · `1` below threshold · `2` tool error  
+
+## Library
+
+```ts
+import { scorePaths } from '@qaguardian/playwright-score';
+
+const result = await scorePaths({
+  paths: ['tests/login.spec.ts'],
+  profile: 'standard',
+  threshold: 80,
+});
+
+console.log(result.score, result.grade, result.pass, result.findings);
+```
+
+## CI example
+
+```yaml
+- name: Playwright Spec Score
+  run: npx playwright-score ./tests --profile standard --threshold 80 --format text
+```
+
+## QA Guardian integration
+
+When used from `playwright_runner`, set:
+
+| Env | Values | Default |
+|---|---|---|
+| `SPEC_SCORE_MODE` | `off` \| `warn` \| `gate` | `warn` |
+| `SPEC_SCORE_PROFILE` | `standard` \| `guardian` | `guardian` |
+| `SPEC_SCORE_THRESHOLD` | `0`–`100` | `75` (guardian) / `80` (standard) |
+
+- **`warn`**: log score; never fail the run; findings still inject into heal/generate repair prompts  
+- **`gate`**: fail the run when score &lt; threshold  
+- After AI **generate**, specs below threshold get one automatic **heal** pass with score findings  
+- **Heal** prompts always include score findings when the scorer reports issues  
+
+```bash
+# Local from monorepo
+cd playwright-score && npm run build
+cd ../playwright_runner && npm install
+SPEC_SCORE_MODE=warn SPEC_SCORE_PROFILE=guardian node ...
+```
+
+## Profiles
+
+| Profile | Use when |
+|---|---|
+| `standard` | Any Playwright repo — best practices + locator mix + empty-expect detection |
+| `guardian` | QA Guardian (or similar) codegen quality gate |
+
+Disable a guardian secrets false positive:
+
+```ts
+// spec-score-disable-next-line guardian/no-hardcoded-secrets
+const password = 'intentional-fixture-value-xyz';
+```
+
+## Development
+
+```bash
+npm install
+npm run build
+npm test
+npm run score -- ./fixtures/good-standard.spec.ts
+```
+
+## License
+
+MIT

package/bin/playwright-score.js

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+import '../dist/cli.js';

package/dist/cli.d.ts

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+export {};

package/dist/cli.js

@@ -0,0 +1,113 @@
+#!/usr/bin/env node
+/**
+ * playwright-score CLI
+ */
+import fs from 'node:fs';
+import path from 'node:path';
+import { scorePaths } from './index.js';
+import { formatJson } from './formatters/json.js';
+import { formatText } from './formatters/text.js';
+import { formatMarkdown } from './formatters/markdown.js';
+import { formatSarif } from './formatters/sarif.js';
+function printHelp() {
+    console.log(`playwright-score — deterministic Playwright spec quality score (sqs-v1)
+
+Usage:
+  playwright-score <paths...> [options]
+
+Options:
+  --profile <standard|guardian>   Scoring profile (default: standard)
+  --threshold <n>                 Pass threshold 0-100 (default: 80 standard / 75 guardian)
+  --format <text|json|markdown|sarif>  Output format (default: text)
+  --out <file>                    Write report to file
+  --help                          Show help
+
+Exit codes:
+  0  score >= threshold
+  1  score < threshold
+  2  tool error
+
+See METHODOLOGY.md for sqs-v1 formula.
+`);
+}
+async function main() {
+    const args = process.argv.slice(2);
+    if (args.length === 0 || args.includes('--help') || args.includes('-h')) {
+        printHelp();
+        process.exit(args.length === 0 ? 2 : 0);
+    }
+    let profile = 'standard';
+    let threshold;
+    let format = 'text';
+    let out;
+    const paths = [];
+    for (let i = 0; i < args.length; i++) {
+        const a = args[i];
+        if (a === '--profile') {
+            profile = args[++i];
+            if (profile !== 'standard' && profile !== 'guardian') {
+                console.error(`Invalid profile: ${profile}`);
+                process.exit(2);
+            }
+        }
+        else if (a === '--threshold') {
+            threshold = Number(args[++i]);
+            if (Number.isNaN(threshold) || threshold < 0 || threshold > 100) {
+                console.error('threshold must be 0-100');
+                process.exit(2);
+            }
+        }
+        else if (a === '--format') {
+            const f = args[++i];
+            if (f !== 'text' && f !== 'json' && f !== 'markdown' && f !== 'sarif') {
+                console.error(`Invalid format: ${f}`);
+                process.exit(2);
+            }
+            format = f;
+        }
+        else if (a === '--out') {
+            out = args[++i];
+        }
+        else if (a.startsWith('-')) {
+            console.error(`Unknown option: ${a}`);
+            process.exit(2);
+        }
+        else {
+            paths.push(a);
+        }
+    }
+    if (paths.length === 0) {
+        console.error('At least one path is required');
+        process.exit(2);
+    }
+    try {
+        const result = await scorePaths({ paths, profile, threshold });
+        let body;
+        switch (format) {
+            case 'json':
+                body = formatJson(result);
+                break;
+            case 'markdown':
+                body = formatMarkdown(result);
+                break;
+            case 'sarif':
+                body = formatSarif(result);
+                break;
+            default:
+                body = formatText(result);
+        }
+        if (out) {
+            fs.mkdirSync(path.dirname(path.resolve(out)), { recursive: true });
+            fs.writeFileSync(out, body + '\n', 'utf8');
+        }
+        else {
+            console.log(body);
+        }
+        process.exit(result.pass ? 0 : 1);
+    }
+    catch (err) {
+        console.error(err instanceof Error ? err.message : err);
+        process.exit(2);
+    }
+}
+main();

package/dist/eslint-runner.d.ts

@@ -0,0 +1,10 @@
+import type { Finding, ProfileName } from './types.js';
+/**
+ * Run ESLint once per scorePaths invocation with embedded config only.
+ * Does not load host-repo eslint.config.js.
+ */
+export declare function runEslint(options: {
+    files: string[];
+    profile: ProfileName;
+    cwd?: string;
+}): Promise<Finding[]>;

package/dist/eslint-runner.js

@@ -0,0 +1,120 @@
+import path from 'node:path';
+import { ESLint } from 'eslint';
+import playwright from 'eslint-plugin-playwright';
+import { guardianPlugin, guardianRuleConfigs, } from './rules/guardian-plugin.js';
+import { mapRule } from './profiles.js';
+function buildConfig(profile) {
+    const playwrightRecommended = playwright.configs?.['flat/recommended'] ??
+        playwright.configs?.recommended;
+    const base = [
+        {
+            files: ['**/*.{ts,tsx,js,jsx,mjs,cjs}'],
+            languageOptions: {
+                ecmaVersion: 2022,
+                sourceType: 'module',
+                parser: undefined, // set below if available
+            },
+            plugins: {
+                playwright,
+                ...(profile === 'guardian' ? { guardian: guardianPlugin } : {}),
+            },
+            rules: {
+                // Ensure key hygiene rules even if recommended set differs by version
+                'playwright/no-wait-for-timeout': 'error',
+                'playwright/no-networkidle': 'error',
+                'playwright/no-force-option': 'error',
+                'playwright/no-wait-for-selector': 'error',
+                'playwright/no-wait-for-navigation': 'error',
+                'playwright/missing-playwright-await': 'error',
+                'playwright/expect-expect': 'error',
+                'playwright/no-focused-test': 'error',
+                'playwright/no-skipped-test': 'warn',
+                'playwright/no-raw-locators': 'warn',
+                'playwright/prefer-web-first-assertions': 'warn',
+                ...(profile === 'guardian' ? { ...guardianRuleConfigs } : {}),
+            },
+        },
+    ];
+    // Prefer spreading official recommended flat config when available
+    if (Array.isArray(playwrightRecommended)) {
+        const configs = playwrightRecommended;
+        if (profile === 'guardian') {
+            return [
+                ...configs,
+                {
+                    plugins: { guardian: guardianPlugin },
+                    rules: { ...guardianRuleConfigs },
+                },
+                // our severity tweaks
+                base[0],
+            ];
+        }
+        return [...configs, base[0]];
+    }
+    return base;
+}
+function toSeverity(eslintSeverity, override) {
+    if (override)
+        return override;
+    if (eslintSeverity === 2)
+        return 'error';
+    if (eslintSeverity === 1)
+        return 'warning';
+    return 'info';
+}
+/**
+ * Run ESLint once per scorePaths invocation with embedded config only.
+ * Does not load host-repo eslint.config.js.
+ */
+export async function runEslint(options) {
+    const cwd = options.cwd ?? process.cwd();
+    const overrideConfig = buildConfig(options.profile);
+    // Try to attach typescript-eslint parser for .ts files
+    try {
+        const tsParser = await import('@typescript-eslint/parser');
+        for (const cfg of overrideConfig) {
+            if (cfg.files) {
+                cfg.languageOptions = {
+                    ...cfg.languageOptions,
+                    parser: tsParser,
+                    parserOptions: {
+                        ecmaVersion: 2022,
+                        sourceType: 'module',
+                    },
+                };
+            }
+        }
+    }
+    catch {
+        // parser optional for pure JS
+    }
+    const eslint = new ESLint({
+        cwd,
+        // Prevent loading ambient project ESLint configs (ESLint 9 flat API)
+        overrideConfigFile: true,
+        overrideConfig: overrideConfig,
+        // Ignore host ignore files that might exclude generated specs
+        ignore: false,
+    });
+    const results = await eslint.lintFiles(options.files);
+    const findings = [];
+    for (const result of results) {
+        const file = path.resolve(result.filePath);
+        for (const msg of result.messages) {
+            if (!msg.ruleId)
+                continue;
+            const mapping = mapRule(msg.ruleId);
+            findings.push({
+                rule: msg.ruleId,
+                severity: toSeverity(msg.severity, mapping.severityOverride),
+                message: msg.message,
+                file,
+                line: msg.line,
+                column: msg.column,
+                dimension: mapping.dimension,
+                reportOnly: mapping.reportOnly,
+            });
+        }
+    }
+    return findings;
+}

package/dist/formatters/json.d.ts

@@ -0,0 +1,2 @@
+import type { ScoreResult } from '../types.js';
+export declare function formatJson(result: ScoreResult): string;

package/dist/formatters/json.js

@@ -0,0 +1,3 @@
+export function formatJson(result) {
+    return JSON.stringify(result, null, 2);
+}

package/dist/formatters/markdown.d.ts

@@ -0,0 +1,2 @@
+import type { ScoreResult } from '../types.js';
+export declare function formatMarkdown(result: ScoreResult): string;

package/dist/formatters/markdown.js

@@ -0,0 +1,26 @@
+export function formatMarkdown(result) {
+    const lines = [];
+    lines.push(`# Playwright Spec Score (\`${result.scoreVersion}\`)`);
+    lines.push('');
+    lines.push(`**${result.score}/100 (${result.grade})** — ${result.pass ? 'PASS' : 'FAIL'} · profile \`${result.profile}\` · threshold ${result.threshold}`);
+    lines.push('');
+    lines.push('| Dimension | Score |');
+    lines.push('|---|---|');
+    for (const [k, v] of Object.entries(result.dimensions)) {
+        if (v === undefined)
+            continue;
+        lines.push(`| ${k} | ${v} |`);
+    }
+    lines.push('');
+    lines.push(`Files: ${result.summary.files} · Tests: ${result.summary.tests} · SLOC: ${result.summary.sloc} · Findings: ${result.summary.findings}`);
+    if (result.findings.length) {
+        lines.push('');
+        lines.push('## Findings');
+        lines.push('');
+        for (const f of result.findings.slice(0, 40)) {
+            const loc = f.line ? `:${f.line}` : '';
+            lines.push(`- **${f.severity}** \`${f.rule}\` \`${f.file}${loc}\` — ${f.message}`);
+        }
+    }
+    return lines.join('\n');
+}

package/dist/formatters/sarif.d.ts

@@ -0,0 +1,3 @@
+import type { ScoreResult } from '../types.js';
+/** Minimal SARIF 2.1.0 for CI integrations */
+export declare function formatSarif(result: ScoreResult): string;

package/dist/formatters/sarif.js

@@ -0,0 +1,55 @@
+/** Minimal SARIF 2.1.0 for CI integrations */
+export function formatSarif(result) {
+    const rulesMap = new Map();
+    for (const f of result.findings) {
+        if (!rulesMap.has(f.rule)) {
+            rulesMap.set(f.rule, {
+                id: f.rule,
+                shortDescription: { text: f.rule },
+            });
+        }
+    }
+    const sarif = {
+        $schema: 'https://raw.githubusercontent.com/oasis-tcs/sarif-spec/master/Schemata/sarif-schema-2.1.0.json',
+        version: '2.1.0',
+        runs: [
+            {
+                tool: {
+                    driver: {
+                        name: 'playwright-score',
+                        informationUri: 'https://github.com/qaguardian/playwright-score',
+                        version: result.scoreVersion,
+                        rules: [...rulesMap.values()],
+                    },
+                },
+                results: result.findings.map((f) => ({
+                    ruleId: f.rule,
+                    level: f.severity === 'error'
+                        ? 'error'
+                        : f.severity === 'warning'
+                            ? 'warning'
+                            : 'note',
+                    message: { text: f.message },
+                    locations: [
+                        {
+                            physicalLocation: {
+                                artifactLocation: { uri: f.file },
+                                region: f.line
+                                    ? { startLine: f.line, startColumn: f.column ?? 1 }
+                                    : undefined,
+                            },
+                        },
+                    ],
+                })),
+                properties: {
+                    score: result.score,
+                    grade: result.grade,
+                    pass: result.pass,
+                    profile: result.profile,
+                    threshold: result.threshold,
+                },
+            },
+        ],
+    };
+    return JSON.stringify(sarif, null, 2);
+}

package/dist/formatters/text.d.ts

@@ -0,0 +1,2 @@
+import type { ScoreResult } from '../types.js';
+export declare function formatText(result: ScoreResult): string;