playwright-mutation-gate 0.0.1 → 0.1.0

package/README.md

@@ -1,39 +1,239 @@
 # playwright-mutation-gate
 
+[![npm version](https://img.shields.io/npm/v/playwright-mutation-gate)](https://www.npmjs.com/package/playwright-mutation-gate)
+[![CI](https://github.com/VladyslavDmitriiev/playwright-mutation-gate/actions/workflows/ci.yml/badge.svg)](https://github.com/VladyslavDmitriiev/playwright-mutation-gate/actions)
+[![license](https://img.shields.io/npm/l/playwright-mutation-gate)](./LICENSE)
+
 > A passing test only means something if it can fail.
 
-**Status: v0.0.1 is a name reservation.** The actual CLI is in development and ships as v0.1.0.
+`playwright-mutation-gate` inverts the assertion you marked as primary and
+re-runs the test. If it stays green, the assertion is hollow and the gate
+fails your CI run instead of reporting a lie.
+
+<p align="center">
+  <img src="media/demo.svg" alt="Terminal recording: 3 killed, 1 survived, gate fails" width="840">
+</p>
+
+<details><summary>Text version (if the SVG above does not render)</summary>
+
+```
+fixtures/demo.spec.ts
+  killed         shows error for invalid email  :8
+  killed         hides error for valid email    :16
+  killed         shows success for valid email  :24
+  survived       hides success initially        :31
+
+3 killed, 1 survived, 0 cannot-mutate, 0 errors
+Gate FAILED: hollow, unverified or broken assertions above.
+```
+</details>
 
-## What it will do
+## Quick start
 
-`playwright-mutation-gate` checks that your green Playwright tests are green for a reason.
-You mark the primary assertion in each test:
+Requires **Node >= 20** and a working **Playwright** setup (any version that
+supports `playwright test`).
+
+```bash
+npm install -D playwright-mutation-gate
+```
+
+Mark the primary assertion in each test with `// @primary-assert`. Put it at
+the end of the assertion line, or on its own line right above it. The assertion
+itself must be a single-line `expect(...)` call.
 
 ```ts
-test('checkout shows order confirmation', async ({ page }) => {
+test('checkout shows confirmation', async ({ page }) => {
   // ...
+  await expect(page.getByText('Order confirmed')).toBeVisible(); // @primary-assert
+});
+
+// Also valid: the marker on its own line above the assertion.
+test('cart badge updates', async ({ page }) => {
   // @primary-assert
-  await expect(page.getByText('Order confirmed')).toBeVisible();
+  await expect(page.getByTestId('cart-count')).toHaveText('1');
+});
+```
+
+Run the gate:
+
+```bash
+npx playwright-mutation-gate "tests/**/*.spec.ts"
+```
+
+## How marking works
+
+The gate inverts the marked assertion
+(`expect(x).toBeVisible()` becomes `expect(x).not.toBeVisible()`) in a
+temporary copy of the spec and re-runs the test in isolation.
+
+- **killed** -- the test failed under inversion. The assertion is alive.
+- **survived** -- the test still passed. The assertion checks nothing useful.
+- **cannot-mutate** -- the assertion shape could not be inverted safely. The
+  gate reports it instead of guessing.
+
+One marker per `test()` block. Markers inside `test.describe` or `test.use`
+callbacks work the same way -- the gate scans each `test()` individually,
+nesting does not matter.
+
+By default, tests without a marker fail the gate (sentinel mode). This means
+a newly added test without `// @primary-assert` shows up as:
+
+```
+  unmarked       adds item to cart  :42
+```
+
+Pass `--no-require-sentinel` to opt out while you're marking an existing suite.
+
+## Custom test fixtures
+
+If your tests are declared through a custom fixture instead of the bare `test`,
+the gate finds it automatically in two cases: a local `.extend` chain and a
+renamed import.
+
+```ts
+import { test as base } from '@playwright/test';
+const authedTest = base.extend({ /* ... */ });   // auto-detected
+
+authedTest('sees dashboard', async ({ page }) => {
+  await expect(page.getByText('Welcome')).toBeVisible(); // @primary-assert
 });
 ```
 
-The gate then inverts that assertion (`expect(x).toBeVisible()` → `expect(x).not.toBeVisible()`)
-in a temporary copy of the spec and re-runs the test:
+When the fixture is imported by its own name from another module, name it with
+`--test-id` (repeatable):
+
+```bash
+npx playwright-mutation-gate tests/dashboard.spec.ts --test-id authedTest
+```
+
+A runtime `authedTest.skip(condition)` inside a test is respected: if the test
+skips, the mutation reports `cannot-mutate` rather than a false `survived`.
+
+## CI recipe
 
-- test fails → the assertion is alive (**killed** mutation, good)
-- test still passes → the assertion checks nothing (**survived**, a hollow test), so the
-  gate exits 1 and fails your CI run
+```yaml
+# .github/workflows/mutation-gate.yml
+name: mutation-gate
 
-Tests without a `@primary-assert` marker also fail the gate by default, so nothing slips
-through unmarked.
+on: [push, pull_request]
+
+jobs:
+  gate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+          cache: npm
+      - run: npm ci
+      - run: npx playwright install --with-deps chromium
+      - run: npx playwright-mutation-gate "tests/**/*.spec.ts"
+```
+
+The gate exits 0 only when every marked assertion is killed and sentinel
+mode is satisfied. Any other outcome exits 1.
+
+Cost scales linearly: every marked assertion is one extra `playwright test`
+run. That is fine for a handful of specs, but on a large suite you probably
+do not want the full sweep on every push. Two patterns that work well:
+
+- **Gate only what changed.** The CLI takes plain file paths, so feed it the
+  spec files the PR touched:
+
+  ```yaml
+      - run: |
+          SPECS=$(git diff --name-only origin/${{ github.base_ref }}... -- 'tests/**/*.spec.ts')
+          [ -n "$SPECS" ] && npx playwright-mutation-gate $SPECS || echo "no specs changed"
+  ```
+
+- **Full sweep on a schedule.** Run the whole glob nightly instead of per
+  commit. A hollow assertion does not get less hollow between merges;
+  catching it a day later is fine. `--playwright-args` is available if you
+  need to shard or point at a specific config.
+
+## CLI reference
+
+```
+npx playwright-mutation-gate [options] <specs...>
+
+Options:
+  --json                    machine-readable JSON report
+  --no-require-sentinel     do not fail on tests without a marker
+  --playwright-args <args>  extra arguments for playwright test
+  --test-id <name>          treat <name> as a test builder (repeatable)
+  -V, --version             output the version number
+  -h, --help                display help for command
+```
+
+The package is CLI-only for now. Programmatic API is planned for v0.2.
+
+## Limitations
+
+**The gate proves an assertion can fail, not that it checks the right thing.**
+A weak but falsifiable assertion passes: a test that asserts a site-wide header
+(one that also renders on the error page) is killed by inversion and clears the
+gate, even though it would not catch the feature breaking. What the gate
+guarantees is narrower and mechanical: every marked assertion actually runs and
+is load-bearing, so the test cannot stay green regardless of what the app does.
+Judging whether the assertion is the *right* one still takes review, or
+application-code mutation testing (Stryker-style), which is a different tool.
+
+**A kill is slightly generous by construction.** `killed` means the mutated
+run failed, not that the flipped assertion is what failed. Playwright actions
+carry implicit assertions (`click()` fails on its own if the element never
+appears), and an ordinarily flaky step also counts, so a test can die before
+it ever reaches the inverted line. The asymmetry works in your favor:
+`survived` has no such excuse. A test that stays green with its own primary
+assertion inverted is hollow, full stop.
+
+String-based inversion, not AST. This keeps the tool small and dependency-free
+but means some shapes report `cannot-mutate`:
+
+- **Single line only.** `expect(...)` and its matcher must start on the marked
+  line. Multi-line chains report cannot-mutate.
+- **One assertion per line.** Two `expect` calls on the same line (including
+  asymmetric matchers in arguments, nested expects inside `toPass`) report
+  cannot-mutate.
+- **No aliases.** Assertions using a re-exported or aliased `expect` are not
+  recognized.
+- **Custom matchers** must follow the `toXxx` convention. Names that shadow
+  built-ins (`toString`, `toFixed`, ...) are denylisted.
+- **Supported chain:**
+  `[await] expect[.soft|.poll](...)[.resolves|.rejects][.not].toXxx(`
+
+`cannot-mutate` is always safe: the gate tells you it could not verify the
+assertion, never silently skips it.
+
+Skipped tests get the same honesty. A test disabled with `test.skip` never
+executes, so there is nothing to observe when its assertion is inverted.
+Rather than guess, the gate reports `cannot-mutate` with the reason ("the
+test was skipped at runtime (conditional test.skip), so the inverted
+assertion never ran"). In
+practice this is a feature: a `test.skip(true, ...)` from months ago looks
+identical to a healthy passing spec in normal CI output, and the gate is
+often what surfaces it.
 
 ## Why
 
-Stryker and friends mutate your application code to score unit tests. E2E suites have the
-opposite blind spot: the assertion itself is often hollow. It passes against a broken app.
-This tool mutates the assertions, not the app.
+AI code generators write tests fast. The tests look right: good names, proper
+selectors, green checkmarks. But the assertions are often hollow: a forgotten
+`await` whose result nobody checks, an assertion guarded behind a branch that
+never executes, a `not.toBeVisible` against a selector that never existed, a
+test silently disabled with `skip`.
+
+Stryker and friends mutate your *application code* to score unit tests. E2E
+suites have the opposite blind spot: the *assertion itself* is often the weak
+link. It passes against a broken app. `playwright-mutation-gate` mutates the
+assertions, not the app.
+
+If a green test cannot turn red when its assertion is inverted, it was never
+testing anything. The gate catches that before it reaches production.
+
+## Contributing
 
-v0.1.0 coming soon.
+Issues and PRs are welcome. Run `npm test` before submitting -- it covers
+lint, typecheck, unit, e2e and integration tests in one pass.
 
 ## License
 

package/dist/cli.js

@@ -0,0 +1,64 @@
+#!/usr/bin/env node
+import { existsSync, readFileSync } from 'node:fs';
+import { Command, CommanderError } from 'commander';
+import { jsonReport, renderTable, summarize } from './report.js';
+import { runSpec } from './run.js';
+// Exit codes: 0 = gate passed, 1 = gate failed (survived / unverified /
+// marker mistakes), 2 = usage or environment error (bad flags, missing file).
+const pkg = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8'));
+const program = new Command()
+    .name('playwright-mutation-gate')
+    .description('Invert each @primary-assert assertion and re-run the test.\n' +
+    'A test that stays green under inversion is hollow and fails the gate.')
+    .version(pkg.version)
+    .argument('<specs...>', 'Playwright spec files to gate')
+    .option('--json', 'print a machine-readable JSON report instead of the table', false)
+    .option('--no-require-sentinel', 'do not fail the gate when a test() has no @primary-assert marker')
+    .option('--playwright-args <args>', 'extra arguments passed to `npx playwright test`, split on spaces\n' +
+    '(values containing spaces are not supported)')
+    .option('--test-id <name>', 'treat <name> as a test builder in addition to `test` (repeatable);\n' +
+    'for custom fixtures imported by name (import { authedTest } from ...).\n' +
+    'Local `const x = test.extend(...)` and `test as x` renames are auto-detected', (value, acc) => acc.concat(value), [])
+    .showHelpAfterError()
+    .addHelpText('after', '\nMarking:\n' +
+    '  Put `// @primary-assert` at the end of the primary assertion line,\n' +
+    '  or on its own line right above it. One marker per test().\n' +
+    '    await expect(page.getByText(\'Saved\')).toBeVisible(); // @primary-assert')
+    .action(async (specPaths, opts) => {
+    const missing = specPaths.filter((p) => !existsSync(p));
+    if (missing.length > 0) {
+        console.error(`spec file not found: ${missing.join(', ')}`);
+        process.exit(2);
+    }
+    const playwrightArgs = (opts.playwrightArgs ?? '').split(/\s+/).filter(Boolean);
+    const results = [];
+    for (const spec of specPaths) {
+        process.stderr.write(`gating ${spec}\n`);
+        try {
+            results.push(await runSpec(spec, { playwrightArgs, testIds: opts.testId }));
+        }
+        catch (err) {
+            console.error(`failed to gate ${spec}: ${err instanceof Error ? err.message : String(err)}`);
+            process.exit(2);
+        }
+    }
+    const gate = { requireSentinel: opts.requireSentinel };
+    if (opts.json) {
+        console.log(jsonReport(results, gate));
+    }
+    else {
+        const color = process.stdout.isTTY === true && process.env.NO_COLOR === undefined;
+        console.log(renderTable(results, { color, ...gate }));
+    }
+    process.exit(summarize(results, gate).gatePassed ? 0 : 1);
+});
+program.exitOverride();
+try {
+    await program.parseAsync();
+}
+catch (err) {
+    // commander has already printed its message; map usage errors to exit 2,
+    // keep 0 for --help / --version.
+    const code = err instanceof CommanderError && err.exitCode === 0 ? 0 : 2;
+    process.exit(code);
+}

package/dist/invert.js

@@ -0,0 +1,116 @@
+import { CODE, findCloseMasked, lex } from './lex.js';
+// Built-in toXxx methods that look like matchers but are not assertions.
+const NOT_MATCHERS = new Set([
+    'toString',
+    'toLocaleString',
+    'toJSON',
+    'toISOString',
+    'toFixed',
+    'toExponential',
+    'toPrecision',
+    'toDateString',
+    'toTimeString',
+    'toUTCString',
+    'toLocaleDateString',
+    'toLocaleTimeString',
+    'toLocaleLowerCase',
+    'toLocaleUpperCase',
+    'toLowerCase',
+    'toUpperCase',
+    'toReversed',
+    'toSorted',
+    'toSpliced',
+]);
+const isWord = (c) => c !== undefined && /[\w$]/.test(c);
+/**
+ * Invert the assertion on a single line of a Playwright spec:
+ * `expect(x).toY()` <-> `expect(x).not.toY()`.
+ *
+ * Purely string-based. Supported chain shape:
+ * `[await] expect[.soft|.poll](...) [.resolves|.rejects] [.not] .toXxx(`
+ * The matcher call must start on this line; its arguments may continue on
+ * following lines. Everything else returns `cannotMutate` with a reason
+ * rather than guessing.
+ */
+export function invertAssertLine(line) {
+    const kind = lex(line);
+    // First CODE index at or after j: skips whitespace, comments and other
+    // non-code chars (so a comment inside the chain does not break the walk).
+    const skip = (j) => {
+        while (j < line.length && (kind[j] !== CODE || /\s/.test(line[j])))
+            j++;
+        return j;
+    };
+    const anchors = [...line.matchAll(/\bexpect\s*[.(]/g)].filter((m) => kind[m.index] === CODE);
+    if (anchors.length === 0) {
+        return { cannotMutate: 'no expect() call found on this line' };
+    }
+    if (anchors.length > 1) {
+        return { cannotMutate: 'multiple expect calls on one line' };
+    }
+    // Position of the `.` or `(` right after the `expect` token.
+    let i = anchors[0].index + anchors[0][0].length - 1;
+    if (line[i] === '.') {
+        i = skip(i + 1);
+        const start = i;
+        while (isWord(line[i]))
+            i++;
+        const form = line.slice(start, i);
+        if (form !== 'soft' && form !== 'poll') {
+            return { cannotMutate: `unsupported expect form "expect.${form}"` };
+        }
+        i = skip(i);
+        if (line[i] !== '(') {
+            return { cannotMutate: `expect.${form} is not called on this line` };
+        }
+    }
+    const close = findCloseMasked(line, kind, i);
+    if (close === -1) {
+        return { cannotMutate: 'expect(...) does not close on this line (multi-line expect)' };
+    }
+    // Walk the chain after expect(...): modifiers, optional .not, then the matcher.
+    i = skip(close + 1);
+    let notStart = -1;
+    let notEnd = -1;
+    for (;;) {
+        if (i >= line.length) {
+            return { cannotMutate: 'no matcher call after expect(...) on this line' };
+        }
+        if (line[i] !== '.') {
+            return { cannotMutate: `unexpected "${line[i]}" after expect(...) chain` };
+        }
+        const dot = i;
+        i = skip(i + 1);
+        const start = i;
+        while (isWord(line[i]))
+            i++;
+        const id = line.slice(start, i);
+        if (id === '') {
+            return { cannotMutate: 'unexpected character in expect(...) chain' };
+        }
+        const lookahead = skip(i);
+        if (line[lookahead] === '(') {
+            if (!/^to[A-Z]/.test(id) || NOT_MATCHERS.has(id)) {
+                return { cannotMutate: `".${id}" does not look like an assertion matcher` };
+            }
+            const inverted = notStart !== -1
+                ? line.slice(0, notStart) + line.slice(notEnd)
+                : line.slice(0, dot) + '.not' + line.slice(dot);
+            return { ok: inverted };
+        }
+        if (id === 'not') {
+            if (notStart !== -1) {
+                return { cannotMutate: 'double .not in expect(...) chain' };
+            }
+            notStart = dot;
+            notEnd = i;
+            i = skip(i);
+            continue;
+        }
+        if (id === 'resolves' || id === 'rejects') {
+            i = skip(i);
+            continue;
+        }
+        return { cannotMutate: `unsupported chain segment ".${id}"` };
+    }
+}

package/dist/lex.js

@@ -0,0 +1,185 @@
+/**
+ * Shared single-pass lexer for the string-based spec analysis. Classifies
+ * every char of a source (a whole file or a single line) so the scanners in
+ * invert.ts and mutate.ts agree about what is code. One implementation on
+ * purpose: two hand-rolled lexers had already diverged once.
+ */
+/** Char inside a string, template literal, block comment or regex literal. */
+export const NON_CODE = 0;
+/** Executable source. */
+export const CODE = 1;
+/** Char of a `//` comment that starts in code context. */
+export const LINE_COMMENT = 2;
+// Keywords after which a `/` starts a regex literal, not division.
+const REGEX_PRECEDING_KEYWORDS = new Set([
+    'return',
+    'typeof',
+    'instanceof',
+    'in',
+    'of',
+    'new',
+    'delete',
+    'void',
+    'do',
+    'else',
+    'case',
+    'yield',
+    'await',
+]);
+const isWordChar = (c) => c !== undefined && /[\w$]/.test(c);
+/**
+ * A `/` in code context is a regex literal when the last significant code
+ * token cannot end an expression: after a value (identifier, `)`, `]`,
+ * string) it is division.
+ */
+function startsRegex(lastSig, lastWord) {
+    if (lastSig === '')
+        return true;
+    if (isWordChar(lastSig))
+        return REGEX_PRECEDING_KEYWORDS.has(lastWord);
+    return lastSig !== ')' && lastSig !== ']' && lastSig !== '"' && lastSig !== "'" && lastSig !== '`';
+}
+/**
+ * Classify every char of `src` as CODE, LINE_COMMENT or NON_CODE. Handles
+ * strings with escapes, template literals with nested `${}` expressions,
+ * `//` and block comments, and regex literals (heuristic: see startsRegex).
+ */
+export function lex(src) {
+    const kind = new Uint8Array(src.length).fill(CODE);
+    // '`' = inside a template literal, '{' = inside a `${}` expression.
+    const stack = [];
+    // Last significant code char, its index, and the word it ends, for regex detection.
+    let lastSig = '';
+    let lastSigIndex = -2;
+    let lastWord = '';
+    let i = 0;
+    while (i < src.length) {
+        const c = src[i];
+        if (stack[stack.length - 1] === '`') {
+            kind[i] = NON_CODE;
+            if (c === '\\') {
+                if (i + 1 < src.length)
+                    kind[i + 1] = NON_CODE;
+                i++;
+            }
+            else if (c === '`') {
+                stack.pop();
+                lastSig = '`';
+                lastWord = '';
+            }
+            else if (c === '$' && src[i + 1] === '{') {
+                kind[i + 1] = NON_CODE;
+                stack.push('{');
+                lastSig = '';
+                lastWord = '';
+                i++;
+            }
+            i++;
+            continue;
+        }
+        if (c === "'" || c === '"') {
+            kind[i] = NON_CODE;
+            i++;
+            while (i < src.length && src[i] !== '\n') {
+                kind[i] = NON_CODE;
+                if (src[i] === '\\' && i + 1 < src.length) {
+                    kind[i + 1] = NON_CODE;
+                    i += 2;
+                    continue;
+                }
+                i++;
+                if (src[i - 1] === c)
+                    break;
+            }
+            lastSig = c;
+            lastWord = '';
+            continue;
+        }
+        if (c === '`') {
+            kind[i] = NON_CODE;
+            stack.push('`');
+            i++;
+            continue;
+        }
+        if (c === '/' && src[i + 1] === '/') {
+            while (i < src.length && src[i] !== '\n')
+                kind[i++] = LINE_COMMENT;
+            continue;
+        }
+        if (c === '/' && src[i + 1] === '*') {
+            kind[i] = NON_CODE;
+            kind[i + 1] = NON_CODE;
+            i += 2;
+            while (i < src.length && !(src[i] === '*' && src[i + 1] === '/'))
+                kind[i++] = NON_CODE;
+            if (i < src.length) {
+                kind[i] = NON_CODE;
+                kind[i + 1] = NON_CODE;
+                i += 2;
+            }
+            continue;
+        }
+        if (c === '/' && startsRegex(lastSig, lastWord)) {
+            kind[i] = NON_CODE;
+            i++;
+            let inClass = false;
+            while (i < src.length && src[i] !== '\n') {
+                kind[i] = NON_CODE;
+                if (src[i] === '\\' && i + 1 < src.length) {
+                    kind[i + 1] = NON_CODE;
+                    i += 2;
+                    continue;
+                }
+                if (src[i] === '[')
+                    inClass = true;
+                else if (src[i] === ']')
+                    inClass = false;
+                else if (src[i] === '/' && !inClass) {
+                    i++;
+                    break;
+                }
+                i++;
+            }
+            while (i < src.length && /[a-z]/.test(src[i]))
+                kind[i++] = NON_CODE; // flags
+            lastSig = '/';
+            lastWord = '';
+            continue;
+        }
+        if (!/\s/.test(c)) {
+            if (isWordChar(c)) {
+                lastWord = isWordChar(lastSig) && lastSigIndex === i - 1 ? lastWord + c : c;
+            }
+            else {
+                lastWord = '';
+            }
+            lastSig = c;
+            lastSigIndex = i;
+            if (stack.length > 0) {
+                if (c === '{')
+                    stack.push('{');
+                else if (c === '}')
+                    stack.pop();
+            }
+        }
+        i++;
+        continue;
+    }
+    return kind;
+}
+/**
+ * Find the `)` matching the `(` at `open`, counting only CODE chars.
+ * Returns -1 when it does not close within `src`.
+ */
+export function findCloseMasked(src, kind, open) {
+    let depth = 0;
+    for (let i = open; i < src.length; i++) {
+        if (kind[i] !== CODE)
+            continue;
+        if (src[i] === '(')
+            depth++;
+        else if (src[i] === ')' && --depth === 0)
+            return i;
+    }
+    return -1;
+}

package/dist/mutate.js

@@ -0,0 +1,266 @@
+import { writeFileSync } from 'node:fs';
+import { basename, dirname, join } from 'node:path';
+import { invertAssertLine } from './invert.js';
+import { CODE, LINE_COMMENT, findCloseMasked, lex } from './lex.js';
+export const MARKER = '// @primary-assert';
+/** Read the test title: the first argument when it is a literal without interpolation. */
+function readTitle(src, open) {
+    let i = open + 1;
+    while (i < src.length && /\s/.test(src[i]))
+        i++;
+    const quote = src[i];
+    if (quote !== "'" && quote !== '"' && quote !== '`')
+        return '(dynamic title)';
+    let title = '';
+    for (i++; i < src.length; i++) {
+        const c = src[i];
+        if (c === '\\' && i + 1 < src.length) {
+            title += src[i + 1];
+            i++;
+        }
+        else if (quote === '`' && c === '$' && src[i + 1] === '{') {
+            return '(dynamic title)';
+        }
+        else if (c === quote) {
+            return title;
+        }
+        else {
+            title += c;
+        }
+    }
+    return title;
+}
+const IDENT = '[A-Za-z_$][\\w$]*';
+/** True when the first argument at `open` (the call's `(`) is a string literal. */
+function firstArgIsStringLiteral(src, open) {
+    let i = open + 1;
+    while (i < src.length && /\s/.test(src[i]))
+        i++;
+    const c = src[i];
+    return c === "'" || c === '"' || c === '`';
+}
+/**
+ * The set of identifiers that declare a test in this file. Always `test`, plus
+ * any passed in explicitly (`--test-id`), plus auto-detected custom fixtures:
+ * a local `const X = <known>.extend(...)` chain, or a renamed import
+ * (`import { test as X }`). Resolved to a fixpoint so chained extends and
+ * renames of just-discovered ids are picked up too.
+ */
+function collectTestIds(src, kind, extra) {
+    const ids = new Set(['test', ...extra]);
+    const renameRe = new RegExp(`\\b(${IDENT})\\s+as\\s+(${IDENT})`, 'g');
+    const extendRe = new RegExp(`\\b(?:const|let|var)\\s+(${IDENT})\\s*=\\s*(${IDENT})\\s*\\.\\s*extend\\s*\\(`, 'g');
+    let changed = true;
+    while (changed) {
+        changed = false;
+        for (const m of src.matchAll(renameRe)) {
+            if (kind[m.index] !== CODE)
+                continue;
+            if (ids.has(m[1]) && !ids.has(m[2])) {
+                ids.add(m[2]);
+                changed = true;
+            }
+        }
+        for (const m of src.matchAll(extendRe)) {
+            if (kind[m.index] !== CODE)
+                continue;
+            if (ids.has(m[2]) && !ids.has(m[1])) {
+                ids.add(m[1]);
+                changed = true;
+            }
+        }
+    }
+    return ids;
+}
+function findTestBlocks(src, kind, ids) {
+    const alt = [...ids].sort((a, b) => b.length - a.length).join('|');
+    const anchor = new RegExp(`\\b(?:${alt})\\s*(?:\\.\\s*(only|skip|fixme|fail)\\s*)?\\(`, 'g');
+    const blocks = [];
+    for (const m of src.matchAll(anchor)) {
+        if (kind[m.index] !== CODE)
+            continue;
+        // Excludes member calls like `t.test(...)`; `test.describe(` and
+        // `test.step(` never match the regex in the first place.
+        if (m.index > 0 && src[m.index - 1] === '.')
+            continue;
+        const mode = (m[1] ?? '');
+        const open = m.index + m[0].length - 1;
+        // `test.skip('title', fn)` declares a skipped test; `test.skip(!cond)` is a
+        // runtime modifier statement inside a test body. Only the declaration form
+        // (string title) is a block; the modifier is left to sit inside its test.
+        if (UNGATEABLE_MODES.has(mode) && !firstArgIsStringLiteral(src, open))
+            continue;
+        const close = findCloseMasked(src, kind, open);
+        blocks.push({
+            title: readTitle(src, open),
+            mode,
+            anchor: m.index,
+            open,
+            close: close === -1 ? src.length : close,
+            markers: [],
+        });
+    }
+    return blocks;
+}
+// test.skip/fixme never run and test.fail inverts pass/fail semantics, so a
+// mutation verdict on them would be a lie; they are also exempt from sentinel
+// accounting. test.only runs and is treated like a plain test.
+const UNGATEABLE_MODES = new Set(['skip', 'fixme', 'fail']);
+/**
+ * Parse a spec source and plan its mutations: one inverted copy per marked
+ * test. Anything the gate cannot honestly mutate is reported with a reason,
+ * never skipped silently.
+ */
+export function planSpecMutations(source, opts = {}) {
+    const kind = lex(source);
+    const ids = collectTestIds(source, kind, opts.testIds ?? []);
+    const blocks = findTestBlocks(source, kind, ids);
+    const lines = source.split('\n');
+    // Char offset of each line start, to map offsets to 1-based line numbers.
+    const starts = [0];
+    for (let i = 0; i < source.length; i++)
+        if (source[i] === '\n')
+            starts.push(i + 1);
+    const lineOf = (offset) => {
+        let n = 0;
+        while (n + 1 < starts.length && starts[n + 1] <= offset)
+            n++;
+        return n + 1;
+    };
+    // A line "has code" when any of its chars is CODE and not whitespace;
+    // blank, comment-only and string-interior lines do not.
+    const lineHasCode = (n) => {
+        for (let j = starts[n], end = starts[n] + lines[n].length; j < end; j++) {
+            if (kind[j] === CODE && !/\s/.test(source[j]))
+                return true;
+        }
+        return false;
+    };
+    // The `//` line comment on a line (in code context), or null when the line
+    // has none. `codeBefore` distinguishes a trailing marker (assertion code
+    // precedes the comment) from a whole-line one.
+    const lineComment = (n) => {
+        const start = starts[n];
+        const end = start + lines[n].length;
+        let c = -1;
+        for (let j = start; j < end; j++) {
+            if (kind[j] === LINE_COMMENT) {
+                c = j;
+                break;
+            }
+        }
+        if (c === -1)
+            return null;
+        let codeBefore = false;
+        for (let j = start; j < c; j++) {
+            if (kind[j] === CODE && !/\s/.test(source[j])) {
+                codeBefore = true;
+                break;
+            }
+        }
+        return { codeBefore, text: source.slice(c, end).trim() };
+    };
+    const strayMarkerLines = [];
+    const malformedMarkerLines = [];
+    for (let n = 0; n < lines.length; n++) {
+        const line = lines[n];
+        if (!line.includes('@primary-assert'))
+            continue;
+        // A valid marker is a `// @primary-assert` line comment in code context,
+        // either on its own line (assertion is the next code line) or trailing the
+        // assertion. The same text inside a block comment, string, template, or as
+        // a near-miss (`// @primary-assert: foo`) is not a marker.
+        const lc = lineComment(n);
+        if (!lc || lc.text !== MARKER) {
+            malformedMarkerLines.push(n + 1);
+            continue;
+        }
+        const firstNonWs = starts[n] + (line.length - line.trimStart().length);
+        const block = blocks.find((b) => firstNonWs > b.open && firstNonWs < b.close);
+        if (block)
+            block.markers.push({ line: n + 1, sameLine: lc.codeBefore });
+        else
+            strayMarkerLines.push(n + 1);
+    }
+    const mutations = [];
+    const unmarkedTests = [];
+    const onlyTestLines = [];
+    for (const block of blocks) {
+        const testLine = lineOf(block.anchor);
+        if (block.mode === 'only')
+            onlyTestLines.push(testLine);
+        if (block.markers.length === 0) {
+            if (!UNGATEABLE_MODES.has(block.mode)) {
+                unmarkedTests.push({ title: block.title, line: testLine });
+            }
+            continue;
+        }
+        const marker = block.markers[0];
+        const base = { testTitle: block.title, testLine, markerLine: marker.line };
+        if (UNGATEABLE_MODES.has(block.mode)) {
+            const why = block.mode === 'fail'
+                ? 'test.fail() expects failure, so a mutation verdict would be meaningless'
+                : `test.${block.mode}() never runs, so a mutation cannot be observed`;
+            mutations.push({ ...base, assertLine: null, cannotMutate: why });
+            continue;
+        }
+        if (block.markers.length > 1) {
+            mutations.push({
+                ...base,
+                assertLine: null,
+                cannotMutate: `found ${block.markers.length} @primary-assert markers in one test, expected exactly one`,
+            });
+            continue;
+        }
+        // A trailing marker sits on the assertion itself; a whole-line marker
+        // points at the next line that contains code (blank and comment-only lines
+        // are skipped, not mutated).
+        let target; // 0-based index of the assertion line
+        if (marker.sameLine) {
+            target = marker.line - 1;
+        }
+        else {
+            target = marker.line; // 0-based index of the line after the marker
+            while (target < lines.length && !lineHasCode(target))
+                target++;
+            if (target >= lines.length) {
+                mutations.push({ ...base, assertLine: null, cannotMutate: 'no code line follows the marker' });
+                continue;
+            }
+        }
+        const inverted = invertAssertLine(lines[target]);
+        if ('cannotMutate' in inverted) {
+            mutations.push({ ...base, assertLine: target + 1, cannotMutate: inverted.cannotMutate });
+            continue;
+        }
+        const mutatedLines = lines.slice();
+        mutatedLines[target] = inverted.ok;
+        mutations.push({ ...base, assertLine: target + 1, mutatedSource: mutatedLines.join('\n') });
+    }
+    return { mutations, unmarkedTests, strayMarkerLines, malformedMarkerLines, onlyTestLines };
+}
+/**
+ * Name for a mutated copy: the marker segment goes right after the first
+ * dot-segment of the basename, so the ENTIRE original tail is preserved and
+ * the copy matches whatever testMatch convention picked up the original
+ * (`checkout.e2e.ts` -> `checkout.pmg-mutation-0.e2e.ts`).
+ */
+export function mutationFilePath(specPath, index, tag = '') {
+    const base = basename(specPath);
+    const dot = base.indexOf('.', 1); // a leading dot (dotfile) stays with the stem
+    const stem = dot === -1 ? base : base.slice(0, dot);
+    const tail = dot === -1 ? '' : base.slice(dot);
+    const marker = tag === '' ? `pmg-mutation-${index}` : `pmg-mutation-${index}-${tag}`;
+    return join(dirname(specPath), `${stem}.${marker}${tail}`);
+}
+/**
+ * Write the mutated copy next to the original spec (same dir keeps relative
+ * imports and the Playwright testDir working) and return its path. The name
+ * carries the process id so overlapping gate runs in one checkout cannot
+ * collide. The caller owns cleanup.
+ */
+export function writeMutationFile(specPath, index, mutatedSource) {
+    const path = mutationFilePath(specPath, index, process.pid.toString(36));
+    writeFileSync(path, mutatedSource);
+    return path;
+}

package/dist/report.js

@@ -0,0 +1,99 @@
+export function summarize(specs, opts) {
+    const s = {
+        killed: 0,
+        survived: 0,
+        cannotMutate: 0,
+        errors: 0,
+        unmarkedTests: 0,
+        strayMarkers: 0,
+        malformedMarkers: 0,
+        gatePassed: true,
+    };
+    for (const spec of specs) {
+        for (const m of spec.results) {
+            if (m.verdict === 'killed')
+                s.killed++;
+            else if (m.verdict === 'survived')
+                s.survived++;
+            else if (m.verdict === 'cannot-mutate')
+                s.cannotMutate++;
+            else
+                s.errors++;
+        }
+        s.unmarkedTests += spec.unmarkedTests.length;
+        s.strayMarkers += spec.strayMarkerLines.length;
+        s.malformedMarkers += spec.malformedMarkerLines.length;
+    }
+    s.gatePassed =
+        s.survived === 0 &&
+            s.cannotMutate === 0 &&
+            s.errors === 0 &&
+            s.strayMarkers === 0 &&
+            s.malformedMarkers === 0 &&
+            (!opts.requireSentinel || s.unmarkedTests === 0);
+    return s;
+}
+export function jsonReport(specs, opts) {
+    return JSON.stringify({ specs, summary: summarize(specs, opts) }, null, 2);
+}
+const RESET = '\x1b[0m';
+const VERDICT_COLOR = {
+    killed: '\x1b[32m',
+    survived: '\x1b[31m',
+    'cannot-mutate': '\x1b[33m',
+    error: '\x1b[33m',
+};
+const TITLE_WIDTH = 48;
+function clip(title) {
+    return title.length <= TITLE_WIDTH ? title : title.slice(0, TITLE_WIDTH - 3) + '...';
+}
+/**
+ * Human-readable report: per spec, one row per verdict plus sentinel findings,
+ * then a one-line summary and the gate outcome.
+ */
+export function renderTable(specs, opts) {
+    const paint = (color, text) => opts.color ? color + text + RESET : text;
+    const labelWidth = 'cannot-mutate'.length;
+    const titleWidth = Math.min(TITLE_WIDTH, Math.max(4, ...specs.flatMap((spec) => [
+        ...spec.results.map((m) => clip(m.testTitle).length),
+        ...spec.unmarkedTests.map((t) => clip(t.title).length),
+    ])));
+    const row = (label, color, title, line) => '  ' + paint(color, label.padEnd(labelWidth)) + '  ' + clip(title).padEnd(titleWidth) + '  ' + line;
+    const lines = [];
+    for (const spec of specs) {
+        lines.push(spec.specPath);
+        for (const m of spec.results) {
+            const at = m.assertLine === null ? ':?' : `:${m.assertLine}`;
+            lines.push(row(m.verdict, VERDICT_COLOR[m.verdict], m.testTitle, at));
+            if (m.reason !== undefined) {
+                lines.push('  ' + ' '.repeat(labelWidth) + '  ' + m.reason);
+            }
+        }
+        for (const t of spec.unmarkedTests) {
+            lines.push(row('unmarked', VERDICT_COLOR['cannot-mutate'], t.title, `:${t.line}`));
+        }
+        for (const line of spec.strayMarkerLines) {
+            lines.push(`  marker outside any test() block at line ${line}`);
+        }
+        for (const line of spec.malformedMarkerLines) {
+            lines.push(`  malformed @primary-assert marker at line ${line}`);
+        }
+        lines.push('');
+    }
+    const s = summarize(specs, opts);
+    const counts = [
+        `${s.killed} killed`,
+        `${s.survived} survived`,
+        `${s.cannotMutate} cannot-mutate`,
+        `${s.errors} errors`,
+    ];
+    if (s.unmarkedTests > 0) {
+        counts.push(`${s.unmarkedTests} unmarked ${s.unmarkedTests === 1 ? 'test' : 'tests'}` +
+            (opts.requireSentinel ? '' : ' (not enforced)'));
+    }
+    lines.push(counts.join(', '));
+    lines.push(s.gatePassed
+        ? paint('\x1b[1;32m', 'Gate passed: every marked assertion can fail.')
+        : paint('\x1b[1;31m', 'Gate FAILED: hollow, unverified or broken assertions above.'));
+    return lines.join('\n');
+}

package/dist/run.js

@@ -0,0 +1,240 @@
+import { spawn } from 'node:child_process';
+import { mkdtempSync, readdirSync, readFileSync, rmSync, unlinkSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { basename, dirname, join, relative } from 'node:path';
+import { planSpecMutations, writeMutationFile } from './mutate.js';
+export const DEFAULT_TIMEOUT_MS = 5 * 60 * 1000;
+const DEFAULT_COMMAND = ['npx', 'playwright', 'test'];
+/** Matches the copy names produced by mutationFilePath, pid token included. */
+const MUTATION_FILE_RE = /\.pmg-mutation-\d+-([a-z0-9]+)\./;
+const OUTPUT_TAIL_LIMIT = 4000;
+function runChild(command, cwd, env, timeoutMs) {
+    return new Promise((resolve) => {
+        // A detached child gets its own process group, so on timeout the kill
+        // reaches Playwright's browser and web-server descendants too.
+        const detached = process.platform !== 'win32';
+        const child = spawn(command[0], command.slice(1), {
+            cwd,
+            env,
+            detached,
+            stdio: ['ignore', 'pipe', 'pipe'],
+        });
+        let output = '';
+        let timedOut = false;
+        const append = (chunk) => {
+            output = (output + chunk.toString()).slice(-OUTPUT_TAIL_LIMIT);
+        };
+        child.stdout.on('data', append);
+        child.stderr.on('data', append);
+        const timer = setTimeout(() => {
+            timedOut = true;
+            try {
+                if (detached && child.pid !== undefined)
+                    process.kill(-child.pid, 'SIGKILL');
+                else
+                    child.kill('SIGKILL');
+            }
+            catch {
+                // Already gone.
+            }
+        }, timeoutMs);
+        child.on('error', (err) => {
+            clearTimeout(timer);
+            resolve({ code: null, timedOut, spawnError: err.message, output });
+        });
+        child.on('close', (code) => {
+            clearTimeout(timer);
+            resolve({ code, timedOut, output });
+        });
+    });
+}
+/**
+ * Decide a verdict from a Playwright JSON report. The exit code alone cannot
+ * be trusted: Playwright exits 1 for a failed test, for "no tests found" and
+ * for many crashes alike, and only the report tells those apart.
+ */
+export function verdictFromReport(raw) {
+    let report;
+    try {
+        report = JSON.parse(raw);
+    }
+    catch {
+        return { verdict: 'error', reason: 'Playwright JSON report is not valid JSON' };
+    }
+    const stats = report.stats;
+    const expected = typeof stats?.expected === 'number' ? stats.expected : null;
+    const unexpected = typeof stats?.unexpected === 'number' ? stats.unexpected : null;
+    const flaky = typeof stats?.flaky === 'number' ? stats.flaky : null;
+    const skipped = typeof stats?.skipped === 'number' ? stats.skipped : 0;
+    if (expected === null || unexpected === null || flaky === null) {
+        return { verdict: 'error', reason: 'Playwright JSON report has no stats block' };
+    }
+    if (expected + unexpected + flaky === 0) {
+        // A test that matched the filter but was skipped at runtime (a conditional
+        // test.skip, e.g. a fixture with no captured session) never executed the
+        // inverted assertion. Reporting survived would be a lie; it is cannot-mutate.
+        if (skipped > 0) {
+            return {
+                verdict: 'cannot-mutate',
+                reason: 'the test was skipped at runtime (conditional test.skip), so the inverted assertion never ran',
+            };
+        }
+        const message = report.errors?.map((e) => e.message).find((m) => typeof m === 'string');
+        const detail = message ? `: ${message.split('\n', 1)[0]}` : '';
+        return { verdict: 'error', reason: `no tests ran under the mutation filter${detail}` };
+    }
+    // A flaky test failed at least once under the inversion, so the assertion
+    // demonstrably can fail; retries just masked it. That counts as killed.
+    if (unexpected + flaky > 0)
+        return { verdict: 'killed' };
+    return { verdict: 'survived' };
+}
+/**
+ * Run one planned mutation: write the inverted copy next to the spec, run
+ * Playwright against exactly that copy and test line, and read the verdict
+ * from the JSON report. The copy and the report directory are always removed.
+ */
+export async function runMutation(specPath, index, mutation, opts = {}) {
+    const cwd = opts.cwd ?? process.cwd();
+    const timeoutMs = opts.timeoutMs ?? DEFAULT_TIMEOUT_MS;
+    const command = opts.playwrightCommand ?? DEFAULT_COMMAND;
+    const copyPath = writeMutationFile(specPath, index, mutation.mutatedSource);
+    const reportDir = mkdtempSync(join(tmpdir(), 'pmg-'));
+    const reportPath = join(reportDir, 'report.json');
+    try {
+        // The copy replaces a single line, so line numbers match the original
+        // and the test declaration line targets exactly the mutated test.
+        const target = `${relative(cwd, copyPath) || copyPath}:${mutation.testLine}`;
+        const outcome = await runChild([...command, target, '--reporter=json', ...(opts.playwrightArgs ?? [])], cwd, { ...process.env, PLAYWRIGHT_JSON_OUTPUT_NAME: reportPath }, timeoutMs);
+        if (outcome.spawnError) {
+            return {
+                verdict: 'error',
+                reason: `could not launch ${command.join(' ')}: ${outcome.spawnError}`,
+            };
+        }
+        if (outcome.timedOut) {
+            return {
+                verdict: 'error',
+                reason: `Playwright run timed out after ${Math.round(timeoutMs / 1000)}s and was killed`,
+            };
+        }
+        let raw;
+        try {
+            raw = readFileSync(reportPath, 'utf8');
+        }
+        catch {
+            const tail = outcome.output.trim();
+            return {
+                verdict: 'error',
+                reason: `Playwright exited with code ${outcome.code ?? 'unknown'} without producing a report` +
+                    (tail === '' ? '' : `; output tail: ${tail.slice(-500)}`),
+            };
+        }
+        return verdictFromReport(raw);
+    }
+    finally {
+        try {
+            unlinkSync(copyPath);
+        }
+        catch {
+            // Already gone.
+        }
+        rmSync(reportDir, { recursive: true, force: true });
+    }
+}
+function pidIsAlive(pid) {
+    try {
+        process.kill(pid, 0);
+        return true;
+    }
+    catch (err) {
+        // EPERM means the pid exists but belongs to someone else: alive.
+        return err.code === 'EPERM';
+    }
+}
+/**
+ * Delete leftover mutation copies in a directory: files whose pid token
+ * points at a process that no longer exists. Copies of live concurrent gate
+ * runs are left alone. Returns the paths that were removed.
+ */
+export function sweepStaleMutationFiles(dir) {
+    const removed = [];
+    let entries;
+    try {
+        entries = readdirSync(dir);
+    }
+    catch {
+        return removed;
+    }
+    for (const entry of entries) {
+        const token = MUTATION_FILE_RE.exec(entry)?.[1];
+        if (token === undefined)
+            continue;
+        const pid = parseInt(token, 36);
+        if (Number.isNaN(pid) || pidIsAlive(pid))
+            continue;
+        const path = join(dir, entry);
+        try {
+            unlinkSync(path);
+            removed.push(path);
+        }
+        catch {
+            // Raced with another sweep.
+        }
+    }
+    return removed;
+}
+/**
+ * Gate one spec file end to end: plan its mutations, sweep stale copies from
+ * earlier crashed runs, run every mutable mutation and collect verdicts.
+ * A spec containing test.only is refused outright: .only filters out every
+ * other test, so verdicts from such a run would be fiction.
+ */
+export async function runSpec(specPath, opts = {}) {
+    const source = readFileSync(specPath, 'utf8');
+    const plan = planSpecMutations(source, { testIds: opts.testIds });
+    const results = [];
+    if (plan.onlyTestLines.length > 0) {
+        const reason = `test.only on line ${plan.onlyTestLines.join(', ')} of ${basename(specPath)} ` +
+            'filters out every other test, so no mutation verdict would be honest; remove .only first';
+        for (const m of plan.mutations) {
+            results.push({
+                testTitle: m.testTitle,
+                testLine: m.testLine,
+                assertLine: m.assertLine,
+                verdict: 'error',
+                reason,
+            });
+        }
+    }
+    else {
+        sweepStaleMutationFiles(dirname(specPath));
+        for (const [index, m] of plan.mutations.entries()) {
+            if ('cannotMutate' in m) {
+                results.push({
+                    testTitle: m.testTitle,
+                    testLine: m.testLine,
+                    assertLine: m.assertLine,
+                    verdict: 'cannot-mutate',
+                    reason: m.cannotMutate,
+                });
+                continue;
+            }
+            const run = await runMutation(specPath, index, m, opts);
+            results.push({
+                testTitle: m.testTitle,
+                testLine: m.testLine,
+                assertLine: m.assertLine,
+                verdict: run.verdict,
+                ...('reason' in run ? { reason: run.reason } : {}),
+            });
+        }
+    }
+    return {
+        specPath,
+        results,
+        unmarkedTests: plan.unmarkedTests,
+        strayMarkerLines: plan.strayMarkerLines,
+        malformedMarkerLines: plan.malformedMarkerLines,
+    };
+}

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "playwright-mutation-gate",
-  "version": "0.0.1",
-  "description": "Mutation testing for Playwright assertions: invert each test's primary assertion and fail the run if the test stays green. Name reservation; v0.1.0 in development.",
+  "version": "0.1.0",
+  "description": "Mutation testing for Playwright assertions: invert each test's primary assertion and fail the run if the test stays green.",
   "keywords": [
     "playwright",
     "mutation-testing",
@@ -15,14 +15,36 @@
   ],
   "license": "MIT",
   "author": "Vladyslav Dmitriiev <vladyslav.dmitriiev@pm.me>",
-  "main": "index.js",
+  "type": "module",
   "bin": {
-    "playwright-mutation-gate": "index.js"
+    "playwright-mutation-gate": "dist/cli.js"
   },
   "files": [
-    "index.js"
+    "dist"
   ],
+  "scripts": {
+    "build": "tsc -p tsconfig.build.json",
+    "typecheck": "tsc --noEmit",
+    "lint": "eslint .",
+    "test:unit": "vitest run",
+    "test:e2e": "playwright test",
+    "test:integration": "vitest run --config vitest.integration.config.ts",
+    "test": "npm run lint && npm run typecheck && npm run build && npm run test:unit && npm run test:e2e && npm run test:integration"
+  },
   "engines": {
-    "node": ">=18"
+    "node": ">=20"
+  },
+  "devDependencies": {
+    "@eslint/js": "^10.0.1",
+    "@playwright/test": "^1.61.1",
+    "@types/node": "^26.1.1",
+    "eslint": "^10.7.0",
+    "globals": "^17.7.0",
+    "typescript": "^6.0.3",
+    "typescript-eslint": "^8.63.0",
+    "vitest": "^4.1.10"
+  },
+  "dependencies": {
+    "commander": "^15.0.0"
   }
 }

package/index.js

@@ -1,6 +0,0 @@
-#!/usr/bin/env node
-console.error(
-  'playwright-mutation-gate 0.0.1 is a name reservation.\n' +
-    'The actual CLI ships as v0.1.0. Watch the npm page for updates.'
-);
-process.exit(1);