playwright-mutation-gate 0.0.1 → 0.2.0

package/README.md

@@ -1,39 +1,267 @@
 # playwright-mutation-gate
 
+[![npm version](https://img.shields.io/npm/v/playwright-mutation-gate)](https://www.npmjs.com/package/playwright-mutation-gate)
+[![CI](https://github.com/VladyslavDmitriiev/playwright-mutation-gate/actions/workflows/ci.yml/badge.svg)](https://github.com/VladyslavDmitriiev/playwright-mutation-gate/actions)
+[![license](https://img.shields.io/npm/l/playwright-mutation-gate)](./LICENSE)
+
 > A passing test only means something if it can fail.
 
-**Status: v0.0.1 is a name reservation.** The actual CLI is in development and ships as v0.1.0.
+`playwright-mutation-gate` inverts the assertion you marked as primary and
+re-runs the test. If it stays green, the assertion is hollow and the gate
+fails your CI run instead of reporting a lie.
+
+<p align="center">
+  <img src="media/demo.svg" alt="Terminal recording: 3 killed, 1 survived, gate fails" width="840">
+</p>
 
-## What it will do
+<details><summary>Text version (if the SVG above does not render)</summary>
 
-`playwright-mutation-gate` checks that your green Playwright tests are green for a reason.
-You mark the primary assertion in each test:
+```
+fixtures/demo.spec.ts
+  killed         shows error for invalid email  :8
+  killed         hides error for valid email    :16
+  killed         shows success for valid email  :24
+  survived       hides success initially        :31
+
+3 killed, 1 survived, 0 cannot-mutate, 0 errors
+Gate FAILED: hollow, unverified or broken assertions above.
+```
+</details>
+
+## Quick start
+
+Requires **Node >= 20** and a working **Playwright** setup (any version that
+supports `playwright test`).
+
+```bash
+npm install -D playwright-mutation-gate
+```
+
+Mark the primary assertion in each test with `// @primary-assert`. Put it at
+the end of the assertion line, or on its own line right above it. The assertion
+itself must be a single-line `expect(...)` call.
 
 ```ts
-test('checkout shows order confirmation', async ({ page }) => {
+test('checkout shows confirmation', async ({ page }) => {
   // ...
+  await expect(page.getByText('Order confirmed')).toBeVisible(); // @primary-assert
+});
+
+// Also valid: the marker on its own line above the assertion.
+test('cart badge updates', async ({ page }) => {
   // @primary-assert
-  await expect(page.getByText('Order confirmed')).toBeVisible();
+  await expect(page.getByTestId('cart-count')).toHaveText('1');
+});
+```
+
+Run the gate:
+
+```bash
+npx playwright-mutation-gate "tests/**/*.spec.ts"
+```
+
+## How marking works
+
+The gate inverts the marked assertion
+(`expect(x).toBeVisible()` becomes `expect(x).not.toBeVisible()`) in a
+temporary copy of the spec and re-runs the test in isolation.
+
+- **killed** -- the test failed under inversion. The assertion is alive.
+- **survived** -- the test still passed. The assertion checks nothing useful.
+- **cannot-mutate** -- the assertion shape could not be inverted safely. The
+  gate reports it instead of guessing.
+
+One marker per `test()` block. Markers inside `test.describe` or `test.use`
+callbacks work the same way -- the gate scans each `test()` individually,
+nesting does not matter.
+
+By default, tests without a marker fail the gate (sentinel mode). This means
+a newly added test without `// @primary-assert` shows up as:
+
+```
+  unmarked       adds item to cart  :42
+```
+
+Pass `--no-require-sentinel` to opt out while you're marking an existing suite.
+
+## Custom test fixtures
+
+If your tests are declared through a custom fixture instead of the bare `test`,
+the gate finds it automatically in two cases: a local `.extend` chain and a
+renamed import.
+
+```ts
+import { test as base } from '@playwright/test';
+const authedTest = base.extend({ /* ... */ });   // auto-detected
+
+authedTest('sees dashboard', async ({ page }) => {
+  await expect(page.getByText('Welcome')).toBeVisible(); // @primary-assert
+});
+```
+
+When the fixture is imported by its own name from another module, name it with
+`--test-id` (repeatable):
+
+```bash
+npx playwright-mutation-gate tests/dashboard.spec.ts --test-id authedTest
+```
+
+A runtime `authedTest.skip(condition)` inside a test is respected: if the test
+skips, the mutation reports `cannot-mutate` rather than a false `survived`.
+
+## CI recipe
+
+```yaml
+# .github/workflows/mutation-gate.yml
+name: mutation-gate
+
+on: [push, pull_request]
+
+jobs:
+  gate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+          cache: npm
+      - run: npm ci
+      - run: npx playwright install --with-deps chromium
+      - run: npx playwright-mutation-gate "tests/**/*.spec.ts"
+```
+
+The gate exits 0 only when every marked assertion is killed and sentinel
+mode is satisfied. Any other outcome exits 1.
+
+Cost scales linearly: every marked assertion is one extra `playwright test`
+run. That is fine for a handful of specs, but on a large suite you probably
+do not want the full sweep on every push. Two patterns that work well:
+
+- **Gate only what changed.** The CLI takes plain file paths, so feed it the
+  spec files the PR touched:
+
+  ```yaml
+      - run: |
+          SPECS=$(git diff --name-only origin/${{ github.base_ref }}... -- 'tests/**/*.spec.ts')
+          [ -n "$SPECS" ] && npx playwright-mutation-gate $SPECS || echo "no specs changed"
+  ```
+
+- **Full sweep on a schedule.** Run the whole glob nightly instead of per
+  commit. A hollow assertion does not get less hollow between merges;
+  catching it a day later is fine. `--playwright-args` is available if you
+  need to shard or point at a specific config.
+
+## CLI reference
+
+```
+npx playwright-mutation-gate [options] <specs...>
+
+Options:
+  --json                    machine-readable JSON report
+  --no-require-sentinel     do not fail on tests without a marker
+  --playwright-args <args>  extra arguments for playwright test
+  --test-id <name>          treat <name> as a test builder (repeatable)
+  -V, --version             output the version number
+  -h, --help                display help for command
+```
+
+## Programmatic API
+
+Since v0.2 the package also exposes a library entry, so a tool can run the gate
+in-process and inspect the structured report instead of shelling out to the CLI
+and parsing `--json`.
+
+```ts
+import { runSpec, summarize } from 'playwright-mutation-gate';
+
+const spec = await runSpec('tests/checkout.spec.ts', {
+  testIds: ['authedTest'],            // extra test builders, same as --test-id
+  playwrightArgs: ['--project=chromium', '--retries=0'],
 });
+const summary = summarize([spec], { requireSentinel: true });
+
+if (!summary.gatePassed) {
+  const hollow = spec.results.filter((r) => r.verdict === 'survived');
+  console.error(`${hollow.length} assertion(s) do not gate the result`);
+  process.exit(1);
+}
 ```
 
-The gate then inverts that assertion (`expect(x).toBeVisible()` → `expect(x).not.toBeVisible()`)
-in a temporary copy of the spec and re-runs the test:
+`runSpec(specPath, opts?)` returns a `SpecResult` (per-assertion `verdict`:
+`killed` / `survived` / `cannot-mutate` / `error`, plus unmarked/stray/malformed
+markers). `summarize(specs, opts)` folds one or more results into a `Summary`
+with the `gatePassed` verdict. `jsonReport` and `renderTable` produce the CLI's
+two output formats. Advanced: `planSpecMutations` and `invertAssertLine` expose
+the mutation planner and the single-line inverter directly. Full types ship with
+the package.
 
-- test fails → the assertion is alive (**killed** mutation, good)
-- test still passes → the assertion checks nothing (**survived**, a hollow test), so the
-  gate exits 1 and fails your CI run
+## Limitations
 
-Tests without a `@primary-assert` marker also fail the gate by default, so nothing slips
-through unmarked.
+**The gate proves an assertion can fail, not that it checks the right thing.**
+A weak but falsifiable assertion passes: a test that asserts a site-wide header
+(one that also renders on the error page) is killed by inversion and clears the
+gate, even though it would not catch the feature breaking. What the gate
+guarantees is narrower and mechanical: every marked assertion actually runs and
+is load-bearing, so the test cannot stay green regardless of what the app does.
+Judging whether the assertion is the *right* one still takes review, or
+application-code mutation testing (Stryker-style), which is a different tool.
+
+**A kill is slightly generous by construction.** `killed` means the mutated
+run failed, not that the flipped assertion is what failed. Playwright actions
+carry implicit assertions (`click()` fails on its own if the element never
+appears), and an ordinarily flaky step also counts, so a test can die before
+it ever reaches the inverted line. The asymmetry works in your favor:
+`survived` has no such excuse. A test that stays green with its own primary
+assertion inverted is hollow, full stop.
+
+String-based inversion, not AST. This keeps the tool small and dependency-free
+but means some shapes report `cannot-mutate`:
+
+- **Single line only.** `expect(...)` and its matcher must start on the marked
+  line. Multi-line chains report cannot-mutate.
+- **One assertion per line.** Two `expect` calls on the same line (including
+  asymmetric matchers in arguments, nested expects inside `toPass`) report
+  cannot-mutate.
+- **No aliases.** Assertions using a re-exported or aliased `expect` are not
+  recognized.
+- **Custom matchers** must follow the `toXxx` convention. Names that shadow
+  built-ins (`toString`, `toFixed`, ...) are denylisted.
+- **Supported chain:**
+  `[await] expect[.soft|.poll](...)[.resolves|.rejects][.not].toXxx(`
+
+`cannot-mutate` is always safe: the gate tells you it could not verify the
+assertion, never silently skips it.
+
+Skipped tests get the same honesty. A test disabled with `test.skip` never
+executes, so there is nothing to observe when its assertion is inverted.
+Rather than guess, the gate reports `cannot-mutate` with the reason ("the
+test was skipped at runtime (conditional test.skip), so the inverted
+assertion never ran"). In
+practice this is a feature: a `test.skip(true, ...)` from months ago looks
+identical to a healthy passing spec in normal CI output, and the gate is
+often what surfaces it.
 
 ## Why
 
-Stryker and friends mutate your application code to score unit tests. E2E suites have the
-opposite blind spot: the assertion itself is often hollow. It passes against a broken app.
-This tool mutates the assertions, not the app.
+AI code generators write tests fast. The tests look right: good names, proper
+selectors, green checkmarks. But the assertions are often hollow: a forgotten
+`await` whose result nobody checks, an assertion guarded behind a branch that
+never executes, a `not.toBeVisible` against a selector that never existed, a
+test silently disabled with `skip`.
+
+Stryker and friends mutate your *application code* to score unit tests. E2E
+suites have the opposite blind spot: the *assertion itself* is often the weak
+link. It passes against a broken app. `playwright-mutation-gate` mutates the
+assertions, not the app.
+
+If a green test cannot turn red when its assertion is inverted, it was never
+testing anything. The gate catches that before it reaches production.
+
+## Contributing
 
-v0.1.0 coming soon.
+Issues and PRs are welcome. Run `npm test` before submitting -- it covers
+lint, typecheck, unit, e2e and integration tests in one pass.
 
 ## License
 

package/dist/cli.d.ts

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+export {};

package/dist/cli.js

@@ -0,0 +1,64 @@
+#!/usr/bin/env node
+import { existsSync, readFileSync } from 'node:fs';
+import { Command, CommanderError } from 'commander';
+import { jsonReport, renderTable, summarize } from './report.js';
+import { runSpec } from './run.js';
+// Exit codes: 0 = gate passed, 1 = gate failed (survived / unverified /
+// marker mistakes), 2 = usage or environment error (bad flags, missing file).
+const pkg = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8'));
+const program = new Command()
+    .name('playwright-mutation-gate')
+    .description('Invert each @primary-assert assertion and re-run the test.\n' +
+    'A test that stays green under inversion is hollow and fails the gate.')
+    .version(pkg.version)
+    .argument('<specs...>', 'Playwright spec files to gate')
+    .option('--json', 'print a machine-readable JSON report instead of the table', false)
+    .option('--no-require-sentinel', 'do not fail the gate when a test() has no @primary-assert marker')
+    .option('--playwright-args <args>', 'extra arguments passed to `npx playwright test`, split on spaces\n' +
+    '(values containing spaces are not supported)')
+    .option('--test-id <name>', 'treat <name> as a test builder in addition to `test` (repeatable);\n' +
+    'for custom fixtures imported by name (import { authedTest } from ...).\n' +
+    'Local `const x = test.extend(...)` and `test as x` renames are auto-detected', (value, acc) => acc.concat(value), [])
+    .showHelpAfterError()
+    .addHelpText('after', '\nMarking:\n' +
+    '  Put `// @primary-assert` at the end of the primary assertion line,\n' +
+    '  or on its own line right above it. One marker per test().\n' +
+    '    await expect(page.getByText(\'Saved\')).toBeVisible(); // @primary-assert')
+    .action(async (specPaths, opts) => {
+    const missing = specPaths.filter((p) => !existsSync(p));
+    if (missing.length > 0) {
+        console.error(`spec file not found: ${missing.join(', ')}`);
+        process.exit(2);
+    }
+    const playwrightArgs = (opts.playwrightArgs ?? '').split(/\s+/).filter(Boolean);
+    const results = [];
+    for (const spec of specPaths) {
+        process.stderr.write(`gating ${spec}\n`);
+        try {
+            results.push(await runSpec(spec, { playwrightArgs, testIds: opts.testId }));
+        }
+        catch (err) {
+            console.error(`failed to gate ${spec}: ${err instanceof Error ? err.message : String(err)}`);
+            process.exit(2);
+        }
+    }
+    const gate = { requireSentinel: opts.requireSentinel };
+    if (opts.json) {
+        console.log(jsonReport(results, gate));
+    }
+    else {
+        const color = process.stdout.isTTY === true && process.env.NO_COLOR === undefined;
+        console.log(renderTable(results, { color, ...gate }));
+    }
+    process.exit(summarize(results, gate).gatePassed ? 0 : 1);
+});
+program.exitOverride();
+try {
+    await program.parseAsync();
+}
+catch (err) {
+    // commander has already printed its message; map usage errors to exit 2,
+    // keep 0 for --help / --version.
+    const code = err instanceof CommanderError && err.exitCode === 0 ? 0 : 2;
+    process.exit(code);
+}

package/dist/index.d.ts

@@ -0,0 +1,4 @@
+export { runSpec, sweepStaleMutationFiles, DEFAULT_TIMEOUT_MS, type SpecResult, type MutationResult, type Verdict, type RunOptions, } from './run.js';
+export { summarize, jsonReport, renderTable, type Summary, type GateOptions, type RenderOptions, } from './report.js';
+export { planSpecMutations, MARKER, type SpecPlan, type Mutation, type PlanOptions, } from './mutate.js';
+export { invertAssertLine, type InvertResult } from './invert.js';

package/dist/index.js

@@ -0,0 +1,15 @@
+// Programmatic entry point.
+//
+// The CLI (`dist/cli.js`) is one consumer of this library; another is any tool
+// that wants to run the gate in-process and inspect the report itself — e.g. a
+// custom test-pipeline that already knows which specs changed and wants the
+// structured `SpecResult`/`Summary` without shelling out and parsing `--json`.
+//
+// Curated surface: run a spec, summarize/render the results, and (advanced) plan
+// or invert individual mutations. The tokenizer (`lex.ts`) and the on-disk
+// mutant-file mechanics stay internal — `runSpec` orchestrates them for you.
+export { runSpec, sweepStaleMutationFiles, DEFAULT_TIMEOUT_MS, } from './run.js';
+export { summarize, jsonReport, renderTable, } from './report.js';
+// Advanced: build or inspect the mutation plan directly.
+export { planSpecMutations, MARKER, } from './mutate.js';
+export { invertAssertLine } from './invert.js';

package/dist/invert.d.ts

@@ -0,0 +1,16 @@
+export type InvertResult = {
+    ok: string;
+} | {
+    cannotMutate: string;
+};
+/**
+ * Invert the assertion on a single line of a Playwright spec:
+ * `expect(x).toY()` <-> `expect(x).not.toY()`.
+ *
+ * Purely string-based. Supported chain shape:
+ * `[await] expect[.soft|.poll](...) [.resolves|.rejects] [.not] .toXxx(`
+ * The matcher call must start on this line; its arguments may continue on
+ * following lines. Everything else returns `cannotMutate` with a reason
+ * rather than guessing.
+ */
+export declare function invertAssertLine(line: string): InvertResult;

package/dist/invert.js

@@ -0,0 +1,116 @@
+import { CODE, findCloseMasked, lex } from './lex.js';
+// Built-in toXxx methods that look like matchers but are not assertions.
+const NOT_MATCHERS = new Set([
+    'toString',
+    'toLocaleString',
+    'toJSON',
+    'toISOString',
+    'toFixed',
+    'toExponential',
+    'toPrecision',
+    'toDateString',
+    'toTimeString',
+    'toUTCString',
+    'toLocaleDateString',
+    'toLocaleTimeString',
+    'toLocaleLowerCase',
+    'toLocaleUpperCase',
+    'toLowerCase',
+    'toUpperCase',
+    'toReversed',
+    'toSorted',
+    'toSpliced',
+]);
+const isWord = (c) => c !== undefined && /[\w$]/.test(c);
+/**
+ * Invert the assertion on a single line of a Playwright spec:
+ * `expect(x).toY()` <-> `expect(x).not.toY()`.
+ *
+ * Purely string-based. Supported chain shape:
+ * `[await] expect[.soft|.poll](...) [.resolves|.rejects] [.not] .toXxx(`
+ * The matcher call must start on this line; its arguments may continue on
+ * following lines. Everything else returns `cannotMutate` with a reason
+ * rather than guessing.
+ */
+export function invertAssertLine(line) {
+    const kind = lex(line);
+    // First CODE index at or after j: skips whitespace, comments and other
+    // non-code chars (so a comment inside the chain does not break the walk).
+    const skip = (j) => {
+        while (j < line.length && (kind[j] !== CODE || /\s/.test(line[j])))
+            j++;
+        return j;
+    };
+    const anchors = [...line.matchAll(/\bexpect\s*[.(]/g)].filter((m) => kind[m.index] === CODE);
+    if (anchors.length === 0) {
+        return { cannotMutate: 'no expect() call found on this line' };
+    }
+    if (anchors.length > 1) {
+        return { cannotMutate: 'multiple expect calls on one line' };
+    }
+    // Position of the `.` or `(` right after the `expect` token.
+    let i = anchors[0].index + anchors[0][0].length - 1;
+    if (line[i] === '.') {
+        i = skip(i + 1);
+        const start = i;
+        while (isWord(line[i]))
+            i++;
+        const form = line.slice(start, i);
+        if (form !== 'soft' && form !== 'poll') {
+            return { cannotMutate: `unsupported expect form "expect.${form}"` };
+        }
+        i = skip(i);
+        if (line[i] !== '(') {
+            return { cannotMutate: `expect.${form} is not called on this line` };
+        }
+    }
+    const close = findCloseMasked(line, kind, i);
+    if (close === -1) {
+        return { cannotMutate: 'expect(...) does not close on this line (multi-line expect)' };
+    }
+    // Walk the chain after expect(...): modifiers, optional .not, then the matcher.
+    i = skip(close + 1);
+    let notStart = -1;
+    let notEnd = -1;
+    for (;;) {
+        if (i >= line.length) {
+            return { cannotMutate: 'no matcher call after expect(...) on this line' };
+        }
+        if (line[i] !== '.') {
+            return { cannotMutate: `unexpected "${line[i]}" after expect(...) chain` };
+        }
+        const dot = i;
+        i = skip(i + 1);
+        const start = i;
+        while (isWord(line[i]))
+            i++;
+        const id = line.slice(start, i);
+        if (id === '') {
+            return { cannotMutate: 'unexpected character in expect(...) chain' };
+        }
+        const lookahead = skip(i);
+        if (line[lookahead] === '(') {
+            if (!/^to[A-Z]/.test(id) || NOT_MATCHERS.has(id)) {
+                return { cannotMutate: `".${id}" does not look like an assertion matcher` };
+            }
+            const inverted = notStart !== -1
+                ? line.slice(0, notStart) + line.slice(notEnd)
+                : line.slice(0, dot) + '.not' + line.slice(dot);
+            return { ok: inverted };
+        }
+        if (id === 'not') {
+            if (notStart !== -1) {
+                return { cannotMutate: 'double .not in expect(...) chain' };
+            }
+            notStart = dot;
+            notEnd = i;
+            i = skip(i);
+            continue;
+        }
+        if (id === 'resolves' || id === 'rejects') {
+            i = skip(i);
+            continue;
+        }
+        return { cannotMutate: `unsupported chain segment ".${id}"` };
+    }
+}

package/dist/lex.d.ts

@@ -0,0 +1,23 @@
+/**
+ * Shared single-pass lexer for the string-based spec analysis. Classifies
+ * every char of a source (a whole file or a single line) so the scanners in
+ * invert.ts and mutate.ts agree about what is code. One implementation on
+ * purpose: two hand-rolled lexers had already diverged once.
+ */
+/** Char inside a string, template literal, block comment or regex literal. */
+export declare const NON_CODE = 0;
+/** Executable source. */
+export declare const CODE = 1;
+/** Char of a `//` comment that starts in code context. */
+export declare const LINE_COMMENT = 2;
+/**
+ * Classify every char of `src` as CODE, LINE_COMMENT or NON_CODE. Handles
+ * strings with escapes, template literals with nested `${}` expressions,
+ * `//` and block comments, and regex literals (heuristic: see startsRegex).
+ */
+export declare function lex(src: string): Uint8Array;
+/**
+ * Find the `)` matching the `(` at `open`, counting only CODE chars.
+ * Returns -1 when it does not close within `src`.
+ */
+export declare function findCloseMasked(src: string, kind: Uint8Array, open: number): number;