failfirst 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ben Malaga
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,173 @@
+<div align="center">
+
+# failfirst
+
+**A CI gate that proves new tests actually test the change.**
+
+A new test that passes on the base commit would pass without your change.
+It is not testing your change. failfirst catches it.
+
+[![release](https://img.shields.io/github/v/release/BenMalaga/failfirst?color=blue&label=release)](https://github.com/BenMalaga/failfirst/releases)
+[![CI](https://github.com/BenMalaga/failfirst/actions/workflows/test.yml/badge.svg)](https://github.com/BenMalaga/failfirst/actions/workflows/test.yml)
+[![node](https://img.shields.io/badge/node-%3E%3D18-brightgreen)](https://nodejs.org)
+[![dependencies](https://img.shields.io/badge/dependencies-0-brightgreen)](package.json)
+[![license](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
+
+</div>
+
+## The problem
+
+A pull request adds a feature and a test. The test passes. CI is green. Reviewer approves.
+
+But did the test ever depend on the feature? If you reverted the code change, would the test notice? With AI-generated PRs this failure mode went from occasional to routine: plausible-looking tests that assert things the old code already did. They pass before the change, they pass after the change, they pass after the next regression too. Green forever, guarding nothing.
+
+The classic discipline is "watch the test fail first." failfirst turns that discipline into a CI gate: it runs your branch's new tests against the code from before your branch. Any new test that passes there is flagged **vacuous**, and the gate fails.
+
+## See it
+
+A PR implements `multiply()` and adds two tests. One imports `multiply` and checks it. The other looks multiply-related but only exercises the old `add()` function. Both are green in a normal test run. failfirst tells them apart:
+
+```
+$ failfirst main
+failfirst v0.1.0
+  base     main (merge-base 4dbdb3a)
+  runner   node-test
+  changed  2 test file(s)
+
+  running 2 file(s) against base 4dbdb3a...
+  running 2 file(s) against HEAD...
+
+  TEST                                                                           BASE    HEAD   VERDICT
+  test/multiply-props.test.js > multiply: adding a number to itself doubles it   pass    pass   VACUOUS
+  test/multiply.test.js > multiply multiplies two numbers                        error   pass   GOOD
+
+  1 good, 1 vacuous
+
+FAIL: 1 new test passes on the base commit.
+A test that passes without your change is not testing your change.
+
+$ echo $?
+1
+```
+
+That transcript is a real run against the fixture repository used in this project's integration tests.
+
+## Install
+
+No install needed:
+
+```sh
+npx failfirst
+```
+
+Or add it to the project:
+
+```sh
+npm install --save-dev failfirst
+```
+
+Requires Node 18 or newer and git. Zero dependencies.
+
+## Usage
+
+```
+failfirst [base-ref] [options]
+
+Arguments:
+  base-ref          Base to compare against (default: origin/main, main,
+                    origin/master, or master, first one that exists)
+
+Options:
+  --runner <name>   node-test | vitest | jest (default: auto-detect)
+  --json            Machine-readable JSON output
+  --color           Force colored output
+  --no-color        Disable colored output
+  -h, --help        Show help
+  -v, --version     Show version
+
+Exit codes:
+  0  no vacuous tests (or no changed test files)
+  1  at least one vacuous test
+  2  usage or environment error
+```
+
+### GitHub Actions
+
+```yaml
+jobs:
+  failfirst:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0   # failfirst needs the merge-base commit
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+      - run: npm ci        # only needed for the vitest/jest runners
+      - run: npx failfirst origin/${{ github.base_ref }}
+```
+
+## How it works
+
+1. **Find the changed tests.** `git diff --name-status <merge-base>..HEAD`, filtered to test files (`*.test.*`, `*.spec.*`, `*_test.*`, `test-*.*`, and files under `test/`, `tests/`, `__tests__/`, `spec/` directories).
+2. **Rebuild the world before your change.** A temporary detached git worktree is created at the merge-base of the base ref and HEAD.
+3. **Overlay only the new tests.** The added and modified test files (plus changed support files from test directories) are copied into that worktree. Production code stays old; tests are new.
+4. **Handle modified test files fairly.** Before the overlay, failfirst runs the base version of each modified test file and records its test names. Those pre-existing tests are reported but never gated; only tests your branch actually added are judged.
+5. **Run twice, compare per test.** The changed test files run against the base worktree and against HEAD. Each new test gets a verdict:
+
+| Verdict | Base | HEAD | Meaning |
+|---|---|---|---|
+| `GOOD` | fail / error | pass | The test fails without your change. It proves something. |
+| `VACUOUS` | pass | pass | The test passes without your change. It proves nothing. Gate fails. |
+| `BROKEN` | fail | fail | The test fails everywhere. Your normal CI run will catch it. |
+| `PRE-EXISTING` | any | any | Already existed in the base version of a modified file. Not gated. |
+| `SKIPPED` | any | skip | Skipped on HEAD. Nothing to judge. |
+
+6. **Clean up.** The worktree is removed even when a run blows up.
+
+A `BASE` of `error` means the test file could not even load on the old code, typically because it imports something your branch introduced. That is strong evidence the test targets the change, so it counts as failing on base.
+
+## Runners
+
+| Runner | Detection | Notes |
+|---|---|---|
+| `node-test` | default | Built into Node. Parses TAP from `node --test`, including nested describe/it suites. |
+| `vitest` | test script or dependency mentions vitest | Uses the JSON reporter. |
+| `jest` | test script or dependency mentions jest | Uses `--json` with `--runTestsByPath`. |
+
+All three adapters are validated against real fixture repositories (base commit, then a PR adding one good and one vacuous test) as part of this project's development. For vitest and jest, the base worktree borrows the main checkout's `node_modules` via symlink, so run `npm ci` first in CI.
+
+## Why not X
+
+**Why not [tdd-guard](https://github.com/nizos/tdd-guard) or other live TDD enforcers?** Those hook into an agent's or developer's editing session and enforce red-green discipline while the code is being written. Great when you control the session. failfirst works at the other end: it gates the finished PR in CI, no matter what tool, agent, or human produced it, with nothing installed on the author's side.
+
+**Why not mutation testing (Stryker and friends)?** Mutation testing mutates your production code and checks that existing tests notice. It measures the strength of the whole suite, costs minutes to hours, and does not know what a specific PR changed. failfirst answers one narrow question per PR in seconds: do the new tests depend on the new code? The two are complementary; mutation testing for depth, failfirst for the PR loop.
+
+**Why not SWE-bench style fail-to-pass checks?** The fail-to-pass concept is exactly right, and SWE-bench uses it to build benchmark datasets with internal tooling tied to their harness and container images. failfirst is that concept packaged as a generic, zero-dependency CLI for any git repo with a supported runner.
+
+**Why not just review the tests?** Vacuity is invisible in review. The test imports the right module, asserts plausible things, and passes. The only way to know whether it would pass without the change is to run it without the change, which is tedious by hand and trivial for a tool.
+
+## Scope and limitations
+
+- Tests are matched between base and HEAD runs by file path plus full test name. A renamed test in a modified file is treated as new.
+- A pre-existing test whose body was edited (same name) is not re-gated. Gating it would re-flag every harmless refactor of an old test.
+- New tests in modified files that pass on base are flagged even if they sit next to legitimate ones; the table tells you exactly which test to fix.
+- Support files outside test directories (e.g. a changed helper in `src/`) are not overlaid onto the base worktree. If a new test needs them, it will fail or error on base, which is the safe direction: failfirst never produces a false VACUOUS from a missing dependency, only a conservative GOOD.
+- For vitest and jest, the base run uses the head checkout's `node_modules`. If your PR also changed dependency versions, the base run sees the new versions.
+- Flaky tests that pass on base by luck will be flagged. They deserve it.
+
+## Roadmap
+
+- `--require-tests`: fail PRs that change source code but add no tests.
+- Adapters: pytest, mocha, bun test.
+- GitHub Action wrapper with inline PR annotations on vacuous tests.
+- Optional gating of edited pre-existing tests.
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md). The short version: zero dependencies, pure logic stays pure and unit-tested, and every new behavior needs a test that fails without it.
+
+## License
+
+[MIT](LICENSE), Ben Malaga.

package/bin/failfirst.js

@@ -0,0 +1,10 @@
+#!/usr/bin/env node
+import { main } from '../src/cli.js';
+
+main(process.argv.slice(2)).then(
+  (code) => process.exit(code),
+  (err) => {
+    console.error(`failfirst: ${err && err.message ? err.message : err}`);
+    process.exit(2);
+  },
+);

package/package.json

@@ -0,0 +1,44 @@
+{
+  "name": "failfirst",
+  "version": "0.1.0",
+  "description": "CI gate that proves new tests actually test the change: a new test that passes on the base commit is vacuous.",
+  "type": "module",
+  "bin": {
+    "failfirst": "bin/failfirst.js"
+  },
+  "files": [
+    "bin",
+    "src",
+    "LICENSE",
+    "README.md"
+  ],
+  "engines": {
+    "node": ">=18"
+  },
+  "scripts": {
+    "test": "node --test"
+  },
+  "keywords": [
+    "testing",
+    "ci",
+    "git",
+    "tdd",
+    "fail-first",
+    "vacuous-tests",
+    "test-quality",
+    "code-review",
+    "ai",
+    "llm",
+    "cli"
+  ],
+  "author": "Ben Malaga",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/BenMalaga/failfirst.git"
+  },
+  "bugs": {
+    "url": "https://github.com/BenMalaga/failfirst/issues"
+  },
+  "homepage": "https://github.com/BenMalaga/failfirst#readme"
+}

package/src/cli.js

@@ -0,0 +1,197 @@
+// CLI entry: argument parsing and orchestration.
+import { copyFileSync, mkdirSync, readFileSync } from 'node:fs';
+import { dirname, join } from 'node:path';
+import { fileURLToPath } from 'node:url';
+import {
+  addWorktree,
+  diffNameStatus,
+  mergeBase,
+  removeWorktree,
+  repoRoot,
+  resolveBaseRef,
+} from './git.js';
+import { selectTestChanges } from './detect.js';
+import {
+  detectRunnerFromDir,
+  ensureNodeModules,
+  normalizeRunnerName,
+  runTests,
+} from './runners.js';
+import { computeVerdicts, preExistingKey, VERDICTS } from './verdict.js';
+import { formatSummary, formatTable, makePainter } from './report.js';
+
+const HELP = `failfirst: prove that new tests actually test the change
+
+Usage: failfirst [base-ref] [options]
+
+Runs the test files your branch added or modified against the merge-base
+(the old code). A new test that PASSES there is vacuous: it would pass
+without your change, so it is not testing it.
+
+Arguments:
+  base-ref          Base to compare against (default: origin/main, main,
+                    origin/master, or master, first one that exists)
+
+Options:
+  --runner <name>   node-test | vitest | jest (default: auto-detect)
+  --json            Machine-readable JSON output
+  --color           Force colored output (default: only when stdout is a TTY)
+  --no-color        Disable colored output
+  -h, --help        Show this help
+  -v, --version     Show version
+
+Exit codes:
+  0  no vacuous tests (or no changed test files)
+  1  at least one vacuous test
+  2  usage or environment error
+`;
+
+export function parseArgs(argv) {
+  const opts = { base: null, runner: null, json: false, color: null, help: false, version: false };
+  for (let i = 0; i < argv.length; i += 1) {
+    const a = argv[i];
+    if (a === '-h' || a === '--help') opts.help = true;
+    else if (a === '-v' || a === '--version') opts.version = true;
+    else if (a === '--json') opts.json = true;
+    else if (a === '--color') opts.color = true;
+    else if (a === '--no-color') opts.color = false;
+    else if (a === '--runner') {
+      i += 1;
+      if (i >= argv.length) throw new Error('--runner requires a value');
+      opts.runner = normalizeRunnerName(argv[i]);
+      if (!opts.runner) throw new Error(`unknown runner '${argv[i]}' (use node-test, vitest, or jest)`);
+    } else if (a.startsWith('--runner=')) {
+      const v = a.slice('--runner='.length);
+      opts.runner = normalizeRunnerName(v);
+      if (!opts.runner) throw new Error(`unknown runner '${v}' (use node-test, vitest, or jest)`);
+    } else if (a.startsWith('-')) {
+      throw new Error(`unknown option '${a}'`);
+    } else if (opts.base === null) {
+      opts.base = a;
+    } else {
+      throw new Error(`unexpected argument '${a}'`);
+    }
+  }
+  return opts;
+}
+
+function ownVersion() {
+  const pkgPath = join(dirname(fileURLToPath(import.meta.url)), '..', 'package.json');
+  return JSON.parse(readFileSync(pkgPath, 'utf8')).version;
+}
+
+function copyInto(files, fromRoot, toRoot) {
+  for (const f of files) {
+    const dest = join(toRoot, f);
+    mkdirSync(dirname(dest), { recursive: true });
+    copyFileSync(join(fromRoot, f), dest);
+  }
+}
+
+export async function main(argv) {
+  let opts;
+  try {
+    opts = parseArgs(argv);
+  } catch (err) {
+    console.error(`failfirst: ${err.message}`);
+    console.error(`Run 'failfirst --help' for usage.`);
+    return 2;
+  }
+  if (opts.help) {
+    process.stdout.write(HELP);
+    return 0;
+  }
+  if (opts.version) {
+    console.log(ownVersion());
+    return 0;
+  }
+
+  const useColor = opts.color ?? (Boolean(process.stdout.isTTY) && !process.env.NO_COLOR);
+  const paint = makePainter(useColor);
+  const log = (s = '') => {
+    if (!opts.json) console.log(s);
+  };
+
+  const root = repoRoot(process.cwd());
+  const baseRef = resolveBaseRef(opts.base, root);
+  const mb = mergeBase(baseRef, root);
+  const changes = selectTestChanges(diffNameStatus(mb, root));
+  const runner = opts.runner || detectRunnerFromDir(root);
+
+  log(`failfirst v${ownVersion()}`);
+  log(`  base     ${baseRef} (merge-base ${mb.slice(0, 7)})`);
+  log(`  runner   ${runner}`);
+  log(`  changed  ${changes.run.length} test file(s)`);
+  log();
+
+  if (changes.run.length === 0) {
+    if (opts.json) {
+      console.log(JSON.stringify({ base: baseRef, mergeBase: mb, runner, files: [], tests: [], summary: {}, vacuous: 0 }, null, 2));
+    } else {
+      log('No added or modified test files in this diff. Nothing to check.');
+    }
+    return 0;
+  }
+
+  const support = changes.copy.filter((f) => !changes.run.includes(f));
+  if (support.length > 0) {
+    log(`  copying ${support.length} changed support file(s) from test dirs: ${support.join(', ')}`);
+  }
+
+  const worktree = addWorktree(mb, root);
+  let headResults;
+  let baseResults;
+  const preExisting = new Set();
+  try {
+    if (runner !== 'node-test') ensureNodeModules(worktree, root);
+
+    // 1. For modified test files, enumerate the tests that already exist on
+    //    base, so we only gate the tests this branch actually added.
+    if (changes.modified.length > 0) {
+      for (const r of runTests(runner, worktree, changes.modified)) {
+        preExisting.add(preExistingKey(r.file, r.name));
+      }
+    }
+
+    // 2. Overlay the head versions of the changed test files onto old code.
+    copyInto(changes.copy, root, worktree);
+
+    // 3. Run them against the base, then against HEAD.
+    log(`  running ${changes.run.length} file(s) against base ${mb.slice(0, 7)}...`);
+    baseResults = runTests(runner, worktree, changes.run);
+    log(`  running ${changes.run.length} file(s) against HEAD...`);
+    headResults = runTests(runner, root, changes.run);
+  } finally {
+    removeWorktree(worktree, root);
+  }
+
+  const { rows, summary, vacuousCount } = computeVerdicts(headResults, baseResults, preExisting);
+
+  if (opts.json) {
+    console.log(JSON.stringify(
+      { base: baseRef, mergeBase: mb, runner, files: changes.run, tests: rows, summary, vacuous: vacuousCount },
+      null,
+      2,
+    ));
+    return vacuousCount > 0 ? 1 : 0;
+  }
+
+  log();
+  log(formatTable(rows, useColor));
+  log();
+  log(`  ${formatSummary(summary, useColor)}`);
+  log();
+
+  if (vacuousCount > 0) {
+    const noun = vacuousCount === 1 ? 'test passes' : 'tests pass';
+    console.log(paint('red', `FAIL: ${vacuousCount} new ${noun} on the base commit.`));
+    console.log('A test that passes without your change is not testing your change.');
+    return 1;
+  }
+  const checked = rows.filter((r) => r.verdict !== VERDICTS.PRE_EXISTING && r.verdict !== VERDICTS.SKIPPED).length;
+  const msg = checked === 1
+    ? 'PASS: the 1 new test fails on the base commit.'
+    : `PASS: all ${checked} new tests fail on the base commit.`;
+  console.log(paint('green', msg));
+  return 0;
+}

package/src/detect.js

@@ -0,0 +1,78 @@
+// Pure logic: classify changed files as test files, parse `git diff --name-status`.
+
+const EXT = String.raw`\.[cm]?[jt]sx?$`;
+
+// File NAME patterns that mark a file as a runnable test file.
+const RUNNABLE_RES = [
+  new RegExp(String.raw`[._-](test|spec)${EXT}`, 'i'), // foo.test.js, foo_test.ts, foo-spec.mjs
+  new RegExp(String.raw`(^|/)(test|spec)[._-][^/]*${EXT}`, 'i'), // test-foo.js, spec_bar.ts
+  new RegExp(String.raw`(^|/)(test|spec)${EXT}`, 'i'), // test.js, spec.ts
+];
+
+const TEST_DIR_RE = /(^|\/)(__tests__|tests?|specs?)\//i;
+const CODE_EXT_RE = new RegExp(EXT, 'i');
+
+/**
+ * A file we should RUN as a test (its name says "I am a test").
+ */
+export function isRunnableTestFile(path) {
+  const p = path.replace(/\\/g, '/');
+  return RUNNABLE_RES.some((re) => re.test(p));
+}
+
+/**
+ * A file that belongs to the test surface (runnable tests plus support files
+ * living in test directories, e.g. helpers/fixtures that new tests import).
+ */
+export function isTestFile(path) {
+  const p = path.replace(/\\/g, '/');
+  if (isRunnableTestFile(p)) return true;
+  return TEST_DIR_RE.test(p) && CODE_EXT_RE.test(p);
+}
+
+/**
+ * Parse `git diff --name-status -M` output.
+ * Returns [{ status: 'A'|'M'|'D'|'R'|'C'|'T', path, oldPath? }]
+ * For renames/copies, `path` is the NEW path.
+ */
+export function parseNameStatus(text) {
+  const entries = [];
+  for (const line of text.split('\n')) {
+    if (!line.trim()) continue;
+    const parts = line.split('\t');
+    const status = parts[0].trim();
+    const kind = status[0];
+    if (kind === 'R' || kind === 'C') {
+      if (parts.length >= 3) entries.push({ status: kind, path: parts[2], oldPath: parts[1] });
+    } else if (parts.length >= 2) {
+      entries.push({ status: kind, path: parts[1] });
+    }
+  }
+  return entries;
+}
+
+/**
+ * From a name-status diff, pick the test files added or changed by the PR.
+ * Returns { copy, run, added, modified }
+ *  - copy:     test-surface files to overlay onto the base worktree
+ *  - run:      subset of copy that we execute as tests
+ *  - added:    runnable files that did not exist on base
+ *  - modified: runnable files that existed on base (we enumerate their old tests)
+ */
+export function selectTestChanges(diffText) {
+  const copy = [];
+  const run = [];
+  const added = [];
+  const modified = [];
+  for (const e of parseNameStatus(diffText)) {
+    if (e.status === 'D') continue;
+    if (!isTestFile(e.path)) continue;
+    copy.push(e.path);
+    if (!isRunnableTestFile(e.path)) continue;
+    run.push(e.path);
+    // A=new file, R/C=new path on base. M/T existed on base.
+    if (e.status === 'M' || e.status === 'T') modified.push(e.path);
+    else added.push(e.path);
+  }
+  return { copy, run, added, modified };
+}

package/src/git.js

@@ -0,0 +1,84 @@
+// Thin wrappers around the git CLI. No dependencies, no shell.
+import { spawnSync } from 'node:child_process';
+import { mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+
+export class GitError extends Error {}
+
+export function git(args, cwd) {
+  const res = spawnSync('git', args, {
+    cwd,
+    encoding: 'utf8',
+    maxBuffer: 64 * 1024 * 1024,
+  });
+  if (res.error) throw new GitError(`git ${args[0]}: ${res.error.message}`);
+  if (res.status !== 0) {
+    const detail = (res.stderr || res.stdout || '').trim().split('\n')[0];
+    throw new GitError(`git ${args.join(' ')} failed: ${detail}`);
+  }
+  return res.stdout;
+}
+
+export function repoRoot(cwd) {
+  try {
+    return git(['rev-parse', '--show-toplevel'], cwd).trim();
+  } catch {
+    throw new GitError('not inside a git repository');
+  }
+}
+
+function refExists(ref, cwd) {
+  const res = spawnSync('git', ['rev-parse', '--verify', '--quiet', `${ref}^{commit}`], {
+    cwd,
+    encoding: 'utf8',
+  });
+  return res.status === 0;
+}
+
+const DEFAULT_BASES = ['origin/main', 'main', 'origin/master', 'master'];
+
+export function resolveBaseRef(given, cwd) {
+  if (given) {
+    if (refExists(given, cwd)) return given;
+    throw new GitError(`base ref '${given}' not found (try fetching it first)`);
+  }
+  for (const ref of DEFAULT_BASES) {
+    if (refExists(ref, cwd)) return ref;
+  }
+  throw new GitError(`no default base ref found (tried ${DEFAULT_BASES.join(', ')}); pass one explicitly`);
+}
+
+export function mergeBase(baseRef, cwd) {
+  try {
+    return git(['merge-base', baseRef, 'HEAD'], cwd).trim();
+  } catch {
+    throw new GitError(
+      `cannot find merge-base of '${baseRef}' and HEAD (shallow clone? use fetch-depth: 0 in CI)`,
+    );
+  }
+}
+
+export function diffNameStatus(mergeBaseSha, cwd) {
+  return git(['diff', '--name-status', '-M', `${mergeBaseSha}..HEAD`], cwd);
+}
+
+export function addWorktree(sha, cwd) {
+  const dir = mkdtempSync(join(tmpdir(), 'failfirst-'));
+  git(['worktree', 'add', '--detach', '--quiet', dir, sha], cwd);
+  return dir;
+}
+
+export function removeWorktree(dir, cwd) {
+  try {
+    git(['worktree', 'remove', '--force', dir], cwd);
+  } catch {
+    // fall through to manual cleanup
+  }
+  rmSync(dir, { recursive: true, force: true });
+  try {
+    git(['worktree', 'prune'], cwd);
+  } catch {
+    // best effort
+  }
+}

package/src/report.js

@@ -0,0 +1,63 @@
+// Render the verdict table and summary.
+import { VERDICTS } from './verdict.js';
+
+const CODES = { red: 31, green: 32, yellow: 33, dim: 2, bold: 1 };
+
+export function makePainter(useColor) {
+  if (!useColor) return (_style, s) => s;
+  return (style, s) => `\u001b[${CODES[style]}m${s}\u001b[0m`;
+}
+
+const VERDICT_STYLE = {
+  [VERDICTS.VACUOUS]: 'red',
+  [VERDICTS.GOOD]: 'green',
+  [VERDICTS.BROKEN]: 'yellow',
+  [VERDICTS.PRE_EXISTING]: 'dim',
+  [VERDICTS.SKIPPED]: 'dim',
+};
+
+export function formatTable(rows, useColor) {
+  const paint = makePainter(useColor);
+  const header = ['TEST', 'BASE', 'HEAD', 'VERDICT'];
+  const cells = rows.map((r) => [
+    `${r.file} > ${r.name}`,
+    r.base,
+    r.head,
+    r.verdict,
+  ]);
+  const widths = header.map((h, i) =>
+    Math.max(h.length, ...cells.map((c) => c[i].length)),
+  );
+  const lines = [];
+  lines.push(
+    '  ' + header.map((h, i) => paint('bold', h.padEnd(widths[i]))).join('   '),
+  );
+  rows.forEach((r, idx) => {
+    const c = cells[idx];
+    const style = VERDICT_STYLE[r.verdict] || 'dim';
+    const baseCell = r.base === 'pass' && r.verdict === VERDICTS.VACUOUS
+      ? paint('red', c[1].padEnd(widths[1]))
+      : c[1].padEnd(widths[1]);
+    lines.push(
+      '  ' +
+        [
+          c[0].padEnd(widths[0]),
+          baseCell,
+          c[2].padEnd(widths[2]),
+          paint(style, c[3].padEnd(widths[3])),
+        ].join('   '),
+    );
+  });
+  return lines.join('\n');
+}
+
+export function formatSummary(summary, useColor) {
+  const paint = makePainter(useColor);
+  const parts = [];
+  if (summary.good) parts.push(paint('green', `${summary.good} good`));
+  if (summary.vacuous) parts.push(paint('red', `${summary.vacuous} vacuous`));
+  if (summary.broken) parts.push(paint('yellow', `${summary.broken} broken`));
+  if (summary.preExisting) parts.push(paint('dim', `${summary.preExisting} pre-existing`));
+  if (summary.skipped) parts.push(paint('dim', `${summary.skipped} skipped`));
+  return parts.join(', ');
+}

package/src/runners.js

@@ -0,0 +1,178 @@
+// Test runner adapters: node:test, vitest, jest.
+// Each adapter runs a set of test files in a directory and returns
+// normalized results: [{ file, name, status: 'pass'|'fail'|'skip' }].
+import { spawnSync } from 'node:child_process';
+import { existsSync, mkdtempSync, readFileSync, realpathSync, rmSync, symlinkSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { isAbsolute, join, relative } from 'node:path';
+import { parseTap } from './tap.js';
+import { FILE_LOAD_ERROR } from './verdict.js';
+
+export const RUNNERS = ['node-test', 'vitest', 'jest'];
+
+export function normalizeRunnerName(name) {
+  const n = String(name).toLowerCase();
+  if (n === 'node' || n === 'node:test' || n === 'node-test' || n === 'nodetest') return 'node-test';
+  if (n === 'vitest') return 'vitest';
+  if (n === 'jest') return 'jest';
+  return null;
+}
+
+/**
+ * Pure: pick a runner from a parsed package.json ({} if none).
+ */
+export function detectRunner(pkg) {
+  const deps = { ...(pkg.dependencies || {}), ...(pkg.devDependencies || {}) };
+  const script = ((pkg.scripts || {}).test || '').toLowerCase();
+  if (script.includes('vitest') || 'vitest' in deps) return 'vitest';
+  if (script.includes('jest') || 'jest' in deps) return 'jest';
+  return 'node-test';
+}
+
+export function detectRunnerFromDir(dir) {
+  try {
+    return detectRunner(JSON.parse(readFileSync(join(dir, 'package.json'), 'utf8')));
+  } catch {
+    return 'node-test';
+  }
+}
+
+/**
+ * vitest/jest need node_modules; a fresh worktree has none.
+ * Borrow the main checkout's node_modules via symlink.
+ */
+export function ensureNodeModules(worktreeDir, sourceRoot) {
+  const target = join(worktreeDir, 'node_modules');
+  const source = join(sourceRoot, 'node_modules');
+  if (!existsSync(target) && existsSync(source)) {
+    symlinkSync(source, target, 'junction');
+  }
+}
+
+// If failfirst itself runs under `node --test` (e.g. in an integration test),
+// children inherit NODE_TEST_CONTEXT and nested runners refuse to run files.
+function cleanEnv() {
+  const env = { ...process.env };
+  delete env.NODE_TEST_CONTEXT;
+  return env;
+}
+
+function runNodeTest(cwd, files) {
+  const results = [];
+  for (const file of files) {
+    const res = spawnSync(
+      process.execPath,
+      ['--test', '--test-reporter=tap', file],
+      { cwd, encoding: 'utf8', maxBuffer: 64 * 1024 * 1024, env: cleanEnv() },
+    );
+    if (res.error) throw new Error(`node --test failed to start: ${res.error.message}`);
+    const leaves = parseTap(res.stdout || '');
+    if (leaves.length === 0) {
+      // File produced no test points (load error before any test, or empty file).
+      results.push({
+        file,
+        name: FILE_LOAD_ERROR,
+        status: res.status === 0 ? 'skip' : 'fail',
+      });
+      continue;
+    }
+    // When a file crashes before running any test (e.g. it imports something
+    // that does not exist on this commit), node --test emits a single failing
+    // test point named after the file itself. Normalize that.
+    if (leaves.length === 1 && !leaves[0].pass && leaves[0].name === file) {
+      results.push({ file, name: FILE_LOAD_ERROR, status: 'fail' });
+      continue;
+    }
+    for (const t of leaves) {
+      results.push({
+        file,
+        name: t.name,
+        status: t.skip || t.todo ? 'skip' : t.pass ? 'pass' : 'fail',
+      });
+    }
+  }
+  return results;
+}
+
+function findBin(cwd, name) {
+  const p = join(cwd, 'node_modules', '.bin', name);
+  return existsSync(p) ? p : null;
+}
+
+function runJsonReporter(runner, cwd, files) {
+  const bin = findBin(cwd, runner);
+  if (!bin) {
+    throw new Error(
+      `cannot find ${runner} in ${join(cwd, 'node_modules', '.bin')}; install dependencies first`,
+    );
+  }
+  const outDir = mkdtempSync(join(tmpdir(), 'failfirst-json-'));
+  const outFile = join(outDir, 'results.json');
+  const args =
+    runner === 'vitest'
+      ? [bin, 'run', '--reporter=json', `--outputFile=${outFile}`, ...files]
+      : [bin, '--runTestsByPath', '--json', `--outputFile=${outFile}`, ...files];
+  try {
+    const res = spawnSync(process.execPath, args, {
+      cwd,
+      encoding: 'utf8',
+      maxBuffer: 64 * 1024 * 1024,
+      env: cleanEnv(),
+    });
+    if (res.error) throw new Error(`${runner} failed to start: ${res.error.message}`);
+    if (!existsSync(outFile)) {
+      const detail = (res.stderr || res.stdout || '').trim().slice(0, 400);
+      throw new Error(`${runner} produced no JSON report. Output:\n${detail}`);
+    }
+    return parseJestJson(readFileSync(outFile, 'utf8'), cwd);
+  } finally {
+    rmSync(outDir, { recursive: true, force: true });
+  }
+}
+
+/**
+ * Pure: parse jest-style JSON (vitest's json reporter is jest-compatible).
+ */
+export function parseJestJson(text, cwd) {
+  const data = JSON.parse(text);
+  // Resolve symlinks (e.g. /tmp -> /private/tmp on macOS) so reported absolute
+  // paths and our cwd agree before computing relative paths.
+  let realCwd = cwd;
+  try {
+    realCwd = realpathSync(cwd);
+  } catch {
+    // keep cwd as-is
+  }
+  const results = [];
+  for (const tr of data.testResults || []) {
+    const abs = tr.name || tr.testFilePath || '';
+    const file = isAbsolute(abs) ? relative(realCwd, abs) : abs;
+    const assertions = tr.assertionResults || [];
+    if (assertions.length === 0) {
+      // Suite-level failure (e.g. the file could not load on this commit).
+      const failed = tr.status === 'failed' || Boolean(tr.message);
+      results.push({ file, name: FILE_LOAD_ERROR, status: failed ? 'fail' : 'skip' });
+      continue;
+    }
+    for (const a of assertions) {
+      const name =
+        a.ancestorTitles && a.ancestorTitles.length > 0
+          ? [...a.ancestorTitles, a.title].join(' > ')
+          : a.fullName || a.title;
+      const status =
+        a.status === 'passed' ? 'pass' : a.status === 'failed' ? 'fail' : 'skip';
+      results.push({ file, name, status });
+    }
+  }
+  return results;
+}
+
+/**
+ * Run `files` (paths relative to `cwd`) with the chosen runner.
+ */
+export function runTests(runner, cwd, files) {
+  if (files.length === 0) return [];
+  if (runner === 'node-test') return runNodeTest(cwd, files);
+  if (runner === 'vitest' || runner === 'jest') return runJsonReporter(runner, cwd, files);
+  throw new Error(`unknown runner '${runner}'`);
+}

package/src/tap.js

@@ -0,0 +1,88 @@
+// Pure logic: parse TAP output from `node --test` into leaf test results.
+//
+// node:test TAP shape (children precede their parent, 4-space indent per level):
+//   # Subtest: suite
+//       # Subtest: inner
+//       ok 1 - inner
+//       1..1
+//   ok 1 - suite
+// YAML diagnostic blocks sit between `---` and `...` lines.
+
+const RESULT_RE = /^(\s*)(not )?ok\s+\d+(?:\s+-\s+(.*?))?\s*$/;
+const DIRECTIVE_RE = /\s+#\s+(SKIP|TODO)\b.*$/i;
+
+/**
+ * Parse TAP text into leaf results: [{ name, pass, skip, todo }]
+ * Suite/parent entries are folded into their children's names ("suite > inner")
+ * and excluded from the result list.
+ */
+export function parseTap(text) {
+  const all = [];
+  const byDepth = new Map(); // depth -> entries awaiting a parent
+  let inYaml = false;
+  let yamlIndent = 0;
+
+  for (const line of text.split('\n')) {
+    const trimmed = line.trim();
+    const indent = line.length - line.trimStart().length;
+
+    if (inYaml) {
+      if (trimmed === '...' && indent === yamlIndent) inYaml = false;
+      continue;
+    }
+    if (trimmed === '---') {
+      inYaml = true;
+      yamlIndent = indent;
+      continue;
+    }
+    if (trimmed.startsWith('#')) continue;
+
+    const m = RESULT_RE.exec(line);
+    if (!m) continue;
+
+    let name = m[3] ?? '';
+    let skip = false;
+    let todo = false;
+    const d = DIRECTIVE_RE.exec(name);
+    if (d) {
+      skip = /skip/i.test(d[1]);
+      todo = /todo/i.test(d[1]);
+      name = name.slice(0, d.index);
+    }
+
+    const depth = Math.floor((m[1] ? m[1].length : 0) / 4);
+    const entry = {
+      name: name.trim(),
+      fullName: name.trim(),
+      pass: !m[2],
+      skip,
+      todo,
+      depth,
+      isParent: false,
+      subtree: [],
+    };
+    entry.subtree.push(entry);
+
+    // Adopt any pending entries one level deeper: they are this entry's children.
+    const children = byDepth.get(depth + 1) || [];
+    byDepth.set(depth + 1, []);
+    if (children.length > 0) {
+      entry.isParent = true;
+      for (const child of children) {
+        for (const node of child.subtree) {
+          node.fullName = `${entry.name} > ${node.fullName}`;
+        }
+        entry.subtree.push(...child.subtree);
+      }
+    }
+
+    const siblings = byDepth.get(depth) || [];
+    siblings.push(entry);
+    byDepth.set(depth, siblings);
+    all.push(entry);
+  }
+
+  return all
+    .filter((e) => !e.isParent)
+    .map(({ fullName, pass, skip, todo }) => ({ name: fullName, pass, skip, todo }));
+}