@t09tanaka/test-gates 0.3.4 → 0.4.1

package/CHANGELOG.md

@@ -1,5 +1,38 @@
 # Changelog
 
+## 0.4.1
+
+### Fixed
+
+- `test-gates mutation --affected` reuses every result after no change also where it cannot tell what a gate depends on (a file a gate or a spec loads reads files at run time, for one). Before, such a subproject ran every mutant on every run with the option, while a run without it reused the results after no change. The first line says so; a change still runs every mutant.
+- `test-gates doctor` reports what keeps `--affected` from telling the gates apart: as a problem when a hook relies on `--affected`, else as a note. In such a subproject it no longer asks for `--affected` in a hook. Before, it said OK although a run with `--affected` was always one of every mutant.
+
+## 0.4.0
+
+Most of this release is about runs that were full ones although little had changed. `test-gates doctor` tells why a project's runs are full ones. `mutation --affected` runs again only the gates a change can affect. The comparison before a run covers only what the run loads. Two changes of behaviour: Stryker's `concurrency` now defaults to half of the cores, and the record of a run has a new format, so the first run after the update is one of every mutant.
+
+### Added
+
+- `test-gates doctor` tells why `test-gates mutation` would run every mutant where it need not, and what to change: Stryker not in incremental mode, a path that turns reuse off, the file and the path that make every file of the repository part of the comparison, and a git hook that runs mutation without `--affected`. It also shows how many files are compared, whether the next run reuses the last one, and what a run with `--affected` would run again. It runs nothing and changes nothing; the exit code is 1 when there is a problem.
+- `test-gates mutation --affected` runs again only the gates that a change since the last run can affect, and keeps the results of the others. The unit is the gate: every mutant of a gate that may have changed is run again, and Stryker's judgement per mutant is not relied on. A gate depends on itself, its own spec, every spec that imports it or whose tests covered or killed one of its mutants, and what all of those import. A change of a config, a lock file, a tool or the arguments, a file whose imports cannot be followed or that reads files at run time, a run with `--force`, and a run in CI (`CI` set) run every mutant, and the first line says why; otherwise the first lines say how many gates were reused and which file made each of the others run again. A package of the same repository linked into `node_modules` also runs every mutant: its files are not followed. A component (`.vue`, `.svelte`, `.astro`, `.mdx`) that a gate or a spec loads runs every mutant as well: it can use others that it does not import. A gate with an unallowed survivor or timeout always runs again. A mutant that timed out in the last run is run again on its own (`--mutate` with its range), also in a reused gate, since a timeout on a loaded machine can stand for a survivor. It is an option: without it a change of any compared file still runs every mutant. See "`--affected`" in the README.
+
+### Changed
+
+- `createStrykerGatesConfig` sets `concurrency` when the options do not: half of the cores (`os.availableParallelism()`), at least 1 and at most 8, and 2 when `CI` is set. Stryker's own default was one fewer than the cores. A `concurrency` in the options wins, and `--concurrency` on Stryker's command line wins over both. See "How many test runners Stryker starts" in the README.
+- `test-gates selfcheck --jobs <n>` decides for each control whether it runs next to others. Before, one file anywhere in the gate run that reached outside of its process (the clock, the environment, a built-in module, a package not vouched for) made every control run one at a time. Now only the control of a gate is kept apart that a spec of another gate loads while that spec, or a file it loads, reaches outside, or that a spec may load because its imports cannot all be followed; a file the gate config names that reaches outside still keeps every control apart. The controls kept apart run one at a time after the others, and the first lines name every one of them with every reason, instead of the first reason only. A spec that reaches outside but loads only its own gate keeps nothing apart. The other checks (the gate command, the gate config, `failurePattern`, and the judgement of each run next to others) are unchanged, and the default stays 1.
+- `test-gates mutation` no longer takes over a timeout from the last run after no change: the mutants that timed out are run again on their own, the other results are reused as before.
+- In a git hook of a linked worktree (`GIT_DIR` set, `GIT_WORK_TREE` not), `test-gates mutation` lists the files from the top of the repository. Before, git took the subproject for it, a path to a sibling directory led "outside of the repository", and every run was a full one.
+- The record next to the incremental file is of a new format (2), with what a run with `--affected` writes down about each gate. The first run after the update is one of every mutant.
+- `test-gates mutation` compares fewer files outside the subproject before it reuses the results of the last run, so that a change elsewhere in a repository no longer turns every run into a full one:
+  - A path that leads up from a file and back into the subproject (`'../helper'` from `src/` to a file at the top of the subproject) no longer makes every file of the repository part of the comparison. It was taken for a path to a directory above the subproject.
+  - A path that starts at the directory of its own file is followed: `` `${__dirname}/../shared` ``, and `__dirname, '..', 'shared'` handed to `path.join` or `path.resolve` (also with `import.meta.dirname`). Before, every file of the repository was compared. When such a path ends in a directory above the subproject, only the files directly in it are compared, as they always are; when an unknown part follows it (`join(__dirname, '..', name)`, or something added to what the call returns), every file still is. A way up after a placeholder (`` `../x/${name}/../..` ``) now also makes every file part of the comparison; before, only the part before the placeholder was followed.
+  - Paths in `.md`, `.markdown`, `.txt`, `.rst` and `.adoc` files are no longer followed. The files themselves are still compared.
+  - A plain relative path to a file or a directory above the subproject (`'../tsconfig.base.json'`), a path whose start is not known, and `'..'` after anything but the directory of the file still make every file of the repository part of the comparison.
+- In a project where the comparison is now smaller, the first run after the update is one of every mutant: the record of the last run holds files that are no longer compared, and the message lists them as changed.
+- When a path leads outside of the repository and the results are therefore not reused, the message names the path as it is written, next to the file it is in.
+- With a runner config made by `createJestGatesConfig` or `createVitestGatesConfig`, `test-gates mutation` follows the paths only in the files of the subproject that the run loads: the configs and the files they name, the gates, their specs, the configs that the tools read by name (`tsconfig*.json`, Babel, SWC), mocks, and everything those import. The other files, `package.json` among them, are still compared, but a `'../'` in a lint config, a script in `package.json`, `$PROJECT_DIR$/..` in the settings of an editor or `<rootDir>/../shared` in the config of another test run no longer makes every file of the repository part of the comparison. An alias that `settings.importAliases` does not name loads the files of the subproject whose paths end as the import does. A loaded file that imports a module whose path is not written out, and a runner config made otherwise, keep the search in every file.
+- A path that leads outside of the repository to nothing (one meant for the sandbox of Stryker, such as `path.resolve(__dirname, '../../../shared')` in a runner config) is compared as missing, instead of turning reuse off. Once something is there, reuse is off as before.
+
 ## 0.3.4
 
 One change of behaviour: `test-gates mutation` no longer lets Stryker reuse old results after a change. The rest is new options of `selfcheck`; without them it runs as before.

package/README.md

@@ -105,10 +105,17 @@ export default createStrykerGatesConfig({
   testRunner: 'jest',
   jest: { projectType: 'custom', configFile: 'jest.gates.config.js' },
   // testRunner: 'vitest', vitest: { configFile: 'vitest.gates.config.ts' },
-  concurrency: 2,
 });
 ```
 
+**How many test runners Stryker starts (`concurrency`).** When neither the config nor the command line says, the helper sets half of the cores (`os.availableParallelism()`), at least 1 and at most 8, and 2 when `CI` is set. Stryker's own default is one fewer than the cores, but each of its runners is a Jest or Vitest that does work besides the tests (transforms, coverage), and the machine of a developer is rarely free for the run alone: half of the cores leaves room, and the cap of 8 keeps the memory of the runners in bounds on large machines. On a machine with 12 cores that is 6; going from 2 to 6 took about half the time for a full run on two projects, and on this repository's own gates (about 2,400 mutants, other jobs running) 143 and 184 s instead of 307 and 389 s, each time with the same numbers of killed, survived and timed-out mutants. In CI 2 is the number of cores of the usual hosted runner; more runners than cores only make mutants run out of time. What wins, from the first:
+
+1. `--concurrency <n>` after `--` on the command line (`npm run test:gates:mutation -- --concurrency 1`), which Stryker takes over its config;
+2. `concurrency` in the options of `createStrykerGatesConfig`;
+3. the default above.
+
+A config that sets `concurrency: 2` keeps 2 everywhere; remove the key to get the default. Setting it to `undefined` gives Stryker's own default instead.
+
 Add `reports/`, `.stryker-tmp/` and `.test-gates-touch-*` (see `test-gates selfcheck`) to `.gitignore`.
 
 ### 4. Scripts
@@ -157,7 +164,7 @@ done
 
 for project in backend frontend; do
   printf '%s\n' "$changed" | grep -q "^$project/" || continue
-  (cd "$project" && npm run --silent test:gates && npm run --silent test:gates:mutation) || exit 1
+  (cd "$project" && npm run --silent test:gates && npm run --silent test:gates:mutation -- --affected) || exit 1
 done
 ```
 
@@ -227,7 +234,7 @@ Static checks, before the gate run. Reports every violation as `file:line: reaso
 - The Stryker config does not use `excludedMutations`, `ignoreStatic` or `ignorers`.
 - A Vitest gate run below Vitest 4 does not use the plain v8 coverage provider (since 0.3.0), see [Vitest: v8 or istanbul](#vitest-v8-or-istanbul).
 
-### `test-gates mutation [--gate <path>…] [-- <stryker arguments>]`
+### `test-gates mutation [--gate <path>…] [--affected] [-- <stryker arguments>]`
 
 Runs `stryker run <config>` from the project's own `node_modules`, then judges the JSON report (see `mutation-result`). Unknown arguments, and everything after `--`, are handed to Stryker. If Stryker itself fails, the exit code is 1 and nothing is judged. A report from an earlier run is deleted first, so a run that writes none cannot pass on old results.
 
@@ -250,15 +257,19 @@ What is compared:
 
 - Every file of the subproject: gates, specs, other sources, helpers of the specs, configs, `test-gates.json`, the lock file.
 - The files directly in the directories above the subproject, up to the root of the repository: the lock file and the shared configs of a monorepo.
-- Packages of the same repository that are linked into `node_modules`, and directories outside the subproject that a file leads to, with a link or with a relative path (`../../pricing-kit/rounding/half-up`). When a path cannot be followed (`<rootDir>/../pricing-kit`, a way up handed to `path.join` as separate `'..'` arguments, a path to a file or a directory above the subproject), every file of the repository is compared.
+- Packages of the same repository that are linked into `node_modules`, and directories outside the subproject that a file leads to, with a link or with a relative path (`../../pricing-kit/rounding/half-up`). A path that starts at the directory of its own file is followed as well, whether it is written as a template (`` `${__dirname}/../pricing-kit` ``) or as parts handed to `path.join` or `path.resolve` (`__dirname, '..', 'pricing-kit'`, also with `import.meta.dirname`). A path that leads back into the subproject adds nothing.
+- When a path cannot be followed, every file of the repository is compared: when its start is not known (`<rootDir>/../pricing-kit`, an alias such as `@/../shared`, `'..'` after anything but the directory of the file), when a part after it is not known (`join(__dirname, '..', name)`, `join(resolve(__dirname, '..'), name)`, a way up after a placeholder), and when a plain relative path leads to a file or a directory above the subproject (`'../tsconfig.base.json'`).
+- Files written for people to read (`.md`, `.markdown`, `.txt`, `.rst`, `.adoc`) are compared, but the paths in them are not followed. Every other file is searched, whatever its extension.
+- When the config of the test runner is made with this package's helper (`createJestGatesConfig` or `createVitestGatesConfig`, which run the specs of the gates and no others), only the files of the subproject that the run loads are searched: the Stryker config, the runner config and the files it names (a setup file), the gates and their specs, `test-gates.json`, `tsconfig*.json`, `jsconfig*.json`, the Babel and SWC configs, the files below a root `__mocks__/` and the mock next to a loaded file, what those configs name (in a script, any path as text, such as a setup file; in `tsconfig.json` or `.babelrc`, what it extends and the plugins and presets it names, and what those name in turn), and everything those import, also what a component (`.vue`, `.svelte`, `.astro`, `.mdx`) imports. The other files of the subproject are compared, but a path in them is not followed: `package.json` (its scripts, and a runner config in it that the helper's config replaces), a lint config, the settings of an editor, the config of another test run, a component no gate loads. An import of an alias that `settings.importAliases` does not name (one that the runner config maps) loads every file of the subproject whose path ends as the import does (`@shared/money/fee` loads `src/money/fee.ts`); when none does, it leads out of the subproject, to where the config that maps it points. An import of an alias with the extension of the compiled file (`@shared/fee.js`) leads to `fee.ts` as well, and one of a single part (`@helper`, `#config`) to the files of that name. Every file is searched when a loaded file imports a module whose path is not written out, when the runner config is not made with the helper, when what the runner config exports is not what the helper returns (merged with another config after it, changed after the call), and when the runner config or a file it is put together from names `projects` or `workspace` outside of a comment: then it may run more than the specs of the gates. Another config spread into the helper's options is fine: the helper sets the specs after them.
+- A path that leads outside of the repository to nothing is compared as missing: a path from a config that is meant for the sandbox of Stryker (`path.resolve(__dirname, '../../../shared')`, which from the subproject itself leads above the repository). Once something is there, reuse is off, as for any path outside the repository.
 - What is installed (`node_modules/.package-lock.json`, or what yarn and pnpm write in its place), the versions of Node.js, of this package, of Stryker and of the test runners, and the arguments handed to Stryker.
 - The incremental file itself: it has to be the one the recorded run wrote.
 
-With git, "every file" is what git tracks plus the untracked files it does not ignore. Without git, or when git ignores the subproject, every file is read except `node_modules`, `.git` and what the run writes itself, up to the nearest directory that has a `.git` entry, else the topmost one that declares workspaces, else only the subproject. Such a listing also contains what git would ignore (coverage output, an HTML report, logs), so a file that changes between two runs makes every run a full one; reuse is meant for a checkout with git.
+With git, "every file" is what git tracks plus the untracked files it does not ignore. In a git hook of a linked worktree, git sets `GIT_DIR` and not `GIT_WORK_TREE`; then git would take the subproject for the top of the repository, so test-gates calls git without `GIT_DIR` in that case and lets it find the repository from the subproject. Without git, or when git ignores the subproject, every file is read except `node_modules`, `.git` and what the run writes itself, up to the nearest directory that has a `.git` entry, else the topmost one that declares workspaces, else only the subproject. Such a listing also contains what git would ignore (coverage output, an HTML report, logs), so a file that changes between two runs makes every run a full one; reuse is meant for a checkout with git.
 
 The record is removed before each run and written only after a run that Stryker ended without an error and in which it wrote the incremental file, so a run that failed, was interrupted or was a dry run is followed by a full one. A run that is given `--force` is not changed. When Stryker does not run in incremental mode, nothing is recorded or compared.
 
-The paths that leave the subproject are found by text, in every file that is compared. The search errs towards comparing too much: text that merely looks like such a path widens the comparison, and a path or a link to a directory that exists outside the repository turns reuse off, with the file named in the message.
+The paths that leave the subproject are found by text, in every file that is compared but those written for people and, with the helper's runner config, those of the subproject that the run does not load. The search errs towards comparing too much: text that merely looks like such a path widens the comparison, and a path or a link to a directory that exists outside the repository turns reuse off, with the file and the path as it is written named in the message. There is one exception, where the path is known: a path from the directory of the file that ends in a directory above the subproject (`path.resolve(__dirname, '..')` in a file at the top of the subproject) adds only the files directly in that directory, which are compared anyway (see the last point of the list below).
 
 The comparison loads the Stryker config in a process of its own, to see where the incremental file is, and reads and hashes those files before each run. Timed with a stand-in for Stryker, so that only this tool is measured, in a project of 5,200 files written for the measurement: 0.13 s without the comparison, 0.44 to 0.55 s with it (medians of three rounds; other jobs were running on the machine).
 
@@ -267,13 +278,18 @@ What it does not see, and what therefore still needs `-- --force` by hand, or CI
 - files git ignores (build output that a spec or a config loads, `.env`),
 - environment variables, the clock, the network, and a Stryker config that reads them or the command line to decide about incremental mode,
 - files outside the subproject that are reached without a link and without `..` in a path: an absolute path, an alias defined in a config above the subproject (in that case every file of the repository is compared only if the subproject points at that config with a relative path),
-- changes inside `node_modules` that leave the files of the package manager as they were.
+- changes inside `node_modules` that leave the files of the package manager as they were,
+- paths in files written for people (`.md`, `.txt` and the like): a script that reads a path out of a README,
+- with the helper's runner config, paths in a file of the subproject that the run does not load as a module: a file that a loaded one reads as data (`readFileSync`) and takes a path out of, and a config that a package reads by its name and that is not among those listed above (`next.config.js` read by `next/jest`, for example, or a Babel config kept in the `babel` key of `package.json`),
+- with the helper's runner config, what a component uses without importing it (a component that a plugin registers), and the paths in it,
+- a path outside the repository to nothing, to which something is added that starts with the same text and continues the last part of its name (`'../../../shared' + '-kit'`),
+- what is read below a directory above the subproject when the path to that directory is kept in a variable and the rest is added elsewhere: `const top = path.resolve(__dirname, '..')` in one place and `path.join(top, 'shared', name)` in another, or a path to such a directory handed to a tool that reads below it.
 
 #### `--gate <path>`: only the mutants of one gate
 
 While working on one gate, `test-gates mutation --gate src/pricing/shipping-fee.ts` runs the mutants of that gate only (it hands the path to Stryker as `--mutate`) and judges that gate only. The option can be given more than once; the path is written as in `test-gates.json`.
 
-**It is not the final check, and the last line says so.** A change in one gate, in its spec or in a file they import can make a mutant of **another** gate survive, and the mutants of that other gate are not run. `test/incremental.test.ts` has such a project, with a real Stryker: the spec of one gate compares its result through a function of another gate; that function is changed to tell less apart, together with its own spec. Every mutant of the changed gate is still killed. Running only those and taking over the old results of the other gate from Stryker's incremental file passes. The run of every gate finds the survivor in the other gate.
+**It is not the final check, and the last line says so.** A change in one gate, in its spec or in a file they import can make a mutant of **another** gate survive, and the mutants of that other gate are not run. `test/incremental-reuse.test.ts` has such a project, with a real Stryker: the spec of one gate compares its result through a function of another gate; that function is changed to tell less apart, together with its own spec. Every mutant of the changed gate is still killed. Running only those and taking over the old results of the other gate from Stryker's incremental file passes. The run of every gate finds the survivor in the other gate.
 
 So a run with `--gate`
 
@@ -285,6 +301,54 @@ So a run with `--gate`
 
 `--gate` is written before `--`, and cannot be combined with a `--mutate` of your own.
 
+#### `--affected`: run again only the gates a change can affect
+
+Without the option, a change to any file that is compared makes the run one of every mutant. That is the safe answer, but in a pre-push hook that only runs for the subprojects a push touches, it is the answer every time. `test-gates mutation --affected` (`npm run test:gates:mutation -- --affected`) decides gate by gate instead: a gate whose results cannot have changed keeps them, and every mutant of a gate that may have changed is run again. It never relies on Stryker's judgement per mutant, which is what reused the old "Killed" in the measurements above.
+
+The results of a gate depend on
+
+- the gate and every file it imports,
+- its own spec, every spec that imports it (directly or through other files), and every spec whose tests covered or killed one of its mutants in the last run, with every file those specs import. A spec that does not load the gate cannot see its mutants: Jest and Vitest give each spec file a module registry of its own. This is the way a change reaches another gate: when the spec of `next` checks its result through the gate `same`, a change of `same` runs `next` again, and so does a weakened spec of `next` for `same`,
+- snapshots next to a spec (`__snapshots__/<spec>.snap`) and data a spec or a gate imports.
+
+Imports are followed when they are relative, absolute, or start with a prefix of `settings.importAliases`; type imports count too. Paths written as text for the runner (`vi.mock('./rates')`) are followed as well. A package is taken to be what the lock file and the versions of the tools say it is, unless it is a link to a package of the same repository: see below.
+
+A change elsewhere decides for every gate:
+
+- **Every mutant is run** after a change of a config (the Stryker config and the config of the test runner, what they import, the setup files and other files they name, `package.json`, `test-gates.json`, `tsconfig*.json`, `*.config.*`, Babel configs, dotfiles, declaration files `.d.ts`, anything below `__mocks__`), of a lock file or of what is installed, of the tools or of the arguments; after a change of a file at the top of the subproject or outside of it that no gate depends on; and after a change of a file whose kind is not known (`.yaml`, `.html` and the like) that no gate depends on. Such a file that a gate does depend on runs that gate again, as any other file does.
+- **Nothing is run again** for a change of a source, data, a snapshot, a style or an image below the subproject that no gate depends on, or of a document (`.md`, `.txt` and the like).
+
+When what a gate depends on cannot be told from the text, a change runs every mutant, and the first line says why. After no change at all the results are reused as they are without the option, so `--affected` is never slower than a run without it. Gate by gate it cannot tell when it meets an import whose path is not written out (`import(name)`, `require(name)`), an import that leads to no file, a specifier that is neither a file nor an installed package (an alias defined only in a config: name it in `settings.importAliases`), a file read at run time (`node:fs`, `readFileSync`, `__dirname`, `import.meta.url`, `process.cwd`, `toMatchFileSnapshot`), a file that is loaded but not compared (git ignores it), a gate loaded for every spec by a setup file, a package of the same repository linked into `node_modules` (`@acme/shared`, also when it lies in the subproject), a component (`.vue`, `.svelte`, `.astro`, `.mdx`), which can use others that it does not import (registered by a plugin), and a test runner other than Jest and Vitest. Every mutant is also run when there is no record of a run with `--affected`, when the record cannot be read, with `--force`, and **when `CI` is set**: CI keeps running every mutant, and is the final check.
+
+A gate with a survivor or a timeout that the allow lists do not cover is not reused: it runs again until it is clean. A mutant that timed out in the last run is never taken over either, also not in a gate whose other results are, and also not in a run without `--affected` after no change: on a loaded machine a mutant that would survive can time out instead, and a reused timeout would keep that. Those mutants alone are run again (`--mutate` with their ranges); the first lines count them. The run is judged as one of every gate, and its first lines say what was reused:
+
+```
+test-gates mutation --affected: 1 of 3 gate(s) reused, 2 run again
+  src/same.ts: changed src/same.ts
+  src/next.ts: changed src/same.ts
+```
+
+To run only some gates, `--affected` hands them to Stryker as `--mutate` with `--force`, and Stryker takes the results of the others over from its incremental file, which has to be the one the recorded run wrote. `--affected` cannot be combined with `--gate` or with a `--mutate` of your own, and is written before `--`. `test/affected-reuse.test.ts` runs each of these cases with a real Stryker.
+
+What it does not see, besides what is listed for the comparison above: a file that a spec or a gate reaches in a way the text does not show and that is not a config by its name (a script that a tool loads by a name built at run time), and a spec that changes what another spec sees (Vitest with `isolate: false`, a shared global set by one spec file and read by another). Keep the CI run without the option.
+
+### `test-gates doctor`
+
+Tells why `test-gates mutation` would run every mutant where it need not, and what to change. It runs neither Stryker nor a test and changes no file: it takes the same snapshot as a mutation run, reads the record of the last run, and looks at the git hooks of the repository (`.husky/pre-push` and `pre-commit`, the hooks git runs, `lefthook.yml`).
+
+```
+test-gates doctor: /work/shop/fee-rules (stryker.gates.config.mjs)
+  compared: 640 files, 16 outside the subproject, paths followed in the 31 files the run loads
+  next run: every mutant, changed since the last run: src/fee.ts
+  next run with --affected: 3 of 4 gate(s) reused, 1 run again
+    src/fee.ts: changed src/fee.ts
+problem: .husky/pre-push runs test-gates mutation without --affected: a change of any file of the subproject runs every mutant
+  fix: pass --affected in the hook (npm run test:gates:mutation -- --affected); keep the run in CI without it
+test-gates doctor: 1 problem(s) that make runs full ones
+```
+
+A **problem** makes runs full ones that need not be: Stryker not in incremental mode, a path that turns reuse off (one that leads outside of the repository to something, or a Stryker config that cannot be loaded), a path that makes every file of the repository part of the comparison (it names the file and the path), a Stryker config that sets `force`, a hook that runs `test-gates mutation` of the subproject without `--affected`, also through its `test:gates:mutation` script, and, when a hook relies on `--affected`, what keeps it from telling the gates apart (a file a gate or a spec loads that reads files at run time, for one). Where it cannot tell them apart, a hook without `--affected` is not taken for a problem, and what keeps it from doing so is a note. A run in a hook is taken to be one of the subproject that its line names (`--dir`, `--prefix`, `-w`, `cd`), else the nearest `cd` before it; a name with a variable in it (`cd "$project"` in a loop over the subprojects), or none, may be one of any subproject. Each run is judged on its own line, and comments are left out. A **note** is worth knowing: that every file of the subproject is searched for paths, and why (a runner config that is not made with the helper, an import whose path is not written out), and that no hook runs a mutation run. The exit code is 1 when there is a problem, so that the doctor can run in CI next to the other checks.
+
 ### `test-gates mutation-result [--report <path>]`
 
 Judges an existing Stryker JSON report without running Stryker.
@@ -378,7 +442,7 @@ So a control that `--quick` reports as holding is not the last word; one that it
 
 The notes are taken from what the runner itself counts for a spec file, after the file has run to its end: with Jest, the coverage a reporter is given for the file; with Vitest, what the worker hands to the istanbul coverage provider. All of it rests on the notes being complete: an execution that is missing from the note of a spec could make a gate look untouched. So the notes are used only within three limits, and outside of them every gate gets a run of its own, as before, with the reason on the first line (`test-gates selfcheck: a run of the gate command for every gate (…)`).
 
-1. **Only for the runs it is verified with.** This table is the list; it is also `VERIFIED_RUNS` in the code, and `test/selfcheck-runners.test.ts` compares the shared run with a run per gate for every row.
+1. **Only for the runs it is verified with.** This table is the list; it is also `VERIFIED_RUNS` in the code, and `test/selfcheck-runners-*.test.ts` compares the shared run with a run per gate for every row.
 
    | Runner | Versions | Coverage                       | Pool                         | Other                                  |
    | ------ | -------- | ------------------------------ | ---------------------------- | -------------------------------------- |
@@ -399,26 +463,39 @@ A gate also keeps a run of its own when another spec touches it (it imports the
 
 When the gate command is `jest` or `vitest` and the version installed is outside of the table, or the gate command is the one derived from the gate config and that config does not name the helper (or, for Vitest, `istanbul`), this is known beforehand and the run for the notes is not started. Otherwise that run was one run for nothing, also in every part of `--shard`.
 
-With Jest and Vitest, and specs that do not depend on each other, the lines printed and the exit code are those of the run per gate. (A gate command of your own whose exit code says what failed would show the code of the shared run in the `✓` lines.) The two are compared with a real Jest and a real Vitest in `test/selfcheck-runners.test.ts`, for a gate that the spec of another gate covers, one that two other specs cover between them, one that another spec covers in part, one whose top-level code runs when another spec loads it, one that a setup file calls, one that another spec replaces with a mock, one that another spec calls in `afterAll` (with Vitest's hooks in the order written), and one that another spec calls in a cleanup function returned by `beforeAll`.
+With Jest and Vitest, and specs that do not depend on each other, the lines printed and the exit code are those of the run per gate. (A gate command of your own whose exit code says what failed would show the code of the shared run in the `✓` lines.) The two are compared with a real Jest and a real Vitest in `test/selfcheck-runners-*.test.ts`, for a gate that the spec of another gate covers, one that two other specs cover between them, one that another spec covers in part, one whose top-level code runs when another spec loads it, one that a setup file calls, one that another spec replaces with a mock, one that another spec calls in `afterAll` (with Vitest's hooks in the order written), and one that another spec calls in a cleanup function returned by `beforeAll`.
 
 During the first run a directory `.test-gates-touch-…` exists in the subproject (Vitest loads the wrapped provider from there). It is removed when the run ends. A run that is interrupted leaves it behind; a later `selfcheck` removes what is older than an hour. Add `.test-gates-touch-*` to `.gitignore`.
 
 **`--jobs <n>`: `<n>` negative controls at the same time.** The default is 1, one at a time. With `--jobs 4`, four runs of the gate command are under way at any moment, each for another gate. A control stays what it is: a run of the gate command with the spec of one gate left out, and of that one only. What `--jobs` adds is other runs on the same machine, and a run can be disturbed by them. Two things are therefore asked, one before the runs start and one of each run.
 
-_Before the runs start_, the files a gate run loads must not be seen to reach outside of their process. A spec whose helper takes a lock on a file, listens on a fixed port or looks at the clock can quietly do less while another run is there, and then a gate that this spec covers looks as if only its own spec covered it. Nothing in the output of such a run tells. The files are: the spec of every gate, every script the gate config names (a setup file, a reporter; also in the project files the config imports), and all the project files those import, the gates among them. When one of them
+_Before the runs start_, the gate command and its config must be known, and for each control the files of its run that can change what it reports must not be seen to reach outside of their process. A spec whose helper takes a lock on a file, listens on a fixed port or looks at the clock can quietly do less while another run is there, and then a gate that this spec covers looks as if only its own spec covered it. Nothing in the output of such a run tells. A spec can only cover a gate it loads, and the control of a gate leaves out the spec of that gate. So the control of a gate is started next to others unless, in its run,
+
+- a spec of another gate loads the gate (directly or through other files), and that spec or a file it loads reaches outside;
+- a spec of another gate may load any gate: one of its imports cannot be followed, or one of its files loads a module whose name is computed;
+- a script the gate config names (a setup file, a reporter; also in the project files the config imports) reaches outside or cannot be followed: those run with every spec, so this keeps every control on its own.
+
+A spec that reaches outside but loads no other gate, which is what the spec of a gate that needs a clock, a file or a package usually is, keeps no control from running next to others; it is left out of the run of its own gate. A file reaches outside when it
 
 - mentions `process.`, `globalThis`, `global.` or `import.meta.env`; the clock or a wait (`Date.now`, `new Date()`, `performance.now`, `setTimeout`, `setInterval`, `setImmediate`, `waitFor`, `waitUntil`, `expect.poll`, `retryTimes`, `retry:`); the network (`fetch(`, `WebSocket`, `XMLHttpRequest`); or `navigator`, `Atomics`, `WeakRef`, `FinalizationRegistry`, `eval(`, `Function(`,
 - imports a built-in module of Node.js, also one that only computes (`path`): with Jest the spec files of one worker share the objects of the built-in modules, so a setting that one spec changes there is seen by the next, and which spec comes next depends on Jest's cache of run times,
 - imports a package other than the runner's (`vitest`, `@jest/globals`). Packages are not looked into. With `settings.imports.mode: "allowlist"`, the packages that `settings.imports.allow` lists by name are taken too: the project already vouches for them as imports of a gate,
-- loads a module whose name is computed, or imports something that is neither a file nor an installed package (an alias that `settings.importAliases` does not have),
+- loads a module whose name is computed.
 
-every control runs on its own, and the first line says why:
+The controls kept apart run one at a time after the others. The first lines name every one of them: a line for each spec (or the gate config) that keeps controls apart, with the gates whose controls it keeps apart and every file of it that reaches outside:
 
 ```
-test-gates selfcheck: one negative control at a time (src/support/lock.ts imports node:fs, and may meet what another run is doing)
+test-gates selfcheck: 2 of 30 negative control(s) run one at a time, since in their runs something that loads the gate also loads a file that may meet what another run is doing:
+  src/tax.ts, src/age.ts: loaded by src/fee.spec.ts, in which src/support/lock.ts imports node:fs
 ```
 
-The same happens when the gate command is anything but `jest` or `vitest run` with `--config` and options for the number of workers (`--runInBand`, `-i`, `--ci`, `--maxWorkers`, `-w` for Jest; `--no-file-parallelism`, `--fileParallelism=false`, `--maxWorkers`, `--minWorkers`, `--pool=forks`, `--pool=threads` for Vitest), when the gate config does not use the helper, and with `settings.selfcheck.failurePattern`. This is a search by text: a comment that names `setTimeout` counts, and `new Date('2024-01-01')` does not.
+Every control runs on its own, and the first line says why, when the gate command is anything but `jest` or `vitest run` with `--config` and options for the number of workers (`--runInBand`, `-i`, `--ci`, `--maxWorkers`, `-w` for Jest; `--no-file-parallelism`, `--fileParallelism=false`, `--maxWorkers`, `--minWorkers`, `--pool=forks`, `--pool=threads` for Vitest), when the gate config does not use the helper, and with `settings.selfcheck.failurePattern`:
+
+```
+test-gates selfcheck: one negative control at a time (the gate config does not use createVitestGatesConfig)
+```
+
+This is a search by text: a comment that names `setTimeout` counts, and `new Date('2024-01-01')` does not. Which spec loads which gate is also read from the text: the imports, also those of a component (`.vue`, `.svelte`, `.astro`, `.mdx`), and paths handed to the runner (`vi.mock('./rates')`). A spec that loads modules by a name it computes, or by `import.meta.glob`, or that loads a component (which can use others that it does not import), may load any gate. A package that loads files of the project by itself (by a glob, for one) is not seen to load them.
 
 _Of each run next to others_, the result is taken only when the control held and the runner itself sums the run up as undisturbed: every spec file but the one left out passed, every test passed and none was skipped (` Test Files  13 passed (13)` and `Tests  14 passed (14)` of Vitest, `Test Suites: 13 passed, 13 total` and `Tests: 14 passed, 14 total` of Jest), every threshold the run reports is that of this gate, the process ended by itself, and no other line speaks of a failure, a timeout, an error or a retry. Anything less (a test that timed out under the load, a reporter that prints no summary) and the control is run again on its own after the others have ended, and that run counts. So a control that does **not** hold is always reported from a run of its own, with the output of that run. The first line says how many controls were run again:
 
@@ -428,21 +505,23 @@ test-gates selfcheck: 4 of 14 negative control(s) did not hold cleanly next to o
 
 Each of the `<n>` lines of runs has a directory of its own, outside of the project, for what the coverage tool writes and for the cache of the runner. It is given on the command line (`--coverage.reportsDirectory` and `--no-cache` for Vitest, `--coverageDirectory` and `--cacheDirectory` for Jest), which wins over the gate config, and removed when the command ends. Vitest keeps the coverage of every spec file in that directory until the run ends, and empties it when a run starts: two runs in one directory read each other's files and report a gate at 0% whose spec ran. That is why a gate command with any other argument is not run at the same time; it could name such a place itself.
 
-**Why the default stays 1.** What `--jobs` reports was compared with one control at a time in `test/selfcheck-runners.test.ts`, with a real Jest and a real Vitest (both pools), for the project described above for `--quick`: the same lines and the same exit code, TBD-STRESS. A hunt for projects in which the two differ found six, all of the same kind: a control that does not hold on its own, reported as holding with `--jobs`.
+**Why the default stays 1.** What `--jobs` reports was compared with one control at a time in `test/selfcheck-runners-*.test.ts`, with a real Jest and a real Vitest (both pools), for the project described above for `--quick`: the same lines and the same exit code, TBD-STRESS. A hunt for projects in which the two differ found six, all of the same kind: a control that does not hold on its own, reported as holding with `--jobs`.
 
 | What the project does                                                                                                | What now keeps it from being reported as holding                             |
 | -------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
-| a helper of a spec takes a lock on a file and, when another run has it, leaves out the calls that cover another gate | the helper imports `fs`: one control at a time                               |
-| the same with a server on a fixed port                                                                               | the helper imports `net`: one control at a time                              |
-| the same in a setup file named in the gate config                                                                    | the files the gate config names are read too                                 |
+| a helper of a spec takes a lock on a file and, when another run has it, leaves out the calls that cover another gate | the helper imports `fs`: that control on its own                             |
+| the same with a server on a fixed port                                                                               | the helper imports `net`: that control on its own                            |
+| the same in a setup file named in the gate config                                                                    | the files the gate config names are read too: every control on its own       |
 | a test skips itself when it does not get the lock (the lock is taken inside a package)                               | the summary of the run says `1 skipped`: that control runs again on its own  |
 | with Jest, a test fails there, and a reporter of the project prints no failures                                      | there is no summary that says all passed: that control runs again on its own |
 | with Vitest, the gate command names the coverage directory, so all runs share it                                     | a gate command with such an argument: one control at a time                  |
 
-The first three and the last are in `test/cli.test.ts`, the other two in `test/selfcheck-runners.test.ts` with the real runners. One kind is left, and it is the reason for the default: **a package the project vouches for that reaches outside of its process after all**, and quietly. With the lock of the fourth row taken inside a package listed in `settings.imports.allow`, and the test neither skipped nor failed but simply doing less, `--jobs` still reports the control as holding. Code that hides what it uses (a property of `globalThis` is found, a name put together from strings is not) is the same kind. No search by text closes that, so `--jobs` is for projects whose specs are what the specs of pure modules usually are: calls of the gate with fixed values.
+The first three and the last are in `test/cli.test.ts`, the other two in `test/selfcheck-runners-*.test.ts` with the real runners. One kind is left, and it is the reason for the default: **a package the project vouches for that reaches outside of its process after all**, and quietly. With the lock of the fourth row taken inside a package listed in `settings.imports.allow`, and the test neither skipped nor failed but simply doing less, `--jobs` still reports the control as holding. Code that hides what it uses (a property of `globalThis` is found, a name put together from strings is not) is the same kind. No search by text closes that, so `--jobs` is for projects whose specs are what the specs of pure modules usually are: calls of the gate with fixed values.
 
 A second kind came out of a check afterwards. `--jobs` leaves out of the shared runs the specs that use the clock or timers, but it finds them by a list of names, so a spec that measures elapsed time in a way that is not on the list (`+new Date`, `AbortSignal.timeout`, `performance.mark`, or a package the project vouches for) is not seen. If which path such a spec takes depends on how long something took, the lines it covers change with the load, and a control that does not hold one at a time can be counted as holding with `--jobs`. Do the final check without `--jobs`.
 
+That the controls kept apart are chosen one by one, rather than all of them for any file that reaches outside, does not make `--jobs` safer or less safe for a control that runs next to others: what can disturb it is looked for in the same files as before, those of the specs that load its gate and those the gate config names. It makes the option work in projects where some specs need a clock, a file or a package. Before, a single such file anywhere kept every control on its own. `test/selfcheck-runners-*.test.ts` has a project of five gates, with a real Jest and a real Vitest, in which a spec that reads files loads another gate: that control runs on its own, the others next to each other, and the lines and the exit code are those of one control at a time, also for a gate that another spec covers.
+
 **The number of workers is the gate command's.** Every run starts its own workers, and both runners take about as many as the machine has cores: `--jobs 4` is then four times that many processes. Say how many a run may take in the gate command: `"gateCommand": ["jest", "--config", "jest.gates.config.js", "--runInBand"]` (or `--maxWorkers=2`), and for Vitest `--no-file-parallelism` or `--maxWorkers=2`. `selfcheck` does not add such an option on its own: one worker per run is slower for a single run, and which is faster in sum depends on the project. TBD-SPEED
 
 **With the other options.** `--jobs` applies to the gates that are selected: to the part of `--shard`, to the gates of `--gate`. With `--first` or a single `--gate` there is one control and nothing to run at the same time. With `--quick`, the shared run comes first and `--jobs` applies to the controls it leaves. Unlike `--quick`, `--jobs` is not refused in CI: within the limits above a control is judged from a run that left out its spec and nothing else. On machines of a hosted CI with two cores, `--shard` over parallel jobs gains more.
@@ -711,11 +790,11 @@ Mutation testing is the expensive step. Numbers for this repository's own gates:
 
 The time grows with the number of mutants and depends heavily on cores and on `concurrency`. Twenty gates of this size on a slow machine are well past ten minutes for a full run. The setup that has held up in practice:
 
-- **pre-push: incremental.** `test-gates mutation` as is. Stryker's incremental file (`reports/stryker-incremental.json`, not committed) makes a run after no change take seconds; the first run after a clone is a full one, and so is every run after a change in the subproject or in what it depends on (see [When the results of the last run are reused](#when-the-results-of-the-last-run-are-reused)).
+- **pre-push: incremental.** `test-gates mutation --affected` (or `test-gates mutation` as is, which runs every mutant after any change). Stryker's incremental file (`reports/stryker-incremental.json`, not committed) makes a run after no change take seconds; the first run after a clone is a full one, and so is every run after a change in the subproject or in what it depends on (see [When the results of the last run are reused](#when-the-results-of-the-last-run-are-reused)).
 - **CI: full.** `npm run test:gates:mutation -- --force`. CI has no incremental file to start from anyway, and the full run is the one that counts.
 - If even CI is too slow, split by subproject (each has its own `test-gates.json`) before you think about sampling.
 
-The negative control (`selfcheck`) is the step that grows fastest: a run of the gate command per gate, each executing the specs of all the other gates, grows up to the square of the number of gates. With `--quick` it uses one run for the gates that only their own spec touches (see [`selfcheck`](#commands) for what that takes for granted). Measured on a project written for this measurement: N gates of 20 lines each, one spec of five tests per gate, Vitest 3.2 with istanbul or Jest 30 with ts-jest, Stryker 10 without a `concurrency` setting; an Apple M2 Max (12 cores), Node 22.22, other jobs running. Read the growth, not the seconds:
+The negative control (`selfcheck`) is the step that grows fastest: a run of the gate command per gate, each executing the specs of all the other gates, grows up to the square of the number of gates. With `--quick` it uses one run for the gates that only their own spec touches (see [`selfcheck`](#commands) for what that takes for granted). Measured on a project written for this measurement: N gates of 20 lines each, one spec of five tests per gate, Vitest 3.2 with istanbul or Jest 30 with ts-jest, Stryker 10 with its own default `concurrency` (one fewer than the cores, before the helper set half of them); an Apple M2 Max (12 cores), Node 22.22, other jobs running. Read the growth, not the seconds:
 
 | Gates | Runner | Gate run | `selfcheck` | `selfcheck --quick` | `mutation -- --force` |
 | ----- | ------ | -------- | ----------- | ------------------- | --------------------- |
@@ -728,7 +807,7 @@ The negative control (`selfcheck`) is the step that grows fastest: a run of the
 
 In this project no spec touches the gate of another, so `selfcheck --quick` is two runs of the gate command. (That column was measured later, with fewer other jobs running than for the other columns; the difference in load is far smaller than the difference between the two columns.) Every gate that another spec touches adds a run of its own. For the plain `selfcheck`, which is the one for CI, `--shard <index>/<total>` splits the runs over parallel jobs: each part runs its share of the gates. That does not reduce the total work; only the waiting time gets shorter.
 
-How the incremental file and this tool interact (covered by `test/incremental.test.ts`, which runs a real Stryker):
+How the incremental file and this tool interact (covered by `test/incremental-*.test.ts`, which runs a real Stryker):
 
 - `test-gates mutation` deletes the JSON **report** before each run. It does not touch the incremental file; they are different files. It decides whether Stryker may read it: see [When the results of the last run are reused](#when-the-results-of-the-last-run-are-reused).
 - A run in which every result is reused still writes a complete JSON report, and the verdict is the same as that of the run it reuses.
@@ -759,7 +838,7 @@ How the incremental file and this tool interact (covered by `test/incremental.te
 
 ## Contributing
 
-Run `npm run check:quick` (lint, typecheck, format check and the light tests; about 20 seconds) before you push. CI runs the same checks as a separate fast job, and the heavy tests, gates and mutation test as parallel jobs.
+Run `npm run check:quick` (lint, typecheck, format check and the light tests; about 20 seconds) before you push. CI runs the same checks as a separate fast job, and the heavy tests (spread over four shards, on Node 22 and on Node 20), gates and mutation test as parallel jobs. `npm run test:heavy` runs all of them in one process.
 
 ## License
 

package/dist/{chunk-XMZ6NBV7.js → chunk-AK2NTFQO.js}

@@ -489,6 +489,10 @@ function judgeSelfcheck(input) {
   const reported = output.split("\n").some((line) => line.includes(gatePath) && THRESHOLD_MESSAGE.test(line));
   return reported ? null : `the gate failed (exit ${status}) but not on the coverage threshold of ${gatePath}. It may be failing for another reason`;
 }
+function inCi(env) {
+  const ci = (env.CI ?? "").trim().toLowerCase();
+  return ci !== "" && ci !== "false" && ci !== "0";
+}
 
 // src/load.ts
 import fs2 from "fs";
@@ -626,6 +630,7 @@ export {
   resolveSettings,
   EXCLUDE_ENV,
   judgeSelfcheck,
+  inCi,
   gateExtensionOf,
   looksLikeSpec,
   existsExact,