@t09tanaka/test-gates 0.3.3 → 0.3.4

package/CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## 0.3.4
+
+One change of behaviour: `test-gates mutation` no longer lets Stryker reuse old results after a change. The rest is new options of `selfcheck`; without them it runs as before.
+
+### Changed
+
+- `test-gates mutation` lets Stryker reuse the results of the last run only when nothing the run depends on has changed since. Stryker's incremental mode reuses the result of a mutant when the text at its place and a test that killed it are the same; a change on another line of the gate, or in a file the spec imports, could turn a killed mutant into a survivor and the old "Killed" was reused. A record next to the incremental file (`reports/stryker-incremental.json.test-gates.json`) now holds a hash of every file of the subproject and of what it leads to, the versions of the tools and the arguments for Stryker. When anything differs, or there is no record, the run gets `--force` and the first line says why. The first run after the update is therefore one of every mutant. A run after no change is reused as before. See "When the results of the last run are reused" in the README for what is compared and what is not seen.
+- `test-gates mutation` takes `--gate` for itself. Before, it was handed to Stryker like every unknown argument (Stryker has no such option).
+- `test-gates check`, `lcov` and the config helpers read each directory once while they look for the gates and their specs, instead of once for every gate and every place its spec may be in. What they report is unchanged.
+
+### Added
+
+- `test-gates selfcheck --gate <path>` (may be given more than once) and `--shard <index>/<total>` run the negative control of a part of the gates: one gate while working on it, or one of several parts for parallel CI jobs. The last line says how many gates were not selected. More parts than gates, a path that is not a gate, and either option after `--` are errors (exit code 2).
+- `test-gates selfcheck --quick` uses one run of the gate command for all the gates that only their own spec touches, as far as a full gate run with the config helpers shows it (Jest 30.5; Vitest 3.2 with the istanbul provider). It is for work in progress and not the final check: it takes for granted that a spec executes the same code whichever other specs run with it. Controls judged in the shared run are marked in the output, the last line asks for a run without the option, and the option is refused when `CI` is set (exit code 2).
+- README: Dart and Flutter projects are pointed to the package for Dart; `lcov --file` is shown as optional.
+- `test-gates mutation --gate <path>` runs and judges the mutants of the named gates only, for the time a gate is being worked on. It is not the final check: a change in one gate can make a mutant of another gate survive. The last line says so, the run takes nothing over from an earlier one and leaves neither a record nor a JSON report behind, and the option is refused when `CI` is set.
+- `test-gates selfcheck --jobs <n>` runs `<n>` negative controls at the same time. The default stays 1. A control is taken from a run next to others only when the runner sums that run up as undisturbed (every other spec file passed, every test passed, none skipped, no threshold but that of its gate, no line that speaks of a failure); any other control is run again on its own, and that run counts, so a control that does not hold is always reported from a run of its own. The controls are started at the same time only when the gate command is `jest` or `vitest run` with `--config` and options for the number of workers, the gate config uses the helper, and no file a gate run loads (specs, scripts the gate config names, and the project files those import) is seen to use the environment, the clock, the network, a built-in module of Node.js or a package the project does not vouch for; otherwise the first line says why and the run is one control at a time. Each line of runs writes its coverage and the cache of the runner to a directory of its own outside of the project. See the README for what the search by text cannot see, which is why it is an option.
+
 ## 0.3.3
 
 Documentation only; the commands and the helpers behave as in 0.3.2.

package/README.md

@@ -109,7 +109,7 @@ export default createStrykerGatesConfig({
 });
 ```
 
-Add `reports/` and `.stryker-tmp/` to `.gitignore`.
+Add `reports/`, `.stryker-tmp/` and `.test-gates-touch-*` (see `test-gates selfcheck`) to `.gitignore`.
 
 ### 4. Scripts
 
@@ -182,9 +182,28 @@ test-gates:
     - run: npm run test:gates:mutation
 ```
 
+When the negative control takes too long with many gates, run it as a job of its own, in parts (`--shard`, see `test-gates selfcheck` under [Commands](#commands)):
+
+```yaml
+test-gates-selfcheck:
+  runs-on: ubuntu-latest
+  strategy:
+    fail-fast: false
+    matrix:
+      part: [1, 2, 3, 4]
+  defaults:
+    run:
+      working-directory: backend
+  steps:
+    # checkout, setup-node and npm ci as above
+    - run: npm run test:gates:selfcheck -- --shard ${{ matrix.part }}/4
+```
+
+The number of parts is written twice, in `matrix.part` and after the `/`; keep the two in step. The `test:gates:selfcheck` script must not itself end in `-- <gate command>`, or the option would land behind it (which is refused); put the command into `settings.gateCommand` instead.
+
 ## Commands
 
-All commands take `--dir <subproject>` (default: the current directory).
+All commands take `--dir <subproject>` (default: the current directory). `test-gates -h` (`--help`) prints the usage and `test-gates -v` (`--version`) the version.
 
 | Exit code | Meaning                                                                    |
 | --------- | -------------------------------------------------------------------------- |
@@ -208,10 +227,64 @@ Static checks, before the gate run. Reports every violation as `file:line: reaso
 - The Stryker config does not use `excludedMutations`, `ignoreStatic` or `ignorers`.
 - A Vitest gate run below Vitest 4 does not use the plain v8 coverage provider (since 0.3.0), see [Vitest: v8 or istanbul](#vitest-v8-or-istanbul).
 
-### `test-gates mutation [-- <stryker arguments>]`
+### `test-gates mutation [--gate <path>…] [-- <stryker arguments>]`
 
 Runs `stryker run <config>` from the project's own `node_modules`, then judges the JSON report (see `mutation-result`). Unknown arguments, and everything after `--`, are handed to Stryker. If Stryker itself fails, the exit code is 1 and nothing is judged. A report from an earlier run is deleted first, so a run that writes none cannot pass on old results.
 
+#### When the results of the last run are reused
+
+In Stryker's incremental mode (the helper turns it on) a run reuses the old result of a mutant when the text at the place of the mutant is the same and a test that killed it is the same. That is less than it sounds. Measured with Stryker 10, with Jest and with Vitest, in projects written for this; each time the incremental run reused the old "Killed" and passed, and `--force` found survivors:
+
+- an assertion weakened in a file that the spec imports (111 of 111 results reused; three survivors),
+- **the gate itself changed, on another line than the mutant**: a helper in the same file that the mutated comparison goes through (10 of 12 results reused; two survivors).
+
+So the incremental file can only be trusted when nothing at all changed. `test-gates mutation` keeps a record next to it (`reports/stryker-incremental.json.test-gates.json`) of everything the run depended on, and compares it before the next run. When something differs, or there is no record, it adds `--force` and says why:
+
+```
+test-gates mutation: running every mutant again (--force), the results of the last run are not reused: changed since the last run: src/expect-true.ts
+```
+
+What is left of incremental mode is the run after no change: a push that does not touch the subproject, a run repeated after a failure elsewhere. That one takes seconds.
+
+What is compared:
+
+- Every file of the subproject: gates, specs, other sources, helpers of the specs, configs, `test-gates.json`, the lock file.
+- The files directly in the directories above the subproject, up to the root of the repository: the lock file and the shared configs of a monorepo.
+- Packages of the same repository that are linked into `node_modules`, and directories outside the subproject that a file leads to, with a link or with a relative path (`../../pricing-kit/rounding/half-up`). When a path cannot be followed (`<rootDir>/../pricing-kit`, a way up handed to `path.join` as separate `'..'` arguments, a path to a file or a directory above the subproject), every file of the repository is compared.
+- What is installed (`node_modules/.package-lock.json`, or what yarn and pnpm write in its place), the versions of Node.js, of this package, of Stryker and of the test runners, and the arguments handed to Stryker.
+- The incremental file itself: it has to be the one the recorded run wrote.
+
+With git, "every file" is what git tracks plus the untracked files it does not ignore. Without git, or when git ignores the subproject, every file is read except `node_modules`, `.git` and what the run writes itself, up to the nearest directory that has a `.git` entry, else the topmost one that declares workspaces, else only the subproject. Such a listing also contains what git would ignore (coverage output, an HTML report, logs), so a file that changes between two runs makes every run a full one; reuse is meant for a checkout with git.
+
+The record is removed before each run and written only after a run that Stryker ended without an error and in which it wrote the incremental file, so a run that failed, was interrupted or was a dry run is followed by a full one. A run that is given `--force` is not changed. When Stryker does not run in incremental mode, nothing is recorded or compared.
+
+The paths that leave the subproject are found by text, in every file that is compared. The search errs towards comparing too much: text that merely looks like such a path widens the comparison, and a path or a link to a directory that exists outside the repository turns reuse off, with the file named in the message.
+
+The comparison loads the Stryker config in a process of its own, to see where the incremental file is, and reads and hashes those files before each run. Timed with a stand-in for Stryker, so that only this tool is measured, in a project of 5,200 files written for the measurement: 0.13 s without the comparison, 0.44 to 0.55 s with it (medians of three rounds; other jobs were running on the machine).
+
+What it does not see, and what therefore still needs `-- --force` by hand, or CI, where the run should always be a full one:
+
+- files git ignores (build output that a spec or a config loads, `.env`),
+- environment variables, the clock, the network, and a Stryker config that reads them or the command line to decide about incremental mode,
+- files outside the subproject that are reached without a link and without `..` in a path: an absolute path, an alias defined in a config above the subproject (in that case every file of the repository is compared only if the subproject points at that config with a relative path),
+- changes inside `node_modules` that leave the files of the package manager as they were.
+
+#### `--gate <path>`: only the mutants of one gate
+
+While working on one gate, `test-gates mutation --gate src/pricing/shipping-fee.ts` runs the mutants of that gate only (it hands the path to Stryker as `--mutate`) and judges that gate only. The option can be given more than once; the path is written as in `test-gates.json`.
+
+**It is not the final check, and the last line says so.** A change in one gate, in its spec or in a file they import can make a mutant of **another** gate survive, and the mutants of that other gate are not run. `test/incremental.test.ts` has such a project, with a real Stryker: the spec of one gate compares its result through a function of another gate; that function is changed to tell less apart, together with its own spec. Every mutant of the changed gate is still killed. Running only those and taking over the old results of the other gate from Stryker's incremental file passes. The run of every gate finds the survivor in the other gate.
+
+So a run with `--gate`
+
+- judges the named gates and no other (the report of such a run can hold old results of the other gates, which Stryker takes over from its incremental file),
+- takes nothing over from an earlier run: it adds `--force`, and removes the record of the last run, so the next run without `--gate` is one of every mutant,
+- removes the JSON report after it is judged, so that `mutation-result` cannot take it for the report of a full run,
+- ends its last line with `Only 1 of 9 gate(s) were run and judged (--gate), and a change in one gate can make a mutant of another survive: run without --gate for the final check`,
+- **is refused when `CI` is set** in the environment (to anything but an empty value, `false` or `0`), with exit code 2, like `selfcheck --quick`.
+
+`--gate` is written before `--`, and cannot be combined with a `--mutate` of your own.
+
 ### `test-gates mutation-result [--report <path>]`
 
 Judges an existing Stryker JSON report without running Stryker.
@@ -237,7 +310,7 @@ The run also fails when
 On failure every unallowed survivor is listed as `file:line:column / mutator / original → replacement / occurrence`, followed by a JSON fragment that can be pasted into `equivalentMutants`. The last line gives the real numbers, never a percentage. For the gates of this repository:
 
 ```
-test-gates mutation: OK (mutants 1017 / detected 1017 (timeout 2) / allowed equivalent 0 / not evaluable 0 / unallowed survivors 0)
+test-gates mutation: OK (mutants 1702 / detected 1702 (timeout 2) / allowed equivalent 0 / not evaluable 0 / unallowed survivors 0)
 ```
 
 #### Timeouts
@@ -284,11 +357,108 @@ When the run fails on a timeout, first run again with less load (`npm run test:g
 
 0.3.1 set `timeoutMS` to 30000 instead. That was the wrong fix and 0.3.2 takes it back: the longer limit let most slow mutants finish, but the same full run then took about three times as long as without it, and in another project mutants that exhaust the heap crashed the test runner before the limit and came back as `RuntimeError`, which is only counted, not failed. `timeoutMS` is Stryker's default (5000) again unless the project sets it.
 
-### `test-gates selfcheck [--all | --first] [-- <gate command>]`
+### `test-gates selfcheck [--all | --first | --gate <path>… | --shard <index>/<total>] [--jobs <n>] [--quick] [-- <gate command>]`
 
 Negative control. For each gate, runs the gate with that gate's spec left out and requires the run to **fail on the coverage threshold of that very file**. A gate that stays green without its spec measures nothing; a run that fails for another reason (a config that does not load) proves nothing either.
 
-The spec to leave out is passed in the environment variable `TEST_GATES_EXCLUDE_SPEC`. The config helpers read it. The gate command is `jest --config jest.gates.config.*` or `vitest run --config vitest.gates.config.*`, whichever config exists; override it with `settings.gateCommand` or after `--`. `--all` (default) checks every gate in turn, `--first` only the first.
+The spec to leave out is passed in the environment variable `TEST_GATES_EXCLUDE_SPEC`. The config helpers read it. The gate command is `jest --config jest.gates.config.*` or `vitest run --config vitest.gates.config.*`, whichever config exists; override it with `settings.gateCommand` or after `--`. `--all` (default) checks every gate, `--first` only the first.
+
+**`--quick`: one run for the gates that only their own spec touches.** A run of the gate command for every gate executes the specs of all the other gates each time, and takes minutes with many gates (see [Cost](#cost-and-how-to-use-strykers-incremental-mode)). With `--quick` and three gates or more, `selfcheck` starts with a full gate run in which the config helpers write down, for each spec file, what it executed of each gate. Leaving out the spec of a gate changes the coverage of that gate only by what its spec executed of it. So for a gate that no other spec touched, the result does not depend on which other specs are left out with it, and all those gates lose their specs in **one** run. Each of them is judged in that run as it is in a run of its own: the run has to fail, and it has to report the coverage threshold of that very file.
+
+**It is an option, not the default, because it takes one thing for granted: that a spec executes the same code whichever other specs run with it.** A spec that covers another gate only while that gate's spec is away is not caught by any note of a full run; the run per gate catches it, because it really leaves that one spec out. Specs of pure modules do not behave that way, and the ones that could (they read `process.env`, also the variables of `selfcheck` itself, or files, a child process, the network) are recognised by their text and switch the shared run off. That is a search by text in the spec files only, not in what they import, and it cannot be made complete. Two hunts for counterexamples found projects in which `--quick` reports every control as holding and the plain run does not, with nothing in the text of a spec: a helper that keeps state in the process (it changes a setting of a Node.js core module once; with Jest that state is shared by the spec files of one worker, so whether a later spec has to do it again, and thereby executes another gate, depends on which spec ran before it, and with the default number of workers even on Jest's cache of run times), and a helper that keeps a cache on disk which one spec fills and another reads. In each of them a spec covers a gate only while that gate's spec is away. No way of taking notes from one full run can see that.
+
+So a control that `--quick` reports as holding is not the last word; one that it reports as not holding is, because that one was run on its own. The output says so: such a line ends in `, in the shared run)`, and the last line names how many there were and asks for a run without `--quick`. **With `CI` set in the environment (to anything but an empty value, `false` or `0`), `--quick` is refused** (exit code 2): where the run is the gate itself, it has to be the full one. Most hosted CI services set `CI`; some CI systems do not set it at all, and there the job has to export `CI=true` itself, or simply not use the option.
+
+| Where                                                   | Run                                                                                   |
+| ------------------------------------------------------- | ------------------------------------------------------------------------------------- |
+| while working on one gate                               | `selfcheck --gate <path>`                                                             |
+| before a push, when CI runs the gates too               | `selfcheck --quick`                                                                   |
+| before a push, when that hook is the only gate there is | `selfcheck`                                                                           |
+| CI                                                      | `selfcheck`, with `--shard <index>/<total>` over parallel jobs when it takes too long |
+
+The notes are taken from what the runner itself counts for a spec file, after the file has run to its end: with Jest, the coverage a reporter is given for the file; with Vitest, what the worker hands to the istanbul coverage provider. All of it rests on the notes being complete: an execution that is missing from the note of a spec could make a gate look untouched. So the notes are used only within three limits, and outside of them every gate gets a run of its own, as before, with the reason on the first line (`test-gates selfcheck: a run of the gate command for every gate (…)`).
+
+1. **Only for the runs it is verified with.** This table is the list; it is also `VERIFIED_RUNS` in the code, and `test/selfcheck-runners.test.ts` compares the shared run with a run per gate for every row.
+
+   | Runner | Versions | Coverage                       | Pool                         | Other                                  |
+   | ------ | -------- | ------------------------------ | ---------------------------- | -------------------------------------- |
+   | Jest   | 30.5     | `babel` (what the helper sets) | —                            | one project                            |
+   | Vitest | 3.2      | `istanbul`                     | `forks` (default), `threads` | `isolate: true` (default), one project |
+
+   Another minor version, another provider or pool, spec files that share their modules, a workspace with several projects: a run per gate. The Vitest wrapper also checks what it relies on while it runs (the provider has `onAfterSuiteRun` and `initialize`; every hand-over of coverage names exactly one spec file) and reports a deviation, which has the same effect. Vitest 4.1 was tried by hand on three of the cases below and agreed; it is not in the table because the tests of this repository do not run it.
+
+2. **Only when the notes account for what the runner counted.** The same run writes the runner's own total (its JSON coverage report). For every gate, the executions in the notes of all specs are added up and compared with that total: every function, every branch arm and every statement. An execution that happened after a note was taken, or that no note has, shows as a total above the sum; then no note is trusted. (Checked with notes taken too early, the way the first version of this took them: a cleanup function returned by `beforeAll`, and an `afterAll` with the hooks in the order written, are both refused.) Where the two cannot be compared for a gate, because the report puts the counters back onto the source through a source map and the functions do not line up, that gate gets its own run.
+
+3. **Only on a gate run that passes.** A failing test or a gate below 100% in the full run: a run per gate.
+
+A gate also keeps a run of its own when another spec touches it (it imports the gate, a setup file calls it, an `afterAll` or a cleanup function calls it), when its own spec is not seen to touch it, or when the shared run does not confirm it. And every gate does when
+
+- the gate config does not use the helpers (the list of specs to leave out arrives in `TEST_GATES_EXCLUDE_SPECS` as a JSON list, the directory for the notes in `TEST_GATES_TOUCH_DIR`),
+- Vitest measures with the v8 provider, which the helper cannot wrap. **v8 is the default of `createVitestGatesConfig`**: the shared run needs `coverageProvider: 'istanbul'` (see [Vitest: v8 or istanbul](#vitest-v8-or-istanbul), which recommends it for another reason),
+- `settings.selfcheck.failurePattern` does not contain `{gate}`, or a gate has no single spec.
+
+When the gate command is `jest` or `vitest` and the version installed is outside of the table, or the gate command is the one derived from the gate config and that config does not name the helper (or, for Vitest, `istanbul`), this is known beforehand and the run for the notes is not started. Otherwise that run was one run for nothing, also in every part of `--shard`.
+
+With Jest and Vitest, and specs that do not depend on each other, the lines printed and the exit code are those of the run per gate. (A gate command of your own whose exit code says what failed would show the code of the shared run in the `✓` lines.) The two are compared with a real Jest and a real Vitest in `test/selfcheck-runners.test.ts`, for a gate that the spec of another gate covers, one that two other specs cover between them, one that another spec covers in part, one whose top-level code runs when another spec loads it, one that a setup file calls, one that another spec replaces with a mock, one that another spec calls in `afterAll` (with Vitest's hooks in the order written), and one that another spec calls in a cleanup function returned by `beforeAll`.
+
+During the first run a directory `.test-gates-touch-…` exists in the subproject (Vitest loads the wrapped provider from there). It is removed when the run ends. A run that is interrupted leaves it behind; a later `selfcheck` removes what is older than an hour. Add `.test-gates-touch-*` to `.gitignore`.
+
+**`--jobs <n>`: `<n>` negative controls at the same time.** The default is 1, one at a time. With `--jobs 4`, four runs of the gate command are under way at any moment, each for another gate. A control stays what it is: a run of the gate command with the spec of one gate left out, and of that one only. What `--jobs` adds is other runs on the same machine, and a run can be disturbed by them. Two things are therefore asked, one before the runs start and one of each run.
+
+_Before the runs start_, the files a gate run loads must not be seen to reach outside of their process. A spec whose helper takes a lock on a file, listens on a fixed port or looks at the clock can quietly do less while another run is there, and then a gate that this spec covers looks as if only its own spec covered it. Nothing in the output of such a run tells. The files are: the spec of every gate, every script the gate config names (a setup file, a reporter; also in the project files the config imports), and all the project files those import, the gates among them. When one of them
+
+- mentions `process.`, `globalThis`, `global.` or `import.meta.env`; the clock or a wait (`Date.now`, `new Date()`, `performance.now`, `setTimeout`, `setInterval`, `setImmediate`, `waitFor`, `waitUntil`, `expect.poll`, `retryTimes`, `retry:`); the network (`fetch(`, `WebSocket`, `XMLHttpRequest`); or `navigator`, `Atomics`, `WeakRef`, `FinalizationRegistry`, `eval(`, `Function(`,
+- imports a built-in module of Node.js, also one that only computes (`path`): with Jest the spec files of one worker share the objects of the built-in modules, so a setting that one spec changes there is seen by the next, and which spec comes next depends on Jest's cache of run times,
+- imports a package other than the runner's (`vitest`, `@jest/globals`). Packages are not looked into. With `settings.imports.mode: "allowlist"`, the packages that `settings.imports.allow` lists by name are taken too: the project already vouches for them as imports of a gate,
+- loads a module whose name is computed, or imports something that is neither a file nor an installed package (an alias that `settings.importAliases` does not have),
+
+every control runs on its own, and the first line says why:
+
+```
+test-gates selfcheck: one negative control at a time (src/support/lock.ts imports node:fs, and may meet what another run is doing)
+```
+
+The same happens when the gate command is anything but `jest` or `vitest run` with `--config` and options for the number of workers (`--runInBand`, `-i`, `--ci`, `--maxWorkers`, `-w` for Jest; `--no-file-parallelism`, `--fileParallelism=false`, `--maxWorkers`, `--minWorkers`, `--pool=forks`, `--pool=threads` for Vitest), when the gate config does not use the helper, and with `settings.selfcheck.failurePattern`. This is a search by text: a comment that names `setTimeout` counts, and `new Date('2024-01-01')` does not.
+
+_Of each run next to others_, the result is taken only when the control held and the runner itself sums the run up as undisturbed: every spec file but the one left out passed, every test passed and none was skipped (` Test Files  13 passed (13)` and `Tests  14 passed (14)` of Vitest, `Test Suites: 13 passed, 13 total` and `Tests: 14 passed, 14 total` of Jest), every threshold the run reports is that of this gate, the process ended by itself, and no other line speaks of a failure, a timeout, an error or a retry. Anything less (a test that timed out under the load, a reporter that prints no summary) and the control is run again on its own after the others have ended, and that run counts. So a control that does **not** hold is always reported from a run of its own, with the output of that run. The first line says how many controls were run again:
+
+```
+test-gates selfcheck: 4 of 14 negative control(s) did not hold cleanly next to others and run again on their own
+```
+
+Each of the `<n>` lines of runs has a directory of its own, outside of the project, for what the coverage tool writes and for the cache of the runner. It is given on the command line (`--coverage.reportsDirectory` and `--no-cache` for Vitest, `--coverageDirectory` and `--cacheDirectory` for Jest), which wins over the gate config, and removed when the command ends. Vitest keeps the coverage of every spec file in that directory until the run ends, and empties it when a run starts: two runs in one directory read each other's files and report a gate at 0% whose spec ran. That is why a gate command with any other argument is not run at the same time; it could name such a place itself.
+
+**Why the default stays 1.** What `--jobs` reports was compared with one control at a time in `test/selfcheck-runners.test.ts`, with a real Jest and a real Vitest (both pools), for the project described above for `--quick`: the same lines and the same exit code, TBD-STRESS. A hunt for projects in which the two differ found six, all of the same kind: a control that does not hold on its own, reported as holding with `--jobs`.
+
+| What the project does                                                                                                | What now keeps it from being reported as holding                             |
+| -------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
+| a helper of a spec takes a lock on a file and, when another run has it, leaves out the calls that cover another gate | the helper imports `fs`: one control at a time                               |
+| the same with a server on a fixed port                                                                               | the helper imports `net`: one control at a time                              |
+| the same in a setup file named in the gate config                                                                    | the files the gate config names are read too                                 |
+| a test skips itself when it does not get the lock (the lock is taken inside a package)                               | the summary of the run says `1 skipped`: that control runs again on its own  |
+| with Jest, a test fails there, and a reporter of the project prints no failures                                      | there is no summary that says all passed: that control runs again on its own |
+| with Vitest, the gate command names the coverage directory, so all runs share it                                     | a gate command with such an argument: one control at a time                  |
+
+The first three and the last are in `test/cli.test.ts`, the other two in `test/selfcheck-runners.test.ts` with the real runners. One kind is left, and it is the reason for the default: **a package the project vouches for that reaches outside of its process after all**, and quietly. With the lock of the fourth row taken inside a package listed in `settings.imports.allow`, and the test neither skipped nor failed but simply doing less, `--jobs` still reports the control as holding. Code that hides what it uses (a property of `globalThis` is found, a name put together from strings is not) is the same kind. No search by text closes that, so `--jobs` is for projects whose specs are what the specs of pure modules usually are: calls of the gate with fixed values.
+
+A second kind came out of a check afterwards. `--jobs` leaves out of the shared runs the specs that use the clock or timers, but it finds them by a list of names, so a spec that measures elapsed time in a way that is not on the list (`+new Date`, `AbortSignal.timeout`, `performance.mark`, or a package the project vouches for) is not seen. If which path such a spec takes depends on how long something took, the lines it covers change with the load, and a control that does not hold one at a time can be counted as holding with `--jobs`. Do the final check without `--jobs`.
+
+**The number of workers is the gate command's.** Every run starts its own workers, and both runners take about as many as the machine has cores: `--jobs 4` is then four times that many processes. Say how many a run may take in the gate command: `"gateCommand": ["jest", "--config", "jest.gates.config.js", "--runInBand"]` (or `--maxWorkers=2`), and for Vitest `--no-file-parallelism` or `--maxWorkers=2`. `selfcheck` does not add such an option on its own: one worker per run is slower for a single run, and which is faster in sum depends on the project. TBD-SPEED
+
+**With the other options.** `--jobs` applies to the gates that are selected: to the part of `--shard`, to the gates of `--gate`. With `--first` or a single `--gate` there is one control and nothing to run at the same time. With `--quick`, the shared run comes first and `--jobs` applies to the controls it leaves. Unlike `--quick`, `--jobs` is not refused in CI: within the limits above a control is judged from a run that left out its spec and nothing else. On machines of a hosted CI with two cores, `--shard` over parallel jobs gains more.
+
+**A part of the gates.** Two options run only a part. A gate gets the same verdict in a part as in the whole run.
+
+- `--shard <index>/<total>` splits the gates into `<total>` parts and checks part `<index>` (from 1). The parts are disjoint, and together they are every gate: the gates are dealt out in the order of `test-gates.json`. This is for parallel CI jobs; **every index has to run**, a single part says nothing about the gates of the others. More parts than gates is an error (exit code 2), because a part without a gate would pass without checking anything.
+- `--gate <path>` checks only that gate and can be given more than once. The path is written as in `test-gates.json`; a path that is not a gate there is an error (exit code 2).
+
+`--all`, `--first`, `--gate` and `--shard` cannot be combined, and `--shard` can be given only once. `--gate` and `--shard` win over `settings.selfcheck.mode`. Both are written before `--`: after it they would go to the gate command (Jest and Vitest have a `--shard` of their own, which would leave specs out of every run), so that is an error too (exit code 2). The last line of a run with `--gate` or `--shard` says how many gates were left out, and by what:
+
+```
+test-gates selfcheck: OK (2 negative control(s) failed as they should, 6 gate(s) not selected by --shard 1/4)
+```
+
+`--gate` is meant for a quick look while working on one gate, not as the check before a push: the negative control of a gate also changes when the spec of **another** gate starts to cover it (see the next paragraph), and that is a change in a file `--gate` was not pointed at. Run every gate in CI.
 
 **When a gate imports another gate.** If `delivery-price.ts` imports `shipping-fee.ts` and both are gates, the spec of `delivery-price` executes every line of `shipping-fee` as well. Leaving out `shipping-fee.spec.ts` then does not lower its coverage, and `selfcheck` fails for `shipping-fee` ("the gate passed although the spec … was left out"), naming the importing gate in a hint. The coverage floor of `shipping-fee` is being held up by somebody else's spec. Fix it in the importer's spec: mock the imported gate and assert **how it is called**, so that each spec covers only its own gate:
 
@@ -298,9 +468,11 @@ jest.mock('./shipping-fee');
 expect(shippingFeeCents).toHaveBeenCalledWith(weightGrams); // called with the weight, not with the price
 ```
 
-### `test-gates lcov --file <lcov.info>`
+### `test-gates lcov [--file <lcov.info>]`
+
+A line coverage floor from an lcov tracefile, without a mutation test: requires `LH == LF` for every gate in the tracefile. `--file` names the tracefile and can be left out when `settings.lcov.file` is set; with neither, the command exits with code 2. A gate without a record, or with no instrumented line, fails. Paths are compared as written in the tracefile (`lib/shipping_fee.dart`); absolute paths under the subproject are made relative.
 
-For stacks without mutation testing (Flutter / Dart): requires `LH == LF` for every gate in an lcov tracefile. A gate without a record, or with no instrumented line, fails. Paths are compared as written in the tracefile (`lib/shipping_fee.dart`); absolute paths under the subproject are made relative.
+For Dart and Flutter, use [`test_gates`](https://github.com/t09tanaka/test-gates-dart) instead. It reads the same `test-gates.json`, requires 100% of lines and of branches, has a negative control and judges a mutation test. The example below shows what this command alone does with a Dart package.
 
 ```bash
 flutter test --coverage && npx test-gates lcov --file coverage/lcov.info
@@ -492,7 +664,7 @@ Why these choices:
 - Jest measures with istanbul (`babel`), because the v8 provider does not count the untaken side of an `if` without `else`.
 - Jest fails on a `coverageThreshold` path key that matches no file, so a typo in `test-gates.json` cannot pass. Vitest does the opposite: a glob or path key in `thresholds` that matches nothing passes. The Vitest helper therefore narrows `coverage.include` to the gates and uses `perFile`.
 - Stryker never fails on its own score (`break: null`). The verdict comes from `test-gates mutation`, which knows the allow list.
-- The Stryker helper also sets defaults that can be overridden: `coverageAnalysis: 'perTest'`, `incremental` with its file under `reports/`, and `maxTestRunnerReuse: 50` (see [Timeouts](#timeouts)).
+- The Stryker helper also sets defaults that can be overridden: `coverageAnalysis: 'perTest'`, `incremental` with its file under `reports/` (reused by `test-gates mutation` only after no change, see [When the results of the last run are reused](#when-the-results-of-the-last-run-are-reused)), and `maxTestRunnerReuse: 50` (see [Timeouts](#timeouts)).
 
 `loadGates(dir)` returns the validated manifest: `gates` (each with its resolved `spec`), `candidates` and the resolved `settings`. It throws when the manifest has any problem.
 
@@ -539,16 +711,28 @@ Mutation testing is the expensive step. Numbers for this repository's own gates:
 
 The time grows with the number of mutants and depends heavily on cores and on `concurrency`. Twenty gates of this size on a slow machine are well past ten minutes for a full run. The setup that has held up in practice:
 
-- **pre-push: incremental.** `test-gates mutation` as is. Stryker's incremental file (`reports/stryker-incremental.json`, not committed) makes the second and later runs take seconds; the first run after a clone is a full one.
+- **pre-push: incremental.** `test-gates mutation` as is. Stryker's incremental file (`reports/stryker-incremental.json`, not committed) makes a run after no change take seconds; the first run after a clone is a full one, and so is every run after a change in the subproject or in what it depends on (see [When the results of the last run are reused](#when-the-results-of-the-last-run-are-reused)).
 - **CI: full.** `npm run test:gates:mutation -- --force`. CI has no incremental file to start from anyway, and the full run is the one that counts.
 - If even CI is too slow, split by subproject (each has its own `test-gates.json`) before you think about sampling.
 
+The negative control (`selfcheck`) is the step that grows fastest: a run of the gate command per gate, each executing the specs of all the other gates, grows up to the square of the number of gates. With `--quick` it uses one run for the gates that only their own spec touches (see [`selfcheck`](#commands) for what that takes for granted). Measured on a project written for this measurement: N gates of 20 lines each, one spec of five tests per gate, Vitest 3.2 with istanbul or Jest 30 with ts-jest, Stryker 10 without a `concurrency` setting; an Apple M2 Max (12 cores), Node 22.22, other jobs running. Read the growth, not the seconds:
+
+| Gates | Runner | Gate run | `selfcheck` | `selfcheck --quick` | `mutation -- --force` |
+| ----- | ------ | -------- | ----------- | ------------------- | --------------------- |
+| 5     | Vitest | 0.9 s    | 3.8 s       | 1.3 s               | 13 s                  |
+| 30    | Vitest | 2.0 s    | 49 s        | 2.1 s               | 38 s                  |
+| 100   | Vitest | 5.1 s    | 8 min 4 s   | 5.0 s               | 1 min 55 s            |
+| 5     | Jest   | 0.8 s    | 6.3 s       | 1.4 s               | 8 s                   |
+| 30    | Jest   | 2.4 s    | 1 min 19 s  | 3.1 s               | 22 s                  |
+| 100   | Jest   | 3.1 s    | 6 min 2 s   | 4.5 s               | 1 min 8 s             |
+
+In this project no spec touches the gate of another, so `selfcheck --quick` is two runs of the gate command. (That column was measured later, with fewer other jobs running than for the other columns; the difference in load is far smaller than the difference between the two columns.) Every gate that another spec touches adds a run of its own. For the plain `selfcheck`, which is the one for CI, `--shard <index>/<total>` splits the runs over parallel jobs: each part runs its share of the gates. That does not reduce the total work; only the waiting time gets shorter.
+
 How the incremental file and this tool interact (covered by `test/incremental.test.ts`, which runs a real Stryker):
 
-- `test-gates mutation` deletes the JSON **report** before each run. It does not touch the incremental file; they are different files.
+- `test-gates mutation` deletes the JSON **report** before each run. It does not touch the incremental file; they are different files. It decides whether Stryker may read it: see [When the results of the last run are reused](#when-the-results-of-the-last-run-are-reused).
 - A run in which every result is reused still writes a complete JSON report, and the verdict is the same as that of the run it reuses.
-- **A stale "Survived" can be reused.** After adding a test that kills a _static_ mutant (code that runs when the module is loaded, such as a module-level constant), the incremental run kept reporting the mutant as survived; `--force` reported it killed. The error is on the safe side, a failure that is not real, never a pass that is not real. When a survivor makes no sense after you fixed the test, run once with `-- --force`.
-- **After removing a gate, its old results stay in the incremental file** and come back in the report. They are ignored: only the gates of `test-gates.json` are judged. Delete the incremental file (or run with `--force`) when you change the list of gates, so that it does not carry files that are no longer mutated.
+- **After removing a gate, its old results stay in the incremental file** and come back in the report. They are ignored: only the gates of `test-gates.json` are judged. (A change of `test-gates.json` makes the next run a full one, and the old results still come back.) Delete the incremental file when you change the list of gates, so that it does not carry files that are no longer mutated.
 
 ## Limitations and things measured
 
@@ -560,13 +744,23 @@ How the incremental file and this tool interact (covered by `test/incremental.te
 - **`process.env` is found by text too.** A comment that mentions it is reported, and reading the environment through another name (`const { env } = process`) is not. The clock (`Date.now()`, `new Date()`) and `Math.random()` are not checked: a default argument such as `now = new Date()` is a legitimate way to pass them in, and text cannot tell it from a direct read. Add a `forbiddenSource` rule if the project wants one.
 - **Imports are found by text, not by parsing.** The checks run without the project's dependencies installed, so an import inside a comment is reported too, and only the gate's own imports are looked at, not what those modules import in turn.
 - **An ordinary `import { Request } from 'express'` used only as a type is reported.** Write `import type`, or exempt the module with `impureImports.allow`.
-- **`test-gates lcov` checks lines only.** lcov from Flutter carries no branch data, and Dart has no established mutation testing tool; a Dart gate is guarded by coverage alone.
+- **`test-gates lcov` checks lines only.** It does not read the branch entries (`BRDA:`) of a tracefile and runs no mutation test, so a gate that only this command checks is guarded by line coverage alone. Dart and Flutter code is guarded by [`test_gates`](https://github.com/t09tanaka/test-gates-dart), which checks lines and branches and judges a mutation test.
 - **knip reports the Stryker runner as unused** once no npm script calls `stryker run` directly. Tell knip where the config is: `"stryker": { "config": ["stryker.gates.config.mjs"] }` in `knip.json`.
 - **A directory or worktree whose name contains `test-`** (for example a checkout called `test-gates-migration`) makes `@darraghor/eslint-plugin-nestjs-typed` treat every file as a test file, because it matches the whole path against a test pattern. Not caused by this package, but a name like that is easy to choose while adopting it.
 - **A comment line that starts with `@t09tanaka/…`** in a config file can be read by ESLint's JSDoc rules as a tag. Put the package name in the middle of the sentence.
 - **Vitest 4's `text` reporter can print only the summary line** and no per-file table for the gate run. The thresholds and the negative control still work.
 - **The checks do not judge what belongs in `gates`.** Whether every important decision has been extracted and registered is for people to review; `candidates` is where the known gaps are written down.
 
+## Related tools
+
+- [test_gates](https://github.com/t09tanaka/test-gates-dart) (pub.dev): the same gates for Dart and Flutter.
+- [dart_mutant](https://github.com/t09tanaka/dart_mutant): mutation testing for Dart, the tool that writes the mutation report those gates judge.
+- [contract-fixtures](https://github.com/t09tanaka/contract-fixtures) (TypeScript and Dart): shared request and response fixtures that keep an API and its clients in step.
+
+## Contributing
+
+Run `npm run check:quick` (lint, typecheck, format check and the light tests; about 20 seconds) before you push. CI runs the same checks as a separate fast job, and the heavy tests, gates and mutation test as parallel jobs.
+
 ## License
 
 MIT

package/dist/chunk-TOKHAUFX.js

@@ -0,0 +1,163 @@
+// src/touch.ts
+import { createHash } from "crypto";
+import fs from "fs";
+import path from "path";
+import { fileURLToPath } from "url";
+var TOUCH_ENV = "TEST_GATES_TOUCH_DIR";
+function isObject(value) {
+  return typeof value === "object" && value !== null && !Array.isArray(value);
+}
+function countersOf(coverage) {
+  return {
+    s: Object.values(coverage.s ?? {}),
+    f: Object.values(coverage.f ?? {}),
+    b: Object.values(coverage.b ?? {}).flat()
+  };
+}
+function countersOfAll(coverage) {
+  if (!isObject(coverage)) {
+    return null;
+  }
+  const files = {};
+  for (const [file, counters] of Object.entries(coverage)) {
+    if (!isObject(counters)) {
+      return null;
+    }
+    files[file] = countersOf(counters);
+  }
+  return files;
+}
+function add(before, now) {
+  return before.length === now.length ? before.map((count, index) => count + (now[index] ?? 0)) : [-1];
+}
+function writeSpecNote(directory, spec, coverage) {
+  const files = countersOfAll(coverage) ?? {};
+  const specs = path.join(directory, "specs");
+  fs.mkdirSync(specs, { recursive: true });
+  const file = path.join(specs, `${createHash("sha256").update(spec).digest("hex")}.json`);
+  let before = {};
+  try {
+    before = JSON.parse(fs.readFileSync(file, "utf8")).files;
+  } catch {
+  }
+  for (const [name, counters] of Object.entries(before)) {
+    const now = files[name];
+    files[name] = now === void 0 ? counters : { s: add(counters.s, now.s), f: add(counters.f, now.f), b: add(counters.b, now.b) };
+  }
+  const note = { spec, files };
+  fs.writeFileSync(file, JSON.stringify(note));
+}
+function writeRunInfo(directory, run) {
+  fs.mkdirSync(directory, { recursive: true });
+  fs.writeFileSync(path.join(directory, "run.json"), JSON.stringify(run));
+}
+function jestReporterFile(moduleUrl) {
+  const directory = path.dirname(fileURLToPath(moduleUrl));
+  const found = [".cjs", ".ts"].map((extension) => path.join(directory, `jest-touch-reporter${extension}`)).find((file) => fs.existsSync(file));
+  if (found === void 0) {
+    throw new Error(`test-gates: jest-touch-reporter is missing in ${directory}`);
+  }
+  return found;
+}
+var VITEST_PROVIDER = `import { createHash } from 'node:crypto';
+import fs from 'node:fs';
+import path from 'node:path';
+import istanbul from '@vitest/coverage-istanbul';
+
+const directory = "__DIRECTORY__";
+const run = { runner: 'vitest', version: 'unknown', provider: 'istanbul', pool: 'unknown', isolate: false, projects: 0, problem: null };
+const save = () => fs.writeFileSync(path.join(directory, 'run.json'), JSON.stringify(run));
+const problem = (text) => {
+  run.problem ??= text;
+  save();
+};
+
+const counts = (coverage) =>
+  Object.fromEntries(
+    Object.entries(coverage).map(([file, counters]) => [
+      file,
+      {
+        s: Object.values(counters.s ?? {}),
+        f: Object.values(counters.f ?? {}),
+        b: Object.values(counters.b ?? {}).flat(),
+      },
+    ])
+  );
+const add = (before, now) =>
+  before.length === now.length ? before.map((count, index) => count + now[index]) : [-1];
+
+function note(spec, files) {
+  const file = path.join(directory, 'specs', createHash('sha256').update(spec).digest('hex') + '.json');
+  let before = {};
+  try {
+    before = JSON.parse(fs.readFileSync(file, 'utf8')).files;
+  } catch {
+    // the first note of this spec
+  }
+  const merged = { ...files };
+  for (const [name, counters] of Object.entries(before)) {
+    const now = files[name];
+    merged[name] =
+      now === undefined
+        ? counters
+        : { s: add(counters.s, now.s), f: add(counters.f, now.f), b: add(counters.b, now.b) };
+  }
+  fs.writeFileSync(file, JSON.stringify({ spec, files: merged }));
+}
+
+export default {
+  ...istanbul,
+  async getProvider() {
+    const provider = await istanbul.getProvider();
+    fs.mkdirSync(path.join(directory, 'specs'), { recursive: true });
+    save();
+    if (typeof provider.onAfterSuiteRun !== 'function' || typeof provider.initialize !== 'function') {
+      problem('the coverage provider of this Vitest has no onAfterSuiteRun or no initialize');
+      return provider;
+    }
+    const initialize = provider.initialize.bind(provider);
+    provider.initialize = (ctx) => {
+      // A project can set its own pool and isolation: all of them have to say the same.
+      const projects = Array.isArray(ctx?.projects) ? ctx.projects : [];
+      const configs = [ctx?.config ?? {}, ...projects.map((project) => project?.config ?? {})];
+      const pools = [...new Set(configs.map((config) => (typeof config.pool === 'string' ? config.pool : 'unknown')))];
+      run.version = typeof ctx?.version === 'string' ? ctx.version : 'unknown';
+      run.pool = pools.length === 1 ? pools[0] : 'mixed';
+      run.isolate = configs.every(
+        (config) => (config.poolOptions?.[config.pool]?.isolate ?? config.isolate) === true
+      );
+      run.projects = projects.length;
+      save();
+      return initialize(ctx);
+    };
+    const handOver = provider.onAfterSuiteRun.bind(provider);
+    provider.onAfterSuiteRun = (meta) => {
+      const coverage = meta?.coverage;
+      if (coverage !== undefined && coverage !== null) {
+        const specs = meta.testFiles;
+        if (typeof coverage !== 'object' || !Array.isArray(specs) || specs.length !== 1 || typeof specs[0] !== 'string') {
+          problem('Vitest handed over coverage that is not that of exactly one spec file');
+        } else {
+          note(specs[0], counts(coverage));
+        }
+      }
+      return handOver(meta);
+    };
+    return provider;
+  },
+};
+`;
+function vitestProviderFile(directory) {
+  const file = path.join(directory, "vitest-provider.mjs");
+  fs.mkdirSync(directory, { recursive: true });
+  fs.writeFileSync(file, VITEST_PROVIDER.replace('"__DIRECTORY__"', JSON.stringify(directory)));
+  return file;
+}
+
+export {
+  TOUCH_ENV,
+  writeSpecNote,
+  writeRunInfo,
+  jestReporterFile,
+  vitestProviderFile
+};

package/dist/{chunk-F7WBEFIL.js → chunk-XMZ6NBV7.js}

@@ -452,6 +452,24 @@ function resolveSettings(raw) {
 
 // src/core/selfcheck.ts
 var EXCLUDE_ENV = "TEST_GATES_EXCLUDE_SPEC";
+var EXCLUDE_ALL_ENV = "TEST_GATES_EXCLUDE_SPECS";
+function excludedSpecs(env) {
+  const one = env[EXCLUDE_ENV];
+  const several = env[EXCLUDE_ALL_ENV];
+  const excluded = one === void 0 || one === "" ? [] : [one];
+  if (several === void 0 || several === "") {
+    return excluded;
+  }
+  let listed = null;
+  try {
+    listed = JSON.parse(several);
+  } catch {
+  }
+  if (!Array.isArray(listed) || !listed.every((spec) => typeof spec === "string")) {
+    throw new Error(`test-gates: ${EXCLUDE_ALL_ENV} must be a JSON list of spec paths`);
+  }
+  return [...excluded, ...listed];
+}
 var THRESHOLD_MESSAGE = /threshold|coverage data for/i;
 function escapeRegExp(text) {
   return text.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
@@ -503,19 +521,22 @@ function looksLikeSpec(gatePath, settings) {
 // src/fs.ts
 import fs from "fs";
 import path from "path";
-function existsExact(rootDir, relativePath, kind = "file") {
+function existsExact(rootDir, relativePath, kind = "file", listing = /* @__PURE__ */ new Map()) {
   let current = rootDir;
   for (const segment of relativePath.split("/")) {
     if (segment === "" || segment === ".") {
       continue;
     }
-    let entries;
-    try {
-      entries = fs.readdirSync(current);
-    } catch {
-      return false;
+    let entries = listing.get(current);
+    if (entries === void 0) {
+      try {
+        entries = fs.readdirSync(current);
+      } catch {
+        entries = null;
+      }
+      listing.set(current, entries);
     }
-    if (!entries.includes(segment)) {
+    if (entries === null || !entries.includes(segment)) {
       return false;
     }
     current = path.join(current, segment);
@@ -553,9 +574,12 @@ function readManifest(dir) {
     }
     throw error;
   }
+  const listing = /* @__PURE__ */ new Map();
   const gates = parsed.gates.map((gate) => {
     const specCandidates = specCandidatesFor(gate.path, settings);
-    const existingSpecs = specCandidates.filter((candidate) => existsExact(absoluteDir, candidate));
+    const existingSpecs = specCandidates.filter(
+      (candidate) => existsExact(absoluteDir, candidate, "file", listing)
+    );
     return {
       ...gate,
       specCandidates,
@@ -582,8 +606,8 @@ function loadGates(dir = process.cwd()) {
   return manifest;
 }
 function gateSpecs(manifest, env = process.env) {
-  const excluded = env[EXCLUDE_ENV];
-  return manifest.gates.map((gate) => gate.spec).filter((spec) => spec !== null && spec !== excluded);
+  const excluded = excludedSpecs(env);
+  return manifest.gates.map((gate) => gate.spec).filter((spec) => spec !== null && !excluded.includes(spec));
 }
 var NO_SPEC_PATTERN = "__test_gates_no_spec_left__";