@t09tanaka/test-gates 0.4.1 → 0.4.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +69 -0
- package/README.md +66 -19
- package/dist/cli.js +2392 -786
- package/dist/index.cjs +61 -44
- package/dist/index.d.cts +9 -3
- package/dist/index.d.ts +9 -3
- package/dist/index.js +61 -44
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,74 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.4.3
|
|
4
|
+
|
|
5
|
+
This release spares more full runs of `test-gates mutation`: `--affected` keeps telling the gates apart where only some of them read files, a new linked worktree starts from the record of the main worktree, and timeouts on a loaded machine are run again before they fail the run. Three things to know when updating:
|
|
6
|
+
|
|
7
|
+
- The record of a run has a new format (3: it says for each gate whether it is blind), and a record of an older format is not read. So the first run after the update is one of every mutant.
|
|
8
|
+
- A new linked worktree that has no record of its own starts from the incremental file and the record of the main worktree. Every file that differs from the main worktree counts as changed.
|
|
9
|
+
- Before it starts Stryker, `test-gates mutation` removes the sandboxes an interrupted run left in the subproject. A Stryker started by hand has no marker of a run, so if one runs in the same subproject at the same time, its sandbox is removed.
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- A new linked worktree (`git worktree add`) that has neither an incremental file nor a record starts from those of the main worktree, at the same path. The incremental file is copied into place, and the record goes through the usual comparison, so a file that differs here (another branch, a change) counts as changed. The files of the main worktree are never written. The first line of the run says so, and `test-gates doctor` says beforehand whether the next run starts from the main worktree. Nothing is taken over with `--gate` or `--force`, or inside a Stryker sandbox. A blind gate (see below) never takes its results over from the main worktree, since a file git ignores, such as an `.env`, is not compared and differs between worktrees as a rule: with `--affected` it is run again, and without `--affected` the run is one of every mutant.
|
|
14
|
+
- `test-gates mutation` runs again, once and one at a time (`--mutate` with their ranges, `--concurrency 1`, `--force`), the mutants that timed out outside `expectedTimeouts` when there are more of them than `settings.mutation.maxTimeouts` allows. Their new result takes the place of the timeout before the report is judged: killed is killed, a survivor fails the run, and a timeout again fails as before. In incremental mode the next run takes over the new result. If the run again fails or writes no report, the first run is judged, and the output says so. This is done in CI too. `mutation-result` runs nothing again.
|
|
15
|
+
- `test-gates mutation` removes the sandboxes (`<tempDirName>/sandbox-*`) that an interrupted Stryker run left in the subproject, before it starts Stryker. Tools that walk the subproject no longer find a second copy of every file. `backup-*` of `inPlace` is never removed. Nothing is removed when the temp directory is outside the subproject, when `cleanTempDir` is `false`, or while another `test-gates mutation` runs there: each run keeps a marker with its pid and the start time of its process while Stryker runs. `test-gates doctor` notes the sandboxes that are left, and whether the next run removes them.
|
|
16
|
+
|
|
17
|
+
### Changed
|
|
18
|
+
|
|
19
|
+
- `mutation --affected` runs again only the gates that cannot be told apart. A file that reads files no import names (`readFileSync`, `readFile`, a glob, jsdom `fromFile` / `fromURL`, a path from `__dirname` handed to a reading function or to a package such as `dotenv`) now makes only the gates that depend on it blind. A blind gate runs again after a change of any compared file and keeps its results when nothing changed; the other gates are still told apart. What may load modules that are not followed still runs every gate. `test-gates doctor` names the blind gates in a note.
|
|
20
|
+
- `mutation --affected` takes a `.html` or `.htm` file for a document, as the comparison does since 0.4.2, so a page that no gate depends on (the HTML report of Stryker, a page of the docs) no longer runs every mutant. `index.html` and `index.htm` still do, since Vite starts from one. To keep this safe:
|
|
21
|
+
- a path from the top of the subproject to a `.html`, `.htm`, `.md` or `.json` file, written as text (`'src/page.html'`), is now a file of the one that names it;
|
|
22
|
+
- `readFile` and jsdom `fromFile` / `fromURL` count as reading files, as `readFileSync` does;
|
|
23
|
+
- the query of a Vite import (`?raw`, `?url`, `?inline`, `?worker`) is cut off before the file is looked up, so `import page from './page.html?raw'` leads to `page.html`.
|
|
24
|
+
- `test-gates doctor` reads `case "$var" in` and `for var in a b; do … done` in a hook. A hook that loops over the subprojects and gives `--affected` to some of them only is no longer reported as running every subproject without it. A pattern that may stand for other names (`*`, `app*`, `{a,b}`) still stands for any subproject, and a hook that cannot be read this way is read as before.
|
|
25
|
+
- `selfcheck --jobs` shares the cores between the runs at the same time: each one gets `--maxWorkers=max(1, floor(cores / runs))` of Jest or Vitest. Nothing is added when the gate command or the config of the runner already sets the workers. `--jobs 1` is unchanged.
|
|
26
|
+
- `selfcheck --jobs` no longer takes the text of a string in quotes (`'retry: {count}'` in a file of messages) for a use that keeps a gate out of the runs at the same time. Template literals, and strings that hold `${`, still count.
|
|
27
|
+
|
|
28
|
+
### Fixed
|
|
29
|
+
|
|
30
|
+
- `selfcheck --jobs` reported "N of N negative control(s) did not hold cleanly next to others", and ran every control on its own, when another `selfcheck --jobs` on the same machine removed its temporary directory at the wrong moment. The tidying of old directories now passes over an entry it cannot look at. When no directory for the runs can be made, the output now says so. The same fix applies to the shared run of `--quick`.
|
|
31
|
+
|
|
32
|
+
## 0.4.2
|
|
33
|
+
|
|
34
|
+
Most of this release closes cases where `test-gates mutation` took over the results of the last run although a file the run depends on had changed, with `--affected` and without it. The rest spares full runs that were not needed. Two things to know when updating: the first run after it is one of every mutant, since the record holds the version of test-gates; and in a project whose runner config maps modules in a way that `settings.importAliases` does not name, `--affected` no longer tells the gates apart until the aliases are added (`test-gates doctor` names them).
|
|
35
|
+
|
|
36
|
+
### Fixed
|
|
37
|
+
|
|
38
|
+
These made a run take over results that a change could have turned into survivors.
|
|
39
|
+
|
|
40
|
+
- `mutation --affected` kept the results of a gate after a change of a file it depends on:
|
|
41
|
+
- An import put together from strings (`import('./locales/' + lang + '.ts')`, `require('./rules/' + lang)`) was not followed at all. Any `require()` / `import()` whose argument is not one string literal is now one whose path is not written out, and what the gates depend on is unknown. Allowlist mode of `test-gates check` now rejects such an import too.
|
|
42
|
+
- With `x.ts` and `x.js` next to each other, `./x` was taken for `x.ts` alone while the runner may load `x.js`. Every script that exists for a path is now followed (also by the comparison and by `selfcheck --jobs`).
|
|
43
|
+
- Module maps of the runner (`resolve.alias` / `test.alias` of Vite, `moduleNameMapper` of Jest, also in package.json, `jsc.paths` of `.swcrc`, tsconfig `paths` for Vitest, Babel `module-resolver`) can send an import somewhere else than the path says. Such a map now makes what the gates depend on unknown, unless it does what an alias of `settings.importAliases` does, is a `moduleNameMapper` entry to one fixed module (a style stub), or is a `paths` entry of one name that leads to an installed package. The reason names every such map and the alias to add; `test-gates doctor` shows it in its note.
|
|
44
|
+
- Where the run drops imports of types, a spec with `import type { Rate } from './rate'` and `jest.createMockFromModule<Rate>('./rate')` left `./rate` out of what the gate loads. `createMockFromModule` and `genMockFromModule` are now followed, also with a type argument.
|
|
45
|
+
- `test-gates mutation` (without `--affected` too) reused the results after a change in files the comparison did not reach:
|
|
46
|
+
- A directory whose name holds glob characters (`shared/[rates]`, a `[locale]` of Next.js) was listed as empty by git and dropped out of the comparison. The path handed to git is now escaped.
|
|
47
|
+
- A plain `..` path that exists only from the subproject, where the run starts (`readFileSync('../shared-data/x.json')` in a spec, `path.resolve('../…')` in a setup file), was resolved from the directory of the file only. It is now resolved from the subproject as well. One from the subproject that lies outside of what can be listed is refused, and one that ends in a directory holding the repository makes every file of the repository part of the comparison.
|
|
48
|
+
- A file git ignores that the run loads (generated code, downloaded fixtures a spec imports) was not compared. With the helper's runner config it is now hashed and its paths followed.
|
|
49
|
+
- `patches/` of patch-package, in the subproject and in the directories above it, is compared.
|
|
50
|
+
- The node on PATH, which the shebang of `node_modules/.bin/stryker` starts, is recorded as `Node.js on PATH`; a change of it runs every mutant.
|
|
51
|
+
- A path in a file above the subproject that the comparison reaches (`"extends": "./configs/strict.json"` in a base tsconfig at the top of the repository) is followed by the same rules as a path in the subproject.
|
|
52
|
+
|
|
53
|
+
### Changed
|
|
54
|
+
|
|
55
|
+
These spare full runs, or runs of more gates, that were not needed.
|
|
56
|
+
|
|
57
|
+
- `mutation --affected` gives up telling the gates apart in fewer cases:
|
|
58
|
+
- An import of a module that only has a declaration file (`@/types/schema` for `src/types/schema.d.ts`, `index.d.ts` of a directory) leads to that file. A script of the same name still comes first.
|
|
59
|
+
- An import of types only (`import type`, `export type … from`, an import whose names are all marked `type`) is left out of what a gate depends on, but only where the run is known not to check types: no checkers in the Stryker config, and Vitest without `typecheck`, or Jest with ts-jest and `isolatedModules: true` or `diagnostics: false`, or with `@swc/jest`, `babel-jest`, `esbuild-jest` or `@sucrase/jest` and neither ts-jest nor a preset. `import { type A } from './a'` is left out only where no compiler config keeps imports as written (`verbatimModuleSyntax`, `preserveValueImports`, `importsNotUsedAsValues`, `onlyRemoveTypeImports`). Anything else keeps the import.
|
|
60
|
+
- A file other than a script that a runner config names (`coverage.exclude: ['src/components/Foo.vue']`, a JSON stub of `moduleNameMapper`) is taken as a config by itself: a change of it runs every gate again, but a component named there no longer makes `--affected` refuse.
|
|
61
|
+
- A file that uses `__dirname`, `__filename` or `import.meta.url` / `dirname` / `filename` only to find where it lies keeps the gates apart. It is refused only when it may also read by that path: a call that reads or looks at a file or a directory, a mock named by a computed path, a `Worker`, or an import of a package or built-in module other than `path`, `url`, `vitest` and `@jest/globals`.
|
|
62
|
+
- `require('./rates')` of `rates.json` leads to the JSON file (also `<dir>/index.json`), after the scripts, as Node, Jest and Vite do.
|
|
63
|
+
- An import or a word such as `readFileSync` in a comment no longer counts. A relative path in quotes is still followed wherever it is written.
|
|
64
|
+
- A mock in a `__mocks__` next to a module (`src/__mocks__/dep.ts`) is loaded with that module, so a change of it runs again only the gates that load the module. The `__mocks__` at the top of the subproject, and a mock named as an installed package or a built-in module, are still configs.
|
|
65
|
+
- A run without `--affected` writes what each gate depends on too, whenever it writes a record. Before, the next `--affected` run after a run without it, or after `-- --force`, had nothing to go on and ran every mutant. Runs with `--gate` and runs that fail still write none.
|
|
66
|
+
- A mutant that timed out in the last run and that an `expectedTimeouts` entry of its gate points at is taken over, with and without `--affected`. Before, such mutants that never end were run again on every run. Any other timeout is still run again.
|
|
67
|
+
- A plain path to a file directly in a directory above the subproject (`"extends": "../../tsconfig.base.json"`) no longer makes every file of the repository part of the comparison: that file is compared anyway. A path to a directory above, or to a missing file there, still does.
|
|
68
|
+
- The files of Stryker's reporters (`htmlReporter`, `jsonReporter`, or their defaults) and the files that `ignorePatterns` leaves out (`.eslintcache`, `tsconfig.tsbuildinfo`) are no longer taken for files the run reads. A new report or cache no longer runs every mutant, and the paths in an HTML report no longer make every file of the repository part of the comparison. Where a pattern of `ignorePatterns` starts with `!`, nothing is left out by it. The paths in `.html` files are not followed.
|
|
69
|
+
- `test-gates check` keeps a `..` of an import that leaves the subproject (`resolveSpecifier`): `../../shared/x` from `src/a.ts` is `../shared/x`, not `shared/x`. An import outside is judged by the path it points at, a mock of a module outside is no longer taken for a mock of the gate, and `selfcheck` no longer lists a file that imports a module outside as an importer of a gate of the same path inside. A root-anchored `impurePaths` rule (`^shared/`) no longer matches an import outside.
|
|
70
|
+
- The comparison and the work of `--affected` before a run are faster: the files of each path are looked up once in a run (on this repository the snapshot went from about 140 to 90 ms).
|
|
71
|
+
|
|
3
72
|
## 0.4.1
|
|
4
73
|
|
|
5
74
|
### Fixed
|
package/README.md
CHANGED
|
@@ -256,35 +256,57 @@ What is left of incremental mode is the run after no change: a push that does no
|
|
|
256
256
|
What is compared:
|
|
257
257
|
|
|
258
258
|
- Every file of the subproject: gates, specs, other sources, helpers of the specs, configs, `test-gates.json`, the lock file.
|
|
259
|
-
- The files directly in the directories above the subproject, up to the root of the repository: the lock file and the shared configs of a monorepo.
|
|
260
|
-
- Packages of the same repository that are linked into `node_modules`, and directories outside the subproject that a file leads to, with a link or with a relative path (`../../pricing-kit/rounding/half-up`). A path that starts at the directory of its own file is followed as well, whether it is written as a template (`` `${__dirname}/../pricing-kit` ``) or as parts handed to `path.join` or `path.resolve` (`__dirname, '..', 'pricing-kit'`, also with `import.meta.dirname`). A path that leads back into the subproject adds nothing.
|
|
261
|
-
- When a path cannot be followed, every file of the repository is compared: when its start is not known (`<rootDir>/../pricing-kit`, an alias such as `@/../shared`, `'..'` after anything but the directory of the file), when a part after it is not known (`join(__dirname, '..', name)`, `join(resolve(__dirname, '..'), name)`, a way up after a placeholder), and when a plain relative path leads to a
|
|
262
|
-
- Files written for people to read (`.md`, `.markdown`, `.txt`, `.rst`, `.adoc
|
|
263
|
-
- When the config of the test runner is made with this package's helper (`createJestGatesConfig` or `createVitestGatesConfig`, which run the specs of the gates and no others), only the files of the subproject that the run loads are searched: the Stryker config, the runner config and the files it names (a setup file), the gates and their specs, `test-gates.json`, `tsconfig*.json`, `jsconfig*.json`, the Babel and SWC configs, the files below a root `__mocks__/` and the mock next to a loaded file, what those configs name (in a script, any path as text, such as a setup file; in `tsconfig.json` or `.babelrc`, what it extends and the plugins and presets it names, and what those name in turn), and everything those import, also what a component (`.vue`, `.svelte`, `.astro`, `.mdx`) imports. The other files of the subproject are compared, but a path in them is not followed: `package.json` (its scripts, and a runner config in it that the helper's config replaces), a lint config, the settings of an editor, the config of another test run, a component no gate loads. An import of an alias that `settings.importAliases` does not name (one that the runner config maps) loads every file of the subproject whose path ends as the import does (`@shared/money/fee` loads `src/money/fee.ts`); when none does, it leads out of the subproject, to where the config that maps it points. An import of an alias with the extension of the compiled file (`@shared/fee.js`) leads to `fee.ts` as well, and one of a single part (`@helper`, `#config`) to the files of that name. Every file is searched when a loaded file imports a module whose path is not written out, when the runner config is not made with the helper, when what the runner config exports is not what the helper returns (merged with another config after it, changed after the call), and when the runner config or a file it is put together from names `projects` or `workspace` outside of a comment: then it may run more than the specs of the gates. Another config spread into the helper's options is fine: the helper sets the specs after them.
|
|
259
|
+
- The files directly in the directories above the subproject, up to the root of the repository: the lock file and the shared configs of a monorepo. Also the files in a `patches/` directory of the subproject or of a directory above it (patch-package).
|
|
260
|
+
- Packages of the same repository that are linked into `node_modules`, and directories outside the subproject that a file leads to, with a link or with a relative path (`../../pricing-kit/rounding/half-up`). A path that starts at the directory of its own file is followed as well, whether it is written as a template (`` `${__dirname}/../pricing-kit` ``) or as parts handed to `path.join` or `path.resolve` (`__dirname, '..', 'pricing-kit'`, also with `import.meta.dirname`). A plain relative path is taken both from the directory of its file (an import) and from the subproject, where the run starts (`readFileSync`, `path.resolve`), wherever something is there: `readFileSync('../shared-data/rates.json')` in `src/fee.spec.ts` reads the `shared-data` next to the subproject. Where what is there from the subproject lies outside the repository, reuse is off, as for any path there; where it is a directory that holds the repository (a way up to the top in a nested file), every file of the repository is compared. When nothing is there from either, a file that appears later at one of them makes the run a full one. A path that leads back into the subproject adds nothing, and so does a plain path to a file directly in a directory above the subproject (`"extends": "../../tsconfig.base.json"`): that file is compared anyway.
|
|
261
|
+
- When a path cannot be followed, every file of the repository is compared: when its start is not known (`<rootDir>/../pricing-kit`, an alias such as `@/../shared`, `'..'` after anything but the directory of the file), when a part after it is not known (`join(__dirname, '..', name)`, `join(resolve(__dirname, '..'), name)`, a way up after a placeholder), and when a plain relative path leads to a directory above the subproject (`'..'`, `'../..'`), or to a file in one that is not there (`'../tsconfig.missing.json'`).
|
|
262
|
+
- Files written for people to read (`.md`, `.markdown`, `.txt`, `.rst`, `.adoc`, `.html`, such as the report of Stryker) are compared, but the paths in them are not followed. Every other file is searched, whatever its extension.
|
|
263
|
+
- When the config of the test runner is made with this package's helper (`createJestGatesConfig` or `createVitestGatesConfig`, which run the specs of the gates and no others), only the files of the subproject that the run loads are searched: the Stryker config, the runner config and the files it names (a setup file), the gates and their specs, `test-gates.json`, `tsconfig*.json`, `jsconfig*.json`, the Babel and SWC configs, the files below a root `__mocks__/` and the mock next to a loaded file, what those configs name (in a script, any path as text, such as a setup file; in `tsconfig.json` or `.babelrc`, what it extends and the plugins and presets it names, and what those name in turn), and everything those import, also what a component (`.vue`, `.svelte`, `.astro`, `.mdx`) imports. The other files of the subproject are compared, but a path in them is not followed: `package.json` (its scripts, and a runner config in it that the helper's config replaces), a lint config, the settings of an editor, the config of another test run, a component no gate loads. An import of an alias that `settings.importAliases` does not name (one that the runner config maps) loads every file of the subproject whose path ends as the import does (`@shared/money/fee` loads `src/money/fee.ts`); when none does, it leads out of the subproject, to where the config that maps it points. An import of an alias with the extension of the compiled file (`@shared/fee.js`) leads to `fee.ts` as well, and one of a single part (`@helper`, `#config`) to the files of that name. Every file is searched when a loaded file imports a module whose path is not written out, when the runner config is not made with the helper, when what the runner config exports is not what the helper returns (merged with another config after it, changed after the call), and when the runner config or a file it is put together from names `projects` or `workspace` outside of a comment: then it may run more than the specs of the gates. Another config spread into the helper's options is fine: the helper sets the specs after them. A file the run loads in this way is compared also when git ignores it (generated code, fixtures that are downloaded), and the paths in it are followed. The reports of Stryker (the files of its HTML and JSON reporters, `reports/mutation/mutation.html` and `reports/mutation/mutation.json` unless the config names others) and what its `ignorePatterns` leave out (`.eslintcache`, `tsconfig.tsbuildinfo`; nothing, where one of them takes files back with `!`) are not taken for files the run loads, though the Stryker config names them; the reports are not compared at all.
|
|
264
264
|
- A path that leads outside of the repository to nothing is compared as missing: a path from a config that is meant for the sandbox of Stryker (`path.resolve(__dirname, '../../../shared')`, which from the subproject itself leads above the repository). Once something is there, reuse is off, as for any path outside the repository.
|
|
265
|
-
- What is installed (`node_modules/.package-lock.json`, or what yarn and pnpm write in its place), the versions of Node.js, of this package, of Stryker and of the test runners, and the arguments handed to Stryker.
|
|
265
|
+
- What is installed (`node_modules/.package-lock.json`, or what yarn and pnpm write in its place), the versions of Node.js (the one this tool runs on, and the one on `PATH`, which starts `node_modules/.bin/stryker`), of this package, of Stryker and of the test runners, and the arguments handed to Stryker.
|
|
266
266
|
- The incremental file itself: it has to be the one the recorded run wrote.
|
|
267
267
|
|
|
268
|
-
With git, "every file" is what git tracks plus the untracked files it does not ignore. In a git hook of a linked worktree, git sets `GIT_DIR` and not `GIT_WORK_TREE`; then git would take the subproject for the top of the repository, so test-gates calls git without `GIT_DIR` in that case and lets it find the repository from the subproject. Without git, or when git ignores the subproject, every file is read except `node_modules`, `.git` and what the run writes itself, up to the nearest directory that has a `.git` entry, else the topmost one that declares workspaces, else only the subproject. Such a listing also contains what git would ignore (coverage output, an HTML report, logs), so a file that changes between two runs makes every run a full one; reuse is meant for a checkout with git.
|
|
268
|
+
With git, "every file" is what git tracks plus the untracked files it does not ignore, and, with the helper's runner config, the files git ignores that the run loads. In a git hook of a linked worktree, git sets `GIT_DIR` and not `GIT_WORK_TREE`; then git would take the subproject for the top of the repository, so test-gates calls git without `GIT_DIR` in that case and lets it find the repository from the subproject. Without git, or when git ignores the subproject, every file is read except `node_modules`, `.git` and what the run writes itself, up to the nearest directory that has a `.git` entry, else the topmost one that declares workspaces, else only the subproject. Such a listing also contains what git would ignore (coverage output, an HTML report, logs), so a file that changes between two runs makes every run a full one; reuse is meant for a checkout with git.
|
|
269
269
|
|
|
270
270
|
The record is removed before each run and written only after a run that Stryker ended without an error and in which it wrote the incremental file, so a run that failed, was interrupted or was a dry run is followed by a full one. A run that is given `--force` is not changed. When Stryker does not run in incremental mode, nothing is recorded or compared.
|
|
271
271
|
|
|
272
|
-
|
|
272
|
+
**A new linked worktree** (`git worktree add`) has neither an incremental file nor a record, since git ignores both. When it has neither, `test-gates mutation` starts from those of the main worktree, so that the first run there need not be one of every mutant. The incremental file of the main worktree is copied into place, and its record is compared with the files of the new worktree as if it were its own: a file that differs (another branch, a change) counts as changed, with or without `--affected`. The files of the main worktree are never written; the run writes its own record, and the next one uses that. The first line of the run says so:
|
|
273
|
+
|
|
274
|
+
```
|
|
275
|
+
test-gates mutation: this worktree has no record of its own, so it starts from the one of the main worktree (/work/shop)
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
Nothing is taken from the main worktree with `--gate`, with `--force` (or `force` in the Stryker config), or inside a Stryker sandbox; other linked worktrees are not looked at. Only the files git lists are compared, and a file git ignores that a blind gate reads (an `.env` read with `readFileSync`, see `--affected` below) is not, while each worktree has its own as a rule. So a blind gate never takes its results over from the main worktree: with `--affected` it is run again, and the others are told apart as usual. Without `--affected`, a blind gate makes the run one of every mutant, and so does a project where the gates cannot be told apart (a test runner `--affected` does not know, for one). `test-gates doctor` says beforehand whether the next run starts from the main worktree.
|
|
279
|
+
|
|
280
|
+
The paths that leave the subproject are found by text, in every file that is compared but those written for people and, with the helper's runner config, those of the subproject that the run does not load. The search errs towards comparing too much: text that merely looks like such a path widens the comparison, and a path or a link to a directory that exists outside the repository turns reuse off, with the file and the path as it is written named in the message. There are two exceptions, where the path is known: a path from the directory of the file that ends in a directory above the subproject (`path.resolve(__dirname, '..')` in a file at the top of the subproject) adds only the files directly in that directory, and a plain path to a file that is directly in such a directory (`'../../tsconfig.base.json'`) adds only that file, and what that file points at with a path from its own directory (`"extends": "./configs/strict.json"`) or with `..`, followed in the same way. Both are compared anyway (see the last two points of the list below).
|
|
273
281
|
|
|
274
282
|
The comparison loads the Stryker config in a process of its own, to see where the incremental file is, and reads and hashes those files before each run. Timed with a stand-in for Stryker, so that only this tool is measured, in a project of 5,200 files written for the measurement: 0.13 s without the comparison, 0.44 to 0.55 s with it (medians of three rounds; other jobs were running on the machine).
|
|
275
283
|
|
|
276
284
|
What it does not see, and what therefore still needs `-- --force` by hand, or CI, where the run should always be a full one:
|
|
277
285
|
|
|
278
|
-
- files git ignores (build output that a spec or a config loads
|
|
286
|
+
- files git ignores that the run reads without importing them (`.env` read by `dotenv`, a JSON file read with `readFileSync`), and every file git ignores when the runner config is not made with the helper (build output that a spec or a config loads),
|
|
279
287
|
- environment variables, the clock, the network, and a Stryker config that reads them or the command line to decide about incremental mode,
|
|
280
288
|
- files outside the subproject that are reached without a link and without `..` in a path: an absolute path, an alias defined in a config above the subproject (in that case every file of the repository is compared only if the subproject points at that config with a relative path),
|
|
281
289
|
- changes inside `node_modules` that leave the files of the package manager as they were,
|
|
282
|
-
- paths in files written for people (`.md`, `.txt` and the like): a script that reads a path out of a README,
|
|
290
|
+
- paths in files written for people (`.md`, `.txt`, `.html` and the like): a script that reads a path out of a README,
|
|
283
291
|
- with the helper's runner config, paths in a file of the subproject that the run does not load as a module: a file that a loaded one reads as data (`readFileSync`) and takes a path out of, and a config that a package reads by its name and that is not among those listed above (`next.config.js` read by `next/jest`, for example, or a Babel config kept in the `babel` key of `package.json`),
|
|
284
292
|
- with the helper's runner config, what a component uses without importing it (a component that a plugin registers), and the paths in it,
|
|
285
293
|
- a path outside the repository to nothing, to which something is added that starts with the same text and continues the last part of its name (`'../../../shared' + '-kit'`),
|
|
286
294
|
- what is read below a directory above the subproject when the path to that directory is kept in a variable and the rest is added elsewhere: `const top = path.resolve(__dirname, '..')` in one place and `path.join(top, 'shared', name)` in another, or a path to such a directory handed to a tool that reads below it.
|
|
287
295
|
|
|
296
|
+
#### Sandboxes left by an interrupted run
|
|
297
|
+
|
|
298
|
+
Stryker runs the tests in copies of the subproject (`.stryker-tmp/sandbox-*`, below its `tempDirName`) and removes them at the end. A run that is interrupted (Ctrl-C, a killed process) leaves them, and every tool that walks the subproject finds a copy of each file in them: a test runner, a linter, a script. So before Stryker starts, `test-gates mutation` removes every `sandbox-*` there and says how many. While Stryker runs, it keeps a marker `test-gates-run-<pid>` in the same directory, with the pid of its process and when that process started, and removes it at the end.
|
|
299
|
+
|
|
300
|
+
It removes nothing:
|
|
301
|
+
|
|
302
|
+
- when the Stryker config sets `cleanTempDir: false`: the sandboxes are kept on purpose,
|
|
303
|
+
- when the temp directory is not a directory inside the subproject (a `tempDirName` outside it, or a link out of it),
|
|
304
|
+
- while another run of `test-gates mutation` is going on in the subproject: a marker whose process is still there, and started when the marker says (within a few seconds: on Linux two prints of the same start can differ by a second). The names of the sandboxes are random, so which one belongs to that run cannot be told, and none is removed. A process that has the pid of a marker but started at another time is not that run; where the start cannot be read (no `ps`), a process with the pid is taken for the run.
|
|
305
|
+
|
|
306
|
+
**A Stryker started by hand (`npx stryker run`) has no marker.** If `test-gates mutation` starts in the same subproject while it runs, it removes the sandbox of that Stryker, which then fails. Do not run the two at the same time; they would also write the same report and incremental file.
|
|
307
|
+
|
|
308
|
+
`test-gates doctor` notes sandboxes that are left: that the next run removes them, that they are in use by a run going on (with its pid), or, with `cleanTempDir: false` or a temp directory outside the subproject, that they have to be removed by hand.
|
|
309
|
+
|
|
288
310
|
#### `--gate <path>`: only the mutants of one gate
|
|
289
311
|
|
|
290
312
|
While working on one gate, `test-gates mutation --gate src/pricing/shipping-fee.ts` runs the mutants of that gate only (it hands the path to Stryker as `--mutate`) and judges that gate only. The option can be given more than once; the path is written as in `test-gates.json`.
|
|
@@ -311,16 +333,26 @@ The results of a gate depend on
|
|
|
311
333
|
- its own spec, every spec that imports it (directly or through other files), and every spec whose tests covered or killed one of its mutants in the last run, with every file those specs import. A spec that does not load the gate cannot see its mutants: Jest and Vitest give each spec file a module registry of its own. This is the way a change reaches another gate: when the spec of `next` checks its result through the gate `same`, a change of `same` runs `next` again, and so does a weakened spec of `next` for `same`,
|
|
312
334
|
- snapshots next to a spec (`__snapshots__/<spec>.snap`) and data a spec or a gate imports.
|
|
313
335
|
|
|
314
|
-
Imports are followed when they are relative, absolute, or start with a prefix of `settings.importAliases
|
|
336
|
+
Imports are followed when they are relative, absolute, or start with a prefix of `settings.importAliases`. An import of types only (`import type`, and `import { type Rate }` where a `tsconfig.json` tells that such imports are dropped: no `verbatimModuleSyntax` or `preserveValueImports` in it or in what it extends) counts where the run may check types: with a checker in the Stryker config, with the typecheck of Vitest, with ts-jest that reports type errors, and where that cannot be told. Where the run only strips types (Vitest, `@swc/jest`, `babel-jest`, ts-jest with `isolatedModules: true` or `diagnostics: false`), it does not count. Paths written as text for the runner (`vi.mock('./rates')`, `jest.createMockFromModule('./rates')`) are followed as well. The query of an import of Vite is not a part of the path: `import page from './page.html?raw'` (also `?url`, `?inline`, `?worker`) leads to `page.html`. A path from the top of the subproject to a page, a document or data, written as text (`'src/page.html'`, `'docs/rates.md'`, `'fixtures/rates.json'`), is followed where such a file is there: a test may read it by that path. A path without an extension leads to the scripts of that name, and to a JSON file of that name (`require('./rates')` to `rates.json`); where there is no script of that name, to its declaration file (`schema.d.ts` for `./schema`), which counts as a config (see below). An import or a name such as `readFileSync` in a comment does not count, but a relative path in quotes is followed wherever it is written, except in an import of types only that does not count. A package is taken to be what the lock file and the versions of the tools say it is, unless it is a link to a package of the same repository: see below.
|
|
315
337
|
|
|
316
338
|
A change elsewhere decides for every gate:
|
|
317
339
|
|
|
318
|
-
- **Every mutant is run** after a change of a config (the Stryker config and the config of the test runner, what they import, the setup files and other files they name, `package.json`, `test-gates.json`, `tsconfig*.json`, `*.config.*`, Babel configs, dotfiles, declaration files `.d.ts`, anything below `__mocks__`), of a lock file or of what is installed, of the tools or of the arguments; after a change of a file at the top of the subproject or outside of it that no gate depends on;
|
|
319
|
-
- **Nothing is run again** for a change of a source, data, a snapshot, a style or an image below the subproject that no gate depends on, or of a document (`.md`, `.txt` and the like).
|
|
340
|
+
- **Every mutant is run** after a change of a config (the Stryker config and the config of the test runner, what they import, the setup files and other files they name, `package.json`, `test-gates.json`, `tsconfig*.json`, `*.config.*`, Babel configs, dotfiles, declaration files `.d.ts`, anything below the `__mocks__` at the top of the subproject, and a mock below another `__mocks__` that is named as a package or a builtin module, such as `src/__mocks__/axios.ts`, which Jest can take for the package in every spec), of a lock file or of what is installed, of the tools or of the arguments; after a change of a file at the top of the subproject or outside of it that no gate depends on; after a change of a file whose kind is not known (`.yaml`, `.csv` and the like) that no gate depends on; and after a change of an `index.html` or `index.htm` at any depth that no gate depends on, since a build tool or a test runner may start from a page of that name (Vite, Vitest in a browser). This goes by the name alone: an `index.html` in a report that git does not ignore (`coverage/index.html`) runs every mutant too, and another entry page (`admin.html` of a site of several pages) is a document unless a config names it. Such a file that a gate does depend on runs that gate again, as any other file does. A mock next to a module (`src/__mocks__/rates.ts` for `src/rates.ts`, which `jest.mock('./rates')` and `vi.mock('./rates')` without a factory take) is loaded with that module: a change of it runs again the gates that load the module. One that no gate loads, or one that is gone, runs every mutant.
|
|
341
|
+
- **Nothing is run again** for a change of a source, data, a snapshot, a style or an image below the subproject that no gate depends on, or of a document (`.md`, `.txt`, `.html`, `.htm` and the like, such as a page of the docs or an HTML report the run does not write itself). A document that a gate or a spec imports runs that gate again; a page that the runner config names (`browser.testerHtmlPath`) is a config.
|
|
342
|
+
|
|
343
|
+
When what the gates depend on cannot be told from the text, the run falls back in one of two ways, and the first lines say why.
|
|
320
344
|
|
|
321
|
-
|
|
345
|
+
A file that only reads files no import names makes the gates that depend on it **blind**: a file read at run time (`node:fs`, `node:fs/promises`, `readFileSync`, `readFile`, `fromFile` and `fromURL` of jsdom, `readdirSync`, `toMatchFileSnapshot`, `glob`), or a file that finds where it lies (`__dirname`, `import.meta.url`) and then may read by that path (it calls a function such as `existsSync` or `readFile`, or imports a package other than `path`, `url` and the test runner's, such as `dotenv`: a package is not looked into, so it is taken to read by the path; finding where it lies alone is fine). A gate depends on the file when the gate, a spec of it or what they import loads it. A blind gate is run again after a change of any compared file, and keeps its results only when nothing changed; the other gates are still told apart by what they load. In a subproject where only a few gates load a module that reads a `.env` by a path from `__dirname`, a change of a source runs those few again next to the gates it affects, not every gate (a change of the `.env` itself, a file whose name starts with a dot, still runs every mutant). The first lines count the blind gates that run again, with why:
|
|
322
346
|
|
|
323
|
-
|
|
347
|
+
```text
|
|
348
|
+
test-gates mutation --affected: 1 of 4 gate(s) reused, 3 run again
|
|
349
|
+
src/next.ts: changed src/next.ts
|
|
350
|
+
2 gate(s) run again for any change, as what they depend on cannot be told: src/settings.ts uses readFileSync, which can reach files that no import names
|
|
351
|
+
```
|
|
352
|
+
|
|
353
|
+
What may load modules that are not followed, or what concerns every spec, keeps every gate from being told apart: a change runs every mutant. After no change at all the results are reused as they are without the option, so `--affected` is never slower than a run without it. A spec that may load any module may load any gate, so the gates it sees cannot be told. That is the case for an import whose path is not written out (`import(name)`, `require(name)`, strings put together as in `import('./locales/' + lang)`, a template with a value in it, a second argument), an import that leads to no file, a specifier that is neither a file nor an installed package (an alias defined only in a config: name it in `settings.importAliases`), `process.cwd`, `createRequire`, `require.context`, `require.resolve`, `import.meta.glob` and `import.meta.resolve`, an import of `child_process`, `module` or `worker_threads`, a file that finds where it lies and names a mock or starts a worker by that path, a file that is loaded but not compared (git ignores it, and the comparison does not find that the run loads it: see above; a blind gate would not see its change either), a setup file that reads files no import names or loads a gate for every spec, a package of the same repository linked into `node_modules` (`@acme/shared`, also when it lies in the subproject), a component (`.vue`, `.svelte`, `.astro`, `.mdx`), which can use others that it does not import (registered by a plugin), a map of modules that `settings.importAliases` does not name (`resolve.alias` or `test.alias` of Vite, `moduleNameMapper` of Jest, `jsc.paths` of `.swcrc`, and with Vitest tsconfig `paths` other than those of a prefix the runner config maps itself; a map may lead an installed package to a file of the project or give an import more than one target. One that maps the same prefix to the same directory as an alias of the settings, or a part of it, is fine: `'@': path.resolve(__dirname, 'src')`, `{ find: /^@\//, replacement: resolve(__dirname, 'src') + '/' }`, `'^@/(.*)$': '<rootDir>/src/$1'`, `"@/*": ["./src/*"]`. So is a `moduleNameMapper` entry to one fixed module such as a style stub, and a `paths` entry to an installed package. `test-gates doctor` names each map that is not fine, and the alias to add where there is one), a Babel config with `module-resolver`, and a test runner other than Jest and Vitest. Where it is not clear which of the two a cause is, it is taken for one of every gate. Where one name stands for both a script and a TypeScript file (`rate.js` and `rate.ts` for `./rate`), both are followed: runners do not agree on which comes first. A run without `--affected` writes down what each gate depends on as well, so a full run by hand (`-- --force`) does not make the next run with it a full one. Every mutant is also run when there is no record of the last run (a run with `--gate` leaves none), when the record cannot be read or does not say what each gate depends on (a run in CI writes no such part), with `--force`, and **when `CI` is set**: CI keeps running every mutant, and is the final check.
|
|
354
|
+
|
|
355
|
+
A gate with a survivor or a timeout that the allow lists do not cover is not reused: it runs again until it is clean. A mutant that timed out in the last run is not taken over either, also not in a gate whose other results are, and also not in a run without `--affected` after no change: on a loaded machine a mutant that would survive can time out instead, and a reused timeout would keep that. Those mutants alone are run again (`--mutate` with their ranges); the first lines count them. Only a timeout that an `expectedTimeouts` entry of its gate points at is taken over (matched as the verdict matches it): such a mutant times out on every run, and running it again would only cost the wait for its timeout. An entry says the mutant cannot end, so a listed mutant that would survive is kept as well: list only mutants that never end. The first lines count these apart (`2 mutant(s) that timed out in the last run are taken over: expectedTimeouts lists them`). The run is judged as one of every gate, and its first lines say what was reused:
|
|
324
356
|
|
|
325
357
|
```
|
|
326
358
|
test-gates mutation --affected: 1 of 3 gate(s) reused, 2 run again
|
|
@@ -347,7 +379,7 @@ problem: .husky/pre-push runs test-gates mutation without --affected: a change o
|
|
|
347
379
|
test-gates doctor: 1 problem(s) that make runs full ones
|
|
348
380
|
```
|
|
349
381
|
|
|
350
|
-
A **problem** makes runs full ones that need not be: Stryker not in incremental mode, a path that turns reuse off (one that leads outside of the repository to something, or a Stryker config that cannot be loaded), a path that makes every file of the repository part of the comparison (it names the file and the path), a Stryker config that sets `force`, a hook that runs `test-gates mutation` of the subproject without `--affected`, also through its `test:gates:mutation` script, and, when a hook relies on `--affected`, what keeps it from telling the gates apart (a
|
|
382
|
+
A **problem** makes runs full ones that need not be: Stryker not in incremental mode, a path that turns reuse off (one that leads outside of the repository to something, or a Stryker config that cannot be loaded), a path that makes every file of the repository part of the comparison (it names the file and the path), a Stryker config that sets `force`, a hook that runs `test-gates mutation` of the subproject without `--affected`, also through its `test:gates:mutation` script, and, when a hook relies on `--affected`, what keeps it from telling the gates apart (a spec that imports a module whose path is not written out, for one, or every gate blind). Where it cannot tell any of them apart, a hook without `--affected` is not taken for a problem, and what keeps it from doing so is a note. Blind gates next to others that are told apart are a note that names them, with why (`with --affected, a change of any file runs 2 of 35 gate(s) again (…)`). A run in a hook is taken to be one of the subproject that its line names (`--dir`, `--prefix`, `-w`, `cd`), else the nearest `cd` before it; a name with a variable in it (`cd "$project"` in a loop over the subprojects), or none, may be one of any subproject. In a branch of `case "$project" in`, and in `for project in fee-rules catalog-site; do … done`, the variable is one of the names written out there, so a hook can give `--affected` to some subprojects and not to others (`fee-rules|catalog-site) … -- --affected ;;`). A pattern or a word that may stand for others (`*`, `app*`, `{a,b}`, `"$@"`) leaves it any subproject, and so does a hook that cannot be read this way (a fall-through `;&`, a `case` without its `esac`); a `cd` in another branch of the same `case` is not taken for the nearest `cd` before the run, and the variable of a `cd` on a line before the run is narrowed only when that line is in the same branch. Each run is judged on its own line, or on its own branch where a line has more than one, and comments are left out. A **note** is worth knowing: that every file of the subproject is searched for paths, and why (a runner config that is not made with the helper, an import whose path is not written out), that no hook runs a mutation run, and that sandboxes are left in the temp directory (see [Sandboxes left by an interrupted run](#sandboxes-left-by-an-interrupted-run)). The exit code is 1 when there is a problem, so that the doctor can run in CI next to the other checks.
|
|
351
383
|
|
|
352
384
|
### `test-gates mutation-result [--report <path>]`
|
|
353
385
|
|
|
@@ -415,7 +447,16 @@ How an `expectedTimeouts` entry is judged:
|
|
|
415
447
|
|
|
416
448
|
`Killed` is accepted on purpose, unlike a stale `equivalentMutants` entry. A mutant that hangs is reported as `Timeout` when Stryker's timer fires first and as `Killed` when the test runner's own per-test timeout fires first; which one wins depends on the load. Failing on `Killed` would make the list flap between two runs of the same code. And nothing is hidden by accepting it: `Killed` is the best possible outcome.
|
|
417
449
|
|
|
418
|
-
|
|
450
|
+
**`test-gates mutation` runs the timeouts of the run once more, one at a time** (since 0.4.3). When more mutants timed out outside `expectedTimeouts` than `settings.mutation.maxTimeouts` allows, so that the run would fail on them, Stryker is started again for those mutants only (`--mutate` with their ranges, `--concurrency 1`, `--force`, after the arguments of the project), and each takes the result of that run before the report is judged:
|
|
451
|
+
|
|
452
|
+
```
|
|
453
|
+
test-gates mutation: 3 mutant(s) timed out outside expectedTimeouts, running them again one at a time (--concurrency 1)
|
|
454
|
+
test-gates mutation: 3 mutant(s) that timed out were run again one at a time (--concurrency 1): 3 Killed
|
|
455
|
+
```
|
|
456
|
+
|
|
457
|
+
A mutant that is killed in the run again counts as killed. One that survives there is a survivor and fails the run as any other; the run again hides nothing. One that times out again fails as before: there is only one run again, so a mutant that can never finish costs its timeout twice, and then belongs in `expectedTimeouts`. In incremental mode the run again also writes the incremental file, so the next run takes over the new result and not the old timeout. It runs in CI as well: a hosted runner with two cores times out under its own load too, and the run again only costs the time of the mutants that timed out. Within `maxTimeouts` nothing is run again: the run passes as it is, and a project that tolerates a few timeouts would otherwise pay for the run again every time. If the run again cannot finish, the first run is judged. `mutation-result` judges a report as it is and runs nothing again.
|
|
458
|
+
|
|
459
|
+
When the run still fails on a timeout, the mutant timed out also when it ran alone: look at it before anything else. Add an entry only for a mutant that can never finish.
|
|
419
460
|
|
|
420
461
|
`createStrykerGatesConfig` sets Stryker's `maxTestRunnerReuse` to 50 (since 0.3.2; Stryker's own default is 0, one test runner process for the whole run). A mutant that would be killed can also time out for a reason that has nothing to do with the mutant: in a Jest project with a few mutants that never end, the test runner handled about a third as many mutants per minute from the first of them on, for the rest of the run. Mutants that are killed in seconds on their own then timed out, a different set on each full run, and the run took about a quarter of an hour. With a new process every 50 mutants the speed stayed the same to the end, nothing timed out that should not, and the run took less than half as long. Why the runner slows down was not found; the restart only cuts it short. Pass `maxTestRunnerReuse` to the helper to change it.
|
|
421
462
|
|
|
@@ -495,7 +536,7 @@ Every control runs on its own, and the first line says why, when the gate comman
|
|
|
495
536
|
test-gates selfcheck: one negative control at a time (the gate config does not use createVitestGatesConfig)
|
|
496
537
|
```
|
|
497
538
|
|
|
498
|
-
This is a search by text: a comment that names `setTimeout` counts, and `new Date('2024-01-01')` does not. Which spec loads which gate is also read from the text: the imports, also those of a component (`.vue`, `.svelte`, `.astro`, `.mdx`), and paths handed to the runner (`vi.mock('./rates')`). A spec that loads modules by a name it computes, or by `import.meta.glob`, or that loads a component (which can use others that it does not import), may load any gate. A package that loads files of the project by itself (by a glob, for one) is not seen to load them.
|
|
539
|
+
This is a search by text: a comment that names `setTimeout` counts, and `new Date('2024-01-01')` does not. Nor does the text of a string in quotes, such as `'retry: {count}'` in a file of messages, when it is surely a string: it opens where a value starts, closes on its line, and a value may end after it. The text of a template (`` `retry: 1` ``) counts, since it may hold code, and so does a string in quotes that holds `${`, which may be part of a template. Which spec loads which gate is also read from the text: the imports, also those of a component (`.vue`, `.svelte`, `.astro`, `.mdx`), and paths handed to the runner (`vi.mock('./rates')`). A spec that loads modules by a name it computes, or by `import.meta.glob`, or that loads a component (which can use others that it does not import), may load any gate. A package that loads files of the project by itself (by a glob, for one) is not seen to load them.
|
|
499
540
|
|
|
500
541
|
_Of each run next to others_, the result is taken only when the control held and the runner itself sums the run up as undisturbed: every spec file but the one left out passed, every test passed and none was skipped (` Test Files 13 passed (13)` and `Tests 14 passed (14)` of Vitest, `Test Suites: 13 passed, 13 total` and `Tests: 14 passed, 14 total` of Jest), every threshold the run reports is that of this gate, the process ended by itself, and no other line speaks of a failure, a timeout, an error or a retry. Anything less (a test that timed out under the load, a reporter that prints no summary) and the control is run again on its own after the others have ended, and that run counts. So a control that does **not** hold is always reported from a run of its own, with the output of that run. The first line says how many controls were run again:
|
|
501
542
|
|
|
@@ -505,6 +546,12 @@ test-gates selfcheck: 4 of 14 negative control(s) did not hold cleanly next to o
|
|
|
505
546
|
|
|
506
547
|
Each of the `<n>` lines of runs has a directory of its own, outside of the project, for what the coverage tool writes and for the cache of the runner. It is given on the command line (`--coverage.reportsDirectory` and `--no-cache` for Vitest, `--coverageDirectory` and `--cacheDirectory` for Jest), which wins over the gate config, and removed when the command ends. Vitest keeps the coverage of every spec file in that directory until the run ends, and empties it when a run starts: two runs in one directory read each other's files and report a gate at 0% whose spec ran. That is why a gate command with any other argument is not run at the same time; it could name such a place itself.
|
|
507
548
|
|
|
549
|
+
The directories are made in the temporary directory of the system (`TMPDIR`). When none can be made there, every control runs one at a time, and the first line says why:
|
|
550
|
+
|
|
551
|
+
```
|
|
552
|
+
test-gates selfcheck: one negative control at a time (no directory for the runs next to others could be made in /tmp)
|
|
553
|
+
```
|
|
554
|
+
|
|
508
555
|
**Why the default stays 1.** What `--jobs` reports was compared with one control at a time in `test/selfcheck-runners-*.test.ts`, with a real Jest and a real Vitest (both pools), for the project described above for `--quick`: the same lines and the same exit code, TBD-STRESS. A hunt for projects in which the two differ found six, all of the same kind: a control that does not hold on its own, reported as holding with `--jobs`.
|
|
509
556
|
|
|
510
557
|
| What the project does | What now keeps it from being reported as holding |
|
|
@@ -522,7 +569,7 @@ A second kind came out of a check afterwards. `--jobs` leaves out of the shared
|
|
|
522
569
|
|
|
523
570
|
That the controls kept apart are chosen one by one, rather than all of them for any file that reaches outside, does not make `--jobs` safer or less safe for a control that runs next to others: what can disturb it is looked for in the same files as before, those of the specs that load its gate and those the gate config names. It makes the option work in projects where some specs need a clock, a file or a package. Before, a single such file anywhere kept every control on its own. `test/selfcheck-runners-*.test.ts` has a project of five gates, with a real Jest and a real Vitest, in which a spec that reads files loads another gate: that control runs on its own, the others next to each other, and the lines and the exit code are those of one control at a time, also for a gate that another spec covers.
|
|
524
571
|
|
|
525
|
-
**The
|
|
572
|
+
**The runs share the cores.** Every run starts its own workers, and both runners take about as many as the machine has cores, so `--jobs 4` alone would start four times that many processes. So each run next to others gets `--maxWorkers=<cores / runs>` (at least 1) added to its gate command, where the cores are what `os.availableParallelism()` says and the runs are those under way at the same time (`<n>`, or the number of controls when there are fewer). On 12 cores with `--jobs 4` that is `--maxWorkers=3`; on a hosted CI runner with 2 cores it is 1. When the gate command already says how many workers a run may use (Jest `--runInBand`, `-i`, `--maxWorkers`, `-w`; Vitest `--no-file-parallelism`, `--fileParallelism=false`, `--maxWorkers`, `--minWorkers`), or the gate config names one of `maxWorkers`, `minWorkers`, `maxThreads`, `minThreads`, `maxForks`, `minForks`, `fileParallelism`, `singleThread` or `singleFork`, nothing is added and that number counts. (Only the file the gate command names with `--config` is read for that: a setting in a file it imports is not seen, and `--maxWorkers` on the command line then wins over it.) With `--jobs 1`, and for a control that runs again on its own, the gate command is run as it is.
|
|
526
573
|
|
|
527
574
|
**With the other options.** `--jobs` applies to the gates that are selected: to the part of `--shard`, to the gates of `--gate`. With `--first` or a single `--gate` there is one control and nothing to run at the same time. With `--quick`, the shared run comes first and `--jobs` applies to the controls it leaves. Unlike `--quick`, `--jobs` is not refused in CI: within the limits above a control is judged from a run that left out its spec and nothing else. On machines of a hosted CI with two cores, `--shard` over parallel jobs gains more.
|
|
528
575
|
|