@t09tanaka/test-gates 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +26 -0
- package/README.md +191 -19
- package/dist/{chunk-LMD3ANJ2.js → chunk-XEKQKLKY.js} +48 -4
- package/dist/cli.js +364 -133
- package/dist/index.cjs +225 -81
- package/dist/index.d.cts +40 -0
- package/dist/index.d.ts +40 -0
- package/dist/index.js +178 -78
- package/dist/jest.cjs +48 -4
- package/dist/jest.js +1 -1
- package/dist/stryker.cjs +48 -4
- package/dist/stryker.js +1 -1
- package/dist/vitest.cjs +50 -5
- package/dist/vitest.d.cts +6 -0
- package/dist/vitest.d.ts +6 -0
- package/dist/vitest.js +3 -2
- package/package.json +9 -3
- package/schema/test-gates.schema.json +164 -27
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.2.0
|
|
4
|
+
|
|
5
|
+
Everything added here is opt-in. With an unchanged `test-gates.json` and unchanged config files, 0.2.0 gives the same verdicts, exit codes and summary line as 0.1.0. The one visible difference is a warning on stderr from `test-gates check` for Vitest 3 with the v8 provider (see below); it does not change the exit code.
|
|
6
|
+
|
|
7
|
+
A manifest that uses the new settings needs 0.2.0 or later, because 0.1.0 rejects unknown settings.
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
|
|
11
|
+
- `settings.mutation.maxTimeouts`: fail the mutation run when more mutants time out than this. Not set means no limit, as before.
|
|
12
|
+
- `expectedTimeouts` per gate: mutants that can never finish (a loop that no longer ends), matched like `equivalentMutants`. They are not counted against `maxTimeouts`. An entry whose mutant survives or does not exist fails the run; one that is killed is accepted.
|
|
13
|
+
- `settings.imports.mode: "allowlist"` with `settings.imports.allow`: a gate may import at runtime only relative paths, import aliases, listed packages (optionally only listed names, e.g. `Prisma` from `@prisma/client`) and types. The default stays `blocklist`.
|
|
14
|
+
- `createVitestGatesConfig({ coverageProvider: 'istanbul' })`, and `@vitest/coverage-istanbul` as an optional peer dependency.
|
|
15
|
+
- `test-gates check` warns when the gate run uses Vitest below 4 with the v8 provider, which does not count the skipped side of an `if` without `else`.
|
|
16
|
+
- `test-gates selfcheck` names the gates that import a gate whose spec could be left out without the gate failing.
|
|
17
|
+
- CI for this repository (lint, tests, gates, negative control, full mutation run; build and tests on Node 20).
|
|
18
|
+
- README: when the package is worth using, timeouts, allowlist mode, the Vitest v8 / istanbul measurements, cost and incremental mode, settings for Dart, and notes from migrating six repositories.
|
|
19
|
+
|
|
20
|
+
### Changed
|
|
21
|
+
|
|
22
|
+
- The package's own gates now run with the allowlist, `maxTimeouts: 0` and istanbul, and include the two new decision modules (8 gates).
|
|
23
|
+
|
|
24
|
+
## 0.1.0
|
|
25
|
+
|
|
26
|
+
First release: `test-gates check`, `mutation`, `mutation-result`, `selfcheck`, `lcov`; config helpers for Jest, Vitest and Stryker; JSON Schema for `test-gates.json`.
|
package/README.md
CHANGED
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# test-gates
|
|
2
2
|
|
|
3
|
+
[](https://github.com/t09tanaka/test-gates/actions/workflows/ci.yml)
|
|
4
|
+
|
|
3
5
|
Test gates for the code that must not be wrong.
|
|
4
6
|
|
|
5
7
|
This package guards the handful of modules where a wrong decision costs money or trust: amounts and rounding, state transitions of payments and orders, authentication and role checks, expiry and remaining-quota rules.
|
|
@@ -10,6 +12,15 @@ This package guards the handful of modules where a wrong decision costs money or
|
|
|
10
12
|
|
|
11
13
|
`test-gates` is a CLI plus three small config helpers. It has no runtime dependencies. Jest, Vitest and Stryker are the ones already installed in your project.
|
|
12
14
|
|
|
15
|
+
### When this is worth it, and when it is not
|
|
16
|
+
|
|
17
|
+
It pays off when the same rule has to hold in several places: a monorepo with several subprojects, or several repositories, where a copied script would drift. One `test-gates.json` per subproject, the same commands everywhere, the same verdict.
|
|
18
|
+
|
|
19
|
+
It does little when
|
|
20
|
+
|
|
21
|
+
- you have one project and one script already does the job. A script you understand is fine; this package is that script, made uniform.
|
|
22
|
+
- the decisions are not in pure modules yet. In a codebase that is mostly decorators and dependency injection, the important logic ends up under `candidates`, and **nothing is enforced for a candidate**. The package then only gives you the list of what to extract first. The enforcement starts when a decision has been moved into a pure module.
|
|
23
|
+
|
|
13
24
|
## Install
|
|
14
25
|
|
|
15
26
|
```bash
|
|
@@ -217,6 +228,7 @@ The run also fails when
|
|
|
217
228
|
- two allowances point at the same mutant, or an allowance has no `reason`,
|
|
218
229
|
- a gate is missing from the report or has no mutants, or no mutant was evaluated at all,
|
|
219
230
|
- the report was made from a different version of a gate than the one on disk.
|
|
231
|
+
- more mutants timed out than `settings.mutation.maxTimeouts` allows (only when that setting is present, see [Timeouts](#timeouts)).
|
|
220
232
|
|
|
221
233
|
`Timeout` counts as detected, but its count is always printed on its own (`detected N (timeout M)`). On a loaded machine mutants time out that would otherwise survive, so a rising timeout count means the run may be hiding survivors: rerun with lower `--concurrency`.
|
|
222
234
|
|
|
@@ -226,12 +238,61 @@ On failure every unallowed survivor is listed as `file:line:column / mutator / o
|
|
|
226
238
|
test-gates mutation: OK (mutants 317 / detected 308 (timeout 0) / allowed equivalent 8 / not evaluable 1 / unallowed survivors 0)
|
|
227
239
|
```
|
|
228
240
|
|
|
241
|
+
#### Timeouts
|
|
242
|
+
|
|
243
|
+
A timeout is a detection only if the mutant really cannot finish. On a loaded machine a mutant that would have survived can time out instead, and the run looks better than it is. Two opt-in controls (since 0.2.0):
|
|
244
|
+
|
|
245
|
+
```json
|
|
246
|
+
{
|
|
247
|
+
"gates": [
|
|
248
|
+
{
|
|
249
|
+
"path": "src/core/args.ts",
|
|
250
|
+
"decides": "…",
|
|
251
|
+
"impact": "…",
|
|
252
|
+
"expectedTimeouts": [
|
|
253
|
+
{
|
|
254
|
+
"mutator": "BlockStatement",
|
|
255
|
+
"original": "{ index = consume(parsed, argv, index, allowed); }",
|
|
256
|
+
"replacement": "{}",
|
|
257
|
+
"reason": "With an empty body the loop index never advances, so the function never returns"
|
|
258
|
+
}
|
|
259
|
+
]
|
|
260
|
+
}
|
|
261
|
+
],
|
|
262
|
+
"settings": { "mutation": { "maxTimeouts": 0 } }
|
|
263
|
+
}
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
- `settings.mutation.maxTimeouts`: how many timeouts outside `expectedTimeouts` are tolerated. Above it the run fails with exit code 1 and lists every such mutant as `file:line:column / mutator / original → replacement`, with a fragment to paste. Not set means no limit, which is the 0.1.0 behaviour.
|
|
267
|
+
- `expectedTimeouts` (per gate): mutants that turn a loop into one that never ends. Same key as `equivalentMutants` (`mutator` + `original` + `replacement`, `occurrence` when ambiguous, `reason` required).
|
|
268
|
+
|
|
269
|
+
How an `expectedTimeouts` entry is judged:
|
|
270
|
+
|
|
271
|
+
| The mutant is… | Result |
|
|
272
|
+
| ------------------------ | --------------------------------------------------- |
|
|
273
|
+
| `Timeout` | Accepted, not counted against `maxTimeouts` |
|
|
274
|
+
| `Killed` | Accepted |
|
|
275
|
+
| `Survived`, `NoCoverage` | Failure: it is not a timeout at all, write the test |
|
|
276
|
+
| not in this run | Failure: stale entry, remove it |
|
|
277
|
+
|
|
278
|
+
`Killed` is accepted on purpose, unlike a stale `equivalentMutants` entry. A mutant that hangs is reported as `Timeout` when Stryker's timer fires first and as `Killed` when the test runner's own per-test timeout fires first; which one wins depends on the load. Failing on `Killed` would make the list flap between two runs of the same code. And nothing is hidden by accepting it: `Killed` is the best possible outcome.
|
|
279
|
+
|
|
280
|
+
When the limit is exceeded, first run again with less load (`npm run test:gates:mutation -- --concurrency 1`). Add an entry only for a mutant that can never finish.
|
|
281
|
+
|
|
229
282
|
### `test-gates selfcheck [--all | --first] [-- <gate command>]`
|
|
230
283
|
|
|
231
284
|
Negative control. For each gate, runs the gate with that gate's spec left out and requires the run to **fail on the coverage threshold of that very file**. A gate that stays green without its spec measures nothing; a run that fails for another reason (a config that does not load) proves nothing either.
|
|
232
285
|
|
|
233
286
|
The spec to leave out is passed in the environment variable `TEST_GATES_EXCLUDE_SPEC`. The config helpers read it. The gate command is `jest --config jest.gates.config.*` or `vitest run --config vitest.gates.config.*`, whichever config exists; override it with `settings.gateCommand` or after `--`. `--all` (default) checks every gate in turn, `--first` only the first.
|
|
234
287
|
|
|
288
|
+
**When a gate imports another gate.** If `api-key-auth.ts` imports `safe-compare.ts` and both are gates, the spec of `api-key-auth` executes every line of `safe-compare` as well. Leaving out `safe-compare.spec.ts` then does not lower its coverage, and `selfcheck` fails for `safe-compare` ("the gate passed although the spec … was left out"), naming the importing gate in a hint. The coverage floor of `safe-compare` is being held up by somebody else's spec. Fix it in the importer's spec: mock the imported gate and assert **how it is called**, so that each spec covers only its own gate:
|
|
289
|
+
|
|
290
|
+
```ts
|
|
291
|
+
jest.mock('./safe-compare');
|
|
292
|
+
// …
|
|
293
|
+
expect(safeCompare).toHaveBeenCalledWith(presentedKey, storedKey); // right arguments, right order
|
|
294
|
+
```
|
|
295
|
+
|
|
235
296
|
### `test-gates lcov --file <lcov.info>`
|
|
236
297
|
|
|
237
298
|
For stacks without mutation testing (Flutter / Dart): requires `LH == LF` for every gate in an lcov tracefile. A gate without a record, or with no instrumented line, fails. Paths are compared as written in the tracefile (`lib/models/coupon.dart`); absolute paths under the subproject are made relative.
|
|
@@ -240,6 +301,24 @@ For stacks without mutation testing (Flutter / Dart): requires `LH == LF` for ev
|
|
|
240
301
|
flutter test --coverage && npx test-gates lcov --file coverage/lcov.info
|
|
241
302
|
```
|
|
242
303
|
|
|
304
|
+
Settings for a Dart package that uses only `lcov`:
|
|
305
|
+
|
|
306
|
+
```json
|
|
307
|
+
{
|
|
308
|
+
"gates": [{ "path": "lib/models/coupon.dart", "decides": "…", "impact": "…" }],
|
|
309
|
+
"candidates": [],
|
|
310
|
+
"settings": {
|
|
311
|
+
"gateExtensions": [".dart"],
|
|
312
|
+
"spec": { "suffixes": [] },
|
|
313
|
+
"lcov": { "file": "coverage/lcov.info", "summaryExclude": ["\\.g\\.dart$", "lib/l10n/"] }
|
|
314
|
+
}
|
|
315
|
+
}
|
|
316
|
+
```
|
|
317
|
+
|
|
318
|
+
Do not run `test-gates check` on such a manifest: Dart annotations such as `@JsonSerializable()` look like decorators to it.
|
|
319
|
+
|
|
320
|
+
Lines are counted from the `DA:` entries, not taken from `LH:` / `LF:`. A tracefile whose `LH:` was edited but whose `DA:` lines still show an unexecuted line fails; `LH:` / `LF:` are only used for a record that has no `DA:` entry at all.
|
|
321
|
+
|
|
243
322
|
## Allowing an equivalent mutant
|
|
244
323
|
|
|
245
324
|
An equivalent mutant changes the code without changing anything observable: same return value, same exception, same side effects, for every input. Only those may be allowed, in `test-gates.json`, with the reason:
|
|
@@ -271,25 +350,28 @@ Expected values in the tests come from the specification, not from the implement
|
|
|
271
350
|
|
|
272
351
|
All optional. Patterns are regular expressions written as JSON strings; a rule is either a string or `{ "pattern", "flags", "reason" }`. An unknown key is an error (exit code 2), so a typo cannot silently fall back to a default.
|
|
273
352
|
|
|
274
|
-
| Setting
|
|
275
|
-
|
|
|
276
|
-
| `gateExtensions`
|
|
277
|
-
| `spec.suffixes`
|
|
278
|
-
| `spec.rewrite`
|
|
279
|
-
| `impureImports.defaults`
|
|
280
|
-
| `impureImports.add`
|
|
281
|
-
| `impureImports.allow`
|
|
282
|
-
| `impureNamedImports`
|
|
283
|
-
| `impurePaths`
|
|
284
|
-
| `forbiddenSource`
|
|
285
|
-
| `importAliases`
|
|
286
|
-
| `gateCommand`
|
|
287
|
-
| `selfcheck.mode`
|
|
288
|
-
| `selfcheck.failurePattern`
|
|
289
|
-
| `stryker.configFile`
|
|
290
|
-
| `stryker.reportFile`
|
|
291
|
-
| `lcov.file`
|
|
292
|
-
| `lcov.summaryExclude`
|
|
353
|
+
| Setting | Default | Purpose |
|
|
354
|
+
| ------------------------------ | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
355
|
+
| `gateExtensions` | `.ts .mts .cts .js .mjs .cjs` | Extensions a gate may have |
|
|
356
|
+
| `spec.suffixes` | `.spec<ext>` and `.test<ext>` of the gate | Replaces the gate's extension to form the spec path. Exactly one must exist. `[]` means no spec convention (spec checks are skipped) |
|
|
357
|
+
| `spec.rewrite` | none | `[{ "from", "to" }]` applied to the gate path first, for specs in another directory (`^app/` → `tests/`) |
|
|
358
|
+
| `impureImports.defaults` | `true` | Built-in list, see below |
|
|
359
|
+
| `impureImports.add` | none | More module patterns a gate must not import at runtime |
|
|
360
|
+
| `impureImports.allow` | none | Module patterns exempt from every rule (e.g. `^express$` where `express` is imported only for its types without `import type`) |
|
|
361
|
+
| `impureNamedImports` | `PrismaClient` from `@prisma/client` | `{ "defaults", "add": [{ "module", "names", "reason" }] }`: names banned from a module that is otherwise fine |
|
|
362
|
+
| `impurePaths` | `.vue` `.jsx` `.tsx`, `.d.ts` | `{ "defaults", "add" }`: gate paths that are rejected (`\\.service\\.ts$`, `^src/hooks/`) |
|
|
363
|
+
| `forbiddenSource` | `'use client'` / `'use server'` | `{ "defaults", "add" }`: lines that must not appear in a gate, tested one line at a time |
|
|
364
|
+
| `importAliases` | none | `[{ "prefix": "@/", "target": "src/" }]`, to recognise `jest.mock('@/x')` of the module under test |
|
|
365
|
+
| `gateCommand` | derived from `jest.gates.config.*` / `vitest.gates.config.*` | Command `selfcheck` runs, as an array (`["jest", "--config", "jest.gates.config.js", "--maxWorkers=2"]`) |
|
|
366
|
+
| `selfcheck.mode` | `all` | `all` or `first` |
|
|
367
|
+
| `selfcheck.failurePattern` | a threshold message on a line naming the gate | Regular expression the failing run's output must match; `{gate}` stands for the gate path |
|
|
368
|
+
| `stryker.configFile` | `stryker.gates.config.mjs` (`.js`, `.cjs`, `.json`) | Stryker config |
|
|
369
|
+
| `stryker.reportFile` | `reports/mutation/mutation.json` | Where the JSON report is written (by the helper) and read (by the judge) |
|
|
370
|
+
| `lcov.file` | none | Tracefile for `test-gates lcov` when `--file` is not given |
|
|
371
|
+
| `lcov.summaryExclude` | none | Paths left out of the reference total (`\\.g\\.dart$`) |
|
|
372
|
+
| `imports.mode` (0.2.0) | `blocklist` | `allowlist` turns the import check around, see [Allowlist mode](#allowlist-mode) |
|
|
373
|
+
| `imports.allow` (0.2.0) | none | Packages a gate may import in allowlist mode: `"decimal.js"`, `{ "pattern": "^@acme/pure-" }`, `{ "module": "@prisma/client", "names": ["Prisma"] }` |
|
|
374
|
+
| `mutation.maxTimeouts` (0.2.0) | none (no limit) | Timeouts tolerated outside `expectedTimeouts`, see [Timeouts](#timeouts) |
|
|
293
375
|
|
|
294
376
|
Built-in impure imports: `@nestjs/*`, `class-validator`, `class-transformer`, `typeorm`, `sequelize`, `mongoose`, `knex`, `pg`, `mysql`, `mysql2`, `ioredis`, `redis`, `@prisma/adapter-*`, `react`, `react-dom`, `next`, `server-only`, `client-only`, `vue`, `vue-router`, `vue-i18n`, `pinia`, `nuxt`, `#app`, `#imports`, `*.vue`, `express`, `express-jwt`, `fastify`, `koa`, `rxjs`, `passport`, `passport-*`, `axios`, `openapi-fetch`, `stripe`, `nodemailer`, `firebase-admin`, `@aws-sdk/*`, `@sentry/*`, `@slack/*`, `@stripe/*`, and the Node.js modules `fs`, `http`, `https`, `http2`, `net`, `tls`, `dgram`, `dns`, `child_process`, `worker_threads`.
|
|
295
377
|
|
|
@@ -323,6 +405,38 @@ Project-specific layers go into `add`:
|
|
|
323
405
|
}
|
|
324
406
|
```
|
|
325
407
|
|
|
408
|
+
### Allowlist mode
|
|
409
|
+
|
|
410
|
+
The default import check is a blocklist: it knows the frameworks and SDKs listed above, and a new SDK passes until somebody adds it. `settings.imports.mode: "allowlist"` (since 0.2.0) turns it around. A gate may then import at runtime only
|
|
411
|
+
|
|
412
|
+
- relative paths and paths through `importAliases` (the project's own files),
|
|
413
|
+
- what `settings.imports.allow` lists,
|
|
414
|
+
- anything with `import type` (types leave no trace at runtime).
|
|
415
|
+
|
|
416
|
+
Everything else is a violation, including Node.js built-ins (`fs` and `node:fs` alike; list `node:path` if a gate needs it) and a `require()` / `import()` whose argument is not a string literal.
|
|
417
|
+
|
|
418
|
+
```json
|
|
419
|
+
{
|
|
420
|
+
"settings": {
|
|
421
|
+
"importAliases": [{ "prefix": "@/", "target": "src/" }],
|
|
422
|
+
"imports": {
|
|
423
|
+
"mode": "allowlist",
|
|
424
|
+
"allow": ["decimal.js", "date-fns", { "module": "@prisma/client", "names": ["Prisma"] }]
|
|
425
|
+
}
|
|
426
|
+
}
|
|
427
|
+
}
|
|
428
|
+
```
|
|
429
|
+
|
|
430
|
+
- A string is a package name and covers its subpaths (`date-fns`, `date-fns/locale`).
|
|
431
|
+
- `{ "pattern" }` is a regular expression tested against the whole specifier.
|
|
432
|
+
- `{ "module", "names" }` allows only those names, and only as named imports. `import { Prisma } from '@prisma/client'` passes; `import { PrismaClient }`, a default import, `import * as` and `require()` do not.
|
|
433
|
+
|
|
434
|
+
The blocklist stays in force underneath. A project file is not safe just because it is relative: `./user.service` is still rejected by an `impureImports` pattern, and in allowlist mode also when the file it points at matches `impurePaths` (a service, a component). `impureImports.allow` (exempt a module from the blocklist) and `imports.allow` (the list of permitted packages) are different settings.
|
|
435
|
+
|
|
436
|
+
What it does not do: follow the imports. Only the gate's own import statements are read. If `./rates` is allowed and `rates.ts` itself imports a database client, that is not noticed unless `rates.ts` is a gate too. A transitive check would need real module resolution (extensionless imports, `index` files, `paths` of tsconfig, package `exports`), and a resolver that guesses wrong reports violations that are not there. It was left out for that reason; register the imported file as a gate when it matters.
|
|
437
|
+
|
|
438
|
+
A manifest that uses `settings.imports` or `settings.mutation` needs 0.2.0 or later: 0.1.0 rejects unknown settings.
|
|
439
|
+
|
|
326
440
|
## Config helpers
|
|
327
441
|
|
|
328
442
|
```ts
|
|
@@ -348,6 +462,60 @@ Why these choices:
|
|
|
348
462
|
|
|
349
463
|
`loadGates(dir)` returns the validated manifest: `gates` (each with its resolved `spec`), `candidates` and the resolved `settings`. It throws when the manifest has any problem.
|
|
350
464
|
|
|
465
|
+
## Vitest: v8 or istanbul
|
|
466
|
+
|
|
467
|
+
The Jest helper forces istanbul. The Vitest helper defaults to `v8`, and whether that is as strict depends on the Vitest version. Measured with one module per construct and a test that takes only one side; "found" means branches were reported below 100%:
|
|
468
|
+
|
|
469
|
+
| Only one side tested | Vitest 3.2.7 v8 | 3.2.7 v8 + `experimentalAstAwareRemapping` | 3.2.7 istanbul | Vitest 4.1.11 v8 | 4.1.11 istanbul |
|
|
470
|
+
| -------------------------------------------- | --------------- | ------------------------------------------ | -------------- | ---------------- | --------------- |
|
|
471
|
+
| `if (a) { … }` without `else`, never skipped | **missed** | found | found | found | found |
|
|
472
|
+
| `if (!a) return …` (early return) | found | found | found | found | found |
|
|
473
|
+
| `a ? x : y` | found | found | found | found | found |
|
|
474
|
+
| `a ?? b` | found | found | found | found | found |
|
|
475
|
+
| `a && f()` with `a` always false | found | found | found | found | found |
|
|
476
|
+
| default parameter `(a = 3)`, always passed | **missed** | **missed** | found | **missed** | found |
|
|
477
|
+
| `o?.y` with `o` always undefined | found | not a branch | not a branch | not a branch | not a branch |
|
|
478
|
+
|
|
479
|
+
So the asymmetry is real on Vitest 3: with the default v8 provider a gate can show 100% branches while the skipped side of an `if` was never tested. Vitest 4 closes that gap; an unused default parameter is still only seen by istanbul. The mutation test catches both cases anyway (a mutant in untested code survives), but the floor should not have a hole in it.
|
|
480
|
+
|
|
481
|
+
Recommended: istanbul for the gate run.
|
|
482
|
+
|
|
483
|
+
```bash
|
|
484
|
+
npm install -D @vitest/coverage-istanbul # same major version as vitest
|
|
485
|
+
```
|
|
486
|
+
|
|
487
|
+
```ts
|
|
488
|
+
export default defineConfig(
|
|
489
|
+
createVitestGatesConfig({ rootDir: import.meta.url, coverageProvider: 'istanbul' })
|
|
490
|
+
);
|
|
491
|
+
```
|
|
492
|
+
|
|
493
|
+
The default stays `v8` because changing it would break every project that has only `@vitest/coverage-v8` installed. Instead `test-gates check` prints a warning (the exit code does not change) when it finds a `vitest.gates.config.*`, an installed Vitest below 4, and neither `istanbul` nor `experimentalAstAwareRemapping: true` in that config.
|
|
494
|
+
|
|
495
|
+
## Cost, and how to use Stryker's incremental mode
|
|
496
|
+
|
|
497
|
+
Mutation testing is the expensive step. Numbers for this repository's own gates:
|
|
498
|
+
|
|
499
|
+
| Run | Machine | Time |
|
|
500
|
+
| ------------------------------------------------- | ---------------------------------------------------- | ---------- |
|
|
501
|
+
| 6 gates, about 1,000 lines, 809 mutants, full run | an outside evaluator's machine (not specified) | 4 min 24 s |
|
|
502
|
+
| the same 6 gates, 809 mutants, full run | Apple M2 Max (12 cores), Node 22.22, `concurrency 2` | 41 s |
|
|
503
|
+
| 8 gates, 1,136 lines, 1,021 mutants, full run | same machine, other jobs running | 1 min 19 s |
|
|
504
|
+
| the same, incremental, nothing changed | same machine | 8–15 s |
|
|
505
|
+
|
|
506
|
+
The time grows with the number of mutants and depends heavily on cores and on `concurrency`. Twenty gates of this size on a slow machine are well past ten minutes for a full run. The setup that has held up in practice:
|
|
507
|
+
|
|
508
|
+
- **pre-push: incremental.** `test-gates mutation` as is. Stryker's incremental file (`reports/stryker-incremental.json`, not committed) makes the second and later runs take seconds; the first run after a clone is a full one.
|
|
509
|
+
- **CI: full.** `npm run test:gates:mutation -- --force`. CI has no incremental file to start from anyway, and the full run is the one that counts.
|
|
510
|
+
- If even CI is too slow, split by subproject (each has its own `test-gates.json`) before you think about sampling.
|
|
511
|
+
|
|
512
|
+
How the incremental file and this tool interact (covered by `test/incremental.test.ts`, which runs a real Stryker):
|
|
513
|
+
|
|
514
|
+
- `test-gates mutation` deletes the JSON **report** before each run. It does not touch the incremental file; they are different files.
|
|
515
|
+
- A run in which every result is reused still writes a complete JSON report, and the verdict is the same as that of the run it reuses.
|
|
516
|
+
- **A stale "Survived" can be reused.** After adding a test that kills a _static_ mutant (code that runs when the module is loaded, such as a module-level constant), the incremental run kept reporting the mutant as survived; `--force` reported it killed. The error is on the safe side, a failure that is not real, never a pass that is not real. When a survivor makes no sense after you fixed the test, run once with `-- --force`.
|
|
517
|
+
- **After removing a gate, its old results stay in the incremental file** and come back in the report. They are ignored: only the gates of `test-gates.json` are judged. Delete the incremental file (or run with `--force`) when you change the list of gates, so that it does not carry files that are no longer mutated.
|
|
518
|
+
|
|
351
519
|
## Limitations and things measured
|
|
352
520
|
|
|
353
521
|
- **A file with decorators cannot reach 100% branches under Jest.** The type metadata emitted for decorators contains inline conditionals (`typeof X === "undefined" ? Object : X`) that no test can take both ways. That is why decorated files are rejected as gates instead of being given a lower threshold.
|
|
@@ -358,6 +526,10 @@ Why these choices:
|
|
|
358
526
|
- **Imports are found by text, not by parsing.** The checks run without the project's dependencies installed, so an import inside a comment is reported too, and only the gate's own imports are looked at, not what those modules import in turn.
|
|
359
527
|
- **An ordinary `import { Request } from 'express'` used only as a type is reported.** Write `import type`, or exempt the module with `impureImports.allow`.
|
|
360
528
|
- **`test-gates lcov` checks lines only.** lcov from Flutter carries no branch data, and Dart has no established mutation testing tool; a Dart gate is guarded by coverage alone.
|
|
529
|
+
- **knip reports the Stryker runner as unused** once no npm script calls `stryker run` directly. Tell knip where the config is: `"stryker": { "config": ["stryker.gates.config.mjs"] }` in `knip.json`.
|
|
530
|
+
- **A directory or worktree whose name contains `test-`** (for example a checkout called `test-gates-migration`) makes `@darraghor/eslint-plugin-nestjs-typed` treat every file as a test file, because it matches the whole path against a test pattern. Not caused by this package, but a name like that is easy to choose while adopting it.
|
|
531
|
+
- **A comment line that starts with `@t09tanaka/…`** in a config file can be read by ESLint's JSDoc rules as a tag. Put the package name in the middle of the sentence.
|
|
532
|
+
- **Vitest 4's `text` reporter can print only the summary line** and no per-file table for the gate run. The thresholds and the negative control still work.
|
|
361
533
|
- **The checks do not judge what belongs in `gates`.** Whether every important decision has been extracted and registered is for people to review; `candidates` is where the known gaps are written down.
|
|
362
534
|
|
|
363
535
|
## License
|
|
@@ -30,17 +30,17 @@ function allowanceProblems(entry) {
|
|
|
30
30
|
}
|
|
31
31
|
return problems;
|
|
32
32
|
}
|
|
33
|
-
function validateEquivalentMutants(gatePath, value) {
|
|
33
|
+
function validateEquivalentMutants(gatePath, value, field = "equivalentMutants") {
|
|
34
34
|
if (value === void 0) {
|
|
35
35
|
return [];
|
|
36
36
|
}
|
|
37
37
|
if (!Array.isArray(value)) {
|
|
38
|
-
return [{ file: MANIFEST_FILE, message: `${gatePath}:
|
|
38
|
+
return [{ file: MANIFEST_FILE, message: `${gatePath}: ${field} must be an array` }];
|
|
39
39
|
}
|
|
40
40
|
return value.flatMap(
|
|
41
41
|
(entry, index) => allowanceProblems(entry).map((problem) => ({
|
|
42
42
|
file: MANIFEST_FILE,
|
|
43
|
-
message: `${gatePath}:
|
|
43
|
+
message: `${gatePath}: ${field}[${index}]${problem}`
|
|
44
44
|
}))
|
|
45
45
|
);
|
|
46
46
|
}
|
|
@@ -123,6 +123,9 @@ function parseManifest(text) {
|
|
|
123
123
|
}
|
|
124
124
|
for (const gate of gates) {
|
|
125
125
|
violations.push(...validateEquivalentMutants(gate.path, gate.equivalentMutants));
|
|
126
|
+
violations.push(
|
|
127
|
+
...validateEquivalentMutants(gate.path, gate.expectedTimeouts, "expectedTimeouts")
|
|
128
|
+
);
|
|
126
129
|
}
|
|
127
130
|
return { gates, candidates, settings: raw.settings, violations };
|
|
128
131
|
}
|
|
@@ -272,7 +275,9 @@ function resolveSettings(raw) {
|
|
|
272
275
|
"gateCommand",
|
|
273
276
|
"selfcheck",
|
|
274
277
|
"stryker",
|
|
275
|
-
"lcov"
|
|
278
|
+
"lcov",
|
|
279
|
+
"imports",
|
|
280
|
+
"mutation"
|
|
276
281
|
],
|
|
277
282
|
"settings"
|
|
278
283
|
);
|
|
@@ -373,6 +378,42 @@ function resolveSettings(raw) {
|
|
|
373
378
|
"settings.selfcheck.failurePattern"
|
|
374
379
|
);
|
|
375
380
|
}
|
|
381
|
+
const importsSection = section(settings.imports, ["mode", "allow"], "settings.imports");
|
|
382
|
+
if (importsSection.mode !== void 0 && importsSection.mode !== "blocklist" && importsSection.mode !== "allowlist") {
|
|
383
|
+
throw new SettingsError('settings.imports.mode: must be "blocklist" or "allowlist"');
|
|
384
|
+
}
|
|
385
|
+
const importAllowRaw = importsSection.allow ?? [];
|
|
386
|
+
if (!Array.isArray(importAllowRaw)) {
|
|
387
|
+
throw new SettingsError("settings.imports.allow: must be an array");
|
|
388
|
+
}
|
|
389
|
+
const importAllow = importAllowRaw.map((item, index) => {
|
|
390
|
+
const at = `settings.imports.allow[${index}]`;
|
|
391
|
+
if (typeof item === "string" && item !== "") {
|
|
392
|
+
return { module: item };
|
|
393
|
+
}
|
|
394
|
+
if (isRecord2(item) && typeof item.pattern === "string" && item.pattern !== "") {
|
|
395
|
+
rejectUnknownKeys(item, ["pattern", "flags"], at);
|
|
396
|
+
if (item.flags !== void 0 && typeof item.flags !== "string") {
|
|
397
|
+
throw new SettingsError(`${at}.flags: must be a string`);
|
|
398
|
+
}
|
|
399
|
+
return { regex: compile(item.pattern, item.flags, at) };
|
|
400
|
+
}
|
|
401
|
+
if (isRecord2(item) && typeof item.module === "string" && item.module !== "") {
|
|
402
|
+
rejectUnknownKeys(item, ["module", "names"], at);
|
|
403
|
+
const names = stringArray(item.names, `${at}.names`);
|
|
404
|
+
if (names.length === 0) {
|
|
405
|
+
throw new SettingsError(`${at}.names: must list at least one name`);
|
|
406
|
+
}
|
|
407
|
+
return { module: item.module, names };
|
|
408
|
+
}
|
|
409
|
+
throw new SettingsError(
|
|
410
|
+
`${at}: must be a package name, { pattern, flags? } or { module, names }`
|
|
411
|
+
);
|
|
412
|
+
});
|
|
413
|
+
const mutation = section(settings.mutation, ["maxTimeouts"], "settings.mutation");
|
|
414
|
+
if (mutation.maxTimeouts !== void 0 && !(Number.isInteger(mutation.maxTimeouts) && mutation.maxTimeouts >= 0)) {
|
|
415
|
+
throw new SettingsError("settings.mutation.maxTimeouts: must be an integer >= 0");
|
|
416
|
+
}
|
|
376
417
|
const stryker = section(settings.stryker, ["configFile", "reportFile"], "settings.stryker");
|
|
377
418
|
const lcov = section(settings.lcov, ["file", "summaryExclude"], "settings.lcov");
|
|
378
419
|
return {
|
|
@@ -394,6 +435,9 @@ function resolveSettings(raw) {
|
|
|
394
435
|
selfcheckFailurePattern,
|
|
395
436
|
strykerConfigFile: optionalString(stryker.configFile, "settings.stryker.configFile"),
|
|
396
437
|
strykerReportFile: optionalString(stryker.reportFile, "settings.stryker.reportFile") ?? DEFAULT_REPORT_FILE,
|
|
438
|
+
importMode: importsSection.mode ?? "blocklist",
|
|
439
|
+
importAllow,
|
|
440
|
+
maxTimeouts: mutation.maxTimeouts ?? null,
|
|
397
441
|
lcovFile: optionalString(lcov.file, "settings.lcov.file"),
|
|
398
442
|
lcovSummaryExclude: lcov.summaryExclude === void 0 ? [] : stringArray(lcov.summaryExclude, "settings.lcov.summaryExclude").map(
|
|
399
443
|
(pattern, index) => compile(pattern, void 0, `settings.lcov.summaryExclude[${index}]`)
|