@t09tanaka/test-gates 0.3.1 → 0.3.3

package/CHANGELOG.md

@@ -1,10 +1,25 @@
 # Changelog
 
+## 0.3.3
+
+Documentation only; the commands and the helpers behave as in 0.3.2.
+
+- The paths and outputs shown in the README are those of a small example written for it and of this repository's own gates.
+
+## 0.3.2
+
+### Changed
+
+- `createStrykerGatesConfig` sets Stryker's `maxTestRunnerReuse` to 50 unless the project passes its own (Stryker's default is 0: one test runner process for the whole run). In a project with mutants that never end, the runner became about three times slower from the first of them on, and mutants that are killed in seconds on their own timed out. Starting a new process every 50 mutants kept the speed: a full run of about 800 mutants went from about a quarter of an hour with stray timeouts to less than half of that with none.
+- `timeoutMS` is no longer set by the helper; Stryker's default (5000) applies again. The 30000 of 0.3.1 was the wrong fix. It rested on the idea that a restarted runner makes the next mutant slow, when in fact a runner that is not restarted stays slow. With 30000 the same full run took about three times as long and still failed, and mutants that exhaust the heap crashed the runner and were reported as `RuntimeError` instead of `Timeout`.
+
+A project that set `timeoutMS` itself to work around stray timeouts can remove it.
+
 ## 0.3.1
 
 ### Changed
 
-- `createStrykerGatesConfig` sets Stryker's `timeoutMS` to 30000 unless the project passes its own (Stryker's default is 5000). With `maxTimeouts` at 0 since 0.3.0, a mutant that would be killed but ran out of time fails the run. That happened on every full run in two projects with mutants that really never end: after each of them Stryker starts a new test runner, and the next mutant timed out while it was starting. A full run takes longer by up to 25 seconds for each mutant that hangs without tripping Stryker's loop counter.
+- `createStrykerGatesConfig` sets Stryker's `timeoutMS` to 30000 unless the project passes its own (Stryker's default is 5000). With `maxTimeouts` at 0 since 0.3.0, a mutant that would be killed but ran out of time fails the run. That happened on every full run in projects with mutants that really never end: after each of them Stryker starts a new test runner, and the next mutant timed out while it was starting. A full run takes longer by up to 25 seconds for each mutant that hangs without tripping Stryker's loop counter.
 
 ## 0.3.0
 
@@ -35,7 +50,7 @@ A manifest that uses the new settings needs 0.2.0 or later, because 0.1.0 reject
 - `test-gates check` warns when the gate run uses Vitest below 4 with the v8 provider, which does not count the skipped side of an `if` without `else`.
 - `test-gates selfcheck` names the gates that import a gate whose spec could be left out without the gate failing.
 - CI for this repository (lint, tests, gates, negative control, full mutation run; build and tests on Node 20).
-- README: when the package is worth using, timeouts, allowlist mode, the Vitest v8 / istanbul measurements, cost and incremental mode, settings for Dart, and notes from migrating six repositories.
+- README: when the package is worth using, timeouts, allowlist mode, the Vitest v8 / istanbul measurements, cost and incremental mode, settings for Dart, and notes from adopting it in several repositories.
 
 ### Changed
 

package/README.md

@@ -44,22 +44,22 @@ Everything below is per subproject (the directory that has a `package.json`: `ba
   "$schema": "./node_modules/@t09tanaka/test-gates/schema/test-gates.schema.json",
   "gates": [
     {
-      "path": "src/common/utils/money.ts",
-      "decides": "Validation and rounding of amounts",
-      "impact": "Wrong invoice and refund amounts"
+      "path": "src/shipping-fee.ts",
+      "decides": "The fee for sending a parcel, by weight, and its maximum",
+      "impact": "A heavy parcel is sent for too little, or the customer pays more than the advertised maximum"
     }
   ],
   "candidates": [
     {
-      "path": "src/admin/guards/admin-role.guard.ts",
-      "decides": "Who is an administrator",
+      "path": "src/warehouse/dispatch.guard.ts",
+      "decides": "Who may release an order for dispatch",
       "blocker": "Has decorators. The decision has to be extracted into a pure module"
     }
   ]
 }
 ```
 
-- `gates`: pure modules that hold an important decision. Each one needs a spec next to it (`money.spec.ts` or `money.test.ts`), 100% on statements, branches, functions and lines, and no surviving mutant outside its allow list.
+- `gates`: pure modules that hold an important decision. Each one needs a spec next to it (`shipping-fee.spec.ts` or `shipping-fee.test.ts`), 100% on statements, branches, functions and lines, and no surviving mutant outside its allow list.
 - `candidates`: important logic that cannot be a gate yet, with what blocks it. Nothing is enforced for these except that the path exists.
 - `settings`: optional project conventions, see [Settings](#settings).
 
@@ -234,10 +234,10 @@ The run also fails when
 
 `Timeout` counts as detected, but its count is always printed on its own (`detected N (timeout M)`). On a loaded machine mutants time out that would otherwise survive, so a timeout nobody has explained fails the run.
 
-On failure every unallowed survivor is listed as `file:line:column / mutator / original → replacement / occurrence`, followed by a JSON fragment that can be pasted into `equivalentMutants`. The last line gives the real numbers, never a percentage:
+On failure every unallowed survivor is listed as `file:line:column / mutator / original → replacement / occurrence`, followed by a JSON fragment that can be pasted into `equivalentMutants`. The last line gives the real numbers, never a percentage. For the gates of this repository:
 
 ```
-test-gates mutation: OK (mutants 317 / detected 308 (timeout 0) / allowed equivalent 8 / not evaluable 1 / unallowed survivors 0)
+test-gates mutation: OK (mutants 1017 / detected 1017 (timeout 2) / allowed equivalent 0 / not evaluable 0 / unallowed survivors 0)
 ```
 
 #### Timeouts
@@ -280,7 +280,9 @@ How an `expectedTimeouts` entry is judged:
 
 When the run fails on a timeout, first run again with less load (`npm run test:gates:mutation -- --concurrency 1`). Add an entry only for a mutant that can never finish.
 
-`createStrykerGatesConfig` sets Stryker's `timeoutMS` to 30000 (since 0.3.1; Stryker's own default is 5000). After a mutant that really never ends, Stryker starts a new test runner, and with 5000 the mutant that comes next can run out of time while that runner is still starting: in one project with 11 such mutants, 2 to 7 others timed out on every full run, a different set each time, and none in a run with 30000. A longer limit hides nothing, because a timeout that is not listed fails the run anyway. The price is the wait for each mutant that hangs without tripping Stryker's loop counter. Pass `timeoutMS` to the helper to change it.
+`createStrykerGatesConfig` sets Stryker's `maxTestRunnerReuse` to 50 (since 0.3.2; Stryker's own default is 0, one test runner process for the whole run). A mutant that would be killed can also time out for a reason that has nothing to do with the mutant: in a Jest project with a few mutants that never end, the test runner handled about a third as many mutants per minute from the first of them on, for the rest of the run. Mutants that are killed in seconds on their own then timed out, a different set on each full run, and the run took about a quarter of an hour. With a new process every 50 mutants the speed stayed the same to the end, nothing timed out that should not, and the run took less than half as long. Why the runner slows down was not found; the restart only cuts it short. Pass `maxTestRunnerReuse` to the helper to change it.
+
+0.3.1 set `timeoutMS` to 30000 instead. That was the wrong fix and 0.3.2 takes it back: the longer limit let most slow mutants finish, but the same full run then took about three times as long as without it, and in another project mutants that exhaust the heap crashed the test runner before the limit and came back as `RuntimeError`, which is only counted, not failed. `timeoutMS` is Stryker's default (5000) again unless the project sets it.
 
 ### `test-gates selfcheck [--all | --first] [-- <gate command>]`
 
@@ -288,17 +290,17 @@ Negative control. For each gate, runs the gate with that gate's spec left out an
 
 The spec to leave out is passed in the environment variable `TEST_GATES_EXCLUDE_SPEC`. The config helpers read it. The gate command is `jest --config jest.gates.config.*` or `vitest run --config vitest.gates.config.*`, whichever config exists; override it with `settings.gateCommand` or after `--`. `--all` (default) checks every gate in turn, `--first` only the first.
 
-**When a gate imports another gate.** If `api-key-auth.ts` imports `safe-compare.ts` and both are gates, the spec of `api-key-auth` executes every line of `safe-compare` as well. Leaving out `safe-compare.spec.ts` then does not lower its coverage, and `selfcheck` fails for `safe-compare` ("the gate passed although the spec … was left out"), naming the importing gate in a hint. The coverage floor of `safe-compare` is being held up by somebody else's spec. Fix it in the importer's spec: mock the imported gate and assert **how it is called**, so that each spec covers only its own gate:
+**When a gate imports another gate.** If `delivery-price.ts` imports `shipping-fee.ts` and both are gates, the spec of `delivery-price` executes every line of `shipping-fee` as well. Leaving out `shipping-fee.spec.ts` then does not lower its coverage, and `selfcheck` fails for `shipping-fee` ("the gate passed although the spec … was left out"), naming the importing gate in a hint. The coverage floor of `shipping-fee` is being held up by somebody else's spec. Fix it in the importer's spec: mock the imported gate and assert **how it is called**, so that each spec covers only its own gate:
 
 ```ts
-jest.mock('./safe-compare');
+jest.mock('./shipping-fee');
 // …
-expect(safeCompare).toHaveBeenCalledWith(presentedKey, storedKey); // right arguments, right order
+expect(shippingFeeCents).toHaveBeenCalledWith(weightGrams); // called with the weight, not with the price
 ```
 
 ### `test-gates lcov --file <lcov.info>`
 
-For stacks without mutation testing (Flutter / Dart): requires `LH == LF` for every gate in an lcov tracefile. A gate without a record, or with no instrumented line, fails. Paths are compared as written in the tracefile (`lib/models/coupon.dart`); absolute paths under the subproject are made relative.
+For stacks without mutation testing (Flutter / Dart): requires `LH == LF` for every gate in an lcov tracefile. A gate without a record, or with no instrumented line, fails. Paths are compared as written in the tracefile (`lib/shipping_fee.dart`); absolute paths under the subproject are made relative.
 
 ```bash
 flutter test --coverage && npx test-gates lcov --file coverage/lcov.info
@@ -308,7 +310,7 @@ Settings for a Dart package that uses only `lcov`:
 
 ```json
 {
-  "gates": [{ "path": "lib/models/coupon.dart", "decides": "…", "impact": "…" }],
+  "gates": [{ "path": "lib/shipping_fee.dart", "decides": "…", "impact": "…" }],
   "candidates": [],
   "settings": {
     "gateExtensions": [".dart"],
@@ -324,19 +326,47 @@ Lines are counted from the `DA:` entries, not taken from `LH:` / `LF:`. A tracef
 
 ## Allowing an equivalent mutant
 
-An equivalent mutant changes the code without changing anything observable: same return value, same exception, same side effects, for every input. Only those may be allowed, in `test-gates.json`, with the reason:
+An equivalent mutant changes the code without changing anything observable: same return value, same exception, same side effects, for every input. Only those may be allowed, in `test-gates.json`, with the reason.
+
+An example. A gate limits a fee to a maximum:
+
+```ts
+let fee = Math.ceil(weightGrams / 1000) * 500;
+if (fee > MAX_FEE_CENTS) {
+  fee = MAX_FEE_CENTS;
+}
+return fee;
+```
+
+The specs check every boundary, and one mutant still survives:
+
+```
+test-gates mutation: 1 surviving mutant(s) not in the allow list
+  - src/shipping-fee.ts:9:7 / EqualityOperator / fee > MAX_FEE_CENTS → fee >= MAX_FEE_CENTS (Survived)
+
+Fix the test so that it detects the mutant. Only when the mutant cannot be observed from
+outside (same return value, exception and side effects for every input), add it to
+"equivalentMutants" of the gate in test-gates.json and fill in "reason":
+
+  src/shipping-fee.ts
+    {"mutator":"EqualityOperator","original":"fee > MAX_FEE_CENTS","replacement":"fee >= MAX_FEE_CENTS","reason":""}
+
+test-gates mutation: FAILED (mutants 14 / detected 13 (timeout 0) / allowed equivalent 0 / not evaluable 0 / unallowed survivors 1)
+```
+
+With `>=` the only input that is treated differently is a fee equal to the maximum, and that fee is set to the value it already has. No test can tell the two apart, so the gate lists the mutant:
 
 ```json
 {
-  "path": "src/lib/auth/safe-compare.ts",
-  "decides": "Constant-time comparison of tokens",
-  "impact": "Token comparison leaks timing or accepts a wrong token",
+  "path": "src/shipping-fee.ts",
+  "decides": "The fee for sending a parcel, by weight, and its maximum",
+  "impact": "A heavy parcel is sent for too little, or the customer pays more than the advertised maximum",
   "equivalentMutants": [
     {
-      "mutator": "ConditionalExpression",
-      "original": "a.length !== b.length",
-      "replacement": "false",
-      "reason": "timingSafeEqual below throws on a length mismatch and the catch returns false, so the result is the same"
+      "mutator": "EqualityOperator",
+      "original": "fee > MAX_FEE_CENTS",
+      "replacement": "fee >= MAX_FEE_CENTS",
+      "reason": "A fee that is equal to the maximum is set to the maximum, which is the value it has"
     }
   ]
 }
@@ -390,17 +420,17 @@ Project-specific layers go into `add`:
     "impureImports": {
       "add": [
         {
-          "pattern": "(^|/)(repository|wrapper|handler)(/|$)",
-          "reason": "DB / external API layer"
+          "pattern": "(^|/)(persistence|gateways)(/|$)",
+          "reason": "talks to the database or to a remote API"
         },
-        { "pattern": "\\.(service|guard|controller|module)$", "reason": "class managed by DI" }
+        { "pattern": "\\.(service|controller)$", "reason": "class created by the DI container" }
       ]
     },
     "impurePaths": {
       "add": [
         {
-          "pattern": "\\.(service|guard|controller|module|dto)\\.ts$",
-          "reason": "NestJS class file"
+          "pattern": "\\.(service|controller|module)\\.ts$",
+          "reason": "a class with decorators"
         }
       ]
     }
@@ -462,7 +492,7 @@ Why these choices:
 - Jest measures with istanbul (`babel`), because the v8 provider does not count the untaken side of an `if` without `else`.
 - Jest fails on a `coverageThreshold` path key that matches no file, so a typo in `test-gates.json` cannot pass. Vitest does the opposite: a glob or path key in `thresholds` that matches nothing passes. The Vitest helper therefore narrows `coverage.include` to the gates and uses `perFile`.
 - Stryker never fails on its own score (`break: null`). The verdict comes from `test-gates mutation`, which knows the allow list.
-- The Stryker helper also sets defaults that can be overridden: `coverageAnalysis: 'perTest'`, `incremental` with its file under `reports/`, and `timeoutMS: 30000` (see [Timeouts](#timeouts)).
+- The Stryker helper also sets defaults that can be overridden: `coverageAnalysis: 'perTest'`, `incremental` with its file under `reports/`, and `maxTestRunnerReuse: 50` (see [Timeouts](#timeouts)).
 
 `loadGates(dir)` returns the validated manifest: `gates` (each with its resolved `spec`), `candidates` and the resolved `settings`. It throws when the manifest has any problem.
 

package/dist/stryker.cjs

@@ -30,7 +30,7 @@ var __toCommonJS = (mod) => __copyProps(__defProp({}, "__esModule", { value: tru
 // src/stryker.ts
 var stryker_exports = {};
 __export(stryker_exports, {
-  DEFAULT_TIMEOUT_MS: () => DEFAULT_TIMEOUT_MS,
+  DEFAULT_MAX_TEST_RUNNER_REUSE: () => DEFAULT_MAX_TEST_RUNNER_REUSE,
   createStrykerGatesConfig: () => createStrykerGatesConfig
 });
 module.exports = __toCommonJS(stryker_exports);
@@ -602,7 +602,7 @@ function resolveRootDir(rootDir) {
 
 // src/stryker.ts
 var WEAKENING_OPTIONS = ["ignoreStatic", "ignorers"];
-var DEFAULT_TIMEOUT_MS = 3e4;
+var DEFAULT_MAX_TEST_RUNNER_REUSE = 50;
 function createStrykerGatesConfig(options = {}) {
   const { rootDir: rootDirOption, ...overrides } = options;
   for (const option of WEAKENING_OPTIONS) {
@@ -627,7 +627,7 @@ function createStrykerGatesConfig(options = {}) {
     incremental: true,
     incrementalFile: "reports/stryker-incremental.json",
     tempDirName: ".stryker-tmp",
-    timeoutMS: DEFAULT_TIMEOUT_MS,
+    maxTestRunnerReuse: DEFAULT_MAX_TEST_RUNNER_REUSE,
     clearTextReporter: { logTests: false, reportTests: false },
     ...overrides,
     mutate: manifest.gates.map((gate) => gate.path),
@@ -638,6 +638,6 @@ function createStrykerGatesConfig(options = {}) {
 }
 // Annotate the CommonJS export names for ESM import in node:
 0 && (module.exports = {
-  DEFAULT_TIMEOUT_MS,
+  DEFAULT_MAX_TEST_RUNNER_REUSE,
   createStrykerGatesConfig
 });

package/dist/stryker.d.cts

@@ -5,11 +5,13 @@ interface StrykerGatesOptions {
     [option: string]: unknown;
 }
 /**
- * Stryker's own default is 5000. After a mutant that really never ends, Stryker starts a new
- * test runner, and the next mutant then pays that start-up inside its time limit: with 5000 a
- * mutant that would be killed is reported as `Timeout`, which fails the run since 0.3.0.
+ * Stryker's own default is 0: one test runner process for the whole run. Measured on a Jest
+ * project with a few mutants that never end (`Hit limit reached`): from the first of them on
+ * the runner handled about a third as many mutants per minute for the rest of the run, and
+ * mutants that are killed in seconds on their own ran out of time. With a new process every 50
+ * mutants the speed stayed the same to the end.
  */
-declare const DEFAULT_TIMEOUT_MS = 30000;
+declare const DEFAULT_MAX_TEST_RUNNER_REUSE = 50;
 /**
  * Stryker config for the gates: mutate exactly the gates, never fail on the score
  * (`thresholds.break: null`), and write the JSON report that `test-gates mutation` judges.
@@ -19,4 +21,4 @@ declare const DEFAULT_TIMEOUT_MS = 30000;
  */
 declare function createStrykerGatesConfig(options?: StrykerGatesOptions): Record<string, unknown>;
 
-export { DEFAULT_TIMEOUT_MS, type StrykerGatesOptions, createStrykerGatesConfig };
+export { DEFAULT_MAX_TEST_RUNNER_REUSE, type StrykerGatesOptions, createStrykerGatesConfig };

package/dist/stryker.d.ts

@@ -5,11 +5,13 @@ interface StrykerGatesOptions {
     [option: string]: unknown;
 }
 /**
- * Stryker's own default is 5000. After a mutant that really never ends, Stryker starts a new
- * test runner, and the next mutant then pays that start-up inside its time limit: with 5000 a
- * mutant that would be killed is reported as `Timeout`, which fails the run since 0.3.0.
+ * Stryker's own default is 0: one test runner process for the whole run. Measured on a Jest
+ * project with a few mutants that never end (`Hit limit reached`): from the first of them on
+ * the runner handled about a third as many mutants per minute for the rest of the run, and
+ * mutants that are killed in seconds on their own ran out of time. With a new process every 50
+ * mutants the speed stayed the same to the end.
  */
-declare const DEFAULT_TIMEOUT_MS = 30000;
+declare const DEFAULT_MAX_TEST_RUNNER_REUSE = 50;
 /**
  * Stryker config for the gates: mutate exactly the gates, never fail on the score
  * (`thresholds.break: null`), and write the JSON report that `test-gates mutation` judges.
@@ -19,4 +21,4 @@ declare const DEFAULT_TIMEOUT_MS = 30000;
  */
 declare function createStrykerGatesConfig(options?: StrykerGatesOptions): Record<string, unknown>;
 
-export { DEFAULT_TIMEOUT_MS, type StrykerGatesOptions, createStrykerGatesConfig };
+export { DEFAULT_MAX_TEST_RUNNER_REUSE, type StrykerGatesOptions, createStrykerGatesConfig };

package/dist/stryker.js

@@ -7,7 +7,7 @@ import {
 
 // src/stryker.ts
 var WEAKENING_OPTIONS = ["ignoreStatic", "ignorers"];
-var DEFAULT_TIMEOUT_MS = 3e4;
+var DEFAULT_MAX_TEST_RUNNER_REUSE = 50;
 function createStrykerGatesConfig(options = {}) {
   const { rootDir: rootDirOption, ...overrides } = options;
   for (const option of WEAKENING_OPTIONS) {
@@ -32,7 +32,7 @@ function createStrykerGatesConfig(options = {}) {
     incremental: true,
     incrementalFile: "reports/stryker-incremental.json",
     tempDirName: ".stryker-tmp",
-    timeoutMS: DEFAULT_TIMEOUT_MS,
+    maxTestRunnerReuse: DEFAULT_MAX_TEST_RUNNER_REUSE,
     clearTextReporter: { logTests: false, reportTests: false },
     ...overrides,
     mutate: manifest.gates.map((gate) => gate.path),
@@ -42,6 +42,6 @@ function createStrykerGatesConfig(options = {}) {
   };
 }
 export {
-  DEFAULT_TIMEOUT_MS,
+  DEFAULT_MAX_TEST_RUNNER_REUSE,
   createStrykerGatesConfig
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@t09tanaka/test-gates",
-  "version": "0.3.1",
+  "version": "0.3.3",
   "description": "Test gates for the code that must not be wrong: per-file 100% coverage as the floor, mutation testing with a per-mutant allow list as the real check.",
   "author": "t09tanaka",
   "license": "MIT",