@t09tanaka/test-gates 0.3.0 → 0.3.2

package/CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## 0.3.2
+
+### Changed
+
+- `createStrykerGatesConfig` sets Stryker's `maxTestRunnerReuse` to 50 unless the project passes its own (Stryker's default is 0: one test runner process for the whole run). In a project with mutants that never end, the runner became about three times slower from the first of them on, and mutants that are killed in seconds on their own timed out. Starting a new process every 50 mutants kept the speed: a full run of 781 mutants went from 14 to 18 minutes with stray timeouts to under 7 minutes with none.
+- `timeoutMS` is no longer set by the helper; Stryker's default (5000) applies again. The 30000 of 0.3.1 was the wrong fix. It rested on the idea that a restarted runner makes the next mutant slow, when in fact a runner that is not restarted stays slow. With 30000 the same full run took 47 minutes and still failed, and mutants that exhaust the heap crashed the runner and were reported as `RuntimeError` instead of `Timeout`.
+
+A project that set `timeoutMS` itself to work around stray timeouts can remove it.
+
+## 0.3.1
+
+### Changed
+
+- `createStrykerGatesConfig` sets Stryker's `timeoutMS` to 30000 unless the project passes its own (Stryker's default is 5000). With `maxTimeouts` at 0 since 0.3.0, a mutant that would be killed but ran out of time fails the run. That happened on every full run in two projects with mutants that really never end: after each of them Stryker starts a new test runner, and the next mutant timed out while it was starting. A full run takes longer by up to 25 seconds for each mutant that hangs without tripping Stryker's loop counter.
+
 ## 0.3.0
 
 Three defaults move to the strict side. A project that passed with 0.2.0 can fail with 0.3.0 without any change of its own; each case below says what to do.

package/README.md

@@ -280,6 +280,10 @@ How an `expectedTimeouts` entry is judged:
 
 When the run fails on a timeout, first run again with less load (`npm run test:gates:mutation -- --concurrency 1`). Add an entry only for a mutant that can never finish.
 
+`createStrykerGatesConfig` sets Stryker's `maxTestRunnerReuse` to 50 (since 0.3.2; Stryker's own default is 0, one test runner process for the whole run). A mutant that would be killed can also time out for a reason that has nothing to do with the mutant: in a Jest project with three mutants that never end, the test runner handled about a third as many mutants per minute from the first of them on, for the rest of the run. Mutants that are killed in seconds on their own then timed out, a different set on each full run, and the run took 14 to 18 minutes. With a new process every 50 mutants the speed stayed the same to the end, nothing timed out that should not, and the run took under 7 minutes. Why the runner slows down was not found; the restart only cuts it short. Pass `maxTestRunnerReuse` to the helper to change it.
+
+0.3.1 set `timeoutMS` to 30000 instead. That was the wrong fix and 0.3.2 takes it back: the longer limit let most slow mutants finish, but the run above took 47 minutes, and in another project mutants that exhaust the heap crashed the test runner before the limit and came back as `RuntimeError`, which is only counted, not failed. `timeoutMS` is Stryker's default (5000) again unless the project sets it.
+
 ### `test-gates selfcheck [--all | --first] [-- <gate command>]`
 
 Negative control. For each gate, runs the gate with that gate's spec left out and requires the run to **fail on the coverage threshold of that very file**. A gate that stays green without its spec measures nothing; a run that fails for another reason (a config that does not load) proves nothing either.
@@ -460,6 +464,7 @@ Why these choices:
 - Jest measures with istanbul (`babel`), because the v8 provider does not count the untaken side of an `if` without `else`.
 - Jest fails on a `coverageThreshold` path key that matches no file, so a typo in `test-gates.json` cannot pass. Vitest does the opposite: a glob or path key in `thresholds` that matches nothing passes. The Vitest helper therefore narrows `coverage.include` to the gates and uses `perFile`.
 - Stryker never fails on its own score (`break: null`). The verdict comes from `test-gates mutation`, which knows the allow list.
+- The Stryker helper also sets defaults that can be overridden: `coverageAnalysis: 'perTest'`, `incremental` with its file under `reports/`, and `maxTestRunnerReuse: 50` (see [Timeouts](#timeouts)).
 
 `loadGates(dir)` returns the validated manifest: `gates` (each with its resolved `spec`), `candidates` and the resolved `settings`. It throws when the manifest has any problem.
 

package/dist/stryker.cjs

@@ -30,6 +30,7 @@ var __toCommonJS = (mod) => __copyProps(__defProp({}, "__esModule", { value: tru
 // src/stryker.ts
 var stryker_exports = {};
 __export(stryker_exports, {
+  DEFAULT_MAX_TEST_RUNNER_REUSE: () => DEFAULT_MAX_TEST_RUNNER_REUSE,
   createStrykerGatesConfig: () => createStrykerGatesConfig
 });
 module.exports = __toCommonJS(stryker_exports);
@@ -601,6 +602,7 @@ function resolveRootDir(rootDir) {
 
 // src/stryker.ts
 var WEAKENING_OPTIONS = ["ignoreStatic", "ignorers"];
+var DEFAULT_MAX_TEST_RUNNER_REUSE = 50;
 function createStrykerGatesConfig(options = {}) {
   const { rootDir: rootDirOption, ...overrides } = options;
   for (const option of WEAKENING_OPTIONS) {
@@ -625,6 +627,7 @@ function createStrykerGatesConfig(options = {}) {
     incremental: true,
     incrementalFile: "reports/stryker-incremental.json",
     tempDirName: ".stryker-tmp",
+    maxTestRunnerReuse: DEFAULT_MAX_TEST_RUNNER_REUSE,
     clearTextReporter: { logTests: false, reportTests: false },
     ...overrides,
     mutate: manifest.gates.map((gate) => gate.path),
@@ -635,5 +638,6 @@ function createStrykerGatesConfig(options = {}) {
 }
 // Annotate the CommonJS export names for ESM import in node:
 0 && (module.exports = {
+  DEFAULT_MAX_TEST_RUNNER_REUSE,
   createStrykerGatesConfig
 });

package/dist/stryker.d.cts

@@ -4,6 +4,14 @@ interface StrykerGatesOptions {
     /** Any other Stryker option: testRunner, jest / vitest, concurrency, ignorePatterns, … */
     [option: string]: unknown;
 }
+/**
+ * Stryker's own default is 0: one test runner process for the whole run. Measured on a Jest
+ * project with three mutants that never end (`Hit limit reached`): from the first of them on
+ * the runner handled about a third as many mutants per minute for the rest of the run, and
+ * mutants that are killed in seconds on their own ran out of time. With a new process every 50
+ * mutants the speed stayed the same to the end.
+ */
+declare const DEFAULT_MAX_TEST_RUNNER_REUSE = 50;
 /**
  * Stryker config for the gates: mutate exactly the gates, never fail on the score
  * (`thresholds.break: null`), and write the JSON report that `test-gates mutation` judges.
@@ -13,4 +21,4 @@ interface StrykerGatesOptions {
  */
 declare function createStrykerGatesConfig(options?: StrykerGatesOptions): Record<string, unknown>;
 
-export { type StrykerGatesOptions, createStrykerGatesConfig };
+export { DEFAULT_MAX_TEST_RUNNER_REUSE, type StrykerGatesOptions, createStrykerGatesConfig };

package/dist/stryker.d.ts

@@ -4,6 +4,14 @@ interface StrykerGatesOptions {
     /** Any other Stryker option: testRunner, jest / vitest, concurrency, ignorePatterns, … */
     [option: string]: unknown;
 }
+/**
+ * Stryker's own default is 0: one test runner process for the whole run. Measured on a Jest
+ * project with three mutants that never end (`Hit limit reached`): from the first of them on
+ * the runner handled about a third as many mutants per minute for the rest of the run, and
+ * mutants that are killed in seconds on their own ran out of time. With a new process every 50
+ * mutants the speed stayed the same to the end.
+ */
+declare const DEFAULT_MAX_TEST_RUNNER_REUSE = 50;
 /**
  * Stryker config for the gates: mutate exactly the gates, never fail on the score
  * (`thresholds.break: null`), and write the JSON report that `test-gates mutation` judges.
@@ -13,4 +21,4 @@ interface StrykerGatesOptions {
  */
 declare function createStrykerGatesConfig(options?: StrykerGatesOptions): Record<string, unknown>;
 
-export { type StrykerGatesOptions, createStrykerGatesConfig };
+export { DEFAULT_MAX_TEST_RUNNER_REUSE, type StrykerGatesOptions, createStrykerGatesConfig };

package/dist/stryker.js

@@ -7,6 +7,7 @@ import {
 
 // src/stryker.ts
 var WEAKENING_OPTIONS = ["ignoreStatic", "ignorers"];
+var DEFAULT_MAX_TEST_RUNNER_REUSE = 50;
 function createStrykerGatesConfig(options = {}) {
   const { rootDir: rootDirOption, ...overrides } = options;
   for (const option of WEAKENING_OPTIONS) {
@@ -31,6 +32,7 @@ function createStrykerGatesConfig(options = {}) {
     incremental: true,
     incrementalFile: "reports/stryker-incremental.json",
     tempDirName: ".stryker-tmp",
+    maxTestRunnerReuse: DEFAULT_MAX_TEST_RUNNER_REUSE,
     clearTextReporter: { logTests: false, reportTests: false },
     ...overrides,
     mutate: manifest.gates.map((gate) => gate.path),
@@ -40,5 +42,6 @@ function createStrykerGatesConfig(options = {}) {
   };
 }
 export {
+  DEFAULT_MAX_TEST_RUNNER_REUSE,
   createStrykerGatesConfig
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@t09tanaka/test-gates",
-  "version": "0.3.0",
+  "version": "0.3.2",
   "description": "Test gates for the code that must not be wrong: per-file 100% coverage as the floor, mutation testing with a per-mutant allow list as the real check.",
   "author": "t09tanaka",
   "license": "MIT",