@t09tanaka/test-gates 0.3.1 → 0.3.2

package/CHANGELOG.md

@@ -1,5 +1,14 @@
 # Changelog
 
+## 0.3.2
+
+### Changed
+
+- `createStrykerGatesConfig` sets Stryker's `maxTestRunnerReuse` to 50 unless the project passes its own (Stryker's default is 0: one test runner process for the whole run). In a project with mutants that never end, the runner became about three times slower from the first of them on, and mutants that are killed in seconds on their own timed out. Starting a new process every 50 mutants kept the speed: a full run of 781 mutants went from 14 to 18 minutes with stray timeouts to under 7 minutes with none.
+- `timeoutMS` is no longer set by the helper; Stryker's default (5000) applies again. The 30000 of 0.3.1 was the wrong fix. It rested on the idea that a restarted runner makes the next mutant slow, when in fact a runner that is not restarted stays slow. With 30000 the same full run took 47 minutes and still failed, and mutants that exhaust the heap crashed the runner and were reported as `RuntimeError` instead of `Timeout`.
+
+A project that set `timeoutMS` itself to work around stray timeouts can remove it.
+
 ## 0.3.1
 
 ### Changed

package/README.md

@@ -280,7 +280,9 @@ How an `expectedTimeouts` entry is judged:
 
 When the run fails on a timeout, first run again with less load (`npm run test:gates:mutation -- --concurrency 1`). Add an entry only for a mutant that can never finish.
 
-`createStrykerGatesConfig` sets Stryker's `timeoutMS` to 30000 (since 0.3.1; Stryker's own default is 5000). After a mutant that really never ends, Stryker starts a new test runner, and with 5000 the mutant that comes next can run out of time while that runner is still starting: in one project with 11 such mutants, 2 to 7 others timed out on every full run, a different set each time, and none in a run with 30000. A longer limit hides nothing, because a timeout that is not listed fails the run anyway. The price is the wait for each mutant that hangs without tripping Stryker's loop counter. Pass `timeoutMS` to the helper to change it.
+`createStrykerGatesConfig` sets Stryker's `maxTestRunnerReuse` to 50 (since 0.3.2; Stryker's own default is 0, one test runner process for the whole run). A mutant that would be killed can also time out for a reason that has nothing to do with the mutant: in a Jest project with three mutants that never end, the test runner handled about a third as many mutants per minute from the first of them on, for the rest of the run. Mutants that are killed in seconds on their own then timed out, a different set on each full run, and the run took 14 to 18 minutes. With a new process every 50 mutants the speed stayed the same to the end, nothing timed out that should not, and the run took under 7 minutes. Why the runner slows down was not found; the restart only cuts it short. Pass `maxTestRunnerReuse` to the helper to change it.
+
+0.3.1 set `timeoutMS` to 30000 instead. That was the wrong fix and 0.3.2 takes it back: the longer limit let most slow mutants finish, but the run above took 47 minutes, and in another project mutants that exhaust the heap crashed the test runner before the limit and came back as `RuntimeError`, which is only counted, not failed. `timeoutMS` is Stryker's default (5000) again unless the project sets it.
 
 ### `test-gates selfcheck [--all | --first] [-- <gate command>]`
 
@@ -462,7 +464,7 @@ Why these choices:
 - Jest measures with istanbul (`babel`), because the v8 provider does not count the untaken side of an `if` without `else`.
 - Jest fails on a `coverageThreshold` path key that matches no file, so a typo in `test-gates.json` cannot pass. Vitest does the opposite: a glob or path key in `thresholds` that matches nothing passes. The Vitest helper therefore narrows `coverage.include` to the gates and uses `perFile`.
 - Stryker never fails on its own score (`break: null`). The verdict comes from `test-gates mutation`, which knows the allow list.
-- The Stryker helper also sets defaults that can be overridden: `coverageAnalysis: 'perTest'`, `incremental` with its file under `reports/`, and `timeoutMS: 30000` (see [Timeouts](#timeouts)).
+- The Stryker helper also sets defaults that can be overridden: `coverageAnalysis: 'perTest'`, `incremental` with its file under `reports/`, and `maxTestRunnerReuse: 50` (see [Timeouts](#timeouts)).
 
 `loadGates(dir)` returns the validated manifest: `gates` (each with its resolved `spec`), `candidates` and the resolved `settings`. It throws when the manifest has any problem.
 

package/dist/stryker.cjs

@@ -30,7 +30,7 @@ var __toCommonJS = (mod) => __copyProps(__defProp({}, "__esModule", { value: tru
 // src/stryker.ts
 var stryker_exports = {};
 __export(stryker_exports, {
-  DEFAULT_TIMEOUT_MS: () => DEFAULT_TIMEOUT_MS,
+  DEFAULT_MAX_TEST_RUNNER_REUSE: () => DEFAULT_MAX_TEST_RUNNER_REUSE,
   createStrykerGatesConfig: () => createStrykerGatesConfig
 });
 module.exports = __toCommonJS(stryker_exports);
@@ -602,7 +602,7 @@ function resolveRootDir(rootDir) {
 
 // src/stryker.ts
 var WEAKENING_OPTIONS = ["ignoreStatic", "ignorers"];
-var DEFAULT_TIMEOUT_MS = 3e4;
+var DEFAULT_MAX_TEST_RUNNER_REUSE = 50;
 function createStrykerGatesConfig(options = {}) {
   const { rootDir: rootDirOption, ...overrides } = options;
   for (const option of WEAKENING_OPTIONS) {
@@ -627,7 +627,7 @@ function createStrykerGatesConfig(options = {}) {
     incremental: true,
     incrementalFile: "reports/stryker-incremental.json",
     tempDirName: ".stryker-tmp",
-    timeoutMS: DEFAULT_TIMEOUT_MS,
+    maxTestRunnerReuse: DEFAULT_MAX_TEST_RUNNER_REUSE,
     clearTextReporter: { logTests: false, reportTests: false },
     ...overrides,
     mutate: manifest.gates.map((gate) => gate.path),
@@ -638,6 +638,6 @@ function createStrykerGatesConfig(options = {}) {
 }
 // Annotate the CommonJS export names for ESM import in node:
 0 && (module.exports = {
-  DEFAULT_TIMEOUT_MS,
+  DEFAULT_MAX_TEST_RUNNER_REUSE,
   createStrykerGatesConfig
 });

package/dist/stryker.d.cts

@@ -5,11 +5,13 @@ interface StrykerGatesOptions {
     [option: string]: unknown;
 }
 /**
- * Stryker's own default is 5000. After a mutant that really never ends, Stryker starts a new
- * test runner, and the next mutant then pays that start-up inside its time limit: with 5000 a
- * mutant that would be killed is reported as `Timeout`, which fails the run since 0.3.0.
+ * Stryker's own default is 0: one test runner process for the whole run. Measured on a Jest
+ * project with three mutants that never end (`Hit limit reached`): from the first of them on
+ * the runner handled about a third as many mutants per minute for the rest of the run, and
+ * mutants that are killed in seconds on their own ran out of time. With a new process every 50
+ * mutants the speed stayed the same to the end.
  */
-declare const DEFAULT_TIMEOUT_MS = 30000;
+declare const DEFAULT_MAX_TEST_RUNNER_REUSE = 50;
 /**
  * Stryker config for the gates: mutate exactly the gates, never fail on the score
  * (`thresholds.break: null`), and write the JSON report that `test-gates mutation` judges.
@@ -19,4 +21,4 @@ declare const DEFAULT_TIMEOUT_MS = 30000;
  */
 declare function createStrykerGatesConfig(options?: StrykerGatesOptions): Record<string, unknown>;
 
-export { DEFAULT_TIMEOUT_MS, type StrykerGatesOptions, createStrykerGatesConfig };
+export { DEFAULT_MAX_TEST_RUNNER_REUSE, type StrykerGatesOptions, createStrykerGatesConfig };

package/dist/stryker.d.ts

@@ -5,11 +5,13 @@ interface StrykerGatesOptions {
     [option: string]: unknown;
 }
 /**
- * Stryker's own default is 5000. After a mutant that really never ends, Stryker starts a new
- * test runner, and the next mutant then pays that start-up inside its time limit: with 5000 a
- * mutant that would be killed is reported as `Timeout`, which fails the run since 0.3.0.
+ * Stryker's own default is 0: one test runner process for the whole run. Measured on a Jest
+ * project with three mutants that never end (`Hit limit reached`): from the first of them on
+ * the runner handled about a third as many mutants per minute for the rest of the run, and
+ * mutants that are killed in seconds on their own ran out of time. With a new process every 50
+ * mutants the speed stayed the same to the end.
  */
-declare const DEFAULT_TIMEOUT_MS = 30000;
+declare const DEFAULT_MAX_TEST_RUNNER_REUSE = 50;
 /**
  * Stryker config for the gates: mutate exactly the gates, never fail on the score
  * (`thresholds.break: null`), and write the JSON report that `test-gates mutation` judges.
@@ -19,4 +21,4 @@ declare const DEFAULT_TIMEOUT_MS = 30000;
  */
 declare function createStrykerGatesConfig(options?: StrykerGatesOptions): Record<string, unknown>;
 
-export { DEFAULT_TIMEOUT_MS, type StrykerGatesOptions, createStrykerGatesConfig };
+export { DEFAULT_MAX_TEST_RUNNER_REUSE, type StrykerGatesOptions, createStrykerGatesConfig };

package/dist/stryker.js

@@ -7,7 +7,7 @@ import {
 
 // src/stryker.ts
 var WEAKENING_OPTIONS = ["ignoreStatic", "ignorers"];
-var DEFAULT_TIMEOUT_MS = 3e4;
+var DEFAULT_MAX_TEST_RUNNER_REUSE = 50;
 function createStrykerGatesConfig(options = {}) {
   const { rootDir: rootDirOption, ...overrides } = options;
   for (const option of WEAKENING_OPTIONS) {
@@ -32,7 +32,7 @@ function createStrykerGatesConfig(options = {}) {
     incremental: true,
     incrementalFile: "reports/stryker-incremental.json",
     tempDirName: ".stryker-tmp",
-    timeoutMS: DEFAULT_TIMEOUT_MS,
+    maxTestRunnerReuse: DEFAULT_MAX_TEST_RUNNER_REUSE,
     clearTextReporter: { logTests: false, reportTests: false },
     ...overrides,
     mutate: manifest.gates.map((gate) => gate.path),
@@ -42,6 +42,6 @@ function createStrykerGatesConfig(options = {}) {
   };
 }
 export {
-  DEFAULT_TIMEOUT_MS,
+  DEFAULT_MAX_TEST_RUNNER_REUSE,
   createStrykerGatesConfig
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@t09tanaka/test-gates",
-  "version": "0.3.1",
+  "version": "0.3.2",
   "description": "Test gates for the code that must not be wrong: per-file 100% coverage as the floor, mutation testing with a per-mutant allow list as the real check.",
   "author": "t09tanaka",
   "license": "MIT",