@t09tanaka/test-gates 0.4.0 → 0.4.1

package/CHANGELOG.md

@@ -1,5 +1,12 @@
 # Changelog
 
+## 0.4.1
+
+### Fixed
+
+- `test-gates mutation --affected` reuses every result after no change also where it cannot tell what a gate depends on (a file a gate or a spec loads reads files at run time, for one). Before, such a subproject ran every mutant on every run with the option, while a run without it reused the results after no change. The first line says so; a change still runs every mutant.
+- `test-gates doctor` reports what keeps `--affected` from telling the gates apart: as a problem when a hook relies on `--affected`, else as a note. In such a subproject it no longer asks for `--affected` in a hook. Before, it said OK although a run with `--affected` was always one of every mutant.
+
 ## 0.4.0
 
 Most of this release is about runs that were full ones although little had changed. `test-gates doctor` tells why a project's runs are full ones. `mutation --affected` runs again only the gates a change can affect. The comparison before a run covers only what the run loads. Two changes of behaviour: Stryker's `concurrency` now defaults to half of the cores, and the record of a run has a new format, so the first run after the update is one of every mutant.

package/README.md

@@ -318,7 +318,7 @@ A change elsewhere decides for every gate:
 - **Every mutant is run** after a change of a config (the Stryker config and the config of the test runner, what they import, the setup files and other files they name, `package.json`, `test-gates.json`, `tsconfig*.json`, `*.config.*`, Babel configs, dotfiles, declaration files `.d.ts`, anything below `__mocks__`), of a lock file or of what is installed, of the tools or of the arguments; after a change of a file at the top of the subproject or outside of it that no gate depends on; and after a change of a file whose kind is not known (`.yaml`, `.html` and the like) that no gate depends on. Such a file that a gate does depend on runs that gate again, as any other file does.
 - **Nothing is run again** for a change of a source, data, a snapshot, a style or an image below the subproject that no gate depends on, or of a document (`.md`, `.txt` and the like).
 
-When what a gate depends on cannot be told from the text, every mutant is run, and the first line says why: an import whose path is not written out (`import(name)`, `require(name)`), an import that leads to no file, a specifier that is neither a file nor an installed package (an alias defined only in a config: name it in `settings.importAliases`), a file read at run time (`node:fs`, `readFileSync`, `__dirname`, `import.meta.url`, `process.cwd`, `toMatchFileSnapshot`), a file that is loaded but not compared (git ignores it), a gate loaded for every spec by a setup file, a package of the same repository linked into `node_modules` (`@acme/shared`, also when it lies in the subproject), a component (`.vue`, `.svelte`, `.astro`, `.mdx`), which can use others that it does not import (registered by a plugin), and a test runner other than Jest and Vitest. Every mutant is also run when there is no record of a run with `--affected`, when the record cannot be read, with `--force`, and **when `CI` is set**: CI keeps running every mutant, and is the final check.
+When what a gate depends on cannot be told from the text, a change runs every mutant, and the first line says why. After no change at all the results are reused as they are without the option, so `--affected` is never slower than a run without it. Gate by gate it cannot tell when it meets an import whose path is not written out (`import(name)`, `require(name)`), an import that leads to no file, a specifier that is neither a file nor an installed package (an alias defined only in a config: name it in `settings.importAliases`), a file read at run time (`node:fs`, `readFileSync`, `__dirname`, `import.meta.url`, `process.cwd`, `toMatchFileSnapshot`), a file that is loaded but not compared (git ignores it), a gate loaded for every spec by a setup file, a package of the same repository linked into `node_modules` (`@acme/shared`, also when it lies in the subproject), a component (`.vue`, `.svelte`, `.astro`, `.mdx`), which can use others that it does not import (registered by a plugin), and a test runner other than Jest and Vitest. Every mutant is also run when there is no record of a run with `--affected`, when the record cannot be read, with `--force`, and **when `CI` is set**: CI keeps running every mutant, and is the final check.
 
 A gate with a survivor or a timeout that the allow lists do not cover is not reused: it runs again until it is clean. A mutant that timed out in the last run is never taken over either, also not in a gate whose other results are, and also not in a run without `--affected` after no change: on a loaded machine a mutant that would survive can time out instead, and a reused timeout would keep that. Those mutants alone are run again (`--mutate` with their ranges); the first lines count them. The run is judged as one of every gate, and its first lines say what was reused:
 
@@ -347,7 +347,7 @@ problem: .husky/pre-push runs test-gates mutation without --affected: a change o
 test-gates doctor: 1 problem(s) that make runs full ones
 ```
 
-A **problem** makes runs full ones that need not be: Stryker not in incremental mode, a path that turns reuse off (one that leads outside of the repository to something, or a Stryker config that cannot be loaded), a path that makes every file of the repository part of the comparison (it names the file and the path), a Stryker config that sets `force`, and a hook that runs `test-gates mutation` of the subproject without `--affected`, also through its `test:gates:mutation` script. A run in a hook is taken to be one of the subproject that its line names (`--dir`, `--prefix`, `-w`, `cd`), else the nearest `cd` before it; a name with a variable in it (`cd "$project"` in a loop over the subprojects), or none, may be one of any subproject. Each run is judged on its own line, and comments are left out. A **note** is worth knowing: that every file of the subproject is searched for paths, and why (a runner config that is not made with the helper, an import whose path is not written out), and that no hook runs a mutation run. The exit code is 1 when there is a problem, so that the doctor can run in CI next to the other checks.
+A **problem** makes runs full ones that need not be: Stryker not in incremental mode, a path that turns reuse off (one that leads outside of the repository to something, or a Stryker config that cannot be loaded), a path that makes every file of the repository part of the comparison (it names the file and the path), a Stryker config that sets `force`, a hook that runs `test-gates mutation` of the subproject without `--affected`, also through its `test:gates:mutation` script, and, when a hook relies on `--affected`, what keeps it from telling the gates apart (a file a gate or a spec loads that reads files at run time, for one). Where it cannot tell them apart, a hook without `--affected` is not taken for a problem, and what keeps it from doing so is a note. A run in a hook is taken to be one of the subproject that its line names (`--dir`, `--prefix`, `-w`, `cd`), else the nearest `cd` before it; a name with a variable in it (`cd "$project"` in a loop over the subprojects), or none, may be one of any subproject. Each run is judged on its own line, and comments are left out. A **note** is worth knowing: that every file of the subproject is searched for paths, and why (a runner config that is not made with the helper, an import whose path is not written out), and that no hook runs a mutation run. The exit code is 1 when there is a problem, so that the doctor can run in CI next to the other checks.
 
 ### `test-gates mutation-result [--report <path>]`
 

package/dist/cli.js

@@ -1697,6 +1697,13 @@ function diagnose(facts) {
       fix: "write such a path from the directory of its file (path.resolve(__dirname, \u2026)) so that it can be followed, or keep it out of the files the run loads"
     });
   }
+  if (facts.blind !== null) {
+    findings.push({
+      level: facts.hooks.some((hook) => hook.affected) ? "problem" : "note",
+      text: `with --affected, a change of any file runs every mutant: what a gate depends on cannot be told (${facts.blind})`,
+      fix: "keep what reads files or loads modules by a computed name out of what the gates and their specs import, or run without --affected"
+    });
+  }
   if (facts.scope !== null && "every" in facts.scope) {
     findings.push({
       level: "note",
@@ -1711,7 +1718,7 @@ function diagnose(facts) {
       fix: null
     });
   }
-  for (const hook of facts.hooks.filter((one) => !one.affected)) {
+  for (const hook of facts.hooks.filter((one) => !one.affected && facts.blind === null)) {
     findings.push({
       level: "problem",
       text: `${hook.file} runs test-gates mutation without --affected: a change of any file of the subproject runs every mutant`,
@@ -2297,12 +2304,6 @@ function snapshotOf(input) {
   return { snapshot: { mode, environment, args: input.args, files }, root: listRoot, widened };
 }
 
-// src/commands/mutation.ts
-import { spawnSync as spawnSync2 } from "child_process";
-import fs8 from "fs";
-import path8 from "path";
-import { pathToFileURL } from "url";
-
 // src/affected-files.ts
 import fs6 from "fs";
 import { builtinModules as builtinModules2 } from "module";
@@ -2762,6 +2763,12 @@ function withoutComments(text) {
   return out;
 }
 
+// src/commands/mutation.ts
+import { spawnSync as spawnSync2 } from "child_process";
+import fs8 from "fs";
+import path8 from "path";
+import { pathToFileURL } from "url";
+
 // src/commands/mutation-result.ts
 import fs7 from "fs";
 import path7 from "path";
@@ -3186,6 +3193,7 @@ function decideReuse(manifest, config, strykerArgs, part, affected) {
     plan: null,
     last: void 0,
     ci: affected ? affectedRefused(process.env) : null,
+    unchanged: null,
     configs
   };
   if (loaded !== void 0 && !options.incremental) {
@@ -3287,7 +3295,12 @@ function decideReuse(manifest, config, strykerArgs, part, affected) {
           plan = { full: `what a gate depends on cannot be told: ${error.message}` };
         }
         if ("full" in plan) {
-          force = plan.full;
+          if (refuseReuse({ record: stored, incremental, snapshot }) === null) {
+            decision.unchanged = plan.full;
+            plan = null;
+          } else {
+            force = plan.full;
+          }
         }
       }
     }
@@ -3342,6 +3355,11 @@ function guardIncremental(manifest, config, strykerArgs, part, affected) {
       said = describePlan({ full: force });
     } else if (plan !== null) {
       said = describePlan(plan, 5, ranges.length);
+    } else if (decision.unchanged !== null) {
+      said = [
+        `test-gates mutation --affected: nothing changed since the last run, every result is reused (gate by gate, ${decision.unchanged.replace(/^what a gate/, "what each gate")})`,
+        ...ranges.length > 0 ? [`  and ${ranges.length} mutant(s) that timed out in the last run are run again`] : []
+      ];
     }
   } else if (ranges.length > 0) {
     said = [
@@ -3562,11 +3580,30 @@ function runDoctor(dir, io2) {
         `  next run with --affected: ${first.replace(/^test-gates mutation --affected: /, "")}`
       );
       rest.forEach((line) => io2.out(`  ${line}`));
+    } else if (affected.unchanged !== null) {
+      io2.out("  next run with --affected: reuses the results of the last run (nothing changed)");
     } else {
       const why = affected.forced ? "the Stryker config sets force" : affected.force;
       io2.out(`  next run with --affected: every mutant, ${why ?? "there is nothing to reuse"}`);
     }
   }
+  let blind = null;
+  if (plain2.snapshot !== null) {
+    try {
+      affectedInputs(manifest, {
+        root: plain2.root,
+        files: plain2.snapshot.files,
+        configs: plain2.configs(),
+        tests: [],
+        covering: {}
+      });
+    } catch (error) {
+      if (!(error instanceof AffectedError)) {
+        throw error;
+      }
+      blind = error.message;
+    }
+  }
   const { lines, status } = formatFindings(
     diagnose({
       incremental: plain2.incremental,
@@ -3574,7 +3611,8 @@ function runDoctor(dir, io2) {
       failure: plain2.failure,
       widened: plain2.widened,
       scope: plain2.scope,
-      hooks: hooksOf(fs9.realpathSync(manifest.dir))
+      hooks: hooksOf(fs9.realpathSync(manifest.dir)),
+      blind
     })
   );
   lines.forEach((line) => io2.out(line));

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@t09tanaka/test-gates",
-  "version": "0.4.0",
+  "version": "0.4.1",
   "description": "Test gates for the code that must not be wrong: per-file 100% coverage as the floor, mutation testing with a per-mutant allow list as the real check.",
   "author": "t09tanaka",
   "license": "MIT",