@t09tanaka/test-gates 0.4.3 → 0.5.1

package/CHANGELOG.md

@@ -1,5 +1,26 @@
 # Changelog
 
+## 0.5.1
+
+### Fixed
+
+- `selfcheck --jobs` no longer keeps every other control apart because the word `require` appears inside a string, a template, a comment or a regular expression (`'sslmode=require'`, `// pass it to require`). Only code is looked at now. Handing `require` on under another name (`const r = require`) still counts as loading a module whose name is computed.
+
+## 0.5.0
+
+This release makes the gates quicker to check in CI: `test-gates mutation --shard <index>/<total>` splits the mutation run over parallel jobs, and `selfcheck --jobs` runs more controls side by side. Nothing changes for a run without the new option:
+
+- `--shard` deals the gates out in the same way as `selfcheck --shard`. A passing part says nothing about the others, so run every part and make each one a required check.
+- `selfcheck --jobs` judges every control as before. Only the reason that kept a control apart because of an import of `path`, `url`, `crypto` and the like is gone.
+
+### Added
+
+- `test-gates mutation --shard <index>/<total>` runs the mutants of one part of the gates, so that the mutation run can be split over parallel CI jobs. The assignment is the one of `selfcheck --shard`: the gates are dealt out in turn in the order of `test-gates.json`, so the same `--shard i/n` takes the same gates in both commands. More parts than gates is an error. One passing part is not a guarantee for the whole: run every part (the number of matrix entries has to be `<total>`) and make each one a required check. Unlike `--gate` it is accepted when `CI` is set, and it cannot be combined with `--gate`, `--affected` or `--mutate`.
+
+### Changed
+
+- `selfcheck --jobs` no longer keeps a control apart because a file its run loads imports `path`, `url`, `querystring`, `string_decoder`, `punycode`, `assert`, `util/types` or `crypto` (with or without `node:`, and the subpaths `path/posix`, `path/win32` and `assert/strict`). They only compute. A use of `crypto.setFips`, `setEngine`, `setDefaultEncoding` or `DEFAULT_ENCODING` still counts. Every other built-in module (`fs`, `os`, `util`, `events`, `buffer`, `timers/promises`, ...) keeps its controls apart as before, and a project cannot vouch for one by its name. `--jobs` is still 1 by default, and `--quick` is still refused when `CI` is set.
+
 ## 0.4.3
 
 This release spares more full runs of `test-gates mutation`: `--affected` keeps telling the gates apart where only some of them read files, a new linked worktree starts from the record of the main worktree, and timeouts on a loaded machine are run again before they fail the run. Three things to know when updating:

package/README.md

@@ -234,7 +234,7 @@ Static checks, before the gate run. Reports every violation as `file:line: reaso
 - The Stryker config does not use `excludedMutations`, `ignoreStatic` or `ignorers`.
 - A Vitest gate run below Vitest 4 does not use the plain v8 coverage provider (since 0.3.0), see [Vitest: v8 or istanbul](#vitest-v8-or-istanbul).
 
-### `test-gates mutation [--gate <path>…] [--affected] [-- <stryker arguments>]`
+### `test-gates mutation [--gate <path>… | --shard <index>/<total>] [--affected] [-- <stryker arguments>]`
 
 Runs `stryker run <config>` from the project's own `node_modules`, then judges the JSON report (see `mutation-result`). Unknown arguments, and everything after `--`, are handed to Stryker. If Stryker itself fails, the exit code is 1 and nothing is judged. A report from an earlier run is deleted first, so a run that writes none cannot pass on old results.
 
@@ -323,6 +323,34 @@ So a run with `--gate`
 
 `--gate` is written before `--`, and cannot be combined with a `--mutate` of your own.
 
+#### `--shard <index>/<total>`: the mutants of one part of the gates, for parallel CI jobs
+
+When the mutation run takes too long for one CI job, split it: `test-gates mutation --shard 2/4` runs the mutants of part 2 of 4 (the index starts at 1) and judges those gates. Unlike `--gate` it is for the final check, so **it is accepted when `CI` is set**. **One part that passes says nothing about the gates of the other parts**: the check is complete only when every part has run and passed.
+
+- The gates are dealt out in turn in the order of `test-gates.json`: the gate at position `k` (from 0) belongs to part `k % total + 1`. This is the assignment of `selfcheck --shard` too, so the same `--shard i/n` takes the same gates in both commands. The parts are disjoint and together every gate exactly once.
+- More parts than gates is an error (exit code 2), as in `selfcheck`: a part without a gate would pass without checking anything.
+- The gates of the part are handed to Stryker as `--mutate` and judged; the last line says which part it was (`Only part 2/4 (3 of 11 gate(s)) was run and judged (--shard)`). `--shard 1/1` is a run of every gate, and says so. Every mutant of the gates is run: the handling of an incremental file is the one of a run without `--shard`, which is not what `--gate` does. The timeouts of the part are run again one at a time (`--concurrency 1`) as in any run, and only for the gates of the part.
+- `<index>/<total>` outside `1 <= index <= total`, or not written as two plain numbers, is an error (exit code 2). `--shard` cannot be combined with `--gate`, `--affected` or a `--mutate` of your own, and is written before `--`.
+
+A matrix of a hosted CI, in parts:
+
+```yaml
+test-gates-mutation:
+  runs-on: ubuntu-latest
+  strategy:
+    fail-fast: false
+    matrix:
+      part: [1, 2, 3, 4]
+  defaults:
+    run:
+      working-directory: backend
+  steps:
+    # checkout, setup-node and npm ci as above
+    - run: npm run test:gates:mutation -- --shard ${{ matrix.part }}/4
+```
+
+The number of parts is written twice, in `matrix.part` and after the `/`: the two have to be the same, or some gates are run by no job. Make every part a required check (one for each entry of the matrix), so that no part can be left out.
+
 #### `--affected`: run again only the gates a change can affect
 
 Without the option, a change to any file that is compared makes the run one of every mutant. That is the safe answer, but in a pre-push hook that only runs for the subprojects a push touches, it is the answer every time. `test-gates mutation --affected` (`npm run test:gates:mutation -- --affected`) decides gate by gate instead: a gate whose results cannot have changed keeps them, and every mutant of a gate that may have changed is run again. It never relies on Stryker's judgement per mutant, which is what reused the old "Killed" in the measurements above.
@@ -519,7 +547,7 @@ _Before the runs start_, the gate command and its config must be known, and for
 A spec that reaches outside but loads no other gate, which is what the spec of a gate that needs a clock, a file or a package usually is, keeps no control from running next to others; it is left out of the run of its own gate. A file reaches outside when it
 
 - mentions `process.`, `globalThis`, `global.` or `import.meta.env`; the clock or a wait (`Date.now`, `new Date()`, `performance.now`, `setTimeout`, `setInterval`, `setImmediate`, `waitFor`, `waitUntil`, `expect.poll`, `retryTimes`, `retry:`); the network (`fetch(`, `WebSocket`, `XMLHttpRequest`); or `navigator`, `Atomics`, `WeakRef`, `FinalizationRegistry`, `eval(`, `Function(`,
-- imports a built-in module of Node.js, also one that only computes (`path`): with Jest the spec files of one worker share the objects of the built-in modules, so a setting that one spec changes there is seen by the next, and which spec comes next depends on Jest's cache of run times,
+- imports a built-in module of Node.js other than `path`, `url`, `querystring`, `string_decoder`, `punycode`, `assert`, `util/types` and `crypto` (these only compute, and a use of `crypto.setFips`, `setEngine`, `setDefaultEncoding` or `DEFAULT_ENCODING` still counts; a project cannot vouch for another built-in module by its name): with Jest the spec files of one worker share the objects of the built-in modules, so a setting that one spec changes there is seen by the next, and which spec comes next depends on Jest's cache of run times,
 - imports a package other than the runner's (`vitest`, `@jest/globals`). Packages are not looked into. With `settings.imports.mode: "allowlist"`, the packages that `settings.imports.allow` lists by name are taken too: the project already vouches for them as imports of a gate,
 - loads a module whose name is computed.
 

package/dist/cli.js

@@ -28,7 +28,7 @@ function parseShard(value) {
 }
 var OPTIONS_BY_COMMAND = {
   check: ["--dir"],
-  mutation: ["--dir", "--gate", "--affected"],
+  mutation: ["--dir", "--gate", "--shard", "--affected"],
   "mutation-result": ["--dir", "--report"],
   selfcheck: ["--dir", "--all", "--first", "--gate", "--shard", "--quick", "--jobs"],
   lcov: ["--dir", "--file"],
@@ -146,13 +146,18 @@ function parseArgs(argv) {
       "--affected and --gate cannot be combined: --gate runs the gates it names, --affected decides which gates run"
     );
   }
+  if (parsed.affected && parsed.shard !== null) {
+    throw new UsageError(
+      "--affected and --shard cannot be combined: --shard runs every mutant of its gates, --affected decides which gates run"
+    );
+  }
   const misplaced = parsed.rest.find((arg) => /^--(gate|shard|quick|jobs)(=|$)/.test(arg));
   if (parsed.command === "selfcheck" && misplaced !== void 0) {
     throw new UsageError(
       `${misplaced.split("=")[0]} after "--" would go to the gate command. Write it before "--"`
     );
   }
-  const forStryker = parsed.rest.find((arg) => /^--(gate|affected)(=|$)/.test(arg));
+  const forStryker = parsed.rest.find((arg) => /^--(gate|shard|affected)(=|$)/.test(arg));
   if (forStryker !== void 0) {
     throw new UsageError(
       `${forStryker.split("=")[0]} after "--" would go to Stryker. Write it before "--"`
@@ -1083,6 +1088,12 @@ function gateRunRefused(env) {
 function affectedRefused(env) {
   return inCi(env) ? "CI is set, and there every mutant is run (the final check)" : null;
 }
+function shardRunCaveat(shard, judged, all) {
+  if (shard.total === 1) {
+    return `. All ${all} gate(s) were run and judged (--shard 1/1 is the only part)`;
+  }
+  return `. Only part ${shard.index}/${shard.total} (${judged} of ${all} gate(s)) was run and judged (--shard): every part has to pass for the final check`;
+}
 function gateRunCaveat(judged, all) {
   return `. Only ${judged} of ${all} gate(s) were run and judged (--gate), and a change in one gate can make a mutant of another survive: run without --gate for the final check`;
 }
@@ -4326,10 +4337,10 @@ import path9 from "path";
 function reportPathOf(manifest, override) {
   return path9.resolve(manifest.dir, override ?? manifest.settings.strykerReportFile);
 }
-function runMutationResult(dir, reportOverride, io2, only = [], seen = () => void 0, retried = null) {
+function runMutationResult(dir, reportOverride, io2, only = [], seen = () => void 0, retried = null, shard = null) {
   const manifest = readManifest(dir);
   const gates = only.length === 0 ? manifest.gates : manifest.gates.filter((gate) => only.includes(gate.path));
-  const caveat = only.length === 0 ? "" : gateRunCaveat(gates.length, manifest.gates.length);
+  const caveat = only.length === 0 ? "" : shard === null ? gateRunCaveat(gates.length, manifest.gates.length) : shardRunCaveat(shard, gates.length, manifest.gates.length);
   const reportPath = reportPathOf(manifest, reportOverride);
   const shown = path9.relative(manifest.dir, reportPath) || reportPath;
   let report;
@@ -4879,11 +4890,11 @@ function guardIncremental(manifest, config, strykerArgs, part, affected, found)
     }
   }
 }
-function runMutation(dir, strykerArgs, io2, only = [], affected = false) {
+function runMutation(dir, strykerArgs, io2, only = [], affected = false, shard = null) {
   const part = only.length > 0;
-  if (affected && strykerArgs.some((arg) => /^(?:--mutate|-m)(?:=|$)/.test(arg))) {
+  if ((affected || shard !== null) && strykerArgs.some((arg) => /^(?:--mutate|-m)(?:=|$)/.test(arg))) {
     throw new UsageError(
-      "--affected and --mutate cannot be combined: --affected sets --mutate itself"
+      `${affected ? "--affected" : "--shard"} and --mutate cannot be combined: ${affected ? "--affected" : "--shard"} sets --mutate itself`
     );
   }
   if (part) {
@@ -4900,11 +4911,12 @@ function runMutation(dir, strykerArgs, io2, only = [], affected = false) {
     gates: manifest.gates.map((gate) => gate.path),
     mode: "all",
     only,
-    shard: null
+    shard
   });
   if ("error" in chosen) {
     throw new UsageError(chosen.error);
   }
+  const shardGates = shard === null ? null : chosen.selected;
   const config = findStrykerConfig(manifest);
   if (config === null) {
     throw new ManifestError(
@@ -4930,7 +4942,8 @@ function runMutation(dir, strykerArgs, io2, only = [], affected = false) {
   }
   removed.forEach((line) => io2.out(line));
   const guardArgs = guard.force === null && guard.mutate === null ? [] : ["--force"];
-  const partArgs = part ? ["--mutate", chosen.selected.join(",")] : guard.mutate !== null ? ["--mutate", guard.mutate.join(",")] : [];
+  const partArgs = part || shardGates !== null ? ["--mutate", (shardGates ?? chosen.selected).join(",")] : guard.mutate !== null ? ["--mutate", guard.mutate.join(",")] : [];
+  const judged = part ? chosen.selected : shardGates ?? [];
   const unmark = markRun(manifest.dir, guard.temp);
   let retried;
   try {
@@ -4954,24 +4967,25 @@ function runMutation(dir, strykerArgs, io2, only = [], affected = false) {
       [config, ...strykerArgs],
       manifest,
       reportFile,
-      part ? manifest.gates.filter((gate) => chosen.selected.includes(gate.path)) : manifest.gates,
+      judged.length > 0 ? manifest.gates.filter((gate) => judged.includes(gate.path)) : manifest.gates,
       io2
     );
   } finally {
     unmark();
   }
-  let judged = null;
+  let lastSeen = null;
   const status = runMutationResult(
     dir,
     null,
     io2,
-    part ? chosen.selected : [],
+    judged,
     (seen) => {
-      judged = seen;
+      lastSeen = seen;
     },
-    retried
+    retried,
+    shard
   );
-  guard.record(judged);
+  guard.record(lastSeen);
   if (part) {
     fs10.rmSync(reportFile, { force: true });
   }
@@ -5420,6 +5434,11 @@ var REACHES = [
   "WebSocket",
   "XMLHttpRequest",
   "navigator",
+  // What crypto keeps as a setting of its module, for every spec file of a worker after it.
+  "setFips",
+  "setEngine",
+  "setDefaultEncoding",
+  "DEFAULT_ENCODING",
   "Atomics",
   "WeakRef",
   "FinalizationRegistry",
@@ -5439,9 +5458,86 @@ function withoutStrings(source) {
     )
   );
 }
+var BEFORE_REGEX = /(?:^|[(,=:[!&|?{};+\-*%<>~^]|\b(?:return|typeof|case|in|of|void|delete|throw|new|yield|await))\s*$/;
+var COMMENT_OR_STRING = /\/\/[^\n]*|\/\*[\s\S]*?(?:\*\/|$)|(['"])(?:\\.|(?!\1)[^\\\n])*\1?/y;
+var REGEX = /\/(?:\\.|\[(?:\\.|[^\]\\\n])*\]?|[^/\\\n[])*\/?/y;
+var TEMPLATE_TEXT = /(?:\\[\s\S]|[^`\\$]|\$(?!\{))*/y;
+function withoutLiterals(source) {
+  let out = "";
+  let i = 0;
+  const blank = (text) => text.replace(/[^\n]/g, " ");
+  const enclosed = (literal) => {
+    const closed = literal.length > 1 && literal.endsWith(literal.charAt(0));
+    return literal.charAt(0) + blank(literal.slice(1, closed ? -1 : void 0)) + (closed ? literal.charAt(0) : "");
+  };
+  const take = (pattern) => {
+    pattern.lastIndex = i;
+    const found = pattern.exec(source)?.[0] ?? null;
+    if (found !== null) {
+      i += found.length;
+    }
+    return found;
+  };
+  const code = (inBraces) => {
+    while (i < source.length) {
+      const c = source[i];
+      const literal = take(COMMENT_OR_STRING);
+      if (literal !== null) {
+        out += literal.startsWith("/") ? blank(literal) : enclosed(literal);
+      } else if (c === "/" && BEFORE_REGEX.test(out)) {
+        out += enclosed(take(REGEX));
+      } else if (c === "`") {
+        i += 1;
+        out += "`";
+        template();
+      } else if (c === "}" && inBraces) {
+        return;
+      } else if (c === "{" && inBraces) {
+        braces("{");
+      } else {
+        out += c;
+        i += 1;
+      }
+    }
+  };
+  const braces = (open) => {
+    i += open.length;
+    out += open;
+    code(true);
+    if (i < source.length) {
+      i += 1;
+      out += "}";
+    }
+  };
+  const template = () => {
+    out += blank(take(TEMPLATE_TEXT));
+    if (source.startsWith("${", i)) {
+      braces("${");
+      template();
+    } else if (i < source.length) {
+      i += 1;
+      out += "`";
+    }
+  };
+  code(false);
+  return out;
+}
+var PURE_BUILTINS = [
+  "assert",
+  "assert/strict",
+  "crypto",
+  "path",
+  "path/posix",
+  "path/win32",
+  "punycode",
+  "querystring",
+  "string_decoder",
+  "url",
+  "util/types"
+];
 var RUNNER_PACKAGES = ["vitest", "@jest/globals"];
 function reachesOutside(file, trusted) {
-  const dense = file.source.replace(/\s/g, "");
+  const dense = withoutLiterals(file.source).replace(/\s/g, "");
   const code = withoutStrings(file.source).replace(/\s/g, "");
   const mark = REACHES.find((text) => code.includes(text));
   if (mark !== void 0) {
@@ -5452,7 +5548,9 @@ function reachesOutside(file, trusted) {
   }
   for (const { specifier, builtin } of file.imports) {
     if (builtin) {
-      return `imports ${specifier}`;
+      if (!PURE_BUILTINS.includes(specifier.replace(/^node:/, ""))) {
+        return `imports ${specifier}`;
+      }
     } else if (![...RUNNER_PACKAGES, ...trusted].some(
       (name) => specifier === name || specifier.startsWith(`${name}/`)
     )) {
@@ -6040,7 +6138,9 @@ Options:
   --affected                  mutation: run again only the gates that a change since the last
                               run can affect, and reuse the results of the others. Every
                               mutant is run when CI is set
-  --shard <index>/<total>     selfcheck: one of <total> parts of the gates, for parallel CI jobs
+  --shard <index>/<total>     selfcheck, mutation: one of <total> parts of the gates, for parallel CI
+                              jobs. For mutation the gates are sorted and dealt out in turn; every
+                              part has to run. Accepted when CI is set
   --jobs <n>                  selfcheck: run <n> negative controls at the same time. One that does
                               not hold cleanly there is run again on its own
   --quick                     selfcheck: one run for the gates only their own spec touches.
@@ -6075,7 +6175,7 @@ function main(argv) {
     case "doctor":
       return runDoctor(args.dir, io);
     case "mutation":
-      return runMutation(args.dir, args.rest, io, args.gates, args.affected);
+      return runMutation(args.dir, args.rest, io, args.gates, args.affected, args.shard);
     case "mutation-result":
       return runMutationResult(args.dir, args.report, io);
     case "selfcheck":

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@t09tanaka/test-gates",
-  "version": "0.4.3",
+  "version": "0.5.1",
   "description": "Test gates for the code that must not be wrong: per-file 100% coverage as the floor, mutation testing with a per-mutant allow list as the real check.",
   "author": "t09tanaka",
   "license": "MIT",