probatio 0.1.2 → 0.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,21 +1,38 @@
1
1
  # Probatio
2
2
 
3
- Probatio (Latin: a testing, a proof) is a free testing toolkit under the MIT license, built for AI agents
4
- (Claude, Grok, ChatGPT, others), working alone or as swarms. Every claim it makes is checkable.
5
- It remembers what was fixed so agents with small context do not redo or undo work, and it shows
6
- which tests already touch a change.
3
+ [![npm version](https://img.shields.io/npm/v/probatio)](https://www.npmjs.com/package/probatio)
7
4
 
8
- Status (2026-10-06): `mutate run` discovers the project suite and scores it. Node, pytest, C (when the binary was built with LLVM coverage), Go, Maven/Java, Rust, C#, and Mocha collect a line map on the baseline. Mocha's map comes from its own root hooks. Loading the node:test collector under Mocha does not name Mocha tests. A line no test executes is `no coverage`, and the suite is not started. A plain Node script can write the lines it ran, and the parent test that reads that dump stores them under its own name. A child the parent does not read stays `no coverage`, and the suite is not started. A suite with no line map, a baseline of 5 seconds or more, and more than 30 mutants stops before the first mutant. `verify-change` names the tests the line map ran. It scores `src/`, a `lib/` file when the package tests import `lib/`, and a Go file beside `go.mod`. Tests, docs, `dist/`, and `node_modules` stay unseen. `ran` stays the direct-importer list. Covered mutants of one Node file run one after another inside the suite process that is already running. `--workers` stays 1. A Node test that sets its own `timeout` keeps that time when `--test-timeout-ms` is shorter. A mutant that crashes or leaves that process dirty ends it, and the next mutant starts clean. An uncovered line still does not start the suite. `mutate tally` does not delete a test. A kill is an assertion the suite already had. A sealed miss means the suite did not see the bug. The kill count is not a merge gate. On Commons CSV (Apache-2.0, commit `2c83a308`), two sealed runs agreed: with the new test file hidden, `CSVFormatTest.testFormatThrowsNullPointerException` failed on the reverted printer. `deletedTests` stayed 0. That older JUnit platform did not write a line map, so the first pass was the whole suite and confirm reran the older test. Auspex, markupsafe, and a hand-planted fixture are not that result. The package version in this repository is 0.1.2. npm latest is 0.1.0. The BugsInPy search row is still open, and the rows that apply only when a sealed run misses do not apply, because this catch held. The MCP server speaks one JSON object per line and returns the same JSON as the CLI.
5
+ Probatio scores a test suite by the bugs it already catches. A kill is an assertion the suite had. The kill count is not a merge gate.
9
6
 
10
- Read in this order:
7
+ Node 22.6 or newer:
11
8
 
12
- 1. `HANDOFF.md` — why, the evidence from Auspex, the design, and the lessons that cost time.
13
- 2. `FOR-GROK.md` — the builder brief: build order with acceptance tests, the agent contract,
14
- memory for agents, anti-gaming, swarm protocol, traps.
15
- 3. `seed/` — code copied from the Auspex experiment to start from (Auspex paths inside).
9
+ ```bash
10
+ npm install probatio
11
+ ```
12
+
13
+ ```bash
14
+ npx probatio mutate generate --package . --out .probatio/generate
15
+ npx probatio mutate run --package . --patches .probatio/generate/mutants --out .probatio/runs
16
+ npx probatio mutate tally --out .probatio/runs
17
+ ```
16
18
 
17
- Roles: Grok builds most of it; Claude refines and finishes. Grok keeps `NOTES-FOR-CLAUDE.md` here
18
- (decisions, guesses, shortcuts with file:line, acceptance output actually run).
19
+ Each command prints one JSON object and exits 0 only when `ok` is true.
20
+
21
+ ```json
22
+ {
23
+ "schemaVersion": 1,
24
+ "ok": true,
25
+ "summary": "0 no coverage, 0 survived, 1 killed, 0 flaky, 0 timed out, 0 errored, of 1 finished.",
26
+ "next": "No survivor in this batch.",
27
+ "nextCall": { "argv": ["mutate", "tally", "--out", ".probatio/runs"] }
28
+ }
29
+ ```
30
+
31
+ Agents changing this repo should read [AGENTS.md](AGENTS.md). The MCP server is `probatio mcp`. Snippets are in the MCP section below. People building Probatio should read [docs/builders.md](docs/builders.md). Small suites live in [examples/](examples/).
32
+
33
+ ## Status
34
+
35
+ Status (2026-10-06): `mutate run` discovers the project suite and scores it. Node, pytest, C (when the binary was built with LLVM coverage), Go, Maven/Java, Rust, C#, and Mocha collect a line map on the baseline. Mocha's map comes from its own root hooks. Loading the node:test collector under Mocha does not name Mocha tests. A line no test executes is `no coverage`, and the suite is not started. A plain Node script can write the lines it ran, and the parent test that reads that dump stores them under its own name. A child the parent does not read stays `no coverage`, and the suite is not started. A suite with no line map, a baseline of 5 seconds or more, and more than 30 mutants stops before the first mutant. `verify-change` names the tests the line map ran. It scores `src/`, a `lib/` file when the package tests import `lib/`, and a Go file beside `go.mod`. Tests, docs, `dist/`, and `node_modules` stay unseen. `ran` stays the direct-importer list. Covered mutants of one Node file run one after another inside the suite process that is already running. `--workers` stays 1. A Node test that sets its own `timeout` keeps that time when `--test-timeout-ms` is shorter. A mutant that crashes or leaves that process dirty ends it, and the next mutant starts clean. An uncovered line still does not start the suite. `mutate tally` does not delete a test. A kill is an assertion the suite already had. A sealed miss means the suite did not see the bug. The kill count is not a merge gate. On Commons CSV (Apache-2.0, commit `2c83a308`), two sealed runs agreed: with the new test file hidden, `CSVFormatTest.testFormatThrowsNullPointerException` failed on the reverted printer. `deletedTests` stayed 0. That older JUnit platform did not write a line map, so the first pass was the whole suite and confirm reran the older test. Auspex, markupsafe, and a hand-planted fixture are not that result. The package version in this repository is 0.1.3. npm latest is 0.1.2 (`29301bc`). The BugsInPy search row is still open, and the rows that apply only when a sealed run misses do not apply, because this catch held. The MCP server speaks one JSON object per line and returns the same JSON as the CLI.
19
36
 
20
37
  ## mutate
21
38
 
@@ -81,4 +98,23 @@ Lists the tests that can see the diff, runs mutants on the changed lines, and re
81
98
 
82
99
  An item is `.probatio/queue/<id>.json`. `queue claim` renames it to `claimed/<agent>-<id>.json` with a lease. `queue reap` moves an expired lease back. `check-kill <id>` runs the tests that directly import the mutated file and confirms the killing test again. It is done only when that mutant dies and that suite is still green. `verify-change` is the edit-sized run: it uses the line map and does not confirm a second time.
83
100
 
84
- The MCP server speaks stdio JSON-RPC. Its one tool runs the CLI and returns that command's JSON.
101
+ ## MCP
102
+
103
+ `probatio mcp` and the `probatio-mcp` bin speak stdio JSON-RPC, one JSON object per line. The tool name is `probatio`. `argv` is the CLI words. `ping` returns `{}`. An unknown method returns JSON-RPC `-32601` with the same id.
104
+
105
+ ```json
106
+ {
107
+ "mcpServers": {
108
+ "probatio": {
109
+ "command": "probatio",
110
+ "args": ["mcp"]
111
+ }
112
+ }
113
+ }
114
+ ```
115
+
116
+ That block is the Claude, Cursor, and Grok shape. The tool runs the CLI and returns that command's JSON. `nextCall.argv` keeps absolute paths so a later spawn does not have to expand `~`.
117
+
118
+ `mutate generate` reads the commit (`HEAD` unless you pass `--commit`). `--working-tree` reads the checkout on disk and says so. `mutate run` scores the commit either way.
119
+
120
+ `verify-change` does not score uncommitted edits when `--base` and `--commit` are the same. A clean empty diff says `No diff-scoped mutant.`
package/dist/cli.js CHANGED
@@ -10,65 +10,82 @@ import { appendFinding, chooseSealed, fixGap, guardAllows, lineHash, loadState,
10
10
  import { git } from "./mutate/patch.js";
11
11
  import { generateMutants } from "./mutate/generate.js";
12
12
  import { runMutants } from "./mutate/run.js";
13
- import { LABEL_LEAK_SUMMARY, labelLeaked, readSealedLabel } from "./mutate/sealed.js";
13
+ import { LABEL_LEAK_SUMMARY, labelLeaked, readSealedLabel, sealedTreeLeaked } from "./mutate/sealed.js";
14
14
  import { generateNext } from "./mutate/suite-decision.js";
15
15
  import { discoverSuite } from "./mutate/suites.js";
16
16
  import { tallyRun } from "./mutate/tally.js";
17
17
  import { checkKill } from "./swarm/check-kill.js";
18
18
  import { claimItem, reapClaims, seedQueue } from "./swarm/queue.js";
19
19
  import { verifyChange } from "./verify/change.js";
20
- const parsed = parseArgs(process.argv.slice(2));
21
- const [group, action] = parsed.command;
22
- try {
23
- if (group === "mutate" && action === "generate")
24
- finish(await generateCommand(parsed.flags), parsed.human);
25
- if (group === "mutate" && action === "run")
26
- finish(await runCommand(parsed.flags), parsed.human);
27
- if (group === "mutate" && action === "sealed")
28
- finish(await sealedCommand(parsed.flags), parsed.human);
29
- if (group === "mutate" && action === "tally")
30
- finish(tallyCommand(parsed.flags), parsed.human);
31
- if (group === "ledger" && action === "build")
32
- finish(await ledgerCommand(parsed.flags), parsed.human);
33
- if (group === "matrix" && action === "report")
34
- finish(matrixCommand(parsed.flags), parsed.human);
35
- if (group === "golden" && action === "check")
36
- finish(goldenCommand(parsed.flags), parsed.human);
37
- if (group === "gap" && action === "fix")
38
- finish(gapFixCommand(parsed.flags), parsed.human);
39
- if (group === "gap" && action === "revert")
40
- finish(gapRevertCommand(parsed.flags), parsed.human);
41
- if (group === "guard" && action === "check")
42
- finish(guardCommand(parsed.flags), parsed.human);
43
- if (group === "findings" && action === "add")
44
- finish(findingsCommand(parsed.flags), parsed.human);
45
- if (group === "status")
46
- finish(statusCommand(parsed.flags), parsed.human);
47
- if (group === "seal")
48
- finish(sealCommand(parsed.flags), parsed.human);
49
- if (group === "queue" && action === "seed")
50
- finish(queueSeedCommand(parsed.flags), parsed.human);
51
- if (group === "queue" && action === "claim")
52
- finish(queueClaimCommand(parsed.flags), parsed.human);
53
- if (group === "queue" && action === "reap")
54
- finish(queueReapCommand(parsed.flags), parsed.human);
55
- if (group === "check-kill")
56
- finish(await checkKillCommand(parsed.flags, action), parsed.human);
57
- if (group === "verify-change")
58
- finish(await verifyCommand(parsed.flags), parsed.human);
59
- finish(usage(false, "Use mutate, ledger, matrix, golden, gap, guard, findings, status, queue, check-kill, or verify-change."), parsed.human);
60
- }
61
- catch (error) {
62
- const message = error instanceof Error ? error.message : String(error);
63
- finish({
20
+ import { nodeTooOld } from "./node-version.js";
21
+ const tooOld = nodeTooOld(process.versions.node);
22
+ if (tooOld) {
23
+ process.stdout.write(`${JSON.stringify({
64
24
  schemaVersion: SCHEMA_VERSION,
65
25
  ok: false,
66
- command: `${group ?? "mutate"}.${action ?? "help"}`,
67
- summary: message,
68
- next: "Fix the command and run it again.",
26
+ command: "mutate.help",
27
+ summary: tooOld,
28
+ next: "Install Node 22.6 or newer and run the command again.",
69
29
  nextCall: null,
70
- }, parsed.human);
30
+ })}\n`);
31
+ process.exit(1);
71
32
  }
33
+ const parsed = parseArgs(process.argv.slice(2));
34
+ const [group, action] = parsed.command;
35
+ if (group === "mcp" && !action) {
36
+ await import("./mcp.js");
37
+ }
38
+ else
39
+ try {
40
+ if (group === "mutate" && action === "generate")
41
+ finish(await generateCommand(parsed.flags), parsed.human);
42
+ if (group === "mutate" && action === "run")
43
+ finish(await runCommand(parsed.flags), parsed.human);
44
+ if (group === "mutate" && action === "sealed")
45
+ finish(await sealedCommand(parsed.flags), parsed.human);
46
+ if (group === "mutate" && action === "tally")
47
+ finish(tallyCommand(parsed.flags), parsed.human);
48
+ if (group === "ledger" && action === "build")
49
+ finish(await ledgerCommand(parsed.flags), parsed.human);
50
+ if (group === "matrix" && action === "report")
51
+ finish(matrixCommand(parsed.flags), parsed.human);
52
+ if (group === "golden" && action === "check")
53
+ finish(goldenCommand(parsed.flags), parsed.human);
54
+ if (group === "gap" && action === "fix")
55
+ finish(gapFixCommand(parsed.flags), parsed.human);
56
+ if (group === "gap" && action === "revert")
57
+ finish(gapRevertCommand(parsed.flags), parsed.human);
58
+ if (group === "guard" && action === "check")
59
+ finish(guardCommand(parsed.flags), parsed.human);
60
+ if (group === "findings" && action === "add")
61
+ finish(findingsCommand(parsed.flags), parsed.human);
62
+ if (group === "status")
63
+ finish(statusCommand(parsed.flags), parsed.human);
64
+ if (group === "seal")
65
+ finish(sealCommand(parsed.flags), parsed.human);
66
+ if (group === "queue" && action === "seed")
67
+ finish(queueSeedCommand(parsed.flags), parsed.human);
68
+ if (group === "queue" && action === "claim")
69
+ finish(queueClaimCommand(parsed.flags), parsed.human);
70
+ if (group === "queue" && action === "reap")
71
+ finish(queueReapCommand(parsed.flags), parsed.human);
72
+ if (group === "check-kill")
73
+ finish(await checkKillCommand(parsed.flags, action), parsed.human);
74
+ if (group === "verify-change")
75
+ finish(await verifyCommand(parsed.flags), parsed.human);
76
+ finish(usage(false, "Use mutate, ledger, matrix, golden, gap, guard, findings, status, queue, check-kill, or verify-change."), parsed.human);
77
+ }
78
+ catch (error) {
79
+ const message = error instanceof Error ? error.message : String(error);
80
+ finish({
81
+ schemaVersion: SCHEMA_VERSION,
82
+ ok: false,
83
+ command: `${group ?? "mutate"}.${action ?? "help"}`,
84
+ summary: message,
85
+ next: "Fix the command and run it again.",
86
+ nextCall: null,
87
+ }, parsed.human);
88
+ }
72
89
  function usage(ok, summary) {
73
90
  return {
74
91
  schemaVersion: SCHEMA_VERSION,
@@ -93,7 +110,19 @@ async function generateCommand(flags) {
93
110
  maxMutants: int(flags, "max-mutants") ?? null,
94
111
  maxMinutes: int(flags, "max-minutes") ?? null,
95
112
  skipFiles: texts(flags, "skip-file"),
113
+ commit: text(flags, "commit") ?? "HEAD",
114
+ workingTree: bool(flags, "working-tree", false),
96
115
  });
116
+ if (result.error) {
117
+ return {
118
+ schemaVersion: SCHEMA_VERSION,
119
+ ok: false,
120
+ command: "mutate.generate",
121
+ summary: result.error,
122
+ next: "Commit the file, or pass --working-tree to mutate the checkout on disk. mutate run scores the commit.",
123
+ nextCall: null,
124
+ };
125
+ }
97
126
  if (result.violations.length > 0) {
98
127
  const first = result.violations[0];
99
128
  return {
@@ -117,9 +146,14 @@ async function generateCommand(flags) {
117
146
  summary: stopped
118
147
  ? `Stopped by the time budget after ${result.filesVisited} files. ${result.mutants.length} mutants written.`
119
148
  : `${result.mutants.length} mutants in ${result.filesVisited} files. No string-literal mutant.`,
120
- next: stopped
121
- ? "Narrow --src or raise --max-minutes, then run the same command."
122
- : generateNext(result.mutants.length, suite ? { kind: suite.kind, command: suite.command } : null),
149
+ next: [
150
+ stopped
151
+ ? "Narrow --src or raise --max-minutes, then run the same command."
152
+ : generateNext(result.mutants.length, suite ? { kind: suite.kind, command: suite.command } : null),
153
+ result.source === "working-tree"
154
+ ? "These patches match the working tree. mutate run scores the commit, so commit the files before mutate run."
155
+ : "",
156
+ ].filter((line) => line.length > 0).join(" "),
123
157
  nextCall: result.mutants.length === 0
124
158
  ? null
125
159
  : { argv: ["mutate", "run", "--package", packageDir, "--patches", patches, "--out", path.join(outDir, "runs")] },
@@ -133,6 +167,19 @@ async function generateCommand(flags) {
133
167
  };
134
168
  }
135
169
  async function runCommand(flags, hide = null) {
170
+ const hidden = hide ?? texts(flags, "hide");
171
+ const escaped = hidden.find((rel) => rel.split(/[\\/]/).includes(".."));
172
+ if (escaped) {
173
+ return {
174
+ schemaVersion: SCHEMA_VERSION,
175
+ ok: false,
176
+ command: "mutate.run",
177
+ summary: `--hide stays inside the package. Refused ${escaped}.`,
178
+ next: "Pass a relative path with no .. segment.",
179
+ nextCall: null,
180
+ killed: 0,
181
+ };
182
+ }
136
183
  const onlyNames = texts(flags, "only-test");
137
184
  if (onlyNames.some((id) => id === "::command" || id.endsWith("::command"))) {
138
185
  return {
@@ -234,9 +281,10 @@ function tallyCommand(flags) {
234
281
  async function sealedCommand(flags) {
235
282
  const label = readSealedLabel(path.resolve(requireText(flags, "label")));
236
283
  const hide = texts(flags, "hide");
284
+ const outDir = path.resolve(requireText(flags, "out"));
237
285
  const report = await runCommand(flags, hide);
238
286
  const body = JSON.stringify(report);
239
- if (!labelLeaked(body, label))
287
+ if (!labelLeaked(body, label) && !sealedTreeLeaked(outDir, label))
240
288
  return report;
241
289
  return {
242
290
  schemaVersion: SCHEMA_VERSION,
package/dist/contract.js CHANGED
@@ -1,18 +1,20 @@
1
1
  import { homedir } from "node:os";
2
2
  export const SCHEMA_VERSION = 1;
3
3
  /** Replace the home directory so a result never carries a user's name or home path. */
4
- export function scrub(value, home = homedir()) {
4
+ export function scrub(value, home = homedir(), keepPaths = false) {
5
5
  if (typeof value === "string") {
6
- if (!home || !value.includes(home))
6
+ if (keepPaths || !home || !value.includes(home))
7
7
  return value;
8
8
  return value.split(home).join("~");
9
9
  }
10
10
  if (Array.isArray(value))
11
- return value.map((item) => scrub(item, home));
11
+ return value.map((item) => scrub(item, home, keepPaths));
12
12
  if (value && typeof value === "object") {
13
13
  const out = {};
14
- for (const [key, item] of Object.entries(value))
15
- out[key] = scrub(item, home);
14
+ for (const [key, item] of Object.entries(value)) {
15
+ // nextCall.argv is executed. A tilde is not expanded by spawn or by MCP.
16
+ out[key] = scrub(item, home, keepPaths || key === "nextCall");
17
+ }
16
18
  return out;
17
19
  }
18
20
  return value;
package/dist/mcp.js CHANGED
@@ -3,6 +3,19 @@ import { spawnSync } from "node:child_process";
3
3
  import { existsSync, readFileSync } from "node:fs";
4
4
  import path from "node:path";
5
5
  import { fileURLToPath } from "node:url";
6
+ import { nodeTooOld } from "./node-version.js";
7
+ const tooOld = nodeTooOld(process.versions.node);
8
+ if (tooOld) {
9
+ process.stdout.write(`${JSON.stringify({
10
+ schemaVersion: 1,
11
+ ok: false,
12
+ command: "mcp",
13
+ summary: tooOld,
14
+ next: "Install Node 22.6 or newer and run the command again.",
15
+ nextCall: null,
16
+ })}\n`);
17
+ process.exit(1);
18
+ }
6
19
  const here = path.dirname(fileURLToPath(import.meta.url));
7
20
  const version = readVersion();
8
21
  const tool = {
@@ -63,6 +76,10 @@ function handle(message) {
63
76
  send({ jsonrpc: "2.0", id: message.id, result: { tools: [tool] } });
64
77
  return;
65
78
  }
79
+ if (message.method === "ping") {
80
+ send({ jsonrpc: "2.0", id: message.id, result: {} });
81
+ return;
82
+ }
66
83
  if (message.method === "tools/call") {
67
84
  const argv = message.params?.arguments?.argv;
68
85
  if (!Array.isArray(argv) || argv.some((item) => typeof item !== "string")) {
@@ -92,7 +109,13 @@ function handle(message) {
92
109
  isError: envelope.ok !== true,
93
110
  },
94
111
  });
112
+ return;
95
113
  }
114
+ send({
115
+ jsonrpc: "2.0",
116
+ id: message.id,
117
+ error: { code: -32601, message: `method not found: ${message.method}` },
118
+ });
96
119
  }
97
120
  function cliCommand() {
98
121
  const compiled = path.join(here, "cli.js");
@@ -104,17 +104,10 @@ export function writeLlvmCoverage(pkg, profileDir, binary, dest) {
104
104
  return { ok: false, detail: "no profraw" };
105
105
  if (!binary)
106
106
  return { ok: false, detail: "no instrumented binary" };
107
- const llvmCov = findTool("llvm-cov");
108
- const llvmProf = findTool("llvm-profdata");
109
- if (!llvmCov || !llvmProf)
110
- return { ok: false, detail: `llvm tool missing (${llvmCov ? "llvm-profdata" : "llvm-cov"})` };
111
107
  const merged = path.join(profileDir, "baseline.profdata");
112
- const merge = spawnSync(llvmProf, ["merge", "-sparse", "-o", merged, ...raws], { encoding: "utf8" });
113
- if (merge.status !== 0)
114
- return { ok: false, detail: (merge.stderr || merge.stdout || "llvm-profdata failed").trim() };
115
- const exported = spawnSync(llvmCov, ["export", binary, `-instr-profile=${merged}`], { encoding: "utf8", maxBuffer: 64 * 1024 * 1024 });
116
- if (exported.status !== 0)
117
- return { ok: false, detail: (exported.stderr || exported.stdout || "llvm-cov failed").trim() };
108
+ const exported = exportLlvmProfile(binary, raws, merged);
109
+ if (!exported.ok)
110
+ return exported;
118
111
  const map = llvmExportToMap(exported.stdout, pkg);
119
112
  if (!map)
120
113
  return { ok: false, detail: "llvm-cov export was not a coverage map" };
@@ -159,14 +152,78 @@ function llvmExportToMap(stdout, pkg) {
159
152
  return null;
160
153
  return { files };
161
154
  }
162
- function findTool(name) {
155
+ /**
156
+ * clang and rustc write different profraw versions. Each pair is one LLVM install.
157
+ * The first pair that can merge the profile wins.
158
+ */
159
+ export function llvmToolsPresent() {
160
+ return llvmToolPairs().length > 0;
161
+ }
162
+ export function exportLlvmProfile(binary, raws, merged) {
163
+ const pairs = llvmToolPairs();
164
+ if (pairs.length === 0)
165
+ return { ok: false, detail: "llvm tool missing (llvm-cov)" };
166
+ let detail = "llvm-profdata failed";
167
+ for (const { cov, prof } of pairs) {
168
+ const merge = spawnSync(prof, ["merge", "-sparse", "-o", merged, ...raws], { encoding: "utf8" });
169
+ if (merge.status !== 0) {
170
+ detail = (merge.stderr || merge.stdout || "llvm-profdata failed").trim();
171
+ continue;
172
+ }
173
+ const exported = spawnSync(cov, ["export", binary, `-instr-profile=${merged}`], { encoding: "utf8", maxBuffer: 64 * 1024 * 1024 });
174
+ if (exported.status !== 0) {
175
+ detail = (exported.stderr || exported.stdout || "llvm-cov failed").trim();
176
+ continue;
177
+ }
178
+ return { ok: true, stdout: exported.stdout };
179
+ }
180
+ return { ok: false, detail };
181
+ }
182
+ function llvmToolPairs() {
183
+ const covs = toolCandidates("llvm-cov");
184
+ const profs = toolCandidates("llvm-profdata");
185
+ const pairs = [];
186
+ const seen = new Set();
187
+ for (const cov of covs) {
188
+ const prof = profs.find((item) => path.dirname(item) === path.dirname(cov));
189
+ if (!prof)
190
+ continue;
191
+ const key = `${cov}\0${prof}`;
192
+ if (seen.has(key))
193
+ continue;
194
+ seen.add(key);
195
+ pairs.push({ cov, prof });
196
+ }
197
+ return pairs;
198
+ }
199
+ function toolCandidates(name) {
200
+ const out = [];
201
+ const add = (file) => {
202
+ if (!file || out.includes(file) || !existsSync(file))
203
+ return;
204
+ out.push(file);
205
+ };
163
206
  const fromXcrun = spawnSync("xcrun", ["--find", name], { encoding: "utf8" });
164
- const xcrunPath = fromXcrun.stdout?.trim() ?? "";
165
- if (fromXcrun.status === 0 && xcrunPath && existsSync(xcrunPath))
166
- return xcrunPath;
207
+ if (fromXcrun.status === 0)
208
+ add((fromXcrun.stdout ?? "").trim());
167
209
  const which = spawnSync("sh", ["-c", `command -v ${name}`], { encoding: "utf8" });
168
- const found = which.stdout?.trim() ?? "";
169
- if (which.status === 0 && found)
170
- return found;
171
- return null;
210
+ if (which.status === 0)
211
+ add((which.stdout ?? "").trim());
212
+ const rustc = spawnSync("rustc", ["--print", "sysroot"], { encoding: "utf8" });
213
+ if (rustc.status === 0) {
214
+ const root = path.join(rustc.stdout.trim(), "lib", "rustlib");
215
+ if (existsSync(root)) {
216
+ for (const host of readdirSync(root))
217
+ add(path.join(root, host, "bin", name));
218
+ }
219
+ }
220
+ const usr = "/usr/lib";
221
+ if (existsSync(usr)) {
222
+ for (const dir of readdirSync(usr)) {
223
+ if (dir.startsWith("llvm-"))
224
+ add(path.join(usr, dir, "bin", name));
225
+ }
226
+ }
227
+ add(path.join("/usr/bin", name));
228
+ return out;
172
229
  }
@@ -8,8 +8,9 @@ export function generateMutants(options) {
8
8
  const started = Date.now();
9
9
  const limit = options.maxMinutes;
10
10
  const over = () => limit != null && Date.now() - started >= limit * 60_000;
11
+ const sourceDefault = options.workingTree ? "working-tree" : "commit";
11
12
  if (over())
12
- return { mutants: [], violations: [], filesVisited: 0, budgetHit: true };
13
+ return { mutants: [], violations: [], filesVisited: 0, budgetHit: true, source: sourceDefault };
13
14
  const srcRoot = path.resolve(options.packageDir, options.srcDir);
14
15
  const walked = walkSources(srcRoot, over);
15
16
  const files = walked.files.filter((file) => !options.skipFiles.includes(path.basename(file)));
@@ -18,13 +19,18 @@ export function generateMutants(options) {
18
19
  const chosen = [];
19
20
  let filesVisited = 0;
20
21
  let budgetHit = walked.stopped;
22
+ let source = sourceDefault;
21
23
  for (const file of files) {
22
24
  if (over()) {
23
25
  budgetHit = true;
24
26
  break;
25
27
  }
26
28
  filesVisited += 1;
27
- const text = readFileSync(file, "utf8");
29
+ const loaded = fileText(options.packageDir, file, options);
30
+ if ("error" in loaded)
31
+ return { mutants: [], violations: [], filesVisited, budgetHit, source: "commit", error: loaded.error };
32
+ source = loaded.source;
33
+ const text = loaded.text;
28
34
  const rel = posix(path.relative(options.packageDir, file));
29
35
  const found = findInSource(rel, text);
30
36
  violations.push(...found.violations);
@@ -34,7 +40,7 @@ export function generateMutants(options) {
34
40
  chosen.push({ rel, text, point });
35
41
  }
36
42
  if (violations.length > 0)
37
- return { mutants: [], violations, filesVisited, budgetHit };
43
+ return { mutants: [], violations, filesVisited, budgetHit, source };
38
44
  const capped = options.maxMutants === null ? chosen : chosen.slice(0, options.maxMutants);
39
45
  const packageReal = real(options.packageDir);
40
46
  const patchRoot = gitRoot(packageReal);
@@ -55,7 +61,22 @@ export function generateMutants(options) {
55
61
  mutants.push({ id, file: item.rel, line: item.point.line, op: item.point.op, patch });
56
62
  }
57
63
  writeFileSync(path.join(mutantsDir, "mutants.json"), `${JSON.stringify({ schemaVersion: 1, seed: options.seed, mutants }, null, 2)}\n`);
58
- return { mutants, violations, filesVisited, budgetHit };
64
+ return { mutants, violations, filesVisited, budgetHit, source };
65
+ }
66
+ function fileText(packageDir, absFile, options) {
67
+ if (options.workingTree)
68
+ return { text: readFileSync(absFile, "utf8"), source: "working-tree" };
69
+ const root = gitRoot(packageDir);
70
+ if (!root)
71
+ return { text: readFileSync(absFile, "utf8"), source: "working-tree" };
72
+ const rel = posix(path.relative(root, real(absFile)));
73
+ const commit = options.commit && options.commit.length > 0 ? options.commit : "HEAD";
74
+ const shown = spawnSync("git", ["-C", root, "show", `${commit}:${rel}`], { encoding: "utf8" });
75
+ if (shown.status !== 0) {
76
+ const shownRel = posix(path.relative(packageDir, absFile));
77
+ return { error: `${shownRel} is not in ${commit}. Pass --working-tree to read the checkout.` };
78
+ }
79
+ return { text: shown.stdout, source: "commit" };
59
80
  }
60
81
  function walkSources(dir, over) {
61
82
  if (!statExists(dir))
@@ -41,7 +41,7 @@ export async function runMutants(options) {
41
41
  return failReport(options.outDir, "", options.commit, "that commit does not resolve");
42
42
  const commit = sha.stdout.trim();
43
43
  mkdirSync(options.outDir, { recursive: true });
44
- const run = loadRun(options.outDir, commit, options.agent);
44
+ const run = loadRun(options, commit);
45
45
  if ("error" in run)
46
46
  return failReport(options.outDir, run.id, commit, run.error);
47
47
  const listed = listPatches(options.patchDirs);
@@ -211,7 +211,7 @@ async function prepare(pkg, options, progress) {
211
211
  if (baseline.compileToken)
212
212
  return { ok: false, error: `baseline build failed (${baseline.compileToken})` };
213
213
  if (!baseline.report)
214
- return { ok: false, error: baselineReportError(baseline.detail) };
214
+ return { ok: false, error: baselineReportError(baseline.detail, baseline.command || spec.command) };
215
215
  baselineFailed = baseline.report.failed.map((item) => item.name);
216
216
  if (collect && !existsSync(mapPath)) {
217
217
  const binary = binaryFromCommand(baseline.command || spec.command, pkg);
@@ -224,6 +224,10 @@ async function prepare(pkg, options, progress) {
224
224
  progress(`suite report tests=${baseline.report.tests}`);
225
225
  progress(baselineFailed.length > 0 ? `baseline failing ${baselineFailed.length}` : "baseline pass");
226
226
  if (baselineFailed.length > 0) {
227
+ const tool = missingToolFrom(baseline.detail, baseline.command || spec.command);
228
+ const unnamed = baselineFailed.every((name) => name === "command" || name === "::command");
229
+ if (tool && unnamed)
230
+ return { ok: false, error: `missing tool: ${tool}. baseline suite did not return a test report` };
227
231
  return { ok: false, error: "baseline already failing", baselineFailed, suiteCommand: spec.command };
228
232
  }
229
233
  if (collect && !existsSync(mapPath) && languageMapKind(spec.kind)) {
@@ -250,12 +254,13 @@ async function runOne(worktree, pkg, patchPath, options, commit, runId, prep, la
250
254
  const raw = readFileSync(patchPath, "utf8");
251
255
  const files = filesInDiff(raw);
252
256
  const edited = changedLines(raw);
257
+ const located = edited.length > 0 ? edited : files;
253
258
  const mutantHash = sha256(raw);
254
259
  const base = () => ({
255
260
  id,
256
261
  outcome: "error",
257
262
  direction: null,
258
- files,
263
+ files: located,
259
264
  killedBy: [],
260
265
  cause: null,
261
266
  next: "",
@@ -319,8 +324,16 @@ async function runOne(worktree, pkg, patchPath, options, commit, runId, prep, la
319
324
  // A compile error can leave the previous run's report on disk. That report is not this mutant.
320
325
  if (main.compileToken)
321
326
  return finishMutant(killed(base, direction, [main.compileToken]), pkg, commands.join(" | "), false, []);
322
- if (!main.report)
323
- return { ...base(), direction, error: "test reporter returned nothing", command: commands.join(" | ") };
327
+ if (!main.report) {
328
+ const detail = main.detail ? clip(main.detail) : "";
329
+ const tool = missingToolFrom(detail, main.command);
330
+ const error = tool
331
+ ? `missing tool: ${tool}`
332
+ : detail
333
+ ? `test reporter returned nothing (${detail})`
334
+ : "test reporter returned nothing";
335
+ return { ...base(), direction, error, command: commands.join(" | ") };
336
+ }
324
337
  const whole = main.wholeSuite || forceWhole;
325
338
  const chosen = main.selectedTests;
326
339
  if (main.report.failed.length === 0)
@@ -397,19 +410,22 @@ async function rerunGroup(pkg, options, file, failures, counts, pattern, prep, t
397
410
  if (rerun.command)
398
411
  commands.push(rerun.command);
399
412
  if (pattern === null && rerun.timedOut) {
400
- for (const failure of failures)
401
- bump(counts, failureId(pkg, failure));
413
+ for (const id of new Set(failures.map((item) => failureId(pkg, item))))
414
+ bump(counts, id);
402
415
  continue;
403
416
  }
404
417
  if (!rerun.report)
405
418
  continue;
419
+ const seen = new Set();
406
420
  for (const hit of rerun.report.failed) {
407
421
  if (pattern === null && !hit.fileTimeout)
408
422
  continue;
409
423
  const id = failureId(pkg, hit);
410
424
  if (failures.some((item) => failureId(pkg, item) === id))
411
- bump(counts, id);
425
+ seen.add(id);
412
426
  }
427
+ for (const id of seen)
428
+ bump(counts, id);
413
429
  }
414
430
  return wholeSuite;
415
431
  }
@@ -708,8 +724,8 @@ function loadMeta(dirs) {
708
724
  }
709
725
  return meta;
710
726
  }
711
- function loadRun(outDir, commit, agent) {
712
- const file = path.join(outDir, "run.json");
727
+ function loadRun(options, commit) {
728
+ const file = path.join(options.outDir, "run.json");
713
729
  if (existsSync(file)) {
714
730
  const stored = JSON.parse(readFileSync(file, "utf8"));
715
731
  if (stored.commit !== commit)
@@ -717,7 +733,24 @@ function loadRun(outDir, commit, agent) {
717
733
  return { id: stored.id };
718
734
  }
719
735
  const id = randomBytes(4).toString("hex");
720
- writeFileSync(file, `${JSON.stringify({ id, commit, agent, started: new Date().toISOString() }, null, 2)}\n`);
736
+ const record = {
737
+ id,
738
+ commit,
739
+ agent: options.agent,
740
+ started: new Date().toISOString(),
741
+ package: options.packageDir,
742
+ patches: options.patchDirs,
743
+ out: options.outDir,
744
+ repo: options.repoDir,
745
+ workers: options.workers,
746
+ confirm: options.confirm,
747
+ testsDir: options.testsDir,
748
+ suiteTimeoutMs: options.suiteTimeoutMs,
749
+ testTimeoutMs: options.testTimeoutMs,
750
+ direction: options.direction,
751
+ suiteCommand: options.suiteCommand ?? null,
752
+ };
753
+ writeFileSync(file, `${JSON.stringify(record, null, 2)}\n`);
721
754
  return { id };
722
755
  }
723
756
  async function createWorktrees(repo, packageDir, runId, commit, workers, outDir) {
@@ -725,6 +758,10 @@ async function createWorktrees(repo, packageDir, runId, commit, workers, outDir)
725
758
  const dirs = [];
726
759
  for (let index = 0; index < workers; index++) {
727
760
  const dir = path.join(tmpdir(), `probatio-${runId}-${index}`);
761
+ if (!probatioWorktreeDir(dir)) {
762
+ removeWorktrees(repo, dirs, outDir);
763
+ return { error: "worktree path is outside the probatio temp prefix" };
764
+ }
728
765
  rmSync(dir, { recursive: true, force: true });
729
766
  const added = git(repo, ["worktree", "add", "--detach", dir, commit]);
730
767
  if (added.status !== 0) {
@@ -750,15 +787,42 @@ async function createWorktrees(repo, packageDir, runId, commit, workers, outDir)
750
787
  function hidePaths(dirs, repo, packageDir, hide) {
751
788
  for (const dir of dirs) {
752
789
  const pkg = packageIn(dir, repo, packageDir);
790
+ const root = path.resolve(pkg);
753
791
  for (const rel of hide) {
754
- if (!rel || path.isAbsolute(rel))
792
+ if (!rel || path.isAbsolute(rel) || rel.split(/[\\/]/).includes(".."))
793
+ continue;
794
+ const target = path.resolve(root, rel);
795
+ if (target !== root && !target.startsWith(root + path.sep))
755
796
  continue;
756
- rmSync(path.join(pkg, rel), { force: true });
797
+ rmSync(target, { force: true });
757
798
  }
758
799
  }
759
800
  }
801
+ /** A worktree Probatio created: one directory under the OS temp dir, named probatio- plus hex and an index. */
802
+ export function probatioWorktreeDir(dir) {
803
+ let resolved = path.resolve(dir);
804
+ try {
805
+ resolved = realpathSync(dir);
806
+ }
807
+ catch {
808
+ // The directory is created on the next line. The unresolved path is still checked.
809
+ }
810
+ const roots = new Set([path.resolve(tmpdir())]);
811
+ try {
812
+ roots.add(realpathSync(tmpdir()));
813
+ }
814
+ catch {
815
+ // tmpdir() itself is enough when realpath is unavailable.
816
+ }
817
+ const inside = [...roots].some((root) => resolved === root || resolved.startsWith(root + path.sep));
818
+ if (!inside)
819
+ return false;
820
+ return /^probatio-[0-9a-f]+-\d+$/.test(path.basename(resolved));
821
+ }
760
822
  function removeWorktrees(repo, dirs, outDir) {
761
823
  for (const dir of dirs) {
824
+ if (!probatioWorktreeDir(dir))
825
+ continue;
762
826
  git(repo, ["worktree", "remove", "--force", dir]);
763
827
  rmSync(dir, { recursive: true, force: true });
764
828
  }
@@ -899,6 +963,9 @@ function reportOf(results, context) {
899
963
  notes.push(resumeNext(context.notStarted));
900
964
  if (results.some((result) => result.wholeSuite))
901
965
  notes.push(WHOLE_SUITE_NEXT);
966
+ const broken = results.find((result) => (result.outcome === "error" || result.outcome === "build-failed") && result.error);
967
+ if (broken?.error)
968
+ notes.push(`${broken.id}: ${broken.error}`);
902
969
  if (notes.length === 0)
903
970
  notes.push("No survivor in this batch.");
904
971
  return {
@@ -1156,13 +1223,27 @@ function testArg(pkg, file, testsDir) {
1156
1223
  function escapeRegExp(value) {
1157
1224
  return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
1158
1225
  }
1159
- function baselineReportError(detail) {
1226
+ function missingToolFrom(detail, command) {
1227
+ const spawnMiss = /spawn ([^\s]+) ENOENT/.exec(detail);
1228
+ if (spawnMiss)
1229
+ return path.basename(spawnMiss[1]);
1230
+ const shellMiss = /(?:^|\s)([A-Za-z0-9_./+-]+): (?:command not found|not found)\b/.exec(detail);
1231
+ if (shellMiss)
1232
+ return path.basename(shellMiss[1]);
1233
+ if (/No such file or directory|ENOENT/.test(detail)) {
1234
+ const bin = command.trim().split(/\s+/)[0];
1235
+ if (bin)
1236
+ return path.basename(bin);
1237
+ }
1238
+ return null;
1239
+ }
1240
+ function baselineReportError(detail, command = "") {
1241
+ const tool = missingToolFrom(detail, command);
1160
1242
  const line = detail.trim();
1161
1243
  // The first line of a Python or Node crash is a stack fragment. The summary
1162
1244
  // states the limit and does not carry that fragment.
1163
- if (!line || isStackFragment(line))
1164
- return "baseline suite did not return a test report";
1165
- return `baseline suite did not return a test report (${line})`;
1245
+ const base = !line || isStackFragment(line) ? "baseline suite did not return a test report" : `baseline suite did not return a test report (${line})`;
1246
+ return tool ? `missing tool: ${tool}. ${base}` : base;
1166
1247
  }
1167
1248
  function isStackFragment(line) {
1168
1249
  return line.startsWith("Traceback") || /^\s+at \S/.test(line) || /File ".+", line \d+/.test(line);
@@ -1,6 +1,7 @@
1
1
  import { spawnSync } from "node:child_process";
2
2
  import { existsSync, mkdirSync, realpathSync, writeFileSync } from "node:fs";
3
3
  import path from "node:path";
4
+ import { exportLlvmProfile, llvmToolsPresent } from "./coverage-map.js";
4
5
  /**
5
6
  * One instrumented test binary, then one profile per test name.
6
7
  * A single cargo test profile does not say which test hit the line.
@@ -9,9 +10,7 @@ export function writeRustCoverage(pkg, testNames, dest) {
9
10
  const names = [...new Set(testNames.map((name) => name.trim()).filter((name) => name.length > 0 && !/\s/.test(name)))];
10
11
  if (names.length === 0)
11
12
  return { ok: false, detail: "rust coverage: no test names" };
12
- const llvmCov = findTool("llvm-cov");
13
- const llvmProf = findTool("llvm-profdata");
14
- if (!llvmCov || !llvmProf)
13
+ if (!llvmToolsPresent())
15
14
  return { ok: false, detail: "rust coverage: llvm tool missing" };
16
15
  const env = { ...process.env, RUSTFLAGS: "-C instrument-coverage", CARGO_INCREMENTAL: "0" };
17
16
  const built = spawnSync("cargo", ["test", "--no-run", "--message-format=json"], {
@@ -43,11 +42,8 @@ export function writeRustCoverage(pkg, testNames, dest) {
43
42
  if (!new RegExp(`^test ${escapeRegExp(name)} \\.\\.\\. ok`, "m").test(`${ran.stdout}\n${ran.stderr}`))
44
43
  continue;
45
44
  const merged = path.join(work, `${safe(name)}-${safe(path.basename(exe))}.profdata`);
46
- const merge = spawnSync(llvmProf, ["merge", "-sparse", "-o", merged, raw], { encoding: "utf8" });
47
- if (merge.status !== 0)
48
- continue;
49
- const exported = spawnSync(llvmCov, ["export", exe, `-instr-profile=${merged}`], { encoding: "utf8", maxBuffer: 64 * 1024 * 1024 });
50
- if (exported.status !== 0)
45
+ const exported = exportLlvmProfile(exe, [raw], merged);
46
+ if (!exported.ok)
51
47
  continue;
52
48
  for (const hit of llvmHits(exported.stdout, pkg))
53
49
  addHit(files, hit.file, hit.line, name);
@@ -136,17 +132,6 @@ function addHit(files, file, line, test) {
136
132
  lines[key] = names;
137
133
  files[file] = lines;
138
134
  }
139
- function findTool(name) {
140
- const fromXcrun = spawnSync("xcrun", ["--find", name], { encoding: "utf8" });
141
- const xcrunPath = fromXcrun.stdout?.trim() ?? "";
142
- if (fromXcrun.status === 0 && xcrunPath && existsSync(xcrunPath))
143
- return xcrunPath;
144
- const which = spawnSync("sh", ["-c", `command -v ${name}`], { encoding: "utf8" });
145
- const found = which.stdout?.trim() ?? "";
146
- if (which.status === 0 && found)
147
- return found;
148
- return null;
149
- }
150
135
  function safe(value) {
151
136
  return value.replace(/[^A-Za-z0-9._-]+/g, "_");
152
137
  }
@@ -1,4 +1,5 @@
1
- import { readFileSync } from "node:fs";
1
+ import { readdirSync, readFileSync, statSync } from "node:fs";
2
+ import path from "node:path";
2
3
  /** The label stays in its own file. The run report must not repeat these three values. */
3
4
  export function readSealedLabel(file) {
4
5
  let text = "";
@@ -18,6 +19,44 @@ export function readSealedLabel(file) {
18
19
  export function labelLeaked(body, label) {
19
20
  return body.includes(label.bugId) || body.includes(label.issueId) || body.includes(label.expectedFail);
20
21
  }
22
+ /** The envelope can be clean while a result file on disk still repeats the label. */
23
+ export function sealedTreeLeaked(dir, label) {
24
+ const visit = (current) => {
25
+ let names;
26
+ try {
27
+ names = readdirSync(current);
28
+ }
29
+ catch {
30
+ return false;
31
+ }
32
+ for (const name of names) {
33
+ const full = path.join(current, name);
34
+ let info;
35
+ try {
36
+ info = statSync(full);
37
+ }
38
+ catch {
39
+ continue;
40
+ }
41
+ if (info.isDirectory()) {
42
+ if (visit(full))
43
+ return true;
44
+ continue;
45
+ }
46
+ if (!info.isFile() || info.size > 2_000_000)
47
+ continue;
48
+ try {
49
+ if (labelLeaked(readFileSync(full, "utf8"), label))
50
+ return true;
51
+ }
52
+ catch {
53
+ // A binary artifact is not a report.
54
+ }
55
+ }
56
+ return false;
57
+ };
58
+ return visit(dir);
59
+ }
21
60
  export const LABEL_LEAK_SUMMARY = "The label was copied into the report.";
22
61
  function field(text, key) {
23
62
  const match = text.match(new RegExp(`^${key}=(.*)$`, "m"));
@@ -386,8 +386,8 @@ async function launch(pkg, spec, files, pattern, names, timeoutMs, env, hooks) {
386
386
  if (spec.kind === "swift")
387
387
  return tag(runSwift(pkg, timeoutMs, env), spec.command, wantsNames, selection);
388
388
  if (spec.kind === "maven") {
389
- const tests = selection.map(surefireTest).filter((name) => name.length > 0);
390
- const args = ["-B", "test", ...(tests.length > 0 ? [`-Dtest=${tests.join(",")}`] : [])];
389
+ const tests = surefireArg(uniqueSurefire(selection));
390
+ const args = ["-B", "test", ...(tests.length > 0 ? [`-Dtest=${tests}`] : [])];
391
391
  const whole = wantsNames && tests.length === 0;
392
392
  // Surefire leaves the previous run's XML in place. A confirm that did not
393
393
  // re-execute a class would otherwise read that class as failed again.
@@ -468,12 +468,47 @@ function goRunFilter(names, pattern) {
468
468
  return null;
469
469
  return `^(${names.map(escapeRegExp).join("|")})$`;
470
470
  }
471
+ function uniqueSurefire(names) {
472
+ const out = [];
473
+ const seen = new Set();
474
+ for (const name of names) {
475
+ const selector = surefireTest(name);
476
+ if (!selector || seen.has(selector))
477
+ continue;
478
+ seen.add(selector);
479
+ out.push(selector);
480
+ }
481
+ return out;
482
+ }
483
+ /** Surefire 2.12 splits `-Dtest` on commas and then reads `#method+method`. `Class#a,Class#b` runs only `a`. */
484
+ export function surefireArg(tests) {
485
+ const groups = new Map();
486
+ for (const test of tests) {
487
+ const hash = test.indexOf("#");
488
+ const klass = hash === -1 ? test : test.slice(0, hash);
489
+ const method = hash === -1 ? "" : test.slice(hash + 1);
490
+ const methods = groups.get(klass);
491
+ if (methods) {
492
+ if (method)
493
+ methods.push(method);
494
+ }
495
+ else {
496
+ groups.set(klass, method ? [method] : []);
497
+ }
498
+ }
499
+ return [...groups.entries()]
500
+ .map(([klass, methods]) => (methods.length > 0 ? `${klass}#${methods.join("+")}` : klass))
501
+ .join(",");
502
+ }
471
503
  function surefireTest(name) {
472
504
  const leaf = name.split("::").pop() ?? name;
473
505
  const dot = leaf.lastIndexOf(".");
474
506
  const body = dot <= 0 ? leaf : `${leaf.slice(0, dot)}#${leaf.slice(dot + 1)}`;
475
- // testShut[0] collects nothing. method(Parser)[1] keeps the parameter types.
476
- return body.replace(/\[[^\]]*\]$/, "");
507
+ // testShut[0] matches nothing. A bare testShut matches on Surefire 3.6 and runs
508
+ // zero tests on 3.5. testShut* matches every invocation on both. method(Parser)[1] keeps the types.
509
+ const invocation = /\[[^\]]*\]$/.test(body);
510
+ const method = body.replace(/\[[^\]]*\]$/, "");
511
+ return invocation ? `${method}*` : method;
477
512
  }
478
513
  function escapeRegExp(value) {
479
514
  return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
@@ -764,7 +799,8 @@ function scriptInvocation(pkg) {
764
799
  if (!suiteNeedsTty(pkg))
765
800
  return { bin: "bash", args: ["run_tests.sh"], command: "bash run_tests.sh" };
766
801
  if (gnuScript()) {
767
- const args = ["-q", "-c", "bash run_tests.sh", "/dev/null"];
802
+ // -e returns the child status. Without it, util-linux script exits 0 and a failing suite looks green.
803
+ const args = ["-q", "-e", "-c", "bash run_tests.sh", "/dev/null"];
768
804
  return { bin: "script", args, command: shown("script", args) };
769
805
  }
770
806
  const args = ["-q", "/dev/null", "bash", "run_tests.sh"];
@@ -1403,6 +1439,11 @@ function mavenReport(pkg, text, code) {
1403
1439
  }
1404
1440
  return parseSurefire(pkg) ?? parseMavenText(text);
1405
1441
  }
1442
+ /** An attribute on the opening tag. `classname` must not be read as `name`. */
1443
+ export function surefireAttr(fragment, key) {
1444
+ const open = fragment.slice(0, fragment.indexOf(">") === -1 ? fragment.length : fragment.indexOf(">"));
1445
+ return new RegExp(`(?:^|\\s)${key}="([^"]*)"`).exec(open)?.[1] ?? "";
1446
+ }
1406
1447
  function parseSurefire(pkg) {
1407
1448
  const dir = path.join(pkg, "target", "surefire-reports");
1408
1449
  if (!existsSync(dir))
@@ -1414,11 +1455,11 @@ function parseSurefire(pkg) {
1414
1455
  continue;
1415
1456
  const xml = readFileSync(path.join(dir, name), "utf8");
1416
1457
  for (const part of xml.split("<testcase ").slice(1)) {
1417
- const test = /name="([^"]+)"/.exec(part)?.[1];
1418
- const klass = /classname="([^"]+)"/.exec(part)?.[1] ?? "";
1458
+ const test = surefireAttr(part, "name");
1459
+ const klass = surefireAttr(part, "classname");
1419
1460
  if (!test)
1420
1461
  continue;
1421
- const id = `${klass}.${test}`;
1462
+ const id = klass ? `${klass}.${test}` : test;
1422
1463
  names.push(id);
1423
1464
  if (part.includes("<failure") || part.includes("<error"))
1424
1465
  failed.push({ name: id, file: klass, line: 0 });
@@ -105,7 +105,7 @@ export function tallyRun(outDir) {
105
105
  command: "mutate.tally",
106
106
  summary,
107
107
  next,
108
- nextCall: keep.length > 0 ? { argv: ["mutate", "run", ...keep.flatMap((id) => ["--only-test", id])] } : null,
108
+ nextCall: resumeArgv(outDir, keep),
109
109
  keep,
110
110
  drop,
111
111
  gaps,
@@ -117,6 +117,43 @@ export function tallyRun(outDir) {
117
117
  writeFileSync(fullPath, `${JSON.stringify(complete, null, 2)}\n`);
118
118
  return { ...complete, gaps: shown, rest: gaps.length - shown.length };
119
119
  }
120
+ function resumeArgv(outDir, keep) {
121
+ if (keep.length === 0)
122
+ return null;
123
+ let stored = {};
124
+ try {
125
+ stored = JSON.parse(readFileSync(path.join(outDir, "run.json"), "utf8"));
126
+ }
127
+ catch {
128
+ stored = {};
129
+ }
130
+ const argv = ["mutate", "run"];
131
+ if (stored.package)
132
+ argv.push("--package", stored.package);
133
+ for (const dir of stored.patches ?? [])
134
+ argv.push("--patches", dir);
135
+ argv.push("--out", stored.out || outDir);
136
+ if (stored.repo)
137
+ argv.push("--repo", stored.repo);
138
+ if (stored.commit)
139
+ argv.push("--commit", stored.commit);
140
+ if (typeof stored.workers === "number")
141
+ argv.push("--workers", String(stored.workers));
142
+ if (stored.confirm === false)
143
+ argv.push("--no-confirm");
144
+ if (stored.testsDir && stored.testsDir !== "tests")
145
+ argv.push("--tests-dir", stored.testsDir);
146
+ if (typeof stored.suiteTimeoutMs === "number")
147
+ argv.push("--suite-timeout-ms", String(stored.suiteTimeoutMs));
148
+ if (typeof stored.testTimeoutMs === "number")
149
+ argv.push("--test-timeout-ms", String(stored.testTimeoutMs));
150
+ if (stored.direction && stored.direction !== "auto")
151
+ argv.push("--direction", stored.direction);
152
+ if (stored.suiteCommand)
153
+ argv.push("--suite-command", stored.suiteCommand);
154
+ argv.push(...keep.flatMap((id) => ["--only-test", id]));
155
+ return { argv };
156
+ }
120
157
  function readResults(dir) {
121
158
  const out = [];
122
159
  for (const name of readdirSync(dir)) {
@@ -0,0 +1,13 @@
1
+ /** `--experimental-strip-types` needs this. Older Node is refused before a command runs. */
2
+ export const MIN_NODE = "22.6";
3
+ export function nodeTooOld(version) {
4
+ const match = /^(\d+)\.(\d+)/.exec(version);
5
+ const required = `Node ${version} is too old. Probatio needs Node ${MIN_NODE} or newer.`;
6
+ if (!match)
7
+ return required;
8
+ const major = Number(match[1]);
9
+ const minor = Number(match[2]);
10
+ if (major > 22 || (major === 22 && minor >= 6))
11
+ return null;
12
+ return required;
13
+ }
@@ -1,13 +1,19 @@
1
1
  import { existsSync, mkdirSync, readdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
2
2
  import path from "node:path";
3
3
  import { loadSealed } from "../memory/state.js";
4
+ /** One path segment. Rejects slashes and `..` so a claim cannot leave the queue directory. */
5
+ export function safeFilePart(value) {
6
+ if (!/^[A-Za-z0-9._-]+$/.test(value) || value === "." || value === "..")
7
+ return null;
8
+ return value;
9
+ }
4
10
  export function seedQueue(root, ids) {
5
11
  const sealed = new Set(loadSealed(root));
6
12
  const queue = path.join(root, "queue");
7
13
  mkdirSync(queue, { recursive: true });
8
14
  const written = [];
9
15
  for (const id of [...ids].sort()) {
10
- if (sealed.has(id))
16
+ if (sealed.has(id) || !safeFilePart(id))
11
17
  continue;
12
18
  const dest = path.join(queue, `${id}.json`);
13
19
  if (existsSync(dest))
@@ -19,23 +25,23 @@ export function seedQueue(root, ids) {
19
25
  }
20
26
  /** Claim by atomic rename. A second claim fails because the queue file is already gone. */
21
27
  export function claimItem(root, id, agent, now, leaseMs) {
28
+ if (!safeFilePart(id) || !safeFilePart(agent))
29
+ return { ok: false, summary: "agent and id must be a single file name.", file: null };
22
30
  const source = path.join(root, "queue", `${id}.json`);
23
31
  const claimed = path.join(root, "claimed");
24
32
  const dest = path.join(claimed, `${agent}-${id}.json`);
25
33
  if (!existsSync(source))
26
34
  return { ok: false, summary: `${id} is not in the queue.`, file: null };
27
35
  mkdirSync(claimed, { recursive: true });
28
- const body = JSON.parse(readFileSync(source, "utf8"));
29
- body.id = id;
30
- body.agent = agent;
31
- body.leasedUntil = now + leaseMs;
32
- writeFileSync(source, `${JSON.stringify(body)}\n`);
36
+ // Rename first. Writing the body before the rename lets the other claim overwrite the winner.
33
37
  try {
34
38
  renameSync(source, dest);
35
39
  }
36
40
  catch {
37
41
  return { ok: false, summary: `${id} was claimed by someone else.`, file: null };
38
42
  }
43
+ const body = { id, agent, leasedUntil: now + leaseMs };
44
+ writeFileSync(dest, `${JSON.stringify(body)}\n`);
39
45
  return { ok: true, summary: `${agent} claimed ${id} until ${body.leasedUntil}.`, file: dest };
40
46
  }
41
47
  /** Move claims whose lease has expired back to the queue. */
@@ -53,7 +59,7 @@ export function reapClaims(root, now) {
53
59
  const body = JSON.parse(readFileSync(file, "utf8"));
54
60
  if (typeof body.leasedUntil !== "number" || body.leasedUntil > now)
55
61
  continue;
56
- if (!body.id)
62
+ if (!body.id || !safeFilePart(body.id))
57
63
  continue;
58
64
  renameSync(file, path.join(queue, `${body.id}.json`));
59
65
  returned.push(body.id);
@@ -23,6 +23,11 @@ export async function verifyChange(options) {
23
23
  const ranges = parseUnified(diff.stdout);
24
24
  const names = git(repo, ["diff", "--name-only", "--no-renames", options.base, options.commit, ...scope]);
25
25
  const changed = names.stdout.split(/\r?\n/).map((line) => line.trim()).filter(Boolean);
26
+ const status = git(repo, ["status", "--porcelain", ...scope]);
27
+ const dirty = status.stdout.trim().length > 0;
28
+ if (dirty && (options.base === options.commit || changed.length === 0)) {
29
+ return fail(options.outDir, "The working tree has uncommitted changes. verify-change did not score them. Commit the edit, or pass --base and --commit that name different commits with a diff.");
30
+ }
26
31
  const goldenContract = goldenDiffs(repo, options.base, options.commit, changed);
27
32
  const packageFiles = [];
28
33
  const chosen = [];
package/package.json CHANGED
@@ -1,10 +1,11 @@
1
1
  {
2
2
  "name": "probatio",
3
- "version": "0.1.2",
3
+ "version": "0.1.3",
4
4
  "description": "Mutation runner for AI agents. Scores a test suite by the bugs it catches.",
5
5
  "type": "module",
6
6
  "bin": {
7
- "probatio": "dist/cli.js"
7
+ "probatio": "dist/cli.js",
8
+ "probatio-mcp": "dist/mcp.js"
8
9
  },
9
10
  "files": [
10
11
  "dist"
@@ -20,11 +21,12 @@
20
21
  },
21
22
  "scripts": {
22
23
  "build": "tsc -p tsconfig.json && mkdir -p dist/mutate && cp src/mutate/fail-reporter.mjs src/mutate/node-coverage.mjs src/mutate/precise-lines.mjs src/mutate/child-lines.mjs src/mutate/mocha-coverage.cjs src/mutate/node-batch.mjs src/mutate/node-batch-loader.mjs src/mutate/py-reporter.py src/mutate/py-pytest-reporter.py src/mutate/probatio-jacoco-run.java dist/mutate/",
24
+ "prepack": "npm run build",
23
25
  "test": "tsx --test tests/*.test.ts",
24
26
  "probatio": "tsx src/cli.ts"
25
27
  },
26
28
  "engines": {
27
- "node": ">=22"
29
+ "node": ">=22.6"
28
30
  },
29
31
  "dependencies": {
30
32
  "typescript": "^5.7.2"