probatio 0.0.1 → 0.0.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -5,7 +5,7 @@ Probatio (Latin: a testing, a proof) is a free, open-source testing toolkit buil
5
5
  it remembers what was fixed so agents with small context do not redo or undo work; and it finds
6
6
  the bugs a test suite would miss instead of asking anyone to read more tests.
7
7
 
8
- Status (2026-10-05): `mutate run` discovers the project suite and scores it. Python, JavaScript, and TypeScript narrow each mutant with a line map collected on the baseline. C does the same when the binary was built with LLVM coverage. A suite with no line map, a baseline of 5 seconds or more, and more than 30 mutants stops before the first mutant. `verify-change` mutates the changed lines and uses that same map. `mutate tally` reads a finished run, names tests that killed nothing, and does not delete a test. The MCP server returns the same JSON as the CLI.
8
+ Status (2026-10-05): `mutate run` discovers the project suite and scores it. Python, JavaScript, and TypeScript narrow each mutant with a line map collected on the baseline. C does the same when the binary was built with LLVM coverage. A suite with no line map, a baseline of 5 seconds or more, and more than 30 mutants stops before the first mutant. `verify-change` mutates the changed lines and uses that same map. `mutate tally` reads a finished run, names tests that killed nothing, and does not delete a test. The MCP server speaks one JSON object per line and returns the same JSON as the CLI.
9
9
 
10
10
  Read in this order:
11
11
 
@@ -23,19 +23,19 @@ One JSON object on stdout. `schemaVersion`, `ok`, `summary`, `next`, `nextCall`.
23
23
 
24
24
  ```bash
25
25
  npx probatio mutate generate --package . --out .probatio/generate
26
- npx probatio mutate run --package . --patches .probatio/generate/mutants --out .probatio/runs --no-build
26
+ npx probatio mutate run --package . --patches .probatio/generate/mutants --out .probatio/runs
27
27
  npx probatio mutate tally --out .probatio/runs
28
28
  ```
29
29
 
30
30
  `generate` writes operator mutants (conditions, `&&`/`||`, `===`/`!==`, boundaries, booleans, a dropped `!`). The same class is written for COBOL (`.cob`, `.cbl`), Rust, C, C++, Java, Go, Python, JavaScript, and C#. Strings and comments are not mutated. A COBOL `*>` comment and a fixed-format line whose column 7 is `*` or `/` are comments. Each patch is forward: apply it to introduce the bug, and the file says so. When mutants were written, `next` states the count, the suite command if one was discovered, and whether the first run collects a line map.
31
31
 
32
- `run` discovers the suite: `run_tests.sh`, Cargo, Go, Swift, Maven, dotnet, pytest, unittest, node:test, Mocha, then `make test` when COBOL tests sit under that Makefile. An unknown layout stops and asks for `--suite-command`. It does not compile one file and call that the suite. `make test` that would curl or wget a missing file stops, and nothing is fetched. A worktree that has no `node_modules` uses the main checkout's.
32
+ `run` discovers the suite: `run_tests.sh`, Cargo, Go, Swift, Maven, dotnet, pytest, unittest, node:test, Mocha, then `make test` when COBOL tests sit under that Makefile. An unknown layout stops and asks for `--suite-command`. It does not compile one file and call that the suite. `make test` that would curl or wget a missing file stops, and nothing is fetched. A worktree that has no `node_modules` uses the main checkout's. Omitting `--build` runs no build. Pass `--build` with a command when the suite needs one first.
33
33
 
34
- Python, JavaScript, and TypeScript collect a line map on the baseline. C collects one when LLVM coverage was instrumented. A later mutant runs the tests that executed its line. A line no test executed is `no coverage`, not a survivor. Go, Java, Rust, and C# can take a test name. Without a line map they still run the suite you gave them. A baseline of 5 seconds or more, with no line map and more than 30 mutants still pending, stops before the first mutant. `next` names that baseline and tells you to narrow `--src` or pass a smaller patch directory.
34
+ Python, JavaScript, and TypeScript collect a line map on the baseline. C collects one when LLVM coverage was instrumented. Go and Maven/Java collect file, line, and the test names that hit that line. A later mutant runs only those tests (`go test -run`, Maven `-Dtest`). A line no test executed is `no coverage`, and the suite is not started. Rust and C# can take a test name and do not collect a line. A baseline of 5 seconds or more, with no line map and more than 30 mutants still pending, stops before the first mutant. `next` names that baseline and tells you to narrow `--src` or pass a smaller patch directory. A one-file C, C++, Java, COBOL, or assembly launcher is not a suite discovery returns.
35
35
 
36
36
  The mutant timeout is the baseline duration times 5, and at least 20 seconds, capped by `--suite-timeout-ms`. A timeout is not a kill. `--budget-ms` stops mutant work after the baseline. The first mutant still runs, except for that no-line-map stop. A kill is the test that failed, or a compiler token when the mutant did not build. Confirm is on by default and reruns the failing names. Pytest names that `-k` cannot express are passed as node ids. A usage error, including pytest exit 4, is not a kill. `verify-change` does not confirm a second time. Add `--affected` to limit the file list to tests that can see the change. The default, once a line map exists, runs the tests on the changed line.
37
37
 
38
- `tally` reads that run directory. It names tests that killed a mutant, tests that killed nothing, and one gap for each survivor and each uncovered line. `deletedTests` is 0. It does not delete a file.
38
+ `tally` reads that run directory. The summary leads with the no-coverage count, then the survivors. It names tests that killed a mutant and tests that killed nothing. A gap is a survivor. An unseen line is not a gap and not a pass. A timeout is neither a kill nor a gap. `deletedTests` is 0. It does not delete a file.
39
39
 
40
40
  ## ledger
41
41
 
package/dist/cli.js CHANGED
@@ -143,12 +143,7 @@ async function runCommand(flags) {
143
143
  if (direction !== "auto" && direction !== "forward" && direction !== "reverse") {
144
144
  throw new Error("--direction must be auto, forward, or reverse");
145
145
  }
146
- const buildValues = flags.get("build");
147
- const build = !buildValues || buildValues.length === 0
148
- ? ["npm", "run", "build:mcp", "--silent"]
149
- : buildValues[buildValues.length - 1] === false
150
- ? null
151
- : String(buildValues[buildValues.length - 1]).split(" ").filter(Boolean);
146
+ const build = buildCommand(flags);
152
147
  const report = await runMutants({
153
148
  packageDir,
154
149
  repoDir,
@@ -444,12 +439,7 @@ async function checkKillCommand(flags, id) {
444
439
  const patchDirs = texts(flags, "patches").map((dir) => path.resolve(dir));
445
440
  if (patchDirs.length === 0)
446
441
  throw new Error("--patches is required");
447
- const buildValues = flags.get("build");
448
- const build = !buildValues || buildValues.length === 0
449
- ? null
450
- : buildValues[buildValues.length - 1] === false
451
- ? null
452
- : String(buildValues[buildValues.length - 1]).split(" ").filter(Boolean);
442
+ const build = buildCommand(flags);
453
443
  return checkKill(id, {
454
444
  packageDir,
455
445
  repoDir: path.resolve(text(flags, "repo") ?? gitRoot(packageDir)),
@@ -496,3 +486,14 @@ function finish(envelope, human) {
496
486
  process.stdout.write(render(envelope, human));
497
487
  process.exit(envelope.ok ? 0 : 1);
498
488
  }
489
+ /** Omitted `--build` and `--no-build` both mean no build. `--build <cmd>` opts in. */
490
+ function buildCommand(flags) {
491
+ const values = flags.get("build");
492
+ if (!values || values.length === 0)
493
+ return null;
494
+ const last = values[values.length - 1];
495
+ if (typeof last !== "string")
496
+ return null;
497
+ const parts = last.split(" ").filter(Boolean);
498
+ return parts.length > 0 ? parts : null;
499
+ }
package/dist/mcp.js CHANGED
@@ -1,35 +1,39 @@
1
1
  #!/usr/bin/env node
2
2
  import { spawnSync } from "node:child_process";
3
- import { existsSync } from "node:fs";
3
+ import { existsSync, readFileSync } from "node:fs";
4
4
  import path from "node:path";
5
5
  import { fileURLToPath } from "node:url";
6
6
  const here = path.dirname(fileURLToPath(import.meta.url));
7
- let buffer = Buffer.alloc(0);
7
+ const version = readVersion();
8
+ const tool = {
9
+ name: "probatio",
10
+ description: "Run one probatio command. argv is the CLI words and flags.",
11
+ inputSchema: {
12
+ type: "object",
13
+ properties: { argv: { type: "array", items: { type: "string" } } },
14
+ required: ["argv"],
15
+ },
16
+ };
17
+ let buffer = "";
18
+ process.stdin.setEncoding("utf8");
8
19
  process.stdin.on("data", (chunk) => {
9
- buffer = Buffer.concat([buffer, chunk]);
20
+ buffer += chunk;
10
21
  drain();
11
22
  });
12
- process.stdin.on("end", () => process.exit(0));
13
- function drain() {
14
- for (;;) {
15
- const headerEnd = buffer.indexOf("\r\n\r\n");
16
- if (headerEnd === -1)
17
- return;
18
- const header = buffer.subarray(0, headerEnd).toString("utf8");
19
- const match = header.match(/Content-Length:\s*(\d+)/i);
20
- const start = headerEnd + 4;
21
- if (!match) {
22
- buffer = buffer.subarray(start);
23
+ process.stdin.on("end", () => {
24
+ drain(true);
25
+ process.exit(0);
26
+ });
27
+ function drain(flush = false) {
28
+ const lines = buffer.split("\n");
29
+ buffer = flush ? "" : lines.pop() ?? "";
30
+ for (const line of lines) {
31
+ const text = line.replace(/\r$/, "").trim();
32
+ if (!text)
23
33
  continue;
24
- }
25
- const length = Number(match[1]);
26
- if (buffer.length < start + length)
27
- return;
28
- const body = buffer.subarray(start, start + length).toString("utf8");
29
- buffer = buffer.subarray(start + length);
30
34
  let message;
31
35
  try {
32
- message = JSON.parse(body);
36
+ message = JSON.parse(text);
33
37
  }
34
38
  catch {
35
39
  continue;
@@ -49,29 +53,14 @@ function handle(message) {
49
53
  result: {
50
54
  protocolVersion: "2024-11-05",
51
55
  capabilities: { tools: {} },
52
- serverInfo: { name: "probatio", version: "0.0.1" },
56
+ serverInfo: { name: "probatio", version },
57
+ tools: [tool],
53
58
  },
54
59
  });
55
60
  return;
56
61
  }
57
62
  if (message.method === "tools/list") {
58
- send({
59
- jsonrpc: "2.0",
60
- id: message.id,
61
- result: {
62
- tools: [
63
- {
64
- name: "probatio",
65
- description: "Run one probatio command. argv is the CLI words and flags.",
66
- inputSchema: {
67
- type: "object",
68
- properties: { argv: { type: "array", items: { type: "string" } } },
69
- required: ["argv"],
70
- },
71
- },
72
- ],
73
- },
74
- });
63
+ send({ jsonrpc: "2.0", id: message.id, result: { tools: [tool] } });
75
64
  return;
76
65
  }
77
66
  if (message.method === "tools/call") {
@@ -112,8 +101,16 @@ function cliCommand() {
112
101
  const tsx = path.resolve(here, "..", "node_modules", ".bin", "tsx");
113
102
  return { bin: tsx, args: [path.join(here, "cli.ts")] };
114
103
  }
104
+ function readVersion() {
105
+ try {
106
+ const raw = JSON.parse(readFileSync(path.resolve(here, "..", "package.json"), "utf8"));
107
+ return typeof raw.version === "string" && raw.version.length > 0 ? raw.version : "0.0.1";
108
+ }
109
+ catch {
110
+ return "0.0.1";
111
+ }
112
+ }
115
113
  function send(message) {
116
- const payload = Buffer.from(JSON.stringify(message), "utf8");
117
- process.stdout.write(`Content-Length: ${payload.length}\r\n\r\n`);
118
- process.stdout.write(payload);
114
+ const line = JSON.stringify(message);
115
+ process.stdout.write(`${line}\n`);
119
116
  }
@@ -0,0 +1,100 @@
1
+ import { spawnSync } from "node:child_process";
2
+ import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
3
+ import path from "node:path";
4
+ /** Blocks in a Go cover profile whose count is above zero. */
5
+ export function parseCoverProfile(text, modulePath) {
6
+ const hits = [];
7
+ const prefix = modulePath ? `${modulePath}/` : "";
8
+ for (const raw of text.split("\n")) {
9
+ const line = raw.trim();
10
+ if (!line || line.startsWith("mode:"))
11
+ continue;
12
+ const match = /^(.+):(\d+)\.\d+,(\d+)\.\d+\s+\d+\s+(\d+)$/.exec(line);
13
+ if (!match)
14
+ continue;
15
+ if (Number(match[4]) <= 0)
16
+ continue;
17
+ let file = match[1];
18
+ if (prefix && file.startsWith(prefix))
19
+ file = file.slice(prefix.length);
20
+ file = file.split("\\").join("/");
21
+ const start = Number(match[2]);
22
+ const end = Number(match[3]);
23
+ for (let number = start; number <= end; number++)
24
+ hits.push({ file, line: number });
25
+ }
26
+ return hits;
27
+ }
28
+ /**
29
+ * One compiled test binary per package, then one profile per test name.
30
+ * A stock cover profile does not name the test, so each test gets its own.
31
+ */
32
+ export function writeGoCoverage(pkg, testNames, dest) {
33
+ const wanted = new Set(testNames.filter((name) => name.length > 0 && !name.includes("/")));
34
+ if (wanted.size === 0)
35
+ return { ok: false, detail: "go coverage: no test names" };
36
+ const mod = run("go", ["list", "-m"], pkg);
37
+ if (mod.status !== 0)
38
+ return { ok: false, detail: clip(mod.stderr || mod.stdout || mod.error?.message || "go list -m failed") };
39
+ const modulePath = mod.stdout.trim().split("\n")[0]?.trim() ?? "";
40
+ const listed = run("go", ["list", "-f", "{{.ImportPath}}\t{{.Dir}}\t{{len .TestGoFiles}}\t{{len .XTestGoFiles}}", "./..."], pkg);
41
+ if (listed.status !== 0)
42
+ return { ok: false, detail: clip(listed.stderr || listed.stdout || listed.error?.message || "go list failed") };
43
+ const work = path.join(path.dirname(dest), "go-cover");
44
+ mkdirSync(work, { recursive: true });
45
+ const files = {};
46
+ const packages = listed.stdout.split("\n").map((line) => line.trim()).filter(Boolean);
47
+ for (const row of packages) {
48
+ const [importPath, dir, testFiles, xTestFiles] = row.split("\t");
49
+ if (!importPath || !dir)
50
+ continue;
51
+ if (Number(testFiles ?? "0") + Number(xTestFiles ?? "0") <= 0)
52
+ continue;
53
+ const bin = path.join(work, `${safe(importPath)}.test`);
54
+ const compiled = run("go", ["test", "-c", "-covermode=set", "-o", bin, importPath], pkg);
55
+ if (compiled.status !== 0 || compiled.error)
56
+ continue;
57
+ const listedTests = run(bin, ["-test.list", "."], dir);
58
+ if (listedTests.status !== 0)
59
+ continue;
60
+ for (const name of listedTests.stdout.split("\n").map((item) => item.trim()).filter((item) => wanted.has(item))) {
61
+ const profile = path.join(work, `${safe(importPath)}-${safe(name)}.out`);
62
+ run(bin, ["-test.run", `^${escapeRegExp(name)}$`, "-test.coverprofile", profile], dir);
63
+ let text = "";
64
+ try {
65
+ text = readFileSync(profile, "utf8");
66
+ }
67
+ catch {
68
+ continue;
69
+ }
70
+ for (const hit of parseCoverProfile(text, modulePath))
71
+ addHit(files, hit.file, hit.line, name);
72
+ }
73
+ }
74
+ if (Object.keys(files).length === 0)
75
+ return { ok: false, detail: "go coverage: no covered lines" };
76
+ writeFileSync(dest, `${JSON.stringify({ files }, null, 2)}\n`);
77
+ return { ok: true, detail: `${Object.keys(files).length} files` };
78
+ }
79
+ function addHit(files, file, line, test) {
80
+ const lines = files[file] ?? {};
81
+ const key = String(line);
82
+ const names = lines[key] ?? [];
83
+ if (!names.includes(test))
84
+ names.push(test);
85
+ lines[key] = names;
86
+ files[file] = lines;
87
+ }
88
+ function safe(value) {
89
+ return value.replace(/[^A-Za-z0-9._-]+/g, "_");
90
+ }
91
+ function escapeRegExp(value) {
92
+ return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
93
+ }
94
+ function run(bin, args, cwd) {
95
+ return spawnSync(bin, args, { cwd, encoding: "utf8", timeout: 120_000 });
96
+ }
97
+ function clip(text) {
98
+ const line = text.trim().split("\n").find((item) => item.trim()) ?? "go coverage failed";
99
+ return line.slice(0, 240);
100
+ }
@@ -0,0 +1,188 @@
1
+ import { spawnSync } from "node:child_process";
2
+ import { existsSync, mkdirSync, readdirSync, readFileSync, writeFileSync } from "node:fs";
3
+ import path from "node:path";
4
+ import { fileURLToPath } from "node:url";
5
+ const JACOCO = "0.8.15";
6
+ const ASM = "9.10.1";
7
+ const RUNNER = fileURLToPath(new URL("./probatio-jacoco-run.java", import.meta.url));
8
+ /**
9
+ * One JVM, one JaCoCo dump per Surefire test method.
10
+ * The default jacoco.exec file does not name the test.
11
+ */
12
+ export function writeJavaCoverage(pkg, testNames, dest) {
13
+ const tests = testNames.map(junitMethod).filter((name) => name !== null);
14
+ if (tests.length === 0)
15
+ return { ok: false, detail: "java coverage: no class.method test names" };
16
+ const classDirs = findClassDirs(pkg);
17
+ if (classDirs.length === 0)
18
+ return { ok: false, detail: "java coverage: no target/classes" };
19
+ const work = path.join(path.dirname(dest), "java-cover");
20
+ mkdirSync(work, { recursive: true });
21
+ const jars = resolveJars(pkg, work);
22
+ if ("error" in jars)
23
+ return { ok: false, detail: jars.error };
24
+ const source = path.join(work, "ProbatioJacocoRun.java");
25
+ writeFileSync(source, readFileSync(RUNNER));
26
+ const compileCp = [jars.agent, jars.junit, jars.core, jars.asm, jars.asmCommons, jars.asmTree].join(path.delimiter);
27
+ const compiled = run(javaTool("javac"), ["-encoding", "UTF-8", "-cp", compileCp, "-d", work, source], pkg);
28
+ if (compiled.status !== 0)
29
+ return { ok: false, detail: clip(compiled.stderr || compiled.stdout || compiled.error?.message || "javac failed") };
30
+ const testsFile = path.join(work, "tests.txt");
31
+ writeFileSync(testsFile, `${tests.join("\n")}\n`);
32
+ const projectCp = buildClasspath(pkg, work);
33
+ if ("error" in projectCp)
34
+ return { ok: false, detail: projectCp.error };
35
+ // Our asm and JaCoCo must win over a project that still ships an older asm. That older reader rejects Java 27 class files.
36
+ const cp = [work, jars.core, jars.asm, jars.asmCommons, jars.asmTree, jars.junit, ...classDirs, ...testClassDirs(pkg), projectCp.cp].filter(Boolean).join(path.delimiter);
37
+ const dump = path.join(work, "hits.txt");
38
+ const ran = run(javaTool("java"), [`-javaagent:${jars.agent}=output=none,excludes=ProbatioJacocoRun`, "-cp", cp, "ProbatioJacocoRun", classDirs.join(path.delimiter), testsFile, dump, pkg], pkg);
39
+ if (ran.error && ran.error.code === "ETIMEDOUT") {
40
+ return { ok: false, detail: "java coverage: timed out" };
41
+ }
42
+ if (ran.status !== 0 || !existsSync(dump)) {
43
+ return { ok: false, detail: clip(ran.stderr || ran.stdout || ran.error?.message || "java coverage runner failed") };
44
+ }
45
+ const files = hitsToMap(readFileSync(dump, "utf8"));
46
+ if (Object.keys(files).length === 0)
47
+ return { ok: false, detail: "java coverage: no covered lines" };
48
+ writeFileSync(dest, `${JSON.stringify({ files }, null, 2)}\n`);
49
+ return { ok: true, detail: `${Object.keys(files).length} files` };
50
+ }
51
+ export function hitsToMap(text) {
52
+ const files = {};
53
+ for (const raw of text.split("\n")) {
54
+ const line = raw.trim();
55
+ if (!line.startsWith("HIT "))
56
+ continue;
57
+ const parts = line.slice(4).split(" ");
58
+ if (parts.length < 3)
59
+ continue;
60
+ const test = parts.pop() ?? "";
61
+ const number = parts.pop() ?? "";
62
+ const file = parts.join(" ");
63
+ if (!file || !/^\d+$/.test(number) || !test)
64
+ continue;
65
+ const lines = files[file] ?? {};
66
+ const names = lines[number] ?? [];
67
+ if (!names.includes(test))
68
+ names.push(test);
69
+ lines[number] = names;
70
+ files[file] = lines;
71
+ }
72
+ return files;
73
+ }
74
+ function junitMethod(name) {
75
+ const leaf = name.split("::").pop() ?? name;
76
+ const dot = leaf.lastIndexOf(".");
77
+ if (dot <= 0 || dot === leaf.length - 1)
78
+ return null;
79
+ if (leaf.startsWith("maven-") || leaf === "javac")
80
+ return null;
81
+ return `${leaf.slice(0, dot)}#${leaf.slice(dot + 1)}`;
82
+ }
83
+ function findClassDirs(root) {
84
+ const found = [];
85
+ const walk = (dir, depth) => {
86
+ if (depth > 4)
87
+ return;
88
+ let entries = [];
89
+ try {
90
+ entries = readdirSync(dir);
91
+ }
92
+ catch {
93
+ return;
94
+ }
95
+ for (const name of entries) {
96
+ if (name === "node_modules" || name === ".git" || name === "java-cover")
97
+ continue;
98
+ const full = path.join(dir, name);
99
+ if (name === "target") {
100
+ const classes = path.join(full, "classes");
101
+ if (existsSync(classes))
102
+ found.push(classes);
103
+ continue;
104
+ }
105
+ if (name.startsWith("."))
106
+ continue;
107
+ walk(full, depth + 1);
108
+ }
109
+ };
110
+ walk(root, 0);
111
+ return found;
112
+ }
113
+ function testClassDirs(root) {
114
+ return findClassDirs(root).map((dir) => path.join(path.dirname(dir), "test-classes")).filter((dir) => existsSync(dir));
115
+ }
116
+ function resolveJars(pkg, work) {
117
+ const copies = [
118
+ [`org.jacoco:org.jacoco.agent:${JACOCO}:jar:runtime`, "agent.jar"],
119
+ [`org.jacoco:org.jacoco.core:${JACOCO}`, "core.jar"],
120
+ [`org.ow2.asm:asm:${ASM}`, "asm.jar"],
121
+ [`org.ow2.asm:asm-commons:${ASM}`, "asm-commons.jar"],
122
+ [`org.ow2.asm:asm-tree:${ASM}`, "asm-tree.jar"],
123
+ ["junit:junit:4.13.2", "junit.jar"],
124
+ ];
125
+ const paths = {};
126
+ for (const [artifact, name] of copies) {
127
+ const dest = path.join(work, name);
128
+ if (!existsSync(dest)) {
129
+ const before = new Set(readdirSync(work));
130
+ const copied = run("mvn", ["-B", "-q", "org.apache.maven.plugins:maven-dependency-plugin:3.6.1:copy", `-Dartifact=${artifact}`, `-DoutputDirectory=${work}`], pkg);
131
+ if (copied.status !== 0)
132
+ return { error: clip(copied.stderr || copied.stdout || copied.error?.message || `mvn copy ${artifact} failed`) };
133
+ const created = readdirSync(work).find((item) => !before.has(item) && item.endsWith(".jar"));
134
+ if (!created)
135
+ return { error: `java coverage: missing ${artifact}` };
136
+ writeFileSync(dest, readFileSync(path.join(work, created)));
137
+ }
138
+ paths[name] = dest;
139
+ }
140
+ return {
141
+ agent: paths["agent.jar"],
142
+ core: paths["core.jar"],
143
+ asm: paths["asm.jar"],
144
+ asmCommons: paths["asm-commons.jar"],
145
+ asmTree: paths["asm-tree.jar"],
146
+ junit: paths["junit.jar"],
147
+ };
148
+ }
149
+ function buildClasspath(pkg, work) {
150
+ const file = path.join(work, "classpath.txt");
151
+ const built = run("mvn", ["-B", "-q", "dependency:build-classpath", "-DincludeScope=test", `-Dmdep.outputFile=${file}`], pkg);
152
+ if (built.status !== 0)
153
+ return { error: clip(built.stderr || built.stdout || built.error?.message || "mvn classpath failed") };
154
+ try {
155
+ return { cp: readFileSync(file, "utf8").trim() };
156
+ }
157
+ catch {
158
+ return { error: "java coverage: classpath file missing" };
159
+ }
160
+ }
161
+ function javaTool(name) {
162
+ const home = process.env.JAVA_HOME;
163
+ if (home) {
164
+ const candidate = path.join(home, "bin", name);
165
+ if (existsSync(candidate) && toolWorks(candidate))
166
+ return candidate;
167
+ }
168
+ if (toolWorks(name))
169
+ return name;
170
+ for (const dir of ["/opt/homebrew/opt/openjdk/bin", "/usr/local/opt/openjdk/bin"]) {
171
+ const candidate = path.join(dir, name);
172
+ if (existsSync(candidate) && toolWorks(candidate))
173
+ return candidate;
174
+ }
175
+ return name;
176
+ }
177
+ function toolWorks(bin) {
178
+ const probed = spawnSync(bin, ["-version"], { encoding: "utf8" });
179
+ const text = `${probed.stdout ?? ""}${probed.stderr ?? ""}`;
180
+ return probed.status === 0 && !/Unable to locate a Java Runtime/i.test(text);
181
+ }
182
+ function run(bin, args, cwd) {
183
+ return spawnSync(bin, args, { cwd, encoding: "utf8", timeout: 180_000 });
184
+ }
185
+ function clip(text) {
186
+ const line = text.trim().split("\n").find((item) => item.trim()) ?? "java coverage failed";
187
+ return line.slice(0, 240);
188
+ }
@@ -0,0 +1,81 @@
1
+ import java.io.ByteArrayInputStream;
2
+ import java.io.File;
3
+ import java.nio.file.Files;
4
+ import java.util.List;
5
+ import org.jacoco.agent.rt.IAgent;
6
+ import org.jacoco.agent.rt.RT;
7
+ import org.jacoco.core.analysis.Analyzer;
8
+ import org.jacoco.core.analysis.CoverageBuilder;
9
+ import org.jacoco.core.analysis.IClassCoverage;
10
+ import org.jacoco.core.analysis.ILine;
11
+ import org.jacoco.core.data.ExecutionDataReader;
12
+ import org.jacoco.core.data.ExecutionDataStore;
13
+ import org.junit.runner.JUnitCore;
14
+ import org.junit.runner.Request;
15
+
16
+ /** One JaCoCo dump per JUnit 4 method. Prints "HIT <file> <line>" lines. */
17
+ public class ProbatioJacocoRun {
18
+ public static void main(String[] args) throws Exception {
19
+ if (args.length != 4) {
20
+ System.err.println("usage: ProbatioJacocoRun classDirs testsFile outFile packageRoot");
21
+ System.exit(2);
22
+ }
23
+ String[] classDirs = args[0].split(File.pathSeparator);
24
+ List<String> tests = Files.readAllLines(new File(args[1]).toPath());
25
+ File out = new File(args[2]);
26
+ File root = new File(args[3]);
27
+ IAgent agent = RT.getAgent();
28
+ StringBuilder text = new StringBuilder();
29
+ for (String spec : tests) {
30
+ int hash = spec.lastIndexOf('#');
31
+ if (hash <= 0) continue;
32
+ String className = spec.substring(0, hash);
33
+ String method = spec.substring(hash + 1);
34
+ agent.getExecutionData(true);
35
+ try {
36
+ new JUnitCore().run(Request.method(Class.forName(className), method));
37
+ } catch (Throwable ignored) {
38
+ continue;
39
+ }
40
+ byte[] data = agent.getExecutionData(false);
41
+ ExecutionDataStore store = new ExecutionDataStore();
42
+ ExecutionDataReader reader = new ExecutionDataReader(new ByteArrayInputStream(data));
43
+ reader.setExecutionDataVisitor(store);
44
+ reader.setSessionInfoVisitor(info -> { });
45
+ reader.read();
46
+ String testName = className + "." + method;
47
+ text.append("TEST ").append(testName).append('\n');
48
+ for (String dir : classDirs) {
49
+ if (dir.isEmpty()) continue;
50
+ File classes = new File(dir);
51
+ if (!classes.isDirectory()) continue;
52
+ CoverageBuilder builder = new CoverageBuilder();
53
+ Analyzer analyzer = new Analyzer(store, builder);
54
+ analyzer.analyzeAll(classes);
55
+ for (IClassCoverage cc : builder.getClasses()) {
56
+ String source = cc.getSourceFileName();
57
+ if (source == null) continue;
58
+ File file = sourceFile(classes, cc.getPackageName(), source);
59
+ if (file == null || !file.isFile()) continue;
60
+ String rel = root.toPath().relativize(file.toPath()).toString().replace('\\', '/');
61
+ int first = cc.getFirstLine();
62
+ int last = cc.getLastLine();
63
+ for (int line = first; line <= last; line++) {
64
+ ILine cov = cc.getLine(line);
65
+ if (cov.getInstructionCounter().getCoveredCount() <= 0) continue;
66
+ text.append("HIT ").append(rel).append(' ').append(line).append(' ').append(testName).append('\n');
67
+ }
68
+ }
69
+ }
70
+ }
71
+ Files.writeString(out.toPath(), text.toString());
72
+ }
73
+
74
+ static File sourceFile(File classes, String pkg, String source) {
75
+ File target = classes.getParentFile();
76
+ File module = target == null ? null : target.getParentFile();
77
+ if (module == null) return null;
78
+ String rel = (pkg == null || pkg.isEmpty() ? "" : pkg + "/") + source;
79
+ return new File(module, "src/main/java/" + rel);
80
+ }
81
+ }
@@ -6,9 +6,11 @@ import path from "node:path";
6
6
  import { fileURLToPath } from "node:url";
7
7
  import { affectedTests } from "./affected.js";
8
8
  import { binaryFromCommand, readCoverageMap, selectionFor, writeLlvmCoverage } from "./coverage-map.js";
9
+ import { writeGoCoverage } from "./go-coverage.js";
10
+ import { writeJavaCoverage } from "./java-coverage.js";
9
11
  import { applyPatch, changedLines, chooseDirection, filesInDiff, git, restoreTree } from "./patch.js";
10
12
  import { killLabel } from "./kill-label.js";
11
- import { TIMEOUT_FLOOR_MS, TIMEOUT_MULTIPLE, TIMEOUT_NEXT, WHOLE_PROGRAM_TEST, WHOLE_SUITE_NEXT, decideSuite, noCoverageNext, resumeNext, scaleStop, timeoutLimitMs, } from "./suite-decision.js";
13
+ import { TIMEOUT_FLOOR_MS, TIMEOUT_MULTIPLE, TIMEOUT_NEXT, WHOLE_PROGRAM_TEST, WHOLE_SUITE_NEXT, campaignSummary, decideSuite, hasLineMap, noCoverageNext, resumeNext, scaleStop, timeoutLimitMs, } from "./suite-decision.js";
12
14
  import { commandSuite, discoverSuite, runDiscoveredSuite, sourceCandidates } from "./suites.js";
13
15
  import { duplicateTitles } from "./titles.js";
14
16
  export const reporterPath = fileURLToPath(new URL("./fail-reporter.mjs", import.meta.url));
@@ -71,7 +73,9 @@ export async function runMutants(options) {
71
73
  if (!prep.ok)
72
74
  return failReport(options.outDir, run.id, commit, prep.error, prep.baselineFailed, prep.next, prep.suiteCommand);
73
75
  // The baseline already ran. A slow suite with no line map does not start the batch.
74
- const scale = scaleStop({ kind: prep.spec.kind, baselineMs: prep.baselineMs, pending: pending.length });
76
+ // A kind that can collect a map still refuses when that map was not written.
77
+ const mapped = prep.coverage !== null && Object.keys(prep.coverage.files).length > 0 && hasLineMap(prep.spec.kind);
78
+ const scale = scaleStop({ kind: mapped ? prep.spec.kind : "unmapped", baselineMs: prep.baselineMs, pending: pending.length });
75
79
  if (scale.action === "stop") {
76
80
  const ids = pending.map((file) => path.basename(file, ".patch"));
77
81
  return failReport(options.outDir, run.id, commit, scale.summary, undefined, scale.next, prep.spec.command, prep.baselineMs, ids);
@@ -189,14 +193,28 @@ async function prepare(pkg, options, progress) {
189
193
  writeFileSync(baselineFile, `${JSON.stringify({ ok: baselineFailed.length === 0, tests: baseline.report.tests, failed: baselineFailed, command: spec.command, durationMs: baselineMs }, null, 2)}\n`);
190
194
  progress(`suite report tests=${baseline.report.tests}`);
191
195
  progress(baselineFailed.length > 0 ? `baseline failing ${baselineFailed.length}` : "baseline pass");
196
+ if (baselineFailed.length > 0) {
197
+ return { ok: false, error: "baseline already failing", baselineFailed, suiteCommand: spec.command };
198
+ }
199
+ if (collect && !existsSync(mapPath) && (spec.kind === "go" || spec.kind === "maven")) {
200
+ progress(spec.kind === "go" ? "go line map" : "java line map");
201
+ const written = spec.kind === "go"
202
+ ? writeGoCoverage(pkg, baseline.report.names, mapPath)
203
+ : writeJavaCoverage(pkg, baseline.report.names, mapPath);
204
+ if (!written.ok)
205
+ writeFileSync(path.join(options.outDir, "coverage-error.txt"), `${written.detail}\n`);
206
+ }
192
207
  }
193
208
  else {
194
209
  const stored = JSON.parse(readFileSync(baselineFile, "utf8"));
195
- baselineFailed = stored.ok === false ? stored.failed ?? [] : [];
196
210
  baselineMs = typeof stored.durationMs === "number" ? stored.durationMs : null;
211
+ if (stored.ok !== true) {
212
+ const failed = Array.isArray(stored.failed) ? stored.failed.filter((name) => typeof name === "string") : [];
213
+ return { ok: false, error: "baseline already failing", baselineFailed: failed, suiteCommand: spec.command };
214
+ }
197
215
  }
198
216
  const affectedMap = options.affected ? affectedTests(pkg, { testsDir: options.testsDir }) : null;
199
- return { ok: true, tests, affectedMap, baselineFailed, baselineMs, spec, coverage: readCoverageMap(mapPath) };
217
+ return { ok: true, tests, affectedMap, baselineMs, spec, coverage: readCoverageMap(mapPath) };
200
218
  }
201
219
  async function runOne(worktree, pkg, patchPath, options, commit, runId, prep) {
202
220
  const id = path.basename(patchPath, ".patch");
@@ -268,15 +286,8 @@ async function runOne(worktree, pkg, patchPath, options, commit, runId, prep) {
268
286
  return finishMutant(killed(base, direction, [main.compileToken]), pkg, commands.join(" | "), false, []);
269
287
  if (!main.report)
270
288
  return { ...base(), direction, error: "test reporter returned nothing", command: commands.join(" | ") };
271
- const now = main.report.failed.map((item) => item.name);
272
289
  const whole = main.wholeSuite || forceWhole;
273
290
  const chosen = main.selectedTests;
274
- if (prep.baselineFailed.length > 0) {
275
- if (sameNames(now, prep.baselineFailed))
276
- return finishMutant({ ...base(), direction, outcome: "survived" }, pkg, commands.join(" | "), whole, chosen);
277
- const noticed = [...new Set([...now.filter((name) => !prep.baselineFailed.includes(name)), ...prep.baselineFailed.filter((name) => !now.includes(name))])];
278
- return finishMutant(killed(base, direction, noticed.slice(0, 20)), pkg, commands.join(" | "), whole, chosen);
279
- }
280
291
  if (main.report.failed.length === 0)
281
292
  return finishMutant({ ...base(), direction, outcome: "survived" }, pkg, commands.join(" | "), whole, chosen);
282
293
  if (!options.confirm)
@@ -342,10 +353,10 @@ async function rerunGroup(pkg, options, file, failures, counts, pattern, prep, t
342
353
  const files = file === "*" ? listTestFiles(pkg, options.testsDir, prep.spec) : [testArg(pkg, file, options.testsDir)];
343
354
  const isolated = { ...options, concurrency: 1 };
344
355
  let wholeSuite = false;
345
- // Pytest confirms by node id. A regex fed to -k is a usage error, not a second look at the failure.
346
- const pytestNames = prep.spec.kind === "pytest" ? failures.map((item) => item.name) : null;
356
+ // Pytest confirms by node id. Maven confirms by class#method. A regex is not a test name for either.
357
+ const directNames = prep.spec.kind === "pytest" || prep.spec.kind === "maven" ? failures.map((item) => item.name) : null;
347
358
  for (let attempt = 0; attempt < 2; attempt++) {
348
- const rerun = await runSuite(pkg, files, isolated, pytestNames ? null : pattern, pytestNames, prep.spec, timeoutMs, false, null, null);
359
+ const rerun = await runSuite(pkg, files, isolated, directNames ? null : pattern, directNames, prep.spec, timeoutMs, false, null, null);
349
360
  if (rerun.wholeSuite)
350
361
  wholeSuite = true;
351
362
  if (rerun.command)
@@ -391,13 +402,6 @@ async function runSuite(pkg, files, options, pattern, names, spec, timeoutMs, fo
391
402
  selectedTests: result.selectedTests,
392
403
  };
393
404
  }
394
- function sameNames(left, right) {
395
- if (left.length !== right.length)
396
- return false;
397
- const a = [...left].sort();
398
- const b = [...right].sort();
399
- return a.every((name, index) => name === b[index]);
400
- }
401
405
  function isNodeTestFile(file) {
402
406
  const base = path.posix.basename(file);
403
407
  return /\.(test|spec)\.(ts|tsx|mts|js|mjs|cjs)$/.test(base) || /_test\.(ts|tsx|js|mjs)$/.test(base);
@@ -623,21 +627,21 @@ function reportOf(results, context) {
623
627
  return { id: result.id, file, line, ...(known?.op ? { op: known.op } : {}) };
624
628
  });
625
629
  const budgetHit = context.budgetHit || context.pendingLeft > 0;
626
- const summary = `${killed} killed, ${survived} survived, ${noCoverage} no coverage, ${flaky} flaky, ${timeouts} timed out, ${errors} errored, of ${results.length} finished.`;
630
+ const summary = campaignSummary({ noCoverage, survived, killed, flaky, timeouts, errors, finished: results.length });
627
631
  const notes = [];
628
- if (budgetHit)
629
- notes.push(resumeNext(context.notStarted));
630
- if (timeouts > 0)
631
- notes.push(TIMEOUT_NEXT);
632
632
  const uncovered = results.find((result) => result.outcome === "no coverage");
633
633
  if (uncovered?.next)
634
634
  notes.push(uncovered.next);
635
+ if (gapResults.length > 0)
636
+ notes.push(`First gap is ${gaps[0].id} at ${gaps[0].file}:${gaps[0].line}. A survivor is not a pass.`);
637
+ if (timeouts > 0)
638
+ notes.push(TIMEOUT_NEXT);
639
+ if (budgetHit)
640
+ notes.push(resumeNext(context.notStarted));
635
641
  if (results.some((result) => result.wholeSuite))
636
642
  notes.push(WHOLE_SUITE_NEXT);
637
- if (!budgetHit && gapResults.length > 0 && timeouts === 0)
638
- notes.push(`First gap is ${gaps[0].id} at ${gaps[0].file}:${gaps[0].line}.`);
639
643
  if (notes.length === 0)
640
- notes.push(gapResults.length > 0 ? `First gap is ${gaps[0].id} at ${gaps[0].file}:${gaps[0].line}.` : "No survivor in this batch.");
644
+ notes.push("No survivor in this batch.");
641
645
  return {
642
646
  ok: true,
643
647
  summary,
@@ -17,13 +17,20 @@ export function resumeNext(ids) {
17
17
  export function noCoverageNext(file, line) {
18
18
  return `No coverage for ${file}:${line}. No test executed that line.`;
19
19
  }
20
+ /** Every agent-facing count starts with unseen lines, then survivors. */
21
+ export function leadSummary(noCoverage, survivors, survivorWord, rest) {
22
+ return `${noCoverage} no coverage, ${survivors} ${survivorWord}, ${rest}`;
23
+ }
24
+ export function campaignSummary(counts) {
25
+ return leadSummary(counts.noCoverage, counts.survived, "survived", `${counts.killed} killed, ${counts.flaky} flaky, ${counts.timeouts} timed out, ${counts.errors} errored, of ${counts.finished} finished.`);
26
+ }
20
27
  export function timeoutLimitMs(baselineMs, multiple, floorMs) {
21
28
  const floor = Math.max(1, floorMs);
22
29
  const scaled = Math.ceil(Math.max(0, baselineMs) * Math.max(1, multiple));
23
30
  return Math.max(floor, scaled);
24
31
  }
25
32
  /** Kinds that collect a per-line map on the baseline. Other kinds pay the whole suite. */
26
- const LINE_MAP_KINDS = new Set(["node", "pytest", "c"]);
33
+ const LINE_MAP_KINDS = new Set(["node", "pytest", "c", "go", "maven"]);
27
34
  /** A baseline this long, with no line map, is too expensive to repeat for a large batch. */
28
35
  export const SLOW_BASELINE_MS = 5_000;
29
36
  /** Pending mutants above this count, on a slow unmapped suite, are not started. */
@@ -274,6 +274,9 @@ async function launch(pkg, spec, files, pattern, names, timeoutMs, env, hooks) {
274
274
  const tests = selection.map(surefireTest).filter((name) => name.length > 0);
275
275
  const args = ["-B", "test", ...(tests.length > 0 ? [`-Dtest=${tests.join(",")}`] : [])];
276
276
  const whole = wantsNames && tests.length === 0;
277
+ // Surefire leaves the previous run's XML in place. A confirm that did not
278
+ // re-execute a class would otherwise read that class as failed again.
279
+ rmSync(path.join(pkg, "target", "surefire-reports"), { recursive: true, force: true });
277
280
  return tag(spawnCollected("mvn", args, pkg, javaEnv(env), timeoutMs), shown("mvn", args), whole, selection);
278
281
  }
279
282
  if (spec.kind === "dotnet") {
@@ -284,9 +287,11 @@ async function launch(pkg, spec, files, pattern, names, timeoutMs, env, hooks) {
284
287
  return tag(runDotnetExe(pkg, spec, files, timeoutMs, env), spec.command, wantsNames, selection);
285
288
  if (spec.kind === "mocha")
286
289
  return tag(runMocha(pkg, files, timeoutMs, env), spec.command, wantsNames, selection);
287
- // macOS `script` allocates a tty. tini's suite calls `docker run -it`, which exits immediately without one.
288
- if (spec.kind === "script")
289
- return tag(runScript(pkg, timeoutMs, env), spec.command, wantsNames, selection);
290
+ // A tty is only for a suite that runs `docker run -it` (tini). Everything else is bash.
291
+ if (spec.kind === "script") {
292
+ const launched = scriptInvocation(pkg);
293
+ return tag(runScript(pkg, timeoutMs, env, launched), launched.command, wantsNames, selection);
294
+ }
290
295
  if (spec.kind === "c" || spec.kind === "cpp")
291
296
  return tag(compileAndRun(pkg, files, spec.kind === "cpp", timeoutMs, env), spec.command, wantsNames, selection);
292
297
  if (spec.kind === "java")
@@ -549,7 +554,72 @@ function clearBytecode(pkg) {
549
554
  }
550
555
  }
551
556
  }
552
- function runScript(pkg, timeoutMs, env) {
557
+ let gnuScriptCache = null;
558
+ function suiteNeedsTty(pkg) {
559
+ const files = new Set();
560
+ const rootScript = path.join(pkg, "run_tests.sh");
561
+ if (existsSync(rootScript))
562
+ files.add(rootScript);
563
+ try {
564
+ for (const name of readdirSync(pkg)) {
565
+ if (name.endsWith(".sh"))
566
+ files.add(path.join(pkg, name));
567
+ }
568
+ }
569
+ catch {
570
+ // A missing directory is not a tty.
571
+ }
572
+ let entry = "";
573
+ try {
574
+ entry = readFileSync(rootScript, "utf8");
575
+ }
576
+ catch {
577
+ entry = "";
578
+ }
579
+ for (const match of entry.matchAll(/(?:^|[\s"'`/])([\w.-]+\.sh)\b/g)) {
580
+ const full = path.resolve(pkg, match[1]);
581
+ if (full.startsWith(pkg + path.sep) && existsSync(full))
582
+ files.add(full);
583
+ }
584
+ for (const file of files) {
585
+ let body = "";
586
+ try {
587
+ body = readFileSync(file, "utf8");
588
+ }
589
+ catch {
590
+ continue;
591
+ }
592
+ if (dockerWantsTty(body))
593
+ return true;
594
+ }
595
+ return false;
596
+ }
597
+ function dockerWantsTty(text) {
598
+ return text.split("\n").some((line) => {
599
+ if (!/\bdocker\s+run\b/.test(line))
600
+ return false;
601
+ return /(?:^|\s)--tty(?:\s|=|$)/.test(line) || /(?:^|\s)-[A-Za-z]*t[A-Za-z]*(?:\s|$)/.test(line);
602
+ });
603
+ }
604
+ function gnuScript() {
605
+ if (gnuScriptCache !== null)
606
+ return gnuScriptCache;
607
+ const probe = spawnSync("script", ["--version"], { encoding: "utf8", timeout: 2000 });
608
+ const text = `${probe.stdout ?? ""}\n${probe.stderr ?? ""}`;
609
+ gnuScriptCache = probe.status === 0 && /util-linux|\bGNU\b/.test(text);
610
+ return gnuScriptCache;
611
+ }
612
+ function scriptInvocation(pkg) {
613
+ if (!suiteNeedsTty(pkg))
614
+ return { bin: "bash", args: ["run_tests.sh"], command: "bash run_tests.sh" };
615
+ if (gnuScript()) {
616
+ const args = ["-q", "-c", "bash run_tests.sh", "/dev/null"];
617
+ return { bin: "script", args, command: shown("script", args) };
618
+ }
619
+ const args = ["-q", "/dev/null", "bash", "run_tests.sh"];
620
+ return { bin: "script", args, command: shown("script", args) };
621
+ }
622
+ function runScript(pkg, timeoutMs, env, launched) {
553
623
  const next = { ...env };
554
624
  const shim = writeDockerShim(pkg);
555
625
  if (shim)
@@ -560,7 +630,7 @@ function runScript(pkg, timeoutMs, env) {
560
630
  if (!next.ARCH_NATIVE && existsSync(inner) && existsSync(build) && readFileSync(build, "utf8").includes("ARCH_NATIVE")) {
561
631
  next.ARCH_NATIVE = "1";
562
632
  }
563
- return spawnCollected("script", ["-q", "/dev/null", "bash", "run_tests.sh"], pkg, next, timeoutMs);
633
+ return spawnCollected(launched.bin, launched.args, pkg, next, timeoutMs);
564
634
  }
565
635
  function writeDockerShim(pkg) {
566
636
  const found = spawnSync("sh", ["-c", "command -v docker"], { encoding: "utf8" });
@@ -1132,7 +1202,9 @@ function parseDotnet(text, code) {
1132
1202
  const names = [];
1133
1203
  const failed = [];
1134
1204
  for (const line of text.split("\n")) {
1135
- const match = line.match(/^\s*(Passed|Failed)\s+(\S+)/);
1205
+ // A result line carries a duration: "Passed StaysClosed [4 ms]".
1206
+ // "Failed to load ..." is a diagnostic, not a test, and must not redden the baseline.
1207
+ const match = line.match(/^\s*(Passed|Failed)\s+(.+?)\s+\[(?:<\s*)?\d+(?:\.\d+)?\s*(?:ms|s)\]\s*$/);
1136
1208
  if (!match)
1137
1209
  continue;
1138
1210
  names.push(match[2]);
@@ -3,11 +3,11 @@ import path from "node:path";
3
3
  import { SCHEMA_VERSION } from "../contract.js";
4
4
  import { advisePrune } from "../matrix/report.js";
5
5
  import { isBuildName } from "./kill-label.js";
6
- import { WHOLE_PROGRAM_TEST } from "./suite-decision.js";
6
+ import { leadSummary, TIMEOUT_NEXT, WHOLE_PROGRAM_TEST } from "./suite-decision.js";
7
7
  const NOT_A_TEST = new Set(["suite", "pytest", "unittest", "command", WHOLE_PROGRAM_TEST]);
8
8
  /**
9
9
  * Read a finished mutate run. Name tests that killed a mutant, tests that killed
10
- * nothing, and one gap per survivor or uncovered line. This never deletes a file.
10
+ * nothing, and one gap per survivor. An unseen line is not a gap. This never deletes a file.
11
11
  */
12
12
  export function tallyRun(outDir) {
13
13
  const resultsDir = path.join(outDir, "results");
@@ -75,18 +75,30 @@ export function tallyRun(outDir) {
75
75
  .map(([name]) => name)
76
76
  .sort();
77
77
  const drop = pruning.advice;
78
+ const noCoverage = results.filter((result) => result.outcome === "no coverage").length;
79
+ const survived = results.filter((result) => result.outcome === "survived").length;
80
+ const timeouts = results.filter((result) => result.outcome === "timeout").length;
78
81
  const gaps = results
79
- .filter((result) => result.outcome === "survived" || result.outcome === "no coverage")
82
+ .filter((result) => result.outcome === "survived")
80
83
  .map((result) => ({
81
84
  id: result.id,
82
85
  file: result.files?.[0]?.file ?? "",
83
86
  line: result.files?.[0]?.line ?? 0,
84
- outcome: result.outcome,
87
+ outcome: "survived",
85
88
  }))
86
89
  .sort((a, b) => a.file.localeCompare(b.file) || a.line - b.line || a.id.localeCompare(b.id));
87
90
  const shown = gaps.slice(0, 10);
88
- const summary = `${keep.length} tests to keep, ${drop.length} tests killed nothing, ${gaps.length} gaps. No test file was deleted.`;
89
- const next = gaps.length === 0 ? "No gap in this run." : `First gap is ${gaps[0].id} at ${gaps[0].file}:${gaps[0].line}.`;
91
+ const summary = leadSummary(noCoverage, survived, "survived", `${keep.length} tests to keep, ${drop.length} tests killed nothing, ${gaps.length} gaps. No test file was deleted.`);
92
+ const notes = [];
93
+ if (noCoverage > 0)
94
+ notes.push(`${noCoverage} no coverage. That line was not seen. It is not a pass and not a gap.`);
95
+ if (gaps.length > 0)
96
+ notes.push(`First gap is ${gaps[0].id} at ${gaps[0].file}:${gaps[0].line}. A survivor is not a pass.`);
97
+ if (timeouts > 0)
98
+ notes.push(TIMEOUT_NEXT);
99
+ if (notes.length === 0)
100
+ notes.push("No survivor in this run. No test file was deleted.");
101
+ const next = notes.join(" ");
90
102
  const complete = {
91
103
  schemaVersion: SCHEMA_VERSION,
92
104
  ok: true,
@@ -5,6 +5,7 @@ import { directImporters } from "../mutate/affected.js";
5
5
  import { filesInDiff } from "../mutate/patch.js";
6
6
  import { killLabel } from "../mutate/kill-label.js";
7
7
  import { runMutants } from "../mutate/run.js";
8
+ import { campaignSummary } from "../mutate/suite-decision.js";
8
9
  /** The gap is done only when this mutant is killed and the baseline suite is green. */
9
10
  export async function checkKill(id, options) {
10
11
  const importers = importersOf(id, options);
@@ -28,6 +29,21 @@ export async function checkKill(id, options) {
28
29
  const baseline = JSON.parse(readFileSync(baselineFile, "utf8"));
29
30
  suiteGreen = baseline.ok === true;
30
31
  }
32
+ // A red baseline stops inside the run. Do not read a result that names the broken test.
33
+ if (!report.ok) {
34
+ return {
35
+ schemaVersion: SCHEMA_VERSION,
36
+ ok: false,
37
+ command: "check-kill",
38
+ summary: `${id} is not done. ${report.summary}`,
39
+ next: "The task is not done.",
40
+ nextCall: null,
41
+ done: false,
42
+ outcome: null,
43
+ suiteGreen,
44
+ killedBy: [],
45
+ };
46
+ }
31
47
  const resultFile = path.join(options.outDir, "results", `${id}.json`);
32
48
  let outcome = null;
33
49
  let killedBy = [];
@@ -38,11 +54,12 @@ export async function checkKill(id, options) {
38
54
  }
39
55
  const done = suiteGreen && outcome === "killed";
40
56
  const label = done ? killLabel(killedBy) : null;
57
+ const head = checkHead(report);
41
58
  return {
42
59
  schemaVersion: SCHEMA_VERSION,
43
60
  ok: done,
44
61
  command: "check-kill",
45
- summary: done ? `${id} is dead and the suite is green.` : `${id} is not done. ${report.summary}`,
62
+ summary: done ? `${head} ${id} is dead and the suite is green.` : `${head} ${id} is not done.`,
46
63
  next: label?.next ?? (done ? "The gap-fixing task is done." : "The task is not done."),
47
64
  nextCall: null,
48
65
  done,
@@ -60,6 +77,18 @@ function importersOf(id, options) {
60
77
  const rels = filesInDiff(readFileSync(patch, "utf8")).map((item) => packageRelative(options.repoDir, options.packageDir, item.file));
61
78
  return directImporters(options.packageDir, rels, options.testsDir);
62
79
  }
80
+ function checkHead(report) {
81
+ const finished = report.killed + report.survived + report.flaky + report.timeouts + report.errors + report.noCoverage;
82
+ return campaignSummary({
83
+ noCoverage: report.noCoverage,
84
+ survived: report.survived,
85
+ killed: report.killed,
86
+ flaky: report.flaky,
87
+ timeouts: report.timeouts,
88
+ errors: report.errors,
89
+ finished,
90
+ });
91
+ }
63
92
  function packageRelative(repoDir, packageDir, file) {
64
93
  const norm = file.split(path.sep).join("/");
65
94
  let repo = repoDir;
@@ -7,6 +7,7 @@ import { findInSource } from "../mutate/find.js";
7
7
  import { mutantId } from "../mutate/ids.js";
8
8
  import { forwardDiff, git } from "../mutate/patch.js";
9
9
  import { runMutants } from "../mutate/run.js";
10
+ import { leadSummary } from "../mutate/suite-decision.js";
10
11
  import { discoverSuite } from "../mutate/suites.js";
11
12
  /** Diff-scoped mutants, the tests that can see the edit, and golden contract rows that changed. */
12
13
  export async function verifyChange(options) {
@@ -112,7 +113,7 @@ export async function verifyChange(options) {
112
113
  function bodyOf(outDir, ok, affected, ran, notRan, caught, missed, other, goldenContract, notRun, suiteCommand) {
113
114
  const uncovered = other.filter((item) => item.outcome === "no coverage");
114
115
  const summary = ok
115
- ? `${caught.length} caught, ${missed.length} missed, ${notRun.length} not run, ${uncovered.length} no coverage, ${affected.length} affected tests, ${goldenContract.length} golden contract changes.`
116
+ ? leadSummary(uncovered.length, missed.length, "missed", `${caught.length} caught, ${notRun.length} not run, ${affected.length} affected tests, ${goldenContract.length} golden contract changes.`)
116
117
  : "verify-change did not finish.";
117
118
  const first = missed[0] ?? uncovered[0] ?? notRun[0] ?? caught[0];
118
119
  return {
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "probatio",
3
- "version": "0.0.1",
3
+ "version": "0.0.2",
4
4
  "description": "Mutation runner for AI agents. Scores a test suite by the bugs it catches.",
5
5
  "type": "module",
6
6
  "bin": {
@@ -18,7 +18,7 @@
18
18
  "url": "https://github.com/IronAdamant/Probatio/issues"
19
19
  },
20
20
  "scripts": {
21
- "build": "tsc -p tsconfig.json && mkdir -p dist/mutate && cp src/mutate/fail-reporter.mjs src/mutate/node-coverage.mjs src/mutate/py-reporter.py src/mutate/py-pytest-reporter.py dist/mutate/",
21
+ "build": "tsc -p tsconfig.json && mkdir -p dist/mutate && cp src/mutate/fail-reporter.mjs src/mutate/node-coverage.mjs src/mutate/py-reporter.py src/mutate/py-pytest-reporter.py src/mutate/probatio-jacoco-run.java dist/mutate/",
22
22
  "test": "tsx --test tests/*.test.ts",
23
23
  "probatio": "tsx src/cli.ts"
24
24
  },