probatio 0.0.1 → 0.0.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +5 -5
- package/dist/cli.js +13 -12
- package/dist/mcp.js +39 -42
- package/dist/mutate/go-coverage.js +100 -0
- package/dist/mutate/java-coverage.js +188 -0
- package/dist/mutate/probatio-jacoco-run.java +81 -0
- package/dist/mutate/run.js +33 -29
- package/dist/mutate/suite-decision.js +8 -1
- package/dist/mutate/suites.js +78 -6
- package/dist/mutate/tally.js +18 -6
- package/dist/swarm/check-kill.js +30 -1
- package/dist/verify/change.js +2 -1
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -5,7 +5,7 @@ Probatio (Latin: a testing, a proof) is a free, open-source testing toolkit buil
|
|
|
5
5
|
it remembers what was fixed so agents with small context do not redo or undo work; and it finds
|
|
6
6
|
the bugs a test suite would miss instead of asking anyone to read more tests.
|
|
7
7
|
|
|
8
|
-
Status (2026-10-05): `mutate run` discovers the project suite and scores it. Python, JavaScript, and TypeScript narrow each mutant with a line map collected on the baseline. C does the same when the binary was built with LLVM coverage. A suite with no line map, a baseline of 5 seconds or more, and more than 30 mutants stops before the first mutant. `verify-change` mutates the changed lines and uses that same map. `mutate tally` reads a finished run, names tests that killed nothing, and does not delete a test. The MCP server returns the same JSON as the CLI.
|
|
8
|
+
Status (2026-10-05): `mutate run` discovers the project suite and scores it. Python, JavaScript, and TypeScript narrow each mutant with a line map collected on the baseline. C does the same when the binary was built with LLVM coverage. A suite with no line map, a baseline of 5 seconds or more, and more than 30 mutants stops before the first mutant. `verify-change` mutates the changed lines and uses that same map. `mutate tally` reads a finished run, names tests that killed nothing, and does not delete a test. The MCP server speaks one JSON object per line and returns the same JSON as the CLI.
|
|
9
9
|
|
|
10
10
|
Read in this order:
|
|
11
11
|
|
|
@@ -23,19 +23,19 @@ One JSON object on stdout. `schemaVersion`, `ok`, `summary`, `next`, `nextCall`.
|
|
|
23
23
|
|
|
24
24
|
```bash
|
|
25
25
|
npx probatio mutate generate --package . --out .probatio/generate
|
|
26
|
-
npx probatio mutate run --package . --patches .probatio/generate/mutants --out .probatio/runs
|
|
26
|
+
npx probatio mutate run --package . --patches .probatio/generate/mutants --out .probatio/runs
|
|
27
27
|
npx probatio mutate tally --out .probatio/runs
|
|
28
28
|
```
|
|
29
29
|
|
|
30
30
|
`generate` writes operator mutants (conditions, `&&`/`||`, `===`/`!==`, boundaries, booleans, a dropped `!`). The same class is written for COBOL (`.cob`, `.cbl`), Rust, C, C++, Java, Go, Python, JavaScript, and C#. Strings and comments are not mutated. A COBOL `*>` comment and a fixed-format line whose column 7 is `*` or `/` are comments. Each patch is forward: apply it to introduce the bug, and the file says so. When mutants were written, `next` states the count, the suite command if one was discovered, and whether the first run collects a line map.
|
|
31
31
|
|
|
32
|
-
`run` discovers the suite: `run_tests.sh`, Cargo, Go, Swift, Maven, dotnet, pytest, unittest, node:test, Mocha, then `make test` when COBOL tests sit under that Makefile. An unknown layout stops and asks for `--suite-command`. It does not compile one file and call that the suite. `make test` that would curl or wget a missing file stops, and nothing is fetched. A worktree that has no `node_modules` uses the main checkout's.
|
|
32
|
+
`run` discovers the suite: `run_tests.sh`, Cargo, Go, Swift, Maven, dotnet, pytest, unittest, node:test, Mocha, then `make test` when COBOL tests sit under that Makefile. An unknown layout stops and asks for `--suite-command`. It does not compile one file and call that the suite. `make test` that would curl or wget a missing file stops, and nothing is fetched. A worktree that has no `node_modules` uses the main checkout's. Omitting `--build` runs no build. Pass `--build` with a command when the suite needs one first.
|
|
33
33
|
|
|
34
|
-
Python, JavaScript, and TypeScript collect a line map on the baseline. C collects one when LLVM coverage was instrumented. A later mutant runs
|
|
34
|
+
Python, JavaScript, and TypeScript collect a line map on the baseline. C collects one when LLVM coverage was instrumented. Go and Maven/Java collect file, line, and the test names that hit that line. A later mutant runs only those tests (`go test -run`, Maven `-Dtest`). A line no test executed is `no coverage`, and the suite is not started. Rust and C# can take a test name and do not collect a line. A baseline of 5 seconds or more, with no line map and more than 30 mutants still pending, stops before the first mutant. `next` names that baseline and tells you to narrow `--src` or pass a smaller patch directory. A one-file C, C++, Java, COBOL, or assembly launcher is not a suite discovery returns.
|
|
35
35
|
|
|
36
36
|
The mutant timeout is the baseline duration times 5, and at least 20 seconds, capped by `--suite-timeout-ms`. A timeout is not a kill. `--budget-ms` stops mutant work after the baseline. The first mutant still runs, except for that no-line-map stop. A kill is the test that failed, or a compiler token when the mutant did not build. Confirm is on by default and reruns the failing names. Pytest names that `-k` cannot express are passed as node ids. A usage error, including pytest exit 4, is not a kill. `verify-change` does not confirm a second time. Add `--affected` to limit the file list to tests that can see the change. The default, once a line map exists, runs the tests on the changed line.
|
|
37
37
|
|
|
38
|
-
`tally` reads that run directory. It names tests that killed a mutant
|
|
38
|
+
`tally` reads that run directory. The summary leads with the no-coverage count, then the survivors. It names tests that killed a mutant and tests that killed nothing. A gap is a survivor. An unseen line is not a gap and not a pass. A timeout is neither a kill nor a gap. `deletedTests` is 0. It does not delete a file.
|
|
39
39
|
|
|
40
40
|
## ledger
|
|
41
41
|
|
package/dist/cli.js
CHANGED
|
@@ -143,12 +143,7 @@ async function runCommand(flags) {
|
|
|
143
143
|
if (direction !== "auto" && direction !== "forward" && direction !== "reverse") {
|
|
144
144
|
throw new Error("--direction must be auto, forward, or reverse");
|
|
145
145
|
}
|
|
146
|
-
const
|
|
147
|
-
const build = !buildValues || buildValues.length === 0
|
|
148
|
-
? ["npm", "run", "build:mcp", "--silent"]
|
|
149
|
-
: buildValues[buildValues.length - 1] === false
|
|
150
|
-
? null
|
|
151
|
-
: String(buildValues[buildValues.length - 1]).split(" ").filter(Boolean);
|
|
146
|
+
const build = buildCommand(flags);
|
|
152
147
|
const report = await runMutants({
|
|
153
148
|
packageDir,
|
|
154
149
|
repoDir,
|
|
@@ -444,12 +439,7 @@ async function checkKillCommand(flags, id) {
|
|
|
444
439
|
const patchDirs = texts(flags, "patches").map((dir) => path.resolve(dir));
|
|
445
440
|
if (patchDirs.length === 0)
|
|
446
441
|
throw new Error("--patches is required");
|
|
447
|
-
const
|
|
448
|
-
const build = !buildValues || buildValues.length === 0
|
|
449
|
-
? null
|
|
450
|
-
: buildValues[buildValues.length - 1] === false
|
|
451
|
-
? null
|
|
452
|
-
: String(buildValues[buildValues.length - 1]).split(" ").filter(Boolean);
|
|
442
|
+
const build = buildCommand(flags);
|
|
453
443
|
return checkKill(id, {
|
|
454
444
|
packageDir,
|
|
455
445
|
repoDir: path.resolve(text(flags, "repo") ?? gitRoot(packageDir)),
|
|
@@ -496,3 +486,14 @@ function finish(envelope, human) {
|
|
|
496
486
|
process.stdout.write(render(envelope, human));
|
|
497
487
|
process.exit(envelope.ok ? 0 : 1);
|
|
498
488
|
}
|
|
489
|
+
/** Omitted `--build` and `--no-build` both mean no build. `--build <cmd>` opts in. */
|
|
490
|
+
function buildCommand(flags) {
|
|
491
|
+
const values = flags.get("build");
|
|
492
|
+
if (!values || values.length === 0)
|
|
493
|
+
return null;
|
|
494
|
+
const last = values[values.length - 1];
|
|
495
|
+
if (typeof last !== "string")
|
|
496
|
+
return null;
|
|
497
|
+
const parts = last.split(" ").filter(Boolean);
|
|
498
|
+
return parts.length > 0 ? parts : null;
|
|
499
|
+
}
|
package/dist/mcp.js
CHANGED
|
@@ -1,35 +1,39 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
import { spawnSync } from "node:child_process";
|
|
3
|
-
import { existsSync } from "node:fs";
|
|
3
|
+
import { existsSync, readFileSync } from "node:fs";
|
|
4
4
|
import path from "node:path";
|
|
5
5
|
import { fileURLToPath } from "node:url";
|
|
6
6
|
const here = path.dirname(fileURLToPath(import.meta.url));
|
|
7
|
-
|
|
7
|
+
const version = readVersion();
|
|
8
|
+
const tool = {
|
|
9
|
+
name: "probatio",
|
|
10
|
+
description: "Run one probatio command. argv is the CLI words and flags.",
|
|
11
|
+
inputSchema: {
|
|
12
|
+
type: "object",
|
|
13
|
+
properties: { argv: { type: "array", items: { type: "string" } } },
|
|
14
|
+
required: ["argv"],
|
|
15
|
+
},
|
|
16
|
+
};
|
|
17
|
+
let buffer = "";
|
|
18
|
+
process.stdin.setEncoding("utf8");
|
|
8
19
|
process.stdin.on("data", (chunk) => {
|
|
9
|
-
buffer
|
|
20
|
+
buffer += chunk;
|
|
10
21
|
drain();
|
|
11
22
|
});
|
|
12
|
-
process.stdin.on("end", () =>
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
const
|
|
21
|
-
if (!
|
|
22
|
-
buffer = buffer.subarray(start);
|
|
23
|
+
process.stdin.on("end", () => {
|
|
24
|
+
drain(true);
|
|
25
|
+
process.exit(0);
|
|
26
|
+
});
|
|
27
|
+
function drain(flush = false) {
|
|
28
|
+
const lines = buffer.split("\n");
|
|
29
|
+
buffer = flush ? "" : lines.pop() ?? "";
|
|
30
|
+
for (const line of lines) {
|
|
31
|
+
const text = line.replace(/\r$/, "").trim();
|
|
32
|
+
if (!text)
|
|
23
33
|
continue;
|
|
24
|
-
}
|
|
25
|
-
const length = Number(match[1]);
|
|
26
|
-
if (buffer.length < start + length)
|
|
27
|
-
return;
|
|
28
|
-
const body = buffer.subarray(start, start + length).toString("utf8");
|
|
29
|
-
buffer = buffer.subarray(start + length);
|
|
30
34
|
let message;
|
|
31
35
|
try {
|
|
32
|
-
message = JSON.parse(
|
|
36
|
+
message = JSON.parse(text);
|
|
33
37
|
}
|
|
34
38
|
catch {
|
|
35
39
|
continue;
|
|
@@ -49,29 +53,14 @@ function handle(message) {
|
|
|
49
53
|
result: {
|
|
50
54
|
protocolVersion: "2024-11-05",
|
|
51
55
|
capabilities: { tools: {} },
|
|
52
|
-
serverInfo: { name: "probatio", version
|
|
56
|
+
serverInfo: { name: "probatio", version },
|
|
57
|
+
tools: [tool],
|
|
53
58
|
},
|
|
54
59
|
});
|
|
55
60
|
return;
|
|
56
61
|
}
|
|
57
62
|
if (message.method === "tools/list") {
|
|
58
|
-
send({
|
|
59
|
-
jsonrpc: "2.0",
|
|
60
|
-
id: message.id,
|
|
61
|
-
result: {
|
|
62
|
-
tools: [
|
|
63
|
-
{
|
|
64
|
-
name: "probatio",
|
|
65
|
-
description: "Run one probatio command. argv is the CLI words and flags.",
|
|
66
|
-
inputSchema: {
|
|
67
|
-
type: "object",
|
|
68
|
-
properties: { argv: { type: "array", items: { type: "string" } } },
|
|
69
|
-
required: ["argv"],
|
|
70
|
-
},
|
|
71
|
-
},
|
|
72
|
-
],
|
|
73
|
-
},
|
|
74
|
-
});
|
|
63
|
+
send({ jsonrpc: "2.0", id: message.id, result: { tools: [tool] } });
|
|
75
64
|
return;
|
|
76
65
|
}
|
|
77
66
|
if (message.method === "tools/call") {
|
|
@@ -112,8 +101,16 @@ function cliCommand() {
|
|
|
112
101
|
const tsx = path.resolve(here, "..", "node_modules", ".bin", "tsx");
|
|
113
102
|
return { bin: tsx, args: [path.join(here, "cli.ts")] };
|
|
114
103
|
}
|
|
104
|
+
function readVersion() {
|
|
105
|
+
try {
|
|
106
|
+
const raw = JSON.parse(readFileSync(path.resolve(here, "..", "package.json"), "utf8"));
|
|
107
|
+
return typeof raw.version === "string" && raw.version.length > 0 ? raw.version : "0.0.1";
|
|
108
|
+
}
|
|
109
|
+
catch {
|
|
110
|
+
return "0.0.1";
|
|
111
|
+
}
|
|
112
|
+
}
|
|
115
113
|
function send(message) {
|
|
116
|
-
const
|
|
117
|
-
process.stdout.write(
|
|
118
|
-
process.stdout.write(payload);
|
|
114
|
+
const line = JSON.stringify(message);
|
|
115
|
+
process.stdout.write(`${line}\n`);
|
|
119
116
|
}
|
|
@@ -0,0 +1,100 @@
|
|
|
1
|
+
import { spawnSync } from "node:child_process";
|
|
2
|
+
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
|
3
|
+
import path from "node:path";
|
|
4
|
+
/** Blocks in a Go cover profile whose count is above zero. */
|
|
5
|
+
export function parseCoverProfile(text, modulePath) {
|
|
6
|
+
const hits = [];
|
|
7
|
+
const prefix = modulePath ? `${modulePath}/` : "";
|
|
8
|
+
for (const raw of text.split("\n")) {
|
|
9
|
+
const line = raw.trim();
|
|
10
|
+
if (!line || line.startsWith("mode:"))
|
|
11
|
+
continue;
|
|
12
|
+
const match = /^(.+):(\d+)\.\d+,(\d+)\.\d+\s+\d+\s+(\d+)$/.exec(line);
|
|
13
|
+
if (!match)
|
|
14
|
+
continue;
|
|
15
|
+
if (Number(match[4]) <= 0)
|
|
16
|
+
continue;
|
|
17
|
+
let file = match[1];
|
|
18
|
+
if (prefix && file.startsWith(prefix))
|
|
19
|
+
file = file.slice(prefix.length);
|
|
20
|
+
file = file.split("\\").join("/");
|
|
21
|
+
const start = Number(match[2]);
|
|
22
|
+
const end = Number(match[3]);
|
|
23
|
+
for (let number = start; number <= end; number++)
|
|
24
|
+
hits.push({ file, line: number });
|
|
25
|
+
}
|
|
26
|
+
return hits;
|
|
27
|
+
}
|
|
28
|
+
/**
|
|
29
|
+
* One compiled test binary per package, then one profile per test name.
|
|
30
|
+
* A stock cover profile does not name the test, so each test gets its own.
|
|
31
|
+
*/
|
|
32
|
+
export function writeGoCoverage(pkg, testNames, dest) {
|
|
33
|
+
const wanted = new Set(testNames.filter((name) => name.length > 0 && !name.includes("/")));
|
|
34
|
+
if (wanted.size === 0)
|
|
35
|
+
return { ok: false, detail: "go coverage: no test names" };
|
|
36
|
+
const mod = run("go", ["list", "-m"], pkg);
|
|
37
|
+
if (mod.status !== 0)
|
|
38
|
+
return { ok: false, detail: clip(mod.stderr || mod.stdout || mod.error?.message || "go list -m failed") };
|
|
39
|
+
const modulePath = mod.stdout.trim().split("\n")[0]?.trim() ?? "";
|
|
40
|
+
const listed = run("go", ["list", "-f", "{{.ImportPath}}\t{{.Dir}}\t{{len .TestGoFiles}}\t{{len .XTestGoFiles}}", "./..."], pkg);
|
|
41
|
+
if (listed.status !== 0)
|
|
42
|
+
return { ok: false, detail: clip(listed.stderr || listed.stdout || listed.error?.message || "go list failed") };
|
|
43
|
+
const work = path.join(path.dirname(dest), "go-cover");
|
|
44
|
+
mkdirSync(work, { recursive: true });
|
|
45
|
+
const files = {};
|
|
46
|
+
const packages = listed.stdout.split("\n").map((line) => line.trim()).filter(Boolean);
|
|
47
|
+
for (const row of packages) {
|
|
48
|
+
const [importPath, dir, testFiles, xTestFiles] = row.split("\t");
|
|
49
|
+
if (!importPath || !dir)
|
|
50
|
+
continue;
|
|
51
|
+
if (Number(testFiles ?? "0") + Number(xTestFiles ?? "0") <= 0)
|
|
52
|
+
continue;
|
|
53
|
+
const bin = path.join(work, `${safe(importPath)}.test`);
|
|
54
|
+
const compiled = run("go", ["test", "-c", "-covermode=set", "-o", bin, importPath], pkg);
|
|
55
|
+
if (compiled.status !== 0 || compiled.error)
|
|
56
|
+
continue;
|
|
57
|
+
const listedTests = run(bin, ["-test.list", "."], dir);
|
|
58
|
+
if (listedTests.status !== 0)
|
|
59
|
+
continue;
|
|
60
|
+
for (const name of listedTests.stdout.split("\n").map((item) => item.trim()).filter((item) => wanted.has(item))) {
|
|
61
|
+
const profile = path.join(work, `${safe(importPath)}-${safe(name)}.out`);
|
|
62
|
+
run(bin, ["-test.run", `^${escapeRegExp(name)}$`, "-test.coverprofile", profile], dir);
|
|
63
|
+
let text = "";
|
|
64
|
+
try {
|
|
65
|
+
text = readFileSync(profile, "utf8");
|
|
66
|
+
}
|
|
67
|
+
catch {
|
|
68
|
+
continue;
|
|
69
|
+
}
|
|
70
|
+
for (const hit of parseCoverProfile(text, modulePath))
|
|
71
|
+
addHit(files, hit.file, hit.line, name);
|
|
72
|
+
}
|
|
73
|
+
}
|
|
74
|
+
if (Object.keys(files).length === 0)
|
|
75
|
+
return { ok: false, detail: "go coverage: no covered lines" };
|
|
76
|
+
writeFileSync(dest, `${JSON.stringify({ files }, null, 2)}\n`);
|
|
77
|
+
return { ok: true, detail: `${Object.keys(files).length} files` };
|
|
78
|
+
}
|
|
79
|
+
function addHit(files, file, line, test) {
|
|
80
|
+
const lines = files[file] ?? {};
|
|
81
|
+
const key = String(line);
|
|
82
|
+
const names = lines[key] ?? [];
|
|
83
|
+
if (!names.includes(test))
|
|
84
|
+
names.push(test);
|
|
85
|
+
lines[key] = names;
|
|
86
|
+
files[file] = lines;
|
|
87
|
+
}
|
|
88
|
+
function safe(value) {
|
|
89
|
+
return value.replace(/[^A-Za-z0-9._-]+/g, "_");
|
|
90
|
+
}
|
|
91
|
+
function escapeRegExp(value) {
|
|
92
|
+
return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
|
93
|
+
}
|
|
94
|
+
function run(bin, args, cwd) {
|
|
95
|
+
return spawnSync(bin, args, { cwd, encoding: "utf8", timeout: 120_000 });
|
|
96
|
+
}
|
|
97
|
+
function clip(text) {
|
|
98
|
+
const line = text.trim().split("\n").find((item) => item.trim()) ?? "go coverage failed";
|
|
99
|
+
return line.slice(0, 240);
|
|
100
|
+
}
|
|
@@ -0,0 +1,188 @@
|
|
|
1
|
+
import { spawnSync } from "node:child_process";
|
|
2
|
+
import { existsSync, mkdirSync, readdirSync, readFileSync, writeFileSync } from "node:fs";
|
|
3
|
+
import path from "node:path";
|
|
4
|
+
import { fileURLToPath } from "node:url";
|
|
5
|
+
const JACOCO = "0.8.15";
|
|
6
|
+
const ASM = "9.10.1";
|
|
7
|
+
const RUNNER = fileURLToPath(new URL("./probatio-jacoco-run.java", import.meta.url));
|
|
8
|
+
/**
|
|
9
|
+
* One JVM, one JaCoCo dump per Surefire test method.
|
|
10
|
+
* The default jacoco.exec file does not name the test.
|
|
11
|
+
*/
|
|
12
|
+
export function writeJavaCoverage(pkg, testNames, dest) {
|
|
13
|
+
const tests = testNames.map(junitMethod).filter((name) => name !== null);
|
|
14
|
+
if (tests.length === 0)
|
|
15
|
+
return { ok: false, detail: "java coverage: no class.method test names" };
|
|
16
|
+
const classDirs = findClassDirs(pkg);
|
|
17
|
+
if (classDirs.length === 0)
|
|
18
|
+
return { ok: false, detail: "java coverage: no target/classes" };
|
|
19
|
+
const work = path.join(path.dirname(dest), "java-cover");
|
|
20
|
+
mkdirSync(work, { recursive: true });
|
|
21
|
+
const jars = resolveJars(pkg, work);
|
|
22
|
+
if ("error" in jars)
|
|
23
|
+
return { ok: false, detail: jars.error };
|
|
24
|
+
const source = path.join(work, "ProbatioJacocoRun.java");
|
|
25
|
+
writeFileSync(source, readFileSync(RUNNER));
|
|
26
|
+
const compileCp = [jars.agent, jars.junit, jars.core, jars.asm, jars.asmCommons, jars.asmTree].join(path.delimiter);
|
|
27
|
+
const compiled = run(javaTool("javac"), ["-encoding", "UTF-8", "-cp", compileCp, "-d", work, source], pkg);
|
|
28
|
+
if (compiled.status !== 0)
|
|
29
|
+
return { ok: false, detail: clip(compiled.stderr || compiled.stdout || compiled.error?.message || "javac failed") };
|
|
30
|
+
const testsFile = path.join(work, "tests.txt");
|
|
31
|
+
writeFileSync(testsFile, `${tests.join("\n")}\n`);
|
|
32
|
+
const projectCp = buildClasspath(pkg, work);
|
|
33
|
+
if ("error" in projectCp)
|
|
34
|
+
return { ok: false, detail: projectCp.error };
|
|
35
|
+
// Our asm and JaCoCo must win over a project that still ships an older asm. That older reader rejects Java 27 class files.
|
|
36
|
+
const cp = [work, jars.core, jars.asm, jars.asmCommons, jars.asmTree, jars.junit, ...classDirs, ...testClassDirs(pkg), projectCp.cp].filter(Boolean).join(path.delimiter);
|
|
37
|
+
const dump = path.join(work, "hits.txt");
|
|
38
|
+
const ran = run(javaTool("java"), [`-javaagent:${jars.agent}=output=none,excludes=ProbatioJacocoRun`, "-cp", cp, "ProbatioJacocoRun", classDirs.join(path.delimiter), testsFile, dump, pkg], pkg);
|
|
39
|
+
if (ran.error && ran.error.code === "ETIMEDOUT") {
|
|
40
|
+
return { ok: false, detail: "java coverage: timed out" };
|
|
41
|
+
}
|
|
42
|
+
if (ran.status !== 0 || !existsSync(dump)) {
|
|
43
|
+
return { ok: false, detail: clip(ran.stderr || ran.stdout || ran.error?.message || "java coverage runner failed") };
|
|
44
|
+
}
|
|
45
|
+
const files = hitsToMap(readFileSync(dump, "utf8"));
|
|
46
|
+
if (Object.keys(files).length === 0)
|
|
47
|
+
return { ok: false, detail: "java coverage: no covered lines" };
|
|
48
|
+
writeFileSync(dest, `${JSON.stringify({ files }, null, 2)}\n`);
|
|
49
|
+
return { ok: true, detail: `${Object.keys(files).length} files` };
|
|
50
|
+
}
|
|
51
|
+
export function hitsToMap(text) {
|
|
52
|
+
const files = {};
|
|
53
|
+
for (const raw of text.split("\n")) {
|
|
54
|
+
const line = raw.trim();
|
|
55
|
+
if (!line.startsWith("HIT "))
|
|
56
|
+
continue;
|
|
57
|
+
const parts = line.slice(4).split(" ");
|
|
58
|
+
if (parts.length < 3)
|
|
59
|
+
continue;
|
|
60
|
+
const test = parts.pop() ?? "";
|
|
61
|
+
const number = parts.pop() ?? "";
|
|
62
|
+
const file = parts.join(" ");
|
|
63
|
+
if (!file || !/^\d+$/.test(number) || !test)
|
|
64
|
+
continue;
|
|
65
|
+
const lines = files[file] ?? {};
|
|
66
|
+
const names = lines[number] ?? [];
|
|
67
|
+
if (!names.includes(test))
|
|
68
|
+
names.push(test);
|
|
69
|
+
lines[number] = names;
|
|
70
|
+
files[file] = lines;
|
|
71
|
+
}
|
|
72
|
+
return files;
|
|
73
|
+
}
|
|
74
|
+
function junitMethod(name) {
|
|
75
|
+
const leaf = name.split("::").pop() ?? name;
|
|
76
|
+
const dot = leaf.lastIndexOf(".");
|
|
77
|
+
if (dot <= 0 || dot === leaf.length - 1)
|
|
78
|
+
return null;
|
|
79
|
+
if (leaf.startsWith("maven-") || leaf === "javac")
|
|
80
|
+
return null;
|
|
81
|
+
return `${leaf.slice(0, dot)}#${leaf.slice(dot + 1)}`;
|
|
82
|
+
}
|
|
83
|
+
function findClassDirs(root) {
|
|
84
|
+
const found = [];
|
|
85
|
+
const walk = (dir, depth) => {
|
|
86
|
+
if (depth > 4)
|
|
87
|
+
return;
|
|
88
|
+
let entries = [];
|
|
89
|
+
try {
|
|
90
|
+
entries = readdirSync(dir);
|
|
91
|
+
}
|
|
92
|
+
catch {
|
|
93
|
+
return;
|
|
94
|
+
}
|
|
95
|
+
for (const name of entries) {
|
|
96
|
+
if (name === "node_modules" || name === ".git" || name === "java-cover")
|
|
97
|
+
continue;
|
|
98
|
+
const full = path.join(dir, name);
|
|
99
|
+
if (name === "target") {
|
|
100
|
+
const classes = path.join(full, "classes");
|
|
101
|
+
if (existsSync(classes))
|
|
102
|
+
found.push(classes);
|
|
103
|
+
continue;
|
|
104
|
+
}
|
|
105
|
+
if (name.startsWith("."))
|
|
106
|
+
continue;
|
|
107
|
+
walk(full, depth + 1);
|
|
108
|
+
}
|
|
109
|
+
};
|
|
110
|
+
walk(root, 0);
|
|
111
|
+
return found;
|
|
112
|
+
}
|
|
113
|
+
function testClassDirs(root) {
|
|
114
|
+
return findClassDirs(root).map((dir) => path.join(path.dirname(dir), "test-classes")).filter((dir) => existsSync(dir));
|
|
115
|
+
}
|
|
116
|
+
function resolveJars(pkg, work) {
|
|
117
|
+
const copies = [
|
|
118
|
+
[`org.jacoco:org.jacoco.agent:${JACOCO}:jar:runtime`, "agent.jar"],
|
|
119
|
+
[`org.jacoco:org.jacoco.core:${JACOCO}`, "core.jar"],
|
|
120
|
+
[`org.ow2.asm:asm:${ASM}`, "asm.jar"],
|
|
121
|
+
[`org.ow2.asm:asm-commons:${ASM}`, "asm-commons.jar"],
|
|
122
|
+
[`org.ow2.asm:asm-tree:${ASM}`, "asm-tree.jar"],
|
|
123
|
+
["junit:junit:4.13.2", "junit.jar"],
|
|
124
|
+
];
|
|
125
|
+
const paths = {};
|
|
126
|
+
for (const [artifact, name] of copies) {
|
|
127
|
+
const dest = path.join(work, name);
|
|
128
|
+
if (!existsSync(dest)) {
|
|
129
|
+
const before = new Set(readdirSync(work));
|
|
130
|
+
const copied = run("mvn", ["-B", "-q", "org.apache.maven.plugins:maven-dependency-plugin:3.6.1:copy", `-Dartifact=${artifact}`, `-DoutputDirectory=${work}`], pkg);
|
|
131
|
+
if (copied.status !== 0)
|
|
132
|
+
return { error: clip(copied.stderr || copied.stdout || copied.error?.message || `mvn copy ${artifact} failed`) };
|
|
133
|
+
const created = readdirSync(work).find((item) => !before.has(item) && item.endsWith(".jar"));
|
|
134
|
+
if (!created)
|
|
135
|
+
return { error: `java coverage: missing ${artifact}` };
|
|
136
|
+
writeFileSync(dest, readFileSync(path.join(work, created)));
|
|
137
|
+
}
|
|
138
|
+
paths[name] = dest;
|
|
139
|
+
}
|
|
140
|
+
return {
|
|
141
|
+
agent: paths["agent.jar"],
|
|
142
|
+
core: paths["core.jar"],
|
|
143
|
+
asm: paths["asm.jar"],
|
|
144
|
+
asmCommons: paths["asm-commons.jar"],
|
|
145
|
+
asmTree: paths["asm-tree.jar"],
|
|
146
|
+
junit: paths["junit.jar"],
|
|
147
|
+
};
|
|
148
|
+
}
|
|
149
|
+
function buildClasspath(pkg, work) {
|
|
150
|
+
const file = path.join(work, "classpath.txt");
|
|
151
|
+
const built = run("mvn", ["-B", "-q", "dependency:build-classpath", "-DincludeScope=test", `-Dmdep.outputFile=${file}`], pkg);
|
|
152
|
+
if (built.status !== 0)
|
|
153
|
+
return { error: clip(built.stderr || built.stdout || built.error?.message || "mvn classpath failed") };
|
|
154
|
+
try {
|
|
155
|
+
return { cp: readFileSync(file, "utf8").trim() };
|
|
156
|
+
}
|
|
157
|
+
catch {
|
|
158
|
+
return { error: "java coverage: classpath file missing" };
|
|
159
|
+
}
|
|
160
|
+
}
|
|
161
|
+
function javaTool(name) {
|
|
162
|
+
const home = process.env.JAVA_HOME;
|
|
163
|
+
if (home) {
|
|
164
|
+
const candidate = path.join(home, "bin", name);
|
|
165
|
+
if (existsSync(candidate) && toolWorks(candidate))
|
|
166
|
+
return candidate;
|
|
167
|
+
}
|
|
168
|
+
if (toolWorks(name))
|
|
169
|
+
return name;
|
|
170
|
+
for (const dir of ["/opt/homebrew/opt/openjdk/bin", "/usr/local/opt/openjdk/bin"]) {
|
|
171
|
+
const candidate = path.join(dir, name);
|
|
172
|
+
if (existsSync(candidate) && toolWorks(candidate))
|
|
173
|
+
return candidate;
|
|
174
|
+
}
|
|
175
|
+
return name;
|
|
176
|
+
}
|
|
177
|
+
function toolWorks(bin) {
|
|
178
|
+
const probed = spawnSync(bin, ["-version"], { encoding: "utf8" });
|
|
179
|
+
const text = `${probed.stdout ?? ""}${probed.stderr ?? ""}`;
|
|
180
|
+
return probed.status === 0 && !/Unable to locate a Java Runtime/i.test(text);
|
|
181
|
+
}
|
|
182
|
+
function run(bin, args, cwd) {
|
|
183
|
+
return spawnSync(bin, args, { cwd, encoding: "utf8", timeout: 180_000 });
|
|
184
|
+
}
|
|
185
|
+
function clip(text) {
|
|
186
|
+
const line = text.trim().split("\n").find((item) => item.trim()) ?? "java coverage failed";
|
|
187
|
+
return line.slice(0, 240);
|
|
188
|
+
}
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
import java.io.ByteArrayInputStream;
|
|
2
|
+
import java.io.File;
|
|
3
|
+
import java.nio.file.Files;
|
|
4
|
+
import java.util.List;
|
|
5
|
+
import org.jacoco.agent.rt.IAgent;
|
|
6
|
+
import org.jacoco.agent.rt.RT;
|
|
7
|
+
import org.jacoco.core.analysis.Analyzer;
|
|
8
|
+
import org.jacoco.core.analysis.CoverageBuilder;
|
|
9
|
+
import org.jacoco.core.analysis.IClassCoverage;
|
|
10
|
+
import org.jacoco.core.analysis.ILine;
|
|
11
|
+
import org.jacoco.core.data.ExecutionDataReader;
|
|
12
|
+
import org.jacoco.core.data.ExecutionDataStore;
|
|
13
|
+
import org.junit.runner.JUnitCore;
|
|
14
|
+
import org.junit.runner.Request;
|
|
15
|
+
|
|
16
|
+
/** One JaCoCo dump per JUnit 4 method. Prints "HIT <file> <line>" lines. */
|
|
17
|
+
public class ProbatioJacocoRun {
|
|
18
|
+
public static void main(String[] args) throws Exception {
|
|
19
|
+
if (args.length != 4) {
|
|
20
|
+
System.err.println("usage: ProbatioJacocoRun classDirs testsFile outFile packageRoot");
|
|
21
|
+
System.exit(2);
|
|
22
|
+
}
|
|
23
|
+
String[] classDirs = args[0].split(File.pathSeparator);
|
|
24
|
+
List<String> tests = Files.readAllLines(new File(args[1]).toPath());
|
|
25
|
+
File out = new File(args[2]);
|
|
26
|
+
File root = new File(args[3]);
|
|
27
|
+
IAgent agent = RT.getAgent();
|
|
28
|
+
StringBuilder text = new StringBuilder();
|
|
29
|
+
for (String spec : tests) {
|
|
30
|
+
int hash = spec.lastIndexOf('#');
|
|
31
|
+
if (hash <= 0) continue;
|
|
32
|
+
String className = spec.substring(0, hash);
|
|
33
|
+
String method = spec.substring(hash + 1);
|
|
34
|
+
agent.getExecutionData(true);
|
|
35
|
+
try {
|
|
36
|
+
new JUnitCore().run(Request.method(Class.forName(className), method));
|
|
37
|
+
} catch (Throwable ignored) {
|
|
38
|
+
continue;
|
|
39
|
+
}
|
|
40
|
+
byte[] data = agent.getExecutionData(false);
|
|
41
|
+
ExecutionDataStore store = new ExecutionDataStore();
|
|
42
|
+
ExecutionDataReader reader = new ExecutionDataReader(new ByteArrayInputStream(data));
|
|
43
|
+
reader.setExecutionDataVisitor(store);
|
|
44
|
+
reader.setSessionInfoVisitor(info -> { });
|
|
45
|
+
reader.read();
|
|
46
|
+
String testName = className + "." + method;
|
|
47
|
+
text.append("TEST ").append(testName).append('\n');
|
|
48
|
+
for (String dir : classDirs) {
|
|
49
|
+
if (dir.isEmpty()) continue;
|
|
50
|
+
File classes = new File(dir);
|
|
51
|
+
if (!classes.isDirectory()) continue;
|
|
52
|
+
CoverageBuilder builder = new CoverageBuilder();
|
|
53
|
+
Analyzer analyzer = new Analyzer(store, builder);
|
|
54
|
+
analyzer.analyzeAll(classes);
|
|
55
|
+
for (IClassCoverage cc : builder.getClasses()) {
|
|
56
|
+
String source = cc.getSourceFileName();
|
|
57
|
+
if (source == null) continue;
|
|
58
|
+
File file = sourceFile(classes, cc.getPackageName(), source);
|
|
59
|
+
if (file == null || !file.isFile()) continue;
|
|
60
|
+
String rel = root.toPath().relativize(file.toPath()).toString().replace('\\', '/');
|
|
61
|
+
int first = cc.getFirstLine();
|
|
62
|
+
int last = cc.getLastLine();
|
|
63
|
+
for (int line = first; line <= last; line++) {
|
|
64
|
+
ILine cov = cc.getLine(line);
|
|
65
|
+
if (cov.getInstructionCounter().getCoveredCount() <= 0) continue;
|
|
66
|
+
text.append("HIT ").append(rel).append(' ').append(line).append(' ').append(testName).append('\n');
|
|
67
|
+
}
|
|
68
|
+
}
|
|
69
|
+
}
|
|
70
|
+
}
|
|
71
|
+
Files.writeString(out.toPath(), text.toString());
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
static File sourceFile(File classes, String pkg, String source) {
|
|
75
|
+
File target = classes.getParentFile();
|
|
76
|
+
File module = target == null ? null : target.getParentFile();
|
|
77
|
+
if (module == null) return null;
|
|
78
|
+
String rel = (pkg == null || pkg.isEmpty() ? "" : pkg + "/") + source;
|
|
79
|
+
return new File(module, "src/main/java/" + rel);
|
|
80
|
+
}
|
|
81
|
+
}
|
package/dist/mutate/run.js
CHANGED
|
@@ -6,9 +6,11 @@ import path from "node:path";
|
|
|
6
6
|
import { fileURLToPath } from "node:url";
|
|
7
7
|
import { affectedTests } from "./affected.js";
|
|
8
8
|
import { binaryFromCommand, readCoverageMap, selectionFor, writeLlvmCoverage } from "./coverage-map.js";
|
|
9
|
+
import { writeGoCoverage } from "./go-coverage.js";
|
|
10
|
+
import { writeJavaCoverage } from "./java-coverage.js";
|
|
9
11
|
import { applyPatch, changedLines, chooseDirection, filesInDiff, git, restoreTree } from "./patch.js";
|
|
10
12
|
import { killLabel } from "./kill-label.js";
|
|
11
|
-
import { TIMEOUT_FLOOR_MS, TIMEOUT_MULTIPLE, TIMEOUT_NEXT, WHOLE_PROGRAM_TEST, WHOLE_SUITE_NEXT, decideSuite, noCoverageNext, resumeNext, scaleStop, timeoutLimitMs, } from "./suite-decision.js";
|
|
13
|
+
import { TIMEOUT_FLOOR_MS, TIMEOUT_MULTIPLE, TIMEOUT_NEXT, WHOLE_PROGRAM_TEST, WHOLE_SUITE_NEXT, campaignSummary, decideSuite, hasLineMap, noCoverageNext, resumeNext, scaleStop, timeoutLimitMs, } from "./suite-decision.js";
|
|
12
14
|
import { commandSuite, discoverSuite, runDiscoveredSuite, sourceCandidates } from "./suites.js";
|
|
13
15
|
import { duplicateTitles } from "./titles.js";
|
|
14
16
|
export const reporterPath = fileURLToPath(new URL("./fail-reporter.mjs", import.meta.url));
|
|
@@ -71,7 +73,9 @@ export async function runMutants(options) {
|
|
|
71
73
|
if (!prep.ok)
|
|
72
74
|
return failReport(options.outDir, run.id, commit, prep.error, prep.baselineFailed, prep.next, prep.suiteCommand);
|
|
73
75
|
// The baseline already ran. A slow suite with no line map does not start the batch.
|
|
74
|
-
|
|
76
|
+
// A kind that can collect a map still refuses when that map was not written.
|
|
77
|
+
const mapped = prep.coverage !== null && Object.keys(prep.coverage.files).length > 0 && hasLineMap(prep.spec.kind);
|
|
78
|
+
const scale = scaleStop({ kind: mapped ? prep.spec.kind : "unmapped", baselineMs: prep.baselineMs, pending: pending.length });
|
|
75
79
|
if (scale.action === "stop") {
|
|
76
80
|
const ids = pending.map((file) => path.basename(file, ".patch"));
|
|
77
81
|
return failReport(options.outDir, run.id, commit, scale.summary, undefined, scale.next, prep.spec.command, prep.baselineMs, ids);
|
|
@@ -189,14 +193,28 @@ async function prepare(pkg, options, progress) {
|
|
|
189
193
|
writeFileSync(baselineFile, `${JSON.stringify({ ok: baselineFailed.length === 0, tests: baseline.report.tests, failed: baselineFailed, command: spec.command, durationMs: baselineMs }, null, 2)}\n`);
|
|
190
194
|
progress(`suite report tests=${baseline.report.tests}`);
|
|
191
195
|
progress(baselineFailed.length > 0 ? `baseline failing ${baselineFailed.length}` : "baseline pass");
|
|
196
|
+
if (baselineFailed.length > 0) {
|
|
197
|
+
return { ok: false, error: "baseline already failing", baselineFailed, suiteCommand: spec.command };
|
|
198
|
+
}
|
|
199
|
+
if (collect && !existsSync(mapPath) && (spec.kind === "go" || spec.kind === "maven")) {
|
|
200
|
+
progress(spec.kind === "go" ? "go line map" : "java line map");
|
|
201
|
+
const written = spec.kind === "go"
|
|
202
|
+
? writeGoCoverage(pkg, baseline.report.names, mapPath)
|
|
203
|
+
: writeJavaCoverage(pkg, baseline.report.names, mapPath);
|
|
204
|
+
if (!written.ok)
|
|
205
|
+
writeFileSync(path.join(options.outDir, "coverage-error.txt"), `${written.detail}\n`);
|
|
206
|
+
}
|
|
192
207
|
}
|
|
193
208
|
else {
|
|
194
209
|
const stored = JSON.parse(readFileSync(baselineFile, "utf8"));
|
|
195
|
-
baselineFailed = stored.ok === false ? stored.failed ?? [] : [];
|
|
196
210
|
baselineMs = typeof stored.durationMs === "number" ? stored.durationMs : null;
|
|
211
|
+
if (stored.ok !== true) {
|
|
212
|
+
const failed = Array.isArray(stored.failed) ? stored.failed.filter((name) => typeof name === "string") : [];
|
|
213
|
+
return { ok: false, error: "baseline already failing", baselineFailed: failed, suiteCommand: spec.command };
|
|
214
|
+
}
|
|
197
215
|
}
|
|
198
216
|
const affectedMap = options.affected ? affectedTests(pkg, { testsDir: options.testsDir }) : null;
|
|
199
|
-
return { ok: true, tests, affectedMap,
|
|
217
|
+
return { ok: true, tests, affectedMap, baselineMs, spec, coverage: readCoverageMap(mapPath) };
|
|
200
218
|
}
|
|
201
219
|
async function runOne(worktree, pkg, patchPath, options, commit, runId, prep) {
|
|
202
220
|
const id = path.basename(patchPath, ".patch");
|
|
@@ -268,15 +286,8 @@ async function runOne(worktree, pkg, patchPath, options, commit, runId, prep) {
|
|
|
268
286
|
return finishMutant(killed(base, direction, [main.compileToken]), pkg, commands.join(" | "), false, []);
|
|
269
287
|
if (!main.report)
|
|
270
288
|
return { ...base(), direction, error: "test reporter returned nothing", command: commands.join(" | ") };
|
|
271
|
-
const now = main.report.failed.map((item) => item.name);
|
|
272
289
|
const whole = main.wholeSuite || forceWhole;
|
|
273
290
|
const chosen = main.selectedTests;
|
|
274
|
-
if (prep.baselineFailed.length > 0) {
|
|
275
|
-
if (sameNames(now, prep.baselineFailed))
|
|
276
|
-
return finishMutant({ ...base(), direction, outcome: "survived" }, pkg, commands.join(" | "), whole, chosen);
|
|
277
|
-
const noticed = [...new Set([...now.filter((name) => !prep.baselineFailed.includes(name)), ...prep.baselineFailed.filter((name) => !now.includes(name))])];
|
|
278
|
-
return finishMutant(killed(base, direction, noticed.slice(0, 20)), pkg, commands.join(" | "), whole, chosen);
|
|
279
|
-
}
|
|
280
291
|
if (main.report.failed.length === 0)
|
|
281
292
|
return finishMutant({ ...base(), direction, outcome: "survived" }, pkg, commands.join(" | "), whole, chosen);
|
|
282
293
|
if (!options.confirm)
|
|
@@ -342,10 +353,10 @@ async function rerunGroup(pkg, options, file, failures, counts, pattern, prep, t
|
|
|
342
353
|
const files = file === "*" ? listTestFiles(pkg, options.testsDir, prep.spec) : [testArg(pkg, file, options.testsDir)];
|
|
343
354
|
const isolated = { ...options, concurrency: 1 };
|
|
344
355
|
let wholeSuite = false;
|
|
345
|
-
// Pytest confirms by node id.
|
|
346
|
-
const
|
|
356
|
+
// Pytest confirms by node id. Maven confirms by class#method. A regex is not a test name for either.
|
|
357
|
+
const directNames = prep.spec.kind === "pytest" || prep.spec.kind === "maven" ? failures.map((item) => item.name) : null;
|
|
347
358
|
for (let attempt = 0; attempt < 2; attempt++) {
|
|
348
|
-
const rerun = await runSuite(pkg, files, isolated,
|
|
359
|
+
const rerun = await runSuite(pkg, files, isolated, directNames ? null : pattern, directNames, prep.spec, timeoutMs, false, null, null);
|
|
349
360
|
if (rerun.wholeSuite)
|
|
350
361
|
wholeSuite = true;
|
|
351
362
|
if (rerun.command)
|
|
@@ -391,13 +402,6 @@ async function runSuite(pkg, files, options, pattern, names, spec, timeoutMs, fo
|
|
|
391
402
|
selectedTests: result.selectedTests,
|
|
392
403
|
};
|
|
393
404
|
}
|
|
394
|
-
function sameNames(left, right) {
|
|
395
|
-
if (left.length !== right.length)
|
|
396
|
-
return false;
|
|
397
|
-
const a = [...left].sort();
|
|
398
|
-
const b = [...right].sort();
|
|
399
|
-
return a.every((name, index) => name === b[index]);
|
|
400
|
-
}
|
|
401
405
|
function isNodeTestFile(file) {
|
|
402
406
|
const base = path.posix.basename(file);
|
|
403
407
|
return /\.(test|spec)\.(ts|tsx|mts|js|mjs|cjs)$/.test(base) || /_test\.(ts|tsx|js|mjs)$/.test(base);
|
|
@@ -623,21 +627,21 @@ function reportOf(results, context) {
|
|
|
623
627
|
return { id: result.id, file, line, ...(known?.op ? { op: known.op } : {}) };
|
|
624
628
|
});
|
|
625
629
|
const budgetHit = context.budgetHit || context.pendingLeft > 0;
|
|
626
|
-
const summary =
|
|
630
|
+
const summary = campaignSummary({ noCoverage, survived, killed, flaky, timeouts, errors, finished: results.length });
|
|
627
631
|
const notes = [];
|
|
628
|
-
if (budgetHit)
|
|
629
|
-
notes.push(resumeNext(context.notStarted));
|
|
630
|
-
if (timeouts > 0)
|
|
631
|
-
notes.push(TIMEOUT_NEXT);
|
|
632
632
|
const uncovered = results.find((result) => result.outcome === "no coverage");
|
|
633
633
|
if (uncovered?.next)
|
|
634
634
|
notes.push(uncovered.next);
|
|
635
|
+
if (gapResults.length > 0)
|
|
636
|
+
notes.push(`First gap is ${gaps[0].id} at ${gaps[0].file}:${gaps[0].line}. A survivor is not a pass.`);
|
|
637
|
+
if (timeouts > 0)
|
|
638
|
+
notes.push(TIMEOUT_NEXT);
|
|
639
|
+
if (budgetHit)
|
|
640
|
+
notes.push(resumeNext(context.notStarted));
|
|
635
641
|
if (results.some((result) => result.wholeSuite))
|
|
636
642
|
notes.push(WHOLE_SUITE_NEXT);
|
|
637
|
-
if (!budgetHit && gapResults.length > 0 && timeouts === 0)
|
|
638
|
-
notes.push(`First gap is ${gaps[0].id} at ${gaps[0].file}:${gaps[0].line}.`);
|
|
639
643
|
if (notes.length === 0)
|
|
640
|
-
notes.push(
|
|
644
|
+
notes.push("No survivor in this batch.");
|
|
641
645
|
return {
|
|
642
646
|
ok: true,
|
|
643
647
|
summary,
|
|
@@ -17,13 +17,20 @@ export function resumeNext(ids) {
|
|
|
17
17
|
export function noCoverageNext(file, line) {
|
|
18
18
|
return `No coverage for ${file}:${line}. No test executed that line.`;
|
|
19
19
|
}
|
|
20
|
+
/** Every agent-facing count starts with unseen lines, then survivors. */
|
|
21
|
+
export function leadSummary(noCoverage, survivors, survivorWord, rest) {
|
|
22
|
+
return `${noCoverage} no coverage, ${survivors} ${survivorWord}, ${rest}`;
|
|
23
|
+
}
|
|
24
|
+
export function campaignSummary(counts) {
|
|
25
|
+
return leadSummary(counts.noCoverage, counts.survived, "survived", `${counts.killed} killed, ${counts.flaky} flaky, ${counts.timeouts} timed out, ${counts.errors} errored, of ${counts.finished} finished.`);
|
|
26
|
+
}
|
|
20
27
|
export function timeoutLimitMs(baselineMs, multiple, floorMs) {
|
|
21
28
|
const floor = Math.max(1, floorMs);
|
|
22
29
|
const scaled = Math.ceil(Math.max(0, baselineMs) * Math.max(1, multiple));
|
|
23
30
|
return Math.max(floor, scaled);
|
|
24
31
|
}
|
|
25
32
|
/** Kinds that collect a per-line map on the baseline. Other kinds pay the whole suite. */
|
|
26
|
-
const LINE_MAP_KINDS = new Set(["node", "pytest", "c"]);
|
|
33
|
+
const LINE_MAP_KINDS = new Set(["node", "pytest", "c", "go", "maven"]);
|
|
27
34
|
/** A baseline this long, with no line map, is too expensive to repeat for a large batch. */
|
|
28
35
|
export const SLOW_BASELINE_MS = 5_000;
|
|
29
36
|
/** Pending mutants above this count, on a slow unmapped suite, are not started. */
|
package/dist/mutate/suites.js
CHANGED
|
@@ -274,6 +274,9 @@ async function launch(pkg, spec, files, pattern, names, timeoutMs, env, hooks) {
|
|
|
274
274
|
const tests = selection.map(surefireTest).filter((name) => name.length > 0);
|
|
275
275
|
const args = ["-B", "test", ...(tests.length > 0 ? [`-Dtest=${tests.join(",")}`] : [])];
|
|
276
276
|
const whole = wantsNames && tests.length === 0;
|
|
277
|
+
// Surefire leaves the previous run's XML in place. A confirm that did not
|
|
278
|
+
// re-execute a class would otherwise read that class as failed again.
|
|
279
|
+
rmSync(path.join(pkg, "target", "surefire-reports"), { recursive: true, force: true });
|
|
277
280
|
return tag(spawnCollected("mvn", args, pkg, javaEnv(env), timeoutMs), shown("mvn", args), whole, selection);
|
|
278
281
|
}
|
|
279
282
|
if (spec.kind === "dotnet") {
|
|
@@ -284,9 +287,11 @@ async function launch(pkg, spec, files, pattern, names, timeoutMs, env, hooks) {
|
|
|
284
287
|
return tag(runDotnetExe(pkg, spec, files, timeoutMs, env), spec.command, wantsNames, selection);
|
|
285
288
|
if (spec.kind === "mocha")
|
|
286
289
|
return tag(runMocha(pkg, files, timeoutMs, env), spec.command, wantsNames, selection);
|
|
287
|
-
//
|
|
288
|
-
if (spec.kind === "script")
|
|
289
|
-
|
|
290
|
+
// A tty is only for a suite that runs `docker run -it` (tini). Everything else is bash.
|
|
291
|
+
if (spec.kind === "script") {
|
|
292
|
+
const launched = scriptInvocation(pkg);
|
|
293
|
+
return tag(runScript(pkg, timeoutMs, env, launched), launched.command, wantsNames, selection);
|
|
294
|
+
}
|
|
290
295
|
if (spec.kind === "c" || spec.kind === "cpp")
|
|
291
296
|
return tag(compileAndRun(pkg, files, spec.kind === "cpp", timeoutMs, env), spec.command, wantsNames, selection);
|
|
292
297
|
if (spec.kind === "java")
|
|
@@ -549,7 +554,72 @@ function clearBytecode(pkg) {
|
|
|
549
554
|
}
|
|
550
555
|
}
|
|
551
556
|
}
|
|
552
|
-
|
|
557
|
+
let gnuScriptCache = null;
|
|
558
|
+
function suiteNeedsTty(pkg) {
|
|
559
|
+
const files = new Set();
|
|
560
|
+
const rootScript = path.join(pkg, "run_tests.sh");
|
|
561
|
+
if (existsSync(rootScript))
|
|
562
|
+
files.add(rootScript);
|
|
563
|
+
try {
|
|
564
|
+
for (const name of readdirSync(pkg)) {
|
|
565
|
+
if (name.endsWith(".sh"))
|
|
566
|
+
files.add(path.join(pkg, name));
|
|
567
|
+
}
|
|
568
|
+
}
|
|
569
|
+
catch {
|
|
570
|
+
// A missing directory is not a tty.
|
|
571
|
+
}
|
|
572
|
+
let entry = "";
|
|
573
|
+
try {
|
|
574
|
+
entry = readFileSync(rootScript, "utf8");
|
|
575
|
+
}
|
|
576
|
+
catch {
|
|
577
|
+
entry = "";
|
|
578
|
+
}
|
|
579
|
+
for (const match of entry.matchAll(/(?:^|[\s"'`/])([\w.-]+\.sh)\b/g)) {
|
|
580
|
+
const full = path.resolve(pkg, match[1]);
|
|
581
|
+
if (full.startsWith(pkg + path.sep) && existsSync(full))
|
|
582
|
+
files.add(full);
|
|
583
|
+
}
|
|
584
|
+
for (const file of files) {
|
|
585
|
+
let body = "";
|
|
586
|
+
try {
|
|
587
|
+
body = readFileSync(file, "utf8");
|
|
588
|
+
}
|
|
589
|
+
catch {
|
|
590
|
+
continue;
|
|
591
|
+
}
|
|
592
|
+
if (dockerWantsTty(body))
|
|
593
|
+
return true;
|
|
594
|
+
}
|
|
595
|
+
return false;
|
|
596
|
+
}
|
|
597
|
+
function dockerWantsTty(text) {
|
|
598
|
+
return text.split("\n").some((line) => {
|
|
599
|
+
if (!/\bdocker\s+run\b/.test(line))
|
|
600
|
+
return false;
|
|
601
|
+
return /(?:^|\s)--tty(?:\s|=|$)/.test(line) || /(?:^|\s)-[A-Za-z]*t[A-Za-z]*(?:\s|$)/.test(line);
|
|
602
|
+
});
|
|
603
|
+
}
|
|
604
|
+
function gnuScript() {
|
|
605
|
+
if (gnuScriptCache !== null)
|
|
606
|
+
return gnuScriptCache;
|
|
607
|
+
const probe = spawnSync("script", ["--version"], { encoding: "utf8", timeout: 2000 });
|
|
608
|
+
const text = `${probe.stdout ?? ""}\n${probe.stderr ?? ""}`;
|
|
609
|
+
gnuScriptCache = probe.status === 0 && /util-linux|\bGNU\b/.test(text);
|
|
610
|
+
return gnuScriptCache;
|
|
611
|
+
}
|
|
612
|
+
function scriptInvocation(pkg) {
|
|
613
|
+
if (!suiteNeedsTty(pkg))
|
|
614
|
+
return { bin: "bash", args: ["run_tests.sh"], command: "bash run_tests.sh" };
|
|
615
|
+
if (gnuScript()) {
|
|
616
|
+
const args = ["-q", "-c", "bash run_tests.sh", "/dev/null"];
|
|
617
|
+
return { bin: "script", args, command: shown("script", args) };
|
|
618
|
+
}
|
|
619
|
+
const args = ["-q", "/dev/null", "bash", "run_tests.sh"];
|
|
620
|
+
return { bin: "script", args, command: shown("script", args) };
|
|
621
|
+
}
|
|
622
|
+
function runScript(pkg, timeoutMs, env, launched) {
|
|
553
623
|
const next = { ...env };
|
|
554
624
|
const shim = writeDockerShim(pkg);
|
|
555
625
|
if (shim)
|
|
@@ -560,7 +630,7 @@ function runScript(pkg, timeoutMs, env) {
|
|
|
560
630
|
if (!next.ARCH_NATIVE && existsSync(inner) && existsSync(build) && readFileSync(build, "utf8").includes("ARCH_NATIVE")) {
|
|
561
631
|
next.ARCH_NATIVE = "1";
|
|
562
632
|
}
|
|
563
|
-
return spawnCollected(
|
|
633
|
+
return spawnCollected(launched.bin, launched.args, pkg, next, timeoutMs);
|
|
564
634
|
}
|
|
565
635
|
function writeDockerShim(pkg) {
|
|
566
636
|
const found = spawnSync("sh", ["-c", "command -v docker"], { encoding: "utf8" });
|
|
@@ -1132,7 +1202,9 @@ function parseDotnet(text, code) {
|
|
|
1132
1202
|
const names = [];
|
|
1133
1203
|
const failed = [];
|
|
1134
1204
|
for (const line of text.split("\n")) {
|
|
1135
|
-
|
|
1205
|
+
// A result line carries a duration: "Passed StaysClosed [4 ms]".
|
|
1206
|
+
// "Failed to load ..." is a diagnostic, not a test, and must not redden the baseline.
|
|
1207
|
+
const match = line.match(/^\s*(Passed|Failed)\s+(.+?)\s+\[(?:<\s*)?\d+(?:\.\d+)?\s*(?:ms|s)\]\s*$/);
|
|
1136
1208
|
if (!match)
|
|
1137
1209
|
continue;
|
|
1138
1210
|
names.push(match[2]);
|
package/dist/mutate/tally.js
CHANGED
|
@@ -3,11 +3,11 @@ import path from "node:path";
|
|
|
3
3
|
import { SCHEMA_VERSION } from "../contract.js";
|
|
4
4
|
import { advisePrune } from "../matrix/report.js";
|
|
5
5
|
import { isBuildName } from "./kill-label.js";
|
|
6
|
-
import { WHOLE_PROGRAM_TEST } from "./suite-decision.js";
|
|
6
|
+
import { leadSummary, TIMEOUT_NEXT, WHOLE_PROGRAM_TEST } from "./suite-decision.js";
|
|
7
7
|
const NOT_A_TEST = new Set(["suite", "pytest", "unittest", "command", WHOLE_PROGRAM_TEST]);
|
|
8
8
|
/**
|
|
9
9
|
* Read a finished mutate run. Name tests that killed a mutant, tests that killed
|
|
10
|
-
* nothing, and one gap per survivor
|
|
10
|
+
* nothing, and one gap per survivor. An unseen line is not a gap. This never deletes a file.
|
|
11
11
|
*/
|
|
12
12
|
export function tallyRun(outDir) {
|
|
13
13
|
const resultsDir = path.join(outDir, "results");
|
|
@@ -75,18 +75,30 @@ export function tallyRun(outDir) {
|
|
|
75
75
|
.map(([name]) => name)
|
|
76
76
|
.sort();
|
|
77
77
|
const drop = pruning.advice;
|
|
78
|
+
const noCoverage = results.filter((result) => result.outcome === "no coverage").length;
|
|
79
|
+
const survived = results.filter((result) => result.outcome === "survived").length;
|
|
80
|
+
const timeouts = results.filter((result) => result.outcome === "timeout").length;
|
|
78
81
|
const gaps = results
|
|
79
|
-
.filter((result) => result.outcome === "survived"
|
|
82
|
+
.filter((result) => result.outcome === "survived")
|
|
80
83
|
.map((result) => ({
|
|
81
84
|
id: result.id,
|
|
82
85
|
file: result.files?.[0]?.file ?? "",
|
|
83
86
|
line: result.files?.[0]?.line ?? 0,
|
|
84
|
-
outcome:
|
|
87
|
+
outcome: "survived",
|
|
85
88
|
}))
|
|
86
89
|
.sort((a, b) => a.file.localeCompare(b.file) || a.line - b.line || a.id.localeCompare(b.id));
|
|
87
90
|
const shown = gaps.slice(0, 10);
|
|
88
|
-
const summary = `${keep.length} tests to keep, ${drop.length} tests killed nothing, ${gaps.length} gaps. No test file was deleted
|
|
89
|
-
const
|
|
91
|
+
const summary = leadSummary(noCoverage, survived, "survived", `${keep.length} tests to keep, ${drop.length} tests killed nothing, ${gaps.length} gaps. No test file was deleted.`);
|
|
92
|
+
const notes = [];
|
|
93
|
+
if (noCoverage > 0)
|
|
94
|
+
notes.push(`${noCoverage} no coverage. That line was not seen. It is not a pass and not a gap.`);
|
|
95
|
+
if (gaps.length > 0)
|
|
96
|
+
notes.push(`First gap is ${gaps[0].id} at ${gaps[0].file}:${gaps[0].line}. A survivor is not a pass.`);
|
|
97
|
+
if (timeouts > 0)
|
|
98
|
+
notes.push(TIMEOUT_NEXT);
|
|
99
|
+
if (notes.length === 0)
|
|
100
|
+
notes.push("No survivor in this run. No test file was deleted.");
|
|
101
|
+
const next = notes.join(" ");
|
|
90
102
|
const complete = {
|
|
91
103
|
schemaVersion: SCHEMA_VERSION,
|
|
92
104
|
ok: true,
|
package/dist/swarm/check-kill.js
CHANGED
|
@@ -5,6 +5,7 @@ import { directImporters } from "../mutate/affected.js";
|
|
|
5
5
|
import { filesInDiff } from "../mutate/patch.js";
|
|
6
6
|
import { killLabel } from "../mutate/kill-label.js";
|
|
7
7
|
import { runMutants } from "../mutate/run.js";
|
|
8
|
+
import { campaignSummary } from "../mutate/suite-decision.js";
|
|
8
9
|
/** The gap is done only when this mutant is killed and the baseline suite is green. */
|
|
9
10
|
export async function checkKill(id, options) {
|
|
10
11
|
const importers = importersOf(id, options);
|
|
@@ -28,6 +29,21 @@ export async function checkKill(id, options) {
|
|
|
28
29
|
const baseline = JSON.parse(readFileSync(baselineFile, "utf8"));
|
|
29
30
|
suiteGreen = baseline.ok === true;
|
|
30
31
|
}
|
|
32
|
+
// A red baseline stops inside the run. Do not read a result that names the broken test.
|
|
33
|
+
if (!report.ok) {
|
|
34
|
+
return {
|
|
35
|
+
schemaVersion: SCHEMA_VERSION,
|
|
36
|
+
ok: false,
|
|
37
|
+
command: "check-kill",
|
|
38
|
+
summary: `${id} is not done. ${report.summary}`,
|
|
39
|
+
next: "The task is not done.",
|
|
40
|
+
nextCall: null,
|
|
41
|
+
done: false,
|
|
42
|
+
outcome: null,
|
|
43
|
+
suiteGreen,
|
|
44
|
+
killedBy: [],
|
|
45
|
+
};
|
|
46
|
+
}
|
|
31
47
|
const resultFile = path.join(options.outDir, "results", `${id}.json`);
|
|
32
48
|
let outcome = null;
|
|
33
49
|
let killedBy = [];
|
|
@@ -38,11 +54,12 @@ export async function checkKill(id, options) {
|
|
|
38
54
|
}
|
|
39
55
|
const done = suiteGreen && outcome === "killed";
|
|
40
56
|
const label = done ? killLabel(killedBy) : null;
|
|
57
|
+
const head = checkHead(report);
|
|
41
58
|
return {
|
|
42
59
|
schemaVersion: SCHEMA_VERSION,
|
|
43
60
|
ok: done,
|
|
44
61
|
command: "check-kill",
|
|
45
|
-
summary: done ? `${id} is dead and the suite is green.` : `${id} is not done
|
|
62
|
+
summary: done ? `${head} ${id} is dead and the suite is green.` : `${head} ${id} is not done.`,
|
|
46
63
|
next: label?.next ?? (done ? "The gap-fixing task is done." : "The task is not done."),
|
|
47
64
|
nextCall: null,
|
|
48
65
|
done,
|
|
@@ -60,6 +77,18 @@ function importersOf(id, options) {
|
|
|
60
77
|
const rels = filesInDiff(readFileSync(patch, "utf8")).map((item) => packageRelative(options.repoDir, options.packageDir, item.file));
|
|
61
78
|
return directImporters(options.packageDir, rels, options.testsDir);
|
|
62
79
|
}
|
|
80
|
+
function checkHead(report) {
|
|
81
|
+
const finished = report.killed + report.survived + report.flaky + report.timeouts + report.errors + report.noCoverage;
|
|
82
|
+
return campaignSummary({
|
|
83
|
+
noCoverage: report.noCoverage,
|
|
84
|
+
survived: report.survived,
|
|
85
|
+
killed: report.killed,
|
|
86
|
+
flaky: report.flaky,
|
|
87
|
+
timeouts: report.timeouts,
|
|
88
|
+
errors: report.errors,
|
|
89
|
+
finished,
|
|
90
|
+
});
|
|
91
|
+
}
|
|
63
92
|
function packageRelative(repoDir, packageDir, file) {
|
|
64
93
|
const norm = file.split(path.sep).join("/");
|
|
65
94
|
let repo = repoDir;
|
package/dist/verify/change.js
CHANGED
|
@@ -7,6 +7,7 @@ import { findInSource } from "../mutate/find.js";
|
|
|
7
7
|
import { mutantId } from "../mutate/ids.js";
|
|
8
8
|
import { forwardDiff, git } from "../mutate/patch.js";
|
|
9
9
|
import { runMutants } from "../mutate/run.js";
|
|
10
|
+
import { leadSummary } from "../mutate/suite-decision.js";
|
|
10
11
|
import { discoverSuite } from "../mutate/suites.js";
|
|
11
12
|
/** Diff-scoped mutants, the tests that can see the edit, and golden contract rows that changed. */
|
|
12
13
|
export async function verifyChange(options) {
|
|
@@ -112,7 +113,7 @@ export async function verifyChange(options) {
|
|
|
112
113
|
function bodyOf(outDir, ok, affected, ran, notRan, caught, missed, other, goldenContract, notRun, suiteCommand) {
|
|
113
114
|
const uncovered = other.filter((item) => item.outcome === "no coverage");
|
|
114
115
|
const summary = ok
|
|
115
|
-
?
|
|
116
|
+
? leadSummary(uncovered.length, missed.length, "missed", `${caught.length} caught, ${notRun.length} not run, ${affected.length} affected tests, ${goldenContract.length} golden contract changes.`)
|
|
116
117
|
: "verify-change did not finish.";
|
|
117
118
|
const first = missed[0] ?? uncovered[0] ?? notRun[0] ?? caught[0];
|
|
118
119
|
return {
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "probatio",
|
|
3
|
-
"version": "0.0.
|
|
3
|
+
"version": "0.0.2",
|
|
4
4
|
"description": "Mutation runner for AI agents. Scores a test suite by the bugs it catches.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -18,7 +18,7 @@
|
|
|
18
18
|
"url": "https://github.com/IronAdamant/Probatio/issues"
|
|
19
19
|
},
|
|
20
20
|
"scripts": {
|
|
21
|
-
"build": "tsc -p tsconfig.json && mkdir -p dist/mutate && cp src/mutate/fail-reporter.mjs src/mutate/node-coverage.mjs src/mutate/py-reporter.py src/mutate/py-pytest-reporter.py dist/mutate/",
|
|
21
|
+
"build": "tsc -p tsconfig.json && mkdir -p dist/mutate && cp src/mutate/fail-reporter.mjs src/mutate/node-coverage.mjs src/mutate/py-reporter.py src/mutate/py-pytest-reporter.py src/mutate/probatio-jacoco-run.java dist/mutate/",
|
|
22
22
|
"test": "tsx --test tests/*.test.ts",
|
|
23
23
|
"probatio": "tsx src/cli.ts"
|
|
24
24
|
},
|