probatio 0.1.2 → 0.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +49 -13
- package/dist/cli.js +101 -53
- package/dist/contract.js +7 -5
- package/dist/mcp.js +23 -0
- package/dist/mutate/coverage-map.js +75 -18
- package/dist/mutate/generate.js +25 -4
- package/dist/mutate/run.js +98 -17
- package/dist/mutate/rust-coverage.js +4 -19
- package/dist/mutate/sealed.js +40 -1
- package/dist/mutate/suites.js +49 -8
- package/dist/mutate/tally.js +38 -1
- package/dist/node-version.js +13 -0
- package/dist/swarm/queue.js +13 -7
- package/dist/verify/change.js +5 -0
- package/package.json +5 -3
package/README.md
CHANGED
|
@@ -1,21 +1,38 @@
|
|
|
1
1
|
# Probatio
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
(Claude, Grok, ChatGPT, others), working alone or as swarms. Every claim it makes is checkable.
|
|
5
|
-
It remembers what was fixed so agents with small context do not redo or undo work, and it shows
|
|
6
|
-
which tests already touch a change.
|
|
3
|
+
[](https://www.npmjs.com/package/probatio)
|
|
7
4
|
|
|
8
|
-
|
|
5
|
+
Probatio scores a test suite by the bugs it already catches. A kill is an assertion the suite had. The kill count is not a merge gate.
|
|
9
6
|
|
|
10
|
-
|
|
7
|
+
Node 22.6 or newer:
|
|
11
8
|
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
9
|
+
```bash
|
|
10
|
+
npm install probatio
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
npx probatio mutate generate --package . --out .probatio/generate
|
|
15
|
+
npx probatio mutate run --package . --patches .probatio/generate/mutants --out .probatio/runs
|
|
16
|
+
npx probatio mutate tally --out .probatio/runs
|
|
17
|
+
```
|
|
16
18
|
|
|
17
|
-
|
|
18
|
-
|
|
19
|
+
Each command prints one JSON object and exits 0 only when `ok` is true.
|
|
20
|
+
|
|
21
|
+
```json
|
|
22
|
+
{
|
|
23
|
+
"schemaVersion": 1,
|
|
24
|
+
"ok": true,
|
|
25
|
+
"summary": "0 no coverage, 0 survived, 1 killed, 0 flaky, 0 timed out, 0 errored, of 1 finished.",
|
|
26
|
+
"next": "No survivor in this batch.",
|
|
27
|
+
"nextCall": { "argv": ["mutate", "tally", "--out", ".probatio/runs"] }
|
|
28
|
+
}
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
Agents changing this repo should read [AGENTS.md](AGENTS.md). The MCP server is `probatio mcp`. Snippets are in the MCP section below. People building Probatio should read [docs/builders.md](docs/builders.md). Small suites live in [examples/](examples/).
|
|
32
|
+
|
|
33
|
+
## Status
|
|
34
|
+
|
|
35
|
+
Status (2026-10-06): `mutate run` discovers the project suite and scores it. Node, pytest, C (when the binary was built with LLVM coverage), Go, Maven/Java, Rust, C#, and Mocha collect a line map on the baseline. Mocha's map comes from its own root hooks. Loading the node:test collector under Mocha does not name Mocha tests. A line no test executes is `no coverage`, and the suite is not started. A plain Node script can write the lines it ran, and the parent test that reads that dump stores them under its own name. A child the parent does not read stays `no coverage`, and the suite is not started. A suite with no line map, a baseline of 5 seconds or more, and more than 30 mutants stops before the first mutant. `verify-change` names the tests the line map ran. It scores `src/`, a `lib/` file when the package tests import `lib/`, and a Go file beside `go.mod`. Tests, docs, `dist/`, and `node_modules` stay unseen. `ran` stays the direct-importer list. Covered mutants of one Node file run one after another inside the suite process that is already running. `--workers` stays 1. A Node test that sets its own `timeout` keeps that time when `--test-timeout-ms` is shorter. A mutant that crashes or leaves that process dirty ends it, and the next mutant starts clean. An uncovered line still does not start the suite. `mutate tally` does not delete a test. A kill is an assertion the suite already had. A sealed miss means the suite did not see the bug. The kill count is not a merge gate. On Commons CSV (Apache-2.0, commit `2c83a308`), two sealed runs agreed: with the new test file hidden, `CSVFormatTest.testFormatThrowsNullPointerException` failed on the reverted printer. `deletedTests` stayed 0. That older JUnit platform did not write a line map, so the first pass was the whole suite and confirm reran the older test. Auspex, markupsafe, and a hand-planted fixture are not that result. The package version in this repository is 0.1.3. npm latest is 0.1.2 (`29301bc`). The BugsInPy search row is still open, and the rows that apply only when a sealed run misses do not apply, because this catch held. The MCP server speaks one JSON object per line and returns the same JSON as the CLI.
|
|
19
36
|
|
|
20
37
|
## mutate
|
|
21
38
|
|
|
@@ -81,4 +98,23 @@ Lists the tests that can see the diff, runs mutants on the changed lines, and re
|
|
|
81
98
|
|
|
82
99
|
An item is `.probatio/queue/<id>.json`. `queue claim` renames it to `claimed/<agent>-<id>.json` with a lease. `queue reap` moves an expired lease back. `check-kill <id>` runs the tests that directly import the mutated file and confirms the killing test again. It is done only when that mutant dies and that suite is still green. `verify-change` is the edit-sized run: it uses the line map and does not confirm a second time.
|
|
83
100
|
|
|
84
|
-
|
|
101
|
+
## MCP
|
|
102
|
+
|
|
103
|
+
`probatio mcp` and the `probatio-mcp` bin speak stdio JSON-RPC, one JSON object per line. The tool name is `probatio`. `argv` is the CLI words. `ping` returns `{}`. An unknown method returns JSON-RPC `-32601` with the same id.
|
|
104
|
+
|
|
105
|
+
```json
|
|
106
|
+
{
|
|
107
|
+
"mcpServers": {
|
|
108
|
+
"probatio": {
|
|
109
|
+
"command": "probatio",
|
|
110
|
+
"args": ["mcp"]
|
|
111
|
+
}
|
|
112
|
+
}
|
|
113
|
+
}
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
That block is the Claude, Cursor, and Grok shape. The tool runs the CLI and returns that command's JSON. `nextCall.argv` keeps absolute paths so a later spawn does not have to expand `~`.
|
|
117
|
+
|
|
118
|
+
`mutate generate` reads the commit (`HEAD` unless you pass `--commit`). `--working-tree` reads the checkout on disk and says so. `mutate run` scores the commit either way.
|
|
119
|
+
|
|
120
|
+
`verify-change` does not score uncommitted edits when `--base` and `--commit` are the same. A clean empty diff says `No diff-scoped mutant.`
|
package/dist/cli.js
CHANGED
|
@@ -10,65 +10,82 @@ import { appendFinding, chooseSealed, fixGap, guardAllows, lineHash, loadState,
|
|
|
10
10
|
import { git } from "./mutate/patch.js";
|
|
11
11
|
import { generateMutants } from "./mutate/generate.js";
|
|
12
12
|
import { runMutants } from "./mutate/run.js";
|
|
13
|
-
import { LABEL_LEAK_SUMMARY, labelLeaked, readSealedLabel } from "./mutate/sealed.js";
|
|
13
|
+
import { LABEL_LEAK_SUMMARY, labelLeaked, readSealedLabel, sealedTreeLeaked } from "./mutate/sealed.js";
|
|
14
14
|
import { generateNext } from "./mutate/suite-decision.js";
|
|
15
15
|
import { discoverSuite } from "./mutate/suites.js";
|
|
16
16
|
import { tallyRun } from "./mutate/tally.js";
|
|
17
17
|
import { checkKill } from "./swarm/check-kill.js";
|
|
18
18
|
import { claimItem, reapClaims, seedQueue } from "./swarm/queue.js";
|
|
19
19
|
import { verifyChange } from "./verify/change.js";
|
|
20
|
-
|
|
21
|
-
const
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
finish(await generateCommand(parsed.flags), parsed.human);
|
|
25
|
-
if (group === "mutate" && action === "run")
|
|
26
|
-
finish(await runCommand(parsed.flags), parsed.human);
|
|
27
|
-
if (group === "mutate" && action === "sealed")
|
|
28
|
-
finish(await sealedCommand(parsed.flags), parsed.human);
|
|
29
|
-
if (group === "mutate" && action === "tally")
|
|
30
|
-
finish(tallyCommand(parsed.flags), parsed.human);
|
|
31
|
-
if (group === "ledger" && action === "build")
|
|
32
|
-
finish(await ledgerCommand(parsed.flags), parsed.human);
|
|
33
|
-
if (group === "matrix" && action === "report")
|
|
34
|
-
finish(matrixCommand(parsed.flags), parsed.human);
|
|
35
|
-
if (group === "golden" && action === "check")
|
|
36
|
-
finish(goldenCommand(parsed.flags), parsed.human);
|
|
37
|
-
if (group === "gap" && action === "fix")
|
|
38
|
-
finish(gapFixCommand(parsed.flags), parsed.human);
|
|
39
|
-
if (group === "gap" && action === "revert")
|
|
40
|
-
finish(gapRevertCommand(parsed.flags), parsed.human);
|
|
41
|
-
if (group === "guard" && action === "check")
|
|
42
|
-
finish(guardCommand(parsed.flags), parsed.human);
|
|
43
|
-
if (group === "findings" && action === "add")
|
|
44
|
-
finish(findingsCommand(parsed.flags), parsed.human);
|
|
45
|
-
if (group === "status")
|
|
46
|
-
finish(statusCommand(parsed.flags), parsed.human);
|
|
47
|
-
if (group === "seal")
|
|
48
|
-
finish(sealCommand(parsed.flags), parsed.human);
|
|
49
|
-
if (group === "queue" && action === "seed")
|
|
50
|
-
finish(queueSeedCommand(parsed.flags), parsed.human);
|
|
51
|
-
if (group === "queue" && action === "claim")
|
|
52
|
-
finish(queueClaimCommand(parsed.flags), parsed.human);
|
|
53
|
-
if (group === "queue" && action === "reap")
|
|
54
|
-
finish(queueReapCommand(parsed.flags), parsed.human);
|
|
55
|
-
if (group === "check-kill")
|
|
56
|
-
finish(await checkKillCommand(parsed.flags, action), parsed.human);
|
|
57
|
-
if (group === "verify-change")
|
|
58
|
-
finish(await verifyCommand(parsed.flags), parsed.human);
|
|
59
|
-
finish(usage(false, "Use mutate, ledger, matrix, golden, gap, guard, findings, status, queue, check-kill, or verify-change."), parsed.human);
|
|
60
|
-
}
|
|
61
|
-
catch (error) {
|
|
62
|
-
const message = error instanceof Error ? error.message : String(error);
|
|
63
|
-
finish({
|
|
20
|
+
import { nodeTooOld } from "./node-version.js";
|
|
21
|
+
const tooOld = nodeTooOld(process.versions.node);
|
|
22
|
+
if (tooOld) {
|
|
23
|
+
process.stdout.write(`${JSON.stringify({
|
|
64
24
|
schemaVersion: SCHEMA_VERSION,
|
|
65
25
|
ok: false,
|
|
66
|
-
command:
|
|
67
|
-
summary:
|
|
68
|
-
next: "
|
|
26
|
+
command: "mutate.help",
|
|
27
|
+
summary: tooOld,
|
|
28
|
+
next: "Install Node 22.6 or newer and run the command again.",
|
|
69
29
|
nextCall: null,
|
|
70
|
-
}
|
|
30
|
+
})}\n`);
|
|
31
|
+
process.exit(1);
|
|
71
32
|
}
|
|
33
|
+
const parsed = parseArgs(process.argv.slice(2));
|
|
34
|
+
const [group, action] = parsed.command;
|
|
35
|
+
if (group === "mcp" && !action) {
|
|
36
|
+
await import("./mcp.js");
|
|
37
|
+
}
|
|
38
|
+
else
|
|
39
|
+
try {
|
|
40
|
+
if (group === "mutate" && action === "generate")
|
|
41
|
+
finish(await generateCommand(parsed.flags), parsed.human);
|
|
42
|
+
if (group === "mutate" && action === "run")
|
|
43
|
+
finish(await runCommand(parsed.flags), parsed.human);
|
|
44
|
+
if (group === "mutate" && action === "sealed")
|
|
45
|
+
finish(await sealedCommand(parsed.flags), parsed.human);
|
|
46
|
+
if (group === "mutate" && action === "tally")
|
|
47
|
+
finish(tallyCommand(parsed.flags), parsed.human);
|
|
48
|
+
if (group === "ledger" && action === "build")
|
|
49
|
+
finish(await ledgerCommand(parsed.flags), parsed.human);
|
|
50
|
+
if (group === "matrix" && action === "report")
|
|
51
|
+
finish(matrixCommand(parsed.flags), parsed.human);
|
|
52
|
+
if (group === "golden" && action === "check")
|
|
53
|
+
finish(goldenCommand(parsed.flags), parsed.human);
|
|
54
|
+
if (group === "gap" && action === "fix")
|
|
55
|
+
finish(gapFixCommand(parsed.flags), parsed.human);
|
|
56
|
+
if (group === "gap" && action === "revert")
|
|
57
|
+
finish(gapRevertCommand(parsed.flags), parsed.human);
|
|
58
|
+
if (group === "guard" && action === "check")
|
|
59
|
+
finish(guardCommand(parsed.flags), parsed.human);
|
|
60
|
+
if (group === "findings" && action === "add")
|
|
61
|
+
finish(findingsCommand(parsed.flags), parsed.human);
|
|
62
|
+
if (group === "status")
|
|
63
|
+
finish(statusCommand(parsed.flags), parsed.human);
|
|
64
|
+
if (group === "seal")
|
|
65
|
+
finish(sealCommand(parsed.flags), parsed.human);
|
|
66
|
+
if (group === "queue" && action === "seed")
|
|
67
|
+
finish(queueSeedCommand(parsed.flags), parsed.human);
|
|
68
|
+
if (group === "queue" && action === "claim")
|
|
69
|
+
finish(queueClaimCommand(parsed.flags), parsed.human);
|
|
70
|
+
if (group === "queue" && action === "reap")
|
|
71
|
+
finish(queueReapCommand(parsed.flags), parsed.human);
|
|
72
|
+
if (group === "check-kill")
|
|
73
|
+
finish(await checkKillCommand(parsed.flags, action), parsed.human);
|
|
74
|
+
if (group === "verify-change")
|
|
75
|
+
finish(await verifyCommand(parsed.flags), parsed.human);
|
|
76
|
+
finish(usage(false, "Use mutate, ledger, matrix, golden, gap, guard, findings, status, queue, check-kill, or verify-change."), parsed.human);
|
|
77
|
+
}
|
|
78
|
+
catch (error) {
|
|
79
|
+
const message = error instanceof Error ? error.message : String(error);
|
|
80
|
+
finish({
|
|
81
|
+
schemaVersion: SCHEMA_VERSION,
|
|
82
|
+
ok: false,
|
|
83
|
+
command: `${group ?? "mutate"}.${action ?? "help"}`,
|
|
84
|
+
summary: message,
|
|
85
|
+
next: "Fix the command and run it again.",
|
|
86
|
+
nextCall: null,
|
|
87
|
+
}, parsed.human);
|
|
88
|
+
}
|
|
72
89
|
function usage(ok, summary) {
|
|
73
90
|
return {
|
|
74
91
|
schemaVersion: SCHEMA_VERSION,
|
|
@@ -93,7 +110,19 @@ async function generateCommand(flags) {
|
|
|
93
110
|
maxMutants: int(flags, "max-mutants") ?? null,
|
|
94
111
|
maxMinutes: int(flags, "max-minutes") ?? null,
|
|
95
112
|
skipFiles: texts(flags, "skip-file"),
|
|
113
|
+
commit: text(flags, "commit") ?? "HEAD",
|
|
114
|
+
workingTree: bool(flags, "working-tree", false),
|
|
96
115
|
});
|
|
116
|
+
if (result.error) {
|
|
117
|
+
return {
|
|
118
|
+
schemaVersion: SCHEMA_VERSION,
|
|
119
|
+
ok: false,
|
|
120
|
+
command: "mutate.generate",
|
|
121
|
+
summary: result.error,
|
|
122
|
+
next: "Commit the file, or pass --working-tree to mutate the checkout on disk. mutate run scores the commit.",
|
|
123
|
+
nextCall: null,
|
|
124
|
+
};
|
|
125
|
+
}
|
|
97
126
|
if (result.violations.length > 0) {
|
|
98
127
|
const first = result.violations[0];
|
|
99
128
|
return {
|
|
@@ -117,9 +146,14 @@ async function generateCommand(flags) {
|
|
|
117
146
|
summary: stopped
|
|
118
147
|
? `Stopped by the time budget after ${result.filesVisited} files. ${result.mutants.length} mutants written.`
|
|
119
148
|
: `${result.mutants.length} mutants in ${result.filesVisited} files. No string-literal mutant.`,
|
|
120
|
-
next:
|
|
121
|
-
|
|
122
|
-
|
|
149
|
+
next: [
|
|
150
|
+
stopped
|
|
151
|
+
? "Narrow --src or raise --max-minutes, then run the same command."
|
|
152
|
+
: generateNext(result.mutants.length, suite ? { kind: suite.kind, command: suite.command } : null),
|
|
153
|
+
result.source === "working-tree"
|
|
154
|
+
? "These patches match the working tree. mutate run scores the commit, so commit the files before mutate run."
|
|
155
|
+
: "",
|
|
156
|
+
].filter((line) => line.length > 0).join(" "),
|
|
123
157
|
nextCall: result.mutants.length === 0
|
|
124
158
|
? null
|
|
125
159
|
: { argv: ["mutate", "run", "--package", packageDir, "--patches", patches, "--out", path.join(outDir, "runs")] },
|
|
@@ -133,6 +167,19 @@ async function generateCommand(flags) {
|
|
|
133
167
|
};
|
|
134
168
|
}
|
|
135
169
|
async function runCommand(flags, hide = null) {
|
|
170
|
+
const hidden = hide ?? texts(flags, "hide");
|
|
171
|
+
const escaped = hidden.find((rel) => rel.split(/[\\/]/).includes(".."));
|
|
172
|
+
if (escaped) {
|
|
173
|
+
return {
|
|
174
|
+
schemaVersion: SCHEMA_VERSION,
|
|
175
|
+
ok: false,
|
|
176
|
+
command: "mutate.run",
|
|
177
|
+
summary: `--hide stays inside the package. Refused ${escaped}.`,
|
|
178
|
+
next: "Pass a relative path with no .. segment.",
|
|
179
|
+
nextCall: null,
|
|
180
|
+
killed: 0,
|
|
181
|
+
};
|
|
182
|
+
}
|
|
136
183
|
const onlyNames = texts(flags, "only-test");
|
|
137
184
|
if (onlyNames.some((id) => id === "::command" || id.endsWith("::command"))) {
|
|
138
185
|
return {
|
|
@@ -234,9 +281,10 @@ function tallyCommand(flags) {
|
|
|
234
281
|
async function sealedCommand(flags) {
|
|
235
282
|
const label = readSealedLabel(path.resolve(requireText(flags, "label")));
|
|
236
283
|
const hide = texts(flags, "hide");
|
|
284
|
+
const outDir = path.resolve(requireText(flags, "out"));
|
|
237
285
|
const report = await runCommand(flags, hide);
|
|
238
286
|
const body = JSON.stringify(report);
|
|
239
|
-
if (!labelLeaked(body, label))
|
|
287
|
+
if (!labelLeaked(body, label) && !sealedTreeLeaked(outDir, label))
|
|
240
288
|
return report;
|
|
241
289
|
return {
|
|
242
290
|
schemaVersion: SCHEMA_VERSION,
|
package/dist/contract.js
CHANGED
|
@@ -1,18 +1,20 @@
|
|
|
1
1
|
import { homedir } from "node:os";
|
|
2
2
|
export const SCHEMA_VERSION = 1;
|
|
3
3
|
/** Replace the home directory so a result never carries a user's name or home path. */
|
|
4
|
-
export function scrub(value, home = homedir()) {
|
|
4
|
+
export function scrub(value, home = homedir(), keepPaths = false) {
|
|
5
5
|
if (typeof value === "string") {
|
|
6
|
-
if (!home || !value.includes(home))
|
|
6
|
+
if (keepPaths || !home || !value.includes(home))
|
|
7
7
|
return value;
|
|
8
8
|
return value.split(home).join("~");
|
|
9
9
|
}
|
|
10
10
|
if (Array.isArray(value))
|
|
11
|
-
return value.map((item) => scrub(item, home));
|
|
11
|
+
return value.map((item) => scrub(item, home, keepPaths));
|
|
12
12
|
if (value && typeof value === "object") {
|
|
13
13
|
const out = {};
|
|
14
|
-
for (const [key, item] of Object.entries(value))
|
|
15
|
-
|
|
14
|
+
for (const [key, item] of Object.entries(value)) {
|
|
15
|
+
// nextCall.argv is executed. A tilde is not expanded by spawn or by MCP.
|
|
16
|
+
out[key] = scrub(item, home, keepPaths || key === "nextCall");
|
|
17
|
+
}
|
|
16
18
|
return out;
|
|
17
19
|
}
|
|
18
20
|
return value;
|
package/dist/mcp.js
CHANGED
|
@@ -3,6 +3,19 @@ import { spawnSync } from "node:child_process";
|
|
|
3
3
|
import { existsSync, readFileSync } from "node:fs";
|
|
4
4
|
import path from "node:path";
|
|
5
5
|
import { fileURLToPath } from "node:url";
|
|
6
|
+
import { nodeTooOld } from "./node-version.js";
|
|
7
|
+
const tooOld = nodeTooOld(process.versions.node);
|
|
8
|
+
if (tooOld) {
|
|
9
|
+
process.stdout.write(`${JSON.stringify({
|
|
10
|
+
schemaVersion: 1,
|
|
11
|
+
ok: false,
|
|
12
|
+
command: "mcp",
|
|
13
|
+
summary: tooOld,
|
|
14
|
+
next: "Install Node 22.6 or newer and run the command again.",
|
|
15
|
+
nextCall: null,
|
|
16
|
+
})}\n`);
|
|
17
|
+
process.exit(1);
|
|
18
|
+
}
|
|
6
19
|
const here = path.dirname(fileURLToPath(import.meta.url));
|
|
7
20
|
const version = readVersion();
|
|
8
21
|
const tool = {
|
|
@@ -63,6 +76,10 @@ function handle(message) {
|
|
|
63
76
|
send({ jsonrpc: "2.0", id: message.id, result: { tools: [tool] } });
|
|
64
77
|
return;
|
|
65
78
|
}
|
|
79
|
+
if (message.method === "ping") {
|
|
80
|
+
send({ jsonrpc: "2.0", id: message.id, result: {} });
|
|
81
|
+
return;
|
|
82
|
+
}
|
|
66
83
|
if (message.method === "tools/call") {
|
|
67
84
|
const argv = message.params?.arguments?.argv;
|
|
68
85
|
if (!Array.isArray(argv) || argv.some((item) => typeof item !== "string")) {
|
|
@@ -92,7 +109,13 @@ function handle(message) {
|
|
|
92
109
|
isError: envelope.ok !== true,
|
|
93
110
|
},
|
|
94
111
|
});
|
|
112
|
+
return;
|
|
95
113
|
}
|
|
114
|
+
send({
|
|
115
|
+
jsonrpc: "2.0",
|
|
116
|
+
id: message.id,
|
|
117
|
+
error: { code: -32601, message: `method not found: ${message.method}` },
|
|
118
|
+
});
|
|
96
119
|
}
|
|
97
120
|
function cliCommand() {
|
|
98
121
|
const compiled = path.join(here, "cli.js");
|
|
@@ -104,17 +104,10 @@ export function writeLlvmCoverage(pkg, profileDir, binary, dest) {
|
|
|
104
104
|
return { ok: false, detail: "no profraw" };
|
|
105
105
|
if (!binary)
|
|
106
106
|
return { ok: false, detail: "no instrumented binary" };
|
|
107
|
-
const llvmCov = findTool("llvm-cov");
|
|
108
|
-
const llvmProf = findTool("llvm-profdata");
|
|
109
|
-
if (!llvmCov || !llvmProf)
|
|
110
|
-
return { ok: false, detail: `llvm tool missing (${llvmCov ? "llvm-profdata" : "llvm-cov"})` };
|
|
111
107
|
const merged = path.join(profileDir, "baseline.profdata");
|
|
112
|
-
const
|
|
113
|
-
if (
|
|
114
|
-
return
|
|
115
|
-
const exported = spawnSync(llvmCov, ["export", binary, `-instr-profile=${merged}`], { encoding: "utf8", maxBuffer: 64 * 1024 * 1024 });
|
|
116
|
-
if (exported.status !== 0)
|
|
117
|
-
return { ok: false, detail: (exported.stderr || exported.stdout || "llvm-cov failed").trim() };
|
|
108
|
+
const exported = exportLlvmProfile(binary, raws, merged);
|
|
109
|
+
if (!exported.ok)
|
|
110
|
+
return exported;
|
|
118
111
|
const map = llvmExportToMap(exported.stdout, pkg);
|
|
119
112
|
if (!map)
|
|
120
113
|
return { ok: false, detail: "llvm-cov export was not a coverage map" };
|
|
@@ -159,14 +152,78 @@ function llvmExportToMap(stdout, pkg) {
|
|
|
159
152
|
return null;
|
|
160
153
|
return { files };
|
|
161
154
|
}
|
|
162
|
-
|
|
155
|
+
/**
|
|
156
|
+
* clang and rustc write different profraw versions. Each pair is one LLVM install.
|
|
157
|
+
* The first pair that can merge the profile wins.
|
|
158
|
+
*/
|
|
159
|
+
export function llvmToolsPresent() {
|
|
160
|
+
return llvmToolPairs().length > 0;
|
|
161
|
+
}
|
|
162
|
+
export function exportLlvmProfile(binary, raws, merged) {
|
|
163
|
+
const pairs = llvmToolPairs();
|
|
164
|
+
if (pairs.length === 0)
|
|
165
|
+
return { ok: false, detail: "llvm tool missing (llvm-cov)" };
|
|
166
|
+
let detail = "llvm-profdata failed";
|
|
167
|
+
for (const { cov, prof } of pairs) {
|
|
168
|
+
const merge = spawnSync(prof, ["merge", "-sparse", "-o", merged, ...raws], { encoding: "utf8" });
|
|
169
|
+
if (merge.status !== 0) {
|
|
170
|
+
detail = (merge.stderr || merge.stdout || "llvm-profdata failed").trim();
|
|
171
|
+
continue;
|
|
172
|
+
}
|
|
173
|
+
const exported = spawnSync(cov, ["export", binary, `-instr-profile=${merged}`], { encoding: "utf8", maxBuffer: 64 * 1024 * 1024 });
|
|
174
|
+
if (exported.status !== 0) {
|
|
175
|
+
detail = (exported.stderr || exported.stdout || "llvm-cov failed").trim();
|
|
176
|
+
continue;
|
|
177
|
+
}
|
|
178
|
+
return { ok: true, stdout: exported.stdout };
|
|
179
|
+
}
|
|
180
|
+
return { ok: false, detail };
|
|
181
|
+
}
|
|
182
|
+
function llvmToolPairs() {
|
|
183
|
+
const covs = toolCandidates("llvm-cov");
|
|
184
|
+
const profs = toolCandidates("llvm-profdata");
|
|
185
|
+
const pairs = [];
|
|
186
|
+
const seen = new Set();
|
|
187
|
+
for (const cov of covs) {
|
|
188
|
+
const prof = profs.find((item) => path.dirname(item) === path.dirname(cov));
|
|
189
|
+
if (!prof)
|
|
190
|
+
continue;
|
|
191
|
+
const key = `${cov}\0${prof}`;
|
|
192
|
+
if (seen.has(key))
|
|
193
|
+
continue;
|
|
194
|
+
seen.add(key);
|
|
195
|
+
pairs.push({ cov, prof });
|
|
196
|
+
}
|
|
197
|
+
return pairs;
|
|
198
|
+
}
|
|
199
|
+
function toolCandidates(name) {
|
|
200
|
+
const out = [];
|
|
201
|
+
const add = (file) => {
|
|
202
|
+
if (!file || out.includes(file) || !existsSync(file))
|
|
203
|
+
return;
|
|
204
|
+
out.push(file);
|
|
205
|
+
};
|
|
163
206
|
const fromXcrun = spawnSync("xcrun", ["--find", name], { encoding: "utf8" });
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
return xcrunPath;
|
|
207
|
+
if (fromXcrun.status === 0)
|
|
208
|
+
add((fromXcrun.stdout ?? "").trim());
|
|
167
209
|
const which = spawnSync("sh", ["-c", `command -v ${name}`], { encoding: "utf8" });
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
210
|
+
if (which.status === 0)
|
|
211
|
+
add((which.stdout ?? "").trim());
|
|
212
|
+
const rustc = spawnSync("rustc", ["--print", "sysroot"], { encoding: "utf8" });
|
|
213
|
+
if (rustc.status === 0) {
|
|
214
|
+
const root = path.join(rustc.stdout.trim(), "lib", "rustlib");
|
|
215
|
+
if (existsSync(root)) {
|
|
216
|
+
for (const host of readdirSync(root))
|
|
217
|
+
add(path.join(root, host, "bin", name));
|
|
218
|
+
}
|
|
219
|
+
}
|
|
220
|
+
const usr = "/usr/lib";
|
|
221
|
+
if (existsSync(usr)) {
|
|
222
|
+
for (const dir of readdirSync(usr)) {
|
|
223
|
+
if (dir.startsWith("llvm-"))
|
|
224
|
+
add(path.join(usr, dir, "bin", name));
|
|
225
|
+
}
|
|
226
|
+
}
|
|
227
|
+
add(path.join("/usr/bin", name));
|
|
228
|
+
return out;
|
|
172
229
|
}
|
package/dist/mutate/generate.js
CHANGED
|
@@ -8,8 +8,9 @@ export function generateMutants(options) {
|
|
|
8
8
|
const started = Date.now();
|
|
9
9
|
const limit = options.maxMinutes;
|
|
10
10
|
const over = () => limit != null && Date.now() - started >= limit * 60_000;
|
|
11
|
+
const sourceDefault = options.workingTree ? "working-tree" : "commit";
|
|
11
12
|
if (over())
|
|
12
|
-
return { mutants: [], violations: [], filesVisited: 0, budgetHit: true };
|
|
13
|
+
return { mutants: [], violations: [], filesVisited: 0, budgetHit: true, source: sourceDefault };
|
|
13
14
|
const srcRoot = path.resolve(options.packageDir, options.srcDir);
|
|
14
15
|
const walked = walkSources(srcRoot, over);
|
|
15
16
|
const files = walked.files.filter((file) => !options.skipFiles.includes(path.basename(file)));
|
|
@@ -18,13 +19,18 @@ export function generateMutants(options) {
|
|
|
18
19
|
const chosen = [];
|
|
19
20
|
let filesVisited = 0;
|
|
20
21
|
let budgetHit = walked.stopped;
|
|
22
|
+
let source = sourceDefault;
|
|
21
23
|
for (const file of files) {
|
|
22
24
|
if (over()) {
|
|
23
25
|
budgetHit = true;
|
|
24
26
|
break;
|
|
25
27
|
}
|
|
26
28
|
filesVisited += 1;
|
|
27
|
-
const
|
|
29
|
+
const loaded = fileText(options.packageDir, file, options);
|
|
30
|
+
if ("error" in loaded)
|
|
31
|
+
return { mutants: [], violations: [], filesVisited, budgetHit, source: "commit", error: loaded.error };
|
|
32
|
+
source = loaded.source;
|
|
33
|
+
const text = loaded.text;
|
|
28
34
|
const rel = posix(path.relative(options.packageDir, file));
|
|
29
35
|
const found = findInSource(rel, text);
|
|
30
36
|
violations.push(...found.violations);
|
|
@@ -34,7 +40,7 @@ export function generateMutants(options) {
|
|
|
34
40
|
chosen.push({ rel, text, point });
|
|
35
41
|
}
|
|
36
42
|
if (violations.length > 0)
|
|
37
|
-
return { mutants: [], violations, filesVisited, budgetHit };
|
|
43
|
+
return { mutants: [], violations, filesVisited, budgetHit, source };
|
|
38
44
|
const capped = options.maxMutants === null ? chosen : chosen.slice(0, options.maxMutants);
|
|
39
45
|
const packageReal = real(options.packageDir);
|
|
40
46
|
const patchRoot = gitRoot(packageReal);
|
|
@@ -55,7 +61,22 @@ export function generateMutants(options) {
|
|
|
55
61
|
mutants.push({ id, file: item.rel, line: item.point.line, op: item.point.op, patch });
|
|
56
62
|
}
|
|
57
63
|
writeFileSync(path.join(mutantsDir, "mutants.json"), `${JSON.stringify({ schemaVersion: 1, seed: options.seed, mutants }, null, 2)}\n`);
|
|
58
|
-
return { mutants, violations, filesVisited, budgetHit };
|
|
64
|
+
return { mutants, violations, filesVisited, budgetHit, source };
|
|
65
|
+
}
|
|
66
|
+
function fileText(packageDir, absFile, options) {
|
|
67
|
+
if (options.workingTree)
|
|
68
|
+
return { text: readFileSync(absFile, "utf8"), source: "working-tree" };
|
|
69
|
+
const root = gitRoot(packageDir);
|
|
70
|
+
if (!root)
|
|
71
|
+
return { text: readFileSync(absFile, "utf8"), source: "working-tree" };
|
|
72
|
+
const rel = posix(path.relative(root, real(absFile)));
|
|
73
|
+
const commit = options.commit && options.commit.length > 0 ? options.commit : "HEAD";
|
|
74
|
+
const shown = spawnSync("git", ["-C", root, "show", `${commit}:${rel}`], { encoding: "utf8" });
|
|
75
|
+
if (shown.status !== 0) {
|
|
76
|
+
const shownRel = posix(path.relative(packageDir, absFile));
|
|
77
|
+
return { error: `${shownRel} is not in ${commit}. Pass --working-tree to read the checkout.` };
|
|
78
|
+
}
|
|
79
|
+
return { text: shown.stdout, source: "commit" };
|
|
59
80
|
}
|
|
60
81
|
function walkSources(dir, over) {
|
|
61
82
|
if (!statExists(dir))
|
package/dist/mutate/run.js
CHANGED
|
@@ -41,7 +41,7 @@ export async function runMutants(options) {
|
|
|
41
41
|
return failReport(options.outDir, "", options.commit, "that commit does not resolve");
|
|
42
42
|
const commit = sha.stdout.trim();
|
|
43
43
|
mkdirSync(options.outDir, { recursive: true });
|
|
44
|
-
const run = loadRun(options
|
|
44
|
+
const run = loadRun(options, commit);
|
|
45
45
|
if ("error" in run)
|
|
46
46
|
return failReport(options.outDir, run.id, commit, run.error);
|
|
47
47
|
const listed = listPatches(options.patchDirs);
|
|
@@ -211,7 +211,7 @@ async function prepare(pkg, options, progress) {
|
|
|
211
211
|
if (baseline.compileToken)
|
|
212
212
|
return { ok: false, error: `baseline build failed (${baseline.compileToken})` };
|
|
213
213
|
if (!baseline.report)
|
|
214
|
-
return { ok: false, error: baselineReportError(baseline.detail) };
|
|
214
|
+
return { ok: false, error: baselineReportError(baseline.detail, baseline.command || spec.command) };
|
|
215
215
|
baselineFailed = baseline.report.failed.map((item) => item.name);
|
|
216
216
|
if (collect && !existsSync(mapPath)) {
|
|
217
217
|
const binary = binaryFromCommand(baseline.command || spec.command, pkg);
|
|
@@ -224,6 +224,10 @@ async function prepare(pkg, options, progress) {
|
|
|
224
224
|
progress(`suite report tests=${baseline.report.tests}`);
|
|
225
225
|
progress(baselineFailed.length > 0 ? `baseline failing ${baselineFailed.length}` : "baseline pass");
|
|
226
226
|
if (baselineFailed.length > 0) {
|
|
227
|
+
const tool = missingToolFrom(baseline.detail, baseline.command || spec.command);
|
|
228
|
+
const unnamed = baselineFailed.every((name) => name === "command" || name === "::command");
|
|
229
|
+
if (tool && unnamed)
|
|
230
|
+
return { ok: false, error: `missing tool: ${tool}. baseline suite did not return a test report` };
|
|
227
231
|
return { ok: false, error: "baseline already failing", baselineFailed, suiteCommand: spec.command };
|
|
228
232
|
}
|
|
229
233
|
if (collect && !existsSync(mapPath) && languageMapKind(spec.kind)) {
|
|
@@ -250,12 +254,13 @@ async function runOne(worktree, pkg, patchPath, options, commit, runId, prep, la
|
|
|
250
254
|
const raw = readFileSync(patchPath, "utf8");
|
|
251
255
|
const files = filesInDiff(raw);
|
|
252
256
|
const edited = changedLines(raw);
|
|
257
|
+
const located = edited.length > 0 ? edited : files;
|
|
253
258
|
const mutantHash = sha256(raw);
|
|
254
259
|
const base = () => ({
|
|
255
260
|
id,
|
|
256
261
|
outcome: "error",
|
|
257
262
|
direction: null,
|
|
258
|
-
files,
|
|
263
|
+
files: located,
|
|
259
264
|
killedBy: [],
|
|
260
265
|
cause: null,
|
|
261
266
|
next: "",
|
|
@@ -319,8 +324,16 @@ async function runOne(worktree, pkg, patchPath, options, commit, runId, prep, la
|
|
|
319
324
|
// A compile error can leave the previous run's report on disk. That report is not this mutant.
|
|
320
325
|
if (main.compileToken)
|
|
321
326
|
return finishMutant(killed(base, direction, [main.compileToken]), pkg, commands.join(" | "), false, []);
|
|
322
|
-
if (!main.report)
|
|
323
|
-
|
|
327
|
+
if (!main.report) {
|
|
328
|
+
const detail = main.detail ? clip(main.detail) : "";
|
|
329
|
+
const tool = missingToolFrom(detail, main.command);
|
|
330
|
+
const error = tool
|
|
331
|
+
? `missing tool: ${tool}`
|
|
332
|
+
: detail
|
|
333
|
+
? `test reporter returned nothing (${detail})`
|
|
334
|
+
: "test reporter returned nothing";
|
|
335
|
+
return { ...base(), direction, error, command: commands.join(" | ") };
|
|
336
|
+
}
|
|
324
337
|
const whole = main.wholeSuite || forceWhole;
|
|
325
338
|
const chosen = main.selectedTests;
|
|
326
339
|
if (main.report.failed.length === 0)
|
|
@@ -397,19 +410,22 @@ async function rerunGroup(pkg, options, file, failures, counts, pattern, prep, t
|
|
|
397
410
|
if (rerun.command)
|
|
398
411
|
commands.push(rerun.command);
|
|
399
412
|
if (pattern === null && rerun.timedOut) {
|
|
400
|
-
for (const
|
|
401
|
-
bump(counts,
|
|
413
|
+
for (const id of new Set(failures.map((item) => failureId(pkg, item))))
|
|
414
|
+
bump(counts, id);
|
|
402
415
|
continue;
|
|
403
416
|
}
|
|
404
417
|
if (!rerun.report)
|
|
405
418
|
continue;
|
|
419
|
+
const seen = new Set();
|
|
406
420
|
for (const hit of rerun.report.failed) {
|
|
407
421
|
if (pattern === null && !hit.fileTimeout)
|
|
408
422
|
continue;
|
|
409
423
|
const id = failureId(pkg, hit);
|
|
410
424
|
if (failures.some((item) => failureId(pkg, item) === id))
|
|
411
|
-
|
|
425
|
+
seen.add(id);
|
|
412
426
|
}
|
|
427
|
+
for (const id of seen)
|
|
428
|
+
bump(counts, id);
|
|
413
429
|
}
|
|
414
430
|
return wholeSuite;
|
|
415
431
|
}
|
|
@@ -708,8 +724,8 @@ function loadMeta(dirs) {
|
|
|
708
724
|
}
|
|
709
725
|
return meta;
|
|
710
726
|
}
|
|
711
|
-
function loadRun(
|
|
712
|
-
const file = path.join(outDir, "run.json");
|
|
727
|
+
function loadRun(options, commit) {
|
|
728
|
+
const file = path.join(options.outDir, "run.json");
|
|
713
729
|
if (existsSync(file)) {
|
|
714
730
|
const stored = JSON.parse(readFileSync(file, "utf8"));
|
|
715
731
|
if (stored.commit !== commit)
|
|
@@ -717,7 +733,24 @@ function loadRun(outDir, commit, agent) {
|
|
|
717
733
|
return { id: stored.id };
|
|
718
734
|
}
|
|
719
735
|
const id = randomBytes(4).toString("hex");
|
|
720
|
-
|
|
736
|
+
const record = {
|
|
737
|
+
id,
|
|
738
|
+
commit,
|
|
739
|
+
agent: options.agent,
|
|
740
|
+
started: new Date().toISOString(),
|
|
741
|
+
package: options.packageDir,
|
|
742
|
+
patches: options.patchDirs,
|
|
743
|
+
out: options.outDir,
|
|
744
|
+
repo: options.repoDir,
|
|
745
|
+
workers: options.workers,
|
|
746
|
+
confirm: options.confirm,
|
|
747
|
+
testsDir: options.testsDir,
|
|
748
|
+
suiteTimeoutMs: options.suiteTimeoutMs,
|
|
749
|
+
testTimeoutMs: options.testTimeoutMs,
|
|
750
|
+
direction: options.direction,
|
|
751
|
+
suiteCommand: options.suiteCommand ?? null,
|
|
752
|
+
};
|
|
753
|
+
writeFileSync(file, `${JSON.stringify(record, null, 2)}\n`);
|
|
721
754
|
return { id };
|
|
722
755
|
}
|
|
723
756
|
async function createWorktrees(repo, packageDir, runId, commit, workers, outDir) {
|
|
@@ -725,6 +758,10 @@ async function createWorktrees(repo, packageDir, runId, commit, workers, outDir)
|
|
|
725
758
|
const dirs = [];
|
|
726
759
|
for (let index = 0; index < workers; index++) {
|
|
727
760
|
const dir = path.join(tmpdir(), `probatio-${runId}-${index}`);
|
|
761
|
+
if (!probatioWorktreeDir(dir)) {
|
|
762
|
+
removeWorktrees(repo, dirs, outDir);
|
|
763
|
+
return { error: "worktree path is outside the probatio temp prefix" };
|
|
764
|
+
}
|
|
728
765
|
rmSync(dir, { recursive: true, force: true });
|
|
729
766
|
const added = git(repo, ["worktree", "add", "--detach", dir, commit]);
|
|
730
767
|
if (added.status !== 0) {
|
|
@@ -750,15 +787,42 @@ async function createWorktrees(repo, packageDir, runId, commit, workers, outDir)
|
|
|
750
787
|
function hidePaths(dirs, repo, packageDir, hide) {
|
|
751
788
|
for (const dir of dirs) {
|
|
752
789
|
const pkg = packageIn(dir, repo, packageDir);
|
|
790
|
+
const root = path.resolve(pkg);
|
|
753
791
|
for (const rel of hide) {
|
|
754
|
-
if (!rel || path.isAbsolute(rel))
|
|
792
|
+
if (!rel || path.isAbsolute(rel) || rel.split(/[\\/]/).includes(".."))
|
|
793
|
+
continue;
|
|
794
|
+
const target = path.resolve(root, rel);
|
|
795
|
+
if (target !== root && !target.startsWith(root + path.sep))
|
|
755
796
|
continue;
|
|
756
|
-
rmSync(
|
|
797
|
+
rmSync(target, { force: true });
|
|
757
798
|
}
|
|
758
799
|
}
|
|
759
800
|
}
|
|
801
|
+
/** A worktree Probatio created: one directory under the OS temp dir, named probatio- plus hex and an index. */
|
|
802
|
+
export function probatioWorktreeDir(dir) {
|
|
803
|
+
let resolved = path.resolve(dir);
|
|
804
|
+
try {
|
|
805
|
+
resolved = realpathSync(dir);
|
|
806
|
+
}
|
|
807
|
+
catch {
|
|
808
|
+
// The directory is created on the next line. The unresolved path is still checked.
|
|
809
|
+
}
|
|
810
|
+
const roots = new Set([path.resolve(tmpdir())]);
|
|
811
|
+
try {
|
|
812
|
+
roots.add(realpathSync(tmpdir()));
|
|
813
|
+
}
|
|
814
|
+
catch {
|
|
815
|
+
// tmpdir() itself is enough when realpath is unavailable.
|
|
816
|
+
}
|
|
817
|
+
const inside = [...roots].some((root) => resolved === root || resolved.startsWith(root + path.sep));
|
|
818
|
+
if (!inside)
|
|
819
|
+
return false;
|
|
820
|
+
return /^probatio-[0-9a-f]+-\d+$/.test(path.basename(resolved));
|
|
821
|
+
}
|
|
760
822
|
function removeWorktrees(repo, dirs, outDir) {
|
|
761
823
|
for (const dir of dirs) {
|
|
824
|
+
if (!probatioWorktreeDir(dir))
|
|
825
|
+
continue;
|
|
762
826
|
git(repo, ["worktree", "remove", "--force", dir]);
|
|
763
827
|
rmSync(dir, { recursive: true, force: true });
|
|
764
828
|
}
|
|
@@ -899,6 +963,9 @@ function reportOf(results, context) {
|
|
|
899
963
|
notes.push(resumeNext(context.notStarted));
|
|
900
964
|
if (results.some((result) => result.wholeSuite))
|
|
901
965
|
notes.push(WHOLE_SUITE_NEXT);
|
|
966
|
+
const broken = results.find((result) => (result.outcome === "error" || result.outcome === "build-failed") && result.error);
|
|
967
|
+
if (broken?.error)
|
|
968
|
+
notes.push(`${broken.id}: ${broken.error}`);
|
|
902
969
|
if (notes.length === 0)
|
|
903
970
|
notes.push("No survivor in this batch.");
|
|
904
971
|
return {
|
|
@@ -1156,13 +1223,27 @@ function testArg(pkg, file, testsDir) {
|
|
|
1156
1223
|
function escapeRegExp(value) {
|
|
1157
1224
|
return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
|
1158
1225
|
}
|
|
1159
|
-
function
|
|
1226
|
+
function missingToolFrom(detail, command) {
|
|
1227
|
+
const spawnMiss = /spawn ([^\s]+) ENOENT/.exec(detail);
|
|
1228
|
+
if (spawnMiss)
|
|
1229
|
+
return path.basename(spawnMiss[1]);
|
|
1230
|
+
const shellMiss = /(?:^|\s)([A-Za-z0-9_./+-]+): (?:command not found|not found)\b/.exec(detail);
|
|
1231
|
+
if (shellMiss)
|
|
1232
|
+
return path.basename(shellMiss[1]);
|
|
1233
|
+
if (/No such file or directory|ENOENT/.test(detail)) {
|
|
1234
|
+
const bin = command.trim().split(/\s+/)[0];
|
|
1235
|
+
if (bin)
|
|
1236
|
+
return path.basename(bin);
|
|
1237
|
+
}
|
|
1238
|
+
return null;
|
|
1239
|
+
}
|
|
1240
|
+
function baselineReportError(detail, command = "") {
|
|
1241
|
+
const tool = missingToolFrom(detail, command);
|
|
1160
1242
|
const line = detail.trim();
|
|
1161
1243
|
// The first line of a Python or Node crash is a stack fragment. The summary
|
|
1162
1244
|
// states the limit and does not carry that fragment.
|
|
1163
|
-
|
|
1164
|
-
|
|
1165
|
-
return `baseline suite did not return a test report (${line})`;
|
|
1245
|
+
const base = !line || isStackFragment(line) ? "baseline suite did not return a test report" : `baseline suite did not return a test report (${line})`;
|
|
1246
|
+
return tool ? `missing tool: ${tool}. ${base}` : base;
|
|
1166
1247
|
}
|
|
1167
1248
|
function isStackFragment(line) {
|
|
1168
1249
|
return line.startsWith("Traceback") || /^\s+at \S/.test(line) || /File ".+", line \d+/.test(line);
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
import { spawnSync } from "node:child_process";
|
|
2
2
|
import { existsSync, mkdirSync, realpathSync, writeFileSync } from "node:fs";
|
|
3
3
|
import path from "node:path";
|
|
4
|
+
import { exportLlvmProfile, llvmToolsPresent } from "./coverage-map.js";
|
|
4
5
|
/**
|
|
5
6
|
* One instrumented test binary, then one profile per test name.
|
|
6
7
|
* A single cargo test profile does not say which test hit the line.
|
|
@@ -9,9 +10,7 @@ export function writeRustCoverage(pkg, testNames, dest) {
|
|
|
9
10
|
const names = [...new Set(testNames.map((name) => name.trim()).filter((name) => name.length > 0 && !/\s/.test(name)))];
|
|
10
11
|
if (names.length === 0)
|
|
11
12
|
return { ok: false, detail: "rust coverage: no test names" };
|
|
12
|
-
|
|
13
|
-
const llvmProf = findTool("llvm-profdata");
|
|
14
|
-
if (!llvmCov || !llvmProf)
|
|
13
|
+
if (!llvmToolsPresent())
|
|
15
14
|
return { ok: false, detail: "rust coverage: llvm tool missing" };
|
|
16
15
|
const env = { ...process.env, RUSTFLAGS: "-C instrument-coverage", CARGO_INCREMENTAL: "0" };
|
|
17
16
|
const built = spawnSync("cargo", ["test", "--no-run", "--message-format=json"], {
|
|
@@ -43,11 +42,8 @@ export function writeRustCoverage(pkg, testNames, dest) {
|
|
|
43
42
|
if (!new RegExp(`^test ${escapeRegExp(name)} \\.\\.\\. ok`, "m").test(`${ran.stdout}\n${ran.stderr}`))
|
|
44
43
|
continue;
|
|
45
44
|
const merged = path.join(work, `${safe(name)}-${safe(path.basename(exe))}.profdata`);
|
|
46
|
-
const
|
|
47
|
-
if (
|
|
48
|
-
continue;
|
|
49
|
-
const exported = spawnSync(llvmCov, ["export", exe, `-instr-profile=${merged}`], { encoding: "utf8", maxBuffer: 64 * 1024 * 1024 });
|
|
50
|
-
if (exported.status !== 0)
|
|
45
|
+
const exported = exportLlvmProfile(exe, [raw], merged);
|
|
46
|
+
if (!exported.ok)
|
|
51
47
|
continue;
|
|
52
48
|
for (const hit of llvmHits(exported.stdout, pkg))
|
|
53
49
|
addHit(files, hit.file, hit.line, name);
|
|
@@ -136,17 +132,6 @@ function addHit(files, file, line, test) {
|
|
|
136
132
|
lines[key] = names;
|
|
137
133
|
files[file] = lines;
|
|
138
134
|
}
|
|
139
|
-
function findTool(name) {
|
|
140
|
-
const fromXcrun = spawnSync("xcrun", ["--find", name], { encoding: "utf8" });
|
|
141
|
-
const xcrunPath = fromXcrun.stdout?.trim() ?? "";
|
|
142
|
-
if (fromXcrun.status === 0 && xcrunPath && existsSync(xcrunPath))
|
|
143
|
-
return xcrunPath;
|
|
144
|
-
const which = spawnSync("sh", ["-c", `command -v ${name}`], { encoding: "utf8" });
|
|
145
|
-
const found = which.stdout?.trim() ?? "";
|
|
146
|
-
if (which.status === 0 && found)
|
|
147
|
-
return found;
|
|
148
|
-
return null;
|
|
149
|
-
}
|
|
150
135
|
function safe(value) {
|
|
151
136
|
return value.replace(/[^A-Za-z0-9._-]+/g, "_");
|
|
152
137
|
}
|
package/dist/mutate/sealed.js
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
|
-
import { readFileSync } from "node:fs";
|
|
1
|
+
import { readdirSync, readFileSync, statSync } from "node:fs";
|
|
2
|
+
import path from "node:path";
|
|
2
3
|
/** The label stays in its own file. The run report must not repeat these three values. */
|
|
3
4
|
export function readSealedLabel(file) {
|
|
4
5
|
let text = "";
|
|
@@ -18,6 +19,44 @@ export function readSealedLabel(file) {
|
|
|
18
19
|
export function labelLeaked(body, label) {
|
|
19
20
|
return body.includes(label.bugId) || body.includes(label.issueId) || body.includes(label.expectedFail);
|
|
20
21
|
}
|
|
22
|
+
/** The envelope can be clean while a result file on disk still repeats the label. */
|
|
23
|
+
export function sealedTreeLeaked(dir, label) {
|
|
24
|
+
const visit = (current) => {
|
|
25
|
+
let names;
|
|
26
|
+
try {
|
|
27
|
+
names = readdirSync(current);
|
|
28
|
+
}
|
|
29
|
+
catch {
|
|
30
|
+
return false;
|
|
31
|
+
}
|
|
32
|
+
for (const name of names) {
|
|
33
|
+
const full = path.join(current, name);
|
|
34
|
+
let info;
|
|
35
|
+
try {
|
|
36
|
+
info = statSync(full);
|
|
37
|
+
}
|
|
38
|
+
catch {
|
|
39
|
+
continue;
|
|
40
|
+
}
|
|
41
|
+
if (info.isDirectory()) {
|
|
42
|
+
if (visit(full))
|
|
43
|
+
return true;
|
|
44
|
+
continue;
|
|
45
|
+
}
|
|
46
|
+
if (!info.isFile() || info.size > 2_000_000)
|
|
47
|
+
continue;
|
|
48
|
+
try {
|
|
49
|
+
if (labelLeaked(readFileSync(full, "utf8"), label))
|
|
50
|
+
return true;
|
|
51
|
+
}
|
|
52
|
+
catch {
|
|
53
|
+
// A binary artifact is not a report.
|
|
54
|
+
}
|
|
55
|
+
}
|
|
56
|
+
return false;
|
|
57
|
+
};
|
|
58
|
+
return visit(dir);
|
|
59
|
+
}
|
|
21
60
|
export const LABEL_LEAK_SUMMARY = "The label was copied into the report.";
|
|
22
61
|
function field(text, key) {
|
|
23
62
|
const match = text.match(new RegExp(`^${key}=(.*)$`, "m"));
|
package/dist/mutate/suites.js
CHANGED
|
@@ -386,8 +386,8 @@ async function launch(pkg, spec, files, pattern, names, timeoutMs, env, hooks) {
|
|
|
386
386
|
if (spec.kind === "swift")
|
|
387
387
|
return tag(runSwift(pkg, timeoutMs, env), spec.command, wantsNames, selection);
|
|
388
388
|
if (spec.kind === "maven") {
|
|
389
|
-
const tests =
|
|
390
|
-
const args = ["-B", "test", ...(tests.length > 0 ? [`-Dtest=${tests
|
|
389
|
+
const tests = surefireArg(uniqueSurefire(selection));
|
|
390
|
+
const args = ["-B", "test", ...(tests.length > 0 ? [`-Dtest=${tests}`] : [])];
|
|
391
391
|
const whole = wantsNames && tests.length === 0;
|
|
392
392
|
// Surefire leaves the previous run's XML in place. A confirm that did not
|
|
393
393
|
// re-execute a class would otherwise read that class as failed again.
|
|
@@ -468,12 +468,47 @@ function goRunFilter(names, pattern) {
|
|
|
468
468
|
return null;
|
|
469
469
|
return `^(${names.map(escapeRegExp).join("|")})$`;
|
|
470
470
|
}
|
|
471
|
+
function uniqueSurefire(names) {
|
|
472
|
+
const out = [];
|
|
473
|
+
const seen = new Set();
|
|
474
|
+
for (const name of names) {
|
|
475
|
+
const selector = surefireTest(name);
|
|
476
|
+
if (!selector || seen.has(selector))
|
|
477
|
+
continue;
|
|
478
|
+
seen.add(selector);
|
|
479
|
+
out.push(selector);
|
|
480
|
+
}
|
|
481
|
+
return out;
|
|
482
|
+
}
|
|
483
|
+
/** Surefire 2.12 splits `-Dtest` on commas and then reads `#method+method`. `Class#a,Class#b` runs only `a`. */
|
|
484
|
+
export function surefireArg(tests) {
|
|
485
|
+
const groups = new Map();
|
|
486
|
+
for (const test of tests) {
|
|
487
|
+
const hash = test.indexOf("#");
|
|
488
|
+
const klass = hash === -1 ? test : test.slice(0, hash);
|
|
489
|
+
const method = hash === -1 ? "" : test.slice(hash + 1);
|
|
490
|
+
const methods = groups.get(klass);
|
|
491
|
+
if (methods) {
|
|
492
|
+
if (method)
|
|
493
|
+
methods.push(method);
|
|
494
|
+
}
|
|
495
|
+
else {
|
|
496
|
+
groups.set(klass, method ? [method] : []);
|
|
497
|
+
}
|
|
498
|
+
}
|
|
499
|
+
return [...groups.entries()]
|
|
500
|
+
.map(([klass, methods]) => (methods.length > 0 ? `${klass}#${methods.join("+")}` : klass))
|
|
501
|
+
.join(",");
|
|
502
|
+
}
|
|
471
503
|
function surefireTest(name) {
|
|
472
504
|
const leaf = name.split("::").pop() ?? name;
|
|
473
505
|
const dot = leaf.lastIndexOf(".");
|
|
474
506
|
const body = dot <= 0 ? leaf : `${leaf.slice(0, dot)}#${leaf.slice(dot + 1)}`;
|
|
475
|
-
// testShut[0]
|
|
476
|
-
|
|
507
|
+
// testShut[0] matches nothing. A bare testShut matches on Surefire 3.6 and runs
|
|
508
|
+
// zero tests on 3.5. testShut* matches every invocation on both. method(Parser)[1] keeps the types.
|
|
509
|
+
const invocation = /\[[^\]]*\]$/.test(body);
|
|
510
|
+
const method = body.replace(/\[[^\]]*\]$/, "");
|
|
511
|
+
return invocation ? `${method}*` : method;
|
|
477
512
|
}
|
|
478
513
|
function escapeRegExp(value) {
|
|
479
514
|
return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
|
@@ -764,7 +799,8 @@ function scriptInvocation(pkg) {
|
|
|
764
799
|
if (!suiteNeedsTty(pkg))
|
|
765
800
|
return { bin: "bash", args: ["run_tests.sh"], command: "bash run_tests.sh" };
|
|
766
801
|
if (gnuScript()) {
|
|
767
|
-
|
|
802
|
+
// -e returns the child status. Without it, util-linux script exits 0 and a failing suite looks green.
|
|
803
|
+
const args = ["-q", "-e", "-c", "bash run_tests.sh", "/dev/null"];
|
|
768
804
|
return { bin: "script", args, command: shown("script", args) };
|
|
769
805
|
}
|
|
770
806
|
const args = ["-q", "/dev/null", "bash", "run_tests.sh"];
|
|
@@ -1403,6 +1439,11 @@ function mavenReport(pkg, text, code) {
|
|
|
1403
1439
|
}
|
|
1404
1440
|
return parseSurefire(pkg) ?? parseMavenText(text);
|
|
1405
1441
|
}
|
|
1442
|
+
/** An attribute on the opening tag. `classname` must not be read as `name`. */
|
|
1443
|
+
export function surefireAttr(fragment, key) {
|
|
1444
|
+
const open = fragment.slice(0, fragment.indexOf(">") === -1 ? fragment.length : fragment.indexOf(">"));
|
|
1445
|
+
return new RegExp(`(?:^|\\s)${key}="([^"]*)"`).exec(open)?.[1] ?? "";
|
|
1446
|
+
}
|
|
1406
1447
|
function parseSurefire(pkg) {
|
|
1407
1448
|
const dir = path.join(pkg, "target", "surefire-reports");
|
|
1408
1449
|
if (!existsSync(dir))
|
|
@@ -1414,11 +1455,11 @@ function parseSurefire(pkg) {
|
|
|
1414
1455
|
continue;
|
|
1415
1456
|
const xml = readFileSync(path.join(dir, name), "utf8");
|
|
1416
1457
|
for (const part of xml.split("<testcase ").slice(1)) {
|
|
1417
|
-
const test =
|
|
1418
|
-
const klass =
|
|
1458
|
+
const test = surefireAttr(part, "name");
|
|
1459
|
+
const klass = surefireAttr(part, "classname");
|
|
1419
1460
|
if (!test)
|
|
1420
1461
|
continue;
|
|
1421
|
-
const id = `${klass}.${test}
|
|
1462
|
+
const id = klass ? `${klass}.${test}` : test;
|
|
1422
1463
|
names.push(id);
|
|
1423
1464
|
if (part.includes("<failure") || part.includes("<error"))
|
|
1424
1465
|
failed.push({ name: id, file: klass, line: 0 });
|
package/dist/mutate/tally.js
CHANGED
|
@@ -105,7 +105,7 @@ export function tallyRun(outDir) {
|
|
|
105
105
|
command: "mutate.tally",
|
|
106
106
|
summary,
|
|
107
107
|
next,
|
|
108
|
-
nextCall:
|
|
108
|
+
nextCall: resumeArgv(outDir, keep),
|
|
109
109
|
keep,
|
|
110
110
|
drop,
|
|
111
111
|
gaps,
|
|
@@ -117,6 +117,43 @@ export function tallyRun(outDir) {
|
|
|
117
117
|
writeFileSync(fullPath, `${JSON.stringify(complete, null, 2)}\n`);
|
|
118
118
|
return { ...complete, gaps: shown, rest: gaps.length - shown.length };
|
|
119
119
|
}
|
|
120
|
+
function resumeArgv(outDir, keep) {
|
|
121
|
+
if (keep.length === 0)
|
|
122
|
+
return null;
|
|
123
|
+
let stored = {};
|
|
124
|
+
try {
|
|
125
|
+
stored = JSON.parse(readFileSync(path.join(outDir, "run.json"), "utf8"));
|
|
126
|
+
}
|
|
127
|
+
catch {
|
|
128
|
+
stored = {};
|
|
129
|
+
}
|
|
130
|
+
const argv = ["mutate", "run"];
|
|
131
|
+
if (stored.package)
|
|
132
|
+
argv.push("--package", stored.package);
|
|
133
|
+
for (const dir of stored.patches ?? [])
|
|
134
|
+
argv.push("--patches", dir);
|
|
135
|
+
argv.push("--out", stored.out || outDir);
|
|
136
|
+
if (stored.repo)
|
|
137
|
+
argv.push("--repo", stored.repo);
|
|
138
|
+
if (stored.commit)
|
|
139
|
+
argv.push("--commit", stored.commit);
|
|
140
|
+
if (typeof stored.workers === "number")
|
|
141
|
+
argv.push("--workers", String(stored.workers));
|
|
142
|
+
if (stored.confirm === false)
|
|
143
|
+
argv.push("--no-confirm");
|
|
144
|
+
if (stored.testsDir && stored.testsDir !== "tests")
|
|
145
|
+
argv.push("--tests-dir", stored.testsDir);
|
|
146
|
+
if (typeof stored.suiteTimeoutMs === "number")
|
|
147
|
+
argv.push("--suite-timeout-ms", String(stored.suiteTimeoutMs));
|
|
148
|
+
if (typeof stored.testTimeoutMs === "number")
|
|
149
|
+
argv.push("--test-timeout-ms", String(stored.testTimeoutMs));
|
|
150
|
+
if (stored.direction && stored.direction !== "auto")
|
|
151
|
+
argv.push("--direction", stored.direction);
|
|
152
|
+
if (stored.suiteCommand)
|
|
153
|
+
argv.push("--suite-command", stored.suiteCommand);
|
|
154
|
+
argv.push(...keep.flatMap((id) => ["--only-test", id]));
|
|
155
|
+
return { argv };
|
|
156
|
+
}
|
|
120
157
|
function readResults(dir) {
|
|
121
158
|
const out = [];
|
|
122
159
|
for (const name of readdirSync(dir)) {
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
/** `--experimental-strip-types` needs this. Older Node is refused before a command runs. */
|
|
2
|
+
export const MIN_NODE = "22.6";
|
|
3
|
+
export function nodeTooOld(version) {
|
|
4
|
+
const match = /^(\d+)\.(\d+)/.exec(version);
|
|
5
|
+
const required = `Node ${version} is too old. Probatio needs Node ${MIN_NODE} or newer.`;
|
|
6
|
+
if (!match)
|
|
7
|
+
return required;
|
|
8
|
+
const major = Number(match[1]);
|
|
9
|
+
const minor = Number(match[2]);
|
|
10
|
+
if (major > 22 || (major === 22 && minor >= 6))
|
|
11
|
+
return null;
|
|
12
|
+
return required;
|
|
13
|
+
}
|
package/dist/swarm/queue.js
CHANGED
|
@@ -1,13 +1,19 @@
|
|
|
1
1
|
import { existsSync, mkdirSync, readdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
|
|
2
2
|
import path from "node:path";
|
|
3
3
|
import { loadSealed } from "../memory/state.js";
|
|
4
|
+
/** One path segment. Rejects slashes and `..` so a claim cannot leave the queue directory. */
|
|
5
|
+
export function safeFilePart(value) {
|
|
6
|
+
if (!/^[A-Za-z0-9._-]+$/.test(value) || value === "." || value === "..")
|
|
7
|
+
return null;
|
|
8
|
+
return value;
|
|
9
|
+
}
|
|
4
10
|
export function seedQueue(root, ids) {
|
|
5
11
|
const sealed = new Set(loadSealed(root));
|
|
6
12
|
const queue = path.join(root, "queue");
|
|
7
13
|
mkdirSync(queue, { recursive: true });
|
|
8
14
|
const written = [];
|
|
9
15
|
for (const id of [...ids].sort()) {
|
|
10
|
-
if (sealed.has(id))
|
|
16
|
+
if (sealed.has(id) || !safeFilePart(id))
|
|
11
17
|
continue;
|
|
12
18
|
const dest = path.join(queue, `${id}.json`);
|
|
13
19
|
if (existsSync(dest))
|
|
@@ -19,23 +25,23 @@ export function seedQueue(root, ids) {
|
|
|
19
25
|
}
|
|
20
26
|
/** Claim by atomic rename. A second claim fails because the queue file is already gone. */
|
|
21
27
|
export function claimItem(root, id, agent, now, leaseMs) {
|
|
28
|
+
if (!safeFilePart(id) || !safeFilePart(agent))
|
|
29
|
+
return { ok: false, summary: "agent and id must be a single file name.", file: null };
|
|
22
30
|
const source = path.join(root, "queue", `${id}.json`);
|
|
23
31
|
const claimed = path.join(root, "claimed");
|
|
24
32
|
const dest = path.join(claimed, `${agent}-${id}.json`);
|
|
25
33
|
if (!existsSync(source))
|
|
26
34
|
return { ok: false, summary: `${id} is not in the queue.`, file: null };
|
|
27
35
|
mkdirSync(claimed, { recursive: true });
|
|
28
|
-
|
|
29
|
-
body.id = id;
|
|
30
|
-
body.agent = agent;
|
|
31
|
-
body.leasedUntil = now + leaseMs;
|
|
32
|
-
writeFileSync(source, `${JSON.stringify(body)}\n`);
|
|
36
|
+
// Rename first. Writing the body before the rename lets the other claim overwrite the winner.
|
|
33
37
|
try {
|
|
34
38
|
renameSync(source, dest);
|
|
35
39
|
}
|
|
36
40
|
catch {
|
|
37
41
|
return { ok: false, summary: `${id} was claimed by someone else.`, file: null };
|
|
38
42
|
}
|
|
43
|
+
const body = { id, agent, leasedUntil: now + leaseMs };
|
|
44
|
+
writeFileSync(dest, `${JSON.stringify(body)}\n`);
|
|
39
45
|
return { ok: true, summary: `${agent} claimed ${id} until ${body.leasedUntil}.`, file: dest };
|
|
40
46
|
}
|
|
41
47
|
/** Move claims whose lease has expired back to the queue. */
|
|
@@ -53,7 +59,7 @@ export function reapClaims(root, now) {
|
|
|
53
59
|
const body = JSON.parse(readFileSync(file, "utf8"));
|
|
54
60
|
if (typeof body.leasedUntil !== "number" || body.leasedUntil > now)
|
|
55
61
|
continue;
|
|
56
|
-
if (!body.id)
|
|
62
|
+
if (!body.id || !safeFilePart(body.id))
|
|
57
63
|
continue;
|
|
58
64
|
renameSync(file, path.join(queue, `${body.id}.json`));
|
|
59
65
|
returned.push(body.id);
|
package/dist/verify/change.js
CHANGED
|
@@ -23,6 +23,11 @@ export async function verifyChange(options) {
|
|
|
23
23
|
const ranges = parseUnified(diff.stdout);
|
|
24
24
|
const names = git(repo, ["diff", "--name-only", "--no-renames", options.base, options.commit, ...scope]);
|
|
25
25
|
const changed = names.stdout.split(/\r?\n/).map((line) => line.trim()).filter(Boolean);
|
|
26
|
+
const status = git(repo, ["status", "--porcelain", ...scope]);
|
|
27
|
+
const dirty = status.stdout.trim().length > 0;
|
|
28
|
+
if (dirty && (options.base === options.commit || changed.length === 0)) {
|
|
29
|
+
return fail(options.outDir, "The working tree has uncommitted changes. verify-change did not score them. Commit the edit, or pass --base and --commit that name different commits with a diff.");
|
|
30
|
+
}
|
|
26
31
|
const goldenContract = goldenDiffs(repo, options.base, options.commit, changed);
|
|
27
32
|
const packageFiles = [];
|
|
28
33
|
const chosen = [];
|
package/package.json
CHANGED
|
@@ -1,10 +1,11 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "probatio",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.3",
|
|
4
4
|
"description": "Mutation runner for AI agents. Scores a test suite by the bugs it catches.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
7
|
-
"probatio": "dist/cli.js"
|
|
7
|
+
"probatio": "dist/cli.js",
|
|
8
|
+
"probatio-mcp": "dist/mcp.js"
|
|
8
9
|
},
|
|
9
10
|
"files": [
|
|
10
11
|
"dist"
|
|
@@ -20,11 +21,12 @@
|
|
|
20
21
|
},
|
|
21
22
|
"scripts": {
|
|
22
23
|
"build": "tsc -p tsconfig.json && mkdir -p dist/mutate && cp src/mutate/fail-reporter.mjs src/mutate/node-coverage.mjs src/mutate/precise-lines.mjs src/mutate/child-lines.mjs src/mutate/mocha-coverage.cjs src/mutate/node-batch.mjs src/mutate/node-batch-loader.mjs src/mutate/py-reporter.py src/mutate/py-pytest-reporter.py src/mutate/probatio-jacoco-run.java dist/mutate/",
|
|
24
|
+
"prepack": "npm run build",
|
|
23
25
|
"test": "tsx --test tests/*.test.ts",
|
|
24
26
|
"probatio": "tsx src/cli.ts"
|
|
25
27
|
},
|
|
26
28
|
"engines": {
|
|
27
|
-
"node": ">=22"
|
|
29
|
+
"node": ">=22.6"
|
|
28
30
|
},
|
|
29
31
|
"dependencies": {
|
|
30
32
|
"typescript": "^5.7.2"
|