probatio 0.0.5 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +5 -5
- package/dist/cli.js +24 -0
- package/dist/mutate/child-lines.mjs +34 -0
- package/dist/mutate/mocha-coverage.cjs +71 -0
- package/dist/mutate/node-batch-loader.mjs +85 -0
- package/dist/mutate/node-batch.mjs +130 -0
- package/dist/mutate/node-coverage.mjs +37 -180
- package/dist/mutate/precise-lines.mjs +188 -0
- package/dist/mutate/probatio-jacoco-run.java +10 -1
- package/dist/mutate/run.js +239 -34
- package/dist/mutate/suite-decision.js +1 -1
- package/dist/mutate/suites.js +75 -7
- package/dist/mutate/tally.js +1 -1
- package/dist/verify/change.js +83 -10
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
# Probatio
|
|
2
2
|
|
|
3
3
|
Probatio (Latin: a testing, a proof) is a free testing toolkit under the MIT license, built for AI agents
|
|
4
|
-
(Claude, Grok, ChatGPT, others), working alone or as swarms. Every claim it makes is checkable
|
|
5
|
-
|
|
6
|
-
|
|
4
|
+
(Claude, Grok, ChatGPT, others), working alone or as swarms. Every claim it makes is checkable.
|
|
5
|
+
It remembers what was fixed so agents with small context do not redo or undo work, and it shows
|
|
6
|
+
which tests already touch a change.
|
|
7
7
|
|
|
8
|
-
Status (2026-10-
|
|
8
|
+
Status (2026-10-06): `mutate run` discovers the project suite and scores it. Node, pytest, C (when the binary was built with LLVM coverage), Go, Maven/Java, Rust, C#, and Mocha collect a line map on the baseline. Mocha's map comes from its own root hooks. Loading the node:test collector under Mocha does not name Mocha tests. A line no test executes is `no coverage`, and the suite is not started. A plain Node script can write the lines it ran, and the parent test that reads that dump stores them under its own name. A child the parent does not read stays `no coverage`, and the suite is not started. A suite with no line map, a baseline of 5 seconds or more, and more than 30 mutants stops before the first mutant. `verify-change` names the tests the line map ran. It scores `src/`, a `lib/` file when the package tests import `lib/`, and a Go file beside `go.mod`. Tests, docs, `dist/`, and `node_modules` stay unseen. `ran` stays the direct-importer list. Covered mutants of one Node file run one after another inside the suite process that is already running. `--workers` stays 1. A Node test that sets its own `timeout` keeps that time when `--test-timeout-ms` is shorter. A mutant that crashes or leaves that process dirty ends it, and the next mutant starts clean. An uncovered line still does not start the suite. `mutate tally` does not delete a test. A kill is an assertion the suite already had. A sealed miss means the suite did not see the bug. The kill count is not a merge gate. On Commons CSV (Apache-2.0, commit `2c83a308`), two sealed runs agreed: with the new test file hidden, `CSVFormatTest.testFormatThrowsNullPointerException` failed on the reverted printer. `deletedTests` stayed 0. That older JUnit platform did not write a line map, so the first pass was the whole suite and confirm reran the older test. Auspex, markupsafe, and a hand-planted fixture are not that result. The package version in this repository is 0.1.2. npm latest is 0.1.0. The BugsInPy search row is still open, and the rows that apply only when a sealed run misses do not apply, because this catch held. The MCP server speaks one JSON object per line and returns the same JSON as the CLI.
|
|
9
9
|
|
|
10
10
|
Read in this order:
|
|
11
11
|
|
|
@@ -31,7 +31,7 @@ npx probatio mutate tally --out .probatio/runs
|
|
|
31
31
|
|
|
32
32
|
`run` discovers the suite: `run_tests.sh`, Cargo, Go, Swift, Maven, dotnet, pytest, unittest, node:test, Mocha, then `make test` when COBOL tests sit under that Makefile. An unknown layout stops and asks for `--suite-command`. It does not compile one file and call that the suite. `make test` that would curl or wget a missing file stops, and nothing is fetched. A worktree that has no `node_modules` uses the main checkout's. Omitting `--build` runs no build. Pass `--build` with a command when the suite needs one first.
|
|
33
33
|
|
|
34
|
-
|
|
34
|
+
Node, pytest, Go, Maven/Java, Rust, C#, and Mocha collect a line map on the baseline. C collects one when LLVM coverage was instrumented. Mocha writes its own `{ files }` record from root hooks, keyed by the Mocha title. The node:test collector is a different file and does not name Mocha tests. The map stores the file, the line, and the test names that hit that line. A later mutant runs only those tests (`go test -run`, Maven `-Dtest`, `cargo test`, `dotnet test --filter`, Mocha `--grep`). A line no test executed is `no coverage`, and the suite is not started. A plain Node script can leave a nameless list of the lines it ran. The parent test that reads that list stores them under its own name. A child the parent does not read stays `no coverage`. Covered mutants of one Node file are applied one after another in the suite process already running, and the line map is reset between them. `--workers` stays 1. A crash or a process left dirty ends that process. The next mutant starts clean. A baseline of 5 seconds or more, with no line map and more than 30 mutants still pending, stops before the first mutant. `next` names that baseline and tells you to narrow `--src` or pass a smaller patch directory. A one-file C, C++, Java, COBOL, or assembly launcher is not a suite discovery returns.
|
|
35
35
|
|
|
36
36
|
The mutant timeout is the baseline duration times 5, and at least 20 seconds, capped by `--suite-timeout-ms`. A timeout is not a kill. `--budget-ms` stops mutant work after the baseline. The first mutant still runs, except for that no-line-map stop. A kill is the test that failed, or a compiler token when the mutant did not build. Confirm is on by default and reruns the failing names. Pytest names that `-k` cannot express are passed as node ids. A usage error, including pytest exit 4, is not a kill. `verify-change` does not confirm a second time. Add `--affected` to limit the file list to tests that can see the change. The default, once a line map exists, runs the tests on the changed line.
|
|
37
37
|
|
package/dist/cli.js
CHANGED
|
@@ -133,6 +133,29 @@ async function generateCommand(flags) {
|
|
|
133
133
|
};
|
|
134
134
|
}
|
|
135
135
|
async function runCommand(flags, hide = null) {
|
|
136
|
+
const onlyNames = texts(flags, "only-test");
|
|
137
|
+
if (onlyNames.some((id) => id === "::command" || id.endsWith("::command"))) {
|
|
138
|
+
return {
|
|
139
|
+
schemaVersion: SCHEMA_VERSION,
|
|
140
|
+
ok: false,
|
|
141
|
+
command: "mutate.run",
|
|
142
|
+
summary: "::command is not a test id.",
|
|
143
|
+
next: "Pass a test name the suite can collect.",
|
|
144
|
+
nextCall: null,
|
|
145
|
+
killed: 0,
|
|
146
|
+
};
|
|
147
|
+
}
|
|
148
|
+
if (onlyNames.length > 0 && text(flags, "suite-command")) {
|
|
149
|
+
return {
|
|
150
|
+
schemaVersion: SCHEMA_VERSION,
|
|
151
|
+
ok: false,
|
|
152
|
+
command: "mutate.run",
|
|
153
|
+
summary: "--only-test cannot be combined with --suite-command.",
|
|
154
|
+
next: "Drop --suite-command and pass the keep ids with --only-test.",
|
|
155
|
+
nextCall: null,
|
|
156
|
+
killed: 0,
|
|
157
|
+
};
|
|
158
|
+
}
|
|
136
159
|
const packageDir = path.resolve(requireText(flags, "package"));
|
|
137
160
|
const patchDirs = texts(flags, "patches").map((dir) => path.resolve(dir));
|
|
138
161
|
if (patchDirs.length === 0)
|
|
@@ -167,6 +190,7 @@ async function runCommand(flags, hide = null) {
|
|
|
167
190
|
affected: bool(flags, "affected", false),
|
|
168
191
|
agent: text(flags, "agent") ?? null,
|
|
169
192
|
suiteCommand: text(flags, "suite-command") ?? null,
|
|
193
|
+
onlyNames: onlyNames.length > 0 ? onlyNames : null,
|
|
170
194
|
historyPath: text(flags, "history") ? path.resolve(text(flags, "history")) : null,
|
|
171
195
|
timeoutMultiple: int(flags, "timeout-multiple") ?? 5,
|
|
172
196
|
timeoutFloorMs: int(flags, "timeout-floor-ms") ?? 20_000,
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
// Nameless line dump for a plain script. This is not a node:test process and it
|
|
2
|
+
// does not write a coverage shard. The parent test's afterEach reads the dump.
|
|
3
|
+
import { randomBytes } from "node:crypto"
|
|
4
|
+
import { mkdirSync, writeFileSync } from "node:fs"
|
|
5
|
+
import path from "node:path"
|
|
6
|
+
import { openLineSampler } from "./precise-lines.mjs"
|
|
7
|
+
|
|
8
|
+
const mapPath = process.env.PROBATIO_COVERAGE_MAP
|
|
9
|
+
if (mapPath) {
|
|
10
|
+
let sampler = null
|
|
11
|
+
try {
|
|
12
|
+
sampler = await openLineSampler((rel) => rel.endsWith("child-lines.mjs") || rel.endsWith("precise-lines.mjs"))
|
|
13
|
+
} catch (err) {
|
|
14
|
+
writeFileSync(`${mapPath}.error`, `${err instanceof Error ? err.message : String(err)}\n`)
|
|
15
|
+
}
|
|
16
|
+
if (sampler) {
|
|
17
|
+
let wrote = false
|
|
18
|
+
const dump = () => {
|
|
19
|
+
if (wrote) return
|
|
20
|
+
wrote = true
|
|
21
|
+
sampler.positiveLines().then((hits) => {
|
|
22
|
+
const dir = `${mapPath}.children`
|
|
23
|
+
mkdirSync(dir, { recursive: true })
|
|
24
|
+
writeFileSync(
|
|
25
|
+
path.join(dir, `${process.pid}-${randomBytes(4).toString("hex")}.json`),
|
|
26
|
+
JSON.stringify({ hits }),
|
|
27
|
+
)
|
|
28
|
+
}).catch((err) => {
|
|
29
|
+
writeFileSync(`${mapPath}.error`, `${err instanceof Error ? err.message : String(err)}\n`)
|
|
30
|
+
})
|
|
31
|
+
}
|
|
32
|
+
process.on("beforeExit", dump)
|
|
33
|
+
}
|
|
34
|
+
}
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
// Mocha root hooks. This file is not node-coverage.mjs. Mocha does not run
|
|
2
|
+
// node:test afterEach/after, so that collector records no Mocha title.
|
|
3
|
+
// These hooks key inspector hits by the current Mocha title and write { files }.
|
|
4
|
+
const { mkdirSync, writeFileSync } = require("node:fs")
|
|
5
|
+
const path = require("node:path")
|
|
6
|
+
|
|
7
|
+
const mapPath = process.env.PROBATIO_COVERAGE_MAP
|
|
8
|
+
if (mapPath) {
|
|
9
|
+
/** @type {Map<string, Map<number, Set<string>>>} */
|
|
10
|
+
const byFile = new Map()
|
|
11
|
+
/** @type {Array<{ file: string, line: number }>} */
|
|
12
|
+
let prelude = []
|
|
13
|
+
/** @type {{ positiveLines: () => Promise<Array<{ file: string, line: number }>> } | null} */
|
|
14
|
+
let sampler = null
|
|
15
|
+
|
|
16
|
+
function add(hits, name) {
|
|
17
|
+
if (!name) return
|
|
18
|
+
for (const hit of hits) {
|
|
19
|
+
let lines = byFile.get(hit.file)
|
|
20
|
+
if (!lines) {
|
|
21
|
+
lines = new Map()
|
|
22
|
+
byFile.set(hit.file, lines)
|
|
23
|
+
}
|
|
24
|
+
let names = lines.get(hit.line)
|
|
25
|
+
if (!names) {
|
|
26
|
+
names = new Set()
|
|
27
|
+
lines.set(hit.line, names)
|
|
28
|
+
}
|
|
29
|
+
names.add(name)
|
|
30
|
+
}
|
|
31
|
+
}
|
|
32
|
+
|
|
33
|
+
function titleOf(test) {
|
|
34
|
+
if (!test) return ""
|
|
35
|
+
if (typeof test.fullTitle === "function") return test.fullTitle()
|
|
36
|
+
return typeof test.title === "string" ? test.title : ""
|
|
37
|
+
}
|
|
38
|
+
|
|
39
|
+
exports.mochaHooks = {
|
|
40
|
+
async beforeAll() {
|
|
41
|
+
try {
|
|
42
|
+
const opened = await import("./precise-lines.mjs")
|
|
43
|
+
sampler = await opened.openLineSampler((rel) => rel.endsWith("mocha-coverage.cjs") || rel.endsWith("precise-lines.mjs") || rel.endsWith("node-coverage.mjs"))
|
|
44
|
+
prelude = await sampler.positiveLines()
|
|
45
|
+
} catch (err) {
|
|
46
|
+
writeFileSync(`${mapPath}.error`, `${err instanceof Error ? err.message : String(err)}\n`)
|
|
47
|
+
}
|
|
48
|
+
},
|
|
49
|
+
async afterEach() {
|
|
50
|
+
if (!sampler) return
|
|
51
|
+
const title = titleOf(this.currentTest)
|
|
52
|
+
try {
|
|
53
|
+
add(prelude, title)
|
|
54
|
+
add(await sampler.positiveLines(), title)
|
|
55
|
+
} catch (err) {
|
|
56
|
+
writeFileSync(`${mapPath}.error`, `${err instanceof Error ? err.message : String(err)}\n`)
|
|
57
|
+
}
|
|
58
|
+
},
|
|
59
|
+
afterAll() {
|
|
60
|
+
if (byFile.size === 0) return
|
|
61
|
+
const files = {}
|
|
62
|
+
for (const [file, lines] of byFile) {
|
|
63
|
+
const bucket = {}
|
|
64
|
+
for (const [line, names] of lines) bucket[String(line)] = [...names].sort()
|
|
65
|
+
files[file] = bucket
|
|
66
|
+
}
|
|
67
|
+
mkdirSync(path.dirname(mapPath), { recursive: true })
|
|
68
|
+
writeFileSync(mapPath, `${JSON.stringify({ files }, null, 2)}\n`)
|
|
69
|
+
},
|
|
70
|
+
}
|
|
71
|
+
}
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
// Cache-busts package files so the next mutant in this process loads the patched source.
|
|
2
|
+
// The generation lives in a file because the loader thread does not see later env edits.
|
|
3
|
+
import { createRequire } from "node:module"
|
|
4
|
+
import { readFileSync, realpathSync } from "node:fs"
|
|
5
|
+
import path from "node:path"
|
|
6
|
+
import { fileURLToPath } from "node:url"
|
|
7
|
+
|
|
8
|
+
const require = createRequire(import.meta.url)
|
|
9
|
+
const ts = require("typescript")
|
|
10
|
+
|
|
11
|
+
function generation() {
|
|
12
|
+
const file = process.env.PB_GEN
|
|
13
|
+
if (!file) return ""
|
|
14
|
+
try {
|
|
15
|
+
return readFileSync(file, "utf8").trim()
|
|
16
|
+
} catch {
|
|
17
|
+
return ""
|
|
18
|
+
}
|
|
19
|
+
}
|
|
20
|
+
|
|
21
|
+
function packageRoot() {
|
|
22
|
+
const root = process.env.PB_ROOT
|
|
23
|
+
if (!root) return ""
|
|
24
|
+
try {
|
|
25
|
+
return realpathSync(root)
|
|
26
|
+
} catch {
|
|
27
|
+
return root
|
|
28
|
+
}
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
function underPackage(abs) {
|
|
32
|
+
const root = packageRoot()
|
|
33
|
+
if (!root) return false
|
|
34
|
+
let real = abs
|
|
35
|
+
try {
|
|
36
|
+
real = realpathSync(abs)
|
|
37
|
+
} catch {
|
|
38
|
+
return false
|
|
39
|
+
}
|
|
40
|
+
return real === root || real.startsWith(root.endsWith(path.sep) ? root : root + path.sep)
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
export async function resolve(specifier, context, nextResolve) {
|
|
44
|
+
const resolved = await nextResolve(specifier, context)
|
|
45
|
+
const gen = generation()
|
|
46
|
+
if (!gen) return resolved
|
|
47
|
+
let abs
|
|
48
|
+
try {
|
|
49
|
+
abs = fileURLToPath(resolved.url.split("?")[0])
|
|
50
|
+
} catch {
|
|
51
|
+
return resolved
|
|
52
|
+
}
|
|
53
|
+
if (!underPackage(abs)) return resolved
|
|
54
|
+
if (abs.split(path.sep).includes("node_modules")) return resolved
|
|
55
|
+
const url = new URL(resolved.url.split("?")[0])
|
|
56
|
+
url.searchParams.set("pb", gen)
|
|
57
|
+
return { url: url.href, shortCircuit: true }
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
export async function load(url, context, nextLoad) {
|
|
61
|
+
if (!url.includes("pb=")) return nextLoad(url, context)
|
|
62
|
+
const clean = url.split("?")[0]
|
|
63
|
+
let abs
|
|
64
|
+
try {
|
|
65
|
+
abs = fileURLToPath(clean)
|
|
66
|
+
} catch {
|
|
67
|
+
return nextLoad(url, context)
|
|
68
|
+
}
|
|
69
|
+
const source = readFileSync(abs, "utf8")
|
|
70
|
+
if (abs.endsWith(".ts") || abs.endsWith(".tsx") || abs.endsWith(".mts")) {
|
|
71
|
+
const transpiled = ts.transpileModule(source, {
|
|
72
|
+
fileName: abs,
|
|
73
|
+
compilerOptions: {
|
|
74
|
+
module: ts.ModuleKind.ESNext,
|
|
75
|
+
target: ts.ScriptTarget.ES2022,
|
|
76
|
+
sourceMap: false,
|
|
77
|
+
},
|
|
78
|
+
})
|
|
79
|
+
return { format: "module", source: transpiled.outputText, shortCircuit: true }
|
|
80
|
+
}
|
|
81
|
+
if (abs.endsWith(".mjs") || abs.endsWith(".js") || source.includes("import ") || source.includes("export ")) {
|
|
82
|
+
return { format: "module", source, shortCircuit: true }
|
|
83
|
+
}
|
|
84
|
+
return { format: "commonjs", source, shortCircuit: true }
|
|
85
|
+
}
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
// One suite process for covered mutants of one file. Jobs arrive on stdin.
|
|
2
|
+
// A mutant that exits this process ends it. The parent starts a clean process for the next mutant.
|
|
3
|
+
import { createInterface } from "node:readline"
|
|
4
|
+
import { realpathSync, writeFileSync } from "node:fs"
|
|
5
|
+
import { register } from "node:module"
|
|
6
|
+
import { run } from "node:test"
|
|
7
|
+
import path from "node:path"
|
|
8
|
+
import { openLineSampler } from "./precise-lines.mjs"
|
|
9
|
+
|
|
10
|
+
const loader = new URL("./node-batch-loader.mjs", import.meta.url).href
|
|
11
|
+
register(loader, { parentURL: import.meta.url })
|
|
12
|
+
|
|
13
|
+
let sampler = null
|
|
14
|
+
try {
|
|
15
|
+
sampler = await openLineSampler()
|
|
16
|
+
} catch {
|
|
17
|
+
sampler = null
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
writeFileSync(process.env.PB_READY, `${JSON.stringify({ pid: process.pid })}\n`)
|
|
21
|
+
|
|
22
|
+
function escapeRegExp(value) {
|
|
23
|
+
return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
let generation = 0
|
|
27
|
+
|
|
28
|
+
async function runJob(job) {
|
|
29
|
+
generation += 1
|
|
30
|
+
writeFileSync(process.env.PB_GEN, String(generation))
|
|
31
|
+
if (sampler) await sampler.reset()
|
|
32
|
+
const files = (job.tests || []).map((rel) => path.join(job.root, rel))
|
|
33
|
+
const patterns = (job.names || []).filter((name) => name.length > 0).map((name) => new RegExp(`^${escapeRegExp(name)}$`))
|
|
34
|
+
const failed = []
|
|
35
|
+
const names = []
|
|
36
|
+
const testTimeout = job.testTimeoutMs > 0 ? job.testTimeoutMs : job.timeoutMs
|
|
37
|
+
const stream = run({
|
|
38
|
+
files,
|
|
39
|
+
isolation: "none",
|
|
40
|
+
timeout: testTimeout,
|
|
41
|
+
...(patterns.length > 0 ? { testNamePatterns: patterns } : {}),
|
|
42
|
+
})
|
|
43
|
+
let lastEvent = Date.now()
|
|
44
|
+
const started = Date.now()
|
|
45
|
+
let ended = false
|
|
46
|
+
let timer = null
|
|
47
|
+
const collected = (async () => {
|
|
48
|
+
for await (const event of stream) {
|
|
49
|
+
lastEvent = Date.now()
|
|
50
|
+
if (event.type !== "test:pass" && event.type !== "test:fail") continue
|
|
51
|
+
const data = event.data || {}
|
|
52
|
+
const name = typeof data.name === "string" ? data.name : ""
|
|
53
|
+
if (!name || files.includes(name)) continue
|
|
54
|
+
if (!names.includes(name)) names.push(name)
|
|
55
|
+
if (event.type === "test:fail") {
|
|
56
|
+
const error = data.details && data.details.error
|
|
57
|
+
const text = error ? `${error.failureType || ""} ${error.message || ""} ${error.cause || ""}` : ""
|
|
58
|
+
const fileTimeout = /testTimeoutFailure|timed out/i.test(text) && fileNameIs(name, data.file || "")
|
|
59
|
+
failed.push({ name, file: data.file || "", line: data.line || 0, ...(fileTimeout ? { fileTimeout: true } : {}) })
|
|
60
|
+
}
|
|
61
|
+
}
|
|
62
|
+
ended = true
|
|
63
|
+
})()
|
|
64
|
+
let dirty = false
|
|
65
|
+
try {
|
|
66
|
+
dirty = await Promise.race([
|
|
67
|
+
collected.then(() => false),
|
|
68
|
+
new Promise((resolve) => {
|
|
69
|
+
timer = setInterval(() => {
|
|
70
|
+
if (ended) {
|
|
71
|
+
resolve(false)
|
|
72
|
+
return
|
|
73
|
+
}
|
|
74
|
+
const idle = Date.now() - lastEvent
|
|
75
|
+
const overdue = Date.now() - started >= testTimeout && idle >= 400 && names.length > 0
|
|
76
|
+
if (!overdue) return
|
|
77
|
+
resolve(true)
|
|
78
|
+
}, 50)
|
|
79
|
+
timer.unref()
|
|
80
|
+
}),
|
|
81
|
+
])
|
|
82
|
+
} finally {
|
|
83
|
+
if (timer) clearInterval(timer)
|
|
84
|
+
}
|
|
85
|
+
if (dirty && failed.length === 0) {
|
|
86
|
+
for (const rel of job.tests || []) {
|
|
87
|
+
const posix = rel.split(path.sep).join("/")
|
|
88
|
+
let file = path.join(job.root, rel)
|
|
89
|
+
try {
|
|
90
|
+
file = realpathSync(file)
|
|
91
|
+
} catch {
|
|
92
|
+
// The reporter uses the path it opened. A missing file keeps the joined path.
|
|
93
|
+
}
|
|
94
|
+
failed.push({ name: posix, file, line: 1, fileTimeout: true })
|
|
95
|
+
if (!names.includes(posix)) names.push(posix)
|
|
96
|
+
}
|
|
97
|
+
}
|
|
98
|
+
if (sampler && !dirty) await sampler.reset()
|
|
99
|
+
const pattern = (job.names || []).join("|")
|
|
100
|
+
const command = ["node", "--test", ...(pattern ? [`--test-name-pattern=${pattern}`] : []), ...(job.tests || [])].join(" ")
|
|
101
|
+
return { id: job.id, pid: process.pid, failed, names, command, dirty }
|
|
102
|
+
}
|
|
103
|
+
|
|
104
|
+
function fileNameIs(name, file) {
|
|
105
|
+
const base = String(file).split(/[/\\]/).pop()
|
|
106
|
+
if (!base) return false
|
|
107
|
+
return name === file || name === base || name.endsWith(`/${base}`) || name.endsWith(`\\${base}`)
|
|
108
|
+
}
|
|
109
|
+
|
|
110
|
+
const lines = createInterface({ input: process.stdin })
|
|
111
|
+
for await (const line of lines) {
|
|
112
|
+
if (!line.trim()) continue
|
|
113
|
+
const job = JSON.parse(line)
|
|
114
|
+
// node:test with isolation "none" waits until the event loop is idle.
|
|
115
|
+
// An open readline on stdin is that handle, so the job would never finish.
|
|
116
|
+
lines.pause()
|
|
117
|
+
process.stdin.unref()
|
|
118
|
+
try {
|
|
119
|
+
const reply = await runJob(job)
|
|
120
|
+
writeFileSync(job.replyFile, `${JSON.stringify(reply)}\n`)
|
|
121
|
+
// An open handle after the tests finished is the same file timeout node --test reports.
|
|
122
|
+
// The process is dirty, so it ends and the next mutant starts clean.
|
|
123
|
+
if (reply.dirty) process.exit(0)
|
|
124
|
+
} catch (err) {
|
|
125
|
+
writeFileSync(job.replyFile, `${JSON.stringify({ id: job.id, pid: process.pid, failed: [], names: [], error: err instanceof Error ? err.message : String(err) })}\n`)
|
|
126
|
+
} finally {
|
|
127
|
+
if (!process.stdin.destroyed) process.stdin.ref()
|
|
128
|
+
lines.resume()
|
|
129
|
+
}
|
|
130
|
+
}
|
|
@@ -1,200 +1,29 @@
|
|
|
1
1
|
// Baseline-only per-test line map for node --test. Loaded with --import in each
|
|
2
2
|
// test-file process. Writes one shard; the runner merges shards after the suite.
|
|
3
3
|
// takePreciseCoverage resets counters, so each sample is only what ran since the last take.
|
|
4
|
+
// A plain-script child does not run these hooks. It may leave a nameless hit list in
|
|
5
|
+
// ${map}.children. afterEach stores that list under the parent test before the shard is written.
|
|
4
6
|
import { randomBytes } from "node:crypto"
|
|
5
|
-
import { mkdirSync, writeFileSync } from "node:fs"
|
|
6
|
-
import { Session } from "node:inspector/promises"
|
|
7
|
-
import { findSourceMap } from "node:module"
|
|
7
|
+
import { existsSync, mkdirSync, readdirSync, readFileSync, unlinkSync, writeFileSync } from "node:fs"
|
|
8
8
|
import { after, afterEach, before } from "node:test"
|
|
9
9
|
import path from "node:path"
|
|
10
|
-
import {
|
|
11
|
-
|
|
12
|
-
const B64 = new Int32Array(128)
|
|
13
|
-
for (let i = 0; i < 64; i++) B64["ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/".charCodeAt(i)] = i
|
|
14
|
-
|
|
15
|
-
function decodeSegments(row) {
|
|
16
|
-
const segments = []
|
|
17
|
-
let i = 0
|
|
18
|
-
while (i < row.length) {
|
|
19
|
-
const fields = []
|
|
20
|
-
while (i < row.length && row.charCodeAt(i) !== 44) {
|
|
21
|
-
let value = 0
|
|
22
|
-
let shift = 0
|
|
23
|
-
let digit = 0
|
|
24
|
-
do {
|
|
25
|
-
digit = B64[row.charCodeAt(i++)]
|
|
26
|
-
value |= (digit & 31) << shift
|
|
27
|
-
shift += 5
|
|
28
|
-
} while (digit & 32)
|
|
29
|
-
fields.push(value & 1 ? -(value >> 1) : value >> 1)
|
|
30
|
-
}
|
|
31
|
-
if (row.charCodeAt(i) === 44) i++
|
|
32
|
-
if (fields.length > 0) segments.push(fields)
|
|
33
|
-
}
|
|
34
|
-
return segments
|
|
35
|
-
}
|
|
10
|
+
import { openLineSampler } from "./precise-lines.mjs"
|
|
36
11
|
|
|
37
12
|
const mapPath = process.env.PROBATIO_COVERAGE_MAP
|
|
38
13
|
if (mapPath) {
|
|
39
|
-
|
|
40
|
-
let started = false
|
|
14
|
+
let sampler = null
|
|
41
15
|
try {
|
|
42
|
-
|
|
43
|
-
await session.post("Profiler.enable")
|
|
44
|
-
await session.post("Debugger.enable")
|
|
45
|
-
await session.post("Profiler.startPreciseCoverage", { callCount: true, detailed: true })
|
|
46
|
-
started = true
|
|
16
|
+
sampler = await openLineSampler((rel) => rel.endsWith("node-coverage.mjs") || rel.endsWith("precise-lines.mjs") || rel.endsWith("child-lines.mjs"))
|
|
47
17
|
} catch (err) {
|
|
48
18
|
writeFileSync(`${mapPath}.error`, `${err instanceof Error ? err.message : String(err)}\n`)
|
|
49
19
|
}
|
|
50
20
|
|
|
51
|
-
if (
|
|
52
|
-
const textCache = new Map()
|
|
53
|
-
const indexCache = new Map()
|
|
21
|
+
if (sampler) {
|
|
54
22
|
/** @type {Map<string, Map<number, Set<string>>>} */
|
|
55
23
|
const byFile = new Map()
|
|
56
24
|
/** @type {Array<{ file: string, line: number }>} */
|
|
57
25
|
let prelude = []
|
|
58
26
|
|
|
59
|
-
async function scriptText(script) {
|
|
60
|
-
if (textCache.has(script.scriptId)) return textCache.get(script.scriptId)
|
|
61
|
-
let text = ""
|
|
62
|
-
try {
|
|
63
|
-
const got = await session.post("Debugger.getScriptSource", { scriptId: script.scriptId })
|
|
64
|
-
text = got.scriptSource || ""
|
|
65
|
-
} catch {
|
|
66
|
-
text = ""
|
|
67
|
-
}
|
|
68
|
-
textCache.set(script.scriptId, text)
|
|
69
|
-
return text
|
|
70
|
-
}
|
|
71
|
-
|
|
72
|
-
function relPath(url) {
|
|
73
|
-
if (!url || url.startsWith("node:")) return null
|
|
74
|
-
let abs
|
|
75
|
-
try {
|
|
76
|
-
abs = fileURLToPath(url.split("?")[0].split("#")[0])
|
|
77
|
-
} catch {
|
|
78
|
-
return null
|
|
79
|
-
}
|
|
80
|
-
const rel = path.relative(process.cwd(), abs).split(path.sep).join("/")
|
|
81
|
-
if (!rel || rel.startsWith("..") || path.isAbsolute(rel)) return null
|
|
82
|
-
if (rel.split("/").includes("node_modules")) return null
|
|
83
|
-
if (rel.endsWith("node-coverage.mjs")) return null
|
|
84
|
-
return rel
|
|
85
|
-
}
|
|
86
|
-
|
|
87
|
-
function indexFor(scriptId, text, sm) {
|
|
88
|
-
if (indexCache.has(scriptId)) return indexCache.get(scriptId)
|
|
89
|
-
const mapped = sm && sm.payload && typeof sm.payload.mappings === "string" ? linesFromMappings(text, sm.payload.mappings) : null
|
|
90
|
-
const lines = mapped && hasLine(mapped) ? mapped : linesFromNewlines(text, sm)
|
|
91
|
-
indexCache.set(scriptId, lines)
|
|
92
|
-
return lines
|
|
93
|
-
}
|
|
94
|
-
|
|
95
|
-
function hasLine(lines) {
|
|
96
|
-
for (let i = 0; i < lines.length; i++) if (lines[i] > 0) return true
|
|
97
|
-
return false
|
|
98
|
-
}
|
|
99
|
-
|
|
100
|
-
function linesFromNewlines(text, sm) {
|
|
101
|
-
const lines = new Int32Array(text.length + 1)
|
|
102
|
-
if (!sm) {
|
|
103
|
-
let line = 1
|
|
104
|
-
for (let i = 0; i < text.length; i++) {
|
|
105
|
-
lines[i] = line
|
|
106
|
-
if (text.charCodeAt(i) === 10) line++
|
|
107
|
-
}
|
|
108
|
-
return lines
|
|
109
|
-
}
|
|
110
|
-
// Node 22 reports SourceMap.findEntry originalLine as 0-based.
|
|
111
|
-
let genLine = 0
|
|
112
|
-
let col = 0
|
|
113
|
-
for (let i = 0; i < text.length; i++) {
|
|
114
|
-
const entry = sm.findEntry(genLine, col)
|
|
115
|
-
if (entry && Number.isInteger(entry.originalLine) && entry.originalLine >= 0) lines[i] = entry.originalLine + 1
|
|
116
|
-
if (text.charCodeAt(i) === 10) {
|
|
117
|
-
genLine++
|
|
118
|
-
col = 0
|
|
119
|
-
} else {
|
|
120
|
-
col++
|
|
121
|
-
}
|
|
122
|
-
}
|
|
123
|
-
return lines
|
|
124
|
-
}
|
|
125
|
-
|
|
126
|
-
function linesFromMappings(text, mappings) {
|
|
127
|
-
const lines = new Int32Array(text.length + 1)
|
|
128
|
-
const starts = [0]
|
|
129
|
-
for (let i = 0; i < text.length; i++) if (text.charCodeAt(i) === 10) starts.push(i + 1)
|
|
130
|
-
let carryLine = 0
|
|
131
|
-
const rows = mappings.split(";")
|
|
132
|
-
for (let genLine = 0; genLine < rows.length && genLine < starts.length; genLine++) {
|
|
133
|
-
const start = starts[genLine]
|
|
134
|
-
const next = genLine + 1 < starts.length ? starts[genLine + 1] : text.length
|
|
135
|
-
let col = 0
|
|
136
|
-
let cursor = start
|
|
137
|
-
let active = -1
|
|
138
|
-
for (const fields of decodeSegments(rows[genLine])) {
|
|
139
|
-
col += fields[0]
|
|
140
|
-
const at = Math.min(next, start + Math.max(0, col))
|
|
141
|
-
if (active >= 0) {
|
|
142
|
-
for (let i = cursor; i < at; i++) lines[i] = active + 1
|
|
143
|
-
}
|
|
144
|
-
cursor = at
|
|
145
|
-
if (fields.length >= 4) {
|
|
146
|
-
carryLine += fields[2]
|
|
147
|
-
active = carryLine
|
|
148
|
-
} else {
|
|
149
|
-
active = -1
|
|
150
|
-
}
|
|
151
|
-
}
|
|
152
|
-
if (active >= 0) {
|
|
153
|
-
for (let i = cursor; i < next; i++) lines[i] = active + 1
|
|
154
|
-
}
|
|
155
|
-
}
|
|
156
|
-
return lines
|
|
157
|
-
}
|
|
158
|
-
|
|
159
|
-
function applyRanges(text, script, lines) {
|
|
160
|
-
const ranges = []
|
|
161
|
-
for (const fn of script.functions || []) {
|
|
162
|
-
for (const range of fn.ranges || []) ranges.push(range)
|
|
163
|
-
}
|
|
164
|
-
// Outer ranges first. A nested block then overwrites its own offsets, so a dead
|
|
165
|
-
// branch stays at 0 while a condition earlier on that same line can stay covered.
|
|
166
|
-
ranges.sort((a, b) => a.startOffset - b.startOffset || b.endOffset - a.endOffset)
|
|
167
|
-
const offsets = new Int32Array(text.length)
|
|
168
|
-
for (const range of ranges) {
|
|
169
|
-
const end = Math.min(text.length, range.endOffset)
|
|
170
|
-
for (let i = Math.max(0, range.startOffset); i < end; i++) offsets[i] = range.count
|
|
171
|
-
}
|
|
172
|
-
const counts = new Map()
|
|
173
|
-
for (let i = 0; i < text.length; i++) {
|
|
174
|
-
const line = lines[i]
|
|
175
|
-
if (line <= 0 || offsets[i] <= 0) continue
|
|
176
|
-
if (offsets[i] > (counts.get(line) || 0)) counts.set(line, offsets[i])
|
|
177
|
-
}
|
|
178
|
-
return counts
|
|
179
|
-
}
|
|
180
|
-
|
|
181
|
-
async function positiveLines() {
|
|
182
|
-
const cov = await session.post("Profiler.takePreciseCoverage")
|
|
183
|
-
/** @type {Array<{ file: string, line: number }>} */
|
|
184
|
-
const found = []
|
|
185
|
-
for (const script of cov.result || []) {
|
|
186
|
-
const file = relPath(script.url)
|
|
187
|
-
if (!file) continue
|
|
188
|
-
const text = await scriptText(script)
|
|
189
|
-
if (!text) continue
|
|
190
|
-
const counts = applyRanges(text, script, indexFor(script.scriptId, text, findSourceMap(script.url)))
|
|
191
|
-
for (const [line, count] of counts) {
|
|
192
|
-
if (count > 0) found.push({ file, line })
|
|
193
|
-
}
|
|
194
|
-
}
|
|
195
|
-
return found
|
|
196
|
-
}
|
|
197
|
-
|
|
198
27
|
function add(hits, name) {
|
|
199
28
|
if (!name) return
|
|
200
29
|
for (const hit of hits) {
|
|
@@ -212,9 +41,36 @@ if (mapPath) {
|
|
|
212
41
|
}
|
|
213
42
|
}
|
|
214
43
|
|
|
44
|
+
function takeChildHits() {
|
|
45
|
+
const dir = `${mapPath}.children`
|
|
46
|
+
if (!existsSync(dir)) return []
|
|
47
|
+
/** @type {Array<{ file: string, line: number }>} */
|
|
48
|
+
const hits = []
|
|
49
|
+
for (const name of readdirSync(dir)) {
|
|
50
|
+
if (!name.endsWith(".json")) continue
|
|
51
|
+
const full = path.join(dir, name)
|
|
52
|
+
try {
|
|
53
|
+
const parsed = JSON.parse(readFileSync(full, "utf8"))
|
|
54
|
+
const list = parsed && Array.isArray(parsed.hits) ? parsed.hits : []
|
|
55
|
+
for (const hit of list) {
|
|
56
|
+
if (!hit || typeof hit.file !== "string" || !Number.isInteger(hit.line)) continue
|
|
57
|
+
hits.push({ file: hit.file, line: hit.line })
|
|
58
|
+
}
|
|
59
|
+
} catch {
|
|
60
|
+
// A torn dump is ignored. The parent does not invent a line.
|
|
61
|
+
}
|
|
62
|
+
try {
|
|
63
|
+
unlinkSync(full)
|
|
64
|
+
} catch {
|
|
65
|
+
// The next test must not inherit this dump.
|
|
66
|
+
}
|
|
67
|
+
}
|
|
68
|
+
return hits
|
|
69
|
+
}
|
|
70
|
+
|
|
215
71
|
before(async () => {
|
|
216
72
|
try {
|
|
217
|
-
prelude = await positiveLines()
|
|
73
|
+
prelude = await sampler.positiveLines()
|
|
218
74
|
} catch (err) {
|
|
219
75
|
writeFileSync(`${mapPath}.error`, `${err instanceof Error ? err.message : String(err)}\n`)
|
|
220
76
|
}
|
|
@@ -223,7 +79,8 @@ if (mapPath) {
|
|
|
223
79
|
afterEach(async (t) => {
|
|
224
80
|
try {
|
|
225
81
|
add(prelude, t.name)
|
|
226
|
-
add(await positiveLines(), t.name)
|
|
82
|
+
add(await sampler.positiveLines(), t.name)
|
|
83
|
+
add(takeChildHits(), t.name)
|
|
227
84
|
} catch (err) {
|
|
228
85
|
writeFileSync(`${mapPath}.error`, `${err instanceof Error ? err.message : String(err)}\n`)
|
|
229
86
|
}
|