probatio 0.0.5 → 0.1.0

package/README.md

@@ -1,11 +1,11 @@
 # Probatio
 
 Probatio (Latin: a testing, a proof) is a free testing toolkit under the MIT license, built for AI agents
-(Claude, Grok, ChatGPT, others), working alone or as swarms. Every claim it makes is checkable;
-it remembers what was fixed so agents with small context do not redo or undo work; and it finds
-the bugs a test suite would miss instead of asking anyone to read more tests.
+(Claude, Grok, ChatGPT, others), working alone or as swarms. Every claim it makes is checkable.
+It remembers what was fixed so agents with small context do not redo or undo work, and it shows
+which tests already touch a change.
 
-Status (2026-10-05): `mutate run` discovers the project suite and scores it. Python, JavaScript, and TypeScript narrow each mutant with a line map collected on the baseline. C does the same when the binary was built with LLVM coverage. A suite with no line map, a baseline of 5 seconds or more, and more than 30 mutants stops before the first mutant. `verify-change` mutates the changed lines and uses that same map. `mutate tally` reads a finished run, names tests that killed nothing, and does not delete a test. The MCP server speaks one JSON object per line and returns the same JSON as the CLI.
+Status (2026-10-06): `mutate run` discovers the project suite and scores it. Node, pytest, C (when the binary was built with LLVM coverage), Go, Maven/Java, Rust, and C# collect a line map on the baseline. A line no test executes is `no coverage`, and the suite is not started. A line that runs only in a child process the map cannot see is `no coverage`. A suite with no line map, a baseline of 5 seconds or more, and more than 30 mutants stops before the first mutant. `verify-change` names the tests the line map ran. `ran` stays the direct-importer list. `mutate tally` does not delete a test. A kill is an assertion the suite already had. A sealed miss means the suite did not see the bug. The kill count is not a merge gate. On Commons CSV (Apache-2.0, commit `2c83a308`), two sealed runs agreed: with the new test file hidden, `CSVFormatTest.testFormatThrowsNullPointerException` failed on the reverted printer. `deletedTests` stayed 0. That older JUnit platform did not write a line map, so the first pass was the whole suite and confirm reran the older test. Auspex, markupsafe, and a hand-planted fixture are not that result. The package version in this repository is 0.1.0. The copy on npm is 0.0.5. The BugsInPy search row is still open, and the rows that apply only when a sealed run misses do not apply, because this catch held. The MCP server speaks one JSON object per line and returns the same JSON as the CLI.
 
 Read in this order:
 
@@ -31,7 +31,7 @@ npx probatio mutate tally --out .probatio/runs
 
 `run` discovers the suite: `run_tests.sh`, Cargo, Go, Swift, Maven, dotnet, pytest, unittest, node:test, Mocha, then `make test` when COBOL tests sit under that Makefile. An unknown layout stops and asks for `--suite-command`. It does not compile one file and call that the suite. `make test` that would curl or wget a missing file stops, and nothing is fetched. A worktree that has no `node_modules` uses the main checkout's. Omitting `--build` runs no build. Pass `--build` with a command when the suite needs one first.
 
-Python, JavaScript, and TypeScript collect a line map on the baseline. C collects one when LLVM coverage was instrumented. Go and Maven/Java collect file, line, and the test names that hit that line. A later mutant runs only those tests (`go test -run`, Maven `-Dtest`). A line no test executed is `no coverage`, and the suite is not started. Rust and C# can take a test name and do not collect a line. A baseline of 5 seconds or more, with no line map and more than 30 mutants still pending, stops before the first mutant. `next` names that baseline and tells you to narrow `--src` or pass a smaller patch directory. A one-file C, C++, Java, COBOL, or assembly launcher is not a suite discovery returns.
+Node, pytest, Go, Maven/Java, Rust, and C# collect a line map on the baseline. C collects one when LLVM coverage was instrumented. The map stores the file, the line, and the test names that hit that line. A later mutant runs only those tests (`go test -run`, Maven `-Dtest`, `cargo test`, `dotnet test --filter`). A line no test executed is `no coverage`, and the suite is not started. A child process the map cannot see is `no coverage`. A baseline of 5 seconds or more, with no line map and more than 30 mutants still pending, stops before the first mutant. `next` names that baseline and tells you to narrow `--src` or pass a smaller patch directory. A one-file C, C++, Java, COBOL, or assembly launcher is not a suite discovery returns.
 
 The mutant timeout is the baseline duration times 5, and at least 20 seconds, capped by `--suite-timeout-ms`. A timeout is not a kill. `--budget-ms` stops mutant work after the baseline. The first mutant still runs, except for that no-line-map stop. A kill is the test that failed, or a compiler token when the mutant did not build. Confirm is on by default and reruns the failing names. Pytest names that `-k` cannot express are passed as node ids. A usage error, including pytest exit 4, is not a kill. `verify-change` does not confirm a second time. Add `--affected` to limit the file list to tests that can see the change. The default, once a line map exists, runs the tests on the changed line.
 

package/dist/cli.js

@@ -133,6 +133,29 @@ async function generateCommand(flags) {
     };
 }
 async function runCommand(flags, hide = null) {
+    const onlyNames = texts(flags, "only-test");
+    if (onlyNames.some((id) => id === "::command" || id.endsWith("::command"))) {
+        return {
+            schemaVersion: SCHEMA_VERSION,
+            ok: false,
+            command: "mutate.run",
+            summary: "::command is not a test id.",
+            next: "Pass a test name the suite can collect.",
+            nextCall: null,
+            killed: 0,
+        };
+    }
+    if (onlyNames.length > 0 && text(flags, "suite-command")) {
+        return {
+            schemaVersion: SCHEMA_VERSION,
+            ok: false,
+            command: "mutate.run",
+            summary: "--only-test cannot be combined with --suite-command.",
+            next: "Drop --suite-command and pass the keep ids with --only-test.",
+            nextCall: null,
+            killed: 0,
+        };
+    }
     const packageDir = path.resolve(requireText(flags, "package"));
     const patchDirs = texts(flags, "patches").map((dir) => path.resolve(dir));
     if (patchDirs.length === 0)
@@ -167,6 +190,7 @@ async function runCommand(flags, hide = null) {
         affected: bool(flags, "affected", false),
         agent: text(flags, "agent") ?? null,
         suiteCommand: text(flags, "suite-command") ?? null,
+        onlyNames: onlyNames.length > 0 ? onlyNames : null,
         historyPath: text(flags, "history") ? path.resolve(text(flags, "history")) : null,
         timeoutMultiple: int(flags, "timeout-multiple") ?? 5,
         timeoutFloorMs: int(flags, "timeout-floor-ms") ?? 20_000,

package/dist/mutate/probatio-jacoco-run.java

@@ -252,11 +252,17 @@ public class ProbatioJacocoRun {
       Class<?> selectorType = Class.forName("org.junit.platform.engine.DiscoverySelector");
       Class<?> selectorsType = Class.forName("org.junit.platform.engine.discovery.DiscoverySelectors");
       java.lang.reflect.Method select = selectMethodWithTypes(selectorsType);
+      boolean typed = select.getParameterCount() == 3;
       Object selectors = Array.newInstance(selectorType, unique.size());
       int index = 0;
       for (JupiterSpec spec : unique.values()) {
         Class<?> type = loadTestClass(spec.className);
-        Object selector = select.invoke(null, new Object[] { type, spec.simple, spec.types });
+        // Platform 1.7 has selectMethod(Class, String) only. 5.14's two-argument
+        // form drops the parameter list and can run nothing, so keep the types
+        // when that overload exists.
+        Object selector = typed
+          ? select.invoke(null, new Object[] { type, spec.simple, spec.types })
+          : select.invoke(null, new Object[] { type, spec.simple });
         Array.set(selectors, index++, selector);
       }
       Class<?> builderType = Class.forName("org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder");
@@ -426,13 +432,16 @@ public class ProbatioJacocoRun {
   }
 
   static java.lang.reflect.Method selectMethodWithTypes(Class<?> selectorsType) throws NoSuchMethodException {
+    java.lang.reflect.Method byName = null;
     for (java.lang.reflect.Method candidate : selectorsType.getMethods()) {
       if (!candidate.getName().equals("selectMethod")) continue;
       Class<?>[] params = candidate.getParameterTypes();
       if (params.length == 3 && params[0] == Class.class && params[1] == String.class && params[2].isArray() && params[2].getComponentType() == Class.class) {
         return candidate;
       }
+      if (params.length == 2 && params[0] == Class.class && params[1] == String.class) byName = candidate;
     }
+    if (byName != null) return byName;
     throw new NoSuchMethodException("selectMethod(Class, String, Class[])");
   }
 

package/dist/mutate/run.js

@@ -263,7 +263,8 @@ async function runOne(worktree, pkg, patchPath, options, commit, runId, prep) {
                 return finishMutant(killed(base, direction, ["build"]), pkg, "", false, []);
         }
         const picked = selectionFor(prep.coverage, edited.length > 0 ? edited : files);
-        if (picked.state === "uncovered") {
+        const forced = options.onlyNames && options.onlyNames.length > 0 ? options.onlyNames : null;
+        if (!forced && picked.state === "uncovered") {
             return {
                 ...base(),
                 direction,
@@ -271,7 +272,7 @@ async function runOne(worktree, pkg, patchPath, options, commit, runId, prep) {
                 next: noCoverageNext(picked.file, picked.line),
             };
         }
-        const names = picked.state === "covered" ? picked.tests.filter((name) => name !== WHOLE_PROGRAM_TEST) : null;
+        const names = forced ?? (picked.state === "covered" ? picked.tests.filter((name) => name !== WHOLE_PROGRAM_TEST) : null);
         const forceWhole = picked.state === "covered" && (names === null || names.length === 0);
         // node --test loads every file it is given, so a name pattern still pays for the other files.
         const selected = narrowNodeFiles(pkg, selectTests(files, prep.tests, prep.affectedMap), names, prep.spec.kind);
@@ -927,15 +928,28 @@ function findTestFile(pkg, rel) {
 function sha256(text) {
     return createHash("sha256").update(text).digest("hex");
 }
+const BARE_SUITE = new Set(["command", "suite", "pytest", "unittest", "cobol", "make-test", "c", "cpp"]);
 function failureId(pkg, failure) {
     // Pytest already reports a node id (tests/test_gate.py::test_low). Prepending the
     // file doubles it, and pytest then exits 4 and collects nothing.
     if (pytestNodeId(failure.name))
         return failure.name;
-    const rel = failure.file ? path.relative(pkg, failure.file) : "";
-    const file = !rel || rel.startsWith("..") ? path.basename(failure.file || "") : rel.split(path.sep).join("/");
+    // Cargo and dotnet report a bare test name. Go reports a package path that is
+    // not a file, and JUnit reports a classname. The runnable id is the name itself.
+    // An unnamed shell suite stays ::command so a tally does not treat it as a test.
+    if (!failure.file)
+        return BARE_SUITE.has(failure.name) ? `::${failure.name}` : failure.name;
+    if (!testFileOnDisk(pkg, failure.file))
+        return failure.name;
+    const rel = path.relative(pkg, failure.file);
+    const file = !rel || rel.startsWith("..") ? path.basename(failure.file) : rel.split(path.sep).join("/");
     return `${file}::${failure.name}`;
 }
+function testFileOnDisk(pkg, file) {
+    if (path.isAbsolute(file))
+        return existsSync(file);
+    return existsSync(path.join(pkg, file));
+}
 function pytestNodeId(name) {
     const mark = name.indexOf("::");
     if (mark <= 0)

package/dist/mutate/suites.js

@@ -435,6 +435,10 @@ function pytestLeaf(name) {
 function pytestPlan(selection) {
     if (selection.length === 0)
         return { pattern: null, nodeIds: null };
+    // A doubled keep id (file.py::tests/file.py::test_low) has more than one ::.
+    // Reducing it to the leaf would run -k test_low and still kill the mutant.
+    if (selection.some((name) => name.split("::").length > 2))
+        return { pattern: null, nodeIds: selection };
     const leaves = selection.map(pytestLeaf);
     if (leaves.every((leaf) => PYTEST_KEYWORD.test(leaf)))
         return { pattern: leaves.join(" or "), nodeIds: null };
@@ -464,9 +468,9 @@ function goRunFilter(names, pattern) {
 function surefireTest(name) {
     const leaf = name.split("::").pop() ?? name;
     const dot = leaf.lastIndexOf(".");
-    if (dot <= 0)
-        return leaf;
-    return `${leaf.slice(0, dot)}#${leaf.slice(dot + 1)}`;
+    const body = dot <= 0 ? leaf : `${leaf.slice(0, dot)}#${leaf.slice(dot + 1)}`;
+    // testShut[0] collects nothing. method(Parser)[1] keeps the parameter types.
+    return body.replace(/\[[^\]]*\]$/, "");
 }
 function escapeRegExp(value) {
     return value.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
@@ -1148,6 +1152,9 @@ function adapt(kind, pkg, stdout, stderr, code) {
         return parseMocha(stdout);
     if (kind === "make-test" || kind === "command") {
         const text = `${stdout}\n${stderr}`;
+        const ctest = parseCtest(text);
+        if (ctest)
+            return ctest;
         if (/Tests run:\s*\d+/.test(text))
             return parseMakeTest(text);
         if (kind === "make-test")
@@ -1267,6 +1274,32 @@ function parseMocha(stdout) {
         return null;
     }
 }
+/** CTest prints `Test #N: name` and `N - name (Failed)`. Those names are the kill, not `::command`. */
+function parseCtest(text) {
+    if (!/Test #\d+:|The following tests FAILED:/.test(text))
+        return null;
+    const names = [];
+    const failed = [];
+    for (const raw of text.split(/\r?\n/)) {
+        const line = raw.trim();
+        const ran = /^\d+\/\d+\s+Test\s+#\d+:\s+(\S+)/.exec(line);
+        if (ran && !names.includes(ran[1]))
+            names.push(ran[1]);
+        if (ran && /\*\*\*Failed/.test(line) && !failed.some((item) => item.name === ran[1])) {
+            failed.push({ name: ran[1], file: "", line: 0 });
+        }
+        const listed = /^\d+\s+-\s+(\S+)\s+\(Failed\)/.exec(line);
+        if (listed && !failed.some((item) => item.name === listed[1]))
+            failed.push({ name: listed[1], file: "", line: 0 });
+    }
+    if (names.length === 0 && failed.length === 0)
+        return null;
+    for (const item of failed) {
+        if (!names.includes(item.name))
+            names.push(item.name);
+    }
+    return { tests: names.length, pass: Math.max(names.length - failed.length, 0), fail: failed.length, failed, names };
+}
 function parseMakeTest(text) {
     const run = text.match(/Tests run:\s*(\d+)/);
     if (!run)

package/dist/mutate/tally.js

@@ -105,7 +105,7 @@ export function tallyRun(outDir) {
         command: "mutate.tally",
         summary,
         next,
-        nextCall: null,
+        nextCall: keep.length > 0 ? { argv: ["mutate", "run", ...keep.flatMap((id) => ["--only-test", id])] } : null,
         keep,
         drop,
         gaps,

package/dist/verify/change.js

@@ -7,7 +7,7 @@ import { findInSource } from "../mutate/find.js";
 import { mutantId } from "../mutate/ids.js";
 import { forwardDiff, git } from "../mutate/patch.js";
 import { runMutants } from "../mutate/run.js";
-import { leadSummary } from "../mutate/suite-decision.js";
+import { WHOLE_PROGRAM_TEST, leadSummary } from "../mutate/suite-decision.js";
 import { discoverSuite } from "../mutate/suites.js";
 /** Diff-scoped mutants, the tests that can see the edit, and golden contract rows that changed. */
 export async function verifyChange(options) {
@@ -102,15 +102,17 @@ export async function verifyChange(options) {
         other = results
             .filter((item) => item.outcome !== "killed" && item.outcome !== "survived")
             .map((item) => ({ id: item.id, outcome: item.outcome, file: byId.get(item.id)?.file ?? "", line: byId.get(item.id)?.line ?? 0 }));
+        const executed = executedNames(results);
+        const envelope = bodyOf(options.outDir, true, affected, ran, notRan, caught, missed, other, goldenContract, notRun, suiteCommand, executed);
+        writeFileSync(path.join(options.outDir, "verify.json"), `${JSON.stringify(envelope, null, 2)}\n`);
+        return envelope;
     }
-    else {
-        notRun.push(...written);
-    }
-    const envelope = bodyOf(options.outDir, true, affected, ran, notRan, caught, missed, other, goldenContract, notRun, suiteCommand);
+    notRun.push(...written);
+    const envelope = bodyOf(options.outDir, true, affected, ran, notRan, caught, missed, other, goldenContract, notRun, suiteCommand, []);
     writeFileSync(path.join(options.outDir, "verify.json"), `${JSON.stringify(envelope, null, 2)}\n`);
     return envelope;
 }
-function bodyOf(outDir, ok, affected, ran, notRan, caught, missed, other, goldenContract, notRun, suiteCommand) {
+function bodyOf(outDir, ok, affected, ran, notRan, caught, missed, other, goldenContract, notRun, suiteCommand, executed) {
     const uncovered = other.filter((item) => item.outcome === "no coverage");
     const summary = ok
         ? leadSummary(uncovered.length, missed.length, "missed", `${caught.length} caught, ${notRun.length} not run, ${affected.length} affected tests, ${goldenContract.length} golden contract changes.`)
@@ -127,6 +129,8 @@ function bodyOf(outDir, ok, affected, ran, notRan, caught, missed, other, golden
         ran,
         notRan,
         suiteCommand,
+        executed,
+        confirmed: false,
         caught,
         missed,
         other,
@@ -158,6 +162,8 @@ function fail(outDir, summary, extra) {
         ran: extra?.ran ?? [],
         notRan: extra?.notRan ?? [],
         suiteCommand: extra?.suiteCommand ?? "",
+        executed: [],
+        confirmed: false,
         caught: [],
         missed: [],
         other: [],
@@ -173,14 +179,17 @@ function readOutcomes(outDir, ids) {
     for (const id of ids) {
         try {
             const result = JSON.parse(readFileSync(path.join(file, `${id}.json`), "utf8"));
-            out.push({ id, outcome: result.outcome });
+            out.push({ id, outcome: result.outcome, selectedTests: result.selectedTests ?? [] });
         }
         catch {
-            out.push({ id, outcome: "missing" });
+            out.push({ id, outcome: "missing", selectedTests: [] });
         }
     }
     return out;
 }
+function executedNames(results) {
+    return [...new Set(results.flatMap((item) => item.selectedTests))].filter((name) => name.length > 0 && name !== WHOLE_PROGRAM_TEST).sort();
+}
 function goldenDiffs(repo, base, commit, files) {
     const found = [];
     for (const file of files) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "probatio",
-  "version": "0.0.5",
+  "version": "0.1.0",
   "description": "Mutation runner for AI agents. Scores a test suite by the bugs it catches.",
   "type": "module",
   "bin": {