supercov 3.0.0 → 3.0.2

package/docs/assertions.md

@@ -43,8 +43,13 @@ times out, or the process crashes. The change depends on the statement:
 | a `when`, `switch`, `try`, `loop` or Kotlin `throw` a function ends on | every value it returns becomes that stand-in (skipping it would leave the function nothing to return) |
 | declaration or assignment with a value | the value becomes that stand-in, without being evaluated |
 | JSX expression | the value becomes undefined |
+| `return true`/`false`, or a declaration of `true`/`false` (JavaScript, TypeScript, Python) | the boolean becomes the other one: `false` becoming `undefined` changes nothing a truthiness check reads |
+| `return undefined` or `return None` | skipped, since returning the stand-in would be the same code |
+| `throw` in TypeScript | throws a bare `new Error()` instead: deleting a throw seldom compiles, because it usually guards the narrowing the next line relies on |
 | anything else | skipped: the statement does not run |
 
+A constructor's `super(...)` is not assessed: skipping it does not compile.
+
 The stand-in is `undefined` in JavaScript and TypeScript, `None` in Python and
 `nil` in Ruby. Go and the JVM languages replace a value after it is computed,
 as their mutation testers do: the zero value in Go, and `null`, `0` or `false`
@@ -96,7 +101,7 @@ npx supercov runs latest assertions --test "closes the stream"  # what one test
 ```text
 Run run_4f2a: 93.8% asserted (2731 of 2913 executed statements)
 
-Files, most statements not asserted first: a statement is not asserted when no test that runs it was judged to fail if it changed.
+Files, most statements not asserted first: a statement is not asserted when none of the tests asked about it was judged to fail if it changed. Up to 15 of the tests that run a statement are asked, so a test never asked can still catch it.
  NOT ASSERTED  ASSERTED  FILE
            41   388/429  src/adapters.ts
            ...
@@ -110,8 +115,13 @@ down, so an agent reads only as much as it needs. `runs latest file` and
 `runs latest line` show the same verdicts beside the coverage, and
 `npx supercov runs latest` and the HTML report show the same share.
 
-A statement that is not asserted is a place to add a check: a test that runs it
-exists, but nothing it asserts depends on what the statement does.
+A statement that is not asserted is a place to add a check: tests run it, but
+nothing the asked tests assert depends on what the statement does. Five of the
+tests that run a statement are asked first, and up to 15 when none of those
+catches it: tests whose file quotes text the statement writes (a log line, a
+message), then tests named like the source file, then one per test file and
+describe block. A statement many tests run can still be caught by one that was
+never asked; the statement view (`assertions <file>:<line>`) says how many were.
 
 ## Which tests check a change
 
@@ -130,6 +140,10 @@ test or its helpers, or to which tests run it, asks again. A dependency or
 configuration change asks everything again. After a typical commit only a few
 dozen questions are asked; the rest are reused.
 
+A reused answer is the one Jev gave before, right or wrong. When you have shown
+an answer wrong, `assess --refresh` asks every question again and keeps the new
+answers.
+
 ## Cost, speed and accuracy
 
 - **Cost:** about 0.006 cents per statement for a first assessment: 14 projects

package/docs/cli.md

@@ -182,6 +182,7 @@ after source changes.
 ```sh supercov-example
 npx supercov runs <run-id> assertions assess --dry-run
 npx supercov runs <run-id> assertions assess
+npx supercov runs <run-id> assertions assess --refresh   # ask again instead of reusing saved answers
 npx supercov runs <run-id> assertions
 npx supercov runs <run-id> assertions --all --json
 ```
@@ -349,13 +350,13 @@ up to ten recent runs. Selecting a run compares it with the previous saved run
 automatically.
 
 ```sh
-npx supercov report --runs 5
+npx supercov report --runs 10
 npx supercov report --output artifacts/supercov-report.html --no-open
 ```
 
-Use `--runs N` to include up to 20 stored runs. Reports are capped at 24 MB so
-they remain suitable for a normal pull-request attachment; use `--runs 1` if a
-large project exceeds that target. See [Portable HTML reports](reports.md) for
+A report includes the five newest runs; `--runs N` includes up to 20. A large
+project's report can pass GitHub's 25 MB attachment limit; `--runs 1` makes
+the smallest. See [Portable HTML reports](reports.md) for
 the privacy and source-integrity rules.
 
 ## Find a smaller test set

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "supercov",
-  "version": "3.0.0",
+  "version": "3.0.2",
   "description": "Coverage, security and code quality for coding agents",
   "keywords": [
     "coverage",
@@ -118,14 +118,14 @@
     "test:windows-tls": "node scripts/windows-tls-test.mjs"
   },
   "optionalDependencies": {
-    "@supercov/cli-darwin-arm64": "3.0.0",
-    "@supercov/cli-darwin-x64": "3.0.0",
-    "@supercov/cli-linux-arm64-gnu": "3.0.0",
-    "@supercov/cli-linux-arm64-musl": "3.0.0",
-    "@supercov/cli-linux-x64-gnu": "3.0.0",
-    "@supercov/cli-linux-x64-musl": "3.0.0",
-    "@supercov/cli-win32-arm64": "3.0.0",
-    "@supercov/cli-win32-x64": "3.0.0"
+    "@supercov/cli-darwin-arm64": "3.0.2",
+    "@supercov/cli-darwin-x64": "3.0.2",
+    "@supercov/cli-linux-arm64-gnu": "3.0.2",
+    "@supercov/cli-linux-arm64-musl": "3.0.2",
+    "@supercov/cli-linux-x64-gnu": "3.0.2",
+    "@supercov/cli-linux-x64-musl": "3.0.2",
+    "@supercov/cli-win32-arm64": "3.0.2",
+    "@supercov/cli-win32-x64": "3.0.2"
   },
   "peerDependencies": {
     "@playwright/test": ">=1.55.0",

package/runtime/javascript/nodeTest.mjs

@@ -93,19 +93,30 @@ function restoreUserError(error, depth = 0) {
 // gives what --test-name-pattern matches. A store rather than a stack: an
 // async suite registers tests after it awaits.
 const suitePath = new AsyncLocalStorage();
+// A suite marked todo or skip marks every test inside it, as node:test counts
+// them: `describe.todo` tests run, and are todo whatever they do.
+const suiteMark = new AsyncLocalStorage();
+function markOf(options, mark) {
+    if (mark === "skipped" || options?.skip)
+        return "skipped";
+    if (mark === "todo" || options?.todo)
+        return "todo";
+    return undefined;
+}
 function suiteName(args, callback) {
     return typeof args[0] === "string" ? args[0] : callback.name || "<anonymous>";
 }
-function wrappedSuite(original) {
+function wrappedSuite(original, mark) {
     const wrapped = function supercovNodeSuite(...args) {
         const index = callbackIndex(args);
         if (index < 0)
             return Reflect.apply(original, this, args);
         const callback = args[index];
         const path = [...(suitePath.getStore() ?? []), suiteName(args, callback)];
+        const marked = suiteMark.getStore() ?? markOf(testOptions(args, index), mark);
         const next = [...args];
         next[index] = function supercovNodeSuiteCallback(...callbackArgs) {
-            return suitePath.run(path, () => Reflect.apply(callback, this, callbackArgs));
+            return suitePath.run(path, () => suiteMark.run(marked, () => Reflect.apply(callback, this, callbackArgs)));
         };
         // node:test reports a suite at its direct caller, as it does a test.
         const registration = registrationAt(callerLocation(supercovNodeSuite));
@@ -119,7 +130,7 @@ function wrappedSuite(original) {
             Object.defineProperty(wrapped, property, {
                 configurable: true,
                 enumerable: true,
-                value: wrappedSuite(member),
+                value: wrappedSuite(member, property === "only" ? undefined : property === "skip" ? "skipped" : "todo"),
             });
     }
     return wrapped;
@@ -128,9 +139,9 @@ const registrationCounts = new Map();
 function wrappedRegistration(original, parentTestId, forcedStatus) {
     const wrapped = function supercovNodeTest(...args) {
         const index = callbackIndex(args);
-        if (index < 0)
-            return Reflect.apply(original, this, args);
-        const callback = args[index];
+        // `test.todo("name")` and `it.todo("name")` have no body and are still
+        // tests the runner counts.
+        const callback = index < 0 ? {} : args[index];
         const location = callerLocation(supercovNodeTest);
         const identity = {
             runner: "node:test",
@@ -153,19 +164,27 @@ function wrappedRegistration(original, parentTestId, forcedStatus) {
         registrationCounts.set(registrationKey, registrationOrdinal + 1);
         identity.registrationOrdinal = registrationOrdinal;
         const scope = runnerExecutionScope(identity);
-        const options = testOptions(args, index);
+        const options = testOptions(args, index < 0 ? args.length : index);
         const evidenceDirectory = process.env["SUPERCOV_EVIDENCE_DIR"];
-        if (options?.skip || options?.todo || forcedStatus)
-            writeRunnerEvidence(identity, "skipped", scope, evidenceDirectory);
+        const marked = suiteMark.getStore() ?? markOf(options, forcedStatus);
+        if (index < 0) {
+            writeRunnerEvidence(identity, marked ?? "passed", scope, evidenceDirectory);
+            return Reflect.apply(original, this, args);
+        }
+        if (marked)
+            writeRunnerEvidence(identity, marked, scope, evidenceDirectory);
         const next = [...args];
         const execute = (callbackThis, context, done) => {
             // A test or its hooks may intentionally modify Supercov's public
             // environment while testing integrations. Keep this attempt's transport
             // destination fixed to the value present when the test was registered.
             beginBufferedServerEvidence(scope);
-            let status = options?.skip || options?.todo || forcedStatus
-                ? "skipped"
-                : "passed";
+            // A todo test runs, and stays todo whether its body passes or fails.
+            let status = marked ?? "passed";
+            const fail = () => {
+                if (status !== "todo")
+                    status = "failed";
+            };
             const contextProxy = new Proxy(context, {
                 get(target, property) {
                     // Read with the real context as receiver: TestContext accessors
@@ -174,7 +193,7 @@ function wrappedRegistration(original, parentTestId, forcedStatus) {
                     const value = Reflect.get(target, property, target);
                     if ((property === "skip" || property === "todo") && typeof value === "function") {
                         return (...callArgs) => {
-                            status = "skipped";
+                            status = property === "todo" ? "todo" : "skipped";
                             return Reflect.apply(value, target, callArgs);
                         };
                     }
@@ -202,7 +221,7 @@ function wrappedRegistration(original, parentTestId, forcedStatus) {
                 if (callback.length >= 2) {
                     const callbackDone = (error) => {
                         if (error) {
-                            status = "failed";
+                            fail();
                             restoreUserError(error);
                         }
                         finishBody();
@@ -216,7 +235,7 @@ function wrappedRegistration(original, parentTestId, forcedStatus) {
                         finishBody();
                         return value;
                     }, (error) => {
-                        status = "failed";
+                        fail();
                         finishBody();
                         throw restoreUserError(error);
                     });
@@ -224,7 +243,7 @@ function wrappedRegistration(original, parentTestId, forcedStatus) {
                 return result;
             }
             catch (error) {
-                status = "failed";
+                fail();
                 finishBody();
                 throw restoreUserError(error);
             }
@@ -249,7 +268,7 @@ function wrappedRegistration(original, parentTestId, forcedStatus) {
             Object.defineProperty(wrapped, property, {
                 configurable: true,
                 enumerable: true,
-                value: wrappedRegistration(member, parentTestId, property === "only" ? undefined : "skipped"),
+                value: wrappedRegistration(member, parentTestId, property === "only" ? undefined : property === "skip" ? "skipped" : "todo"),
             });
     }
     return wrapped;

package/runtime/javascript/playwrightReporter.mjs

@@ -1,13 +1,77 @@
-import { relative, resolve, sep } from "node:path";
+import { readFileSync } from "node:fs";
+import { createRequire } from "node:module";
+import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path";
 import { inferTestProvenance } from "./provenance.mjs";
 import { appendJsonLineDurableSync, appendJsonLineSync } from "./atomic.mjs";
 const GENERATED_EVIDENCE_DIRECTORY = "__SUPERCOV_EVIDENCE_DIRECTORY__";
 const evidenceWriterIdentity = () => (process.env.SUPERCOV_EXECUTION_LOG_SHARD ?? `pid-${process.pid}`)
     .replace(/[^A-Za-z0-9_-]/g, "_");
+// Playwright's own code frame builder, from the copy that runs the tests.
+let frameBuilder;
+function codeFrame() {
+    if (frameBuilder !== undefined)
+        return frameBuilder;
+    try {
+        const require = createRequire(join(process.cwd(), "noop.js"));
+        const root = dirname(require.resolve("playwright/package.json"));
+        frameBuilder = require(join(root, "lib/transform/babelBundle.js")).codeFrameColumns ?? null;
+    }
+    catch {
+        frameBuilder = null;
+    }
+    return frameBuilder;
+}
+/**
+ * Playwright prints the failing line from the file it ran: Supercov's copy in
+ * the workspace, where assertions carry the wrappers that time them. Show the
+ * project's own line instead. Every line keeps its number in the copy.
+ */
+export function originalSnippet(error, workspace = process.env.SUPERCOV_PROJECT_ROOT, source = process.env.SUPERCOV_SOURCE_PROJECT_ROOT, build = codeFrame()) {
+    const location = error?.location;
+    if (!error?.snippet || !location?.file || !workspace || !source || !build)
+        return;
+    const path = relative(workspace, location.file);
+    if (!path || path.startsWith("..") || isAbsolute(path))
+        return;
+    let copied;
+    let original;
+    try {
+        copied = readFileSync(location.file, "utf8");
+        original = readFileSync(join(source, path), "utf8");
+    }
+    catch {
+        return;
+    }
+    if (copied === original)
+        return;
+    const originalLine = original.split("\n")[location.line - 1];
+    if (originalLine === undefined)
+        return;
+    // Playwright reads the copy's inline source map, so the location is the
+    // original line. The map has one point per copied run of text, so the
+    // column is the start of the statement; Playwright points at the matcher,
+    // which the message names (`expect(received).toBe(expected)`).
+    let column = location.column <= originalLine.length + 1
+        ? location.column
+        : originalLine.length - originalLine.trimStart().length + 1;
+    const matcher = /expect\(.*?\)\.(?:not\.|resolves\.|rejects\.|soft\.|poll\.)*(\w+)\(/.exec(String(error.message ?? "").replace(/\u001b\[[0-9;]*m/g, ""))?.[1];
+    const named = matcher ? originalLine.indexOf(`.${matcher}(`, column - 1) : -1;
+    if (named >= 0)
+        column = named + 2;
+    const start = { line: location.line, column };
+    const shown = build(copied, { start: location }, { highlightCode: true });
+    if (!error.snippet.includes(shown))
+        return;
+    error.snippet = error.snippet.replace(shown, build(original, { start }, { highlightCode: true }));
+}
 /** Records outcomes even when browser or fixture startup fails before coverage. */
 export default class SupercovPlaywrightReporter {
     records = [];
     onTestEnd(test, result) {
+        // Reporters print failures at the end of the run, after every
+        // reporter has seen this test.
+        for (const error of result.errors ?? [])
+            originalSnippet(error);
         const evidenceDirectory = process.env["SUPERCOV_EVIDENCE_DIR"] ??
             (GENERATED_EVIDENCE_DIRECTORY.startsWith("__")
                 ? undefined