tewip-ai 0.60.0 → 0.62.0

package/README.md

@@ -23,6 +23,41 @@ Formerly *triage-agent*: the repository moved to `hmamut39/tewip`, and GitHub re
 **Status: phases 1 to 14 are built; the two auto-fix phases (5 and 8) are the ones still in progress.** Tewip collects failures from any framework, diagnoses them with rules first and a model only for what rules cannot settle, and turns that into work for each role: a CI comment and merge gate, test plans and verified tests, a flake dossier, a PR-risk review, security scanning, tickets in Jira, Rally, GitHub Issues, Azure Boards or Linear, checks generated from a Figma design, specifications written for people, and - opt in, and only when proved - fixes to test or application code. It runs live in GitHub CI and answers questions from AI assistants through MCP. On fresh real Gutenberg CI failures, with the pipeline frozen before labelling, accuracy is 93.4%; across six real datasets it ranges from 86.1% to 99.7%, and the caveats are in [eval/README.md](eval/README.md).
 
 
+## If your CI already writes a coverage report, point Tewip at it
+
+Tewip decides what a test reaches by following imports, which is a fact rather than a name match -
+and which cannot see a test that posts a GraphQL query, hits an HTTP route or renders a controller.
+On a Django, Rails or Spring codebase that is most of the suite, so `tewip done` used to answer
+*"cannot say"* to the question that matters most.
+
+A coverage report settles it, and CI almost certainly produces one already. Nothing to configure:
+
+| format | written by | found at |
+|---|---|---|
+| `lcov.info` | JavaScript, TypeScript, Rust, Go, C++ | `coverage/lcov.info`, `lcov.info` |
+| Cobertura `coverage.xml` | pytest-cov, coverlet (.NET), simplecov | `coverage.xml`, `coverage/coverage.xml` |
+| JaCoCo | Java, Kotlin | `target/site/jacoco/jacoco.xml`, `build/reports/...` |
+| Clover | PHPUnit | `clover.xml`, `build/logs/clover.xml` |
+
+With one present, `done` stops guessing:
+
+```
+### Is the changed code tested?
+
+Yes: the suite executes saleor/graphql/utils/sorting.py.
+
+_Decided by coverage.xml, which records what ran rather than what imports what._
+```
+
+and it will say the opposite just as plainly - *"is in coverage.xml with no line executed, so the
+suite runs none of this"* is a **not ready**, not a shrug.
+
+**What a coverage report cannot do.** It records which lines ran during the whole suite, not which
+test ran them, so it answers *"is this code tested"* and not *"which tests would notice a change
+here"*. `tewip pick` still narrows from imports and still refuses to narrow when it cannot show the
+rest are unaffected. A report from a different checkout is ignored rather than believed: if the files
+it names are not on disk, Tewip says nothing instead of reporting the whole change as untested.
+
 ## Running with no data leaving the building
 
 A hospital, a bank or a government department usually cannot send source code to a hosted API, and

package/dist/engine/coverage/read-coverage.js

@@ -0,0 +1,194 @@
+// What the tests actually executed, read from a report the project already produces.
+//
+// Everything else in Tewip follows imports to decide what a test reaches, and Phases 46 and 52 found
+// the same wall twice: a test that posts a GraphQL query, hits an HTTP route or renders a Rails
+// controller never imports the code it exercises. `done` had to answer "cannot say" and `pick` had to
+// refuse to narrow - honest both times, and useless to a Python or Rails team.
+//
+// A coverage report settles it, and CI already makes one. This reads the four formats that between
+// them cover most of what runs anywhere:
+//
+//   lcov.info      JavaScript, TypeScript, C, C++, Rust, Go
+//   coverage.xml   Cobertura: pytest-cov, Jacoco's cobertura output, coverlet for .NET, simplecov
+//   jacoco.xml     Java and Kotlin
+//   clover.xml     PHPUnit
+//
+// What a coverage report can and cannot say is worth being exact about. It records which lines ran
+// during the whole suite - not which test ran them. So it answers "is this code executed by the
+// tests at all", which is what `done` asks, and it does **not** answer "which tests would notice a
+// change here", which is what `pick` asks. Only a per-test context report answers that, and this
+// reads one when the project produced it.
+import { existsSync, readFileSync, readdirSync, statSync } from 'node:fs';
+import { join, relative, resolve } from 'node:path';
+/** The names CI tools write, in the order worth trying. */
+const REPORTS = [
+    'coverage/lcov.info',
+    'lcov.info',
+    'coverage.xml',
+    'coverage/coverage.xml',
+    'target/site/jacoco/jacoco.xml',
+    'build/reports/jacoco/test/jacocoTestReport.xml',
+    'jacoco.xml',
+    'clover.xml',
+    'coverage/clover.xml',
+    'build/logs/clover.xml',
+];
+const slash = (p) => p.split(String.fromCharCode(92)).join('/');
+/** The coverage report this project has produced, or null when it has not produced one. */
+export function findCoverage(root) {
+    for (const name of REPORTS) {
+        const path = join(root, name);
+        if (!existsSync(path))
+            continue;
+        try {
+            if (statSync(path).size > 64 * 1024 * 1024)
+                continue;
+            const text = readFileSync(path, 'utf8');
+            const parsed = name.endsWith('.info') ? readLcov(text, root) : readXml(text, root);
+            if (parsed.mentioned.size > 0)
+                return { from: slash(relative(root, path)), ...parsed };
+        }
+        catch {
+            continue;
+        }
+    }
+    return null;
+}
+/** LCOV: `SF:` opens a file, `DA:<line>,<hits>` records one line. */
+export function readLcov(text, root = '') {
+    const executed = new Map();
+    const mentioned = new Set();
+    let file = '';
+    for (const raw of text.split(String.fromCharCode(10))) {
+        const line = raw.trim();
+        if (line.startsWith('SF:')) {
+            file = normalise(line.slice(3), root);
+            mentioned.add(file);
+            if (!executed.has(file))
+                executed.set(file, new Set());
+            continue;
+        }
+        if (!file || !line.startsWith('DA:'))
+            continue;
+        const [at, hits] = line.slice(3).split(',');
+        if (Number(hits) > 0)
+            executed.get(file)?.add(Number(at));
+    }
+    return { executed, mentioned };
+}
+/**
+ * Cobertura, JaCoCo and Clover in one reader, because all three say the same thing in the same
+ * shape: a filename attribute, then elements carrying a line number and a hit count.
+ *
+ * Read with regular expressions rather than an XML parser on purpose - these files reach hundreds of
+ * megabytes on a large suite, and nothing here needs the tree.
+ */
+export function readXml(text, root = '') {
+    const executed = new Map();
+    const mentioned = new Set();
+    // Cobertura nests <class filename="x">; JaCoCo has <sourcefile name="x"> inside <package name="y">;
+    // Clover has <file path="x">. Each opens a file and owns the line elements until the next one.
+    const opens = /<(?:class[^>]*filename|sourcefile[^>]*name|file[^>]*(?:path|name))\s*=\s*"([^"]+)"/g;
+    const starts = [];
+    for (const m of text.matchAll(opens))
+        starts.push({ at: m.index ?? 0, file: m[1] });
+    // JaCoCo names the package separately and the sourcefile without its directory, so the two are
+    // joined back together. Without this every file reads as a bare name that matches nothing.
+    const packages = [...text.matchAll(/<package[^>]*name\s*=\s*"([^"]+)"/g)].map((m) => ({ at: m.index ?? 0, name: m[1] }));
+    const packageFor = (at) => {
+        let found = '';
+        for (const p of packages) {
+            if (p.at > at)
+                break;
+            found = p.name;
+        }
+        return found;
+    };
+    for (const [i, start] of starts.entries()) {
+        const until = starts[i + 1]?.at ?? text.length;
+        const within = text.slice(start.at, until);
+        const inPackage = /<sourcefile/.test(text.slice(start.at, start.at + 12)) ? packageFor(start.at) : '';
+        const file = normalise(inPackage ? `${inPackage}/${start.file}` : start.file, root);
+        mentioned.add(file);
+        if (!executed.has(file))
+            executed.set(file, new Set());
+        // Cobertura: <line number="3" hits="1"/>  JaCoCo: <line nr="3" ci="2"/>  Clover: <line num="3" count="1"/>
+        for (const line of within.matchAll(/<line[^>]*\b(?:number|nr|num)\s*=\s*"(\d+)"[^>]*\b(?:hits|ci|count)\s*=\s*"(\d+)"/g)) {
+            if (Number(line[2]) > 0)
+                executed.get(file)?.add(Number(line[1]));
+        }
+    }
+    return { executed, mentioned };
+}
+/**
+ * The report's path for a source file, as a path relative to the project and slashed.
+ *
+ * Reports disagree about this more than about anything else: lcov writes an absolute path, Cobertura
+ * a path relative to a `<source>` root it declares separately, JaCoCo a package and a bare file name.
+ * Everything here is reduced to the same shape so that a changed file from a git diff can be looked
+ * up without the caller knowing which tool wrote the report.
+ */
+function normalise(file, root) {
+    const path = slash(file).replace(/^\.\//, '');
+    if (!root)
+        return path;
+    const full = slash(resolve(root, path));
+    const base = slash(resolve(root));
+    return full.startsWith(`${base}/`) ? full.slice(base.length + 1) : path;
+}
+/** True when the report shows this file running at least one line. */
+export function wasExecuted(coverage, file) {
+    return (coverage.executed.get(slash(file))?.size ?? 0) > 0;
+}
+/**
+ * Whether the report covers this project at all.
+ *
+ * A stale report from a different checkout, or one whose paths do not line up, would otherwise be
+ * read as "nothing in this change is tested" - which is the confident wrong answer this whole
+ * exercise exists to avoid. If none of the files it names exist on disk, it is not about this tree.
+ */
+export function describesThisProject(coverage, root) {
+    let found = 0;
+    let checked = 0;
+    for (const file of coverage.mentioned) {
+        if (checked >= 20)
+            break;
+        checked += 1;
+        if (existsSync(join(root, file)))
+            found += 1;
+    }
+    return checked > 0 && found / checked >= 0.5;
+}
+/** Any coverage report under the project, for a report that wants to say where it looked. */
+export function reportsUnder(root, limit = 400) {
+    const out = [];
+    const walk = (dir, depth) => {
+        if (depth > 4 || out.length >= limit)
+            return;
+        let entries;
+        try {
+            entries = readdirSync(dir);
+        }
+        catch {
+            return;
+        }
+        for (const entry of entries) {
+            if (entry === 'node_modules' || entry === '.git' || entry.startsWith('.'))
+                continue;
+            const full = join(dir, entry);
+            let info;
+            try {
+                info = statSync(full);
+            }
+            catch {
+                continue;
+            }
+            if (info.isDirectory())
+                walk(full, depth + 1);
+            else if (/^(lcov\.info|coverage\.xml|jacoco.*\.xml|clover\.xml)$/i.test(entry))
+                out.push(slash(relative(root, full)));
+        }
+    };
+    walk(root, 0);
+    return out;
+}

package/dist/engine/pick/pick-command.js

@@ -115,8 +115,27 @@ function mentionsOf(root, file, tests) {
     return out;
 }
 /** Does anything at all in the project refer to this file? Used only to tell "unknown" from "unreachable". */
+/**
+ * Whether anything in the project refers to this file at all.
+ *
+ * This searched for the file name *with its extension*, which is a JavaScript habit: a JS import
+ * often writes `./sorting.js`, and nothing else does. Python writes `from ..utils.sorting import x`,
+ * Java writes `import com.x.Sorting`, C# writes `using X.Sorting`. In all of them the extension
+ * never appears, so the search found nothing and the file was taken to be referenced nowhere.
+ *
+ * That is the most dangerous answer this file can produce. "Nothing refers to it" is the one
+ * condition that lets selection proceed past a source file no test imports - and on saleor it let
+ * `pick` narrow 1,229 test files down to **one**, skipping every sorting test in the suite, for a
+ * change to the GraphQL sorting code that all of them run through. The command's own promise is to
+ * refuse to narrow when it cannot show the rest are unaffected, and here it could not.
+ *
+ * The extension comes off. A name like `utils` or `index` will now match half the repository, and
+ * that is the right direction to be wrong in: over-matching means "this is referenced", which stops
+ * the narrowing. Under-matching skips the tests that would have caught the change.
+ */
 function mentionedAnywhere(root, file) {
-    const named = file.split('/').pop() ?? '';
+    const base = file.split('/').pop() ?? '';
+    const named = base.replace(/\.[A-Za-z0-9]+$/, '') || base;
     const where = spawnSync('git', ['grep', '-l', '--', named], { cwd: root, encoding: 'utf8', timeout: 60_000 });
     const hits = (where.stdout ?? '')
         .split(String.fromCharCode(10))

package/dist/engine/verdict/verdict-command.js

@@ -4,6 +4,7 @@
 // only collects: what changed, what covers it, whether those tests can fail, whether they would catch a
 // bug, what is failing now, and what a kept plan says is still uncovered. Where a layer has nothing to
 // say, it is left as "not checked" rather than filled in with a guess.
+import { describesThisProject, findCoverage, wasExecuted } from '../coverage/read-coverage.js';
 import { existsSync, mkdirSync, readFileSync, readdirSync, writeFileSync } from 'node:fs';
 import { dirname, join, resolve } from 'node:path';
 import { seal } from '../attest/seal.js';
@@ -74,6 +75,18 @@ export function gather(cmd) {
     const changedSource = changed.filter((f) => SOURCE.test(f) && !IS_TEST.test(f) && existsSync(resolve(cmd.root, f)));
     const changedTests = changed.filter((f) => IS_TEST.test(f) && existsSync(resolve(cmd.root, f)));
     const known = testFiles(cmd.root);
+    // A coverage report the project already produces, if there is one. Imports cannot see a test that
+    // posts a GraphQL query, hits an HTTP route or renders a controller - and Phases 46 and 52 both
+    // ended at "cannot say" on every real Saleor pull request because of it. A coverage report records
+    // what ran, so it answers the question imports could not.
+    const report = findCoverage(cmd.root);
+    const coverage = report && describesThisProject(report, cmd.root)
+        ? {
+            from: report.from,
+            executed: changedSource.filter((f) => wasExecuted(report, f)),
+            notExecuted: changedSource.filter((f) => report.mentioned.has(f.split('\\').join('/')) && !wasExecuted(report, f)),
+        }
+        : undefined;
     const covering = [];
     for (const file of changedSource) {
         const direct = importingTests(cmd.root, coverageFor(cmd.root, [file], known), file);
@@ -107,6 +120,7 @@ export function gather(cmd) {
             changedSource,
             changedTests,
             covering,
+            coverage,
             cannotFail: vouched.cannotFail.map((j) => ({ name: j.name, file: j.file, line: j.line, why: j.why })),
             crashOnly: vouched.crashOnly.length,
             unnoticed: proved ? proved.proofs.flatMap((p) => p.tried.filter((t) => t.fate === 'unnoticed').map((t) => ({ file: p.file, line: t.mutant.line, kind: t.mutant.kind }))) : null,

package/dist/engine/verdict/verdict.js

@@ -25,6 +25,27 @@ export function decide(evidence) {
     if (evidence.changedSource.length === 0) {
         checks.push({ question: 'Is the changed code tested?', standing: 'not-applicable', answer: 'this change touches no source file', decidedBy: 'the diff' });
     }
+    else if (evidence.coverage && evidence.coverage.notExecuted.length > 0) {
+        // The report names these files and never ran a line of them. That is not "cannot say" - it is
+        // the suite being measured and found not to reach this code.
+        checks.push({
+            question: 'Is the changed code tested?',
+            standing: 'not-met',
+            answer: `${evidence.coverage.notExecuted.slice(0, 3).join(', ')} ${evidence.coverage.notExecuted.length === 1 ? 'is' : 'are'} in ${evidence.coverage.from} with no line executed, so the suite runs none of this`,
+            decidedBy: `${evidence.coverage.from}, which records what ran rather than what imports what`,
+        });
+    }
+    else if (evidence.coverage && evidence.coverage.executed.length > 0) {
+        // Settles the question imports could not. Phases 46 and 52 both ended at "cannot say" here, on
+        // every real Saleor pull request, because a test that posts a GraphQL query never imports the
+        // resolver chain it exercises.
+        checks.push({
+            question: 'Is the changed code tested?',
+            standing: 'met',
+            answer: `the suite executes ${evidence.coverage.executed.slice(0, 3).join(', ')}${evidence.coverage.executed.length > 3 ? ` and ${evidence.coverage.executed.length - 3} more` : ''}`,
+            decidedBy: `${evidence.coverage.from}, which records what ran rather than what imports what`,
+        });
+    }
     else if (evidence.covering.length === 0 && evidence.changedTests.length > 0) {
         // Nothing *imports* the changed file, and the same change ships tests. Coverage here is followed
         // through imports, which is a fact rather than a name match - and which cannot see a test that

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "tewip-ai",
-  "version": "0.60.0",
+  "version": "0.62.0",
   "mcpName": "io.github.hmamut39/tewip",
   "description": "Tewip finds why each failing test failed - real bug, flaky test, environment, test data or changed selector - for any stack, from the command line, CI, or your editor.",
   "keywords": [