tewip-ai 0.71.0 → 0.73.0

package/README.md

@@ -1173,9 +1173,25 @@ Two things it refuses to do, which is why the list is worth reading:
 **Recall, measured by execution** (`eval/vouch-recall.ts`): mutate a source file, record which tests
 catch which mutation, remove the last statement of the ones that provably caught something, and run it
 all again. A test that caught a mutation before and catches nothing now has provably lost its check -
-a label that owes nothing to vouch's own patterns. On 17 such cases across seven source/test pairs in
-two projects, **vouch named 12 (70.6%)**, and the five it did not name still hold a real assertion, so
-it is right about those too. Nothing that still caught a mutation was wrongly accused.
+a label that owes nothing to vouch's own patterns.
+
+Run across all 48 source/test pairs in this project (Phase 138): 28 produce labelled cases, 22 of
+those settle, and across them **vouch names 48 of 64 (75.0%)**, with **0 of 157 sound tests wrongly
+accused**. Precision is the strong half, and it is the half that matters: it did not once accuse a
+test that still catches something.
+
+A rate is printed only when two different mutant budgets label the same tests. They often do not,
+and that is the point: the first run of this reported "2 of 4 (50.0%)" at 8 mutants and **0 of 0** at
+14 mutants of the same file, because "caught nothing after the deletion" is evidence of hollowness
+only to the degree that the mutants are many. Six pairs still do not settle and are left out; they
+were the favourable ones, so 75.0% is a floor.
+
+Measuring it this way found **two real defects in vouch**, both reporting a hollow test as sound, and
+neither visible by reading the code: a one-line arrow helper was credited with assertions from the
+next block in the file, and a test fixture that merely *defines* a stub named `verify` was read as
+asserting. Fixing them took recall from 58.7% to 70.3% to 75.0%. The figure is measured on this
+project's own suite rather than on strangers' code, which is weaker evidence than the triage numbers
+above, and is the next thing to fix. See [eval/README.md](eval/README.md).
 
 ## A goal instead of a verb (Phases 19 and 21)
 

package/dist/engine/vouch/assertions.js

@@ -14,6 +14,14 @@ const CHECKS = [
     /\bshould\s*\.\s*not\b/,
     /\bassert\b\s*[.(]/,
     /\bassert(Equals?|True|False|Null|NotNull|That|Throws|Match|Contains|Rejects|Fails)\b/i,
+    // Any call named assertSomething. The list above needs the suffix to end on a word boundary, so
+    // keycloak's own `assertNullAdditionalJdbcProperty(config, null, prop)` matched nothing: `Null` is
+    // followed by `A`. Four of its ConfigurationTest tests delegate to a helper built out of those
+    // calls, and the helper was being credited only because the *string* `--db-tls-mode=verify-server`
+    // happened to contain the word `verify` - so narrowing `verify` to a call turned four correct
+    // verdicts into accusations. The corpus caught it. A call named `assertX(` is the universal
+    // convention for a custom assertion, and reading it as one is both broader and the actual reason.
+    /\bassert[A-Z]\w*\s*\(/,
     /\.\s*to\s*\.\s*(equal|be|deep|have|include|throw|match|exist)\b/,
     /\btoMatch(Snapshot|InlineSnapshot|ImageSnapshot)\s*\(/,
     /\bexpect(Type|TypeOf)\s*[<(]/,
@@ -21,8 +29,19 @@ const CHECKS = [
     // Tewip condemn all 59 tests in chalk - a third repeat of the Phase 8 lesson, so the pattern here
     // is deliberately generous about what an assertion on `t` can be called.
     /\bt\s*\.\s*(is|not|deepEqual|notDeepEqual|like|true|false|truthy|falsy|throws(Async)?|notThrows(Async)?|regex|notRegex|snapshot|pass|equal|same|strictSame|ok|notOk|match|type|has|hasStrict|rejects|resolves)\b/,
-    // Mocking libraries: verifying a call is a real check.
-    /\b(verify|verifyNoMoreInteractions|toHaveBeenCalled)\b/,
+    // Mocking libraries: verifying a call is a real check - but only when it is a call. This was a
+    // bare word match, so a fixture that *defines* a stub named `verify` matched it:
+    //   verify: async () => ({ passed: outcomes[run++], detail: 'run' }),
+    // asserts nothing, and five of this project's own autofix fixtures have exactly that line. A test
+    // whose last statement was removed and whose only remaining call was `deps(...)` was then told
+    // "it checks through deps(), which asserts" - a hollow test reported as sound. Mockito's
+    // `verify(mock)` and `Mockito.verify(mock)` both still match; a property key does not, because
+    // `: async` is not `(`. Found by measuring recall against execution, not by reading this list.
+    /\bverify(NoMoreInteractions)?\s*\(/,
+    // Kept deliberately, though `expect(fn).toHaveBeenCalled()` is already read by the `expect(`
+    // pattern above: this one is what reads the bare `spy.toHaveBeenCalled()` form. Narrowing it from
+    // a bare word to a property access is enough to stop a fixture that *defines* such a key.
+    /\.\s*toHaveBeenCalled/,
     /\bsinon\s*\.\s*assert\b/,
     // Python: pytest, unittest, hypothesis.
     /(^|[^\w.])assert\s+[^=]/m,

package/dist/engine/vouch/text.js

@@ -105,6 +105,31 @@ export function firstRealLine(body) {
     const code = lines.find((l) => !/^(#|\/\/|\/\*|\*|"""|'''|--)/.test(l) && /[A-Za-z0-9]/.test(l));
     return (code ?? lines[0] ?? '').slice(0, 120);
 }
+/**
+ * A concise arrow body: `(a) => a.join(x)` has no braces, so it ends at the end of the statement.
+ *
+ * Without this, a scan for the next `{` walks past the arrow entirely and lands in whatever block
+ * comes next in the file. In a test file that is the next test, full of assertions, so every
+ * one-line helper looked as though it asserted and any test calling one was called sound.
+ */
+export const conciseBody = (text, start) => {
+    let depth = 0;
+    for (let i = start; i < text.length; i += 1) {
+        const ch = text[i];
+        if ('([{'.includes(ch))
+            depth += 1;
+        else if (')]}'.includes(ch)) {
+            depth -= 1;
+            // A closing bracket we never opened belongs to whatever encloses this arrow: the body stops.
+            if (depth < 0)
+                return text.slice(start, i);
+        }
+        else if (depth === 0 && (ch === ';' || ch === String.fromCharCode(10) || ch === ',')) {
+            return text.slice(start, i);
+        }
+    }
+    return text.slice(start);
+};
 export const balanced = (text, open) => {
     if (open === -1)
         return null;

package/dist/engine/vouch/vouch.js

@@ -9,7 +9,7 @@
 // their time and their trust, and a reviewer who is wrong once is ignored afterwards. So a body whose
 // checking happens somewhere this cannot follow is reported as undecided, by name, and never as
 // vacuous.
-import { balanced, firstRealLine, stripLines, withoutComments } from './text.js';
+import { balanced, conciseBody, firstRealLine, stripLines, withoutComments } from './text.js';
 import { swallowsFailure } from './swallowed.js';
 import { checksSomething, tautologies, unfinishedExpectations } from './assertions.js';
 import { languageOf, looksLikeTests, testsIn, textRanges } from './tests-in.js';
@@ -142,12 +142,33 @@ function helpersThatCheck(text, language) {
         if (!name)
             continue;
         const from = (hit.index ?? 0) + hit[0].length;
-        const body = language === 'python' ? indentedFrom(text, from) : balanced(text, text.indexOf('{', from - 1));
+        const body = language === 'python' ? indentedFrom(text, from) : jsHelperBody(text, from, hit[0].trimEnd().endsWith('=>'));
         if (body && checksSomething(body))
             names.add(name);
     }
     return names;
 }
+/**
+ * The body of a helper declared in JavaScript or TypeScript.
+ *
+ * An arrow's body begins where the match ends: `{` for a block, otherwise a single expression. The
+ * distinction is the whole point. This used to look for the next `{` from here regardless, which
+ * for `const file = (...lines) => lines.join(NL)` is not this helper's brace at all but the start
+ * of whatever block follows in the file. In a test file that is the next `describe` or `it`, so
+ * `checksSomething` saw its assertions and credited them to the helper - and then any test whose
+ * only remaining call was `file(...)` was told it checks something. A hollow test reported as
+ * sound, which is the one direction Tewip must never fail in.
+ *
+ * Found by measuring vouch's recall against execution: engine/test/line-diff.test.ts has exactly
+ * this helper, and a test stripped of its only assertion came back as "it checks through file(),
+ * which asserts".
+ */
+function jsHelperBody(text, from, isArrow) {
+    if (!isArrow)
+        return balanced(text, text.indexOf('{', from - 1));
+    const start = from + (/^\s*/.exec(text.slice(from))?.[0].length ?? 0);
+    return text[start] === '{' ? balanced(text, start) : conciseBody(text, start);
+}
 /** Python's block after an offset: every line indented further than the line the offset is on. */
 function indentedFrom(text, from) {
     const lines = text.slice(from).split(String.fromCharCode(10));

package/dist/engine/walk.js

@@ -7,7 +7,7 @@
 //
 // The cap now counts the files the caller actually wants. A budget spent on files the caller will
 // discard is not a budget.
-import { readdirSync, statSync } from 'node:fs';
+import { lstatSync, readdirSync } from 'node:fs';
 import { join } from 'node:path';
 const SKIP_DIR = new Set(['node_modules', '.git', 'dist', 'build', 'out', 'coverage', 'target', 'vendor', '__pycache__', '.venv', 'venv', '.tewip', '.next', '.cache']);
 const BINARY = /\.(?:png|jpe?g|gif|webp|ico|svg|pdf|zip|gz|tar|exe|dll|so|dylib|woff2?|ttf|eot|mp4|mp3|wav|class|jar|pyc|lock)$/i;
@@ -39,11 +39,29 @@ export function scanSourceFiles(root, limit = 5000, keep) {
             const path = join(dir, entry);
             let info;
             try {
-                info = statSync(path);
+                // `lstat`, not `stat`: a symlink must be seen as a symlink. See below.
+                info = lstatSync(path);
             }
             catch {
                 continue;
             }
+            // A symlink is neither walked nor read.
+            //
+            // `stat` follows them, so a link to a directory looked like a directory and was walked. In
+            // immich, `fastlane/metadata` points at `../mobile/android/fastlane/metadata` inside the same
+            // repository: following it read 77 files a second time, under a path they do not really have.
+            // A finding in one of them would have been reported twice, at two different paths, and the
+            // file count no longer meant what the report said it meant.
+            //
+            // Worse than the double count: a link pointing out of the project would have Tewip reading
+            // files outside the directory it was given, which it must never do whatever is on the other
+            // end. Anything genuinely inside the project is still reached by its real path.
+            //
+            // Windows was already doing the right thing by accident - git writes a symlink there as a
+            // small text file - which is exactly why this hid: the numbers were recorded on Windows and
+            // only the Linux corpus run disagreed.
+            if (info.isSymbolicLink())
+                continue;
             if (info.isDirectory()) {
                 if (!SKIP_DIR.has(entry))
                     walk(path);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "tewip-ai",
-  "version": "0.71.0",
+  "version": "0.73.0",
   "mcpName": "io.github.hmamut39/tewip",
   "description": "Tewip finds why each failing test failed - real bug, flaky test, environment, test data or changed selector - for any stack, from the command line, CI, or your editor.",
   "keywords": [