agent-roast 0.20.0 → 0.21.0

package/README.md

@@ -80,8 +80,8 @@ Policy is read from the **base** branch, so a pull request cannot switch off the
 | --- | --- | :-: |
 | `assertion-weakened` | an exact check is replaced by a weaker one: `toBe(42)` → `toBeDefined()`, `toBe(2)` → `toBeLessThanOrEqual(2)`, `toBe(2)` → `not.toBe(0)`, `toEqual(x)` → `toMatchObject(x)`, `assert x == 1` → `assert x` | blocks |
 | `guard-weakened` | a coverage threshold is lowered, `strict` is switched off, a lint rule is downgraded, a CI step may fail, a workflow stops running tests, a `conftest.py` hook sets a report to `passed` | blocks |
-| `test-skip` | a test that was already there is switched off or expected to fail: `it.skip`, `@pytest.mark.skip`, `@pytest.mark.xfail`, `it.fails`, `test.fixme`, `t.Skip`, `#[ignore]`, `@Disabled`, `[Ignore]`, `xit`, ... (a new test that is skipped is not reported) | blocks |
-| `test-focus`, `trivial-assert` | `.only` / `fit`; `expect(true).toBe(true)` and its relatives in ten languages; a test that asserts on a function it mocked itself | blocks |
+| `test-skip` | a test that was already there is switched off or expected to fail: `it.skip`, `@pytest.mark.skip`, `@pytest.mark.skipif(True)`, `@pytest.mark.xfail`, `it.fails`, `test.fixme`, `t.Skip`, `#[ignore]`, `@Disabled`, `[Ignore]`, `xit`, ... (a new test that is skipped is not reported) | blocks |
+| `test-focus`, `trivial-assert` | `.only` / `fit`; `expect(true).toBe(true)` and its relatives in ten languages, `assert x or True`; a test that asserts on a function it mocked itself | blocks |
 | `assertion-removed`, `test-removed`, `expectation-rewritten` | a test loses assertions, disappears, or its expected value is edited next to production code | warns |
 | `swallowed-error`, `type-escape`, `stub-abandonment` | `catch {}` / `except: pass`, `as any` / `# type: ignore` / `# pragma: no cover` / `istanbul ignore`, `throw new Error("Not implemented")` | warns |
 
@@ -112,15 +112,18 @@ Measured, with the denominators and the caveats in [docs/benchmarks.md](docs/ben
 deliberate, and not specific to AI. agent-roast is a tripwire that asks about them. On repositories it had not seen, between a quarter and
 two thirds of its blocks were questions with a harmless answer, depending on the sample.**
 
-- **152 of 152** hand-labeled cheating edits are caught and **136 of 136** harmless look-alikes (reformats, moves, raised
+- **162 of 162** hand-labeled cheating edits are caught and **150 of 150** harmless look-alikes (reformats, moves, raised
   thresholds, environment-gated skips) stay silent. The cases were written with the detector, so they guard against regressions;
   they do not prove accuracy. One gap is documented and still open.
+- **A catalog of cheats that someone else wrote.** Of the nine hacks in [reward-hack-zoo](https://github.com/MatthewWaller/reward-hack-zoo) (hand-written specimens; none from a model) four are reported (a skip, a patched target, a `conftest.py` that rewrites reports,
+  an edited expected value as a warning) and none of its six honest solutions. The five that are not (special-casing, swallowed errors, overfitting) live in the code, and a search of 126,486 commits shows why a tool that reads text cannot report them:
+  their shape is the shape of ordinary code ([section 13](docs/benchmarks.md#13-a-probe-of-what-no-rule-sees-and-a-catalog-of-cheats-that-someone-else-wrote)).
 - **12,645 commits** that touch tests or CI/config in 166 public repositories: blocking rules fired on 0.3% of AI-signed commits
   (Claude, Copilot, Cursor, Codex, Devin, aider) and on 0.3% of the others. Inside the same repositories the odds for an
   AI-signed commit were 0.81 times those of another commit (95% CI 0.40–1.67). There is no sign that AI-signed commits weaken
   checks more often.
-- **On repositories it was not written from, between a quarter and two thirds of the blocked commits were false alarms.** Seven samples of 39
-  or 40 public repositories (278 in all, 79,843 commits) were replayed and every blocked commit was read. The share was 69% for 0.17.0 on the
+- **On repositories it was not written from, between a quarter and two thirds of the blocked commits were false alarms.** Nine samples of 39
+  or 40 public repositories (356 in all, 102,210 commits) were replayed and every blocked commit was read. The share was 69% for 0.17.0 on the
   first sample and on the fourth, 51% for 0.18.0 on the fourth and 27% on the fifth, which it had not seen either. This version, frozen before the
   sixth sample was drawn, blocked 100 of its 11,335 commits: 34 changed a check a reviewer should see (an assertion that got weaker, a gate
   that was relaxed, a test switched off or left focused) and 66 were false alarms, but 51 of the 66 were one defect in one repository, an array of
@@ -128,8 +131,8 @@ two thirds of its blocks were questions with a harmless answer, depending on the
   test that asserts nothing counts as a block for a reviewer; counted as false alarms, 13 of 33 on the fifth sample and 72 of 100 on the sixth. Most of the rest, in the
   earlier measurements too, were deliberate and explained. One reader (an AI assistant) sorted them, and the fixes were made from the same
   sorting, so treat the figures as a first estimate. The published 0.19.0 blocked 31 of the 12,000 commits of the seventh sample: 16 for a reviewer and 15 false alarms (48%, 6 of them one
-  repository's lint configuration); this release fixes the kinds of false alarm that reading it showed, and blocks 22 of those commits ([section 9](docs/benchmarks.md#9-repositories-nobody-had-tuned-on),
-  [section 10](docs/benchmarks.md#10-two-more-samples-and-what-the-fixes-from-sample-4-did), [section 11](docs/benchmarks.md#11-a-seventh-sample-and-two-things-that-were-missing)).
+  repository's lint configuration); the build with the fixes from reading it, tried on an eighth sample that it had not seen, blocked 26 of 11,081 commits: 16 for a reviewer and 10 false alarms (38%), and the fixes from reading that one are in this release too; this release as it was frozen blocked 46 of 11,286 commits of a ninth sample that it had not seen: 28 for a reviewer and 18 false alarms (39%), and five of those are fixed in it ([section 9](docs/benchmarks.md#9-repositories-nobody-had-tuned-on),
+  [section 10](docs/benchmarks.md#10-two-more-samples-and-what-the-fixes-from-sample-4-did), [section 11](docs/benchmarks.md#11-a-seventh-sample-and-two-things-that-were-missing), [section 12](docs/benchmarks.md#12-an-eighth-sample-the-first-measurement-of-the-fixes-from-sample-7), [section 14](docs/benchmarks.md#14-a-ninth-sample-the-shipped-build-on-repositories-it-had-not-seen)).
 - **9,732 commits compared in full** from 164 public repositories that use coding agents (5,454 of them AI-signed) were reviewed again for the
   last two releases, and every new finding was read. 0.15.0's rules reported 30 new findings there and 7 were false alarms (fixed in 0.16.0);
   0.17.0's rule for an exact check turned into a bound reported 18 in 14 commits, all of them loosened checks, after two false alarms were

package/dist/{chunk-BNDNXKBM.js → chunk-XVV5FNDV.js}

@@ -1167,7 +1167,9 @@ var PYTHON_TEST_SKIP_PATTERNS = [
   // A test that is expected to fail cannot fail the run, which is what a skip does too.
   { regex: /(?<![\w.])pytest\s*\.\s*xfail\s*\(/, name: "imperative pytest xfail (pytest.xfail)", imperative: true },
   // `@unittest.expectedFailure`, and the same decorator under another module's name (`@test.expectedFailure` of a test framework that re-exports it).
-  { regex: /@(?:[A-Za-z_]\w*\s*\.\s*)*expectedFailure\b/, name: "test expected to fail (@unittest.expectedFailure)", imperative: false }
+  { regex: /@(?:[A-Za-z_]\w*\s*\.\s*)*expectedFailure\b/, name: "test expected to fail (@unittest.expectedFailure)", imperative: false },
+  // `skipif(True)` is a skip whose condition is always true: it switches the test off wherever it runs, which `skipif` with a condition on the environment does not.
+  { regex: /(?:@\s*|(?<![\w.])marks\s*=\s*\[?\s*|^[ \t]*pytestmark\s*=\s*\[?\s*)(?:pytest\s*\.\s*mark\s*\.\s*skipif|(?:unittest\s*\.\s*)?skipIf)\s*\(\s*True\s*[,)]/, name: "unconditionally skipped test (skipif(True))", imperative: false }
 ];
 var PYTHON_MARKER = /(@\s*|(?<![\w.])marks\s*=\s*\[?\s*|^[ \t]*pytestmark\s*=\s*\[?\s*)pytest\s*\.\s*mark\s*\.\s*(skip|xfail)\b/gm;
 function isConditionalXfail(args) {
@@ -1178,20 +1180,20 @@ function isConditionalXfail(args) {
   return !/\bCI\b|\bisCI\b|GITHUB_ACTIONS/.test(condition);
 }
 function isGuardedSkip(lines) {
-  const indentOf = (text) => text.length - text.trimStart().length;
+  const indentOf2 = (text) => text.length - text.trimStart().length;
   const mentionsCi = (raw) => /\bCI\b|\bGITHUB_ACTIONS\b/.test(raw);
   const own = lines[lines.length - 1];
   if (/^\s*(?:if|elif|else)\b.*:\s*\S/.test(own.code)) return !mentionsCi(own.raw);
-  let limit = indentOf(own.code);
+  let limit = indentOf2(own.code);
   for (let i = lines.length - 2; i >= 0 && lines.length - 1 - i <= 200; i--) {
     if (lines[i].number !== lines[i + 1].number - 1) return false;
     const { code, raw } = lines[i];
-    if (!code.trim() || indentOf(code) >= limit) continue;
+    if (!code.trim() || indentOf2(code) >= limit) continue;
     const keyword = /^\s*(async\s+def|def|class|if|elif|else|try|except|finally|with|for|while)\b/.exec(code)?.[1];
     if (keyword === "def" || keyword === "async def") return !/^\s*(?:async\s+)?def\s+test/.test(code);
     if (keyword === "class") return false;
     if (keyword === "if" || keyword === "elif" || keyword === "else" || keyword === "try" || keyword === "except" || keyword === "finally") return !mentionsCi(raw);
-    limit = indentOf(code) + (keyword ? 0 : 1);
+    limit = indentOf2(code) + (keyword ? 0 : 1);
   }
   return false;
 }
@@ -1289,7 +1291,7 @@ function inTestAttributeList(lines, index) {
   return false;
 }
 function isGuardedBlockSkip(lines, profile) {
-  const indentOf = (text) => text.length - text.trimStart().length;
+  const indentOf2 = (text) => text.length - text.trimStart().length;
   const mentionsCi = (raw) => /\bCI\b|\bGITHUB_ACTIONS\b/.test(raw);
   const own = lines[lines.length - 1];
   if (profile.modifier?.test(own.code) || profile.conditional.test(own.code)) return !mentionsCi(own.raw);
@@ -1303,17 +1305,17 @@ function isGuardedBlockSkip(lines, profile) {
     }
     return depth > 0;
   };
-  let limit = indentOf(own.code);
+  let limit = indentOf2(own.code);
   let i = lines.length - 2;
   for (; i >= 0 && lines.length - 1 - i <= 200; i--) {
     if (lines[i].number !== lines[i + 1].number - 1) return compiledConditionally(i + 1);
     const { code, raw } = lines[i];
-    if (!code.trim() || indentOf(code) >= limit) continue;
+    if (!code.trim() || indentOf2(code) >= limit) continue;
     if (profile.preprocessor && (profile.preprocessor.open.test(code) || profile.preprocessor.close.test(code) || profile.preprocessor.other.test(code))) continue;
     if (profile.conditional.test(code)) return !mentionsCi(raw);
     if (profile.test.test(code)) return compiledConditionally(i + 1);
     if (profile.helper.test(code)) return true;
-    limit = indentOf(code);
+    limit = indentOf2(code);
   }
   return compiledConditionally(i + 1);
 }
@@ -1331,7 +1333,7 @@ function isConditionalSkip(source, raw, open2, start) {
 var TEST_REGISTRATION = /^(?:await\s+)?(?:test|it|describe|suite|beforeAll|beforeEach|afterAll|afterEach)\b/;
 function isUnderRuntimeCondition(source, raw, start) {
   const alwaysTrue = (text) => /^\s*(?:true|1|!\s*0|!\s*false)\s*$/.test(text);
-  const indentOf = (text) => text.length - text.trimStart().length;
+  const indentOf2 = (text) => text.length - text.trimStart().length;
   const lineStartOf = (offset) => Math.max(source.lastIndexOf("\n", offset - 1), source.lastIndexOf("\0", offset - 1)) + 1;
   const mentionsCi = (from, to) => MENTIONS_CI.test(raw.slice(from, to));
   const lineStart = lineStartOf(start);
@@ -1351,20 +1353,20 @@ function isUnderRuntimeCondition(source, raw, start) {
   } else if (/(?:^|[\s}])else$/.test(head)) return true;
   else if (head.endsWith("&&")) return !mentionsCi(lineStart, start) && !alwaysTrue(head.slice(0, -2).replace(/^[\s{}]+/, ""));
   const lineEnd = source.slice(start).search(/[\n\0]/);
-  let limit = indentOf(source.slice(lineStart, lineEnd < 0 ? void 0 : start + lineEnd)), at = lineStart;
+  let limit = indentOf2(source.slice(lineStart, lineEnd < 0 ? void 0 : start + lineEnd)), at = lineStart;
   for (let steps = 0; steps < 200 && at > 0; steps++) {
     if (source[at - 1] === "\0") return false;
     const previous = lineStartOf(at - 1), line = source.slice(previous, at - 1);
     const lineRaw = raw.slice(previous, at - 1);
     at = previous;
-    if (!line.trim() || indentOf(line) >= limit) continue;
+    if (!line.trim() || indentOf2(line) >= limit) continue;
     const header = line.trim();
     if (/^(?:\}\s*)?(?:else\s+)?if\s*\(/.test(header)) {
       return !MENTIONS_CI.test(lineRaw) && !alwaysTrue(/\bif\s*\((.*)\)\s*\{?\s*$/.exec(header)?.[1] ?? "");
     }
     if (/^(?:\}\s*)?(?:else|catch)\b|^(?:case\b|default\s*:)|^\)\s*\{?\s*$/.test(header)) return true;
     if (TEST_REGISTRATION.test(header)) return false;
-    limit = indentOf(line);
+    limit = indentOf2(line);
   }
   return false;
 }
@@ -1436,6 +1438,27 @@ var PYTHON_TRIVIAL_ASSERT_PATTERNS = [
   { regex: /\bself\s*\.\s*assertTrue\s*\(\s*(?:True|1)\s*\)/, name: "trivial assert (self.assertTrue(True))" },
   { regex: /\bself\s*\.\s*assertEqual\s*\(\s*(?:1\s*,\s*1|True\s*,\s*True)\s*\)/, name: "trivial assert (self.assertEqual(1, 1))" }
 ];
+function assertsAlwaysTrue(code) {
+  const body = /^\s*assert\b\s*(.*)$/.exec(code)?.[1] ?? /\bself\s*\.\s*assertTrue\s*\((.*)$/.exec(code)?.[1];
+  if (body === void 0) return false;
+  let depth = 0, outside2 = "";
+  for (const character of body) {
+    if ("([{".includes(character)) {
+      outside2 += depth === 0 ? character : " ";
+      depth++;
+      continue;
+    }
+    if (")]}".includes(character)) {
+      if (depth === 0) break;
+      depth--;
+      outside2 += depth === 0 ? character : " ";
+      continue;
+    }
+    if (depth === 0 && character === ",") break;
+    outside2 += depth === 0 ? character : " ";
+  }
+  return outside2.split(/\bor\b/).some((operand) => /^\s*(?:True|1)\s*$/.test(operand));
+}
 var JS_STUB_PATTERNS = [
   { regex: /\bthrow\s+new\s+Error\s*\(\s*["'`](?:TODO(?=\s*(?:["'`:]|implement\b))|Not implemented\b|to be implemented\b|stub\b|implement me\b)/i, name: "unimplemented stub (throw new Error('Not implemented'))" }
 ];
@@ -1562,19 +1585,17 @@ function auditFileDiffs(files, commitHash = "HEAD", commitDate = (/* @__PURE__ *
         foundEscape = true;
       }
       if (python2 && !isComment && isTestFile(file.path)) {
-        for (const rule of PYTHON_TRIVIAL_ASSERT_PATTERNS) {
-          if (rule.regex.test(cleanedLine)) {
-            infractions.push({
-              type: "trivial-assert",
-              file: file.path,
-              lineNumber: added.lineNumber,
-              lineSnippet: rawLine.trim(),
-              commitHash,
-              commitDate,
-              reason: rule.name
-            });
-            break;
-          }
+        const trivial = PYTHON_TRIVIAL_ASSERT_PATTERNS.find((rule) => rule.regex.test(cleanedLine))?.name ?? (assertsAlwaysTrue(cleanedLine) ? "assertion that cannot fail (assert ... or True)" : void 0);
+        if (trivial) {
+          infractions.push({
+            type: "trivial-assert",
+            file: file.path,
+            lineNumber: added.lineNumber,
+            lineSnippet: rawLine.trim(),
+            commitHash,
+            commitDate,
+            reason: trivial
+          });
         }
       }
     }
@@ -3339,7 +3360,7 @@ function javascript(text) {
     if (!binding || !mock) continue;
     const name = binding.namespace ? call[2] : call[2] ? void 0 : binding.name;
     if (!name) continue;
-    const identifier = call[1].replace(/\$/g, "\\$");
+    const identifier = (binding.namespace ? `${call[1]}\\s*\\.\\s*${call[2]}` : call[1]).replace(/\$/g, "\\$");
     const configures = new RegExp(`(?<![\\w$.])${identifier}\\b[^;]{0,80}?\\.\\s*mock(?:Return|Resolved|Rejected|Implementation)\\w*\\s*\\(`, "g");
     const configured = [...raw.matchAll(configures)].some((config) => tests.every((range) => !within(range, config.index) || within(range, match.index)));
     if (!mock.fake && !configured) continue;
@@ -3605,10 +3626,14 @@ function analyzePythonConfig(file, oldText, newText, report) {
     /^\s*python_version\s*=\s*["']?(\d+(?:\.\d+)*)/gm,
     /\bruff\s*(?:(?:===|==|~=|>=|>)\s*["']?|=\s*["'][\^~>=]*)(\d+(?:\.\d+)*)/g
   ];
+  const effective = (text) => newest(text, versions[1]) ?? newest(text, versions[0]);
   const bumped = versions.some((pattern) => {
     const was2 = newest(oldText, pattern), now = newest(newText, pattern);
     return !!was2 && !!now && higher(was2, now);
-  });
+  }) || (() => {
+    const was2 = effective(oldText), now = effective(newText);
+    return !!was2 && !!now && higher(was2, now);
+  })();
   for (const [tool, bucket] of ignored(after)) {
     if (!configured.has(tool) || tool === "ruff" && bumped) continue;
     const old = [...was.get(tool)?.keys() ?? []];
@@ -3713,13 +3738,13 @@ function testBodies(text, language) {
     const match = startPattern.exec(line);
     if (match) starts.push({ index, name: match[1], length: match[0].length });
   });
-  const indentOf = (line) => /^\s*/.exec(line)[0].length;
+  const indentOf2 = (line) => /^\s*/.exec(line)[0].length;
   return starts.map((start, n) => {
     const limit = n + 1 < starts.length ? starts[n + 1].index : lines.length;
-    const header = lines[start.index], indent = indentOf(header);
+    const header = lines[start.index], indent = indentOf2(header);
     let end = start.index + 1, depth = balance(header);
     while (depth > 0 && end < limit) depth += balance(lines[end++]);
-    while (end < limit && (language === "go" ? lines[end - 1] !== "}" : !lines[end].trim() || indentOf(lines[end]) > indent)) end++;
+    while (end < limit && (language === "go" ? lines[end - 1] !== "}" : !lines[end].trim() || indentOf2(lines[end]) > indent)) end++;
     let top = start.index;
     while (top > 0 && start.index - top < 200 && lines[top - 1].trim() && !/^\s*(?:async\s+)?(?:def|class)\b/.test(lines[top - 1])) top--;
     return {
@@ -4088,7 +4113,9 @@ function isWeaker(rawOld, rawNew, language) {
     const oldBare = unittestAsAssert(oldAssertion) ?? oldAssertion;
     if (WEAK_PY.test(oldAssertion) || BOUND_PY.test(oldAssertion) || WEAK_PY.test(oldBare) || BOUND_PY.test(oldBare) || !PY_SPECIFIC.test(oldAssertion)) return false;
     const limit = /(?:>=|<=|>|<)\s*([^\s,)]+)/.exec(oldAssertion)?.[1];
-    return WEAK_PY.test(newAssertion) || BOUND_PY.test(newAssertion) && !EMPTY_PY.test(oldAssertion) && /==|[<>]=?/.test(oldAssertion) && (limit === void 0 || /^-?\d+(?:\.\d+)?$/.test(limit));
+    const outsideBrackets = oldAssertion.replace(/(['"`])(?:\\.|(?!\1)[^\\])*\1/g, "").replace(/[([{][^()[\]{}]*[)\]}]/g, " ").replace(/[([{][^()[\]{}]*[)\]}]/g, " ").replace(/[([{][^()[\]{}]*[)\]}]/g, " ");
+    const expectsNothing = EMPTY_PY.test(oldAssertion) && !/\b(?:and|or)\b/.test(outsideBrackets);
+    return WEAK_PY.test(newAssertion) && !expectsNothing || BOUND_PY.test(newAssertion) && !EMPTY_PY.test(oldAssertion) && /==|[<>]=?/.test(oldAssertion) && (limit === void 0 || /^-?\d+(?:\.\d+)?$/.test(limit));
   }
   return false;
 }
@@ -4346,7 +4373,10 @@ function newTestMarkers(before, after, oldText, newText) {
     if (nth <= had || had === 0 && (nth > 1 && !test.key.startsWith("js:") || !existed(test))) continue;
     found.push(finding2);
   }
-  return [...found, ...unnamed.slice(unnamedBefore)];
+  const plain = (lines) => lines.map((line) => line.trim().replace(/\.(?:skip|fails|failing|todo|fixme|only)\b/g, ""));
+  const oldPlain = plain(oldLines).join("\n"), newPlain = plain(newLines);
+  const newTable = (finding2) => /\.each\b/.test(finding2.lineSnippet ?? "") && !oldPlain.includes(newPlain.slice((finding2.lineNumber ?? 1) - 1, (finding2.lineNumber ?? 1) + 3).join("\n"));
+  return [...found, ...unnamed.slice(unnamedBefore).filter((finding2) => !newTable(finding2))];
 }
 var GUARD_FILE = /(?:^|\/)(?:package\.json|tsconfig[^/]*\.json|jsconfig\.json|(?:jest|vitest|vite)\.config\.[cm]?[jt]s|\.eslintrc(?:\.[a-z]+)?|eslint\.config\.[cm]?[jt]s|pyproject\.toml|setup\.cfg|pytest\.ini|tox\.ini|\.?ruff\.toml|\.flake8|\.?mypy\.ini|\.?pylintrc|\.coveragerc|codecov\.ya?ml|\.gitlab-ci\.ya?ml|\.github\/workflows\/[^/]+\.ya?ml)$/i;
 function isGuardFile(file) {
@@ -4384,8 +4414,68 @@ var CI_RUNNER = new RegExp(
   String.raw`|(?<![\w./-])(?:deno\s+(?:task|run)|just|task|poe|rake|composer(?:\s+run)?|mix|sbt|lein|stack|cabal|(?:\./)?(?:gradlew|mvnw)(?:\.bat|\.cmd)?)\s+(?:-\S+\s+)*test(?![\w/-])` + String.raw`|(?<![\w./-])(?:node|tsx)(?:\s+-\S+)*\s+--test(?![\w-])` + // A project's own test script, run through its interpreter or by path: `python scripts/run_tests.py`, `./scripts/test.sh`, `python tests/test_cli.py`.
   String.raw`|(?<![\w./-])(?:(?:python[\d.]*|node|bash|sh|zsh|ruby|pwsh|tsx|ts-node)(?:\s+-\S+)*\s+|\./)(?:[\w.-]+/)*(?:(?:run[_-]?)?tests?(?:[_-](?:all|suite|runner|unit|ci))?|test_[\w-]+|[\w-]+_test)\.(?:py|sh|bash|js|mjs|cjs|ts|rb|ps1)(?![\w-])` + String.raw`|\brun:.*(?<=\s)(?:tests?|unit|e2e)(?![\w/.-])`
 );
-var SHELL_TEST = /(?:^|run:\s*|[;&|({]\s*|\b(?:if|then|do|else|elif|while|until)\s+|!\s+)test\s+!?\s*-[bcdefghkLnOGprsStuwxzN](?=\s)/g;
-var ciRuns = (line) => CI_RUNNER.test(line.replace(SHELL_TEST, " ").replace(/\s#.*$/, ""));
+var SHELL_TEST = /(?:^|run:\s*|[;&|({]\s*|\b(?:if|then|do|else|elif|while|until)\s+|!\s+)test\s+!?\s*(?:-[bcdefghkLnOGprsStuwxzN](?=\s)|["$'])/g;
+var NOT_TESTS = new RegExp(
+  String.raw`(?<![\w./-])(?:uv\s+(?:sync|add|tool\s+install|pip\s+install)|(?:python[\d.]*\s+-m\s+)?pip[\d.]*\s+install|pipx\s+install|poetry\s+(?:install|add)|pdm\s+(?:install|sync|add)|npm\s+(?:ci|i|install|add)|pnpm\s+(?:i|install|add)|bun\s+(?:i|install|add)|yarn\s+(?:install|add)|bundle\s+install|gem\s+install|cargo\s+install|go\s+(?:install|get)|brew\s+install|apt(?:-get)?\s+install|conda\s+(?:install|create))\b[^&;|\n]*` + // The gap before the environment is bounded, so that a line of a hundred thousand characters with `tox` in it again and again is read in linear time.
+  String.raw`|(?<![\w./-])(?:tox|nox)\b[^&;|\n]{0,300}?\s(?:-e|-s|--env|--session)[= ]\s*(?:lint|linting|format|fmt|docs?|types?|typing|mypy|pyright|ruff|black|flake8|pylint|style|isort|bandit|build|package|clean)(?![,\w-])[^&;|\n]*`,
+  "g"
+);
+var ciRuns = (line) => !/^\s*(?:-\s+)?name\s*:/.test(line) && CI_RUNNER.test(line.replace(SHELL_TEST, " ").replace(NOT_TESTS, " ").replace(/\s#.*$/, ""));
+var COMMAND_BLOCK = /^(\s*(?:-\s+)?)(run|script|before_script|after_script)\s*:\s*([|>])?[+-]?\d*\s*(?:#.*)?$/;
+var indentOf = (line) => line.length - line.trimStart().length;
+var PRINTS = /^(?:echo|printf)\b/;
+var NOT_A_TOOL = "mkdir|rmdir|cd|rm|ls|cat|cp|mv|touch|chmod|chown|echo|printf|export|source|git|docker|find|tar|zip|unzip|grep|sed|awk|diff|head|tail|wc|stat|du|tree|set|test|curl|wget|ssh|scp|rsync|ping|open|sleep|kill|for|while|until|if|elif|else|then|do|done|case|in|assert|return|import|from|def|class|print|raise|yield|with|local|declare|readonly|unset|eval|await|let|const|var|function|new|delete|throw|and|or|not|is|lambda|try|except|finally|pass|del";
+var TOOL_TEST = new RegExp(
+  String.raw`^(?:(?:sudo|time|exec|command|env)(?:\s+-\S+)*\s+|\w+=(?:"[^"]*"|'[^']*'|[^\s"']*)\s+)*(?!(?:${NOT_A_TOOL})\b)(?!-)[\w./-]+\s+test(?![\w/.=\[(-])`
+);
+function blockCommands(lines, at, key) {
+  const keyIndent = key[1].length, found = [];
+  if (key[3] === ">") {
+    for (let next = at + 1; next < lines.length; next++) {
+      const text = lines[next].trim();
+      if (!text) continue;
+      if (indentOf(lines[next]) <= keyIndent) break;
+      if (!text.startsWith("#")) found.push(text);
+    }
+    return [found.join(" ")];
+  }
+  if (key[3] === "|") {
+    for (let next = at + 1; next < lines.length; next++) {
+      const text = lines[next].trim().replace(/\s*\\$/, "");
+      if (!text) continue;
+      if (indentOf(lines[next]) <= keyIndent) break;
+      if (TOOL_TEST.test(text)) found.push(text);
+    }
+    return found;
+  }
+  if (key[3] || key[2] === "run") return [];
+  let itemIndent = -1;
+  for (let next = at + 1; next < lines.length; next++) {
+    const text = lines[next].trim();
+    if (!text || text.startsWith("#")) continue;
+    const indent = indentOf(lines[next]);
+    if (itemIndent < 0) {
+      if (!text.startsWith("- ") || indent < keyIndent) break;
+      itemIndent = indent;
+    }
+    if (indent < itemIndent || indent === itemIndent && !text.startsWith("- ")) break;
+    if (indent === itemIndent) found.push(text.slice(2).trim());
+  }
+  return found;
+}
+function ciTestEntries(text) {
+  const lines = text.split(/\r?\n/), found = [];
+  for (let i = 0; i < lines.length; i++) {
+    const line = lines[i];
+    if (!line.trim() || line.trim().startsWith("#")) continue;
+    if (ciRuns(line)) found.push({ index: i, command: line });
+    const key = COMMAND_BLOCK.exec(line);
+    if (!key) continue;
+    for (const command of blockCommands(lines, i, key)) if (command && !PRINTS.test(command) && !ciRuns(command) && ciRuns(`run: ${command}`)) found.push({ index: i, command });
+  }
+  return found;
+}
+var ciTestLines = (text) => ciTestEntries(text).map((entry) => entry.command);
 var CI_FILE = /(?:\.github\/workflows\/|\.gitlab-ci)/;
 var CI_REUSABLE = /^\s*-?\s*uses:\s*\S*\.github\/workflows\/\S+/;
 var CI_GATE = /^\s*(?:pull_request|pull_request_target|push|merge_group)\s*:|^\s*on\s*:\s*(?:\[[^\]]*\b(?:push|pull_request|pull_request_target|merge_group)\b[^\]]*\]|(?:push|pull_request|pull_request_target|merge_group)\s*$)|^\s*-\s*(?:push|pull_request|pull_request_target|merge_group)\s*$/m;
@@ -4421,10 +4511,14 @@ function triggerSection(text) {
   while (end < lines.length && (!lines[end].trim() || /^[\s#-]/.test(lines[end]))) end++;
   return lines.slice(start, end).join("\n");
 }
-function gatingTestCommands(file, text) {
+function gatingTestCommands(file, text, skip) {
   if (!(/\.gitlab-ci/.test(file) || CI_GATE.test(gatingTriggers(text) ?? text))) return [];
   const directories = [...new Set([...text.matchAll(/^\s*(?:-\s+)?working-directory\s*:\s*["']?([^\s"'#]+)/gm)].map((match) => match[1]))].sort().join(",");
-  return text.split(/\r?\n/).filter((line) => !line.trim().startsWith("#") && ciRuns(line)).map((line) => `${line.replace(/\s#.*$/, "").replace(/^\s*-?\s*(?:run\s*:\s*)?/, "").replace(/\s+/g, " ").trim()} @${directories}`);
+  return ciTestEntries(text).filter((entry) => !skip?.(entry.index)).map((entry) => `${entry.command.replace(/\s#.*$/, "").replace(/^\s*-?\s*(?:run\s*:\s*)?/, "").replace(/\s+/g, " ").trim()} @${directories}`);
+}
+function enforcedTestCommands(file, text) {
+  const tolerated = [...lenientSteps(text).values()].filter((step) => step.runsChecks).flatMap((step) => step.ranges);
+  return gatingTestCommands(file, text, (line) => tolerated.some(([from, to]) => line >= from && line < to));
 }
 function matchingLines(text, pattern) {
   const matches = typeof pattern === "function" ? pattern : (line) => pattern.test(line);
@@ -4433,9 +4527,9 @@ function matchingLines(text, pattern) {
 function ciTestSteps(file, oldText, newText) {
   if (!CI_FILE.test(file) || oldText === null && newText === null) return null;
   const was = oldText ?? "", now = newText ?? "";
-  const commandsBefore = gatingTestCommands(file, was), commandsAfter = gatingTestCommands(file, now);
+  const commandsBefore = enforcedTestCommands(file, was), commandsAfter = gatingTestCommands(file, now);
   const after = commandsAfter.length;
-  return { file, before: commandsBefore.length, after, reusableAfter: matchingLines(now, CI_REUSABLE), ungatedAfter: after === 0 ? matchingLines(now, ciRuns) : 0, commandsBefore, commandsAfter };
+  return { file, before: commandsBefore.length, after, reusableAfter: matchingLines(now, CI_REUSABLE), ungatedAfter: after === 0 ? ciTestLines(now).length : 0, commandsBefore, commandsAfter };
 }
 function resolveCiTestSteps(counts, commit, elsewhere = /* @__PURE__ */ new Set()) {
   const before = counts.reduce((sum, item) => sum + item.before, 0);
@@ -4484,7 +4578,6 @@ function scopeCommands(scope) {
 }
 function lenientSteps(text) {
   const lines = text.split(/\r?\n/);
-  const indentOf = (line) => line.length - line.trimStart().length;
   const found = /* @__PURE__ */ new Map();
   lines.forEach((line, i) => {
     if (line.trim().startsWith("#") || !LENIENT.test(line)) return;
@@ -4501,7 +4594,7 @@ function lenientSteps(text) {
     const start = /^\s*-\s+if:/.test(line) ? i : Math.max(j, 0), level = indentOf(lines[start]);
     let end = start + 1;
     while (end < lines.length && (!lines[end].trim() || lines[end].trim().startsWith("#") || indentOf(lines[end]) > level)) end++;
-    const scope = lines.slice(start, end), runsChecks = matchingLines(scope.join("\n"), ciRuns) > 0;
+    const scope = lines.slice(start, end), runsChecks = ciTestLines(scope.join("\n")).length > 0;
     const uses = scope.filter((item) => /^\s*(?:-\s+)?uses:/.test(item));
     const nonCheck = !runsChecks && uses.length === 1 && NON_CHECK_ACTION.test(uses[0]) && !scope.some((item) => /^\s*(?:-\s+)?run:/.test(item));
     const commands = scopeCommands(scope);
@@ -4511,7 +4604,8 @@ function lenientSteps(text) {
       entry.runsChecks ||= runsChecks;
       entry.nonCheck &&= nonCheck;
       entry.commands.push(...commands);
-    } else found.set(identity, { count: 1, line: i + 1, runsChecks, nonCheck, commands });
+      entry.ranges.push([start, end]);
+    } else found.set(identity, { count: 1, line: i + 1, runsChecks, nonCheck, commands, ranges: [[start, end]] });
   });
   return found;
 }
@@ -4800,7 +4894,7 @@ async function compareSnapshot(options, cwd, snapshot) {
     try {
       for (const file of (await ciFilesAt(cwd, tree, options.signal)).filter((candidate) => !touched.has(candidate.path)).slice(0, 100)) {
         try {
-          for (const command of gatingTestCommands(file.path, new TextDecoder().decode(await readBlob(cwd, file.oid, options.signal, 512 * 1024)))) elsewhere.add(command);
+          for (const command of enforcedTestCommands(file.path, new TextDecoder().decode(await readBlob(cwd, file.oid, options.signal, 512 * 1024)))) elsewhere.add(command);
         } catch (error) {
           assertNotAborted(options.signal);
         }
@@ -5760,6 +5854,7 @@ export {
   GitResolutionError,
   readGit,
   RULE_GUIDES,
+  assertsAlwaysTrue,
   isCodeFile,
   isTestFile,
   auditFileDiffs,

package/dist/cli.js

@@ -38,7 +38,7 @@ import {
   terminalText,
   uninstallHooks,
   writeBaseline
-} from "./chunk-BNDNXKBM.js";
+} from "./chunk-XVV5FNDV.js";
 
 // src/cli.ts
 import { Command } from "commander";

package/dist/index.d.ts

@@ -257,6 +257,10 @@ declare class GitResolutionError extends Error {
     constructor(message: string);
 }
 
+/** `assert check or True` and `self.assertTrue(check or True)` cannot fail: an operand of the `or` is the constant, and the check beside it decides nothing. Only the operands outside every
+ * bracket are read: `lambda paths: seen.append(paths) or True` inside a call is a callback that says it handled the call, and `identity(x or True)` is a value. `code` has its strings and
+ * comments blanked. */
+declare function assertsAlwaysTrue(code: string): boolean;
 declare function isCodeFile(filePath: string, customIgnorePaths?: string[]): boolean;
 declare function isTestFile(filePath: string): boolean;
 /**
@@ -741,4 +745,4 @@ declare function roastRepository(options?: RoastOptions): Promise<{
 }>;
 declare function auditPullRequest(options: PrAuditOptions): Promise<PrAuditResult>;
 
-export { AGENT_LABELS, type AgentRoastConfig, type AiAttributionResult, CONFIG_FILENAMES, type CheckOptions, type CheckResult, type CommitInfo, ConfigError, DEFAULT_CONFIG, DEFAULT_CONFIG_TEMPLATE, type DemoResult, type DetectorType, FIELD_SEPARATOR, type FindingBaseline, type FindingException, GITHUB_WORKFLOW_TEMPLATE, GitError, GitResolutionError, HERMETIC_GIT_FLAGS, HOOK_AGENTS, type HookAgent, type HookInstallChange, type HookInstallOptions, type HookOptions, type HookOutcome, type HookScope, INSTRUCTIONS_BLOCK, INSTRUCTIONS_END, INSTRUCTIONS_START, type Infraction, type InitOptions, type InitResult, InvalidDetectorError, MAX_CONFIG_BYTES, MCP_TOOLS, type McpServer, type McpServerOptions, type MetricSummary, MetricsCollector, PRE_COMMIT_HOOK_TEMPLATE, type ParseDiffResult, type ParsedCommit, type ParsedDiffLine, type ParsedFileDiff, type PrAuditOptions, type PrAuditResult, RECORD_SEPARATOR, RULE_GUIDES, type ReplayCommit, type ReplayOptions, type ReplayResult, type ReplayStatus, type ReviewFormat, type ReviewOptions, type ReviewResult, type RoastOptions, type RuleGuide, type ServeMcpOptions, type ShieldsEndpointResponse, type StagedReviewOptions, type SuppressedFinding, VALID_DETECTORS, applyBaseline, applyExceptions, auditFileDiffs, auditPullRequest, checkProject, compareFindings, configFromText, createBaseline, createMcpServer, detectAiAttribution, findMergeBase, findingFingerprint, formatAgentFeedback, formatCheckMarkdown, formatCheckTerminal, formatGithubAnnotations, formatGithubStepSummary, formatPrMarkdownComment, formatPrTerminalCard, formatReplayTerminal, formatReviewMarkdown, formatReviewTerminal, formatSarif, formatTerminalCard, generateShareUrl, generateShieldsJson, generateShieldsMarkdown, generateSvgBadge, getHermeticGitEnv, getScoreColor, gitOutputPath, hookCommand, hookConfigPath, hookPausedUntil, hookStatus, initProject, installHooks, installInstructions, isCheckRule, isCodeFile, isTestFile, loadBaseline, loadConfig, matchIgnorePath, mostImportantFirst, packageVersion, parseDiff, parseDuration, parseGitLogStream, pauseHook, replayHistory, repositoryRoot, resolveCommitSha, resumeHook, reviewChanges, reviewStagedChanges, reviewWorktreeChanges, roastRepository, runDemo, runHook, serveMcp, streamGitDiff, streamGitLog, stripStringAndRegexLiterals, terminalText, uninstallHooks, unquoteGitPath, validateExceptions, writeBaseline };
+export { AGENT_LABELS, type AgentRoastConfig, type AiAttributionResult, CONFIG_FILENAMES, type CheckOptions, type CheckResult, type CommitInfo, ConfigError, DEFAULT_CONFIG, DEFAULT_CONFIG_TEMPLATE, type DemoResult, type DetectorType, FIELD_SEPARATOR, type FindingBaseline, type FindingException, GITHUB_WORKFLOW_TEMPLATE, GitError, GitResolutionError, HERMETIC_GIT_FLAGS, HOOK_AGENTS, type HookAgent, type HookInstallChange, type HookInstallOptions, type HookOptions, type HookOutcome, type HookScope, INSTRUCTIONS_BLOCK, INSTRUCTIONS_END, INSTRUCTIONS_START, type Infraction, type InitOptions, type InitResult, InvalidDetectorError, MAX_CONFIG_BYTES, MCP_TOOLS, type McpServer, type McpServerOptions, type MetricSummary, MetricsCollector, PRE_COMMIT_HOOK_TEMPLATE, type ParseDiffResult, type ParsedCommit, type ParsedDiffLine, type ParsedFileDiff, type PrAuditOptions, type PrAuditResult, RECORD_SEPARATOR, RULE_GUIDES, type ReplayCommit, type ReplayOptions, type ReplayResult, type ReplayStatus, type ReviewFormat, type ReviewOptions, type ReviewResult, type RoastOptions, type RuleGuide, type ServeMcpOptions, type ShieldsEndpointResponse, type StagedReviewOptions, type SuppressedFinding, VALID_DETECTORS, applyBaseline, applyExceptions, assertsAlwaysTrue, auditFileDiffs, auditPullRequest, checkProject, compareFindings, configFromText, createBaseline, createMcpServer, detectAiAttribution, findMergeBase, findingFingerprint, formatAgentFeedback, formatCheckMarkdown, formatCheckTerminal, formatGithubAnnotations, formatGithubStepSummary, formatPrMarkdownComment, formatPrTerminalCard, formatReplayTerminal, formatReviewMarkdown, formatReviewTerminal, formatSarif, formatTerminalCard, generateShareUrl, generateShieldsJson, generateShieldsMarkdown, generateSvgBadge, getHermeticGitEnv, getScoreColor, gitOutputPath, hookCommand, hookConfigPath, hookPausedUntil, hookStatus, initProject, installHooks, installInstructions, isCheckRule, isCodeFile, isTestFile, loadBaseline, loadConfig, matchIgnorePath, mostImportantFirst, packageVersion, parseDiff, parseDuration, parseGitLogStream, pauseHook, replayHistory, repositoryRoot, resolveCommitSha, resumeHook, reviewChanges, reviewStagedChanges, reviewWorktreeChanges, roastRepository, runDemo, runHook, serveMcp, streamGitDiff, streamGitLog, stripStringAndRegexLiterals, terminalText, uninstallHooks, unquoteGitPath, validateExceptions, writeBaseline };

package/dist/index.js

@@ -24,6 +24,7 @@ import {
   VALID_DETECTORS,
   applyBaseline,
   applyExceptions,
+  assertsAlwaysTrue,
   auditFileDiffs,
   auditPullRequest,
   checkProject,
@@ -91,7 +92,7 @@ import {
   unquoteGitPath,
   validateExceptions,
   writeBaseline
-} from "./chunk-BNDNXKBM.js";
+} from "./chunk-XVV5FNDV.js";
 export {
   AGENT_LABELS,
   CONFIG_FILENAMES,
@@ -117,6 +118,7 @@ export {
   VALID_DETECTORS,
   applyBaseline,
   applyExceptions,
+  assertsAlwaysTrue,
   auditFileDiffs,
   auditPullRequest,
   checkProject,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "agent-roast",
-  "version": "0.20.0",
+  "version": "0.21.0",
   "description": "Stops AI coding agents (and rushed humans) from turning a build green by weakening tests, coverage gates or CI. Hook for Claude Code, Codex and Cursor, pre-commit, GitHub Action. Offline.",
   "mcpName": "io.github.Nari-Ab/agent-roast",
   "type": "module",