agent-roast 0.8.1 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +62 -23
- package/dist/{chunk-U7OCVA3D.js → chunk-GMBQNL6Z.js} +451 -61
- package/dist/cli.js +1 -1
- package/dist/index.d.ts +1 -1
- package/dist/index.js +1 -1
- package/package.json +12 -8
- package/schema.json +33 -1
package/README.md
CHANGED
|
@@ -17,36 +17,75 @@
|
|
|
17
17
|
|
|
18
18
|
</div>
|
|
19
19
|
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
20
|
+
**Did the AI fix the code, or just make the checks pass?**
|
|
21
|
+
|
|
22
|
+
Coding agents (and tired humans) often get a red build green by changing the *checks*
|
|
23
|
+
instead of the *code*: deleting the failing assertion, weakening it to `toBeDefined()`,
|
|
24
|
+
removing the test, lowering the coverage threshold, switching off `strict`, or letting
|
|
25
|
+
CI steps fail. A linter reading the new file cannot see that, because nothing in the new
|
|
26
|
+
file looks wrong. `agent-roast review` compares the **before and after** of a change and
|
|
27
|
+
flags exactly these moves. It is offline, runs no project code, and needs only Git.
|
|
24
28
|
|
|
25
29
|
```bash
|
|
26
|
-
|
|
27
|
-
npx agent-roast
|
|
30
|
+
npx agent-roast review --base origin/main # a pull request
|
|
31
|
+
npx agent-roast --staged # what you are about to commit
|
|
32
|
+
```
|
|
28
33
|
|
|
29
|
-
|
|
34
|
+
| Rule | Fires when | Blocks by default |
|
|
35
|
+
| --- | --- | --- |
|
|
36
|
+
| `assertion-weakened` | a test keeps its name but loses assertions, or an exact check becomes `toBeDefined`/`toBeTruthy`/`assert x` | yes |
|
|
37
|
+
| `guard-weakened` | coverage threshold lowered, `strict`/`noImplicitAny` off, lint rule downgraded, CI step may fail or no longer runs tests, test script cannot fail | yes |
|
|
38
|
+
| `test-removed` | a test (or test file) disappears and no identical body appears elsewhere | no, reported |
|
|
39
|
+
| `expectation-rewritten` | an expected value changed, test inputs did not, and production code changed in the same PR | no, reported |
|
|
40
|
+
| `test-skip`, `test-focus`, `trivial-assert`, `swallowed-error`, `type-escape`, `stub-abandonment` | the classic shortcuts, compared before/after so old debt does not block | skip/focus/trivial: yes |
|
|
41
|
+
|
|
42
|
+
Renames, moved tests, parametrization, and formatter rewraps (prettier, black) are
|
|
43
|
+
recognized and not reported. Intentional cases can be accepted one by one with an
|
|
44
|
+
expiring, reviewed exception (see below). Policy is read from the base branch, so a PR
|
|
45
|
+
cannot switch the rules off for itself.
|
|
46
|
+
|
|
47
|
+
### How well does it work?
|
|
48
|
+
|
|
49
|
+
Measured, not promised (`npm run build`, then `npm run bench` and `npm run bench:noise`;
|
|
50
|
+
method and raw numbers are in [docs/benchmarks.md](docs/benchmarks.md)):
|
|
51
|
+
|
|
52
|
+
- **Labeled changes** (`benchmarks/cases.mjs`): 32 cheating edits across JS/TS, Python, Go
|
|
53
|
+
and CI/config files are caught, and 15 harmless look-alikes (reformat, rename, move,
|
|
54
|
+
parametrize, snapshot, raised thresholds, CI step split) stay silent. The cases were
|
|
55
|
+
written together with the detector, so they guard against regressions; they do not prove
|
|
56
|
+
accuracy. Three known gaps are listed in the file and still open: mocking the module
|
|
57
|
+
under test, hardcoding the tested value, and vacuous matchers such as `expect.anything()`.
|
|
58
|
+
- **Noise on real human history** (the last 300 commits of commander.js, execa, click,
|
|
59
|
+
requests and zod; 606 of them touched tests or CI/config): the two blocking rules fired on
|
|
60
|
+
7 commits (1.2%). Each was read by hand: one was a real defect (a refactor in `requests`
|
|
61
|
+
dropped the `assert` keyword, leaving a comparison with no effect); the other six were
|
|
62
|
+
deliberate rewrites a reviewer should still see, such as a test-framework migration,
|
|
63
|
+
dropping Python 2, or a documented `continue-on-error`. `test-removed` fired on 6.1% and
|
|
64
|
+
`expectation-rewritten` on 2.8% of those commits, which is why they only report.
|
|
65
|
+
- **Not measured yet:** accuracy on real AI-agent pull requests. The sample above is human
|
|
66
|
+
history, and it is small. Treat the numbers as a floor for noise, not a guarantee.
|
|
67
|
+
|
|
68
|
+
It is a **review aid**. It cannot prove who wrote a change or that code is correct, and it
|
|
69
|
+
reads text, not program semantics. A green result is not a quality certificate.
|
|
70
|
+
|
|
71
|
+
### Quick start
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
# Everything that is true of the code you have now
|
|
75
|
+
npx agent-roast check
|
|
30
76
|
npx agent-roast explain swallowed-error
|
|
31
77
|
|
|
32
|
-
# Adopt
|
|
78
|
+
# Adopt in an existing project without blocking on historical debt
|
|
33
79
|
npx agent-roast check --write-baseline .agentroast-baseline.json
|
|
34
|
-
# Review the findings and commit this policy file, then gate only new problems
|
|
35
80
|
npx agent-roast check --baseline .agentroast-baseline.json
|
|
36
81
|
|
|
37
|
-
#
|
|
38
|
-
npx agent-roast --staged
|
|
39
|
-
|
|
40
|
-
# Audit pull request diffs (CI gate)
|
|
41
|
-
npx agent-roast review --base origin/main --head HEAD --format github
|
|
42
|
-
|
|
43
|
-
# Optional historical audit
|
|
82
|
+
# Optional: score a repository's history
|
|
44
83
|
npx agent-roast --all --verbose
|
|
45
84
|
```
|
|
46
85
|
|
|
47
|
-
A finding includes its file and line, the
|
|
48
|
-
|
|
49
|
-
|
|
86
|
+
A finding includes its file and line, the evidence, and a concrete next step.
|
|
87
|
+
`--format json|markdown|sarif|github` are available; terminal output shows the first
|
|
88
|
+
20 findings.
|
|
50
89
|
|
|
51
90
|
### Context review for pull requests
|
|
52
91
|
|
|
@@ -106,12 +145,12 @@ regressions) may appear in the report/summary rather than as an inline PR annota
|
|
|
106
145
|
Use a separate SARIF category for introduced-findings review; it does not represent
|
|
107
146
|
a full scan of all outstanding alerts.
|
|
108
147
|
|
|
109
|
-
Install 0.
|
|
148
|
+
Install 0.9.0: `npx agent-roast@0.9.0 review --base origin/main`. For local development,
|
|
110
149
|
run `npm run build` and use `node dist/cli.js`.
|
|
111
150
|
|
|
112
151
|
### Precise exceptions (since 0.8.0)
|
|
113
152
|
|
|
114
|
-
Use `npx agent-roast@0.
|
|
153
|
+
Use `npx agent-roast@0.9.0 check --format json`, or build this checkout with
|
|
115
154
|
`npm run build` and run `node dist/cli.js`.
|
|
116
155
|
|
|
117
156
|
Some skipped integrations or abstract stubs are intentional. In 0.8.0 you can accept
|
|
@@ -347,7 +386,7 @@ jobs:
|
|
|
347
386
|
|
|
348
387
|
To pilot the Action without blocking on findings, set `advisory: true`. It passes
|
|
349
388
|
an explicit empty fail-on list; invalid configuration, Git errors and incomplete
|
|
350
|
-
scans still fail. The `v1` Action installs 0.
|
|
389
|
+
scans still fail. The `v1` Action installs 0.9.0. The CLI equivalent is `--fail-on ''`.
|
|
351
390
|
|
|
352
391
|
> **Security Note:** `agent-roast` runs in an isolated runner directory (`$RUNNER_TEMP`) with `--ignore-scripts` to neutralize malicious `.npmrc` files in untrusted pull requests. `pull_request_target` is not supported.
|
|
353
392
|
|
|
@@ -435,7 +474,7 @@ node scripts/verify-user-workflows.mjs dist/cli.js
|
|
|
435
474
|
against current code. Regression tests use disposable Git repositories. The programmatic
|
|
436
475
|
`auditPullRequest({ staged: true })` API retains the legacy added-lines contract;
|
|
437
476
|
use `reviewStagedChanges()` for the new context comparison and `ReviewResult`.
|
|
438
|
-
The composite Action installs exactly 0.
|
|
477
|
+
The composite Action installs exactly 0.9.0. For local development, use
|
|
439
478
|
`node dist/cli.js --staged` after building.
|
|
440
479
|
|
|
441
480
|
### Author & Support
|
|
@@ -786,6 +786,30 @@ var RULE_GUIDES = {
|
|
|
786
786
|
suggestion: "Implement the path or make its unsupported state explicit to callers. Check whether this is an intentional abstract contract.",
|
|
787
787
|
example: "throw new Error('Not implemented') \u2192 implement the operation and add a behavior test"
|
|
788
788
|
},
|
|
789
|
+
"assertion-weakened": {
|
|
790
|
+
title: "Test checks less than before",
|
|
791
|
+
why: "A test that keeps its name but loses assertions, or has one replaced by a weaker check, still passes while verifying less behavior.",
|
|
792
|
+
suggestion: "Restore the removed assertion or explain in the PR why that behavior is no longer required. Fix the code under test instead of weakening its check.",
|
|
793
|
+
example: "expect(total).toBe(42) \u2192 expect(total).toBeDefined() (restore the exact check)"
|
|
794
|
+
},
|
|
795
|
+
"test-removed": {
|
|
796
|
+
title: "Test removed",
|
|
797
|
+
why: "Deleting a failing test makes a suite green without fixing the behavior it covered. Removals are legitimate when the feature itself is gone.",
|
|
798
|
+
suggestion: "Confirm the behavior was intentionally removed. If it still exists, restore the test and fix the code.",
|
|
799
|
+
example: "it('rejects expired tokens', ...) deleted while token validation remains in src/"
|
|
800
|
+
},
|
|
801
|
+
"expectation-rewritten": {
|
|
802
|
+
title: "Expected value edited alongside production code",
|
|
803
|
+
why: "Changing an expected value while the test inputs stay the same can mean the test was adjusted to match new output instead of the output being corrected.",
|
|
804
|
+
suggestion: "Verify the new expectation against the requirement, not against the program's current output. Intentional behavior changes should say so in the PR.",
|
|
805
|
+
example: "expect(price(cart)).toBe(100) \u2192 toBe(90) with price() changed in the same PR"
|
|
806
|
+
},
|
|
807
|
+
"guard-weakened": {
|
|
808
|
+
title: "Quality gate made easier to pass",
|
|
809
|
+
why: "Lower coverage thresholds, disabled compiler strictness, downgraded lint rules, tolerated CI failures or an always-passing test script let defects through without touching the code.",
|
|
810
|
+
suggestion: "Keep the gate and fix the underlying failure. If the change is deliberate, record the reason and a plan to restore it.",
|
|
811
|
+
example: 'coverageThreshold lines: 80 \u2192 60, "strict": true \u2192 false, continue-on-error: true'
|
|
812
|
+
},
|
|
789
813
|
"panic-loop": {
|
|
790
814
|
title: "Rapid overlapping fix commits",
|
|
791
815
|
why: "Repeated short fix cycles can suggest a missing reproduction or feedback loop; timing alone does not prove poor work.",
|
|
@@ -806,7 +830,11 @@ var VALID_DETECTORS = [
|
|
|
806
830
|
"panic-loop",
|
|
807
831
|
"trivial-assert",
|
|
808
832
|
"stub-abandonment",
|
|
809
|
-
"test-focus"
|
|
833
|
+
"test-focus",
|
|
834
|
+
"assertion-weakened",
|
|
835
|
+
"test-removed",
|
|
836
|
+
"expectation-rewritten",
|
|
837
|
+
"guard-weakened"
|
|
810
838
|
];
|
|
811
839
|
|
|
812
840
|
// src/file-read.ts
|
|
@@ -839,13 +867,13 @@ function readRegularFile(filename, limit) {
|
|
|
839
867
|
import { createHash, randomUUID as randomUUID2 } from "crypto";
|
|
840
868
|
import * as fs2 from "fs";
|
|
841
869
|
import path from "path";
|
|
842
|
-
function findingFingerprint(
|
|
843
|
-
return createHash("sha256").update(JSON.stringify([
|
|
870
|
+
function findingFingerprint(finding2) {
|
|
871
|
+
return createHash("sha256").update(JSON.stringify([finding2.type, finding2.file, finding2.lineSnippet?.trim()])).digest("hex");
|
|
844
872
|
}
|
|
845
873
|
function createBaseline(findings) {
|
|
846
874
|
const counts = /* @__PURE__ */ new Map();
|
|
847
|
-
for (const
|
|
848
|
-
const fingerprint = findingFingerprint(
|
|
875
|
+
for (const finding2 of findings) {
|
|
876
|
+
const fingerprint = findingFingerprint(finding2);
|
|
849
877
|
counts.set(fingerprint, (counts.get(fingerprint) || 0) + 1);
|
|
850
878
|
}
|
|
851
879
|
return {
|
|
@@ -877,14 +905,14 @@ function applyBaseline(findings, baseline) {
|
|
|
877
905
|
const remaining = new Map(baseline.entries.map((entry) => [entry.fingerprint, entry.count]));
|
|
878
906
|
const fresh = [];
|
|
879
907
|
let baselineCount = 0;
|
|
880
|
-
for (const
|
|
881
|
-
const fingerprint = findingFingerprint(
|
|
908
|
+
for (const finding2 of findings) {
|
|
909
|
+
const fingerprint = findingFingerprint(finding2);
|
|
882
910
|
const count = remaining.get(fingerprint) || 0;
|
|
883
911
|
if (count > 0) {
|
|
884
912
|
remaining.set(fingerprint, count - 1);
|
|
885
913
|
baselineCount++;
|
|
886
914
|
} else {
|
|
887
|
-
fresh.push(
|
|
915
|
+
fresh.push(finding2);
|
|
888
916
|
}
|
|
889
917
|
}
|
|
890
918
|
return { findings: fresh, baselineCount, staleBaselineCount: [...remaining.values()].reduce((a, b) => a + b, 0) };
|
|
@@ -933,11 +961,11 @@ function applyExceptions(findings, exceptions = [], now = /* @__PURE__ */ new Da
|
|
|
933
961
|
const occurrences = new Map(priorOccurrences);
|
|
934
962
|
const actionable = [];
|
|
935
963
|
const suppressedFindings = [];
|
|
936
|
-
for (const
|
|
937
|
-
const fingerprint = findingFingerprint(
|
|
964
|
+
for (const finding2 of findings) {
|
|
965
|
+
const fingerprint = findingFingerprint(finding2);
|
|
938
966
|
const occurrence = (occurrences.get(fingerprint) ?? 0) + 1;
|
|
939
967
|
occurrences.set(fingerprint, occurrence);
|
|
940
|
-
const enriched = { ...
|
|
968
|
+
const enriched = { ...finding2, fingerprint, occurrence };
|
|
941
969
|
const allowance = active.get(fingerprint);
|
|
942
970
|
if (allowance && allowance.remaining > 0) {
|
|
943
971
|
allowance.remaining--;
|
|
@@ -950,7 +978,7 @@ function applyExceptions(findings, exceptions = [], now = /* @__PURE__ */ new Da
|
|
|
950
978
|
// src/config.ts
|
|
951
979
|
var MAX_CONFIG_BYTES = 64 * 1024;
|
|
952
980
|
var DEFAULT_CONFIG = {
|
|
953
|
-
failOn: ["test-skip", "trivial-assert", "test-focus"],
|
|
981
|
+
failOn: ["test-skip", "trivial-assert", "test-focus", "assertion-weakened", "guard-weakened"],
|
|
954
982
|
ignorePaths: [
|
|
955
983
|
"**/fixtures/**",
|
|
956
984
|
"**/__fixtures__/**",
|
|
@@ -966,6 +994,10 @@ var DEFAULT_CONFIG = {
|
|
|
966
994
|
"test-skip": "error",
|
|
967
995
|
"trivial-assert": "error",
|
|
968
996
|
"test-focus": "error",
|
|
997
|
+
"assertion-weakened": "error",
|
|
998
|
+
"guard-weakened": "error",
|
|
999
|
+
"test-removed": "warn",
|
|
1000
|
+
"expectation-rewritten": "warn",
|
|
969
1001
|
"swallowed-error": "warn",
|
|
970
1002
|
"stub-abandonment": "warn",
|
|
971
1003
|
"type-escape": "warn"
|
|
@@ -1395,7 +1427,7 @@ function auditFileDiffs(files, commitHash = "HEAD", commitDate = (/* @__PURE__ *
|
|
|
1395
1427
|
});
|
|
1396
1428
|
}
|
|
1397
1429
|
const ordered = infractions.splice(firstFinding).sort((left, right) => (left.lineNumber || 0) - (right.lineNumber || 0));
|
|
1398
|
-
for (const
|
|
1430
|
+
for (const finding2 of ordered) infractions.push(finding2);
|
|
1399
1431
|
}
|
|
1400
1432
|
return { infractions: infractions.filter((inf) => options.rules?.[inf.type] !== "off").map((inf) => ({ ...inf, suggestion: RULE_GUIDES[inf.type].suggestion })), hasSupportedChanges, filesInspected, linesAdded };
|
|
1401
1433
|
}
|
|
@@ -1530,7 +1562,7 @@ var MetricsCollector = class {
|
|
|
1530
1562
|
const counts = {};
|
|
1531
1563
|
for (const infraction of audit.infractions) {
|
|
1532
1564
|
const count = counts[infraction.type] || 0;
|
|
1533
|
-
if (count < MAX_INFRACTIONS_PER_COMMIT) buckets[infraction.type]
|
|
1565
|
+
if (count < MAX_INFRACTIONS_PER_COMMIT) buckets[infraction.type]?.push(infraction);
|
|
1534
1566
|
counts[infraction.type] = count + 1;
|
|
1535
1567
|
}
|
|
1536
1568
|
}
|
|
@@ -1976,8 +2008,8 @@ function exceptionTerminal(result) {
|
|
|
1976
2008
|
const accepted = result.suppressedFindings || [];
|
|
1977
2009
|
if (accepted.length) {
|
|
1978
2010
|
lines.push(` ${accepted.length} findings accepted by explicit exceptions (not fixed)`);
|
|
1979
|
-
for (const
|
|
1980
|
-
` ACCEPTED [${
|
|
2011
|
+
for (const finding2 of accepted.slice(0, 20)) lines.push(
|
|
2012
|
+
` ACCEPTED [${finding2.type}] ${terminalText(finding2.file || "unknown")}:${finding2.lineNumber} \u2014 ${terminalText(finding2.suppression.reason)} (expires ${terminalText(finding2.suppression.expires)})`
|
|
1981
2013
|
);
|
|
1982
2014
|
if (accepted.length > 20) lines.push(` ${accepted.length - 20} more accepted findings; use --format json.`);
|
|
1983
2015
|
}
|
|
@@ -1997,7 +2029,7 @@ function exceptionMarkdown(result) {
|
|
|
1997
2029
|
"| Rule | Location | Reason | Expires |",
|
|
1998
2030
|
"| --- | --- | --- | --- |"
|
|
1999
2031
|
);
|
|
2000
|
-
for (const
|
|
2032
|
+
for (const finding2 of accepted) lines.push(`| ${finding2.type} | ${markdownCell(`${finding2.file}:${finding2.lineNumber}`)} | ${markdownCell(finding2.suppression.reason)} | ${markdownCell(finding2.suppression.expires)} |`);
|
|
2001
2033
|
}
|
|
2002
2034
|
if (result.expiredExceptionCount) lines.push("", `${result.expiredExceptionCount} expired exceptions were not applied; review their findings.`);
|
|
2003
2035
|
return lines;
|
|
@@ -2054,12 +2086,12 @@ function formatReviewTerminal(result) {
|
|
|
2054
2086
|
lines.push(...exceptionTerminal(result));
|
|
2055
2087
|
if (result.policyChanges.length) lines.push(" Candidate policy changes were ignored; adopt them separately after review.");
|
|
2056
2088
|
if (result.excludedFiles || result.unsupportedFiles) lines.push(` ${result.excludedFiles} excluded \xB7 ${result.unsupportedFiles} unsupported/non-source changes`);
|
|
2057
|
-
for (const
|
|
2089
|
+
for (const finding2 of result.infractions.slice(0, 20)) {
|
|
2058
2090
|
lines.push(
|
|
2059
2091
|
"",
|
|
2060
|
-
` ${result.failedRules.includes(
|
|
2061
|
-
` ${terminalText(
|
|
2062
|
-
` Next: ${terminalText(
|
|
2092
|
+
` ${result.failedRules.includes(finding2.type) ? "FAIL" : "WARN"} [${finding2.type}] ${terminalText(finding2.file || "unknown")}:${finding2.lineNumber}`,
|
|
2093
|
+
` ${terminalText(finding2.lineSnippet || finding2.reason)}`,
|
|
2094
|
+
` Next: ${terminalText(finding2.suggestion || "Review this occurrence.")}`
|
|
2063
2095
|
);
|
|
2064
2096
|
}
|
|
2065
2097
|
if (result.infractions.length > 20) lines.push(` ${result.infractions.length - 20} more new findings; use --format json for all results.`);
|
|
@@ -2080,7 +2112,7 @@ function formatReviewMarkdown(result) {
|
|
|
2080
2112
|
];
|
|
2081
2113
|
if (result.policyChanges.length) lines.push("Candidate policy changes were ignored. This review uses the supplied base commit's policy.", "");
|
|
2082
2114
|
lines.push("| New finding | Head location | Evidence | Next step |", "| --- | --- | --- | --- |");
|
|
2083
|
-
for (const
|
|
2115
|
+
for (const finding2 of result.infractions) lines.push(`| ${finding2.type} | ${markdownCell(`${finding2.file}:${finding2.lineNumber}`)} | ${markdownCell(finding2.lineSnippet || finding2.reason)} | ${markdownCell(finding2.suggestion || "Review this occurrence.")} |`);
|
|
2084
2116
|
lines.push(...exceptionMarkdown(result));
|
|
2085
2117
|
if (result.removedFindings.length) {
|
|
2086
2118
|
lines.push(
|
|
@@ -2092,7 +2124,7 @@ function formatReviewMarkdown(result) {
|
|
|
2092
2124
|
"| Rule | Previous location | Previous evidence |",
|
|
2093
2125
|
"| --- | --- | --- |"
|
|
2094
2126
|
);
|
|
2095
|
-
for (const
|
|
2127
|
+
for (const finding2 of result.removedFindings) lines.push(`| ${finding2.type} | ${markdownCell(`${finding2.file}:${finding2.lineNumber}`)} | ${markdownCell(finding2.lineSnippet || finding2.reason)} |`);
|
|
2096
2128
|
}
|
|
2097
2129
|
for (const skipped of result.skippedFiles) lines.push("", `Not compared: ${markdownCell(skipped.file)} \u2014 ${markdownCell(skipped.reason)}`);
|
|
2098
2130
|
lines.push("", `Scope: changed supported files only; ${result.excludedFiles} excluded and ${result.unsupportedFiles} unsupported/non-source changes.`);
|
|
@@ -2338,26 +2370,26 @@ var version = require2("../package.json").version;
|
|
|
2338
2370
|
function formatSarif(report) {
|
|
2339
2371
|
const occurrences = /* @__PURE__ */ new Map();
|
|
2340
2372
|
const allFindings = [
|
|
2341
|
-
...report.infractions.map((
|
|
2342
|
-
...(report.suppressedFindings ?? []).map((
|
|
2373
|
+
...report.infractions.map((finding2) => ({ finding: finding2, suppression: void 0 })),
|
|
2374
|
+
...(report.suppressedFindings ?? []).map((finding2) => ({ finding: finding2, suppression: finding2.suppression }))
|
|
2343
2375
|
];
|
|
2344
2376
|
allFindings.sort((a, b) => (a.finding.file ?? "").localeCompare(b.finding.file ?? "") || (a.finding.lineNumber ?? 1) - (b.finding.lineNumber ?? 1) || a.finding.type.localeCompare(b.finding.type));
|
|
2345
|
-
const results = allFindings.map(({ finding, suppression }) => {
|
|
2346
|
-
const filename =
|
|
2377
|
+
const results = allFindings.map(({ finding: finding2, suppression }) => {
|
|
2378
|
+
const filename = finding2.file;
|
|
2347
2379
|
if (!filename || filename.startsWith("/") || filename.split("/").includes("..") || /^[a-z]:/i.test(filename)) {
|
|
2348
2380
|
throw new ConfigError("SARIF needs repository-relative finding paths.");
|
|
2349
2381
|
}
|
|
2350
|
-
const fingerprint = findingFingerprint(
|
|
2351
|
-
const occurrence =
|
|
2382
|
+
const fingerprint = findingFingerprint(finding2);
|
|
2383
|
+
const occurrence = finding2.occurrence ?? (occurrences.get(fingerprint) || 0) + 1;
|
|
2352
2384
|
occurrences.set(fingerprint, occurrence);
|
|
2353
2385
|
return {
|
|
2354
|
-
ruleId:
|
|
2355
|
-
ruleIndex: VALID_DETECTORS.indexOf(
|
|
2356
|
-
level: report.failedRules.includes(
|
|
2357
|
-
message: { text: `${
|
|
2386
|
+
ruleId: finding2.type,
|
|
2387
|
+
ruleIndex: VALID_DETECTORS.indexOf(finding2.type),
|
|
2388
|
+
level: report.failedRules.includes(finding2.type) ? "error" : "warning",
|
|
2389
|
+
message: { text: `${finding2.reason}. ${finding2.suggestion || RULE_GUIDES[finding2.type].suggestion}` },
|
|
2358
2390
|
locations: [{ physicalLocation: {
|
|
2359
2391
|
artifactLocation: { uri: filename.split("/").map((segment) => encodeURIComponent(segment)).join("/") },
|
|
2360
|
-
region: { startLine: Math.max(1,
|
|
2392
|
+
region: { startLine: Math.max(1, finding2.lineNumber || 1) }
|
|
2361
2393
|
} }],
|
|
2362
2394
|
partialFingerprints: { "agentRoastFinding/v1": `${fingerprint}:${occurrence}` },
|
|
2363
2395
|
...suppression ? { suppressions: [{
|
|
@@ -2662,7 +2694,7 @@ async function checkProject(options = {}) {
|
|
|
2662
2694
|
const cwd = path5.resolve(options.cwd || process.cwd());
|
|
2663
2695
|
const root = await fs6.realpath(gitOutputPath(await git(cwd, ["rev-parse", "--show-toplevel"], options.signal)));
|
|
2664
2696
|
const config = loadConfig(root);
|
|
2665
|
-
const failOn = options.failOn ?? config.failOn ?? ["test-skip", "trivial-assert", "test-focus"];
|
|
2697
|
+
const failOn = options.failOn ?? config.failOn ?? ["test-skip", "trivial-assert", "test-focus", "assertion-weakened", "guard-weakened"];
|
|
2666
2698
|
for (const rule of failOn) if (!VALID_DETECTORS.includes(rule)) throw new ConfigError(`Unknown rule '${rule}'.`);
|
|
2667
2699
|
const format = options.format || "terminal";
|
|
2668
2700
|
if (!["terminal", "json", "github", "markdown", "sarif"].includes(format)) throw new ConfigError("Unsupported check output format.");
|
|
@@ -2699,14 +2731,14 @@ async function checkProject(options = {}) {
|
|
|
2699
2731
|
const applied = baseline ? applyBaseline(infractions, baseline) : { findings: infractions, baselineCount: 0, staleBaselineCount: 0 };
|
|
2700
2732
|
const notInBaseline = new Set(applied.findings);
|
|
2701
2733
|
const acceptedCounts = /* @__PURE__ */ new Map();
|
|
2702
|
-
for (const
|
|
2703
|
-
if (notInBaseline.has(
|
|
2704
|
-
const fingerprint = findingFingerprint(
|
|
2734
|
+
for (const finding2 of infractions) {
|
|
2735
|
+
if (notInBaseline.has(finding2)) continue;
|
|
2736
|
+
const fingerprint = findingFingerprint(finding2);
|
|
2705
2737
|
acceptedCounts.set(fingerprint, (acceptedCounts.get(fingerprint) ?? 0) + 1);
|
|
2706
2738
|
}
|
|
2707
2739
|
const exceptions = applyExceptions(applied.findings, options.ignoreExceptions ? [] : config.exceptions, /* @__PURE__ */ new Date(), acceptedCounts);
|
|
2708
2740
|
const findings = exceptions.findings;
|
|
2709
|
-
const failedRules = [...new Set(findings.filter((
|
|
2741
|
+
const failedRules = [...new Set(findings.filter((finding2) => failOn.includes(finding2.type)).map((finding2) => finding2.type))];
|
|
2710
2742
|
const complete = skippedFiles.length === 0;
|
|
2711
2743
|
const passed = complete && failedRules.length === 0;
|
|
2712
2744
|
const result = {
|
|
@@ -2738,24 +2770,355 @@ async function checkProject(options = {}) {
|
|
|
2738
2770
|
|
|
2739
2771
|
// src/review.ts
|
|
2740
2772
|
import path6 from "path";
|
|
2773
|
+
|
|
2774
|
+
// src/tampering.ts
|
|
2775
|
+
import { createHash as createHash3 } from "crypto";
|
|
2776
|
+
var EMPTY = { immediate: [], removedTests: [], addedHashes: [], rewritten: [] };
|
|
2777
|
+
function languageOf(file) {
|
|
2778
|
+
if (/\.(?:[cm]?[jt]sx?)$/i.test(file)) return "js";
|
|
2779
|
+
if (/\.py$/i.test(file)) return "py";
|
|
2780
|
+
if (/_test\.go$/i.test(file)) return "go";
|
|
2781
|
+
return null;
|
|
2782
|
+
}
|
|
2783
|
+
function codeLines(text, language) {
|
|
2784
|
+
let block = false;
|
|
2785
|
+
return text.split(/\r?\n/).map((line) => {
|
|
2786
|
+
const trimmed = line.trim();
|
|
2787
|
+
if (language !== "py") {
|
|
2788
|
+
if (block) {
|
|
2789
|
+
if (trimmed.includes("*/")) block = false;
|
|
2790
|
+
return "";
|
|
2791
|
+
}
|
|
2792
|
+
if (trimmed.startsWith("/*")) {
|
|
2793
|
+
if (!trimmed.includes("*/")) block = true;
|
|
2794
|
+
return "";
|
|
2795
|
+
}
|
|
2796
|
+
if (trimmed.startsWith("//")) return "";
|
|
2797
|
+
} else if (trimmed.startsWith("#")) return "";
|
|
2798
|
+
return line;
|
|
2799
|
+
});
|
|
2800
|
+
}
|
|
2801
|
+
var JS_TEST_START = /(?<![\w$.])(?:it|test)(?:\s*\.\s*(?:only|skip|concurrent|serial))*\s*\(\s*(['"`])((?:\\.|(?!\1)[^\\\n])*)\1/;
|
|
2802
|
+
var PY_TEST_START = /^\s*(?:async\s+)?def\s+(test\w*)\s*\(/;
|
|
2803
|
+
var GO_TEST_START = /^func\s+(Test\w+)\s*\(/;
|
|
2804
|
+
var JS_ASSERTION = /(?<![\w$.])(?:expect|assert|expectTypeOf)\b(?!\s*\.\s*(?:assertions|hasAssertions|extend|anything|any|objectContaining|arrayContaining|stringContaining|stringMatching)\b)(?:\s*\.\s*\w+)*\s*\(|\.should\b/;
|
|
2805
|
+
var PY_ASSERTION = /^\s*assert\b|\bself\s*\.\s*assert\w*\s*\(|\bpytest\s*\.\s*raises\b|\bassertRaises\b/;
|
|
2806
|
+
var GO_ASSERTION = /\bt\s*\.\s*(?:Error|Errorf|Fatal|Fatalf|Fail|FailNow)\b|\b(?:assert|require)\s*\.\s*\w+\s*\(/;
|
|
2807
|
+
function callEnd(text, from, limit) {
|
|
2808
|
+
let depth = 1, quote = "";
|
|
2809
|
+
for (let i = from; i < limit; i++) {
|
|
2810
|
+
const char = text[i];
|
|
2811
|
+
if (quote) {
|
|
2812
|
+
if (char === "\\") i++;
|
|
2813
|
+
else if (char === quote || char === "\n" && quote !== "`") quote = "";
|
|
2814
|
+
} else if (char === '"' || char === "'" || char === "`") quote = char;
|
|
2815
|
+
else if (char === "/" && text[i + 1] === "/" && (i === from || /\s/.test(text[i - 1]))) {
|
|
2816
|
+
const newline = text.indexOf("\n", i);
|
|
2817
|
+
i = newline < 0 || newline > limit ? limit : newline;
|
|
2818
|
+
} else if (char === "(") depth++;
|
|
2819
|
+
else if (char === ")" && --depth === 0) return i;
|
|
2820
|
+
}
|
|
2821
|
+
return limit;
|
|
2822
|
+
}
|
|
2823
|
+
function testBodies(text, language) {
|
|
2824
|
+
const lines = codeLines(text, language);
|
|
2825
|
+
if (language === "js") {
|
|
2826
|
+
const code = lines.join("\n");
|
|
2827
|
+
const matches = [...code.matchAll(new RegExp(JS_TEST_START.source, "g"))];
|
|
2828
|
+
let line = 1, cursor = 0;
|
|
2829
|
+
return matches.map((match, i) => {
|
|
2830
|
+
const at = match.index ?? 0;
|
|
2831
|
+
for (; cursor < at; cursor++) if (code.charCodeAt(cursor) === 10) line++;
|
|
2832
|
+
const from = at + match[0].length;
|
|
2833
|
+
const limit = i + 1 < matches.length ? matches[i + 1].index ?? code.length : code.length;
|
|
2834
|
+
return { name: match[2], line, body: code.slice(from, Math.min(callEnd(code, from, limit) + 1, limit)), parametrized: false };
|
|
2835
|
+
});
|
|
2836
|
+
}
|
|
2837
|
+
const startPattern = language === "py" ? PY_TEST_START : GO_TEST_START;
|
|
2838
|
+
const starts = [];
|
|
2839
|
+
lines.forEach((line, index) => {
|
|
2840
|
+
const match = startPattern.exec(line);
|
|
2841
|
+
if (match) starts.push({ index, name: match[1], length: match[0].length });
|
|
2842
|
+
});
|
|
2843
|
+
const indentOf = (line) => /^\s*/.exec(line)[0].length;
|
|
2844
|
+
return starts.map((start, n) => {
|
|
2845
|
+
const limit = n + 1 < starts.length ? starts[n + 1].index : lines.length;
|
|
2846
|
+
const header = lines[start.index], indent = indentOf(header);
|
|
2847
|
+
let end = start.index + 1, depth = balance(header);
|
|
2848
|
+
while (depth > 0 && end < limit) depth += balance(lines[end++]);
|
|
2849
|
+
while (end < limit && (language === "go" ? lines[end - 1] !== "}" : !lines[end].trim() || indentOf(lines[end]) > indent)) end++;
|
|
2850
|
+
let top = start.index;
|
|
2851
|
+
while (top > 0 && start.index - top < 200 && lines[top - 1].trim() && !/^\s*(?:async\s+)?(?:def|class)\b/.test(lines[top - 1])) top--;
|
|
2852
|
+
return {
|
|
2853
|
+
name: start.name,
|
|
2854
|
+
line: start.index + 1,
|
|
2855
|
+
body: [header.slice(start.length), ...lines.slice(start.index + 1, end)].join("\n"),
|
|
2856
|
+
parametrized: /parametrize/.test(lines.slice(top, start.index).join("\n"))
|
|
2857
|
+
};
|
|
2858
|
+
});
|
|
2859
|
+
}
|
|
2860
|
+
function balance(line) {
|
|
2861
|
+
const bare = line.replace(/(['"`])(?:\\.|(?!\1)[^\\])*\1/g, "");
|
|
2862
|
+
let net = 0;
|
|
2863
|
+
for (const char of bare) net += "([{".includes(char) ? 1 : ")]}".includes(char) ? -1 : 0;
|
|
2864
|
+
return net;
|
|
2865
|
+
}
|
|
2866
|
+
function normalize(statement) {
|
|
2867
|
+
return statement.replace(/\s+/g, "").replace(/'/g, '"').replace(/,([)\]}])/g, "$1").replace(/;$/, "");
|
|
2868
|
+
}
|
|
2869
|
+
function splitStatements(text) {
|
|
2870
|
+
const pieces = [];
|
|
2871
|
+
let quote = "", start = 0;
|
|
2872
|
+
for (let i = 0; i < text.length; i++) {
|
|
2873
|
+
const char = text[i];
|
|
2874
|
+
if (quote) {
|
|
2875
|
+
if (char === "\\") i++;
|
|
2876
|
+
else if (char === quote) quote = "";
|
|
2877
|
+
} else if (char === '"' || char === "'" || char === "`") quote = char;
|
|
2878
|
+
else if (char === ";") {
|
|
2879
|
+
pieces.push(text.slice(start, i));
|
|
2880
|
+
start = i + 1;
|
|
2881
|
+
}
|
|
2882
|
+
}
|
|
2883
|
+
pieces.push(text.slice(start));
|
|
2884
|
+
return pieces.map((piece) => piece.trim()).filter(Boolean);
|
|
2885
|
+
}
|
|
2886
|
+
var CALLBACK_HEADER = /^[,\s]*(?:async\s*)?(?:\([^)]*\)|[\w$]+)\s*=>\s*\{\s*|^[,\s]*(?:async\s+)?function\s*\w*\s*\([^)]*\)\s*\{\s*/;
|
|
2887
|
+
function parseTests(text, language) {
|
|
2888
|
+
const assertionPattern = language === "js" ? JS_ASSERTION : language === "py" ? PY_ASSERTION : GO_ASSERTION;
|
|
2889
|
+
const seen = /* @__PURE__ */ new Map();
|
|
2890
|
+
return testBodies(text, language).map((test) => {
|
|
2891
|
+
const body = test.body.split("\n");
|
|
2892
|
+
const assertions = [], raw = [], setupLines = [];
|
|
2893
|
+
for (let i = 0; i < body.length; i++) {
|
|
2894
|
+
if (!assertionPattern.test(body[i])) {
|
|
2895
|
+
setupLines.push(body[i]);
|
|
2896
|
+
continue;
|
|
2897
|
+
}
|
|
2898
|
+
let statement = body[i].trim(), depth = balance(body[i]);
|
|
2899
|
+
while (depth > 0 && i + 1 < body.length) {
|
|
2900
|
+
i++;
|
|
2901
|
+
statement += " " + body[i].trim();
|
|
2902
|
+
depth += balance(body[i]);
|
|
2903
|
+
}
|
|
2904
|
+
for (const piece of language === "js" ? splitStatements(statement) : [statement]) {
|
|
2905
|
+
const clean = piece.replace(CALLBACK_HEADER, "");
|
|
2906
|
+
if (assertionPattern.test(clean)) {
|
|
2907
|
+
raw.push(clean);
|
|
2908
|
+
assertions.push(normalize(clean));
|
|
2909
|
+
} else setupLines.push(clean);
|
|
2910
|
+
}
|
|
2911
|
+
}
|
|
2912
|
+
const setup = setupLines.join("\n").replace(/\s+/g, "");
|
|
2913
|
+
const bodyHash = createHash3("sha1").update(test.body.replace(/\s+/g, "")).digest("hex");
|
|
2914
|
+
const count = (seen.get(test.name) ?? 0) + 1;
|
|
2915
|
+
seen.set(test.name, count);
|
|
2916
|
+
return { key: `${test.name}#${count}`, parametrized: test.parametrized, name: test.name, line: test.line, assertions, raw, setup, bodyHash };
|
|
2917
|
+
});
|
|
2918
|
+
}
|
|
2919
|
+
function skeleton(assertion) {
|
|
2920
|
+
return assertion.replace(/(['"`])(?:\\.|(?!\1)[^\\])*\1/g, "S").replace(/\b\d+(?:\.\d+)?\b/g, "N").replace(/\b(?:true|false|null|undefined|None|True|False|nil)\b/g, "V").replace(/\s+/g, "");
|
|
2921
|
+
}
|
|
2922
|
+
var WEAK_JS = /\.(?:toBeDefined|toBeTruthy|toBeFalsy|toBeUndefined)\s*\(|\.not\s*\.\s*(?:toBeNull|toBeUndefined|toThrow)\s*\(|\.toBeGreaterThanOrEqual\s*\(\s*0\s*\)/;
|
|
2923
|
+
var WEAK_PY = /^\s*assert\s+[\w.()\[\]]+\s*$|^\s*assert\s+.+\sis\s+not\s+None\s*$|^\s*assert\s+True\s*$/;
|
|
2924
|
+
function isWeaker(oldAssertion, newAssertion, language) {
|
|
2925
|
+
if (language === "js") return WEAK_JS.test(newAssertion) && !WEAK_JS.test(oldAssertion);
|
|
2926
|
+
if (language === "py") return WEAK_PY.test(newAssertion) && !WEAK_PY.test(oldAssertion);
|
|
2927
|
+
return false;
|
|
2928
|
+
}
|
|
2929
|
+
function finding(type, file, line, snippet, reason, commit) {
|
|
2930
|
+
return { type, file, lineNumber: Math.max(1, line), lineSnippet: snippet, commitHash: commit, commitDate: "", reason, suggestion: RULE_GUIDES[type].suggestion };
|
|
2931
|
+
}
|
|
2932
|
+
function analyzeTestFile(oldPath, newPath, oldText, newText, commit) {
|
|
2933
|
+
const language = languageOf(newText === null ? oldPath : newPath);
|
|
2934
|
+
if (!language || !(isTestFile(newPath) || isTestFile(oldPath))) return EMPTY;
|
|
2935
|
+
const before = oldText === null ? [] : parseTests(oldText, language);
|
|
2936
|
+
const after = newText === null ? [] : parseTests(newText, language);
|
|
2937
|
+
const afterByKey = new Map(after.map((test) => [test.key, test]));
|
|
2938
|
+
const newCode = newText === null ? "" : codeLines(newText, language).join("\n");
|
|
2939
|
+
const tally = (tests) => {
|
|
2940
|
+
const totals = /* @__PURE__ */ new Map();
|
|
2941
|
+
for (const test of tests) for (const assertion of test.assertions) totals.set(assertion, (totals.get(assertion) ?? 0) + 1);
|
|
2942
|
+
return totals;
|
|
2943
|
+
};
|
|
2944
|
+
const oldTotals = tally(before), newTotals = tally(after);
|
|
2945
|
+
const beforeKeys = new Set(before.map((test) => test.key));
|
|
2946
|
+
const result = { immediate: [], removedTests: [], addedHashes: [], rewritten: [] };
|
|
2947
|
+
for (const test of after) if (!beforeKeys.has(test.key)) result.addedHashes.push(test.bodyHash);
|
|
2948
|
+
for (const old of before) {
|
|
2949
|
+
const current = afterByKey.get(old.key);
|
|
2950
|
+
if (!current) {
|
|
2951
|
+
if (newCode && ['"', "'", "`"].some((quote) => newCode.includes(quote + old.name + quote))) continue;
|
|
2952
|
+
result.removedTests.push({
|
|
2953
|
+
bodyHash: old.bodyHash,
|
|
2954
|
+
finding: finding(
|
|
2955
|
+
"test-removed",
|
|
2956
|
+
newPath,
|
|
2957
|
+
1,
|
|
2958
|
+
`${old.name}: test no longer present`,
|
|
2959
|
+
newText === null ? "Test file deleted" : "Test removed",
|
|
2960
|
+
commit
|
|
2961
|
+
)
|
|
2962
|
+
});
|
|
2963
|
+
continue;
|
|
2964
|
+
}
|
|
2965
|
+
if (current.parametrized && !old.parametrized) continue;
|
|
2966
|
+
const snapshot = (test) => test.raw.some((statement) => /\.toMatch(?:Inline)?Snapshot\b|\.matchSnapshot\b|\bsnapshot\s*\(/.test(statement));
|
|
2967
|
+
if (snapshot(current) && !snapshot(old)) continue;
|
|
2968
|
+
if (current.assertions.length < old.assertions.length) {
|
|
2969
|
+
const budget = tally([current]);
|
|
2970
|
+
const lost = old.assertions.filter((assertion) => {
|
|
2971
|
+
const left = budget.get(assertion) ?? 0;
|
|
2972
|
+
if (left > 0) {
|
|
2973
|
+
budget.set(assertion, left - 1);
|
|
2974
|
+
return false;
|
|
2975
|
+
}
|
|
2976
|
+
return true;
|
|
2977
|
+
});
|
|
2978
|
+
if (lost.every((assertion) => (newTotals.get(assertion) ?? 0) >= (oldTotals.get(assertion) ?? 0))) continue;
|
|
2979
|
+
result.immediate.push(finding(
|
|
2980
|
+
"assertion-weakened",
|
|
2981
|
+
newPath,
|
|
2982
|
+
current.line,
|
|
2983
|
+
`${old.name}: ${old.assertions.length} \u2192 ${current.assertions.length} assertions`,
|
|
2984
|
+
"Test checks less than before",
|
|
2985
|
+
commit
|
|
2986
|
+
));
|
|
2987
|
+
continue;
|
|
2988
|
+
}
|
|
2989
|
+
if (current.assertions.length === old.assertions.length && old.assertions.length > 0) {
|
|
2990
|
+
const changed = old.assertions.map((assertion, i) => ({ before: assertion, after: current.assertions[i], rawBefore: old.raw[i], rawAfter: current.raw[i] })).filter((pair) => pair.before !== pair.after);
|
|
2991
|
+
const weaker = changed.find((pair) => isWeaker(pair.rawBefore, pair.rawAfter, language));
|
|
2992
|
+
if (weaker) {
|
|
2993
|
+
result.immediate.push(finding(
|
|
2994
|
+
"assertion-weakened",
|
|
2995
|
+
newPath,
|
|
2996
|
+
current.line,
|
|
2997
|
+
`${old.name}: ${weaker.rawBefore.slice(0, 80)} \u2192 ${weaker.rawAfter.slice(0, 80)}`,
|
|
2998
|
+
"Assertion replaced by a weaker check",
|
|
2999
|
+
commit
|
|
3000
|
+
));
|
|
3001
|
+
} else if (changed.length && old.setup === current.setup && changed.every((pair) => skeleton(pair.before) === skeleton(pair.after))) {
|
|
3002
|
+
result.rewritten.push(finding(
|
|
3003
|
+
"expectation-rewritten",
|
|
3004
|
+
newPath,
|
|
3005
|
+
current.line,
|
|
3006
|
+
`${old.name}: ${changed[0].rawBefore.slice(0, 80)} \u2192 ${changed[0].rawAfter.slice(0, 80)}`,
|
|
3007
|
+
"Expected value edited while the test inputs stayed the same",
|
|
3008
|
+
commit
|
|
3009
|
+
));
|
|
3010
|
+
}
|
|
3011
|
+
}
|
|
3012
|
+
}
|
|
3013
|
+
return result;
|
|
3014
|
+
}
|
|
3015
|
+
function resolveTampering(files, productionChanged) {
|
|
3016
|
+
const added = new Set(files.flatMap((file) => file.addedHashes));
|
|
3017
|
+
return [
|
|
3018
|
+
...files.flatMap((file) => file.immediate),
|
|
3019
|
+
...files.flatMap((file) => file.removedTests).filter((removed) => !added.has(removed.bodyHash)).map((removed) => removed.finding),
|
|
3020
|
+
...productionChanged ? files.flatMap((file) => file.rewritten) : []
|
|
3021
|
+
];
|
|
3022
|
+
}
|
|
3023
|
+
var GUARD_FILE = /(?:^|\/)(?:package\.json|tsconfig[^/]*\.json|jsconfig\.json|(?:jest|vitest|vite)\.config\.[cm]?[jt]s|\.eslintrc(?:\.[a-z]+)?|eslint\.config\.[cm]?[jt]s|pyproject\.toml|setup\.cfg|pytest\.ini|tox\.ini|\.coveragerc|codecov\.ya?ml|\.gitlab-ci\.ya?ml|\.github\/workflows\/[^/]+\.ya?ml)$/i;
|
|
3024
|
+
function isGuardFile(file) {
|
|
3025
|
+
return GUARD_FILE.test(file);
|
|
3026
|
+
}
|
|
3027
|
+
function lineOf(text, index) {
|
|
3028
|
+
let line = 1;
|
|
3029
|
+
for (let i = 0; i < index && i < text.length; i++) if (text.charCodeAt(i) === 10) line++;
|
|
3030
|
+
return line;
|
|
3031
|
+
}
|
|
3032
|
+
function numbers(text, pattern) {
|
|
3033
|
+
const seen = /* @__PURE__ */ new Map();
|
|
3034
|
+
return [...text.matchAll(pattern)].map((match) => {
|
|
3035
|
+
const label = match[1] ?? "value";
|
|
3036
|
+
const count = (seen.get(label) ?? 0) + 1;
|
|
3037
|
+
seen.set(label, count);
|
|
3038
|
+
return { key: `${label}#${count}`, value: Number(match[match.length - 1]), index: match.index ?? 0 };
|
|
3039
|
+
});
|
|
3040
|
+
}
|
|
3041
|
+
var THRESHOLD_PATTERNS = [
|
|
3042
|
+
/\b(lines|branches|functions|statements)\b["']?\s*[:=]\s*(\d+(?:\.\d+)?)/g,
|
|
3043
|
+
/\b(fail_under)\s*[:=]\s*(\d+(?:\.\d+)?)/g,
|
|
3044
|
+
/(--cov-fail-under)[= ](\d+(?:\.\d+)?)/g
|
|
3045
|
+
];
|
|
3046
|
+
var SWITCHES = ["strict", "noImplicitAny", "strictNullChecks", "noImplicitReturns", "useUnknownInCatchVariables"];
|
|
3047
|
+
var TEST_COMMAND = /\brun:.*\b(?:tests?|unit|pytest|vitest|jest|mocha|ava|tap|tox|nox|e2e|spec)\b|^\s*-?\s*(?:npm|yarn|pnpm|go|cargo|mvn|gradle|make)\s+(?:run\s+)?(?:\w+\s+)?tests?\b/;
|
|
3048
|
+
function severityOf(value) {
|
|
3049
|
+
return /^(?:error|2)$/.test(value) ? 2 : /^(?:warn|1)$/.test(value) ? 1 : 0;
|
|
3050
|
+
}
|
|
3051
|
+
function analyzeGuardFile(file, oldText, newText, commit) {
|
|
3052
|
+
if (oldText === null || newText === null) return [];
|
|
3053
|
+
const out = [];
|
|
3054
|
+
const add = (index, snippet, reason) => out.push(finding("guard-weakened", file, lineOf(newText, index), snippet, reason, commit));
|
|
3055
|
+
if (/threshold|fail_under|cov-fail-under|coverage/i.test(newText)) {
|
|
3056
|
+
for (const pattern of THRESHOLD_PATTERNS) {
|
|
3057
|
+
const old = new Map(numbers(oldText, pattern).map((item) => [item.key, item.value]));
|
|
3058
|
+
for (const item of numbers(newText, pattern)) {
|
|
3059
|
+
const previous = old.get(item.key);
|
|
3060
|
+
if (previous !== void 0 && item.value < previous) {
|
|
3061
|
+
add(item.index, `${item.key.split("#")[0]}: ${previous} \u2192 ${item.value}`, "Coverage threshold lowered");
|
|
3062
|
+
}
|
|
3063
|
+
}
|
|
3064
|
+
}
|
|
3065
|
+
}
|
|
3066
|
+
if (/(?:^|\/)(?:tsconfig|jsconfig)[^/]*\.json$/i.test(file)) {
|
|
3067
|
+
for (const name of SWITCHES) {
|
|
3068
|
+
const pattern = new RegExp(`"${name}"\\s*:\\s*(true|false)`);
|
|
3069
|
+
const before = pattern.exec(oldText), after = pattern.exec(newText);
|
|
3070
|
+
if (before?.[1] === "true" && after?.[1] === "false") add(after.index, `${name}: true \u2192 false`, "Compiler strictness switched off");
|
|
3071
|
+
}
|
|
3072
|
+
}
|
|
3073
|
+
if (/eslint/i.test(file)) {
|
|
3074
|
+
const rules = (text) => new Map([...text.matchAll(/["']([@\w/.-]+)["']\s*:\s*(?:\[\s*)?["']?(error|warn|off|[012])["']?/g)].map((match) => [match[1], { level: severityOf(match[2]), index: match.index ?? 0 }]));
|
|
3075
|
+
const before = rules(oldText);
|
|
3076
|
+
for (const [name, now] of rules(newText)) {
|
|
3077
|
+
const previous = before.get(name);
|
|
3078
|
+
if (previous && previous.level === 2 && now.level < 2) add(now.index, `${name}: error \u2192 ${now.level ? "warn" : "off"}`, "Lint rule downgraded");
|
|
3079
|
+
}
|
|
3080
|
+
}
|
|
3081
|
+
if (/(?:\.github\/workflows\/|\.gitlab-ci)/.test(file)) {
|
|
3082
|
+
const count = (text, pattern) => text.split(/\r?\n/).filter((line) => !line.trim().startsWith("#") && pattern.test(line)).length;
|
|
3083
|
+
const lenient = /continue-on-error:\s*true|allow_failure:\s*true/;
|
|
3084
|
+
if (count(newText, lenient) > count(oldText, lenient)) {
|
|
3085
|
+
add(newText.search(lenient), "continue-on-error: true", "CI step is now allowed to fail");
|
|
3086
|
+
}
|
|
3087
|
+
if (count(newText, TEST_COMMAND) < count(oldText, TEST_COMMAND)) {
|
|
3088
|
+
add(0, "test command no longer runs in CI", "Test step removed from CI");
|
|
3089
|
+
}
|
|
3090
|
+
}
|
|
3091
|
+
if (/(?:^|\/)package\.json$/i.test(file)) {
|
|
3092
|
+
try {
|
|
3093
|
+
const before = JSON.parse(oldText)?.scripts?.test, after = JSON.parse(newText)?.scripts?.test;
|
|
3094
|
+
if (typeof before === "string" && typeof after === "string" && before !== after && (/^\s*(?:echo\b|true\b|exit\s+0|:)/.test(after) || /\|\|\s*(?:true|exit\s+0)\b/.test(after) || /--passWithNoTests/.test(after) && !/--passWithNoTests/.test(before))) {
|
|
3095
|
+
add(Math.max(0, newText.indexOf('"test"')), `test: ${after}`, "Test script can no longer fail");
|
|
3096
|
+
}
|
|
3097
|
+
} catch {
|
|
3098
|
+
}
|
|
3099
|
+
}
|
|
3100
|
+
return out;
|
|
3101
|
+
}
|
|
3102
|
+
|
|
3103
|
+
// src/review.ts
|
|
2741
3104
|
var GENERATED_PATHS2 = ["**/node_modules/**", "**/dist/**", "**/build/**", "**/vendor/**", "**/*.min.js", "**/*.map"];
|
|
2742
3105
|
var MAX_SCAN_BYTES2 = 50 * 1024 * 1024;
|
|
2743
3106
|
function compareFindings(before, after, newPath) {
|
|
2744
3107
|
const old = /* @__PURE__ */ new Map();
|
|
2745
|
-
for (const
|
|
2746
|
-
const key = findingFingerprint({ ...
|
|
3108
|
+
for (const finding2 of before) {
|
|
3109
|
+
const key = findingFingerprint({ ...finding2, file: newPath });
|
|
2747
3110
|
const bucket = old.get(key) || { findings: [], matched: 0 };
|
|
2748
|
-
bucket.findings.push(
|
|
3111
|
+
bucket.findings.push(finding2);
|
|
2749
3112
|
old.set(key, bucket);
|
|
2750
3113
|
}
|
|
2751
3114
|
const introduced = [];
|
|
2752
3115
|
let unchanged = 0;
|
|
2753
|
-
for (const
|
|
2754
|
-
const bucket = old.get(findingFingerprint(
|
|
3116
|
+
for (const finding2 of after) {
|
|
3117
|
+
const bucket = old.get(findingFingerprint(finding2));
|
|
2755
3118
|
if (bucket && bucket.matched < bucket.findings.length) {
|
|
2756
3119
|
bucket.matched++;
|
|
2757
3120
|
unchanged++;
|
|
2758
|
-
} else introduced.push(
|
|
3121
|
+
} else introduced.push(finding2);
|
|
2759
3122
|
}
|
|
2760
3123
|
return { introduced, removed: [...old.values()].flatMap((bucket) => bucket.findings.slice(bucket.matched)), unchanged };
|
|
2761
3124
|
}
|
|
@@ -2773,7 +3136,7 @@ async function compareSourceChanges(options) {
|
|
|
2773
3136
|
const policy = refs.baseSha === "EMPTY" ? { config: configFromText("{}", "default first-commit policy"), filename: null } : await reviewPolicy(cwd, refs.baseSha, options.signal);
|
|
2774
3137
|
const format = options.format ?? "terminal";
|
|
2775
3138
|
if (!["terminal", "json", "github", "markdown", "sarif"].includes(format)) throw new ConfigError("Unsupported review format.");
|
|
2776
|
-
const failOn = options.failOn ?? policy.config.failOn ?? ["test-skip", "trivial-assert", "test-focus"];
|
|
3139
|
+
const failOn = options.failOn ?? policy.config.failOn ?? ["test-skip", "trivial-assert", "test-focus", "assertion-weakened", "guard-weakened"];
|
|
2777
3140
|
for (const rule of failOn) if (!VALID_DETECTORS.includes(rule)) throw new ConfigError(`Unknown review rule '${rule}'.`);
|
|
2778
3141
|
const changes = staged?.changes ?? await changedTreeFiles(cwd, refs.mergeBaseSha, refs.headSha, options.signal);
|
|
2779
3142
|
const policyChanges = changes.filter((change) => CONFIG_FILENAMES.includes(change.oldPath) || CONFIG_FILENAMES.includes(change.newPath)).map((change) => change.newPath);
|
|
@@ -2805,13 +3168,39 @@ async function compareSourceChanges(options) {
|
|
|
2805
3168
|
output: ""
|
|
2806
3169
|
};
|
|
2807
3170
|
let totalBytes = 0;
|
|
3171
|
+
const tamperingFiles = [];
|
|
3172
|
+
const guardFindings = [];
|
|
3173
|
+
let productionChanged = false;
|
|
3174
|
+
const readText = async (mode, oid) => {
|
|
3175
|
+
if (mode === "000000") return null;
|
|
3176
|
+
if (!["100644", "100755"].includes(mode)) throw new ConfigError("Changed source is a symlink, submodule or unsupported Git mode; comparison is incomplete.");
|
|
3177
|
+
const content = await readBlob(cwd, oid, options.signal);
|
|
3178
|
+
totalBytes += content.length;
|
|
3179
|
+
if (totalBytes > MAX_SCAN_BYTES2) throw new ConfigError("Review exceeds the 50 MiB source comparison limit.");
|
|
3180
|
+
if (content.includes(0)) throw new ConfigError("Binary content in a source comparison.");
|
|
3181
|
+
return new TextDecoder("utf-8", { fatal: true }).decode(content);
|
|
3182
|
+
};
|
|
2808
3183
|
const existingCounts = /* @__PURE__ */ new Map();
|
|
2809
3184
|
for (const change of changes) {
|
|
2810
3185
|
assertNotAborted(options.signal);
|
|
2811
3186
|
const oldSupported = change.oldMode !== "000000" && isCodeFile(change.oldPath);
|
|
2812
3187
|
const newSupported = change.newMode !== "000000" && isCodeFile(change.newPath);
|
|
2813
3188
|
if (!oldSupported && !newSupported) {
|
|
2814
|
-
|
|
3189
|
+
const guarded = (isGuardFile(change.newPath) || isGuardFile(change.oldPath)) && isCodeFile(change.newPath + ".ts", ignores);
|
|
3190
|
+
if (!guarded) {
|
|
3191
|
+
result.unsupportedFiles++;
|
|
3192
|
+
continue;
|
|
3193
|
+
}
|
|
3194
|
+
try {
|
|
3195
|
+
const oldText = await readText(change.oldMode, change.oldOid);
|
|
3196
|
+
const newText = await readText(change.newMode, change.newOid);
|
|
3197
|
+
guardFindings.push(...analyzeGuardFile(change.newPath, oldText, newText, refs.headSha));
|
|
3198
|
+
result.filesInspected++;
|
|
3199
|
+
} catch (error) {
|
|
3200
|
+
assertNotAborted(options.signal);
|
|
3201
|
+
result.skippedFiles.push({ file: change.newPath, reason: error instanceof Error ? error.message : String(error) });
|
|
3202
|
+
if (totalBytes > MAX_SCAN_BYTES2) break;
|
|
3203
|
+
}
|
|
2815
3204
|
continue;
|
|
2816
3205
|
}
|
|
2817
3206
|
const oldInScope = oldSupported && isCodeFile(change.oldPath, ignores);
|
|
@@ -2822,14 +3211,8 @@ async function compareSourceChanges(options) {
|
|
|
2822
3211
|
}
|
|
2823
3212
|
try {
|
|
2824
3213
|
let inspectedLines = 0;
|
|
2825
|
-
const scan =
|
|
2826
|
-
if (
|
|
2827
|
-
if (!["100644", "100755"].includes(mode)) throw new ConfigError("Changed source is a symlink, submodule or unsupported Git mode; comparison is incomplete.");
|
|
2828
|
-
const content = await readBlob(cwd, oid, options.signal);
|
|
2829
|
-
totalBytes += content.length;
|
|
2830
|
-
if (totalBytes > MAX_SCAN_BYTES2) throw new ConfigError("Review exceeds the 50 MiB source comparison limit.");
|
|
2831
|
-
if (content.includes(0)) throw new ConfigError("Binary content in a source comparison.");
|
|
2832
|
-
const text = new TextDecoder("utf-8", { fatal: true }).decode(content);
|
|
3214
|
+
const scan = (text, filename, commit) => {
|
|
3215
|
+
if (text === null) return [];
|
|
2833
3216
|
const lines = text.split(/\r?\n/);
|
|
2834
3217
|
if (lines.at(-1) === "") lines.pop();
|
|
2835
3218
|
inspectedLines += lines.length;
|
|
@@ -2838,13 +3221,18 @@ async function compareSourceChanges(options) {
|
|
|
2838
3221
|
if (oldSupported && !newSupported && change.newMode !== "000000") {
|
|
2839
3222
|
throw new ConfigError("Source moved to an unsupported extension; cannot claim its findings were resolved.");
|
|
2840
3223
|
}
|
|
2841
|
-
const
|
|
2842
|
-
const
|
|
3224
|
+
const oldText = await readText(change.oldMode, change.oldOid);
|
|
3225
|
+
const newText = await readText(change.newMode, change.newOid);
|
|
3226
|
+
const before = scan(oldText, change.oldPath, refs.mergeBaseSha);
|
|
3227
|
+
const after = scan(newText, change.newPath, refs.headSha);
|
|
3228
|
+
if (isTestFile(change.oldPath) || isTestFile(change.newPath)) tamperingFiles.push(analyzeTestFile(change.oldPath, change.newPath, oldText, newText, refs.headSha));
|
|
3229
|
+
else if (change.oldOid !== change.newOid) productionChanged = true;
|
|
3230
|
+
if ((isGuardFile(change.newPath) || isGuardFile(change.oldPath)) && oldText !== null && newText !== null) guardFindings.push(...analyzeGuardFile(change.newPath, oldText, newText, refs.headSha));
|
|
2843
3231
|
const compared = compareFindings(before, after, change.newPath);
|
|
2844
3232
|
const introduced = new Set(compared.introduced);
|
|
2845
|
-
for (const
|
|
2846
|
-
if (introduced.has(
|
|
2847
|
-
const fingerprint = findingFingerprint(
|
|
3233
|
+
for (const finding2 of after) {
|
|
3234
|
+
if (introduced.has(finding2)) continue;
|
|
3235
|
+
const fingerprint = findingFingerprint(finding2);
|
|
2848
3236
|
existingCounts.set(fingerprint, (existingCounts.get(fingerprint) ?? 0) + 1);
|
|
2849
3237
|
}
|
|
2850
3238
|
result.infractions.push(...compared.introduced);
|
|
@@ -2863,11 +3251,13 @@ async function compareSourceChanges(options) {
|
|
|
2863
3251
|
assertNotAborted(options.signal);
|
|
2864
3252
|
await staged?.verify();
|
|
2865
3253
|
result.complete = result.skippedFiles.length === 0;
|
|
3254
|
+
const rules = policy.config.rules ?? {};
|
|
3255
|
+
result.infractions.push(...[...resolveTampering(tamperingFiles, productionChanged), ...guardFindings].filter((finding2) => rules[finding2.type] !== "off"));
|
|
2866
3256
|
const exceptions = applyExceptions(result.infractions, options.ignoreExceptions ? [] : policy.config.exceptions, /* @__PURE__ */ new Date(), existingCounts);
|
|
2867
3257
|
result.infractions = exceptions.findings;
|
|
2868
3258
|
result.suppressedFindings = exceptions.suppressedFindings;
|
|
2869
3259
|
result.expiredExceptionCount = exceptions.expiredExceptionCount;
|
|
2870
|
-
result.failedRules = [...new Set(result.infractions.filter((
|
|
3260
|
+
result.failedRules = [...new Set(result.infractions.filter((finding2) => failOn.includes(finding2.type)).map((finding2) => finding2.type))];
|
|
2871
3261
|
result.passed = result.complete && result.failedRules.length === 0;
|
|
2872
3262
|
result.exitCode = !result.complete ? 2 : result.passed ? 0 : 1;
|
|
2873
3263
|
if (format === "json") result.output = JSON.stringify(result, null, 2);
|
package/dist/cli.js
CHANGED
package/dist/index.d.ts
CHANGED
|
@@ -32,7 +32,7 @@ interface CommitInfo {
|
|
|
32
32
|
isAiAttributed: boolean;
|
|
33
33
|
aiSignatures: string[];
|
|
34
34
|
}
|
|
35
|
-
type DetectorType = "test-skip" | "type-escape" | "swallowed-error" | "panic-loop" | "trivial-assert" | "stub-abandonment" | "test-focus";
|
|
35
|
+
type DetectorType = "test-skip" | "type-escape" | "swallowed-error" | "panic-loop" | "trivial-assert" | "stub-abandonment" | "test-focus" | "assertion-weakened" | "test-removed" | "expectation-rewritten" | "guard-weakened";
|
|
36
36
|
declare const VALID_DETECTORS: readonly DetectorType[];
|
|
37
37
|
interface Infraction {
|
|
38
38
|
fingerprint?: string;
|
package/dist/index.js
CHANGED
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agent-roast",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"description": "Offline
|
|
3
|
+
"version": "0.9.0",
|
|
4
|
+
"description": "Offline review for AI-assisted changes: catches weakened tests, lowered quality gates, skipped tests and swallowed errors",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./dist/index.js",
|
|
7
7
|
"types": "./dist/index.d.ts",
|
|
@@ -20,16 +20,22 @@
|
|
|
20
20
|
"test": "vitest run",
|
|
21
21
|
"prepublishOnly": "npm run build",
|
|
22
22
|
"pretest": "npm run build",
|
|
23
|
-
"typecheck": "tsc --noEmit"
|
|
23
|
+
"typecheck": "tsc --noEmit",
|
|
24
|
+
"bench": "node scripts/benchmark-recall.mjs",
|
|
25
|
+
"bench:noise": "node scripts/benchmark-noise.mjs"
|
|
24
26
|
},
|
|
25
27
|
"keywords": [
|
|
26
28
|
"ai",
|
|
27
29
|
"agents",
|
|
28
|
-
"
|
|
30
|
+
"ai-agents",
|
|
29
31
|
"claude-code",
|
|
30
|
-
"
|
|
32
|
+
"cursor",
|
|
33
|
+
"codex",
|
|
34
|
+
"code-review",
|
|
35
|
+
"test-quality",
|
|
36
|
+
"ci",
|
|
37
|
+
"sarif",
|
|
31
38
|
"git",
|
|
32
|
-
"code-quality",
|
|
33
39
|
"cli"
|
|
34
40
|
],
|
|
35
41
|
"author": "Nari-Ab",
|
|
@@ -47,9 +53,7 @@
|
|
|
47
53
|
"picocolors": "^1.1.0"
|
|
48
54
|
},
|
|
49
55
|
"devDependencies": {
|
|
50
|
-
"@napi-rs/canvas": "^1.0.10",
|
|
51
56
|
"@types/node": "^20.0.0",
|
|
52
|
-
"gifenc": "^1.0.3",
|
|
53
57
|
"tsup": "^8.0.0",
|
|
54
58
|
"typescript": "^5.0.0",
|
|
55
59
|
"vitest": "^4.1.12"
|
package/schema.json
CHANGED
|
@@ -20,7 +20,11 @@
|
|
|
20
20
|
"panic-loop",
|
|
21
21
|
"trivial-assert",
|
|
22
22
|
"stub-abandonment",
|
|
23
|
-
"test-focus"
|
|
23
|
+
"test-focus",
|
|
24
|
+
"assertion-weakened",
|
|
25
|
+
"test-removed",
|
|
26
|
+
"expectation-rewritten",
|
|
27
|
+
"guard-weakened"
|
|
24
28
|
]
|
|
25
29
|
}
|
|
26
30
|
},
|
|
@@ -94,6 +98,34 @@
|
|
|
94
98
|
"warn",
|
|
95
99
|
"off"
|
|
96
100
|
]
|
|
101
|
+
},
|
|
102
|
+
"assertion-weakened": {
|
|
103
|
+
"enum": [
|
|
104
|
+
"error",
|
|
105
|
+
"warn",
|
|
106
|
+
"off"
|
|
107
|
+
]
|
|
108
|
+
},
|
|
109
|
+
"test-removed": {
|
|
110
|
+
"enum": [
|
|
111
|
+
"error",
|
|
112
|
+
"warn",
|
|
113
|
+
"off"
|
|
114
|
+
]
|
|
115
|
+
},
|
|
116
|
+
"expectation-rewritten": {
|
|
117
|
+
"enum": [
|
|
118
|
+
"error",
|
|
119
|
+
"warn",
|
|
120
|
+
"off"
|
|
121
|
+
]
|
|
122
|
+
},
|
|
123
|
+
"guard-weakened": {
|
|
124
|
+
"enum": [
|
|
125
|
+
"error",
|
|
126
|
+
"warn",
|
|
127
|
+
"off"
|
|
128
|
+
]
|
|
97
129
|
}
|
|
98
130
|
}
|
|
99
131
|
},
|