agent-roast 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +46 -6
- package/dist/{chunk-FHRPVXCI.js → chunk-SMZZAGFS.js} +369 -33
- package/dist/cli.js +49 -6
- package/dist/index.d.ts +65 -4
- package/dist/index.js +17 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -18,17 +18,23 @@
|
|
|
18
18
|
|
|
19
19
|
</div>
|
|
20
20
|
|
|
21
|
-
`agent-roast` scans recorded git diffs for shortcuts taken by AI coding assistants (Claude Code, Cursor, Aider). It checks for skipped tests, compiler type escapes (`as any`), swallowed errors, and panic revert loops, then outputs a discipline score and line proofs. 100% offline, zero API keys, zero telemetry.
|
|
21
|
+
`agent-roast` scans recorded git diffs for shortcuts taken by AI coding assistants (Claude Code, Cursor, Codex, Antigravity, Aider). It checks for skipped tests, fake tautological assertions, abandoned stubs, compiler type escapes (`as any`), swallowed errors, and panic revert loops, then outputs a discipline score and line proofs. 100% offline, zero API keys, zero telemetry.
|
|
22
22
|
|
|
23
23
|
```bash
|
|
24
|
+
# Initialize in your repository (creates .agentroastrc.json)
|
|
25
|
+
npx agent-roast init
|
|
26
|
+
npx agent-roast init --ci # generate .github/workflows/agent-roast.yml
|
|
27
|
+
npx agent-roast init --hook # install .git/hooks/pre-commit
|
|
28
|
+
|
|
24
29
|
# Audit repository history
|
|
25
30
|
npx agent-roast # scan AI-attributed commits (last 90 days)
|
|
26
31
|
npx agent-roast --all # scan entire repository history
|
|
27
32
|
npx agent-roast --verbose # show commit SHAs and line proofs
|
|
33
|
+
npx agent-roast --fail-under 70 # exit code 1 if score is below 70
|
|
28
34
|
|
|
29
35
|
# Audit pull request diffs (CI gate)
|
|
30
36
|
npx agent-roast --base origin/main --head HEAD
|
|
31
|
-
npx agent-roast --base origin/main --head HEAD --fail-on test-skip,
|
|
37
|
+
npx agent-roast --base origin/main --head HEAD --fail-on test-skip,trivial-assert,stub-abandonment --format github
|
|
32
38
|
```
|
|
33
39
|
|
|
34
40
|
What a report looks like (a real repository audit, trimmed):
|
|
@@ -50,6 +56,12 @@ INFRACTIONS DETECTED (42)
|
|
|
50
56
|
● test-skip (6) — disabled or skipped test suites
|
|
51
57
|
e.g. tests/test_outbound.py:18 — pytest.skip(...)
|
|
52
58
|
|
|
59
|
+
● trivial-assert (2) — tautological assertions faking test coverage
|
|
60
|
+
e.g. tests/auth.test.ts:31 — expect(true).toBe(true)
|
|
61
|
+
|
|
62
|
+
● stub-abandonment (1) — incomplete stubs left behind
|
|
63
|
+
e.g. src/payment.ts:18 — throw new Error("Not implemented")
|
|
64
|
+
|
|
53
65
|
● swallowed-error (2) — empty catch blocks or except: pass
|
|
54
66
|
```
|
|
55
67
|
|
|
@@ -59,12 +71,38 @@ INFRACTIONS DETECTED (42)
|
|
|
59
71
|
|
|
60
72
|
---
|
|
61
73
|
|
|
74
|
+
### Configuration (`.agentroastrc.json`)
|
|
75
|
+
|
|
76
|
+
Configure custom ignore paths, rule severities, and CI thresholds with a `.agentroastrc.json` file in your repository root:
|
|
77
|
+
|
|
78
|
+
```json
|
|
79
|
+
{
|
|
80
|
+
"$schema": "https://raw.githubusercontent.com/sidzuya/agent-roast/main/schema.json",
|
|
81
|
+
"ignorePaths": [
|
|
82
|
+
"**/*.min.js",
|
|
83
|
+
"**/vendor/**",
|
|
84
|
+
"**/dist/**",
|
|
85
|
+
"**/fixtures/**"
|
|
86
|
+
],
|
|
87
|
+
"rules": {
|
|
88
|
+
"hallucinated-package": "error",
|
|
89
|
+
"empty-catch": "warn",
|
|
90
|
+
"trivial-assert": "warn",
|
|
91
|
+
"stub-abandonment": "warn",
|
|
92
|
+
"mock-leak": "warn"
|
|
93
|
+
},
|
|
94
|
+
"scoreThreshold": 70
|
|
95
|
+
}
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
---
|
|
99
|
+
|
|
62
100
|
### GitHub Actions Integration
|
|
63
101
|
|
|
64
|
-
Add `agent-roast` as a gate in your pull request workflow (`.github/workflows/
|
|
102
|
+
Add `agent-roast` as a gate in your pull request workflow (`.github/workflows/agent-roast.yml`):
|
|
65
103
|
|
|
66
104
|
```yaml
|
|
67
|
-
name: Agent
|
|
105
|
+
name: Agent Roast
|
|
68
106
|
on:
|
|
69
107
|
pull_request:
|
|
70
108
|
branches: [main]
|
|
@@ -83,7 +121,7 @@ jobs:
|
|
|
83
121
|
- name: Run agent-roast audit
|
|
84
122
|
uses: Nari-Ab/agent-roast@v1
|
|
85
123
|
with:
|
|
86
|
-
fail-on: test-skip
|
|
124
|
+
fail-on: test-skip,trivial-assert,stub-abandonment
|
|
87
125
|
format: github
|
|
88
126
|
```
|
|
89
127
|
|
|
@@ -96,7 +134,9 @@ jobs:
|
|
|
96
134
|
Only **added lines** (`+`) in source code files are scanned — deleted lines, file moves, and comments never trigger false alarms.
|
|
97
135
|
|
|
98
136
|
- **Skipped tests (`test-skip`):** `it.skip`, `describe.skip`, `xit`, `pytest.skip`, `test.todo` added to silence failing tests. (Default blocking gate).
|
|
99
|
-
- **
|
|
137
|
+
- **Trivial assertions (`trivial-assert`):** Tautological or dummy assertions added to fabricate test green checks (`expect(true).toBe(true)`, `expect(1).toBe(1)`, `assert True`, `self.assertTrue(True)`).
|
|
138
|
+
- **Stub abandonment (`stub-abandonment`):** Dummy stubs left unimplemented in committed code (`throw new Error("Not implemented")`, `raise NotImplementedError`, Rust `todo!()` / `unimplemented!()`).
|
|
139
|
+
- **Type escapes (`type-escape`):** `as any`, `as unknown as`, `as never`, `// @ts-ignore`, `// @ts-nocheck`, `// @ts-expect-error`, `# type: ignore`, `# noqa`, `eslint-disable`.
|
|
100
140
|
- **Swallowed errors (`swallowed-error`):** Empty `catch {}` or `except: pass` blocks hiding runtime failures.
|
|
101
141
|
- **Panic fix loops (`panic-loop`):** Heuristic chain analysis of rapid-fire fix/revert commits by the same author within 15 minutes touching overlapping files. *(Observational metric for repository audits; not a PR blocker)*.
|
|
102
142
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
|
|
3
3
|
// src/index.ts
|
|
4
|
-
import
|
|
4
|
+
import path3 from "path";
|
|
5
5
|
|
|
6
6
|
// src/git.ts
|
|
7
7
|
import { spawn, spawnSync } from "child_process";
|
|
@@ -16,14 +16,21 @@ var TRAILER_AI_PATTERNS = [
|
|
|
16
16
|
{ regex: /Co-Authored-By:.*(?:Aider|aider)/i, name: "trailer:aider" },
|
|
17
17
|
{ regex: /Co-Authored-By:.*(?:Devin|cognition)/i, name: "trailer:devin" },
|
|
18
18
|
{ regex: /Co-Authored-By:.*(?:Windsurf|Codeium)/i, name: "trailer:windsurf" },
|
|
19
|
-
{ regex: /Co-Authored-By:.*(?:ChatGPT|OpenAI|GPT)/i, name: "trailer:gpt" }
|
|
19
|
+
{ regex: /Co-Authored-By:.*(?:ChatGPT|OpenAI|GPT)/i, name: "trailer:gpt" },
|
|
20
|
+
{ regex: /Co-Authored-By:.*(?:Antigravity|Deepmind|Google)/i, name: "trailer:antigravity" },
|
|
21
|
+
{ regex: /Co-Authored-By:.*(?:Codex)/i, name: "trailer:codex" },
|
|
22
|
+
{ regex: /Co-Authored-By:.*(?:Gemini)/i, name: "trailer:gemini" },
|
|
23
|
+
{ regex: /Co-Authored-By:.*(?:OpenClaw|Claw)/i, name: "trailer:openclaw" }
|
|
20
24
|
];
|
|
21
25
|
var MESSAGE_AI_PATTERNS = [
|
|
22
26
|
{ regex: /(?:Generated|Assisted|Created|Written) by Cursor/i, name: "message:cursor" },
|
|
23
27
|
{ regex: /(?:Generated|Assisted|Created|Written) by Claude/i, name: "message:claude" },
|
|
24
28
|
{ regex: /(?:Generated|Assisted|Created|Written) by Aider/i, name: "message:aider" },
|
|
25
29
|
{ regex: /(?:Generated|Assisted|Created|Written) by Copilot/i, name: "message:copilot" },
|
|
26
|
-
{ regex: /(?:Generated|Assisted|Created|Written) by Devin/i, name: "message:devin" }
|
|
30
|
+
{ regex: /(?:Generated|Assisted|Created|Written) by Devin/i, name: "message:devin" },
|
|
31
|
+
{ regex: /(?:Generated|Assisted|Created|Written) by (?:Antigravity|AGY)/i, name: "message:antigravity" },
|
|
32
|
+
{ regex: /(?:Generated|Assisted|Created|Written) by Codex/i, name: "message:codex" },
|
|
33
|
+
{ regex: /(?:Generated|Assisted|Created|Written) by Gemini/i, name: "message:gemini" }
|
|
27
34
|
];
|
|
28
35
|
var AUTHOR_AI_PATTERNS = [
|
|
29
36
|
{ regex: /(?:claude|anthropic)/i, name: "author:claude" },
|
|
@@ -31,6 +38,9 @@ var AUTHOR_AI_PATTERNS = [
|
|
|
31
38
|
{ regex: /(?:copilot)/i, name: "author:copilot" },
|
|
32
39
|
{ regex: /(?:aider)/i, name: "author:aider" },
|
|
33
40
|
{ regex: /(?:devin)/i, name: "author:devin" },
|
|
41
|
+
{ regex: /(?:antigravity|deepmind)/i, name: "author:antigravity" },
|
|
42
|
+
{ regex: /(?:codex)/i, name: "author:codex" },
|
|
43
|
+
{ regex: /(?:gemini)/i, name: "author:gemini" },
|
|
34
44
|
{ regex: /(?:\[bot\]|bot@|automation)/i, name: "author:bot" }
|
|
35
45
|
];
|
|
36
46
|
function detectAiAttribution(authorName, authorEmail, message) {
|
|
@@ -126,12 +136,12 @@ async function parseGitLogStream(stream, onCommit) {
|
|
|
126
136
|
{ skippedLongLines, skippedUnknownFiles }
|
|
127
137
|
);
|
|
128
138
|
};
|
|
129
|
-
return new Promise((
|
|
139
|
+
return new Promise((resolve2, reject) => {
|
|
130
140
|
let settled = false;
|
|
131
141
|
const safeResolve = (val) => {
|
|
132
142
|
if (!settled) {
|
|
133
143
|
settled = true;
|
|
134
|
-
|
|
144
|
+
resolve2(val);
|
|
135
145
|
}
|
|
136
146
|
};
|
|
137
147
|
const safeReject = (err) => {
|
|
@@ -532,6 +542,90 @@ var GitResolutionError = class extends Error {
|
|
|
532
542
|
}
|
|
533
543
|
};
|
|
534
544
|
|
|
545
|
+
// src/config.ts
|
|
546
|
+
import * as fs from "fs";
|
|
547
|
+
import * as path from "path";
|
|
548
|
+
var DEFAULT_CONFIG = {
|
|
549
|
+
failOn: ["test-skip", "trivial-assert"],
|
|
550
|
+
ignorePaths: [
|
|
551
|
+
"**/fixtures/**",
|
|
552
|
+
"**/__fixtures__/**",
|
|
553
|
+
"**/testdata/**",
|
|
554
|
+
"**/mocks/**",
|
|
555
|
+
"**/test_data/**",
|
|
556
|
+
"**/__snapshots__/**",
|
|
557
|
+
"**/sample/**",
|
|
558
|
+
"**/samples/**"
|
|
559
|
+
],
|
|
560
|
+
scoreThreshold: 70,
|
|
561
|
+
rules: {
|
|
562
|
+
"test-skip": "error",
|
|
563
|
+
"trivial-assert": "error",
|
|
564
|
+
"swallowed-error": "warn",
|
|
565
|
+
"stub-abandonment": "warn",
|
|
566
|
+
"type-escape": "warn"
|
|
567
|
+
}
|
|
568
|
+
};
|
|
569
|
+
var CONFIG_FILENAMES = [
|
|
570
|
+
".agentroastrc.json",
|
|
571
|
+
".agentroastrc",
|
|
572
|
+
"agent-roast.config.json"
|
|
573
|
+
];
|
|
574
|
+
function loadConfig(cwd = process.cwd()) {
|
|
575
|
+
for (const filename of CONFIG_FILENAMES) {
|
|
576
|
+
const filePath = path.join(cwd, filename);
|
|
577
|
+
if (fs.existsSync(filePath)) {
|
|
578
|
+
try {
|
|
579
|
+
const raw = fs.readFileSync(filePath, "utf8");
|
|
580
|
+
const parsed = JSON.parse(raw);
|
|
581
|
+
if (typeof parsed === "object" && parsed !== null) {
|
|
582
|
+
return {
|
|
583
|
+
...DEFAULT_CONFIG,
|
|
584
|
+
...parsed,
|
|
585
|
+
failOn: Array.isArray(parsed.failOn) ? parsed.failOn : DEFAULT_CONFIG.failOn,
|
|
586
|
+
ignorePaths: Array.isArray(parsed.ignorePaths) ? Array.from(/* @__PURE__ */ new Set([...DEFAULT_CONFIG.ignorePaths || [], ...parsed.ignorePaths])) : DEFAULT_CONFIG.ignorePaths,
|
|
587
|
+
rules: {
|
|
588
|
+
...DEFAULT_CONFIG.rules,
|
|
589
|
+
...parsed.rules || {}
|
|
590
|
+
}
|
|
591
|
+
};
|
|
592
|
+
}
|
|
593
|
+
} catch (err) {
|
|
594
|
+
console.warn(`Warning: Failed to parse ${filename}: ${err.message}. Using default configuration.`);
|
|
595
|
+
}
|
|
596
|
+
}
|
|
597
|
+
}
|
|
598
|
+
return { ...DEFAULT_CONFIG };
|
|
599
|
+
}
|
|
600
|
+
function matchIgnorePath(filePath, ignorePatterns) {
|
|
601
|
+
if (!ignorePatterns || ignorePatterns.length === 0) {
|
|
602
|
+
return false;
|
|
603
|
+
}
|
|
604
|
+
const normalized = filePath.replace(/\\/g, "/");
|
|
605
|
+
for (const pattern of ignorePatterns) {
|
|
606
|
+
if (!pattern) continue;
|
|
607
|
+
const cleanPattern = pattern.replace(/\\/g, "/");
|
|
608
|
+
let regexPattern = cleanPattern.replace(/[.+^${}()|[\]]/g, "\\$&").replace(/\*\*\//g, "(?:^|.*?/)").replace(/\/\*\*/g, "(?:/.*)?").replace(/\*/g, "[^/]*");
|
|
609
|
+
if (!regexPattern.startsWith("^") && !regexPattern.startsWith("(?:^")) {
|
|
610
|
+
regexPattern = "(?:^|/)" + regexPattern;
|
|
611
|
+
}
|
|
612
|
+
if (!regexPattern.endsWith("$")) {
|
|
613
|
+
regexPattern = regexPattern + "(?:$|/)";
|
|
614
|
+
}
|
|
615
|
+
try {
|
|
616
|
+
const rx = new RegExp(regexPattern, "i");
|
|
617
|
+
if (rx.test(normalized)) {
|
|
618
|
+
return true;
|
|
619
|
+
}
|
|
620
|
+
} catch {
|
|
621
|
+
if (normalized.includes(cleanPattern.replace(/\*/g, ""))) {
|
|
622
|
+
return true;
|
|
623
|
+
}
|
|
624
|
+
}
|
|
625
|
+
}
|
|
626
|
+
return false;
|
|
627
|
+
}
|
|
628
|
+
|
|
535
629
|
// src/metrics.ts
|
|
536
630
|
var TEST_SKIP_PATTERNS = [
|
|
537
631
|
{ regex: /(?:it|test|describe)\.skip\b/, name: "skipped test (it.skip / describe.skip)" },
|
|
@@ -543,16 +637,35 @@ var TEST_SKIP_PATTERNS = [
|
|
|
543
637
|
var COMMENT_DIRECTIVE_PATTERNS = [
|
|
544
638
|
{ regex: /^\s*\/\/\s*@ts-ignore\b/, name: "compiler ignore (@ts-ignore)" },
|
|
545
639
|
{ regex: /^\s*\/\/\s*@ts-expect-error\b/, name: "compiler suppression (@ts-expect-error)" },
|
|
640
|
+
{ regex: /^\s*\/\/\s*@ts-nocheck\b/, name: "compiler check disable (@ts-nocheck)" },
|
|
546
641
|
{ regex: /(?:^\s*#\s*type:\s*ignore\b|#\s*type:\s*ignore\s*$)/, name: "python type ignore" },
|
|
642
|
+
{ regex: /(?:^\s*#\s*noqa\b|#\s*noqa\s*$)/, name: "python linter suppression (# noqa)" },
|
|
547
643
|
{ regex: /^\s*\/\/\s*eslint-disable(?:-next-line)?\b/, name: "linter suppression (eslint-disable)" }
|
|
548
644
|
];
|
|
549
645
|
var CODE_TYPE_ESCAPE_PATTERNS = [
|
|
550
|
-
{ regex: /\bas\s+any\b/, name: "type bypass (as any)" }
|
|
646
|
+
{ regex: /\bas\s+any\b/, name: "type bypass (as any)" },
|
|
647
|
+
{ regex: /\bas\s+unknown\s+as\b/, name: "double assertion bypass (as unknown as)" },
|
|
648
|
+
{ regex: /\bas\s+never\b/, name: "type escape (as never)" }
|
|
551
649
|
];
|
|
552
650
|
var SWALLOWED_ERROR_PATTERNS = [
|
|
553
651
|
{ regex: /catch\s*(?:\([^)]*\))?\s*\{\s*\}/, name: "empty catch block (catch {})" },
|
|
554
652
|
{ regex: /except(?:\s+[^:]*)?:\s*pass\b/, name: "silent python error swallow (except: pass)" }
|
|
555
653
|
];
|
|
654
|
+
var TRIVIAL_ASSERT_PATTERNS = [
|
|
655
|
+
{ regex: /expect\s*\(\s*(?:true|false|1|0|null|undefined)\s*\)\.toBe\s*\(\s*(?:true|false|1|0|null|undefined)\s*\)/, name: "trivial assertion (expect(val).toBe(val))" },
|
|
656
|
+
{ regex: /expect\s*\(\s*(?:true|false|1|0|null|undefined)\s*\)\.toEqual\s*\(\s*(?:true|false|1|0|null|undefined)\s*\)/, name: "trivial assertion (expect(val).toEqual(val))" },
|
|
657
|
+
{ regex: /\bassert\s+(?:True|False|1\s*==\s*1)\b/, name: "trivial python assert (assert True / 1 == 1)" },
|
|
658
|
+
{ regex: /self\.assertTrue\s*\(\s*(?:True|1)\s*\)/, name: "trivial assert (self.assertTrue(True))" },
|
|
659
|
+
{ regex: /self\.assertEqual\s*\(\s*(?:1\s*,\s*1|True\s*,\s*True)\s*\)/, name: "trivial assert (self.assertEqual(1, 1))" },
|
|
660
|
+
{ regex: /assert\.(?:strictEqual|equal)\s*\(\s*(?:true\s*,\s*true|1\s*,\s*1)\s*\)/, name: "trivial assert (assert.equal(1, 1))" }
|
|
661
|
+
];
|
|
662
|
+
var STUB_ABANDONMENT_PATTERNS = [
|
|
663
|
+
{ regex: /\bthrow\s+new\s+Error\s*\(\s*["'](?:TODO|Not implemented|to be implemented|stub|implement me)/i, name: "unimplemented stub (throw new Error('Not implemented'))" },
|
|
664
|
+
{ regex: /\braise\s+NotImplementedError\b/, name: "unimplemented stub (raise NotImplementedError)" },
|
|
665
|
+
{ regex: /\braise\s+NotImplemented\b/, name: "unimplemented stub (raise NotImplemented)" },
|
|
666
|
+
{ regex: /\btodo!\s*\(\s*\)/, name: "unimplemented rust stub (todo!())" },
|
|
667
|
+
{ regex: /\bunimplemented!\s*\(\s*\)/, name: "unimplemented rust stub (unimplemented!())" }
|
|
668
|
+
];
|
|
556
669
|
var CODE_EXTENSIONS = /* @__PURE__ */ new Set([
|
|
557
670
|
".ts",
|
|
558
671
|
".tsx",
|
|
@@ -574,21 +687,28 @@ var CODE_EXTENSIONS = /* @__PURE__ */ new Set([
|
|
|
574
687
|
".c"
|
|
575
688
|
]);
|
|
576
689
|
var IGNORED_PATH_REGEX = /(?:^|\/)(?:fixtures|__fixtures__|testdata|mocks|test_data|__snapshots__|sample[s]?)\//i;
|
|
577
|
-
|
|
690
|
+
var TEST_PATH_REGEX = /(?:^|\/)(?:tests?|__tests__|specs?)\/|[._-](?:test|spec)\.[a-z0-9]+$/i;
|
|
691
|
+
function isCodeFile(filePath, customIgnorePaths) {
|
|
578
692
|
if (IGNORED_PATH_REGEX.test(filePath)) {
|
|
579
693
|
return false;
|
|
580
694
|
}
|
|
695
|
+
if (customIgnorePaths && matchIgnorePath(filePath, customIgnorePaths)) {
|
|
696
|
+
return false;
|
|
697
|
+
}
|
|
581
698
|
const ext = filePath.slice(filePath.lastIndexOf(".")).toLowerCase();
|
|
582
699
|
return CODE_EXTENSIONS.has(ext);
|
|
583
700
|
}
|
|
584
|
-
function
|
|
701
|
+
function isTestFile(filePath) {
|
|
702
|
+
return TEST_PATH_REGEX.test(filePath);
|
|
703
|
+
}
|
|
704
|
+
function auditFileDiffs(files, commitHash = "HEAD", commitDate = (/* @__PURE__ */ new Date()).toISOString(), options = {}) {
|
|
585
705
|
const infractions = [];
|
|
586
706
|
let hasSupportedChanges = false;
|
|
587
707
|
let filesInspected = 0;
|
|
588
708
|
let linesAdded = 0;
|
|
589
709
|
for (const file of files) {
|
|
590
710
|
if (file.deleted || file.path === "unknown") continue;
|
|
591
|
-
if (!isCodeFile(file.path)) continue;
|
|
711
|
+
if (!isCodeFile(file.path, options.ignorePaths)) continue;
|
|
592
712
|
hasSupportedChanges = true;
|
|
593
713
|
filesInspected++;
|
|
594
714
|
linesAdded += file.addedLines.length;
|
|
@@ -661,6 +781,38 @@ function auditFileDiffs(files, commitHash = "HEAD", commitDate = (/* @__PURE__ *
|
|
|
661
781
|
}
|
|
662
782
|
}
|
|
663
783
|
}
|
|
784
|
+
if (!isComment && isTestFile(file.path)) {
|
|
785
|
+
for (const rule of TRIVIAL_ASSERT_PATTERNS) {
|
|
786
|
+
if (rule.regex.test(cleanedLine)) {
|
|
787
|
+
infractions.push({
|
|
788
|
+
type: "trivial-assert",
|
|
789
|
+
file: file.path,
|
|
790
|
+
lineNumber: added.lineNumber,
|
|
791
|
+
lineSnippet: rawLine.trim(),
|
|
792
|
+
commitHash,
|
|
793
|
+
commitDate,
|
|
794
|
+
reason: rule.name
|
|
795
|
+
});
|
|
796
|
+
break;
|
|
797
|
+
}
|
|
798
|
+
}
|
|
799
|
+
}
|
|
800
|
+
if (!isComment) {
|
|
801
|
+
for (const rule of STUB_ABANDONMENT_PATTERNS) {
|
|
802
|
+
if (rule.regex.test(rawLine)) {
|
|
803
|
+
infractions.push({
|
|
804
|
+
type: "stub-abandonment",
|
|
805
|
+
file: file.path,
|
|
806
|
+
lineNumber: added.lineNumber,
|
|
807
|
+
lineSnippet: rawLine.trim(),
|
|
808
|
+
commitHash,
|
|
809
|
+
commitDate,
|
|
810
|
+
reason: rule.name
|
|
811
|
+
});
|
|
812
|
+
break;
|
|
813
|
+
}
|
|
814
|
+
}
|
|
815
|
+
}
|
|
664
816
|
}
|
|
665
817
|
}
|
|
666
818
|
return { infractions, hasSupportedChanges, filesInspected, linesAdded };
|
|
@@ -683,12 +835,16 @@ var MetricsCollector = class {
|
|
|
683
835
|
typeEscapes = [];
|
|
684
836
|
swallowedErrors = [];
|
|
685
837
|
panicLoops = [];
|
|
838
|
+
trivialAsserts = [];
|
|
839
|
+
stubAbandonments = [];
|
|
686
840
|
skippedLongLines = 0;
|
|
687
841
|
skippedUnknownFiles = 0;
|
|
688
842
|
activePanicChain = null;
|
|
689
843
|
includeAll;
|
|
844
|
+
ignorePaths;
|
|
690
845
|
constructor(options = {}) {
|
|
691
846
|
this.includeAll = options.all ?? false;
|
|
847
|
+
this.ignorePaths = options.ignorePaths;
|
|
692
848
|
}
|
|
693
849
|
recordSkipped(stats) {
|
|
694
850
|
if (stats.skippedLongLines) this.skippedLongLines += stats.skippedLongLines;
|
|
@@ -774,8 +930,10 @@ var MetricsCollector = class {
|
|
|
774
930
|
let commitSkips = 0;
|
|
775
931
|
let commitEscapes = 0;
|
|
776
932
|
let commitSwallowed = 0;
|
|
933
|
+
let commitTrivial = 0;
|
|
934
|
+
let commitStubs = 0;
|
|
777
935
|
for (const file of commit.files) {
|
|
778
|
-
if (!isCodeFile(file.path)) {
|
|
936
|
+
if (!isCodeFile(file.path, this.ignorePaths)) {
|
|
779
937
|
continue;
|
|
780
938
|
}
|
|
781
939
|
for (const added of file.addedLines) {
|
|
@@ -851,6 +1009,40 @@ var MetricsCollector = class {
|
|
|
851
1009
|
}
|
|
852
1010
|
}
|
|
853
1011
|
}
|
|
1012
|
+
if (!isComment && isTestFile(file.path) && commitTrivial < MAX_INFRACTIONS_PER_COMMIT) {
|
|
1013
|
+
for (const rule of TRIVIAL_ASSERT_PATTERNS) {
|
|
1014
|
+
if (rule.regex.test(cleanedLine)) {
|
|
1015
|
+
this.trivialAsserts.push({
|
|
1016
|
+
type: "trivial-assert",
|
|
1017
|
+
file: file.path,
|
|
1018
|
+
lineNumber: added.lineNumber,
|
|
1019
|
+
lineSnippet: rawLine.trim(),
|
|
1020
|
+
commitHash: commit.hash,
|
|
1021
|
+
commitDate: commit.date,
|
|
1022
|
+
reason: rule.name
|
|
1023
|
+
});
|
|
1024
|
+
commitTrivial++;
|
|
1025
|
+
break;
|
|
1026
|
+
}
|
|
1027
|
+
}
|
|
1028
|
+
}
|
|
1029
|
+
if (!isComment && commitStubs < MAX_INFRACTIONS_PER_COMMIT) {
|
|
1030
|
+
for (const rule of STUB_ABANDONMENT_PATTERNS) {
|
|
1031
|
+
if (rule.regex.test(rawLine)) {
|
|
1032
|
+
this.stubAbandonments.push({
|
|
1033
|
+
type: "stub-abandonment",
|
|
1034
|
+
file: file.path,
|
|
1035
|
+
lineNumber: added.lineNumber,
|
|
1036
|
+
lineSnippet: rawLine.trim(),
|
|
1037
|
+
commitHash: commit.hash,
|
|
1038
|
+
commitDate: commit.date,
|
|
1039
|
+
reason: rule.name
|
|
1040
|
+
});
|
|
1041
|
+
commitStubs++;
|
|
1042
|
+
break;
|
|
1043
|
+
}
|
|
1044
|
+
}
|
|
1045
|
+
}
|
|
854
1046
|
}
|
|
855
1047
|
}
|
|
856
1048
|
}
|
|
@@ -858,7 +1050,7 @@ var MetricsCollector = class {
|
|
|
858
1050
|
this.flushPanicChain();
|
|
859
1051
|
const linesSample = this.includeAll ? this.totalLinesAdded : this.aiLinesAdded;
|
|
860
1052
|
const isSufficientData = linesSample >= MIN_SAMPLE_LOC;
|
|
861
|
-
const codeInfractionPoints = this.testSkips.length * 15 + this.swallowedErrors.length * 10 + this.typeEscapes.length * 3;
|
|
1053
|
+
const codeInfractionPoints = this.testSkips.length * 15 + this.trivialAsserts.length * 15 + this.swallowedErrors.length * 10 + this.stubAbandonments.length * 8 + this.typeEscapes.length * 3;
|
|
862
1054
|
let panicInfractionPoints = 0;
|
|
863
1055
|
for (const loop of this.panicLoops) {
|
|
864
1056
|
const countMatch = loop.reason.match(/chain of (\d+) quick fixes/);
|
|
@@ -887,16 +1079,24 @@ var MetricsCollector = class {
|
|
|
887
1079
|
}
|
|
888
1080
|
} else {
|
|
889
1081
|
const testSkipPoints = this.testSkips.length * 15;
|
|
1082
|
+
const trivialPoints = this.trivialAsserts.length * 15;
|
|
890
1083
|
const panicPoints = this.panicLoops.length * 8;
|
|
1084
|
+
const stubPoints = this.stubAbandonments.length * 8;
|
|
891
1085
|
const escapePoints = this.typeEscapes.length * 3;
|
|
892
1086
|
const swallowPoints = this.swallowedErrors.length * 10;
|
|
893
|
-
const maxPoints = Math.max(testSkipPoints, panicPoints, escapePoints, swallowPoints);
|
|
1087
|
+
const maxPoints = Math.max(testSkipPoints, trivialPoints, panicPoints, stubPoints, escapePoints, swallowPoints);
|
|
894
1088
|
if (maxPoints === panicPoints && this.panicLoops.length >= 2) {
|
|
895
1089
|
archetype = "The Panic Looper";
|
|
896
1090
|
archetypeDescription = "Prone to rapid-fire fix and revert thrashing cycles when wrestling stubborn bugs.";
|
|
897
1091
|
} else if (maxPoints === testSkipPoints && this.testSkips.length >= 3) {
|
|
898
1092
|
archetype = "The Silent Vandal";
|
|
899
1093
|
archetypeDescription = "When test suites push back, tends to disable or skip tests to keep pipelines green.";
|
|
1094
|
+
} else if (maxPoints === trivialPoints && this.trivialAsserts.length >= 2) {
|
|
1095
|
+
archetype = "The Fake Asserter";
|
|
1096
|
+
archetypeDescription = "Masks broken tests by replacing real assertions with trivial expect(true).toBe(true) or assert True.";
|
|
1097
|
+
} else if (maxPoints === stubPoints && this.stubAbandonments.length >= 2) {
|
|
1098
|
+
archetype = "The Stub Abandoner";
|
|
1099
|
+
archetypeDescription = "Rushes through implementations by leaving unimplemented placeholders and TODO throws.";
|
|
900
1100
|
} else if (maxPoints === escapePoints && this.typeEscapes.length >= 5) {
|
|
901
1101
|
archetype = "The Any Architect";
|
|
902
1102
|
archetypeDescription = "Leans on compiler bypasses (`as any`, `@ts-ignore`) rather than strict type modeling.";
|
|
@@ -922,6 +1122,8 @@ var MetricsCollector = class {
|
|
|
922
1122
|
typeEscapes: this.typeEscapes,
|
|
923
1123
|
swallowedErrors: this.swallowedErrors,
|
|
924
1124
|
panicLoops: this.panicLoops,
|
|
1125
|
+
trivialAsserts: this.trivialAsserts,
|
|
1126
|
+
stubAbandonments: this.stubAbandonments,
|
|
925
1127
|
rawScore: totalInfractionPoints,
|
|
926
1128
|
score,
|
|
927
1129
|
isSufficientData,
|
|
@@ -972,26 +1174,26 @@ function formatTerminalCard(summary, options) {
|
|
|
972
1174
|
);
|
|
973
1175
|
lines.push(` ${pc.dim(summary.archetypeDescription)}`);
|
|
974
1176
|
lines.push("");
|
|
975
|
-
const totalInfractions = summary.testSkips
|
|
1177
|
+
const totalInfractions = (summary.testSkips?.length || 0) + (summary.typeEscapes?.length || 0) + (summary.swallowedErrors?.length || 0) + (summary.panicLoops?.length || 0) + (summary.trivialAsserts?.length || 0) + (summary.stubAbandonments?.length || 0);
|
|
976
1178
|
lines.push(pc.bold(` INFRACTIONS (${totalInfractions})`));
|
|
977
1179
|
lines.push(pc.dim(" \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500"));
|
|
978
|
-
if (summary.panicLoops.length > 0) {
|
|
1180
|
+
if (summary.panicLoops && summary.panicLoops.length > 0) {
|
|
979
1181
|
lines.push(
|
|
980
1182
|
` ${pc.magenta("\u25CF")} ${pc.bold("panic-loop")} (${summary.panicLoops.length}) ${pc.dim("consecutive quick fixes < 15m apart")}`
|
|
981
1183
|
);
|
|
982
1184
|
const sample = summary.panicLoops[0];
|
|
983
1185
|
lines.push(` ${pc.dim(sample.reason)}`);
|
|
984
1186
|
}
|
|
985
|
-
if (summary.
|
|
1187
|
+
if (summary.trivialAsserts && summary.trivialAsserts.length > 0) {
|
|
986
1188
|
lines.push(
|
|
987
|
-
` ${pc.
|
|
1189
|
+
` ${pc.red("\u25CF")} ${pc.bold("trivial-assert")} (${summary.trivialAsserts.length}) ${pc.dim("vacuous test assertions (expect(true).toBe(true), assert True)")}`
|
|
988
1190
|
);
|
|
989
|
-
const sample = summary.
|
|
1191
|
+
const sample = summary.trivialAsserts[0];
|
|
990
1192
|
if (sample.file) {
|
|
991
1193
|
lines.push(` ${pc.dim(`${sample.file}: ${sample.lineSnippet || ""}`)}`);
|
|
992
1194
|
}
|
|
993
1195
|
}
|
|
994
|
-
if (summary.testSkips.length > 0) {
|
|
1196
|
+
if (summary.testSkips && summary.testSkips.length > 0) {
|
|
995
1197
|
lines.push(
|
|
996
1198
|
` ${pc.red("\u25CF")} ${pc.bold("test-skip")} (${summary.testSkips.length}) ${pc.dim("skipped or disabled tests")}`
|
|
997
1199
|
);
|
|
@@ -1000,7 +1202,7 @@ function formatTerminalCard(summary, options) {
|
|
|
1000
1202
|
lines.push(` ${pc.dim(`${sample.file}: ${sample.lineSnippet || ""}`)}`);
|
|
1001
1203
|
}
|
|
1002
1204
|
}
|
|
1003
|
-
if (summary.swallowedErrors.length > 0) {
|
|
1205
|
+
if (summary.swallowedErrors && summary.swallowedErrors.length > 0) {
|
|
1004
1206
|
lines.push(
|
|
1005
1207
|
` ${pc.red("\u25CF")} ${pc.bold("swallowed-error")} (${summary.swallowedErrors.length}) ${pc.dim("empty catch blocks or except: pass")}`
|
|
1006
1208
|
);
|
|
@@ -1009,6 +1211,24 @@ function formatTerminalCard(summary, options) {
|
|
|
1009
1211
|
lines.push(` ${pc.dim(`${sample.file}: ${sample.lineSnippet || ""}`)}`);
|
|
1010
1212
|
}
|
|
1011
1213
|
}
|
|
1214
|
+
if (summary.stubAbandonments && summary.stubAbandonments.length > 0) {
|
|
1215
|
+
lines.push(
|
|
1216
|
+
` ${pc.yellow("\u25CF")} ${pc.bold("stub-abandonment")} (${summary.stubAbandonments.length}) ${pc.dim("unimplemented placeholders (throw new Error('TODO'), NotImplementedError)")}`
|
|
1217
|
+
);
|
|
1218
|
+
const sample = summary.stubAbandonments[0];
|
|
1219
|
+
if (sample.file) {
|
|
1220
|
+
lines.push(` ${pc.dim(`${sample.file}: ${sample.lineSnippet || ""}`)}`);
|
|
1221
|
+
}
|
|
1222
|
+
}
|
|
1223
|
+
if (summary.typeEscapes && summary.typeEscapes.length > 0) {
|
|
1224
|
+
lines.push(
|
|
1225
|
+
` ${pc.yellow("\u25CF")} ${pc.bold("type-escape")} (${summary.typeEscapes.length}) ${pc.dim("compiler bypasses (as any, @ts-ignore)")}`
|
|
1226
|
+
);
|
|
1227
|
+
const sample = summary.typeEscapes[0];
|
|
1228
|
+
if (sample.file) {
|
|
1229
|
+
lines.push(` ${pc.dim(`${sample.file}: ${sample.lineSnippet || ""}`)}`);
|
|
1230
|
+
}
|
|
1231
|
+
}
|
|
1012
1232
|
if (totalInfractions === 0) {
|
|
1013
1233
|
lines.push(` ${pc.green("\u25CF")} ${pc.dim("zero infractions detected across all inspected lines.")}`);
|
|
1014
1234
|
}
|
|
@@ -1026,10 +1246,12 @@ function formatTerminalCard(summary, options) {
|
|
|
1026
1246
|
lines.push("");
|
|
1027
1247
|
}
|
|
1028
1248
|
const allInfractions = [
|
|
1029
|
-
...summary.testSkips,
|
|
1030
|
-
...summary.
|
|
1031
|
-
...summary.swallowedErrors,
|
|
1032
|
-
...summary.
|
|
1249
|
+
...summary.testSkips || [],
|
|
1250
|
+
...summary.trivialAsserts || [],
|
|
1251
|
+
...summary.swallowedErrors || [],
|
|
1252
|
+
...summary.stubAbandonments || [],
|
|
1253
|
+
...summary.typeEscapes || [],
|
|
1254
|
+
...summary.panicLoops || []
|
|
1033
1255
|
];
|
|
1034
1256
|
if (allInfractions.length > 0) {
|
|
1035
1257
|
lines.push(pc.bold(" EVIDENCE (First 10 occurrences):"));
|
|
@@ -1095,7 +1317,7 @@ function formatPrTerminalCard(result) {
|
|
|
1095
1317
|
}
|
|
1096
1318
|
function sortInfractionsDeterministic(infractions) {
|
|
1097
1319
|
const getLevelRank = (type) => {
|
|
1098
|
-
return type === "test-skip" || type === "swallowed-error" ? 0 : 1;
|
|
1320
|
+
return type === "test-skip" || type === "trivial-assert" || type === "swallowed-error" ? 0 : 1;
|
|
1099
1321
|
};
|
|
1100
1322
|
return [...infractions].sort((a, b) => {
|
|
1101
1323
|
const rankDiff = getLevelRank(a.type) - getLevelRank(b.type);
|
|
@@ -1115,7 +1337,7 @@ function formatGithubAnnotations(result) {
|
|
|
1115
1337
|
let omittedCount = 0;
|
|
1116
1338
|
for (const inf of sorted) {
|
|
1117
1339
|
if (!inf.file || inf.file === "unknown") continue;
|
|
1118
|
-
const isError = result.failedRules.includes(inf.type) || inf.type === "test-skip" || inf.type === "swallowed-error";
|
|
1340
|
+
const isError = result.failedRules.includes(inf.type) || inf.type === "test-skip" || inf.type === "trivial-assert" || inf.type === "swallowed-error";
|
|
1119
1341
|
const line = inf.lineNumber || 1;
|
|
1120
1342
|
const title = inf.reason.replace(/[\r\n]/g, " ");
|
|
1121
1343
|
const msg = (inf.lineSnippet || inf.reason).replace(/[\r\n]/g, " ");
|
|
@@ -1181,9 +1403,115 @@ var VALID_DETECTORS = [
|
|
|
1181
1403
|
"test-skip",
|
|
1182
1404
|
"type-escape",
|
|
1183
1405
|
"swallowed-error",
|
|
1184
|
-
"panic-loop"
|
|
1406
|
+
"panic-loop",
|
|
1407
|
+
"trivial-assert",
|
|
1408
|
+
"stub-abandonment"
|
|
1185
1409
|
];
|
|
1186
1410
|
|
|
1411
|
+
// src/init.ts
|
|
1412
|
+
import * as fs2 from "fs";
|
|
1413
|
+
import * as path2 from "path";
|
|
1414
|
+
var DEFAULT_CONFIG_TEMPLATE = {
|
|
1415
|
+
$schema: "https://raw.githubusercontent.com/sidzuya/agent-roast/main/schema.json",
|
|
1416
|
+
ignorePaths: [
|
|
1417
|
+
"**/*.min.js",
|
|
1418
|
+
"**/vendor/**",
|
|
1419
|
+
"**/dist/**",
|
|
1420
|
+
"**/build/**",
|
|
1421
|
+
"**/fixtures/**"
|
|
1422
|
+
],
|
|
1423
|
+
rules: {
|
|
1424
|
+
"hallucinated-package": "error",
|
|
1425
|
+
"empty-catch": "warn",
|
|
1426
|
+
"trivial-assert": "warn",
|
|
1427
|
+
"stub-abandonment": "warn",
|
|
1428
|
+
"mock-leak": "warn"
|
|
1429
|
+
},
|
|
1430
|
+
scoreThreshold: 70
|
|
1431
|
+
};
|
|
1432
|
+
var GITHUB_WORKFLOW_TEMPLATE = `name: Agent Roast
|
|
1433
|
+
on:
|
|
1434
|
+
pull_request:
|
|
1435
|
+
branches: [main, master]
|
|
1436
|
+
push:
|
|
1437
|
+
branches: [main, master]
|
|
1438
|
+
|
|
1439
|
+
jobs:
|
|
1440
|
+
audit:
|
|
1441
|
+
name: Audit AI Shortcuts
|
|
1442
|
+
runs-on: ubuntu-latest
|
|
1443
|
+
steps:
|
|
1444
|
+
- name: Checkout repository
|
|
1445
|
+
uses: actions/checkout@v4
|
|
1446
|
+
with:
|
|
1447
|
+
fetch-depth: 0
|
|
1448
|
+
|
|
1449
|
+
- name: Setup Node.js
|
|
1450
|
+
uses: actions/setup-node@v4
|
|
1451
|
+
with:
|
|
1452
|
+
node-version: 20
|
|
1453
|
+
|
|
1454
|
+
- name: Run agent-roast PR check
|
|
1455
|
+
if: github.event_name == 'pull_request'
|
|
1456
|
+
run: npx agent-roast --base \${{ github.event.pull_request.base.sha }} --head \${{ github.event.pull_request.head.sha }} --format github
|
|
1457
|
+
|
|
1458
|
+
- name: Run agent-roast repo audit
|
|
1459
|
+
if: github.event_name != 'pull_request'
|
|
1460
|
+
run: npx agent-roast --fail-under 70
|
|
1461
|
+
`;
|
|
1462
|
+
var PRE_COMMIT_HOOK_TEMPLATE = `#!/bin/sh
|
|
1463
|
+
# Run agent-roast pre-commit verification
|
|
1464
|
+
npx agent-roast --fail-under 70
|
|
1465
|
+
`;
|
|
1466
|
+
async function initProject(options = {}) {
|
|
1467
|
+
const cwd = path2.resolve(options.cwd || process.cwd());
|
|
1468
|
+
const created = [];
|
|
1469
|
+
const skipped = [];
|
|
1470
|
+
const configPath = path2.join(cwd, ".agentroastrc.json");
|
|
1471
|
+
const altConfigPath = path2.join(cwd, "agent-roast.config.json");
|
|
1472
|
+
if (fs2.existsSync(configPath) || fs2.existsSync(altConfigPath)) {
|
|
1473
|
+
skipped.push(".agentroastrc.json");
|
|
1474
|
+
} else {
|
|
1475
|
+
fs2.writeFileSync(
|
|
1476
|
+
configPath,
|
|
1477
|
+
JSON.stringify(DEFAULT_CONFIG_TEMPLATE, null, 2) + "\n",
|
|
1478
|
+
"utf-8"
|
|
1479
|
+
);
|
|
1480
|
+
created.push(".agentroastrc.json");
|
|
1481
|
+
}
|
|
1482
|
+
if (options.ci) {
|
|
1483
|
+
const workflowDir = path2.join(cwd, ".github", "workflows");
|
|
1484
|
+
const workflowPath = path2.join(workflowDir, "agent-roast.yml");
|
|
1485
|
+
if (fs2.existsSync(workflowPath)) {
|
|
1486
|
+
skipped.push(".github/workflows/agent-roast.yml");
|
|
1487
|
+
} else {
|
|
1488
|
+
fs2.mkdirSync(workflowDir, { recursive: true });
|
|
1489
|
+
fs2.writeFileSync(workflowPath, GITHUB_WORKFLOW_TEMPLATE, "utf-8");
|
|
1490
|
+
created.push(".github/workflows/agent-roast.yml");
|
|
1491
|
+
}
|
|
1492
|
+
}
|
|
1493
|
+
if (options.hook) {
|
|
1494
|
+
const gitDir = path2.join(cwd, ".git");
|
|
1495
|
+
if (!fs2.existsSync(gitDir) || !fs2.statSync(gitDir).isDirectory()) {
|
|
1496
|
+
skipped.push(".git/hooks/pre-commit (no .git directory found)");
|
|
1497
|
+
} else {
|
|
1498
|
+
const hooksDir = path2.join(gitDir, "hooks");
|
|
1499
|
+
const hookPath = path2.join(hooksDir, "pre-commit");
|
|
1500
|
+
if (fs2.existsSync(hookPath)) {
|
|
1501
|
+
skipped.push(".git/hooks/pre-commit");
|
|
1502
|
+
} else {
|
|
1503
|
+
fs2.mkdirSync(hooksDir, { recursive: true });
|
|
1504
|
+
fs2.writeFileSync(hookPath, PRE_COMMIT_HOOK_TEMPLATE, {
|
|
1505
|
+
encoding: "utf-8",
|
|
1506
|
+
mode: 493
|
|
1507
|
+
});
|
|
1508
|
+
created.push(".git/hooks/pre-commit");
|
|
1509
|
+
}
|
|
1510
|
+
}
|
|
1511
|
+
}
|
|
1512
|
+
return { created, skipped };
|
|
1513
|
+
}
|
|
1514
|
+
|
|
1187
1515
|
// src/index.ts
|
|
1188
1516
|
var InvalidDetectorError = class extends Error {
|
|
1189
1517
|
constructor(message) {
|
|
@@ -1192,10 +1520,10 @@ var InvalidDetectorError = class extends Error {
|
|
|
1192
1520
|
}
|
|
1193
1521
|
};
|
|
1194
1522
|
async function roastRepository(options = {}) {
|
|
1195
|
-
const cwd =
|
|
1523
|
+
const cwd = path3.resolve(options.cwd || process.cwd());
|
|
1196
1524
|
const since = options.since || "90 days ago";
|
|
1197
|
-
const repoName =
|
|
1198
|
-
const collector = new MetricsCollector({ all: options.all });
|
|
1525
|
+
const repoName = path3.basename(cwd);
|
|
1526
|
+
const collector = new MetricsCollector({ all: options.all, ignorePaths: options.ignorePaths });
|
|
1199
1527
|
const gitStream = streamGitLog(cwd, since);
|
|
1200
1528
|
await parseGitLogStream(gitStream, (commit, stats) => {
|
|
1201
1529
|
collector.processCommit(commit);
|
|
@@ -1209,7 +1537,7 @@ async function roastRepository(options = {}) {
|
|
|
1209
1537
|
return { summary, output };
|
|
1210
1538
|
}
|
|
1211
1539
|
async function auditPullRequest(options) {
|
|
1212
|
-
const cwd =
|
|
1540
|
+
const cwd = path3.resolve(options.cwd || process.cwd());
|
|
1213
1541
|
const format = options.format || "terminal";
|
|
1214
1542
|
const failOn = (options.failOn || ["test-skip"]).map((s) => s.trim()).filter(Boolean);
|
|
1215
1543
|
for (const det of failOn) {
|
|
@@ -1224,15 +1552,15 @@ async function auditPullRequest(options) {
|
|
|
1224
1552
|
const mergeBaseSha = findMergeBase(cwd, baseSha, headSha);
|
|
1225
1553
|
const diffStream = streamGitDiff(cwd, mergeBaseSha, headSha);
|
|
1226
1554
|
let rawDiff = "";
|
|
1227
|
-
await new Promise((
|
|
1555
|
+
await new Promise((resolve2, reject) => {
|
|
1228
1556
|
diffStream.on("data", (chunk) => {
|
|
1229
1557
|
rawDiff += chunk.toString();
|
|
1230
1558
|
});
|
|
1231
|
-
diffStream.on("end", () =>
|
|
1559
|
+
diffStream.on("end", () => resolve2());
|
|
1232
1560
|
diffStream.on("error", reject);
|
|
1233
1561
|
});
|
|
1234
1562
|
const { files, skippedLongLines, skippedUnknownFiles } = parseDiff(rawDiff);
|
|
1235
|
-
const audit = auditFileDiffs(files, headSha);
|
|
1563
|
+
const audit = auditFileDiffs(files, headSha, (/* @__PURE__ */ new Date()).toISOString(), { ignorePaths: options.ignorePaths });
|
|
1236
1564
|
const infractions = audit.infractions;
|
|
1237
1565
|
const defectDensityPer100Lines = audit.linesAdded > 0 ? Math.round(infractions.length / audit.linesAdded * 100 * 100) / 100 : 0;
|
|
1238
1566
|
const failedRules = [...new Set(
|
|
@@ -1282,7 +1610,11 @@ export {
|
|
|
1282
1610
|
resolveCommitSha,
|
|
1283
1611
|
findMergeBase,
|
|
1284
1612
|
GitResolutionError,
|
|
1613
|
+
DEFAULT_CONFIG,
|
|
1614
|
+
loadConfig,
|
|
1615
|
+
matchIgnorePath,
|
|
1285
1616
|
isCodeFile,
|
|
1617
|
+
isTestFile,
|
|
1286
1618
|
auditFileDiffs,
|
|
1287
1619
|
stripStringAndRegexLiterals,
|
|
1288
1620
|
MetricsCollector,
|
|
@@ -1292,6 +1624,10 @@ export {
|
|
|
1292
1624
|
formatGithubAnnotations,
|
|
1293
1625
|
formatGithubStepSummary,
|
|
1294
1626
|
VALID_DETECTORS,
|
|
1627
|
+
DEFAULT_CONFIG_TEMPLATE,
|
|
1628
|
+
GITHUB_WORKFLOW_TEMPLATE,
|
|
1629
|
+
PRE_COMMIT_HOOK_TEMPLATE,
|
|
1630
|
+
initProject,
|
|
1295
1631
|
InvalidDetectorError,
|
|
1296
1632
|
roastRepository,
|
|
1297
1633
|
auditPullRequest
|
package/dist/cli.js
CHANGED
|
@@ -5,8 +5,10 @@ import {
|
|
|
5
5
|
InvalidDetectorError,
|
|
6
6
|
auditPullRequest,
|
|
7
7
|
formatGithubStepSummary,
|
|
8
|
+
initProject,
|
|
9
|
+
loadConfig,
|
|
8
10
|
roastRepository
|
|
9
|
-
} from "./chunk-
|
|
11
|
+
} from "./chunk-SMZZAGFS.js";
|
|
10
12
|
|
|
11
13
|
// src/cli.ts
|
|
12
14
|
import { Command } from "commander";
|
|
@@ -15,15 +17,22 @@ import * as fs from "fs";
|
|
|
15
17
|
var require2 = createRequire(import.meta.url);
|
|
16
18
|
var pkg = require2("../package.json");
|
|
17
19
|
var program = new Command();
|
|
18
|
-
program.name("agent-roast").description("Audit your git history or PR diffs for AI coding shortcuts and panic loops").version(pkg.version || "0.
|
|
20
|
+
program.name("agent-roast").description("Audit your git history or PR diffs for AI coding shortcuts and panic loops").version(pkg.version || "0.3.0").argument("[path]", "Path to git repository (default: current directory)", ".").option("-a, --all", "Audit ALL commits regardless of AI attribution tags", false).option("-s, --since <time>", "Time window for git analysis", "90 days ago").option("-v, --verbose", "Show commit SHAs and line snippets for all infractions", false).option("--json", "Output results as raw JSON", false).option("--base <ref>", "Base git ref/branch for PR mode audit (e.g. origin/main)").option("--head <ref>", "Head git ref/SHA for PR mode audit (default: HEAD)", "HEAD").option("--fail-on <rules>", "Comma-separated detectors that trigger CI failure").option("--fail-under <score>", "Fail CI if agent score is below threshold (0-100)").option("--format <format>", "Output format: terminal, json, github", "terminal").option("--authors", "Display author names and emails in report", false).action(async (repoPath, options) => {
|
|
19
21
|
try {
|
|
20
22
|
if (process.env.GITHUB_EVENT_NAME === "pull_request_target") {
|
|
21
23
|
console.warn(
|
|
22
24
|
"Warning: pull_request_target runs with elevated repository privileges. agent-roast recommends running on 'pull_request'."
|
|
23
25
|
);
|
|
24
26
|
}
|
|
27
|
+
const config = await loadConfig(repoPath);
|
|
25
28
|
if (options.base) {
|
|
26
|
-
|
|
29
|
+
let failOnRules;
|
|
30
|
+
if (options.failOn) {
|
|
31
|
+
failOnRules = options.failOn.split(",").map((s) => s.trim()).filter(Boolean);
|
|
32
|
+
} else {
|
|
33
|
+
const errorRules = Object.entries(config.rules || {}).filter(([_, level]) => level === "error").map(([rule]) => rule);
|
|
34
|
+
failOnRules = errorRules.length > 0 ? errorRules : ["test-skip"];
|
|
35
|
+
}
|
|
27
36
|
const chosenFormat = options.json ? "json" : options.format || "terminal";
|
|
28
37
|
const prResult = await auditPullRequest({
|
|
29
38
|
cwd: repoPath,
|
|
@@ -32,7 +41,8 @@ program.name("agent-roast").description("Audit your git history or PR diffs for
|
|
|
32
41
|
failOn: failOnRules,
|
|
33
42
|
format: chosenFormat,
|
|
34
43
|
verbose: options.verbose,
|
|
35
|
-
authors: options.authors
|
|
44
|
+
authors: options.authors,
|
|
45
|
+
ignorePaths: config.ignorePaths
|
|
36
46
|
});
|
|
37
47
|
if (process.env.GITHUB_STEP_SUMMARY) {
|
|
38
48
|
try {
|
|
@@ -65,14 +75,20 @@ has-supported-changes=${prResult.hasSupportedChanges}
|
|
|
65
75
|
since: options.since,
|
|
66
76
|
all: options.all,
|
|
67
77
|
verbose: options.verbose,
|
|
68
|
-
json: options.json
|
|
78
|
+
json: options.json,
|
|
79
|
+
ignorePaths: config.ignorePaths
|
|
69
80
|
});
|
|
70
81
|
if (options.json) {
|
|
71
82
|
console.log(JSON.stringify(result.summary, null, 2));
|
|
72
83
|
} else {
|
|
73
84
|
console.log(result.output);
|
|
74
85
|
}
|
|
75
|
-
|
|
86
|
+
const scoreThreshold = options.failUnder !== void 0 ? Number(options.failUnder) : config.scoreThreshold;
|
|
87
|
+
if (scoreThreshold !== void 0 && !isNaN(scoreThreshold) && result.summary.score !== null && result.summary.score < scoreThreshold) {
|
|
88
|
+
process.exitCode = 1;
|
|
89
|
+
} else {
|
|
90
|
+
process.exitCode = 0;
|
|
91
|
+
}
|
|
76
92
|
}
|
|
77
93
|
} catch (err) {
|
|
78
94
|
if (err instanceof GitResolutionError || err instanceof InvalidDetectorError || err instanceof GitError || err.name === "GitError") {
|
|
@@ -84,4 +100,31 @@ has-supported-changes=${prResult.hasSupportedChanges}
|
|
|
84
100
|
}
|
|
85
101
|
}
|
|
86
102
|
});
|
|
103
|
+
program.command("init").description("Initialize .agentroastrc.json and CI/hook configurations").argument("[dir]", "Directory to initialize", ".").option("-y, --yes", "Skip interactive prompts and use defaults", false).option("--ci", "Generate GitHub Actions workflow file", false).option("--hook", "Install Git pre-commit hook", false).action(async (targetDir, opts) => {
|
|
104
|
+
try {
|
|
105
|
+
const res = await initProject({
|
|
106
|
+
cwd: targetDir,
|
|
107
|
+
yes: opts.yes,
|
|
108
|
+
ci: opts.ci,
|
|
109
|
+
hook: opts.hook
|
|
110
|
+
});
|
|
111
|
+
if (res.created.length > 0) {
|
|
112
|
+
console.log(`Initialized agent-roast in ${targetDir}:`);
|
|
113
|
+
for (const file of res.created) {
|
|
114
|
+
console.log(` + Created ${file}`);
|
|
115
|
+
}
|
|
116
|
+
}
|
|
117
|
+
if (res.skipped.length > 0) {
|
|
118
|
+
for (const file of res.skipped) {
|
|
119
|
+
console.log(` - Skipped ${file} (already exists or not applicable)`);
|
|
120
|
+
}
|
|
121
|
+
}
|
|
122
|
+
if (res.created.length === 0 && res.skipped.length === 0) {
|
|
123
|
+
console.log("No files were created.");
|
|
124
|
+
}
|
|
125
|
+
} catch (err) {
|
|
126
|
+
console.error(`Failed to initialize agent-roast: ${err.message}`);
|
|
127
|
+
process.exitCode = 1;
|
|
128
|
+
}
|
|
129
|
+
});
|
|
87
130
|
program.parse(process.argv);
|
package/dist/index.d.ts
CHANGED
|
@@ -9,7 +9,7 @@ interface CommitInfo {
|
|
|
9
9
|
isAiAttributed: boolean;
|
|
10
10
|
aiSignatures: string[];
|
|
11
11
|
}
|
|
12
|
-
type DetectorType = "test-skip" | "type-escape" | "swallowed-error" | "panic-loop";
|
|
12
|
+
type DetectorType = "test-skip" | "type-escape" | "swallowed-error" | "panic-loop" | "trivial-assert" | "stub-abandonment";
|
|
13
13
|
declare const VALID_DETECTORS: readonly DetectorType[];
|
|
14
14
|
interface Infraction {
|
|
15
15
|
type: DetectorType;
|
|
@@ -30,6 +30,8 @@ interface MetricSummary {
|
|
|
30
30
|
typeEscapes: Infraction[];
|
|
31
31
|
swallowedErrors: Infraction[];
|
|
32
32
|
panicLoops: Infraction[];
|
|
33
|
+
trivialAsserts: Infraction[];
|
|
34
|
+
stubAbandonments: Infraction[];
|
|
33
35
|
rawScore: number;
|
|
34
36
|
score: number | null;
|
|
35
37
|
isSufficientData: boolean;
|
|
@@ -45,6 +47,8 @@ interface RoastOptions {
|
|
|
45
47
|
verbose?: boolean;
|
|
46
48
|
json?: boolean;
|
|
47
49
|
authors?: boolean;
|
|
50
|
+
ignorePaths?: string[];
|
|
51
|
+
scoreThreshold?: number;
|
|
48
52
|
}
|
|
49
53
|
interface PrAuditOptions {
|
|
50
54
|
cwd?: string;
|
|
@@ -54,6 +58,7 @@ interface PrAuditOptions {
|
|
|
54
58
|
format?: "terminal" | "json" | "github";
|
|
55
59
|
verbose?: boolean;
|
|
56
60
|
authors?: boolean;
|
|
61
|
+
ignorePaths?: string[];
|
|
57
62
|
}
|
|
58
63
|
interface PrAuditResult {
|
|
59
64
|
schemaVersion: string;
|
|
@@ -139,11 +144,14 @@ declare class GitResolutionError extends Error {
|
|
|
139
144
|
constructor(message: string);
|
|
140
145
|
}
|
|
141
146
|
|
|
142
|
-
declare function isCodeFile(filePath: string): boolean;
|
|
147
|
+
declare function isCodeFile(filePath: string, customIgnorePaths?: string[]): boolean;
|
|
148
|
+
declare function isTestFile(filePath: string): boolean;
|
|
143
149
|
/**
|
|
144
150
|
* Audits a collection of file diffs (e.g. for PR mode) against detector rules.
|
|
145
151
|
*/
|
|
146
|
-
declare function auditFileDiffs(files: ParsedFileDiff[], commitHash?: string, commitDate?: string
|
|
152
|
+
declare function auditFileDiffs(files: ParsedFileDiff[], commitHash?: string, commitDate?: string, options?: {
|
|
153
|
+
ignorePaths?: string[];
|
|
154
|
+
}): {
|
|
147
155
|
infractions: Infraction[];
|
|
148
156
|
hasSupportedChanges: boolean;
|
|
149
157
|
filesInspected: number;
|
|
@@ -163,12 +171,16 @@ declare class MetricsCollector {
|
|
|
163
171
|
private typeEscapes;
|
|
164
172
|
private swallowedErrors;
|
|
165
173
|
private panicLoops;
|
|
174
|
+
private trivialAsserts;
|
|
175
|
+
private stubAbandonments;
|
|
166
176
|
private skippedLongLines;
|
|
167
177
|
private skippedUnknownFiles;
|
|
168
178
|
private activePanicChain;
|
|
169
179
|
private includeAll;
|
|
180
|
+
private ignorePaths?;
|
|
170
181
|
constructor(options?: {
|
|
171
182
|
all?: boolean;
|
|
183
|
+
ignorePaths?: string[];
|
|
172
184
|
});
|
|
173
185
|
recordSkipped(stats: {
|
|
174
186
|
skippedLongLines?: number;
|
|
@@ -195,6 +207,55 @@ declare function formatGithubAnnotations(result: PrAuditResult): string;
|
|
|
195
207
|
*/
|
|
196
208
|
declare function formatGithubStepSummary(result: PrAuditResult): string;
|
|
197
209
|
|
|
210
|
+
interface AgentRoastConfig {
|
|
211
|
+
/** Detector rules to fail CI on in PR mode */
|
|
212
|
+
failOn?: string[];
|
|
213
|
+
/** Glob patterns or prefixes to ignore from analysis */
|
|
214
|
+
ignorePaths?: string[];
|
|
215
|
+
/** Minimum passing score for full repository audits (0-100) */
|
|
216
|
+
scoreThreshold?: number;
|
|
217
|
+
/** Granular rule configuration (error | warn | off) */
|
|
218
|
+
rules?: Record<string, "error" | "warn" | "off">;
|
|
219
|
+
/** Maximum minutes between fix commits before flagging panic-loop */
|
|
220
|
+
panicLoopWindowMinutes?: number;
|
|
221
|
+
}
|
|
222
|
+
declare const DEFAULT_CONFIG: AgentRoastConfig;
|
|
223
|
+
/**
|
|
224
|
+
* Discovers and parses repository configuration file.
|
|
225
|
+
* Returns DEFAULT_CONFIG if no configuration file is found or if parsing fails.
|
|
226
|
+
*/
|
|
227
|
+
declare function loadConfig(cwd?: string): AgentRoastConfig;
|
|
228
|
+
/**
|
|
229
|
+
* Matches a relative or normalized file path against an array of glob ignore patterns.
|
|
230
|
+
*/
|
|
231
|
+
declare function matchIgnorePath(filePath: string, ignorePatterns?: string[]): boolean;
|
|
232
|
+
|
|
233
|
+
interface InitOptions {
|
|
234
|
+
cwd?: string;
|
|
235
|
+
yes?: boolean;
|
|
236
|
+
ci?: boolean;
|
|
237
|
+
hook?: boolean;
|
|
238
|
+
}
|
|
239
|
+
interface InitResult {
|
|
240
|
+
created: string[];
|
|
241
|
+
skipped: string[];
|
|
242
|
+
}
|
|
243
|
+
declare const DEFAULT_CONFIG_TEMPLATE: {
|
|
244
|
+
$schema: string;
|
|
245
|
+
ignorePaths: string[];
|
|
246
|
+
rules: {
|
|
247
|
+
"hallucinated-package": string;
|
|
248
|
+
"empty-catch": string;
|
|
249
|
+
"trivial-assert": string;
|
|
250
|
+
"stub-abandonment": string;
|
|
251
|
+
"mock-leak": string;
|
|
252
|
+
};
|
|
253
|
+
scoreThreshold: number;
|
|
254
|
+
};
|
|
255
|
+
declare const GITHUB_WORKFLOW_TEMPLATE = "name: Agent Roast\non:\n pull_request:\n branches: [main, master]\n push:\n branches: [main, master]\n\njobs:\n audit:\n name: Audit AI Shortcuts\n runs-on: ubuntu-latest\n steps:\n - name: Checkout repository\n uses: actions/checkout@v4\n with:\n fetch-depth: 0\n\n - name: Setup Node.js\n uses: actions/setup-node@v4\n with:\n node-version: 20\n\n - name: Run agent-roast PR check\n if: github.event_name == 'pull_request'\n run: npx agent-roast --base ${{ github.event.pull_request.base.sha }} --head ${{ github.event.pull_request.head.sha }} --format github\n\n - name: Run agent-roast repo audit\n if: github.event_name != 'pull_request'\n run: npx agent-roast --fail-under 70\n";
|
|
256
|
+
declare const PRE_COMMIT_HOOK_TEMPLATE = "#!/bin/sh\n# Run agent-roast pre-commit verification\nnpx agent-roast --fail-under 70\n";
|
|
257
|
+
declare function initProject(options?: InitOptions): Promise<InitResult>;
|
|
258
|
+
|
|
198
259
|
declare class InvalidDetectorError extends Error {
|
|
199
260
|
constructor(message: string);
|
|
200
261
|
}
|
|
@@ -204,4 +265,4 @@ declare function roastRepository(options?: RoastOptions): Promise<{
|
|
|
204
265
|
}>;
|
|
205
266
|
declare function auditPullRequest(options: PrAuditOptions): Promise<PrAuditResult>;
|
|
206
267
|
|
|
207
|
-
export { type AiAttributionResult, type CommitInfo, type DetectorType, FIELD_SEPARATOR, GitError, GitResolutionError, HERMETIC_GIT_FLAGS, type Infraction, InvalidDetectorError, type MetricSummary, MetricsCollector, type ParseDiffResult, type ParsedCommit, type ParsedDiffLine, type ParsedFileDiff, type PrAuditOptions, type PrAuditResult, RECORD_SEPARATOR, type RoastOptions, VALID_DETECTORS, auditFileDiffs, auditPullRequest, detectAiAttribution, findMergeBase, formatGithubAnnotations, formatGithubStepSummary, formatPrTerminalCard, formatTerminalCard, generateShareUrl, getHermeticGitEnv, isCodeFile, parseDiff, parseGitLogStream, resolveCommitSha, roastRepository, streamGitDiff, streamGitLog, stripStringAndRegexLiterals, unquoteGitPath };
|
|
268
|
+
export { type AgentRoastConfig, type AiAttributionResult, type CommitInfo, DEFAULT_CONFIG, DEFAULT_CONFIG_TEMPLATE, type DetectorType, FIELD_SEPARATOR, GITHUB_WORKFLOW_TEMPLATE, GitError, GitResolutionError, HERMETIC_GIT_FLAGS, type Infraction, type InitOptions, type InitResult, InvalidDetectorError, type MetricSummary, MetricsCollector, PRE_COMMIT_HOOK_TEMPLATE, type ParseDiffResult, type ParsedCommit, type ParsedDiffLine, type ParsedFileDiff, type PrAuditOptions, type PrAuditResult, RECORD_SEPARATOR, type RoastOptions, VALID_DETECTORS, auditFileDiffs, auditPullRequest, detectAiAttribution, findMergeBase, formatGithubAnnotations, formatGithubStepSummary, formatPrTerminalCard, formatTerminalCard, generateShareUrl, getHermeticGitEnv, initProject, isCodeFile, isTestFile, loadConfig, matchIgnorePath, parseDiff, parseGitLogStream, resolveCommitSha, roastRepository, streamGitDiff, streamGitLog, stripStringAndRegexLiterals, unquoteGitPath };
|
package/dist/index.js
CHANGED
|
@@ -1,11 +1,15 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
import {
|
|
3
|
+
DEFAULT_CONFIG,
|
|
4
|
+
DEFAULT_CONFIG_TEMPLATE,
|
|
3
5
|
FIELD_SEPARATOR,
|
|
6
|
+
GITHUB_WORKFLOW_TEMPLATE,
|
|
4
7
|
GitError,
|
|
5
8
|
GitResolutionError,
|
|
6
9
|
HERMETIC_GIT_FLAGS,
|
|
7
10
|
InvalidDetectorError,
|
|
8
11
|
MetricsCollector,
|
|
12
|
+
PRE_COMMIT_HOOK_TEMPLATE,
|
|
9
13
|
RECORD_SEPARATOR,
|
|
10
14
|
VALID_DETECTORS,
|
|
11
15
|
auditFileDiffs,
|
|
@@ -18,7 +22,11 @@ import {
|
|
|
18
22
|
formatTerminalCard,
|
|
19
23
|
generateShareUrl,
|
|
20
24
|
getHermeticGitEnv,
|
|
25
|
+
initProject,
|
|
21
26
|
isCodeFile,
|
|
27
|
+
isTestFile,
|
|
28
|
+
loadConfig,
|
|
29
|
+
matchIgnorePath,
|
|
22
30
|
parseDiff,
|
|
23
31
|
parseGitLogStream,
|
|
24
32
|
resolveCommitSha,
|
|
@@ -27,14 +35,18 @@ import {
|
|
|
27
35
|
streamGitLog,
|
|
28
36
|
stripStringAndRegexLiterals,
|
|
29
37
|
unquoteGitPath
|
|
30
|
-
} from "./chunk-
|
|
38
|
+
} from "./chunk-SMZZAGFS.js";
|
|
31
39
|
export {
|
|
40
|
+
DEFAULT_CONFIG,
|
|
41
|
+
DEFAULT_CONFIG_TEMPLATE,
|
|
32
42
|
FIELD_SEPARATOR,
|
|
43
|
+
GITHUB_WORKFLOW_TEMPLATE,
|
|
33
44
|
GitError,
|
|
34
45
|
GitResolutionError,
|
|
35
46
|
HERMETIC_GIT_FLAGS,
|
|
36
47
|
InvalidDetectorError,
|
|
37
48
|
MetricsCollector,
|
|
49
|
+
PRE_COMMIT_HOOK_TEMPLATE,
|
|
38
50
|
RECORD_SEPARATOR,
|
|
39
51
|
VALID_DETECTORS,
|
|
40
52
|
auditFileDiffs,
|
|
@@ -47,7 +59,11 @@ export {
|
|
|
47
59
|
formatTerminalCard,
|
|
48
60
|
generateShareUrl,
|
|
49
61
|
getHermeticGitEnv,
|
|
62
|
+
initProject,
|
|
50
63
|
isCodeFile,
|
|
64
|
+
isTestFile,
|
|
65
|
+
loadConfig,
|
|
66
|
+
matchIgnorePath,
|
|
51
67
|
parseDiff,
|
|
52
68
|
parseGitLogStream,
|
|
53
69
|
resolveCommitSha,
|