agent-roast 0.1.0 โ 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +113 -47
- package/package.json +2 -1
package/README.md
CHANGED
|
@@ -1,82 +1,148 @@
|
|
|
1
|
-
|
|
1
|
+
<div align="center">
|
|
2
2
|
|
|
3
|
-
>
|
|
3
|
+
<picture>
|
|
4
|
+
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/logo-dark.svg">
|
|
5
|
+
<img src="docs/assets/logo.svg" alt="agent-roast" width="460">
|
|
6
|
+
</picture>
|
|
4
7
|
|
|
5
|
-
|
|
6
|
-
[-1f9d55)](#privacy--security)
|
|
7
|
-
[](LICENSE)
|
|
8
|
+
### Audit your git history for AI coding agent shortcuts, infractions, and panic loops
|
|
8
9
|
|
|
9
|
-
|
|
10
|
+
[](https://www.npmjs.com/package/agent-roast)
|
|
11
|
+
[-1f9d55)](#privacy--zero-telemetry)
|
|
12
|
+
[](#performance)
|
|
13
|
+
[](LICENSE)
|
|
10
14
|
|
|
11
|
-
|
|
15
|
+
<br/>
|
|
12
16
|
|
|
13
17
|
```bash
|
|
14
18
|
npx agent-roast
|
|
15
19
|
```
|
|
16
20
|
|
|
21
|
+
</div>
|
|
22
|
+
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
Did your AI coding assistant write clean, maintainable software โ or did it quietly delete failing tests, slap `as any` everywhere, and spam 6 panic commits in 8 minutes?
|
|
26
|
+
|
|
27
|
+
**`agent-roast`** is an offline, zero-telemetry CLI scorecard that parses git diffs to detect AI shortcuts and calculate an objective **Discipline Score (0โ100)** with a personality archetype.
|
|
28
|
+
|
|
29
|
+
<div align="center">
|
|
30
|
+
<img src="docs/assets/demo.svg" alt="agent-roast terminal audit demo" width="100%">
|
|
31
|
+
</div>
|
|
32
|
+
|
|
17
33
|
---
|
|
18
34
|
|
|
19
|
-
|
|
35
|
+
## โก๏ธ Quickstart
|
|
20
36
|
|
|
21
|
-
|
|
22
|
-
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
23
|
-
โ ๐ค AGENT ROAST DOSSIER โ
|
|
24
|
-
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
37
|
+
Run directly in any repository (no install required):
|
|
25
38
|
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
39
|
+
```bash
|
|
40
|
+
# Audit AI-attributed commits (Claude Code, Cursor, Aider, Devin, etc.)
|
|
41
|
+
npx agent-roast
|
|
29
42
|
|
|
30
|
-
|
|
31
|
-
|
|
43
|
+
# Audit ALL commits across repo history (including unverified human commits)
|
|
44
|
+
npx agent-roast --all
|
|
32
45
|
|
|
33
|
-
|
|
34
|
-
|
|
46
|
+
# Print exact commit SHAs and line snippets for all infractions
|
|
47
|
+
npx agent-roast --verbose
|
|
35
48
|
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
โ Skipped tests: 4 (it.skip, describe.skip, xit)
|
|
39
|
-
โฒ Type escapes: 16 (as any, @ts-ignore, eslint-disable)
|
|
40
|
-
โ Swallowed errors: 3 (empty catch {}, except: pass)
|
|
41
|
-
โป Panic fix loops: 2 (quick fixes < 15m apart)
|
|
49
|
+
# Audit a specific time window (default: last 90 days)
|
|
50
|
+
npx agent-roast --since "30 days ago"
|
|
42
51
|
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
Tip: Run with --verbose to see exact commit SHAs and lines
|
|
52
|
+
# Output structured JSON for CI/CD pipelines
|
|
53
|
+
npx agent-roast --json
|
|
46
54
|
```
|
|
47
55
|
|
|
48
56
|
---
|
|
49
57
|
|
|
50
|
-
|
|
58
|
+
## ๐ What It Detects
|
|
59
|
+
|
|
60
|
+
`agent-roast` analyzes **only added lines** (`+`) in source code files, eliminating false alarms from refactoring or file renames:
|
|
51
61
|
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
62
|
+
| Infraction | Patterns Detected | Penalty Weight | Why It Matters |
|
|
63
|
+
| :--- | :--- | :---: | :--- |
|
|
64
|
+
| **Skipped Tests** | `it.skip`, `describe.skip`, `xit`, `pytest.skip`, `test.todo` | **15 pts** | Disabling failing tests instead of fixing the root bug. |
|
|
65
|
+
| **Swallowed Errors** | Empty `catch {}`, `except: pass` | **10 pts** | Silencing exceptions and sweeping runtime failures under the rug. |
|
|
66
|
+
| **Panic Fix Loops** | Consecutive quickfix commits within 15m by same author | **8 pts** | Frantic thrashing without reading stack traces or logs. |
|
|
67
|
+
| **Type Escapes** | `as any`, `// @ts-ignore`, `// @ts-expect-error`, `# type: ignore` | **3 pts** | Bypassing compiler guarantees to pretend code type-checks. |
|
|
68
|
+
|
|
69
|
+
> [!NOTE]
|
|
70
|
+
> String literals and regex patterns are automatically sanitized before analysis. Test fixtures containing pattern names as string data will never trigger false positives.
|
|
56
71
|
|
|
57
72
|
---
|
|
58
73
|
|
|
59
|
-
|
|
74
|
+
## ๐ญ Agent Archetypes
|
|
60
75
|
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
76
|
+
Depending on the dominant infraction patterns, `agent-roast` assigns your agent an archetype:
|
|
77
|
+
|
|
78
|
+
| Archetype | Icon | Trigger Condition | Personality Dossier |
|
|
79
|
+
| :--- | :---: | :--- | :--- |
|
|
80
|
+
| **The Clean Coder** | ๐งผ | Score $\ge 90$, 0 infractions | Spotless discipline. No skipped tests, zero panic loops, clean type safety. |
|
|
81
|
+
| **The Pragmatic Builder** | ๐ ๏ธ | Score $\ge 90$, minor compromises | Overwhelmingly disciplined. Only rare, isolated shortcuts across thousands of lines. |
|
|
82
|
+
| **The Panic Looper** | โป | Dominant panic loops ($\ge 2$) | Prone to rapid-fire fix and revert thrashing cycles when wrestling stubborn bugs. |
|
|
83
|
+
| **The Silent Vandal** | โ | Dominant skipped tests ($\ge 3$) | When test suites push back, tends to disable or skip tests to keep pipelines green. |
|
|
84
|
+
| **The Any Architect** | โฒ | Dominant type escapes ($\ge 5$) | Leans on compiler bypasses (`as any`, `@ts-ignore`) rather than strict type modeling. |
|
|
85
|
+
| **The Secret Keeper** | ๐ค | Dominant empty catches ($\ge 2$) | Tends to silence errors with empty catch blocks, obscuring runtime failures. |
|
|
86
|
+
| **The Chaos Gremlin** | ๐ฅ | Score $< 50$, mixed infractions | Mixes multiple shortcuts: skipped tests, type bypasses, and quick patches under pressure. |
|
|
87
|
+
|
|
88
|
+
---
|
|
89
|
+
|
|
90
|
+
## ๐ How Scoring Works
|
|
91
|
+
|
|
92
|
+
1. **Volume Dampening:** To prevent large repos from automatically scoring 99/100, penalties scale sub-linearly using a logarithmic volume curve:
|
|
93
|
+
$$\text{Volume Factor} = 1 + 1.2 \cdot \ln\left(\max\left(1, \frac{\text{LOC Added}}{1000}\right)\right)$$
|
|
94
|
+
2. **Commit Caps:** Maximum 10 infractions counted per commit to prevent a single refactoring commit from dominating the entire audit.
|
|
95
|
+
3. **Small Sample Protection:** Repositories with fewer than 200 lines of added code display `INSUFFICIENT DATA` instead of a premature score.
|
|
96
|
+
|
|
97
|
+
---
|
|
98
|
+
|
|
99
|
+
## ๐ค AI Attribution
|
|
100
|
+
|
|
101
|
+
`agent-roast` recognizes commits authored or co-authored by AI tools through:
|
|
102
|
+
- Git trailers (`Co-Authored-By: Claude <noreply@anthropic.com>`, Cursor, Aider, etc.)
|
|
103
|
+
- Commit messages (`generated by aider`, `authored by cursor`)
|
|
104
|
+
- Author emails and usernames containing recognized bot signatures
|
|
105
|
+
|
|
106
|
+
To audit all commits regardless of attribution, pass the `--all` flag.
|
|
68
107
|
|
|
69
108
|
---
|
|
70
109
|
|
|
71
|
-
|
|
110
|
+
## ๐ Privacy & Zero Telemetry
|
|
111
|
+
|
|
112
|
+
- **100% Local Execution:** Runs streaming `git log` directly on your machine in under 200ms.
|
|
113
|
+
- **Zero Network Calls:** No external API requests, no telemetry, no analytics.
|
|
114
|
+
- **Hermetic:** Source code never leaves your computer.
|
|
72
115
|
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
116
|
+
---
|
|
117
|
+
|
|
118
|
+
## ๐ฆ GitHub Actions CI Integration
|
|
119
|
+
|
|
120
|
+
Fail a PR if an agent's discipline score drops below 60:
|
|
121
|
+
|
|
122
|
+
```yaml
|
|
123
|
+
name: Agent Audit
|
|
124
|
+
on: [pull_request]
|
|
125
|
+
|
|
126
|
+
jobs:
|
|
127
|
+
roast:
|
|
128
|
+
runs-on: ubuntu-latest
|
|
129
|
+
steps:
|
|
130
|
+
- uses: actions/checkout@v4
|
|
131
|
+
with:
|
|
132
|
+
fetch-depth: 0
|
|
133
|
+
- name: Run agent-roast
|
|
134
|
+
run: |
|
|
135
|
+
npx agent-roast --since "7 days ago" --json > roast.json
|
|
136
|
+
SCORE=$(jq '.score' roast.json)
|
|
137
|
+
echo "Discipline Score: $SCORE / 100"
|
|
138
|
+
if [ "$SCORE" -lt 60 ]; then
|
|
139
|
+
echo "โ Agent discipline score below 60!"
|
|
140
|
+
exit 1
|
|
141
|
+
fi
|
|
142
|
+
```
|
|
77
143
|
|
|
78
144
|
---
|
|
79
145
|
|
|
80
|
-
|
|
146
|
+
## ๐ License
|
|
81
147
|
|
|
82
148
|
[MIT](LICENSE)
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agent-roast",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.1",
|
|
4
4
|
"description": "Zero-telemetry CLI scorecard analyzing git history for AI coding agent infractions and panic loops",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./dist/index.js",
|
|
@@ -45,6 +45,7 @@
|
|
|
45
45
|
},
|
|
46
46
|
"devDependencies": {
|
|
47
47
|
"@types/node": "^20.0.0",
|
|
48
|
+
"gifenc": "^1.0.3",
|
|
48
49
|
"tsup": "^8.0.0",
|
|
49
50
|
"typescript": "^5.0.0",
|
|
50
51
|
"vitest": "^3.0.0"
|