openpitstop 1.5.2 → 1.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +194 -140
- package/demo/DEMO_SCRIPT.md +81 -0
- package/demo/web-app-broken/src/server.js +50 -0
- package/demo/web-app-fixed/src/server.js +61 -0
- package/dist/analyzers/securityStatic.js +122 -0
- package/dist/analyzers/securityStatic.js.map +1 -1
- package/dist/commands/drive.js +365 -104
- package/dist/commands/drive.js.map +1 -1
- package/dist/commands/next.d.ts +25 -0
- package/dist/commands/next.js +403 -72
- package/dist/commands/next.js.map +1 -1
- package/dist/graph/correlate.js +11 -1
- package/dist/graph/correlate.js.map +1 -1
- package/package.json +1 -1
- package/templates/pitstop-menu.command.md +26 -21
- package/templates/pitstop.command.md +25 -30
- package/templates/pitstop.prompt.md +18 -0
package/README.md
CHANGED
|
@@ -12,14 +12,57 @@ know it's actually done.
|
|
|
12
12
|
[](https://github.com/Krish-1507/OpenPitStop/actions/workflows/ci.yml)
|
|
13
13
|
[](LICENSE)
|
|
14
14
|
|
|
15
|
-
> AI coding agents are brilliant at fixing things
|
|
15
|
+
> AI coding agents are brilliant at fixing things, and just as brilliant at *saying they
|
|
16
16
|
> did* when they didn't. OpenPitStop measures your repo with scans, seals every number so it
|
|
17
|
-
> can't be edited later, attacks your app with a live penetration test, and checks every
|
|
18
|
-
> change your agent makes. The exit codes tell you the truth: `0` clean
|
|
17
|
+
> can't be edited later, attacks your own app with a live penetration test, and checks every
|
|
18
|
+
> change your agent makes. The exit codes tell you the truth: `0` clean, `1` suspicious,
|
|
19
19
|
> `2` confirmed cheat.
|
|
20
20
|
|
|
21
21
|
---
|
|
22
22
|
|
|
23
|
+
## Quick install
|
|
24
|
+
|
|
25
|
+
**Start here:** `npx openpitstop try .` scores any repo in about two seconds with zero
|
|
26
|
+
install, or `npm i -g openpitstop` for daily use. No account, no config, no SaaS.
|
|
27
|
+
|
|
28
|
+
**Zero install, try it now:**
|
|
29
|
+
```bash
|
|
30
|
+
npx openpitstop try .
|
|
31
|
+
```
|
|
32
|
+
Scores any repo in about two seconds of scanning. The first run downloads the
|
|
33
|
+
package once, after that it is instant.
|
|
34
|
+
|
|
35
|
+
**Install it globally (recommended for daily use):**
|
|
36
|
+
```bash
|
|
37
|
+
npm install -g openpitstop
|
|
38
|
+
openpitstop --help
|
|
39
|
+
```
|
|
40
|
+
Now every command starts with `openpitstop` instead of `npx openpitstop`.
|
|
41
|
+
|
|
42
|
+
Requires **Node.js 22+**, that is the only dependency. For the full setup
|
|
43
|
+
(slash command, pre-commit hook, CI), see [Install](#install).
|
|
44
|
+
|
|
45
|
+
## Use-Cases
|
|
46
|
+
|
|
47
|
+
- **You ship with an AI agent and want proof it actually finished.** The gate and
|
|
48
|
+
`verify` turn "the agent says done" into a pass or fail you can block a build on.
|
|
49
|
+
- **You want a security scan that proves findings, not just guesses.** `pen` attacks
|
|
50
|
+
your app in a sandbox and labels each issue PROVEN, indicated, or unproven.
|
|
51
|
+
- **You are tired of agents quietly deleting a failing test.** `integrity` and the
|
|
52
|
+
gate catch focused, deleted, or rewritten tests and exit 2 (confirmed cheat).
|
|
53
|
+
- **You want CI to fail on a regression, not just a new bug.** Drift compares every
|
|
54
|
+
`pen` run to the last sealed one and goes red on a NEW finding.
|
|
55
|
+
- **You need a score you can show your team or an auditor.** `report` and `honesty`
|
|
56
|
+
produce a sealed HTML report and an honesty certificate.
|
|
57
|
+
- **You already pay for a pen tool and want continuous proof for free.** OpenPitStop
|
|
58
|
+
keeps a running ledger of proof in your repo. See the OpenPitStop vs Strix
|
|
59
|
+
comparison above.
|
|
60
|
+
|
|
61
|
+
**Ready to try it?** Run `npx openpitstop try .` on any repo and get your score in
|
|
62
|
+
about two seconds.
|
|
63
|
+
|
|
64
|
+
---
|
|
65
|
+
|
|
23
66
|
## Why I built this
|
|
24
67
|
|
|
25
68
|
I spend my days running coding agents on real repos. They're brilliant at fixing things —
|
|
@@ -40,8 +83,7 @@ your agent.
|
|
|
40
83
|
|
|
41
84
|
| Jump to | |
|
|
42
85
|
|---|---|
|
|
43
|
-
| [Feature tour](#feature-tour) —
|
|
44
|
-
| [See it in 90 seconds](#see-it-in-90-seconds) | [What OpenPitStop actually does](#what-openpitstop-actually-does) · [Every command](#every-command) |
|
|
86
|
+
| [Feature tour](#feature-tour) — every feature, in plain English | [Install](#install) · [Usage](#usage) · [Tool support](#tool-support) |
|
|
45
87
|
| [Architecture](#architecture) | [Known limitations](#known-limitations) · [Contributing](#contributing) · [License](#license) |
|
|
46
88
|
|
|
47
89
|
**Straight to one feature:** [The scan](#the-scan) · [Security fixes](#security-fixes) · [Try it on your repo](#try-it-on-your-repo) · [The test pyramid](#the-test-pyramid) · [The gate](#the-gate) · [Integrity](#integrity) · [The pen test](#the-pen-test) · [Honesty](#honesty) · [Verify](#verify) · [Trends](#trends) · [Inspect](#inspect) · [Repro](#repro) · [Report](#report) · [Share](#share) · [The live shield](#the-live-shield) · [The GitHub Action](#the-github-action) · [The pre-commit hook](#the-pre-commit-hook)
|
|
@@ -79,170 +121,179 @@ loop — and it's the reason to choose the referee over the hacker.
|
|
|
79
121
|
|
|
80
122
|
## Feature tour
|
|
81
123
|
|
|
82
|
-
Every
|
|
83
|
-
|
|
124
|
+
Every feature below is explained in plain English: what it does, and how it
|
|
125
|
+
works. Most of it needs nothing more than a `pitstop scan` first.
|
|
84
126
|
|
|
85
127
|
### The scan
|
|
86
128
|
|
|
87
|
-
`pitstop scan`
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
129
|
+
`pitstop scan` runs every check at once, in parallel, and prints one box with a
|
|
130
|
+
single **OpenPitStop Score** (0 to 100, A to F). It looks at circular imports,
|
|
131
|
+
security issues, duplicated code, test results, build speed, accessibility and
|
|
132
|
+
code quality. Each check gives a real number or prints `skipped` with a hint on
|
|
133
|
+
how to install the tool it needs. It never makes up a number.
|
|
92
134
|
|
|
93
135
|
### Security fixes
|
|
94
136
|
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
<img src="docs/media/pitstop-security.gif" alt="pitstop scan — the indicated security fixes, each with a concrete fix" width="700">
|
|
100
|
-
</p>
|
|
137
|
+
Under the scan box, every security finding ships with a concrete `fix:` line, so
|
|
138
|
+
you get a worklist, not just a list of problems. Findings are labeled
|
|
139
|
+
`[indicated]` with the exact code and the fix. The full matrix is in
|
|
140
|
+
[docs/security.md](docs/security.md).
|
|
101
141
|
|
|
102
142
|
### Try it on your repo
|
|
103
143
|
|
|
104
|
-
`pitstop try .`
|
|
105
|
-
|
|
106
|
-
`npm i -g openpitstop` makes even that instant.)
|
|
107
|
-
|
|
108
|
-
<p align="center">
|
|
109
|
-
<img src="docs/media/pitstop-try.gif" alt="pitstop try — zero-setup score on any repo" width="700">
|
|
110
|
-
</p>
|
|
144
|
+
`pitstop try .` scores any repo in about two seconds, no setup, no config. It is
|
|
145
|
+
the fastest first look, and it seeds a baseline your later runs build on.
|
|
111
146
|
|
|
112
147
|
### The test pyramid
|
|
113
148
|
|
|
114
|
-
`pitstop test`
|
|
115
|
-
suite that
|
|
116
|
-
**DO NOT SHIP
|
|
117
|
-
|
|
118
|
-
<p align="center">
|
|
119
|
-
<img src="docs/media/pitstop-pyramid.gif" alt="pitstop test — the pyramid verdict: DO NOT SHIP on a failing e2e layer" width="700">
|
|
120
|
-
</p>
|
|
149
|
+
`pitstop test` runs your **unit, integration and e2e** layers separately, so a
|
|
150
|
+
suite that passes cannot hide a missing layer. It names every failing test. One
|
|
151
|
+
failing layer and it prints **DO NOT SHIP** and exits 1.
|
|
121
152
|
|
|
122
153
|
### The gate
|
|
123
154
|
|
|
124
|
-
`pitstop gate`
|
|
125
|
-
clean
|
|
126
|
-
|
|
127
|
-
<p align="center">
|
|
128
|
-
<img src="docs/media/pitstop-gate.gif" alt="pitstop gate — GATE FAIL on a confirmed cheat, exit 2" width="700">
|
|
129
|
-
</p>
|
|
155
|
+
`pitstop gate` is the contract for CI and pre-commit hooks. It checks the score,
|
|
156
|
+
regression risk and diff integrity, then exits `0` (clean), `1` (issues) or `2`
|
|
157
|
+
(confirmed cheat). The exit code is the truth a build can block on.
|
|
130
158
|
|
|
131
159
|
### Integrity
|
|
132
160
|
|
|
133
|
-
`pitstop integrity`
|
|
134
|
-
patterns: focused tests, deleted tests, rewritten tests
|
|
135
|
-
|
|
136
|
-
<p align="center">
|
|
137
|
-
<img src="docs/media/pitstop-integrity.gif" alt="pitstop integrity — CONFIRMED_CHEAT: test file deleted" width="700">
|
|
138
|
-
</p>
|
|
161
|
+
`pitstop integrity` diffs your change against the sealed baseline and hunts cheat
|
|
162
|
+
patterns: focused tests, deleted tests, rewritten tests, swallowed errors,
|
|
163
|
+
hardcoded-to-pass values. It exits `0/1/2` the same way.
|
|
139
164
|
|
|
140
165
|
### The pen test
|
|
141
166
|
|
|
142
|
-
`pitstop pen`
|
|
143
|
-
|
|
167
|
+
`pitstop pen` boots your app in a sandbox and fires real attack traffic, so a
|
|
168
|
+
finding is **PROVEN** by a live attack, not just guessed. With `--fix` it writes a
|
|
169
|
+
failing-first repro test and a safe patch. Nothing reaches the real network.
|
|
144
170
|
|
|
145
|
-
|
|
146
|
-
<img src="docs/media/pitstop-pen.gif" alt="pitstop pen — sandboxed attacks with runtime-proof verdicts" width="700">
|
|
147
|
-
</p>
|
|
171
|
+
### Drift (the permanent referee)
|
|
148
172
|
|
|
149
|
-
|
|
173
|
+
`pitstop pen` remembers. Every run seals its verdicts and compares them to the last one, so you
|
|
174
|
+
see exactly what changed between today and last week:
|
|
150
175
|
|
|
151
|
-
|
|
176
|
+
- **NEW** — a finding appeared (or escalated from indicated to proven). This is a regression, so the
|
|
177
|
+
gate exits `1` and your CI goes red.
|
|
178
|
+
- **RESOLVED** — a finding is gone because the fix worked. This is the "prove my fix" loop, and it is
|
|
179
|
+
the most satisfying thing here: run `pitstop repro <id>` to turn a finding into a failing test, ship
|
|
180
|
+
the patch, run `pitstop pen` again, and watch it flip to resolved.
|
|
181
|
+
- **ESCALATIONS** — something that was only *indicated* by static analysis is now *proven* by a live
|
|
182
|
+
attack.
|
|
152
183
|
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
184
|
+
Strix, the enterprise tool, runs a one-off scan. OpenPitStop keeps a running ledger of proof, so a
|
|
185
|
+
fix can never silently rot back into a bug.
|
|
186
|
+
|
|
187
|
+
### Honesty
|
|
188
|
+
|
|
189
|
+
`pitstop honesty` prints an honest self-assessment of what the tool cannot do, with
|
|
190
|
+
the evidence chain behind every number. No SaaS, no telemetry, no dashboard, no
|
|
191
|
+
fixing your code: it tells you its limits in plain words.
|
|
156
192
|
|
|
157
193
|
### Verify
|
|
158
194
|
|
|
159
|
-
`pitstop verify`
|
|
195
|
+
`pitstop verify` re-scans after a change and shows exactly how the score moved, and
|
|
196
|
+
it checks your diff for cheat patterns. The numbers cannot be argued with.
|
|
160
197
|
|
|
161
198
|
### Trends
|
|
162
199
|
|
|
163
|
-
`pitstop trends`
|
|
200
|
+
`pitstop trends` turns your saved scan history into per-category sparklines and a
|
|
201
|
+
score trend, so you can watch a repo actually improve over time.
|
|
164
202
|
|
|
165
203
|
### Inspect
|
|
166
204
|
|
|
167
|
-
`pitstop inspect <finding-id>`
|
|
205
|
+
`pitstop inspect <finding-id>` opens one finding: the code snippet, the root
|
|
206
|
+
cause, whether a repro test exists, and what OpenPitStop remembers about these
|
|
207
|
+
files.
|
|
168
208
|
|
|
169
209
|
### Repro
|
|
170
210
|
|
|
171
|
-
`pitstop repro <finding-id>`
|
|
211
|
+
`pitstop repro <finding-id>` turns any finding into a regression test that FAILS
|
|
212
|
+
while the bug is live and must PASS after the fix. Proof first, fix second.
|
|
172
213
|
|
|
173
214
|
### Report
|
|
174
215
|
|
|
175
|
-
`pitstop report --html`
|
|
216
|
+
`pitstop report --html` writes one self-contained HTML report, sealed with an
|
|
217
|
+
evidence signature, plus a README-ready score badge (`PITSTOP_BADGE.svg`).
|
|
176
218
|
|
|
177
219
|
### Share
|
|
178
220
|
|
|
179
|
-
`pitstop share`
|
|
221
|
+
`pitstop share` renders a single share card (score, trend, top findings) you can
|
|
222
|
+
screenshot and post, or paste into a PR.
|
|
180
223
|
|
|
181
224
|
### The live shield
|
|
182
225
|
|
|
183
|
-
`pitstop watch`
|
|
226
|
+
`pitstop watch` sits in a terminal and re-scans the moment you save a file, printing
|
|
227
|
+
the score delta so you see problems as you type.
|
|
184
228
|
|
|
185
|
-
###
|
|
229
|
+
### Drive the agent
|
|
186
230
|
|
|
187
|
-
`
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
](docs/github-action.md)
|
|
231
|
+
`pitstop drive <finding-id>` hands one finding to your own agent with explicit orders:
|
|
232
|
+
write the failing repro first, fix it, make the repro pass, then verify.
|
|
233
|
+
OpenPitStop referees the result and never edits your code.
|
|
191
234
|
|
|
192
|
-
### The
|
|
235
|
+
### The next step
|
|
193
236
|
|
|
194
|
-
`
|
|
195
|
-
|
|
196
|
-
commits → ](docs/caught-in-the-wild.md#bonus-the-same-catches-as-a-pre-commit-hook)
|
|
237
|
+
`pitstop next` reads the sealed artifacts and prints the single best next command plus
|
|
238
|
+
a checklist of everything still open, so you always know where you are.
|
|
197
239
|
|
|
198
|
-
|
|
240
|
+
### Ask in plain English
|
|
199
241
|
|
|
200
|
-
|
|
242
|
+
`pitstop ask "make this safe"` (or `/pitstop make this safe`) maps a plain-English
|
|
243
|
+
request to the right command. No need to memorize flags.
|
|
201
244
|
|
|
202
|
-
|
|
245
|
+
### Autopilot fix
|
|
203
246
|
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
```
|
|
247
|
+
`pitstop fix` chains **scan to pen --fix to verify to gate** and shows the `next` card
|
|
248
|
+
after each hop, so a clean repo is reachable without touching the agent.
|
|
207
249
|
|
|
208
|
-
|
|
209
|
-
failing suite green without fixing the bug — and the gate catches both attempts:
|
|
250
|
+
### Memory and budget
|
|
210
251
|
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
```
|
|
252
|
+
`pitstop memory` is a repo scratchpad for decisions and rejected approaches that
|
|
253
|
+
survive across sessions. `pitstop budget` shows the token and compute bill of your
|
|
254
|
+
scans and reproves, so a fix loop stays cheap.
|
|
215
255
|
|
|
216
|
-
|
|
217
|
-
of the registry. For a tight re-record, `node scripts/cheat-demo.cjs --fast --no-pitch`
|
|
218
|
-
reuses the cached `node_modules` (skips `npm install`) and ends the arc on the
|
|
219
|
-
CONFIRMED_CHEAT box — no pitch, no dead air.
|
|
256
|
+
### The slash command
|
|
220
257
|
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
ACT 3 agent deletes the test → GATE: CONFIRMED_CHEAT (exit 2) — blocked
|
|
225
|
-
(tamper-evident evidence chain verifies the whole way)
|
|
226
|
-
```
|
|
258
|
+
`/pitstop` in Claude Code, Cursor, OpenCode, Codex and more runs the full loop
|
|
259
|
+
immediately. `pitstop install` writes it into your tools; `pitstop prompt` shows the
|
|
260
|
+
exact prompt it expands to. See [Install](#install).
|
|
227
261
|
|
|
228
|
-
|
|
229
|
-
<img src="docs/media/pitstop-gate.gif" alt="The cheat-catch in the real TUI: the gate reads the diff against the sealed baseline and blocks a CONFIRMED_CHEAT with exit 2 — on real opencode output." width="780">
|
|
230
|
-
</p>
|
|
262
|
+
### The GitHub Action
|
|
231
263
|
|
|
232
|
-
|
|
233
|
-
|
|
264
|
+
`uses: openpitstop/action` (or `Krish-1507/OpenPitStop@main`) puts the gate on every PR
|
|
265
|
+
as a comment and a failing check when it matters. No wiring by hand. See
|
|
266
|
+
[docs/github-action.md](docs/github-action.md).
|
|
234
267
|
|
|
235
|
-
|
|
268
|
+
### The pre-commit hook
|
|
236
269
|
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
270
|
+
`npx openpitstop install --hooks` installs the gate one step earlier: the commit
|
|
271
|
+
cannot land until the gate passes. See
|
|
272
|
+
[docs/caught-in-the-wild.md](docs/caught-in-the-wild.md).
|
|
273
|
+
|
|
274
|
+
### Ledger mode (payment proof)
|
|
275
|
+
|
|
276
|
+
`pitstop scan --ledger` boots your app with every outbound HTTP call rerouted to a mock
|
|
277
|
+
gateway, then replays the classic payment bugs (duplicate webhook, concurrent
|
|
278
|
+
double-submit, delayed retry). If the mock shows more than one charge per idempotency
|
|
279
|
+
key, that is a **proven double-charge**, not a guess.
|
|
280
|
+
|
|
281
|
+
### CI reports
|
|
282
|
+
|
|
283
|
+
`pitstop ci` runs a CI-friendly scan plus verify against the base branch and writes a
|
|
284
|
+
PR-ready markdown report, the gate as a PR comment. This is the engine behind the
|
|
285
|
+
GitHub Action.
|
|
286
|
+
|
|
287
|
+
### Ready-check and doctor
|
|
240
288
|
|
|
241
|
-
|
|
242
|
-
|
|
289
|
+
`pitstop ready-check` answers "is it worth scanning again?" and reuses the baseline when
|
|
290
|
+
nothing changed. `pitstop doctor` explains why a category shows `skipped` and prints
|
|
291
|
+
copy-paste install hints for the tools you are missing.
|
|
243
292
|
|
|
244
|
-
|
|
245
|
-
|
|
293
|
+
### Digest (progress story)
|
|
294
|
+
|
|
295
|
+
`pitstop digest` turns your history into a plain-English progress story: how the score
|
|
296
|
+
moved, what got fixed, what regressed, and every cheat it caught.
|
|
246
297
|
|
|
247
298
|
---
|
|
248
299
|
|
|
@@ -255,9 +306,9 @@ npx openpitstop
|
|
|
255
306
|
```
|
|
256
307
|
|
|
257
308
|
No arguments needed: the CLI detects your AI tools, and asks what you want —
|
|
258
|
-
install `/pitstop` into them, score *this* repo (`try .`),
|
|
259
|
-
|
|
260
|
-
|
|
309
|
+
install `/pitstop` into them, or score *this* repo (`try .`). Pick, and it does it.
|
|
310
|
+
(In a non-interactive terminal it skips the questions and prints the one-line menu
|
|
311
|
+
instead.)
|
|
261
312
|
|
|
262
313
|
Or go straight to the files:
|
|
263
314
|
|
|
@@ -318,7 +369,6 @@ that:
|
|
|
318
369
|
| `/pitstop` (bare) | **default full loop** | Scans right away, prints the boxed report, one confirmation pause, then the autonomous fix loop — repeat until clean. |
|
|
319
370
|
| `/pitstop --menu` | menu | Prints the full mode list below and **waits** — handy if you forgot the flags. |
|
|
320
371
|
| `/pitstop --scan-only` | scan-only | Runs `openpitstop scan`, prints the entire boxed report verbatim, and stops — no fixes, no commentary. |
|
|
321
|
-
| `/pitstop --demo` | demo | Scaffolds OpenPitStop's seeded broken demo repo into a temp dir, then runs the default full loop there. |
|
|
322
372
|
| `/pitstop --ledger` | ledger | Runs `openpitstop scan --ledger` (boots the app with every outbound HTTP call intercepted and replays duplicate-webhook / double-submit / retry traffic), then runs the loop restricted to the payment findings. |
|
|
323
373
|
| `/pitstop --integrity-only` | integrity-only | Runs `openpitstop integrity`, prints the boxed verdict verbatim, and stops — no scanning, no fixes. |
|
|
324
374
|
| `/pitstop --pen` | pen | Live penetration test with proof — see [The pen test](#the-pen-test). |
|
|
@@ -329,9 +379,8 @@ For reference, `/pitstop --menu` shows this list:
|
|
|
329
379
|
```
|
|
330
380
|
OpenPitStop modes:
|
|
331
381
|
(enter) — full autonomous loop (scan, confirm, fix, verify, repeat)
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
--ledger — payment idempotency fuzzing only
|
|
382
|
+
--scan-only — scan and report, no fixes
|
|
383
|
+
--ledger — payment idempotency fuzzing only
|
|
335
384
|
--integrity-only — re-check the last commit for cheat patterns, no scanning
|
|
336
385
|
--pen — penetration test: live attacks + proof + fixes (regression tests, patches)
|
|
337
386
|
(your own ask) — reply with anything else, e.g. "check the security of this app"
|
|
@@ -341,28 +390,34 @@ A flag after `/pitstop` picks a specific mode; any free-form text after it becom
|
|
|
341
390
|
custom ask; bare `/pitstop` is the full loop. If a tool ever fails to substitute arguments,
|
|
342
391
|
`/pitstop` behaves as bare — the default full loop — rather than guessing.
|
|
343
392
|
|
|
344
|
-
No repo handy? `npx openpitstop@latest demo` scaffolds a broken demo repo in a temp dir so
|
|
345
|
-
you can watch the whole loop — self-contained, no installs on the hot path, and it never
|
|
346
|
-
writes into your tool configs (that stays an explicit `pitstop install`).
|
|
347
|
-
|
|
348
393
|
## Tool support
|
|
349
394
|
|
|
350
395
|
| Tool | Installed to | Status |
|
|
351
396
|
|------|--------------|--------|
|
|
352
|
-
| Claude Code | `.claude/commands/pitstop.md` (project + user), plus a Skill at `.claude/skills/pitstop/SKILL.md
|
|
397
|
+
| Claude Code | `.claude/commands/pitstop.md` (project + user), plus a Skill at `.claude/skills/pitstop/SKILL.md | Full support |
|
|
353
398
|
| Cursor | `.cursor/commands/pitstop.md` (project + user) | Full support |
|
|
354
399
|
| OpenCode | `.opencode/commands/pitstop.md` (project), `~/.config/opencode/commands/` (user) | Full support |
|
|
355
400
|
| Kilo Code | `.kilo/commands/pitstop.md` (project), `~/.config/kilo/commands/` (user) | Full support |
|
|
356
401
|
| Antigravity | `.agent/workflows/pitstop.md` (project + user) | Full support |
|
|
357
402
|
| Gemini CLI | `.gemini/commands/pitstop.toml` (project + user) | Full support |
|
|
358
403
|
| Codex CLI | `~/.codex/prompts/pitstop.md` | Full support |
|
|
404
|
+
| FreeBuff CLI | portable `pitstop.md` in your tool's commands folder | Full support (portable) |
|
|
405
|
+
| Grok Build CLI | portable `pitstop.md` in your tool's commands folder | Full support (portable) |
|
|
406
|
+
| MUSE Code CLI | portable `pitstop.md` in your tool's commands folder | Full support (portable) |
|
|
407
|
+
| Any other agent CLI | portable `pitstop.md` (drop it in the commands folder) | Full support (portable) — see below |
|
|
359
408
|
| Codex App / VS Code extension | — (no file written) | **Not supported** — OpenAI hasn't shipped custom slash commands there; install prints a manual-copy note instead |
|
|
360
409
|
| GitHub Action (PRs) | `uses: Krish-1507/OpenPitStop@main` | **Full support** — gate verdict as a PR comment + failing check; see [docs/github-action.md](docs/github-action.md) |
|
|
361
410
|
| git pre-commit hook | `.git/hooks/pre-commit` (installed with `--hooks`) | **Full support** — the gate blocks the commit before it lands |
|
|
362
411
|
|
|
363
|
-
|
|
364
|
-
|
|
365
|
-
|
|
412
|
+
**Works with any agent CLI.** OpenPitStop's `/pitstop` is a portable command file:
|
|
413
|
+
run `pitstop prompt` to print the exact instruction text, then paste it as a custom
|
|
414
|
+
slash command in any coding CLI that supports them (FreeBuff, Grok Build, MUSE Code and
|
|
415
|
+
others included above). The CLIs listed by name also get a dedicated path written
|
|
416
|
+
automatically by `pitstop install` when their commands-folder convention is known. Tell
|
|
417
|
+
us your CLI and we'll add it to the auto-install list. Legacy/alternate locations are
|
|
418
|
+
also written where tool docs are inconsistent across versions (see
|
|
419
|
+
`src/installer/targets.ts`). Existing files are never overwritten unless you pass
|
|
420
|
+
`-y`/`--force`; `npx openpitstop install --uninstall` removes everything.
|
|
366
421
|
|
|
367
422
|
## What OpenPitStop actually does
|
|
368
423
|
|
|
@@ -489,10 +544,9 @@ one-shot.
|
|
|
489
544
|
| Command | What it does |
|
|
490
545
|
|---|---|
|
|
491
546
|
| `pitstop install` / `install --uninstall` | Writes `/pitstop` into every supported tool (project + user level). `--uninstall` removes it all. `--hooks` also installs (or with `--uninstall`, removes) the git pre-commit gate. |
|
|
492
|
-
| `pitstop` (no args) | The guided first-run: detects your AI tools and git repo, then offers to install
|
|
547
|
+
| `pitstop` (no args) | The guided first-run: detects your AI tools and git repo, then offers to install or score this repo (`try .`). Non-TTY prints the one-line menu instead. |
|
|
493
548
|
| `pitstop doctor` | Explains why categories show `skipped`: checks your toolchain (Node, git, jscpd, gitleaks, semgrep, pa11y) and prints copy-paste install hints. |
|
|
494
549
|
| `pitstop prompt [--args …]` | Prints the exact prompt your AI tool expands `/pitstop` into, with your arguments filled in — full transparency into what the agent was told. |
|
|
495
|
-
| `pitstop demo [demo]` | Scaffolds an intentionally-broken demo repo into a fresh temp dir (`demo-repo`, `demo-repo-integrity`, `demo-repo-fintech`, `demo-repo-generators`), initializes git, and scans it immediately. |
|
|
496
550
|
|
|
497
551
|
### The score & badge
|
|
498
552
|
|
|
@@ -538,8 +592,7 @@ shows zero clusters (hard limits: 10 fix rounds or 45 minutes), ending with a
|
|
|
538
592
|
gateway**, then replays the three classic payment bugs: duplicate webhook, concurrent
|
|
539
593
|
double-submit, delayed retry. If the mock gateway's own receipt log shows more than one
|
|
540
594
|
charge per idempotency key, that's a **proven double-charge** — not a guess. The shipped
|
|
541
|
-
|
|
542
|
-
webhook endpoints have no idempotency guard. If the sandbox can't intercept some traffic,
|
|
595
|
+
If the sandbox can't intercept some traffic,
|
|
543
596
|
the run aborts (`exit 77`); nothing ever reaches a real gateway.
|
|
544
597
|
|
|
545
598
|
**Which stacks are covered?** Node/JS apps run under the nock preload, which intercepts
|
|
@@ -563,14 +616,6 @@ value edited to match the buggy output. A caught cheat looks like this: change
|
|
|
563
616
|
`CONFIRMED_CHEAT`, the change is blocked, and a human reviews it (verified against
|
|
564
617
|
`fixtures/assertion-literal-tamper/`). An honest app-side fix sails through `CLEAN`.
|
|
565
618
|
|
|
566
|
-
### Cheat-catch demo
|
|
567
|
-
|
|
568
|
-
Want to *see* it? The scripted arc from **[See it in 90 seconds](#see-it-in-90-seconds)**
|
|
569
|
-
is `scripts/cheat-demo.cjs` — a fully deterministic SUSPICIOUS → CONFIRMED_CHEAT
|
|
570
|
-
sequence against a real repo with a real failing jest test. Point it at a build with
|
|
571
|
-
`PITSTOP_CLI="node /path/to/dist/cli.js"`, or let it use `npx openpitstop`. Great for a
|
|
572
|
-
video or a live judge's demo.
|
|
573
|
-
|
|
574
619
|
## Architecture
|
|
575
620
|
|
|
576
621
|
OpenPitStop is two pieces that never mix: a **CLI that measures**, and **your host agent that
|
|
@@ -582,14 +627,12 @@ cheat its own referee. That separation is the product.
|
|
|
582
627
|
|
|
583
628
|
## Known limitations
|
|
584
629
|
|
|
585
|
-
- **Windows** is CI-verified
|
|
586
|
-
`windows-latest`
|
|
587
|
-
Windows host
|
|
588
|
-
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
caveat is breadth, not correctness: not every exotic repo shape has been manually
|
|
592
|
-
exercised on Windows.
|
|
630
|
+
- **Windows** is a first-class, CI-verified platform (build + smoke on `ubuntu-latest`
|
|
631
|
+
and `windows-latest` every push). `watch`, `pen`, `pen --fix` and `scan --ledger` were
|
|
632
|
+
each run end-to-end on a real Windows host against the demo repos, with a live watch
|
|
633
|
+
delta, PROVEN ledger double-charges (sealed evidence) and honest pen verdicts
|
|
634
|
+
(including the honest "0 patches" case) all verified. The only open caveat is breadth,
|
|
635
|
+
not correctness: not every exotic repo shape has been hand-exercised on Windows yet.
|
|
593
636
|
- **Codex App / VS Code extension** isn't supported and won't be until OpenAI ships custom
|
|
594
637
|
slash commands; use Codex CLI for `/pitstop`.
|
|
595
638
|
- **Graceful degradation:** duplication (`jscpd`), secrets/code scanning (`gitleaks`,
|
|
@@ -640,6 +683,17 @@ how to open a PR. For the launch notes and the "why", read [LAUNCH.md](LAUNCH.md
|
|
|
640
683
|
|
|
641
684
|
[MIT](LICENSE)
|
|
642
685
|
|
|
686
|
+
## Support the project
|
|
687
|
+
|
|
688
|
+
If OpenPitStop saved you from shipping a bug your agent swore was fixed, the best
|
|
689
|
+
support is a star and a real repo:
|
|
690
|
+
|
|
691
|
+
- **Star** it: https://github.com/Krish-1507/OpenPitStop
|
|
692
|
+
- **Report issues or ideas**: https://github.com/Krish-1507/OpenPitStop/issues
|
|
693
|
+
- **Contribute** an analyzer (small, well-scoped): see [CONTRIBUTING.md](CONTRIBUTING.md)
|
|
694
|
+
|
|
695
|
+
No donation, no paywall, no telemetry.
|
|
696
|
+
|
|
643
697
|
---
|
|
644
698
|
|
|
645
699
|
<p align="center">
|
package/demo/DEMO_SCRIPT.md
CHANGED
|
@@ -93,6 +93,9 @@ A sampling of the 31 symptoms PitStop flags in MiniShop:
|
|
|
93
93
|
| database | superuser account, `GRANT ALL`, TLS disabled | `src/db.js` |
|
|
94
94
|
| authentication | `password === stored`, `Math.random()` token, `alg:none` JWT, insecure cookie | `src/auth.js` |
|
|
95
95
|
| hidden-vuln | security `TODO`/`FIXME` comments | several files |
|
|
96
|
+
| price-tampering | checkout totals from CLIENT-supplied prices | `src/server.js` |
|
|
97
|
+
| idor | `/api/orders/:id` returns any order, no ownership check | `src/server.js` |
|
|
98
|
+
| prototype-pollution | `lodash.merge` sinks an untrusted request body | `src/server.js` |
|
|
96
99
|
|
|
97
100
|
The scan also **seals a baseline** — the signed "before" state every later
|
|
98
101
|
check is verified against.
|
|
@@ -169,6 +172,38 @@ to your agent. The agent produces exactly the changes in
|
|
|
169
172
|
|
|
170
173
|
---
|
|
171
174
|
|
|
175
|
+
## 4b. Drift — prove the fix stuck (the permanent referee) — `pitstop pen`
|
|
176
|
+
|
|
177
|
+
Run `pen` again, this time on the *fixed* app:
|
|
178
|
+
|
|
179
|
+
```bash
|
|
180
|
+
cd demo/web-app-fixed
|
|
181
|
+
pitstop pen
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
This run is compared against the last sealed pen run from `web-app-broken`. You
|
|
185
|
+
see the delta, not just a fresh score:
|
|
186
|
+
|
|
187
|
+
```
|
|
188
|
+
PEN DRIFT vs .pitstop/pen-latest.json
|
|
189
|
+
◐ RESOLVED pen-… : command-injection (was PROVEN, now gone)
|
|
190
|
+
◐ RESOLVED pen-… : sql-injection
|
|
191
|
+
(+ any still-open classes, and ESCALATIONS if a static
|
|
192
|
+
indication became a live proof)
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
**Why this matters on camera:** most scanners give you a one-off report and walk
|
|
196
|
+
away. OpenPitStop keeps a running ledger of proof. A finding that flips from
|
|
197
|
+
PROVEN to RESOLVED is evidence the fix worked, not hope. And if someone
|
|
198
|
+
reintroduces the bug later, the next `pen` run prints `+1 NEW` and the gate
|
|
199
|
+
exits `1`, so the regression can't sneak into main.
|
|
200
|
+
|
|
201
|
+
**On-camera:** "Most scanners hand you a PDF and leave. OpenPitStop remembers.
|
|
202
|
+
Watch this finding flip from proven to resolved, and if it ever comes back, the
|
|
203
|
+
gate goes red. That's a referee, not a report."
|
|
204
|
+
|
|
205
|
+
---
|
|
206
|
+
|
|
172
207
|
## 5. Verify it didn't fake it — `pitstop verify`
|
|
173
208
|
|
|
174
209
|
```bash
|
|
@@ -266,3 +301,49 @@ straight to a finding's file:line + fix.
|
|
|
266
301
|
> tamper-evident baselines, runtime penetration tests with failing-first
|
|
267
302
|
> regression contracts, and a verification gate that catches the agent lying.
|
|
268
303
|
> The CLI measures; the agent edits; the numbers can't be cheated.
|
|
304
|
+
|
|
305
|
+
---
|
|
306
|
+
|
|
307
|
+
## 11. Fix it yourself — the full `pitstop` command sequence (cheat sheet)
|
|
308
|
+
|
|
309
|
+
Everything you type to take `web-app-broken` from **35/100 (F)** to
|
|
310
|
+
**VERIFIED**, end to end. Copy-paste top to bottom.
|
|
311
|
+
|
|
312
|
+
```bash
|
|
313
|
+
# 0. baseline — confirm it's broken
|
|
314
|
+
cd demo/web-app-broken
|
|
315
|
+
npm install
|
|
316
|
+
npm test # 5 failing — the app is genuinely broken
|
|
317
|
+
npx openpitstop install -y # register the /pitstop command (one time)
|
|
318
|
+
|
|
319
|
+
# 1. honest baseline — seals the signed "before" state every later check uses
|
|
320
|
+
pitstop scan
|
|
321
|
+
|
|
322
|
+
# 2. live pen test + safe patches + failing-first repro tests
|
|
323
|
+
pitstop pen --fix # writes repro tests + deterministic patches
|
|
324
|
+
pitstop repro SEC-001 # watch a real bug FAIL on camera
|
|
325
|
+
|
|
326
|
+
# 3a. drive the agent to fix each root cause, one cluster at a time
|
|
327
|
+
pitstop next # prints the single best next id + the open checklist
|
|
328
|
+
pitstop drive <id> # fix one root cause; repeat until the checklist clears
|
|
329
|
+
|
|
330
|
+
# 3b. ...or skip the manual loop and let autopilot run scan -> pen -> verify -> gate
|
|
331
|
+
pitstop fix
|
|
332
|
+
|
|
333
|
+
# 4. prove the fix stuck (the permanent referee)
|
|
334
|
+
pitstop verify # exit 0 = VERIFIED; agent-cheat patterns caught
|
|
335
|
+
pitstop pen # drift: RESOLVED findings vs the broken baseline
|
|
336
|
+
|
|
337
|
+
# 5. gate + shareable proof
|
|
338
|
+
pitstop gate --score 60 # 1 before the fix, 0 after
|
|
339
|
+
pitstop honesty # trace every number to a sealed .pitstop/ file
|
|
340
|
+
pitstop report # shareable HTML/markdown card
|
|
341
|
+
pitstop ci # drop the same gate into CI
|
|
342
|
+
|
|
343
|
+
# 6. ship only clean code
|
|
344
|
+
git add -A && git commit -m "fix: remediate MiniShop with OpenPitStop"
|
|
345
|
+
```
|
|
346
|
+
|
|
347
|
+
The fixed result is already committed for reference in
|
|
348
|
+
[`web-app-fixed/`](./web-app-fixed) — `cd demo/web-app-fixed && DB_PASSWORD=dummy
|
|
349
|
+
JWT_SECRET=dummy npm test` shows all 5 passing.
|