openpitstop 1.5.1 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +196 -140
- package/demo/DEMO_SCRIPT.md +81 -0
- package/demo/web-app-broken/src/server.js +50 -0
- package/demo/web-app-fixed/src/server.js +61 -0
- package/dist/commands/drive.js +365 -104
- package/dist/commands/drive.js.map +1 -1
- package/dist/commands/next.d.ts +25 -0
- package/dist/commands/next.js +403 -72
- package/dist/commands/next.js.map +1 -1
- package/dist/commands/pen.js +29 -1
- package/dist/commands/pen.js.map +1 -1
- package/dist/graph/correlate.js +11 -1
- package/dist/graph/correlate.js.map +1 -1
- package/dist/pen/drift.d.ts +16 -0
- package/dist/pen/drift.js +69 -0
- package/dist/pen/drift.js.map +1 -0
- package/dist/pen/report.js +90 -2
- package/dist/pen/report.js.map +1 -1
- package/dist/pen/types.d.ts +32 -0
- package/dist/pen/types.js.map +1 -1
- package/package.json +1 -1
- package/templates/pitstop-menu.command.md +26 -21
- package/templates/pitstop.command.md +25 -30
- package/templates/pitstop.prompt.md +18 -0
package/README.md
CHANGED
|
@@ -12,14 +12,57 @@ know it's actually done.
|
|
|
12
12
|
[](https://github.com/Krish-1507/OpenPitStop/actions/workflows/ci.yml)
|
|
13
13
|
[](LICENSE)
|
|
14
14
|
|
|
15
|
-
> AI coding agents are brilliant at fixing things
|
|
15
|
+
> AI coding agents are brilliant at fixing things, and just as brilliant at *saying they
|
|
16
16
|
> did* when they didn't. OpenPitStop measures your repo with scans, seals every number so it
|
|
17
|
-
> can't be edited later, attacks your app with a live penetration test, and checks every
|
|
18
|
-
> change your agent makes. The exit codes tell you the truth: `0` clean
|
|
17
|
+
> can't be edited later, attacks your own app with a live penetration test, and checks every
|
|
18
|
+
> change your agent makes. The exit codes tell you the truth: `0` clean, `1` suspicious,
|
|
19
19
|
> `2` confirmed cheat.
|
|
20
20
|
|
|
21
21
|
---
|
|
22
22
|
|
|
23
|
+
## Quick install
|
|
24
|
+
|
|
25
|
+
**Start here:** `npx openpitstop try .` scores any repo in about two seconds with zero
|
|
26
|
+
install, or `npm i -g openpitstop` for daily use. No account, no config, no SaaS.
|
|
27
|
+
|
|
28
|
+
**Zero install, try it now:**
|
|
29
|
+
```bash
|
|
30
|
+
npx openpitstop try .
|
|
31
|
+
```
|
|
32
|
+
Scores any repo in about two seconds of scanning. The first run downloads the
|
|
33
|
+
package once, after that it is instant.
|
|
34
|
+
|
|
35
|
+
**Install it globally (recommended for daily use):**
|
|
36
|
+
```bash
|
|
37
|
+
npm install -g openpitstop
|
|
38
|
+
openpitstop --help
|
|
39
|
+
```
|
|
40
|
+
Now every command starts with `openpitstop` instead of `npx openpitstop`.
|
|
41
|
+
|
|
42
|
+
Requires **Node.js 22+**, that is the only dependency. For the full setup
|
|
43
|
+
(slash command, pre-commit hook, CI), see [Install](#install).
|
|
44
|
+
|
|
45
|
+
## Use-Cases
|
|
46
|
+
|
|
47
|
+
- **You ship with an AI agent and want proof it actually finished.** The gate and
|
|
48
|
+
`verify` turn "the agent says done" into a pass or fail you can block a build on.
|
|
49
|
+
- **You want a security scan that proves findings, not just guesses.** `pen` attacks
|
|
50
|
+
your app in a sandbox and labels each issue PROVEN, indicated, or unproven.
|
|
51
|
+
- **You are tired of agents quietly deleting a failing test.** `integrity` and the
|
|
52
|
+
gate catch focused, deleted, or rewritten tests and exit 2 (confirmed cheat).
|
|
53
|
+
- **You want CI to fail on a regression, not just a new bug.** Drift compares every
|
|
54
|
+
`pen` run to the last sealed one and goes red on a NEW finding.
|
|
55
|
+
- **You need a score you can show your team or an auditor.** `report` and `honesty`
|
|
56
|
+
produce a sealed HTML report and an honesty certificate.
|
|
57
|
+
- **You already pay for a pen tool and want continuous proof for free.** OpenPitStop
|
|
58
|
+
keeps a running ledger of proof in your repo. See the OpenPitStop vs Strix
|
|
59
|
+
comparison above.
|
|
60
|
+
|
|
61
|
+
**Ready to try it?** Run `npx openpitstop try .` on any repo and get your score in
|
|
62
|
+
about two seconds.
|
|
63
|
+
|
|
64
|
+
---
|
|
65
|
+
|
|
23
66
|
## Why I built this
|
|
24
67
|
|
|
25
68
|
I spend my days running coding agents on real repos. They're brilliant at fixing things —
|
|
@@ -40,8 +83,7 @@ your agent.
|
|
|
40
83
|
|
|
41
84
|
| Jump to | |
|
|
42
85
|
|---|---|
|
|
43
|
-
| [Feature tour](#feature-tour) —
|
|
44
|
-
| [See it in 90 seconds](#see-it-in-90-seconds) | [What OpenPitStop actually does](#what-openpitstop-actually-does) · [Every command](#every-command) |
|
|
86
|
+
| [Feature tour](#feature-tour) — every feature, in plain English | [Install](#install) · [Usage](#usage) · [Tool support](#tool-support) |
|
|
45
87
|
| [Architecture](#architecture) | [Known limitations](#known-limitations) · [Contributing](#contributing) · [License](#license) |
|
|
46
88
|
|
|
47
89
|
**Straight to one feature:** [The scan](#the-scan) · [Security fixes](#security-fixes) · [Try it on your repo](#try-it-on-your-repo) · [The test pyramid](#the-test-pyramid) · [The gate](#the-gate) · [Integrity](#integrity) · [The pen test](#the-pen-test) · [Honesty](#honesty) · [Verify](#verify) · [Trends](#trends) · [Inspect](#inspect) · [Repro](#repro) · [Report](#report) · [Share](#share) · [The live shield](#the-live-shield) · [The GitHub Action](#the-github-action) · [The pre-commit hook](#the-pre-commit-hook)
|
|
@@ -66,6 +108,8 @@ difference is what happens after a vulnerability is found:
|
|
|
66
108
|
| Secret exfiltration | not emphasized | **Ledger mode** proves the app doesn't phone home with your keys |
|
|
67
109
|
| DevSecOps | Cloud platform (paid tiers) | **Free SARIF → GitHub Security tab** + one-number `pitstop gate` |
|
|
68
110
|
| Proof coverage | — | **`PITSTOP_PROOF` badge**: % of findings that ship a permanent repro test |
|
|
111
|
+
| Continuous proof (drift) | report only — re-run and hope | **Drift gate**: every `pitstop pen` compares to the last sealed run, *proves a fix* (finding gone) and *fails the CI gate* on a new high/critical regression or a hypothesis the live attack just confirmed |
|
|
112
|
+
| Prove-my-fix loop | manual | **`pitstop repro <id>`** re-runs the exact attack and asserts the safe outcome — a PASS means the fix is real, a deleted repro test is flagged as a cheat |
|
|
69
113
|
| Bug classes covered | strong general set | **40+ classes** — race/TOCTOU, IDOR/BOLA, price-tampering, XXE, insecure deserialization, JWT alg-confusion/weak-secret, SSRF, SQL/NoSQLi, command-injection, path traversal, XSS, secrets, CORS, missing headers, rate-limit, and more |
|
|
70
114
|
|
|
71
115
|
The honest pitch: a pen-test that only reports is a list of things to argue about. A
|
|
@@ -77,170 +121,179 @@ loop — and it's the reason to choose the referee over the hacker.
|
|
|
77
121
|
|
|
78
122
|
## Feature tour
|
|
79
123
|
|
|
80
|
-
Every
|
|
81
|
-
|
|
124
|
+
Every feature below is explained in plain English: what it does, and how it
|
|
125
|
+
works. Most of it needs nothing more than a `pitstop scan` first.
|
|
82
126
|
|
|
83
127
|
### The scan
|
|
84
128
|
|
|
85
|
-
`pitstop scan`
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
129
|
+
`pitstop scan` runs every check at once, in parallel, and prints one box with a
|
|
130
|
+
single **OpenPitStop Score** (0 to 100, A to F). It looks at circular imports,
|
|
131
|
+
security issues, duplicated code, test results, build speed, accessibility and
|
|
132
|
+
code quality. Each check gives a real number or prints `skipped` with a hint on
|
|
133
|
+
how to install the tool it needs. It never makes up a number.
|
|
90
134
|
|
|
91
135
|
### Security fixes
|
|
92
136
|
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
<img src="docs/media/pitstop-security.gif" alt="pitstop scan — the indicated security fixes, each with a concrete fix" width="700">
|
|
98
|
-
</p>
|
|
137
|
+
Under the scan box, every security finding ships with a concrete `fix:` line, so
|
|
138
|
+
you get a worklist, not just a list of problems. Findings are labeled
|
|
139
|
+
`[indicated]` with the exact code and the fix. The full matrix is in
|
|
140
|
+
[docs/security.md](docs/security.md).
|
|
99
141
|
|
|
100
142
|
### Try it on your repo
|
|
101
143
|
|
|
102
|
-
`pitstop try .`
|
|
103
|
-
|
|
104
|
-
`npm i -g openpitstop` makes even that instant.)
|
|
105
|
-
|
|
106
|
-
<p align="center">
|
|
107
|
-
<img src="docs/media/pitstop-try.gif" alt="pitstop try — zero-setup score on any repo" width="700">
|
|
108
|
-
</p>
|
|
144
|
+
`pitstop try .` scores any repo in about two seconds, no setup, no config. It is
|
|
145
|
+
the fastest first look, and it seeds a baseline your later runs build on.
|
|
109
146
|
|
|
110
147
|
### The test pyramid
|
|
111
148
|
|
|
112
|
-
`pitstop test`
|
|
113
|
-
suite that
|
|
114
|
-
**DO NOT SHIP
|
|
115
|
-
|
|
116
|
-
<p align="center">
|
|
117
|
-
<img src="docs/media/pitstop-pyramid.gif" alt="pitstop test — the pyramid verdict: DO NOT SHIP on a failing e2e layer" width="700">
|
|
118
|
-
</p>
|
|
149
|
+
`pitstop test` runs your **unit, integration and e2e** layers separately, so a
|
|
150
|
+
suite that passes cannot hide a missing layer. It names every failing test. One
|
|
151
|
+
failing layer and it prints **DO NOT SHIP** and exits 1.
|
|
119
152
|
|
|
120
153
|
### The gate
|
|
121
154
|
|
|
122
|
-
`pitstop gate`
|
|
123
|
-
clean
|
|
124
|
-
|
|
125
|
-
<p align="center">
|
|
126
|
-
<img src="docs/media/pitstop-gate.gif" alt="pitstop gate — GATE FAIL on a confirmed cheat, exit 2" width="700">
|
|
127
|
-
</p>
|
|
155
|
+
`pitstop gate` is the contract for CI and pre-commit hooks. It checks the score,
|
|
156
|
+
regression risk and diff integrity, then exits `0` (clean), `1` (issues) or `2`
|
|
157
|
+
(confirmed cheat). The exit code is the truth a build can block on.
|
|
128
158
|
|
|
129
159
|
### Integrity
|
|
130
160
|
|
|
131
|
-
`pitstop integrity`
|
|
132
|
-
patterns: focused tests, deleted tests, rewritten tests
|
|
133
|
-
|
|
134
|
-
<p align="center">
|
|
135
|
-
<img src="docs/media/pitstop-integrity.gif" alt="pitstop integrity — CONFIRMED_CHEAT: test file deleted" width="700">
|
|
136
|
-
</p>
|
|
161
|
+
`pitstop integrity` diffs your change against the sealed baseline and hunts cheat
|
|
162
|
+
patterns: focused tests, deleted tests, rewritten tests, swallowed errors,
|
|
163
|
+
hardcoded-to-pass values. It exits `0/1/2` the same way.
|
|
137
164
|
|
|
138
165
|
### The pen test
|
|
139
166
|
|
|
140
|
-
`pitstop pen`
|
|
141
|
-
|
|
167
|
+
`pitstop pen` boots your app in a sandbox and fires real attack traffic, so a
|
|
168
|
+
finding is **PROVEN** by a live attack, not just guessed. With `--fix` it writes a
|
|
169
|
+
failing-first repro test and a safe patch. Nothing reaches the real network.
|
|
142
170
|
|
|
143
|
-
|
|
144
|
-
<img src="docs/media/pitstop-pen.gif" alt="pitstop pen — sandboxed attacks with runtime-proof verdicts" width="700">
|
|
145
|
-
</p>
|
|
171
|
+
### Drift (the permanent referee)
|
|
146
172
|
|
|
147
|
-
|
|
173
|
+
`pitstop pen` remembers. Every run seals its verdicts and compares them to the last one, so you
|
|
174
|
+
see exactly what changed between today and last week:
|
|
148
175
|
|
|
149
|
-
|
|
176
|
+
- **NEW** — a finding appeared (or escalated from indicated to proven). This is a regression, so the
|
|
177
|
+
gate exits `1` and your CI goes red.
|
|
178
|
+
- **RESOLVED** — a finding is gone because the fix worked. This is the "prove my fix" loop, and it is
|
|
179
|
+
the most satisfying thing here: run `pitstop repro <id>` to turn a finding into a failing test, ship
|
|
180
|
+
the patch, run `pitstop pen` again, and watch it flip to resolved.
|
|
181
|
+
- **ESCALATIONS** — something that was only *indicated* by static analysis is now *proven* by a live
|
|
182
|
+
attack.
|
|
150
183
|
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
184
|
+
Strix, the enterprise tool, runs a one-off scan. OpenPitStop keeps a running ledger of proof, so a
|
|
185
|
+
fix can never silently rot back into a bug.
|
|
186
|
+
|
|
187
|
+
### Honesty
|
|
188
|
+
|
|
189
|
+
`pitstop honesty` prints an honest self-assessment of what the tool cannot do, with
|
|
190
|
+
the evidence chain behind every number. No SaaS, no telemetry, no dashboard, no
|
|
191
|
+
fixing your code: it tells you its limits in plain words.
|
|
154
192
|
|
|
155
193
|
### Verify
|
|
156
194
|
|
|
157
|
-
`pitstop verify`
|
|
195
|
+
`pitstop verify` re-scans after a change and shows exactly how the score moved, and
|
|
196
|
+
it checks your diff for cheat patterns. The numbers cannot be argued with.
|
|
158
197
|
|
|
159
198
|
### Trends
|
|
160
199
|
|
|
161
|
-
`pitstop trends`
|
|
200
|
+
`pitstop trends` turns your saved scan history into per-category sparklines and a
|
|
201
|
+
score trend, so you can watch a repo actually improve over time.
|
|
162
202
|
|
|
163
203
|
### Inspect
|
|
164
204
|
|
|
165
|
-
`pitstop inspect <finding-id>`
|
|
205
|
+
`pitstop inspect <finding-id>` opens one finding: the code snippet, the root
|
|
206
|
+
cause, whether a repro test exists, and what OpenPitStop remembers about these
|
|
207
|
+
files.
|
|
166
208
|
|
|
167
209
|
### Repro
|
|
168
210
|
|
|
169
|
-
`pitstop repro <finding-id>`
|
|
211
|
+
`pitstop repro <finding-id>` turns any finding into a regression test that FAILS
|
|
212
|
+
while the bug is live and must PASS after the fix. Proof first, fix second.
|
|
170
213
|
|
|
171
214
|
### Report
|
|
172
215
|
|
|
173
|
-
`pitstop report --html`
|
|
216
|
+
`pitstop report --html` writes one self-contained HTML report, sealed with an
|
|
217
|
+
evidence signature, plus a README-ready score badge (`PITSTOP_BADGE.svg`).
|
|
174
218
|
|
|
175
219
|
### Share
|
|
176
220
|
|
|
177
|
-
`pitstop share`
|
|
221
|
+
`pitstop share` renders a single share card (score, trend, top findings) you can
|
|
222
|
+
screenshot and post, or paste into a PR.
|
|
178
223
|
|
|
179
224
|
### The live shield
|
|
180
225
|
|
|
181
|
-
`pitstop watch`
|
|
226
|
+
`pitstop watch` sits in a terminal and re-scans the moment you save a file, printing
|
|
227
|
+
the score delta so you see problems as you type.
|
|
182
228
|
|
|
183
|
-
###
|
|
229
|
+
### Drive the agent
|
|
184
230
|
|
|
185
|
-
`
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
](docs/github-action.md)
|
|
231
|
+
`pitstop drive <finding-id>` hands one finding to your own agent with explicit orders:
|
|
232
|
+
write the failing repro first, fix it, make the repro pass, then verify.
|
|
233
|
+
OpenPitStop referees the result and never edits your code.
|
|
189
234
|
|
|
190
|
-
### The
|
|
235
|
+
### The next step
|
|
191
236
|
|
|
192
|
-
`
|
|
193
|
-
|
|
194
|
-
commits → ](docs/caught-in-the-wild.md#bonus-the-same-catches-as-a-pre-commit-hook)
|
|
237
|
+
`pitstop next` reads the sealed artifacts and prints the single best next command plus
|
|
238
|
+
a checklist of everything still open, so you always know where you are.
|
|
195
239
|
|
|
196
|
-
|
|
240
|
+
### Ask in plain English
|
|
197
241
|
|
|
198
|
-
|
|
242
|
+
`pitstop ask "make this safe"` (or `/pitstop make this safe`) maps a plain-English
|
|
243
|
+
request to the right command. No need to memorize flags.
|
|
199
244
|
|
|
200
|
-
|
|
245
|
+
### Autopilot fix
|
|
201
246
|
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
```
|
|
247
|
+
`pitstop fix` chains **scan to pen --fix to verify to gate** and shows the `next` card
|
|
248
|
+
after each hop, so a clean repo is reachable without touching the agent.
|
|
205
249
|
|
|
206
|
-
|
|
207
|
-
failing suite green without fixing the bug — and the gate catches both attempts:
|
|
250
|
+
### Memory and budget
|
|
208
251
|
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
```
|
|
252
|
+
`pitstop memory` is a repo scratchpad for decisions and rejected approaches that
|
|
253
|
+
survive across sessions. `pitstop budget` shows the token and compute bill of your
|
|
254
|
+
scans and reproves, so a fix loop stays cheap.
|
|
213
255
|
|
|
214
|
-
|
|
215
|
-
of the registry. For a tight re-record, `node scripts/cheat-demo.cjs --fast --no-pitch`
|
|
216
|
-
reuses the cached `node_modules` (skips `npm install`) and ends the arc on the
|
|
217
|
-
CONFIRMED_CHEAT box — no pitch, no dead air.
|
|
256
|
+
### The slash command
|
|
218
257
|
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
ACT 3 agent deletes the test → GATE: CONFIRMED_CHEAT (exit 2) — blocked
|
|
223
|
-
(tamper-evident evidence chain verifies the whole way)
|
|
224
|
-
```
|
|
258
|
+
`/pitstop` in Claude Code, Cursor, OpenCode, Codex and more runs the full loop
|
|
259
|
+
immediately. `pitstop install` writes it into your tools; `pitstop prompt` shows the
|
|
260
|
+
exact prompt it expands to. See [Install](#install).
|
|
225
261
|
|
|
226
|
-
|
|
227
|
-
<img src="docs/media/pitstop-gate.gif" alt="The cheat-catch in the real TUI: the gate reads the diff against the sealed baseline and blocks a CONFIRMED_CHEAT with exit 2 — on real opencode output." width="780">
|
|
228
|
-
</p>
|
|
262
|
+
### The GitHub Action
|
|
229
263
|
|
|
230
|
-
|
|
231
|
-
|
|
264
|
+
`uses: openpitstop/action` (or `Krish-1507/OpenPitStop@main`) puts the gate on every PR
|
|
265
|
+
as a comment and a failing check when it matters. No wiring by hand. See
|
|
266
|
+
[docs/github-action.md](docs/github-action.md).
|
|
232
267
|
|
|
233
|
-
|
|
268
|
+
### The pre-commit hook
|
|
234
269
|
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
270
|
+
`npx openpitstop install --hooks` installs the gate one step earlier: the commit
|
|
271
|
+
cannot land until the gate passes. See
|
|
272
|
+
[docs/caught-in-the-wild.md](docs/caught-in-the-wild.md).
|
|
273
|
+
|
|
274
|
+
### Ledger mode (payment proof)
|
|
275
|
+
|
|
276
|
+
`pitstop scan --ledger` boots your app with every outbound HTTP call rerouted to a mock
|
|
277
|
+
gateway, then replays the classic payment bugs (duplicate webhook, concurrent
|
|
278
|
+
double-submit, delayed retry). If the mock shows more than one charge per idempotency
|
|
279
|
+
key, that is a **proven double-charge**, not a guess.
|
|
280
|
+
|
|
281
|
+
### CI reports
|
|
282
|
+
|
|
283
|
+
`pitstop ci` runs a CI-friendly scan plus verify against the base branch and writes a
|
|
284
|
+
PR-ready markdown report, the gate as a PR comment. This is the engine behind the
|
|
285
|
+
GitHub Action.
|
|
286
|
+
|
|
287
|
+
### Ready-check and doctor
|
|
238
288
|
|
|
239
|
-
|
|
240
|
-
|
|
289
|
+
`pitstop ready-check` answers "is it worth scanning again?" and reuses the baseline when
|
|
290
|
+
nothing changed. `pitstop doctor` explains why a category shows `skipped` and prints
|
|
291
|
+
copy-paste install hints for the tools you are missing.
|
|
241
292
|
|
|
242
|
-
|
|
243
|
-
|
|
293
|
+
### Digest (progress story)
|
|
294
|
+
|
|
295
|
+
`pitstop digest` turns your history into a plain-English progress story: how the score
|
|
296
|
+
moved, what got fixed, what regressed, and every cheat it caught.
|
|
244
297
|
|
|
245
298
|
---
|
|
246
299
|
|
|
@@ -253,9 +306,9 @@ npx openpitstop
|
|
|
253
306
|
```
|
|
254
307
|
|
|
255
308
|
No arguments needed: the CLI detects your AI tools, and asks what you want —
|
|
256
|
-
install `/pitstop` into them, score *this* repo (`try .`),
|
|
257
|
-
|
|
258
|
-
|
|
309
|
+
install `/pitstop` into them, or score *this* repo (`try .`). Pick, and it does it.
|
|
310
|
+
(In a non-interactive terminal it skips the questions and prints the one-line menu
|
|
311
|
+
instead.)
|
|
259
312
|
|
|
260
313
|
Or go straight to the files:
|
|
261
314
|
|
|
@@ -316,7 +369,6 @@ that:
|
|
|
316
369
|
| `/pitstop` (bare) | **default full loop** | Scans right away, prints the boxed report, one confirmation pause, then the autonomous fix loop — repeat until clean. |
|
|
317
370
|
| `/pitstop --menu` | menu | Prints the full mode list below and **waits** — handy if you forgot the flags. |
|
|
318
371
|
| `/pitstop --scan-only` | scan-only | Runs `openpitstop scan`, prints the entire boxed report verbatim, and stops — no fixes, no commentary. |
|
|
319
|
-
| `/pitstop --demo` | demo | Scaffolds OpenPitStop's seeded broken demo repo into a temp dir, then runs the default full loop there. |
|
|
320
372
|
| `/pitstop --ledger` | ledger | Runs `openpitstop scan --ledger` (boots the app with every outbound HTTP call intercepted and replays duplicate-webhook / double-submit / retry traffic), then runs the loop restricted to the payment findings. |
|
|
321
373
|
| `/pitstop --integrity-only` | integrity-only | Runs `openpitstop integrity`, prints the boxed verdict verbatim, and stops — no scanning, no fixes. |
|
|
322
374
|
| `/pitstop --pen` | pen | Live penetration test with proof — see [The pen test](#the-pen-test). |
|
|
@@ -327,9 +379,8 @@ For reference, `/pitstop --menu` shows this list:
|
|
|
327
379
|
```
|
|
328
380
|
OpenPitStop modes:
|
|
329
381
|
(enter) — full autonomous loop (scan, confirm, fix, verify, repeat)
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
--ledger — payment idempotency fuzzing only
|
|
382
|
+
--scan-only — scan and report, no fixes
|
|
383
|
+
--ledger — payment idempotency fuzzing only
|
|
333
384
|
--integrity-only — re-check the last commit for cheat patterns, no scanning
|
|
334
385
|
--pen — penetration test: live attacks + proof + fixes (regression tests, patches)
|
|
335
386
|
(your own ask) — reply with anything else, e.g. "check the security of this app"
|
|
@@ -339,28 +390,34 @@ A flag after `/pitstop` picks a specific mode; any free-form text after it becom
|
|
|
339
390
|
custom ask; bare `/pitstop` is the full loop. If a tool ever fails to substitute arguments,
|
|
340
391
|
`/pitstop` behaves as bare — the default full loop — rather than guessing.
|
|
341
392
|
|
|
342
|
-
No repo handy? `npx openpitstop@latest demo` scaffolds a broken demo repo in a temp dir so
|
|
343
|
-
you can watch the whole loop — self-contained, no installs on the hot path, and it never
|
|
344
|
-
writes into your tool configs (that stays an explicit `pitstop install`).
|
|
345
|
-
|
|
346
393
|
## Tool support
|
|
347
394
|
|
|
348
395
|
| Tool | Installed to | Status |
|
|
349
396
|
|------|--------------|--------|
|
|
350
|
-
| Claude Code | `.claude/commands/pitstop.md` (project + user), plus a Skill at `.claude/skills/pitstop/SKILL.md
|
|
397
|
+
| Claude Code | `.claude/commands/pitstop.md` (project + user), plus a Skill at `.claude/skills/pitstop/SKILL.md | Full support |
|
|
351
398
|
| Cursor | `.cursor/commands/pitstop.md` (project + user) | Full support |
|
|
352
399
|
| OpenCode | `.opencode/commands/pitstop.md` (project), `~/.config/opencode/commands/` (user) | Full support |
|
|
353
400
|
| Kilo Code | `.kilo/commands/pitstop.md` (project), `~/.config/kilo/commands/` (user) | Full support |
|
|
354
401
|
| Antigravity | `.agent/workflows/pitstop.md` (project + user) | Full support |
|
|
355
402
|
| Gemini CLI | `.gemini/commands/pitstop.toml` (project + user) | Full support |
|
|
356
403
|
| Codex CLI | `~/.codex/prompts/pitstop.md` | Full support |
|
|
404
|
+
| FreeBuff CLI | portable `pitstop.md` in your tool's commands folder | Full support (portable) |
|
|
405
|
+
| Grok Build CLI | portable `pitstop.md` in your tool's commands folder | Full support (portable) |
|
|
406
|
+
| MUSE Code CLI | portable `pitstop.md` in your tool's commands folder | Full support (portable) |
|
|
407
|
+
| Any other agent CLI | portable `pitstop.md` (drop it in the commands folder) | Full support (portable) — see below |
|
|
357
408
|
| Codex App / VS Code extension | — (no file written) | **Not supported** — OpenAI hasn't shipped custom slash commands there; install prints a manual-copy note instead |
|
|
358
409
|
| GitHub Action (PRs) | `uses: Krish-1507/OpenPitStop@main` | **Full support** — gate verdict as a PR comment + failing check; see [docs/github-action.md](docs/github-action.md) |
|
|
359
410
|
| git pre-commit hook | `.git/hooks/pre-commit` (installed with `--hooks`) | **Full support** — the gate blocks the commit before it lands |
|
|
360
411
|
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
|
|
412
|
+
**Works with any agent CLI.** OpenPitStop's `/pitstop` is a portable command file:
|
|
413
|
+
run `pitstop prompt` to print the exact instruction text, then paste it as a custom
|
|
414
|
+
slash command in any coding CLI that supports them (FreeBuff, Grok Build, MUSE Code and
|
|
415
|
+
others included above). The CLIs listed by name also get a dedicated path written
|
|
416
|
+
automatically by `pitstop install` when their commands-folder convention is known. Tell
|
|
417
|
+
us your CLI and we'll add it to the auto-install list. Legacy/alternate locations are
|
|
418
|
+
also written where tool docs are inconsistent across versions (see
|
|
419
|
+
`src/installer/targets.ts`). Existing files are never overwritten unless you pass
|
|
420
|
+
`-y`/`--force`; `npx openpitstop install --uninstall` removes everything.
|
|
364
421
|
|
|
365
422
|
## What OpenPitStop actually does
|
|
366
423
|
|
|
@@ -487,10 +544,9 @@ one-shot.
|
|
|
487
544
|
| Command | What it does |
|
|
488
545
|
|---|---|
|
|
489
546
|
| `pitstop install` / `install --uninstall` | Writes `/pitstop` into every supported tool (project + user level). `--uninstall` removes it all. `--hooks` also installs (or with `--uninstall`, removes) the git pre-commit gate. |
|
|
490
|
-
| `pitstop` (no args) | The guided first-run: detects your AI tools and git repo, then offers to install
|
|
547
|
+
| `pitstop` (no args) | The guided first-run: detects your AI tools and git repo, then offers to install or score this repo (`try .`). Non-TTY prints the one-line menu instead. |
|
|
491
548
|
| `pitstop doctor` | Explains why categories show `skipped`: checks your toolchain (Node, git, jscpd, gitleaks, semgrep, pa11y) and prints copy-paste install hints. |
|
|
492
549
|
| `pitstop prompt [--args …]` | Prints the exact prompt your AI tool expands `/pitstop` into, with your arguments filled in — full transparency into what the agent was told. |
|
|
493
|
-
| `pitstop demo [demo]` | Scaffolds an intentionally-broken demo repo into a fresh temp dir (`demo-repo`, `demo-repo-integrity`, `demo-repo-fintech`, `demo-repo-generators`), initializes git, and scans it immediately. |
|
|
494
550
|
|
|
495
551
|
### The score & badge
|
|
496
552
|
|
|
@@ -536,8 +592,7 @@ shows zero clusters (hard limits: 10 fix rounds or 45 minutes), ending with a
|
|
|
536
592
|
gateway**, then replays the three classic payment bugs: duplicate webhook, concurrent
|
|
537
593
|
double-submit, delayed retry. If the mock gateway's own receipt log shows more than one
|
|
538
594
|
charge per idempotency key, that's a **proven double-charge** — not a guess. The shipped
|
|
539
|
-
|
|
540
|
-
webhook endpoints have no idempotency guard. If the sandbox can't intercept some traffic,
|
|
595
|
+
If the sandbox can't intercept some traffic,
|
|
541
596
|
the run aborts (`exit 77`); nothing ever reaches a real gateway.
|
|
542
597
|
|
|
543
598
|
**Which stacks are covered?** Node/JS apps run under the nock preload, which intercepts
|
|
@@ -561,14 +616,6 @@ value edited to match the buggy output. A caught cheat looks like this: change
|
|
|
561
616
|
`CONFIRMED_CHEAT`, the change is blocked, and a human reviews it (verified against
|
|
562
617
|
`fixtures/assertion-literal-tamper/`). An honest app-side fix sails through `CLEAN`.
|
|
563
618
|
|
|
564
|
-
### Cheat-catch demo
|
|
565
|
-
|
|
566
|
-
Want to *see* it? The scripted arc from **[See it in 90 seconds](#see-it-in-90-seconds)**
|
|
567
|
-
is `scripts/cheat-demo.cjs` — a fully deterministic SUSPICIOUS → CONFIRMED_CHEAT
|
|
568
|
-
sequence against a real repo with a real failing jest test. Point it at a build with
|
|
569
|
-
`PITSTOP_CLI="node /path/to/dist/cli.js"`, or let it use `npx openpitstop`. Great for a
|
|
570
|
-
video or a live judge's demo.
|
|
571
|
-
|
|
572
619
|
## Architecture
|
|
573
620
|
|
|
574
621
|
OpenPitStop is two pieces that never mix: a **CLI that measures**, and **your host agent that
|
|
@@ -580,14 +627,12 @@ cheat its own referee. That separation is the product.
|
|
|
580
627
|
|
|
581
628
|
## Known limitations
|
|
582
629
|
|
|
583
|
-
- **Windows** is CI-verified
|
|
584
|
-
`windows-latest`
|
|
585
|
-
Windows host
|
|
586
|
-
|
|
587
|
-
|
|
588
|
-
|
|
589
|
-
caveat is breadth, not correctness: not every exotic repo shape has been manually
|
|
590
|
-
exercised on Windows.
|
|
630
|
+
- **Windows** is a first-class, CI-verified platform (build + smoke on `ubuntu-latest`
|
|
631
|
+
and `windows-latest` every push). `watch`, `pen`, `pen --fix` and `scan --ledger` were
|
|
632
|
+
each run end-to-end on a real Windows host against the demo repos, with a live watch
|
|
633
|
+
delta, PROVEN ledger double-charges (sealed evidence) and honest pen verdicts
|
|
634
|
+
(including the honest "0 patches" case) all verified. The only open caveat is breadth,
|
|
635
|
+
not correctness: not every exotic repo shape has been hand-exercised on Windows yet.
|
|
591
636
|
- **Codex App / VS Code extension** isn't supported and won't be until OpenAI ships custom
|
|
592
637
|
slash commands; use Codex CLI for `/pitstop`.
|
|
593
638
|
- **Graceful degradation:** duplication (`jscpd`), secrets/code scanning (`gitleaks`,
|
|
@@ -638,6 +683,17 @@ how to open a PR. For the launch notes and the "why", read [LAUNCH.md](LAUNCH.md
|
|
|
638
683
|
|
|
639
684
|
[MIT](LICENSE)
|
|
640
685
|
|
|
686
|
+
## Support the project
|
|
687
|
+
|
|
688
|
+
If OpenPitStop saved you from shipping a bug your agent swore was fixed, the best
|
|
689
|
+
support is a star and a real repo:
|
|
690
|
+
|
|
691
|
+
- **Star** it: https://github.com/Krish-1507/OpenPitStop
|
|
692
|
+
- **Report issues or ideas**: https://github.com/Krish-1507/OpenPitStop/issues
|
|
693
|
+
- **Contribute** an analyzer (small, well-scoped): see [CONTRIBUTING.md](CONTRIBUTING.md)
|
|
694
|
+
|
|
695
|
+
No donation, no paywall, no telemetry.
|
|
696
|
+
|
|
641
697
|
---
|
|
642
698
|
|
|
643
699
|
<p align="center">
|
package/demo/DEMO_SCRIPT.md
CHANGED
|
@@ -93,6 +93,9 @@ A sampling of the 31 symptoms PitStop flags in MiniShop:
|
|
|
93
93
|
| database | superuser account, `GRANT ALL`, TLS disabled | `src/db.js` |
|
|
94
94
|
| authentication | `password === stored`, `Math.random()` token, `alg:none` JWT, insecure cookie | `src/auth.js` |
|
|
95
95
|
| hidden-vuln | security `TODO`/`FIXME` comments | several files |
|
|
96
|
+
| price-tampering | checkout totals from CLIENT-supplied prices | `src/server.js` |
|
|
97
|
+
| idor | `/api/orders/:id` returns any order, no ownership check | `src/server.js` |
|
|
98
|
+
| prototype-pollution | `lodash.merge` sinks an untrusted request body | `src/server.js` |
|
|
96
99
|
|
|
97
100
|
The scan also **seals a baseline** — the signed "before" state every later
|
|
98
101
|
check is verified against.
|
|
@@ -169,6 +172,38 @@ to your agent. The agent produces exactly the changes in
|
|
|
169
172
|
|
|
170
173
|
---
|
|
171
174
|
|
|
175
|
+
## 4b. Drift — prove the fix stuck (the permanent referee) — `pitstop pen`
|
|
176
|
+
|
|
177
|
+
Run `pen` again, this time on the *fixed* app:
|
|
178
|
+
|
|
179
|
+
```bash
|
|
180
|
+
cd demo/web-app-fixed
|
|
181
|
+
pitstop pen
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
This run is compared against the last sealed pen run from `web-app-broken`. You
|
|
185
|
+
see the delta, not just a fresh score:
|
|
186
|
+
|
|
187
|
+
```
|
|
188
|
+
PEN DRIFT vs .pitstop/pen-latest.json
|
|
189
|
+
◐ RESOLVED pen-… : command-injection (was PROVEN, now gone)
|
|
190
|
+
◐ RESOLVED pen-… : sql-injection
|
|
191
|
+
(+ any still-open classes, and ESCALATIONS if a static
|
|
192
|
+
indication became a live proof)
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
**Why this matters on camera:** most scanners give you a one-off report and walk
|
|
196
|
+
away. OpenPitStop keeps a running ledger of proof. A finding that flips from
|
|
197
|
+
PROVEN to RESOLVED is evidence the fix worked, not hope. And if someone
|
|
198
|
+
reintroduces the bug later, the next `pen` run prints `+1 NEW` and the gate
|
|
199
|
+
exits `1`, so the regression can't sneak into main.
|
|
200
|
+
|
|
201
|
+
**On-camera:** "Most scanners hand you a PDF and leave. OpenPitStop remembers.
|
|
202
|
+
Watch this finding flip from proven to resolved, and if it ever comes back, the
|
|
203
|
+
gate goes red. That's a referee, not a report."
|
|
204
|
+
|
|
205
|
+
---
|
|
206
|
+
|
|
172
207
|
## 5. Verify it didn't fake it — `pitstop verify`
|
|
173
208
|
|
|
174
209
|
```bash
|
|
@@ -266,3 +301,49 @@ straight to a finding's file:line + fix.
|
|
|
266
301
|
> tamper-evident baselines, runtime penetration tests with failing-first
|
|
267
302
|
> regression contracts, and a verification gate that catches the agent lying.
|
|
268
303
|
> The CLI measures; the agent edits; the numbers can't be cheated.
|
|
304
|
+
|
|
305
|
+
---
|
|
306
|
+
|
|
307
|
+
## 11. Fix it yourself — the full `pitstop` command sequence (cheat sheet)
|
|
308
|
+
|
|
309
|
+
Everything you type to take `web-app-broken` from **35/100 (F)** to
|
|
310
|
+
**VERIFIED**, end to end. Copy-paste top to bottom.
|
|
311
|
+
|
|
312
|
+
```bash
|
|
313
|
+
# 0. baseline — confirm it's broken
|
|
314
|
+
cd demo/web-app-broken
|
|
315
|
+
npm install
|
|
316
|
+
npm test # 5 failing — the app is genuinely broken
|
|
317
|
+
npx openpitstop install -y # register the /pitstop command (one time)
|
|
318
|
+
|
|
319
|
+
# 1. honest baseline — seals the signed "before" state every later check uses
|
|
320
|
+
pitstop scan
|
|
321
|
+
|
|
322
|
+
# 2. live pen test + safe patches + failing-first repro tests
|
|
323
|
+
pitstop pen --fix # writes repro tests + deterministic patches
|
|
324
|
+
pitstop repro SEC-001 # watch a real bug FAIL on camera
|
|
325
|
+
|
|
326
|
+
# 3a. drive the agent to fix each root cause, one cluster at a time
|
|
327
|
+
pitstop next # prints the single best next id + the open checklist
|
|
328
|
+
pitstop drive <id> # fix one root cause; repeat until the checklist clears
|
|
329
|
+
|
|
330
|
+
# 3b. ...or skip the manual loop and let autopilot run scan -> pen -> verify -> gate
|
|
331
|
+
pitstop fix
|
|
332
|
+
|
|
333
|
+
# 4. prove the fix stuck (the permanent referee)
|
|
334
|
+
pitstop verify # exit 0 = VERIFIED; agent-cheat patterns caught
|
|
335
|
+
pitstop pen # drift: RESOLVED findings vs the broken baseline
|
|
336
|
+
|
|
337
|
+
# 5. gate + shareable proof
|
|
338
|
+
pitstop gate --score 60 # 1 before the fix, 0 after
|
|
339
|
+
pitstop honesty # trace every number to a sealed .pitstop/ file
|
|
340
|
+
pitstop report # shareable HTML/markdown card
|
|
341
|
+
pitstop ci # drop the same gate into CI
|
|
342
|
+
|
|
343
|
+
# 6. ship only clean code
|
|
344
|
+
git add -A && git commit -m "fix: remediate MiniShop with OpenPitStop"
|
|
345
|
+
```
|
|
346
|
+
|
|
347
|
+
The fixed result is already committed for reference in
|
|
348
|
+
[`web-app-fixed/`](./web-app-fixed) — `cd demo/web-app-fixed && DB_PASSWORD=dummy
|
|
349
|
+
JWT_SECRET=dummy npm test` shows all 5 passing.
|